Most AI features that disappoint were not badly built. They were badly scoped. Someone said “let’s add AI to the dashboard,” a prototype wowed the room, and six weeks later nobody could say whether the feature was good, because nobody had defined what good meant.
This is the checklist I now run before any engineer opens an editor. It takes about an hour, and it has saved me weeks.
Start with the job, not the model
Before you discuss prompts, models or vendors, write one sentence: who is trying to get what done, and what do they do today instead?
“Summarise support tickets” is a capability. “Help a support lead decide which three tickets to escalate before the 9am standup” is a job. The second tells you who the user is, how much time they have, and what a wrong answer costs them.
Three questions that sharpen the job
- What happens if the feature does nothing? If the answer is “users carry on fine,” you may be solving a problem nobody has.
- What is the current workaround? A spreadsheet, a colleague, a long search. The workaround sets the bar you must beat.
- Who reviews the output? If a human always checks it, you can tolerate more mistakes. If the output goes straight to a customer, you cannot.
Decide how good is good enough
Traditional features either work or they don’t. AI features work some of the time, and your job is to decide what “some” means before you see a demo.
Write the quality bar as a sentence you can test:
For a typical ticket, the summary names the customer’s problem and the product area, and is wrong or misleading in no more than 1 of 20 cases.
That sentence has a task, a standard and a tolerance. Without all three you will argue about opinions in the launch meeting.
Pick the failure you can live with
Every AI feature fails in two directions. A spam filter can let spam through or block real mail. A summary can miss a detail or invent one. Decide which direction is worse for your users, and tune toward the other.
For most business tools, a confident wrong answer is worse than “I’m not sure.” Say that out loud early, because it changes the design: you will want confidence signals, citations or an easy way to see the source.
Build the eval set first
You cannot judge quality by trying a few examples and nodding. Collect a small, honest set of real inputs, with the output a good human would produce for each.
- Gather 30 to 50 real cases from logs, tickets or interviews. Not invented ones.
- Include the ugly ones. Typos, missing context, angry customers, mixed languages.
- Write down what good looks like for each case, or at least a short rubric.
- Keep it versioned. When the prompt or model changes, rerun the same set.
This eval set is the closest thing an AI feature has to a spec. If you want more on this, I wrote about it in Evals are the product spec for AI features.
Plan for the unhappy paths
The demo shows the happy path. Production is mostly everything else. Before building, list what the user sees when:
- The model is slow or times out.
- The answer is low quality and the system knows it.
- The input is empty, enormous or in the wrong language.
- The provider has an outage.
- The user simply disagrees with the answer.
For each one, decide the fallback. Often the best fallback is the old, boring flow: show the original ticket, the manual search box, the plain table. An AI feature that degrades gracefully feels reliable. One that breaks into a blank panel feels broken.
Give users a way to push back
A thumbs-down button is the cheapest quality signal you will ever collect. Pair it with a free-text field and a way to see what the model saw. Those reports become the next batch of eval cases.
Know your costs and limits
AI features have a marginal cost that most software does not. Every use costs money and takes time, and both vary with input size.
Estimate three numbers on a napkin:
| Question | Why it matters |
|---|---|
| Cost per use at typical input size | Decides whether free tiers are safe |
| Response time at the 95th percentile | Decides whether it feels instant or broken |
| Uses per user per day | Turns cost per use into a monthly bill |
If the napkin math is scary, you have found a scoping decision early: shorter inputs, caching, a smaller model for easy cases, or a paid tier.
Decide what you will measure after launch
Pick two or three numbers before launch, and write down what you will do if they move the wrong way.
- Adoption: do people use it a second time?
- Outcome: does the job get done faster or better than the workaround?
- Quality: what share of outputs are accepted without edits?
Avoid vanity numbers like “requests sent to the model.” They go up when the feature is annoying as well as when it is useful.
Also decide the kill criteria. “If fewer than 20 percent of users return in week two, we simplify or remove it.” Writing that line while you are still optimistic makes it much easier to act on later.
The one-page scope doc
Everything above fits on a single page. Mine has six headings:
- The job (one sentence)
- The quality bar (one testable sentence)
- The worse failure and how we handle it
- The eval set (where it lives, how big)
- Fallbacks for the unhappy paths
- Cost, speed and the numbers we will watch
If you cannot fill in a heading, that is the conversation to have before building, not after. The prototype will always look good. The page is what tells you whether the feature is worth shipping.