Evals are the product spec for AI features

Traditional product specs describe behaviour: “when the user clicks X, Y happens.” That breaks down when Y is a model’s answer. The same input can produce five acceptable outputs and three embarrassing ones, and no sentence in a PRD will tell your team which is which.

So I stopped writing specs for AI features and started writing evals.

What an eval set actually is

An eval set is a list of real inputs paired with a definition of “good.” Not a vibe. A definition someone else on the team could apply and get the same answer.

My minimum viable version:

1. 30 to 50 real inputs, pulled from logs, support tickets, or interviews. Not invented ones.
2. A rubric with three to five yes/no questions per input.
3. A named owner who can say “this case matters” or “drop it.”

The shape of a good rubric

Rubrics fail when they are too abstract. “Is the answer helpful?” gets you arguments. “Does the answer cite a source from the provided documents?” gets you a number.

type EvalCase = {
  input: string;
  checks: { id: string; question: string }[];
};

const refundCase: EvalCase = {
  input: 'I was charged twice for my subscription',
  checks: [
    { id: 'cites-policy', question: 'Does it reference the 14-day refund policy?' },
    { id: 'no-promise', question: 'Does it avoid promising a refund it cannot grant?' },
    { id: 'next-step', question: 'Does it give exactly one clear next step?' },
  ],
};

Why this belongs to the PM

Engineers can run evals. Only the product owner can decide what counts as a failure. If that judgement lives in a developer’s head, every model upgrade becomes a negotiation.

Ship the eval before the feature

I now treat “eval set exists and the baseline scores are known” as a launch gate, the same way I once treated “designs approved.” It forces the hard conversation early, when changing direction is cheap.

If you take one thing from this post: write down what good looks like before the demo makes everything look good.