Why AI products don't behave like normal software

The interesting thing about building products with AI isn’t that the models are getting smarter. It’s that the relationship between software and the people using it is changing — and most of our product habits were formed for the old relationship.

Traditional software is a contract. You press the button, the same thing happens every time. Product management grew up around that promise: requirements describe behaviour, QA confirms it, and a bug is any gap between the two.

A model-backed feature breaks the contract on purpose. The same input can produce different outputs. “Correct” becomes a distribution rather than a value. And the most useful behaviour is often the behaviour nobody wrote down.

Requirements become ranges

When I write a spec for an AI feature now, I spend less time describing what it should do and more time describing what it must never do, what “good enough” looks like, and how we’ll know when it drifts.

The spec stops being a description of the product and becomes a description of its boundaries.

In practice that means acceptance criteria turn into evaluation sets. A small, curated list of real inputs — with the outputs we’d be proud of — does more work than a page of prose.

# evals/refund-intent.yaml
- input: "charged twice for march"
  expect: intent == "refund"
  must_not: promise_amount
- input: "can I pause instead?"
  expect: intent == "retention"

Shipping is the start of the experiment

The second shift is temporal. With deterministic software, launch is a finish line. With AI products, launch is when you finally get the data you needed to design the thing properly. The best teams I’ve worked with plan for that explicitly: they budget the weeks after release as carefully as the weeks before it.

None of this makes product management harder, exactly. It makes it more honest. The uncertainty was always there — the models just stopped letting us pretend.