Anyone can ship an agent that returns an answer. You can wire a model, a couple of tools, and a prompt in an afternoon and get a demo that looks great. The trap is that a demo tells you nothing about whether the answers are actually right.
So now, before I build the agent, I build the test. I hand label a small set of real cases where I already know the correct answer, because I checked each one myself. Fifty is plenty to start. That set becomes the gate: a new version has to beat the old one on it, or it does not ship.
The first time you do this it feels slow and a little pointless. Then it catches a 'better' version that was quietly worse on a whole class of inputs, and you stop arguing with it.
The payoff is that you can finally say one version is better than another instead of feeling like it is. Without the eval you are shipping vibes. With it, 'more accurate' means something.