I had a step that needed a yes or no judgment on some messy data, and my first instinct was to hand it to an LLM. It felt like the modern, obvious choice.
On a lark I wrote the dumb version first: a plain set of rules, no model, a few lines of logic. Then I ran both against a labeled set. The rules won. They were more accurate, ran instantly, and cost nothing per call. The model was confidently wrong often enough to lose.
A model is the right tool when the problem is genuinely fuzzy and language heavy. A lot of the steps we reach for it on are not that. They are rules we did not feel like writing.
Now I write the deterministic version first and only bring in a model when the simple thing actually falls short on the eval. Reach for the model last, not first. It is usually cheaper and often better.