A batch finished and the dashboard said the success rate was basically perfect. That is exactly the moment to get suspicious, not relieved.
I pulled the raw rows and recomputed it by hand. A big chunk of the 'successes' were empty results that got counted as wins because the code checked 'did it run' instead of 'did it return something real'. The real number was a lot lower, and a lot more useful.
This pattern shows up everywhere: empty results counted as passes, coverage mistaken for accuracy, a flattering match rate nobody re-derived, an average hiding a broken segment. A number that makes everyone comfortable is the one most likely to be wrong, because nobody wants to poke it.
The habit that saved me: when a metric looks too good, recompute it from the raw data before you repeat it to anyone. The honest number is sometimes worse in the moment and always better in the long run.