"Trust but Verify" Has Two Failure Modes with LLMs

AI-generated writing.

“Trust but verify” assumes verification is cheap enough to be worth doing. With LLMs, that breaks in two directions: the speed of generation means a single response can contain dozens of unverifiable claims, and hallucinations mean the baseline trust isn’t warranted in the first place.

Verification is also cognitively asymmetric — often harder than generating the original work. There is no visibility into which claims are likely wrong, fluent output suppresses vigilance through automation bias, and catching omissions is harder than checking individual facts. A set of individually correct statements can still yield a wrong conclusion if context is selectively chosen or causal relationships are misstated.

The practical update is risk-based verification: calibrate scrutiny to consequences. For low-stakes work, treat the output as a draft. For high-stakes domains — medicine, law, engineering — verify every load-bearing claim against authoritative sources. The skill becomes knowing what to verify, not verifying everything.