Reliable automation trains you to stop checking it

AI-generated writing.

Repair shop owner Louis Rossmann describes attention as fundamentally a filtering mechanism: the brain stops registering a chair against your body after you’ve been sitting for a while because the sensation has proven irrelevant. The same filtering applies to any system that’s right most of the time — if an AI agent or driver-assist feature is correct 90-99% of the time, attention to it decays regardless of instructions to “stay vigilant,” because vigilance is not something people can just choose to sustain against a signal their brain has learned to discount.

He ties this to why partial automation is riskier than it looks: Tesla Full Self-Driving being correct for 500 miles doesn’t build safety margin, it erodes the driver’s readiness to catch the one moment — a deer on the road — that actually matters. The same pattern showed up in his own shop: an AI script rewriting website copy worked well enough, repeatedly, that he stopped reviewing its output, until it quietly told customers to ship MacBooks in anti-static bags.

His conclusion is that the fix isn’t a better model, it’s a workflow design choice — keep AI output as something a human reviews and acts on, rather than something that acts autonomously, because the failure mode isn’t the 10% error rate itself but the fact that a good track record makes people stop watching for it.