Trust Is Recoverability: Error States for Probabilistic Systems
Nov 2026What the AI design guidebooks say about errors, and why I think undo should ship before accuracy improvements.
Why this is on my mind
Every AI product I've built has an error state problem that traditional apps don't. In a normal app, an error means something broke. In an AI product, a wrong answer can look exactly like a right one. The screen gives no signal.
What's true right now
The guidance is explicit about errors being normal. Google's People + AI Guidebook has a whole chapter on errors and graceful failure. It says errors can be opportunities: they support learning by experimentation, help establish correct mental models, and encourage feedback. It also asks designers to define what counts as an "error" versus a "failure," to identify user, system, and context errors, and to plan for fail states where the system can't give a good answer at all.
The distinction has teeth. One summary gives the example that a self-driving car stopping at a green light might be an error, while running a red light is a failure. Different severities need different interfaces.
Testing for failure is part of the toolkit. Microsoft's HAX Toolkit includes a playbook that helps teams generate scenarios to test based on likely human-AI interaction failures, such as a transcription error or a false positive, as AI Magazine described the toolkit.
The unhappy path is where products break. One studio argues that products fail at fallback design: they ship the generation capability before building the layer that handles outputs a user can't act on.
High stakes change the interface itself. A piece on agent-assembled interfaces argues that when an agent is about to trigger a database migration or take an irreversible action, the interface must shift modes, from approachable assistant to something closer to a compliance officer.
What I notice
Almost everything on that list is about what happens after the system is wrong. Very little is about making it less likely to be wrong. That isn't an accident. Accuracy is a model problem, and designers can't move it quickly. Recovery is a design problem, and designers can.
A worked example of a receipt state
Illustrative: my own sketch, not a sourced fact.
Picture an agent that changes a batch of product listings. Today it says "Done" and the user hopes for the best. A receipt state would instead say what changed, how many items it touched, and why it chose them, in plain language. It would also show an undo that stays available for a limited time. Imagine the same product with a bare "Done" message after every action. The user has to trust the system blindly, and any later surprise lands on the whole product, not on one step.
Why this is the trust artifact. The receipt does two jobs. It lets the user check the work without redoing it, and it makes a mistake cheap to reverse. Neither job requires the model to be more accurate. That is the point of my hypothesis: recovery is something a designer can ship now. I would also want to test whether people actually read it. A receipt nobody opens is just decoration, so the first version should be short enough to take in at a glance.
Where I would not trust this
My opinion, based on the tradeoffs above.
Undo that arrives too late. A time-limited undo only helps if the user notices the mistake inside the window. In a flow where errors look exactly like right answers, they may not. So the receipt has to make errors easier to spot, not only easier to reverse. It also means the receipt and the undo should sit close together. A user who spots a problem should not have to hunt for the way back while the window is closing.
Actions that cannot be undone. Some actions have no way back. For those, I would not lean on recovery at all. This is where the shift toward a compliance-officer mode matters: preview first, confirm clearly, and slow the interface down on purpose. Recovery-first thinking is for the reversible majority, not for everything. That is why I treat this as a boundary on my own hypothesis. It holds well for the common case of small, reversible changes, and less well where one mistake carries real consequences for someone.
My hypothesis
This section is my opinion, not a sourced fact.
Users' trust in an AI product is driven more by how easily they can recover from a mistake than by how rarely it makes one.
If that's right, then the right sequence for a team is: ship undo, preview, and visible change history first, then work on accuracy. A product that is 90% accurate with one-click recovery should feel safer than one that is 97% accurate with no way back, because the cost of the 3% is so different.
It also suggests a practical rule for irreversible actions: they get a "receipt" state afterward, with a time-limited undo and a plain-language record of what happened and why. That receipt is the trust artifact.
How I'd test it
Build two versions of one workflow: higher accuracy without undo, lower accuracy with undo and preview. Seed errors at known rates. Measure willingness to delegate a real task to each after three sessions. My bet is the recoverable one wins at moderate accuracy gaps.
What would change my mind
If users in high-stakes domains weigh accuracy overwhelmingly (where even one mistake is unacceptable regardless of undo), then recoverability matters less there, and prevention is the real priority.
What I would do first on a real product
My opinion, a suggested starting order.
List every action by reversibility. I would walk through the product and sort each action into two piles: can be undone and cannot. It is an afternoon of work, and it usually shows how many actions are quietly irreversible. That list decides where previews and receipts go first. It is an unglamorous exercise, but it changes the roadmap. Teams often discover that the actions they thought were safe are the ones with no way back, and those are the ones that need attention first.
Then ship the cheapest recovery. For the reversible pile, I would add undo and visible change history before touching accuracy. For the irreversible pile, I would add a preview and a plain confirmation. Only after that would I ask the model team what accuracy improvements are possible, because the safety net will already be in place. The order matters because it protects users while the model is still improving. If accuracy rises later, the product is better still, and if it does not, people are not left exposed in the meantime.
References
- Google PAIR, People + AI Guidebook: Errors + Graceful Failure
- Daniel Bourke, How the biggest companies in the world design ML-powered applications
- Microsoft Research, Guidelines for Human-AI Interaction
- Fuselab Creative, Generative UI design
- AI for Lifelong Learners, The interface that dissolves when you're done with it