The risk you own
There is a sentence that has never worked, in any forum, in any industry: "the AI made an error." Not with a regulator, not with a customer, not in a tribunal, not with your own board. Understanding precisely why it does not work is what makes the rest of this lesson obvious.
Accountability does not transfer
When your team sends a customer a letter, your organisation said it. The mechanism that produced the draft is an internal implementation detail with no standing outside the building.
This has three practical consequences worth stating plainly.
A vendor's liability cap is not your exposure. Read one: it is typically a refund of fees paid, sometimes twelve months' worth. Your exposure to a mis-handled claim is not bounded by what you paid for the software.
"The model did it" is not a defence and not an explanation. What a regulator or a court asks is what process you had, whether it was reasonable, and whether a competent person could have caught it. Those are questions about your design, not the vendor's.
Being unable to explain a decision is itself a finding. In a growing number of jurisdictions, and already in most financial services regulation, a decision that materially affects a person must be explicable. "The system scored it" does not meet that bar in any regime worth naming.
The three questions per task
Before any AI-touched process goes live, three questions. They take five minutes and they determine your design.
Does this affect a person's money, health, employment, or legal position? If yes, a named human makes the decision. The AI may inform, summarise, or draft. It does not decide, and the record must show a person deciding.
Would we be able to explain a specific instance in six months? You need enough of a record to reconstruct one case: what came in, what the system produced, what the person did. Not a research-grade audit trail — a record that answers "what happened on the 14th".
What is the worst realistic outcome, and who absorbs it? If the answer is "a customer", the checkpoint goes before the customer sees anything.
Where the human checkpoint belongs
Before anything irreversible reaches a customer. A draft letter reviewed before sending is safe; an auto-sent letter is a published statement by your company.
At the decision, not after it. Reviewing a decision already communicated is not a control — it is an apology process. The person must be able to change the outcome.
Where the person can actually see the error. This is the design point most often missed. A reviewer shown a summary and asked to approve it, without access to the source, cannot detect the failure that matters — and a checkpoint that cannot detect the error it exists to catch is worse than none, because it manufactures the appearance of control.
Kavita's flag on files referencing prior claims is exactly this principle: the 15% where the summary can be misleading are the 15% where the reviewer is sent to the source.
The rubber-stamp problem
A checkpoint where the person approves ninety-nine of a hundred without changing anything has stopped being a control, whatever the process document says. This is the same vigilance decay from lesson 7, seen from the governance side, and it is what makes many human-in-the-loop designs fail their first serious audit.
Three things help. Sample deliberately — pull a random sample for genuine deep review rather than relying on the pass-through check. Track the change rate — if the proportion of outputs the checker modifies falls toward zero, either the system improved or the checking stopped, and you need to know which. Rotate, so nobody spends six months approving the same thing.
Do not automate what you cannot explain
The most common serious mistake in this area is not deploying something dangerous. It is deploying something opaque into a process that was previously explicable.
Your claims process today can be described: a person reads, checks these terms, applies this rule. If an AI system is inserted and the description becomes "and then it produces a recommendation", you have lost something you will be asked for later — by a regulator, by a customer's lawyer, or by your own operations team trying to work out why last quarter looks strange.
The test is simple and worth applying to every proposal: could someone in your team, in a meeting, explain to a customer why their case came out the way it did? If not, the design needs changing before the technology does.
Do this today: take the process you are most likely to deploy into and write down what record would exist if someone asked you about one specific case from three months ago. If there is no answer, that is your first requirement.