What the claims actually mean
"95% accurate." "Handles 80% of queries." "Best-in-class on industry benchmarks." You will read these on every website in the category, and each is meaningless until you have asked two or three follow-up questions. This lesson is those questions, and the translation into the only number you care about — how many wrong answers a day your team will absorb.
Accuracy is not one number
"95% accurate" is incomplete in a way that matters enormously, because errors come in two flavours with completely different costs.
Imagine a system that flags incoming complaints as urgent. Two ways it can be wrong: it flags something that was not urgent, or it misses something that was.
Of the things it flagged, how many really were urgent? That is precision, and a low number means your team wastes effort on false alarms and eventually stops trusting the flag — which is the real failure, because an ignored flag is worse than no flag.
Of the things that really were urgent, how many did it catch? That is recall, and a low number means genuinely urgent complaints sit in the ordinary queue. In insurance, that is a regulatory conversation.
The two trade against each other. Any system can be tuned to catch everything at the cost of flagging half the queue, or to flag only the certain ones at the cost of missing others. A single "accuracy" figure hides which choice was made, and the choice is a business decision that belongs to you, not to a vendor's default setting.
So: "What are precision and recall, and can we tune the balance?" If the person cannot answer, ask for someone who can.
The base-rate trap
This one catches experienced managers, and it is worth working through once.
Suppose two per cent of claims are fraudulent, and a vendor's detector is "99% accurate" — meaning it correctly identifies 99% of fraudulent claims and correctly clears 99% of legitimate ones. That sounds excellent.
Run it on 10,000 claims. 200 are fraudulent; it catches 198. Of the 9,800 legitimate ones, it wrongly flags 1% — that is 98 claims.
So it produces 296 flags, of which 198 are real. A third of everything your investigators look at is a false alarm, and each one is a legitimate customer whose claim was delayed and who may have been treated as a suspect.
Nothing was misrepresented. The number was accurate. The consequence is a workload and a customer-experience problem nobody mentioned, and it is entirely predictable from the base rate — which is why the question to ask about any detector is: how common is the thing you are detecting, and what does the flag queue look like at our volume?
"Handles 80% of queries"
Handles means what, exactly? Produced a response? Produced a response the customer accepted? Resolved without a human? Did not escalate within 24 hours?
These differ by a lot. A chatbot that "handles" 80% by responding to everything and deflecting is the deployment pattern that has made automated support widely hated, and it moves cost from the company to the customer rather than removing it.
The question that cuts through: "What percentage of interactions ended with the customer's problem solved and no human involved, measured how?"
Benchmarks
Vendors cite model benchmark scores. These measure general capability on standardised tasks and are close to irrelevant to whether a product works on your claims files.
They are not useless — a much weaker model will do worse at most things. But the gap between two frontier models on a benchmark is smaller than the gap made by how the product retrieves your context, handles your document formats, and manages its failure cases. The product is the thing you are buying, not the model inside it. A good product on a mid-tier model beats a thin wrapper on the best model available, routinely.
Translate everything into daily wrongness
This is the conversion that makes the numbers real, and it is the one to do before every decision.
Kavita's claim summarisation runs at roughly 400 files a week. A vendor claiming 92% "accuracy" on summarisation means, at her volume, about 32 poor summaries a week. Roughly six a day.
Now the questions are answerable by an operations manager rather than a data scientist. Will the reviewer notice a poor summary? Yes — they have the file. What does noticing cost? Three minutes of re-reading. Is six a day acceptable against the time saved on 394 good ones? Comfortably.
Run the same arithmetic on the fraud detector and you get 296 flags and 98 wrongly-suspected customers per 10,000 claims. Same style of number, entirely different answer.
Every claim becomes a decision once you multiply it by your volume and ask who absorbs the error. That single move is most of what this lesson is for, and it does not require you to understand anything about the technology.
Do this today: take the last vendor claim you were given and convert it into errors per day at your volume. Then ask who catches them and what it costs.