Reading a vendor demo
You will sit through several of these. They are carefully built, they are impressive, and they are designed to prevent exactly the questions that matter. This lesson is how to get the information anyway, without being adversarial about it — a good vendor will answer all of this happily, and how they respond is itself the most useful signal you will get.
What a rigged demo looks like
Not fraud. Ordinary sales practice, and knowing the shape of it is enough.
The data is theirs. Clean, well-formed examples chosen because the system handles them well. Your data has scanned attachments, three languages, a template that changed in 2023, and customers who write in paragraphs with no punctuation.
The path is fixed. The demo follows a route walked a hundred times. Any deviation lands somewhere less polished, which is why demos rarely invite you to drive.
The failure cases are absent. You will see what it does when it works. What it does when it is unsure — and how you would even know it was unsure — is the thing you actually need to see.
The integration is assumed. "It connects to your systems" is doing enormous work in that sentence. Connecting to your claims system may be a configuration, a project, or a fantasy, and the difference is months.
The seven questions
"Can we run it on our data, in the meeting?" The single highest-information question. Bring twenty real, messy, anonymised examples. A vendor who says yes is confident; one who needs "a data preparation phase" is telling you your data is a problem, which is useful to know now. Watch the ten that go badly, not the ten that go well.
"Show me it being wrong." Ask directly. Every honest vendor can. The answer tells you their failure mode — whether it fails loudly or quietly — which matters far more than the failure rate.
"What's the accuracy on data like ours, and how was that measured?" Lesson 5 is entirely about parsing the answer. For now, note that "95% accurate" without a stated measurement, dataset, and task is not a number.
"When it's unsure, what happens?" You want a system that flags low confidence and routes it to a human. A system that always produces an answer with no confidence signal puts one hundred per cent of the checking burden on your team, which changes the economics completely.
"Where does our data go, is it used for training, and how long is it kept?" Ask for it in writing. This one is a compliance requirement, not curiosity.
"What does implementation actually involve, and who does it?" Get days of your people's time, not theirs. The hidden cost of most deployments is internal effort nobody budgeted.
"Can I speak to a customer who churned?" Slightly cheeky and revealing. A vendor comfortable with this has a product that fits some people and not others and knows which. Reference customers they choose will all be delighted.
The pilot-data trap
Vendors will offer to run a free pilot on your data. Do it — with two conditions.
Agree the success criteria before it starts, in writing. Otherwise the pilot ends and everyone interprets the result in the direction they were already facing. Lesson 6 is how to write them.
Know what happens to your data afterwards. A free pilot involves handing over real customer information to a company you have not yet contracted with. Get deletion in writing, get the training-use answer in writing, and check whether your own compliance function needs to approve it beforehand — because they will certainly want to have been asked afterwards.
Kavita's three vendors
The first proposed fraud detection, which her audit had already excluded. That the pitch did not change after she said so told her the product was looking for a problem.
The second demoed claim summarisation beautifully on their data. On twenty of hers it produced good summaries for fourteen, mediocre for four, and two confident summaries of the wrong claim, because those files contained references to a prior related claim and it had blended them. That is a genuinely useful result, it appeared only because she brought her own files, and it defined the checkpoint her eventual process needed.
The third answered the data questions in writing within a day, and said plainly that their tool would not help with two of her four tasks. She bought from them eventually. Not because the product was best on a feature grid, but because they were the only one whose answers survived contact with her actual operation.
Do this today: assemble twenty real, messy, anonymised examples from your top task and keep them in a folder. That folder is the most valuable procurement asset you will have all year.