On 22 June 2023, a New York lawyer with thirty years at the bar was humiliated in a federal court. His mistake: filing a brief that cited six court decisions. Judges’ names, docket numbers, excerpts. All false. None of those cases existed. He had asked ChatGPT for them, and it had invented them without the slightest hesitation, and better still, it had confirmed they were genuine when he asked it to check.
That case, Mata v. Avianca, became the textbook example everyone cites. But it is not an isolated one, and that is what a business leader should take from it. Eight months later, in February 2024, a Canadian court ordered Air Canada to pay up because its chatbot had promised a traveller a bereavement fare that did not exist in the airline’s terms. Air Canada’s defence rested on an almost comic idea: the chatbot was a separate entity, responsible for its own statements. The court dismissed it flatly. A company answers for what its machine says.
Two sectors, two countries, the same trap. A generative AI can produce a perfect falsehood with perfect confidence. We call it a hallucination, and until you understand where it comes from, you cannot protect yourself from it.
What the machine actually does when it hallucinates
The word hallucination is misleading, because it suggests a bug, an accident. It is neither. The machine is doing exactly what it was built to do.
A language model does not consult a store of truth before answering. It predicts, word after word, the most probable continuation drawn from the vast quantity of text it was trained on. Most of the time the most probable is also the most true, which is why it works so often. But when the information does not exist, or the model does not have it, it does not stop to say so. It keeps predicting something plausible. A case law that looks like real case law. A fare shaped like a real fare. A well turned quotation attributed to someone who never said it.
The result is convincing because it is designed to be convincing, not to be accurate. That is the whole difference, and the whole danger. An intern’s mistake shows: they hesitate, they stumble, they say they are not sure. An AI’s mistake arrives dressed as certainty, in a flawless sentence.
Why this is a leadership risk, not a technical detail
A hallucination is only dangerous if no one catches it. And in a business under pressure, no one catches what is well written.
This is even truer in a small company, where decisions often rest on a single person, with no legal team to review and no committee to filter. You ask the tool for a summary, an argument, a memo, and you use it as is because it is clean and you are short on time. The day that summary contains an invented figure, the company carries it, not the machine. Air Canada learned this the hard way.
The sensitive zones always come back to the same places. Law and contracts, where a false reference can cost you a case. Numbers, where a plausible amount is not a correct amount. Dated facts, names, quotations, which the model happily reconstructs from memory. And anything that will be read by a customer or a third party, because that is where the error becomes public.
How to protect yourself without giving up the tool
The good news is that the safeguard is not technical. It is organisational, and it fits into a few simple rules a leader can set in one meeting.
The first rule I call the source or silence. An answer that puts forward a new fact must be able to say where it comes from. No verifiable source, no taking it as your own. A claim with no origin is not an answer, it is a well dressed guess, and you treat it as such.
The second is to sort uses by risk. To rephrase, structure, translate or summarise a document you supply yourself, the risk is low and the AI is excellent: use it widely. To produce a fact, a source, a figure, the risk climbs and checking becomes mandatory. Everyone in the company needs to know where that line sits.
The third is to keep a human on the decisions that commit you. The machine prepares, the human decides and signs. This is not distrust of the tool, it is recognising what it is: a brilliant performer with no judgement.
The lawyer in the Avianca case, by the way, did not stop using AI after his ordeal. He learned to check every reference by hand. He did not change tools, he changed his way of using them. That is exactly the lesson.
The real subject is not the machine, it is us
I say it often to my students and to the leaders I train: the problem is not that AI gets things wrong, it is that it gets them wrong with confidence, and we tend to confuse confidence with correctness. Our whole education taught us to trust a well written text. The machine turns that reflex against us.
Learning to use it means learning to separate two things the tool constantly blends: what is said with assurance, and what is true. A company that draws this distinction clearly wins on both counts, the speed of the tool and the safety of its decisions. A company that ignores it takes, without seeing it, the risk Air Canada took: letting a machine speak in its name, and finding out too late that it spoke falsely.