Why does AI sometimes make things up?
The short answer
Because a language model is built to produce a likely answer, and the way models are trained and tested rewards a confident guess more than an honest "I don't know". The industry calls these invented answers hallucinations. They come from how the model works, so you deal with them through the way you design the work, not by hoping the next model will be free of them.
What a hallucination is
OpenAI, the company behind ChatGPT, defines hallucinations as plausible but false statements produced by a language model. The important word is plausible. The invented answer looks exactly like a correct one: the same confident tone, the same tidy sentences, sometimes a reference that does not exist.
Why it happens
A language model produces the most likely continuation of the text in front of it. When it knows the answer, the likely continuation is the right answer. When it does not, the likely continuation is still an answer, and it sounds just as sure.
OpenAI’s 2025 explanation adds a second cause. Models are trained and scored in a way that rewards guessing over admitting uncertainty. Think of an exam where a blank answer always scores zero, while a guess sometimes scores a point. A student who wants the highest mark learns to guess. So does the model.
Where it happens most
The risk is highest for specific facts the model rarely saw, or never saw at all: an exact date, a person’s name, a figure, a clause number. Your own company’s data falls in this group. The model has never seen your contracts or your prices, so any answer it gives about them from its own memory is a guess. The page What is a language model, and why doesn’t it know my company? explains why.
What reduces it
- Make the agent look before it answers. Give it your own documents at the moment it answers, and tell it to answer only from them. Research on this method, called retrieval-augmented generation, found that it produces more factual answers than a model working from memory alone. The page Should I train the AI on my company’s data, or give it the data when it answers? explains it.
- Ask for the source with every answer. An answer that names the document it came from can be checked in a minute.
- Make “I could not find it” an acceptable answer. Say so in the agent’s instructions, and treat it as correct behaviour when it happens.
- Check what matters. For anything that goes to a client or changes a record, a second agent or a person checks the work before it is used.