In May 2025, a group of researchers asked one of the most capable open-source language models available a very simple question: what is Adam Tauman Kalai's birthday? They added a condition. Answer with the day and month, and only if you actually know it. Kalai is one of the researchers, so the team knew the right answer. The model, DeepSeek-V3, answered three times. It gave three different dates. All three were wrong.
Nobody was trying to trick it. The question was plain, the instruction was clear, and the model had an explicit way out. It didn't take it.
That behavior has a name. People in the field call it hallucination: an AI system producing an answer that sounds plausible and confident but is false, irrelevant, or invented outright. The word is a little misleading, since the model isn't seeing things the way a person with a fever might. A better description is confident guessing. Once you understand where the guessing comes from, a lot of strange chatbot behavior stops being mysterious, and you get a much better sense of when to trust the answer on your screen.
A machine built to continue sentences
Start with what a large language model (the technology behind ChatGPT, Claude, Gemini and similar tools) actually does. During training, it reads an enormous amount of text and learns one skill: given some words, predict which word is likely to come next. Do that billions of times across books, websites, code and articles, and the model gets remarkably good at producing fluent, sensible-sounding language. It learns grammar, style, common facts, and the shape of an explanation. Picture an extraordinarily well-read improviser. Hand them the first line of almost any scene and they'll keep it going in a way that feels right. What they don't have is a separate reference shelf they check before speaking. The knowledge and the talk come out of the same place, and the model has no built-in way to mark which sentences it learned from many reliable sources and which it is assembling on the fly because they fit the pattern.
That last point is the heart of the matter. Some facts show up in training text thousands of times. Einstein's birthday is one; the boiling point of water is another. The model sees them so often that the pattern and the truth line up almost perfectly. Other facts appear once, or never. A particular researcher's birthday might appear in a single web page, if at all.
The rare-fact problem
The research team behind that birthday test, writing in a 2025 paper titled Why Language Models Hallucinate, makes this precise. Their argument borrows an idea from Alan Turing's wartime statistics work: the way to estimate how much you haven't seen is to count how many things you've seen exactly once. Applied to language models, it works like this. If 20% of the birthday facts in a model's training data appear exactly once, you should expect the base model to get at least 20% of birthday questions wrong. A single mention isn't enough to separate the true date from any other date that would fit the sentence just as well. So the model fills the blank with something shaped like a birthday.
Here's an analogy that holds up reasonably well. Imagine you overheard one conversation at a party five years ago in which someone mentioned a coworker's anniversary. Ask you about it today and you'll know it was in spring, probably. You won't know the date. A careful person says so. The model, by default, produces a date anyway, because producing a plausible next word is the only thing it was trained to do. Where the analogy breaks: you can feel your own uncertainty. The model has a statistical version of that feeling (some answers are less likely than others), but nothing in its basic training rewards it for acting on it.
The paper also shows that this isn't simply a problem of bad data. Even if every sentence in the training set were true, the statistical goal of the training process would still produce some errors on facts that are rare or that follow no learnable pattern. Clean data helps. It doesn't make the problem go away.
Graded like a multiple-choice exam
After the first round of training, companies spend a great deal of effort refining these models to be more helpful and accurate. So why does the guessing survive? The researchers' answer is uncomfortable for the industry. Most of the tests used to measure and rank models award one point for a correct answer and zero for anything else, and "I don't know" counts as anything else. Under those rules, a guess always beats an honest blank. A model that bluffs on hard questions will score higher than an otherwise identical model that admits uncertainty, and leaderboards reward the bluffer. Anyone who has taken a multiple-choice test with no penalty for wrong answers already knows the winning strategy. Fill in every bubble. The models have learned the same lesson, and for the same reason.
The fix they propose is modest and practical: change the scoring. Tell the model, right in the question, that a wrong answer costs points while "I don't know" costs nothing, and grade accordingly. At a 90% confidence threshold, for instance, a wrong answer would cost nine points against one for a right answer, so guessing only pays when the model is genuinely likely to be correct. The authors argue that until the popular benchmarks change, adding new hallucination tests on the side won't move the field much.
What this means when you're the one asking
The consequences are real. IBM's overview of the problem points to a US lawyer who, in 2023, filed court documents drafted with ChatGPT that cited cases which did not exist, and to an AI search summary that suggested adding glue to pizza sauce, apparently drawn from an old joke on Reddit. The same overview describes a more modern risk: chains of AI agents, where one system's invented number becomes another system's input and eventually turns into an action, such as deleting stored records. A wrong sentence in a chat window is embarrassing. A wrong sentence that a second program acts on is a different category of problem.
Knowing the mechanism tells you where to be careful. Hallucinations cluster around rare specifics: exact dates, citations, quotes, case names, version numbers, statistics, URLs, and details about people who aren't famous. Ask a chatbot for a summary of a well-known concept and it will usually do well. Ask it for the page number of a quote from an obscure report and you are asking for exactly the kind of single-mention fact it handles worst. When an answer matters, ask the tool for its sources, then open them yourself; a link that doesn't load or a paper you can't find is your signal. Tools that search the web or your own documents before answering (engineers call this retrieval) reduce the problem because the model is working from text in front of it rather than memory, though they don't eliminate it. And you can borrow the researchers' idea directly: tell the model that a wrong answer is worse than no answer, and that "I'm not sure" is acceptable. It won't make the model honest by itself. It does change the incentive in the one place you control, which is the question you ask.
References
China, C. R. (2026, July 31). What are AI hallucinations? IBM Think. https://www.ibm.com/think/topics/ai-hallucinations
Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). Why language models hallucinate (arXiv:2509.04664). arXiv. https://arxiv.org/abs/2509.04664
