AI Hallucination or Poisoned Context?

In spring 2025, Google's Gemini 2.5 Pro played Pokémon Blue to the end. An independent developer had built a harness that handed the model the game state, its goals and its own notes on every move. At one point a thirsty guard lets the player pass only after receiving a drink from a vending machine. For many hours the model was convinced it needed tea. Tea only exists in a later remake of the game.
In the Gemini 2.5 technical report, the Google team describes a more stubborn form of the same problem and calls it "context poisoning". Wrong assumptions about the game state then sit in several parts of the context at once, in the goals and in the summary of the run so far. The model goes on to pursue impossible or irrelevant goals, and according to the report the error can often take a very long time to undo.
That was a game. In an office the same thing happens more quietly, and it can start with an error that someone takes for a hallucination.
Hallucination or wrong context?
An AI hallucination is a statement a language model makes up that appears in no source: a court ruling that does not exist, or an item number in the right format that belongs to no product. The model fills a gap with something plausible.
With wrong context, the model invents nothing. It repeats what is in front of it, and that is no longer true, such as an outdated price list, a note from an earlier conversation or a summary in which an exception has turned into the rule. To the reader, both errors look the same. They are stated with confidence and fit the rest of the text.
The difference decides the fix. A hallucination is countered by giving the tool the missing source or by telling it to mark a gap instead of filling it. With wrong context, the source is already there, and it is the problem. Switching to a stronger model is no reliable fix either, because it receives the same material. When switching models pays off is covered in our article The best AI model or better context.
A simple test helps tell them apart. Ask the tool where the statement came from, and look up the place it names yourself. If the wrong statement sits in a stored note, an earlier conversation or a file from your records, the problem lies in the context. If it appears in no source, that points to a hallucination. The tool's answer is a pointer, not proof, and sometimes it is no longer possible to establish what the tool had in front of it.
How an error becomes a rule
Wrong context becomes dangerous when it feeds itself. A result is stored, the next request reads it as fact, the next result confirms it. After a few rounds the same statement sits in several places and looks like established knowledge, although it has only been repeated often.
In everyday office work, three places feed this loop:
- Automatic memory features. According to OpenAI's memory help page, ChatGPT saves details that might be useful in later conversations even without being asked, and can draw on earlier conversations. Deleting a conversation does not delete the memory that came out of it.
- Notes an agent writes about its own work. For Gemini these were the goals and the summary of the run; for an agent in an office they are handover notes, call records and status entries.
- A company's own working instructions and knowledge files. They look checked because they are written down, which gives a reader little reason to question them.
That a model rarely corrects an early error on its own has been measured outside games too. In LLMs Get Lost In Multi-Turn Conversation, Philippe Laban and colleagues analysed more than 200,000 simulated conversations in 2025 in which a task was described piece by piece instead of all at once. Performance dropped by 39 % on average. The models made early assumptions, held on to them and did not find their way back after a wrong turn. The study used simulated users and models from early 2025, not office tasks.
The loop can also be triggered from outside. In September 2024, the security researcher Johann Rehberger showed that a website ChatGPT read on a user's behalf could write an entry into that user's memory. The entry contained instructions and stayed active in every later conversation. OpenAI closed the channel through which data leaked in that demonstration. According to Rehberger, outside content could still write to memory at the time, and he advises reviewing stored memories regularly.
A case from our own work
The clearest case we found was in a file we had written ourselves.
Since August 2026, a file recorded the positions CERTAINCE takes in its writing. Agents drafting articles for this website were given it as the basis for their drafts. The file said explicitly that new content adopts the position stated there, and listed for each position the articles that already argued it.
One of these positions said that in office work the information supplied per task limits the result, not the model's intelligence. The evidence given was our own article on model choice. That article itself names tasks where a stronger model is the right answer; in the file, all that remained was a general rule. The file listed every page that adopted a position as one more place where it was argued; for this position, our homepage was among them. In the end the file recorded that our texts argued these positions consistently. Nobody had measured that. The texts agreed because they came from the same template.
We reviewed and cut the file on 6 October 2026 to remove assumptions that could lock future work onto the wrong path. It now says that our own published texts prove what we said, not that it is true. An old framing is not kept merely because it is consistent. The former blanket statements, such as context always being the limiting factor or review always being the bottleneck, are explicitly no longer standing rules.
A second habit helps us find such sources earlier. When we correct a statement in our knowledge store because a file or message supplied a wrong or outdated version, we record that source with the correction. On 6 October alone, nine corrections carried such a record. Besides the single error, this shows us which file keeps misleading our agents.
How to recognise poisoned context
- The tool insists on a statement that nobody in the company can still back up.
- A correction holds until the end of the conversation and is forgotten in the next one.
- An exception that once applied to one customer appears in drafts for other customers.
- Several results agree, and none names a source outside your own records.
- The same error remains after a model switch.
Each of these signs is a reason to look for the source instead of patching the individual text.
What helps against context poisoning
Separate what a tool stores on its own from what is binding in your company. Automatic memory features are convenient for personal preferences. Prices, commitments and customer rules belong in records where every statement has a date, a source and an owner. In our own work we therefore leave Claude's account memory switched off and keep business facts in dated, versioned files. If you keep memory switched on, review the stored entries regularly.
Write corrections into the source, not into the chat. A correction in a conversation helps that conversation first. Some tools carry it into memory, but if the error still sits in a knowledge file or a stored note, it returns with a later request until it is corrected there.
Start over when a conversation has stalled. According to the measurements by Laban and colleagues, models rarely recover from an early wrong assumption within the same conversation. A new conversation with a complete description of the task starts without the old history. It does receive stored memories and files again, which is why correcting the source belongs with it.
Do not let results flow back into your records unchecked. New rules and decisions that an agent notes stay drafts until the named owner confirms them. Small routine updates that can be undone run without a sign-off for each one once they have a proven track record. Spot checks then cover them. Record who wrote an entry: an employee, an agent or an automatic memory feature.
Require evidence for every rule from outside your own body of text. Three of your own documents saying the same thing prove nothing if they copied from each other. A rule needs a decision with a date and a name, or a measurement.
Review instruction files at fixed intervals and ask of every sentence whether it records a decision someone made or an assumption that settled in through repetition. How such records are set up and who maintains them is described in Context management for business.
We can take on this upkeep for you once an AI agent is running, billed by the hour. Every month we evaluate where results were corrected and draft the correction to the source; the named responsible person in your company confirms new rules and decisions. We test every change on the same real examples. Discuss a concrete workflow
What does the term hallucination mean in the context of AI?
An AI hallucination is a false statement that a language model produces although it appears in no source, such as an invented quote or a plausible but wrong figure. The model fills a gap instead of marking it. Wrong context is different: there the model correctly repeats a statement that was supplied to it incorrectly.
What causes AI hallucinations?
Language models continue text the way it most probably goes on. When information is missing from the context, they often fill the gap with something plausible instead of naming it. Many errors counted as hallucinations have a second cause: the tool follows an outdated file, a stored note or an earlier summary that is no longer true.
How can hallucinations in ChatGPT be avoided?
Supply the sources the task needs and tell ChatGPT to mark missing information instead of filling it in. Also check what ChatGPT has stored; under Personalization in the settings you can view and delete its memories. When a conversation has stalled, a new conversation with a complete task description often helps more than further corrections, provided the error does not sit in a stored memory.