What is a context window, and why does my AI agent forget?
The short answer
The context window is everything the model can see while it works on your request: its instructions, the documents it was given, the conversation so far, and its own answer. It is the model's working memory. It has a fixed size, and as it fills up, the model pays less careful attention to what is in it. What falls outside it, the model does not know.
The model’s working memory
Anthropic’s documentation describes the context window as all the text a language model can refer to when it writes a response, including the response itself. It is different from the huge amount of text the model was trained on. It is the model’s working memory for this one task.
Everything the agent knows about your request has to be inside this window: your instructions, the email it is answering, the contract it is reading, and whatever was said earlier in the conversation.
It has a fixed size
The window is measured in tokens. A token is a small piece of text, often a part of a word. Each model has a maximum number of tokens it can hold at once.
In a long conversation, or a long task, the window eventually fills up. Many chat tools then drop the oldest parts of the conversation, or replace them with a shorter summary. Either way, details that were agreed early on can disappear. This is a common reason an agent “forgets” what you agreed. The question What happens when the AI agent forgets what we agreed? covers what to do about it.
More is not better
It is tempting to put everything into the window, just in case. That makes the work worse, not better.
Anthropic’s documentation says that as the number of tokens grows, accuracy and recall decline, a problem it calls context rot. A 2023 study found the same pattern from another angle: models used information at the start or the end of a long input well, and did much worse when the information they needed sat in the middle.
What this means for you
- Give the agent what this task needs, not everything you have. Choosing what goes into the window matters as much as how big the window is.
- Keep agreements outside the conversation. Decisions that must last belong in a written note or file that the agent reads again at the start of each task. Anthropic describes this practice, agents keeping structured notes outside the context window, as a way to keep long work on track.
- Start long work in fresh sessions. When a task runs for days, end each session with a short written summary, and start the next session from it.