The context window is the maximum number of tokens a model can consider at once — its working memory. Everything must fit: the system prompt, the conversation so far, retrieved documents, and the space left for the reply.
It's a shared budget
[ system prompt ] + [ chat history ] + [ retrieved context ] + [ your message ] + [ room for the answer ]
└──────────────────────── all of this ≤ context window ───────────────────────┘Overflow it and the oldest tokens get dropped — the model "forgets" the start of the conversation.
Why bigger is expensive
Attention compares every token with every other token, so cost (and memory) scales with the square of the length. Doubling context ~quadruples the work — which is why huge windows need lots of VRAM and special tricks (FlashAttention, RoPE scaling, sparse attention).
Managing it
- Summarize old turns instead of carrying them verbatim.
- Retrieve only the relevant chunks (RAG with Local Embeddings) rather than dumping everything.
- Tune
num_ctxdeliberately — see Ollama Parameters Guide and Serving Multiple Models on a Dual-GPU Box for the VRAM cost.
A bigger context window isn't free memory — it's expensive working memory you should spend wisely.
Related: The Attention Mechanism · Tokenization Explained · Ollama Parameters Guide