Context Windows and Model Limits

Learning objective

Plan a request that fits without silently dropping the information the model needs.

Prerequisites: know what a language model predicts, how text is tokenised, and what an instruction prompt contains. If those ideas are new, start with What is Generative AI?, Tokenisation and Word Embeddings, and LLMs — Pretraining, RLHF, and Scaling Laws.

A context window is a request’s working set

A model processes tokenised input as context while producing an output. In a multi-turn application, the request may include instructions, the current question, selected conversation history, retrieved passages, and tool definitions or results. The model does not automatically remember text that your application did not send; some APIs offer managed conversation state, so inspect what that API includes on each turn.

A context-window limit is measured in tokens, not words or characters. Token counts vary with the tokenizer and the text: code, punctuation, uncommon names, and different languages can consume different numbers of tokens. Use the model or provider’s documented counting method when you need an exact budget.

Budget all parts of the request

Treat the following numbers as a planning example, not as a tokenizer estimate. Assume this model’s documented combined request and output limit is 7,000 tokens. The application’s planned request totals 7,350 tokens, so it is 350 over budget.

One possible adjustment is to retrieve fewer, more relevant passages: reducing that component by 600 gives 6,750 planned tokens, leaving 250 tokens below the hypothetical limit. In a real application, keep additional headroom because estimates, tool output, formatting, and provider accounting can differ.

A reliable budgeting workflow

1. Choose the model first. Limits and token accounting are model-specific.
2. Reserve output capacity. Do not spend the entire allowance on input if the answer needs room.
3. Count the actual request. Include instructions, history, retrieved content, tool schemas, and tool results.
4. Trim by value, not by position alone. Remove duplicate history and low-relevance evidence; keep the user’s current task and required constraints.
5. Make loss visible. If history or retrieval must be shortened, record what was omitted or summarised rather than silently dropping content.
6. Evaluate answer quality. Test with relevant evidence in different positions and check whether the response uses it correctly.

Diagnose common failures

Practice: repair an over-budget request

Your request has an 8,000-token combined limit and currently uses 8,450 tokens, including a 1,000-token response reserve. The user’s latest question and a required safety instruction must remain. You have 3,200 tokens of retrieved documents, of which only two passages directly answer the question.

1. Keep the reserve and the required instruction.
2. Remove duplicate conversation turns before summarising the current question.
3. Prefer the two relevant passages over unrelated retrieved text.
4. Recount the actual request, leave headroom, and test that the answer still uses the needed evidence.

Check your result: trimming at least 451 tokens gets the planned request below 8,000, but a production request should leave more than one token of headroom. The exact amount depends on the provider’s accounting and request structure.

Key takeaways

• The context window is a token budget for the model request, not persistent memory.
• Budget instructions, user input, history, retrieved evidence, tools, and output together according to the chosen API’s rules.
• Fitting more text does not guarantee better recall; retrieval quality and evaluation still matter.
• When trimming, protect the current task and constraints, keep useful evidence, and make omissions observable.