Tokens, Context Windows, and Why They Matter

Tokens, Context Windows, and Why They Matter

The prototype was flawless. You pasted a few paragraphs into the prompt and the model answered beautifully. Then the first real customer uploaded a ninety-page PDF, the request either failed outright or cost more per call than the customer pays per month, and the answers turned to mush. Nothing about your code changed. The size of the input did.

Almost every confusing LLM behavior traces back to two quantities: tokens and context windows. They determine what you can send, what it costs, how fast responses come back, and where quality quietly degrades. This is the machinery underneath.

From Letters to Tokens#

Models do not read characters or words. Text is first chopped into tokens: subword pieces learned from a byte-pair encoding over massive training corpora. Frequent words survive as single tokens. Rare words get split. Numbers, code, and non-English text tend to fragment heavily, which is why they cost more than you would guess.

A practical rule of thumb for English prose: one token is roughly four characters or about three quarters of a word. You can measure instead of guessing:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o-mini")
text = "Context windows are rented, not owned."

ids = enc.encode(text)
print(len(ids))
print([enc.decode([i]) for i in ids])

Each integer is an index into the model's fixed vocabulary; decoding any single id gives back its piece. The split looks like this:

Why care beyond billing? Because every limit the API enforces is counted in tokens, not words or characters. Your safety margins need to be in the same unit as the constraint.

The Window Is a Budget You Share#

The context window is the maximum number of tokens the model can consider in a single request. Everything shares that budget whether you like it or not: your system prompt, the entire conversation history, retrieved documents, tool outputs from earlier steps. And critically, the model's response must fit too. Output limits are often smaller than the input window and are enforced separately.

Exceed it and most APIs reject the request with an error. Some clients silently truncate history to cope, which produces a different failure mode: the model forgets things mid-conversation while your logs show nothing wrong. Silent truncation is worse than an error because nobody goes looking for it.

Billing follows the same math. You pay per token on both sides of the exchange. Prompt caching, offered by the major providers, discounts repeated prefixes such as long system prompts, which is worth knowing before you shuffle instructions around unnecessarily.

Bigger Windows Don't Solve Everything#

Million-token windows exist now, so stuffing everything in seems tempting. Three problems ruin the plan.

Cost scales linearly per call with everything you include, and latency grows alongside it. A retrieval step fetching two relevant paragraphs beats pasting a whole wiki on both axes, every single call.

Quality also degrades with length. Research on long-context behavior, notably the "lost in the middle" findings from Liu and colleagues in 2023, showed models recall information best near the beginning and end of the context and weakest in the middle. Benchmark scores for giant windows also tend to reflect simple lookup tasks, not messy real workloads.

The conclusion worth internalizing: a context window is expensive RAM that you rent by the token. Treat it like a memory budget under pressure, not an archive.

Staying Inside the Limit#

There is a preferred ladder here, ordered from cheapest to most involved.

First, measure. Estimate every request's token count before sending, using tiktoken locally or the usage fields returned in each response. Reject or flag oversized inputs early with a clear message instead of letting the API error cryptically.

Second, trim. Old conversation turns about resolved topics can go. System prompts accumulate cruft and deserve aggressive editing. Retrieved HTML full of boilerplate should be cleaned before it enters the window.

Third, summarize. For long-running conversations, have a cheap model compress older turns into a running digest and ship the digest instead of raw history. The main model reads a paragraph instead of forty exchanges.

Fourth, retrieve instead of paste. Embed documents once, store vectors, and fetch only the top few chunks relevant to the current query. This is the core move behind retrieval-augmented generation, and it is why RAG dominates enterprise architectures rather than "upload the whole drive."

Fifth, distill. If the task needs facts rather than prose, extract them into structured JSON once and pass the compact record onward.

Measure Before You Ship#

Token awareness belongs in your engineering hygiene, not just your architecture decisions. Log the usage numbers from every response so cost regressions show up in dashboards rather than invoices. Unit-test prompt templates against expected sizes so a well-meaning edit cannot triple your bill. Alert when p95 session costs climb.

Engineers who count tokens ship products that stay fast, cheap, and coherent under real user behavior. Engineers who assume the window will hold discover the opposite in production, usually during the demo that matters.