Budgeting for a 1M-token context window

A large window is a capability, and capability has a cost

Current-generation models support a context window large enough to hold a substantial codebase or document set in a single request, and that capability is genuinely useful — but it's worth being deliberate about the fact that filling more of the window costs more, linearly, regardless of whether the model actually needed all of that context to answer well. "The window supports it" and "the task needs it" are different questions, and conflating them is the most common way a large context window turns into an unnecessarily large bill.

The difference between "fits" and "should send"

A codebase or document set that technically fits inside the window isn't automatically the right amount of context to send for a given task — most individual questions about a large codebase only actually need a fraction of it to answer correctly, and sending the whole thing "to be safe" pays full input price for content the task never touches. Scoping what you send to what's actually relevant to the specific question, rather than defaulting to everything available, is usually the single largest lever on cost for any workflow built around a large document set or codebase.

Where caching changes the calculus

If the same large context genuinely gets reused across many requests — repeated questions against the same document set, an agentic session that keeps referring back to the same codebase — that context is exactly the kind of large, stable prefix caching is built to reward, and the economics shift substantially once it's cached rather than resent fresh on every request. A workload that resends a large context on every single request without caching is paying full price repeatedly for content that, structured differently, would only be paid for in full once.

Your input budget isn't the whole window

Input is only the portion of the window you're actively choosing to fill — the ceiling you're actually budgeting against is smaller than that, once you subtract whatever's reserved for the response and, on a thinking-capable model, whatever reasoning consumes before that response gets written. Sizing a large codebase or document set against the full window figure rather than against what's actually left for input is a common way a request that looked comfortably safe on paper turns out not to be. See context window exceeded on a request you thought fit for exactly this failure and how the pieces add up against each other.

A large window changes the shape of the trade-off, not just the ceiling

Before large context windows were available, a lot of engineering effort went into aggressive summarisation and retrieval just to fit a task's relevant material into a much smaller budget. A large window removes that hard constraint, but it doesn't remove the cost trade-off underneath it — it just relocates the decision from "does this fit at all" to "is sending this much worth what it costs." Teams that built careful context-scoping discipline under the old, tighter constraint sometimes abandon that discipline once a larger window makes it technically unnecessary, and end up paying for a habit of sending more than the task needs simply because nothing stops them from doing so anymore.

Setting an internal budget below the technical ceiling

Because the model's actual limit and the amount you should reasonably send for a given task are different numbers, it's worth setting your own internal budget somewhere below the technical ceiling and treating that as the real constraint your application respects — rather than treating "does it fit in the window" as the only check that matters. A self-imposed budget forces the scoping discipline that a technical ceiling alone doesn't, and keeps cost proportional to what a task actually needs rather than to whatever happens to fit.

Working the actual numbers for your use case

Rather than estimating from a general sense of "large documents cost more," the context-window planner estimates how much of the window your specific codebase or document set actually occupies and what's realistically left over, and budgeting for a 1M-token context window style questions about a specific model's exact window and output limits are answered directly on that model's own reference page rather than assumed to be uniform across the lineup.