Counting tokens before you send a request

Why counting ahead of time is worth a real step in your pipeline

It's tempting to skip a dedicated counting step and just send the request, catching an oversized one only when it fails — but that approach means every oversized request costs you a full round trip and a failure to handle, instead of being caught locally before it's sent at all. For a pipeline processing any real volume, a counting step ahead of the actual call pays for itself quickly, both in avoided failed requests and in giving you the information to route or truncate proactively rather than reactively.

Count against the model you're actually going to call

The single most common mistake here is counting once and reusing that count across requests to different models, or counting against a default model that isn't the one your routing logic actually ends up selecting. Token count for identical text genuinely differs across the current model lineup, so a count that was accurate for one model can be meaningfully wrong for another — close enough to not error, but wrong enough to throw off a cost estimate or a budget check based on it. If your pipeline routes dynamically, count immediately before sending, using the exact model id that request is about to target, not a count computed earlier against a different assumption.

What counting doesn't tell you

A token count tells you the size of your input; it says nothing about the size of the response you'll get back, and total request cost depends on both. Don't treat a pre-send count as a complete cost estimate on its own — pair it with a reasonable estimate of expected output length for the task at hand, since output pricing and input pricing aren't the same rate, and a request with a small input but a long expected output can cost more overall than the input count alone would suggest.

Building a pre-send budget check

Once you're counting reliably, the natural next step is enforcing a ceiling before a request goes out — rejecting or truncating anything that would exceed a budget you've set, rather than discovering the problem from a failed API call. This is also the right place to account for reserved output space and, on models where it applies, thinking overhead, since a request that passes an input-only check can still exceed the full window once those are included. See context window exceeded on a request you thought fit for exactly this gap and how it shows up in practice.

Counting is cheap enough to do liberally

Because a dedicated counting call is inexpensive relative to the request it's checking, there's little reason to be sparing about when you use it — count before every request in a pipeline handling meaningful volume, not just occasionally as a spot check. The cost of counting too often is negligible; the cost of counting too rarely is an oversized request that fails at send time, after you've already committed to it, instead of being caught and handled beforehand.

Where a naive estimate goes wrong

If you've been estimating rather than counting — using a rule of thumb, or a tokenizer built for a different model family entirely — the gap between that estimate and the real count is usually larger than people expect, and larger still on code or non-English text than on ordinary prose. Replace the estimate with an actual count rather than trying to correct the estimate with a fudge factor; a fudge factor tuned against one kind of content doesn't generalise to another. The token & cost estimator shows exactly how large that gap is on your own real text, side by side with the accurate count.