Why your token-count estimate was wrong

The tool most teams reach for first is the wrong one

If you've been estimating Claude token counts with a tokenizer built for a different model family — most commonly tiktoken, OpenAI's tokenizer — the number you got back describes a different vocabulary than Claude actually uses, and it documentedly undercounts ordinary prose against Claude's real count. That gap widens further on code and non-English text, where the vocabulary mismatch between tokenizers matters even more than it does on plain English prose.

Why "close enough" estimates fail exactly when it matters most

An estimate that's reliably off by a consistent, small margin is manageable — you can pad your budget by that margin and move on. The actual problem is that the gap between an estimate and the real count isn't a fixed percentage across all content types; it varies by what you're counting, which means a padding factor tuned against your typical prose content will still fail on the requests that are heaviest in code or structured data, which also tend to be exactly the requests where an oversized-request failure is most costly to hit unexpectedly.

Token count is also not stable across Claude's own model generations

Even setting aside the wrong-tokenizer problem entirely, token count for identical text isn't a fixed property of the text within Claude's own lineup either — a newer tokenizer introduced with a recent generation counts some text differently than an older one still used elsewhere in the lineup, and a redesigned tokenizer on a recent Sonnet generation counts noticeably more tokens for identical input than its immediate predecessor did. An estimate that was accurate against one model generation can become wrong again purely from a model upgrade, with the underlying text never having changed.

The only estimate worth trusting is a real count

There's no reliable shortcut that avoids an actual counting call against the specific model you're about to use — not a cross-vendor tokenizer, not a fixed conversion ratio, not a character-count heuristic. Each of these introduces its own error, and that error compounds with whichever other approximation you're also using. Counting directly against your actual target model, immediately before sending, is the only approach that doesn't accumulate this kind of compounding inaccuracy.

Making the switch practically

Replacing an estimate-based budget check with a real counting call is usually a small code change with an outsized reduction in unpredictable request failures — the token & cost estimator shows exactly how large the gap was on your own real text, side by side against the accurate figure, which is often the fastest way to convince a team the switch is worth making before they've hit a production failure that forces the issue.

Why this matters more once caching is in the picture

An inaccurate token count doesn't just risk an oversized-request failure — it also throws off any cost projection that depends on knowing how much of a request is eligible for caching versus billed fresh, since that projection is built on top of a count that was already wrong before the caching math was even applied. An estimate that's off by a meaningful margin on the input side propagates into every downstream calculation built on it, which is a good part of why this specific inaccuracy tends to be more consequential than it initially looks.

A habit worth building into code review

Treat a hardcoded token estimate, or a cross-vendor tokenizer import, the same way you'd treat any other known-wrong pattern in code review — worth flagging on sight rather than waiting for it to cause a production incident. Once a team has been burned by this once, it's a fast, cheap thing to catch early in any new pipeline that touches Claude requests, and a much slower, more expensive thing to trace back to its cause after the fact from an unexplained failure rate.

See counting tokens before you send a request for building this into a real pipeline, and count_tokens gave a number that didn't match the bill for the related mistake of counting against the wrong model even when using the real endpoint.