The tokenizer mistake almost every cost estimate makes

A mistake ordinary review processes aren't built to catch

Most categories of bug get caught by something in a normal engineering workflow — a test fails, a type checker complains, a linter flags an unused import. A wrong token count doesn't trip any of those, because there's nothing structurally wrong with the code that computes it. A cross-vendor tokenizer runs, returns a number, and that number gets used exactly as intended by whatever called it. The mistake isn't a bug in the conventional sense; it's a wrong assumption baked into an otherwise-correct piece of code, and that specific category of error is invisible to almost every automated check a team already has in place.

Why code review doesn't reliably catch it either

A reviewer scanning a pull request for a token-counting utility is checking whether the code does what it claims to do — count tokens — and by that narrow standard, code built around the wrong tokenizer passes review cleanly every time. Catching this requires a reviewer who happens to know, specifically, which tokenizer is correct for which model family, and who happens to be looking for that exact mismatch on that exact review. That's a narrow, easy-to-miss condition, which is exactly why this mistake tends to ship and stay shipped rather than get caught at the review stage where most other categories of error get filtered out.

The gap between "technically works" and "quietly wrong"

What makes this worse than an obvious bug is that the resulting system doesn't fail — it just produces numbers that are wrong by an amount nobody's checking. A cost projection built on a wrong count still produces a plausible-looking dashboard. A budget check built on a wrong count still approves and rejects requests, just using the wrong threshold. Every downstream system built on top of the bad number inherits its wrongness silently, and none of them has any way to know the input they're trusting was never actually correct.

Why this is a process gap, not a knowledge gap

It's tempting to frame this as "people don't know Claude has its own tokenizer," and for some teams that's true on day one. But the more durable version of this problem persists even after a team learns the fact, because knowing a fact and having a process that enforces it are different things. A team that fixed one estimate after discovering the mismatch, without changing how future pipelines get built or reviewed, will very plausibly reintroduce the same mistake the next time someone stands up a new integration under time pressure and reaches for whatever's fastest to wire up.

What actually closes the gap

The fix that survives staff turnover and time pressure isn't a wiki page saying "use the right tokenizer" — it's a check that runs automatically and fails loudly when it's violated, the same way a type error fails a build regardless of whether the engineer who introduced it remembered the rule. A repository-wide check for cross-vendor tokenizer imports anywhere near a cost-relevant code path, run as part of normal CI rather than relied on as a fact everyone's supposed to remember, turns this from a recurring mistake into a one-time fix.

What to actually check for, concretely

If you're auditing your own systems for this, the search is specific and mechanical: any import of a tokenizer library associated with a different model vendor, anywhere near code that estimates size or cost for a Claude request. It's a narrow, greppable pattern, and finding even one instance is worth treating as a signal to search the rest of the codebase rather than assuming it was isolated.

Why this is worth writing down even though it sounds obvious in hindsight

Every mistake in this category sounds obvious once it's named, and that's exactly why it keeps recurring — a mistake that sounds obvious in hindsight is assumed to be one nobody on a competent team would actually make, which is precisely the assumption that lets it slip past review unquestioned. Naming it plainly, as a specific pattern with a specific fix, does more to prevent it than trusting general competence to catch something that, by design, doesn't look like a mistake from inside the code that contains it.

See why your token-count estimate was wrong for the full technical breakdown of why no correction factor closes this gap reliably, and the token & cost estimator to see the actual size of the gap on your own content.