The Claude tokenizer, and why tiktoken is the wrong tool

What this page is for

This is the pointer page: the name of the correct counting endpoint, the tool it's most often confused with, and the documented size of the gap between them. For the full explanation of why the gap exists, why it varies by content type, and why no fixed correction factor survives contact with real traffic, see why your token-count estimate was wrong — this page stays deliberately short and doesn't repeat that explanation.

Reading the row above correctly

The undercount figure in the table is the documented range for ordinary prose specifically. It is not a universal correction factor — code and non-English text sit further outside this range, in the same direction, and this site does not publish a single number for either because the gap isn't stable enough across content types to reduce to one figure. Where the exact size of the gap on your own content matters, measure it directly rather than extrapolating from the prose figure.

The one thing worth remembering if you read nothing else here

Count against the exact model id you're about to call, every time, rather than a cached count from an earlier session or a different model. Model id, not just "using the right endpoint," is the variable most likely to be silently wrong in a request-building pipeline that grew over time.

Why this figure alone doesn't make a cross-vendor tokenizer safe to use

Knowing the size of the gap doesn't turn a cross-vendor tool into a usable substitute for the real endpoint — a correction applied after the fact still depends on the gap being a fixed, predictable size, and it isn't one. Treat this row as evidence of how wrong an estimate can be, not as a multiplier to apply to one.

Related

See the token & cost estimator to see this gap on your own text, and counting tokens before you send a request for building a real counting step into a pipeline rather than checking manually.