Prompt caching is not free — here's the math

The part of caching's pitch that gets skipped

"Prompt caching saves money" is true and also incomplete in a way that costs teams real spend when they act on the incomplete version. The full version includes a cost, not just a saving: writing a prefix into the cache for the first time costs more than an ordinary read would have, and that premium has to be earned back through later reads before the arrangement is actually saving anything. Skip that half of the pitch and "turn caching on" reads like a free lever. It isn't one.

A pattern that looks identical to a saving and isn't

The failure mode worth naming specifically is that an unprofitable caching setup doesn't look broken from the outside. Requests succeed. The usage block shows cache activity, which reads as "caching is working." Nothing in that picture distinguishes a prefix that's earning back its write premium many times over from one that's writing a fresh, expensive cache entry on almost every request because it's barely ever reused before expiring. Both look like "caching is on." Only one of them is actually cheaper than not bothering.

Where a real integration accumulates this without anyone deciding to

The most common way teams end up here isn't a single bad decision — it's caching applied uniformly across an integration that has more than one kind of request pattern living inside it. A shared system prompt reused on nearly every call is a clear win. A per-user or per-session context block that happens to be long enough to "look cacheable" but is functionally unique to that one interaction is not — and if the same blanket caching policy covers both, the losing half is quietly offsetting some or all of the gain from the winning half, and the combined bill doesn't obviously tell you that's what's happening.

What caching doesn't touch, which further narrows where the win actually lives

Even on the winning half, the saving is bounded by what caching can discount at all: the input side of a request, and specifically the portion of input that's structured as a stable, repeated prefix. Output tokens are never cacheable, and on a model where reasoning runs before the visible answer, the tokens spent thinking aren't either. A workload where most of the cost sits in long, thinking-heavy output rather than a large repeated input prefix will see a real but proportionally small effect from even a well-designed caching setup, because the majority of the bill was never something caching could touch in the first place.

The audit worth running before trusting a caching rollout

Rather than assuming a caching policy is working because it's switched on, the useful check is per-prefix, not integration-wide: for each distinct thing you're caching, is its actual observed reuse frequency — not its assumed frequency — clearing the point where the write premium gets paid back. A prefix that clears that bar comfortably deserves caching. One that doesn't is better left alone, and treating every candidate prefix as equally deserving of caching is exactly how a rollout ends up quietly worse than doing nothing, even while some individual prefixes inside it are performing exactly as intended.

Getting the actual number for your own setup

The break-even point itself — how many reads a write needs before it pays off, and how that differs between the two available cache lifetimes — is mechanical, model- and pattern-specific, and not worth re-deriving by hand. The prompt-caching savings calculator computes it directly against your real prefix size and request cadence, which is a more reliable starting point than applying a rule of thumb calibrated against someone else's traffic pattern.

Why "it's probably fine in aggregate" isn't good enough

A team that looks only at total spend before and after enabling caching can walk away with the wrong lesson either way — a modest net saving can hide a badly losing prefix offset by a strongly winning one, and a modest net cost increase can look like "caching didn't help" when in fact one specific prefix is actively losing money while the rest of the setup is fine. Aggregate numbers answer "did the bill move," not "which specific decision was right," and the second question is the one that actually tells you what to fix.

See reducing cost with prompt caching, in practice for the implementation side of getting this ratio right across a real integration, and the minimum-cacheable-prefix table for the separate, prior question of whether a given prefix even qualifies for caching before any of this math applies.