Reducing cost with prompt caching, in practice

Caching is a design decision, not a checkbox

Turning caching on doesn't, by itself, reduce cost — what reduces cost is a request shape that genuinely has a large, stable, frequently-reused prefix, structured so that stable content sits ahead of a cache breakpoint. Teams that enable caching without restructuring their actual requests around a stable prefix often see negligible savings, not because caching doesn't work, but because their requests were never shaped to benefit from it in the first place.

The highest-leverage place to look first

For most integrations, the system prompt is the single largest, most stable piece of a request, and it's the first place worth auditing for caching potential — if it's genuinely static across requests (not personalised per user, not carrying a timestamp or session-specific value), it's close to a best-case candidate for a cache breakpoint. Where a system prompt does need to vary, separating the parts that are genuinely stable from the parts that aren't, and structuring the request so the stable portion comes first, usually unlocks caching for the majority of the prompt even when the whole thing can't be cached as one block.

Retrieved context is a harder case, and worth solving deliberately

Retrieval-augmented workflows — pulling in documents, search results, or other dynamic context per request — are a much harder caching target than a static system prompt, because the retrieved content is, by design, different on every request. Where retrieved context repeats across requests within a short window (the same document referenced by several consecutive queries, say), restructuring the request so retrieved content sits in a stable position relative to a breakpoint, rather than being freshly interleaved with the query each time, can recover some caching benefit even on inherently dynamic content.

Multi-turn conversations compound the opportunity

In a multi-turn conversation, everything from earlier turns is, by definition, already fixed and unchanging — which makes a long-running conversation one of the best natural fits for caching, since the growing prior history is exactly the kind of large, stable prefix caching rewards. A chat application that isn't caching the conversation history on each new turn is paying full price to re-process content that hasn't changed since the previous turn, on every single turn of every conversation.

Tool definitions are often the easiest overlooked win

Teams that carefully cache a system prompt sometimes overlook that tool definitions sit even earlier in the request and are just as cacheable when they're stable — a large tool surface that's identical across requests is exactly the kind of content caching rewards, and it's easy to miss specifically because it doesn't feel like "the prompt" the way the system instructions do. If your tool list is large and doesn't vary per request, confirm it's actually inside your cached prefix rather than assumed to be, since a tool list rebuilt dynamically for any reason invalidates everything after it in the request.

Measuring the actual saving, not assuming one

Caching is not a one-time setup you configure and then forget — a prefix that was stable when the caching structure was designed can quietly stop being stable as the surrounding product evolves, a new field gets added near the top of the prompt, or a personalisation feature ships that touches what used to be static content. Revisit the actual structure periodically rather than trusting that whatever earned savings on day one is still earning the same savings months later.

Once a caching structure is in place, verify the saving against real usage rather than assuming the theoretical benefit is being realised — a caching setup that looks correct in code can still underperform in practice for reasons that produce no error, only a lower-than-expected cache-hit ratio. See cache_read_input_tokens is 0 with no error for the specific failure modes worth checking, and the prompt-caching savings calculator to model the expected saving for your actual request pattern before and after a restructuring effort, so you have something concrete to check the real result against.