Prompt caching in production: why your cache hit rate is zero

Every team I've worked with turned on prompt caching and assumed it was working. Most of them had a cache hit rate near zero, because caching is a prefix match and something in the request was changing on every call. Here is how to find that something and fix it.

How caching actually works

The API caches a prefix of the request and reuses it when a later request starts with the exact same bytes. The prefix is built in a fixed order: tools, then the system prompt, then messages. Any change anywhere before the cache breakpoint invalidates everything after it. There is also a minimum cacheable length, so tiny prompts never cache at all.

That means caching is a property of how you construct requests, not a flag you turn on.

The silent invalidators

These are the ones I find over and over.

• A timestamp in the system prompt. "The current date is …" at the top of the prompt guarantees a miss every day, and if it includes the time, every minute. Move it to the end of the user message.
• Per-user values near the top. The user's name, plan or ID rendered into the system prompt. Put identity in a block after the cached prefix, or pass it as a tool result.
• Unstable tool lists. Tools loaded from a database in whatever order the query returns. Sort them, and keep the set stable per route.
• Unsorted JSON. Serialising a settings object with keys in insertion order that differs between processes. Sort keys before you embed.
• Editing history. Rewriting or trimming earlier turns before each call. Append only; summarise into a new message instead of editing old ones.
• A/B prompt variants chosen per request. Each variant is its own cache. Fine if intentional, fatal if it is a random assignment.

A structure that caches

Order the request from most stable to least stable:

1. Tool definitions, sorted, identical per route.
2. A frozen system prompt: instructions, policies, retrieval guidance. No dates, no names.
3. Cache breakpoint here.
4. Retrieved context and conversation history, appended, never rewritten.
5. The current user message, with any volatile values (date, user state) placed last.

Keep at most a few breakpoints, and put the one that matters at the end of the frozen section.

Measure it or it is not real

The response usage reports cached input tokens separately. Log it per route. If the cached count is zero across repeated requests, you have an invalidator. If it is non-zero on the first request of a conversation and zero afterwards, you are editing history.

On the REConnect support agent, fixing three invalidators (a timestamp, a per-user block and an unsorted tool list) took an afternoon and cut input cost by more than half. Nothing about the model changed.

When not to bother

Short, one-shot requests below the minimum cacheable length. Prompts that genuinely change on every call. Batch jobs where latency and cost are already handled by the batch discount. Everywhere else, assume you should be caching, and assume you currently are not.