What caching actually saves: a reproducible side-by-side test

What caching saves

LLM billing is input plus output. In a long conversation or a document Q&A session, input is often far larger than output. The classic cases:

• A very long system prompt: several thousand tokens of role definition plus tool definitions.
• Multi-turn conversation, where the history is resent every turn.
• Document Q&A, where the same document is asked about repeatedly.
• An agent loop, where every step carries the full context.

In all of these, a large share of the input is exactly repeated content. Caching keeps that portion resident on the backend and reuses it next time at a lower price.

A typical billing structure, using the Claude family as the example:

• Cache write: slightly more expensive than ordinary input, roughly 1.25x.
• Cache read: much cheaper than ordinary input, roughly 0.1x.
• Cache lifetime: usually 5 minutes, with longer tiers available.

So for workloads that reuse one long prefix, savings can reach 70% to 90%.

What caching requires

Three conditions, all necessary:

1. The client actually sends a cache marker. Anthropic uses a cache_control block; OpenAI-family models mostly cache automatically without a marker.
2. The relay site passes the marker through to the upstream. This is the link most likely to be broken.
3. The upstream supports it and the cache actually hits, which requires the prefix to be byte-for-byte identical.

When a relay breaks at link 2, the request returns normally, billed at full price. You receive no error at all.

How to tell whether caching is really working

The only reliable method is to read the returned usage field.

With caching enabled, usage gains cache-related counters:

"usage": {
  "input_tokens": 120,
  "cache_creation_input_tokens": 4200,
  "cache_read_input_tokens": 0,
  "output_tokens": 250
}

Send the same request a second time and you should see:

"usage": {
  "input_tokens": 120,
  "cache_creation_input_tokens": 0,
  "cache_read_input_tokens": 4200,
  "output_tokens": 250
}

Criteria:

• cache_read_input_tokens > 0 appears → caching genuinely works.
• The fields are entirely absent → the relay did not pass them through. It may have recomputed usage itself.
• The fields exist but cache_read is always 0 while cache_creation is large every time → writes succeed but never hit. The prefix may be inconsistent, or the site routes to a different backend each time.

A reproducible side-by-side test

Preparation: one fixed prefix longer than the cache minimum. Note that the minimum differs by vendor, 1024 tokens or more in some cases, and anything shorter is not cached.

PREFIX="<repeated long text, ensure > 1500 tokens>"

# first call: you should see cache_creation
curl -s .../v1/messages -H "x-api-key: $KEY" -H "anthropic-version: 2023-06-01" \
 -d "{
   \"model\": \"<model>\",
   \"max_tokens\": 100,
   \"system\": [{\"type\":\"text\",\"text\":\"$PREFIX\",
                 \"cache_control\":{\"type\":\"ephemeral\"}}],
   \"messages\": [{\"role\":\"user\",\"content\":\"Reply with OK only\"}]
 }" | grep -o '\"usage\".*'

# immediately again, within 5 minutes: you should see cache_read

The key: the gap between the two calls must be shorter than the cache lifetime, 5 minutes by default.

Three reasons caching might not work

One: the relay does not pass cache_control through

The most direct cause. How to judge: look at whether cache-related fields appear in the returned usage. If the field is missing entirely, that is close to conclusive.

The knock-on effect: you also lose the pricing structure where writes cost more and reads cost less, meaning you are billed at full price, and you have no idea.

Two: the prefix you send differs each time

A cache hit requires the prefix to be byte-for-byte identical. Common ways to break it:

• the system prompt contains the current time or a random ID;
• history is truncated to a different length each time, for instance by dynamic token-based trimming;
• serialisation order is unstable, with JSON key order changing.

How to check: send the same prefix twice in a row. If the second does not hit, first confirm the prefix bytes are truly identical.

Three: the site routes each request to a different backend

This one is specific to relay sites. If a site maintains an account pool or channel pool and assigns each request at random, the cache may land on a different backend than the one holding your last write.

Signature: cache_creation is present every time, cache_read appears occasionally but inconsistently.

There is essentially no fix for this, short of the site implementing session affinity, pinning one session to one backend. So if you depend heavily on caching, "does it support session or channel affinity" belongs on your evaluation list.

Converting it into money

A conservative estimate, using the usual Claude-family ratios:

Conclusion: caching helps with a long fixed prefix plus repeated calls, and does nothing for short conversations. If your usage is not the former, do not spend effort here.

One practical recommendation

If you depend heavily on caching, put it on your selection checklist and ask the site to confirm in writing. This is not a capability you can establish by experiment, because what you demonstrate is "it worked this time", not that it will hold.

Separately, whether or not caching works, always log usage in your code. Without usage you have no ability to reconcile, and caching is precisely the line item most easily erased from a bill without anyone noticing.