Prompt caching on a relay service: does it actually work, and what is it worth

Why caching is worth this much attention

For any workload with a fixed prefix — a long system prompt, a policy document, a codebase summary — caching is usually the single largest cost lever available, larger than switching providers.

The reason is the size of the discount. Vendors commonly charge around 10% of the standard input price on a cache hit. On a workload that is mostly a repeated prefix, that is not a marginal saving; it changes the order of magnitude.

The catch: the discount belongs to the vendor, and your request is going through a relay service. Whether it reaches you is a separate question.

What has to be true for you to get it

Three conditions, all of which have to hold:

1. The vendor supports caching on that model. Not all do, and the details differ by vendor — some require an explicit cache marker, some cache automatically on long prefixes.
2. The relay forwards the request faithfully. A relay that rewrites the request body, strips unknown fields, or reshapes the prompt will break caching without telling you.
3. The relay applies the discounted price in billing. This is the one most often missed. Even a relay that forwards caching correctly may price your tokens from the standard input rate, quietly keeping the discount as margin.

Condition three is why "does the relay support caching" is the wrong question. The right one is "does my bill reflect the cache hit".

How to test it in about ten minutes

You can find out for yourself without asking the provider anything.

The test:

1. Send a long, identical prompt (say 20,000 tokens) twice in a row
2. Record the usage block from both responses
3. Compare the token fields, and the resulting deduction

What you are looking for: the vendor's API typically reports both total input tokens and a cached-token count. If the cached count appears on the second call, the vendor cached it and the relay forwarded it. Then watch the deduction — that is the part the relay controls.

If the cached-token field is zero on the second call, caching is not reaching you. Either the request is being reshaped on the way through, or the vendor does not cache that model.

Common ways caching breaks in a relay chain

• Request reshaping. The relay normalises the body and drops fields the vendor needs for caching.
• Prompt rewriting. Some relays prepend their own instructions or a safety wrapper. Any change to the prefix, including one character at the start, invalidates the cache.
• Routing across upstreams. If consecutive requests land on different upstream accounts, they cannot share a cache. This is common at aggregators with several redundant routes.
• Different accounts. Resold subscription channels may spread your requests across accounts, which has the same effect.

The third and fourth are structural: they are a consequence of how the service is built, and asking them to change it is asking them to change their architecture.

What it is worth, concretely

Take a workload with a 10,000-token fixed system prompt and 1,000 tokens of variable input per call, at $3 per million input tokens.

Without caching:

• 11,000 tokens per call × 100,000 calls = 1,100,000,000 tokens
• at $3 per million = $3,300

With caching, assuming a 95% hit rate on the fixed prefix:

• cached: 9,500 × 100,000 = 950,000,000 tokens at $0.30 per million = $285
• uncached: 1,500 × 100,000 = 150,000,000 tokens at $3 per million = $450
• total = $735

The saving is $2,565, or 78% of the input cost. Set against a relay rate moving from 0.4 to 0.35, which would save 12.5% of a much smaller number.

This is why the order of optimisation matters: fix the prompt structure before shopping for rates.

What to do about it

If you control the prompt, make caching possible: keep the fixed part genuinely fixed, put it first, and move anything that changes (timestamps, user names, request IDs) to the end. A timestamp at the top of a system prompt is the most common accidental cache-killer.

If you do not control the prefix — say a framework builds it for you — check whether the framework exposes cache control, and whether the relay passes it through.

If the relay does not pass the discount on, work out what it costs you before switching. In the example above, a relay that keeps the cache discount is charging you four times what it needs to for input. That is worth moving for, provided the replacement is stable.

Related reading

• From list price to your invoice: a worked cost conversion (guides)
• 2026 model API pricing review: who actually got cheaper (guides)
• How to choose an aggregating relay service (guides)