Context length and truncation: why relay sites error out sooner than you expect

Separate three numbers first

"Context length" conflates three different quantities:

1. Input ceiling: how many tokens the messages array can hold in one request.
2. Output ceiling (max_tokens): how many tokens this generation may produce.
3. Total window: the number that input plus output must not exceed.

Official documentation usually quotes the total window, 200K for instance. But the input ceiling is normally lower than the total window, because room has to be left for output. This is where many relay sites fail first: they advertise the total window, and then error out the moment your input goes over.

Five ways relay sites limit context

One: per-channel caps (reasonable)

Every upstream channel has its own real ceiling. A gateway that enforces the limit per channel is doing the right thing. You go over, you get an error, and you know why.

Two: silent truncation (dangerous)

When a request exceeds the limit, the gateway quietly drops the earliest messages so the request goes through.

The symptom is that nothing errors, but the model has "forgotten" the start of the conversation. Mid-conversation it starts producing amnesia-style lines like "that thing you mentioned earlier".

This is the hardest class to detect, because there is no error signal. Test it by planting something only the opening message knows, a random number for instance, and asking the model to repeat it some turns later. The point at which it can no longer repeat it is your truncation point.

Three: counting max_tokens against the input ceiling (a common mistake)

Some implementations get the arithmetic wrong: they compare "input tokens plus max_tokens" against the input ceiling, so the request errors out before the input itself is anywhere near the limit. If a site errors at an input size well below what it advertises, this is a candidate.

How to verify: hold the input length fixed and only increase max_tokens. If the error appears as max_tokens grows, this is what you are looking at.

Four: gateway timeouts (an indirect limit)

Long contexts take longer to process. If the gateway's timeout is shorter than the upstream's response time, long requests return 504 — which looks like "long context is unsupported" but is really a timeout.

How to verify: keep the input the same length and swap in a simpler question so the output is shorter. If that succeeds, it is a timeout.

Five: intermediate buffer limits

Settings such as client_max_body_size and proxy_read_timeout on a gateway or reverse proxy limit the request body size indirectly. These limits usually sit around the 1MB mark, and 200K tokens of text is far past 1MB.

This is the one people overlook most. It is especially common in self-hosted setups: the application supports 200K, but nginx's default client_max_body_size 1m returns 413 immediately.

How to measure the length you actually get

Do not trust the advertised figure. Run a step test.

Preparation: a block of filler text whose length you can control precisely. Repeating a fixed English sentence is easiest, because token density stays stable.

Method: start at 10K tokens and add 10K each time, sending the same simple question with max_tokens pinned at 100. Record:

The key is to record the status code.

• 400 with an explicit token-limit message → a channel ceiling, working as intended
• 413 → request body too large, so the reverse proxy layer is limiting you
• 504 or a timeout → a gateway timeout, not a length limit
• 200 every time but the answer is missing content → silent truncation

Four outcomes, four completely different responses.

Why this is worth doing

Because "long context" is the metric you should least take on trust, and it happens to be critical to a lot of workloads:

• Codebase analysis: several files fed in at once.
• Long document processing: contracts, reports, papers.
• RAG context assembly: retrieved passages concatenated together go over the limit easily.
• Long-running conversational agents: history accumulates.

Once a workload depends on a wrong length assumption, it fails in production as an intermittent error, and intermittent failure is the most expensive kind.

Three practical conclusions

1. Treat the advertised context length as a ceiling reference only. Measure what you actually get, and remember that what you measure is "the usable length on this gateway over this path". Change sites and you have to measure again.
2. Prefer sites that give clear errors. A site that tells you plainly "you exceeded X tokens" is far more reliable than one that truncates silently. With the former you can do something; with the latter you can only step on the rake.
3. If your workload genuinely needs long context, rank this above price. Price can be negotiated. Length either works or it does not.

Our detail pages list the model count detected by probing and the verification status, but they do not include a measured context length — the result depends too heavily on the shape of the caller (message structure, whether images are present), and a single universal figure would mislead. Measure this one yourself, using the method above.