Six kinds of streaming (SSE) failure on relay sites, and how to pin down each one

Why streaming is the lie detector

In non-streaming mode a relay site only has to forward the upstream response body as one piece. That is easy to implement and easy to get right. In streaming mode it must parse the upstream SSE chunk by chunk and re-wrap it chunk by chunk. Any misreading of the format shows up in the output immediately.

So: to judge a relay site's engineering, look at streaming, not at non-streaming.

Failure one: duplicated tokens in the output

Symptom: individual words or characters repeat inside a passage, as in "we can we can use a".

Cause: an error in incremental bookkeeping. Some upstream implementations return cumulative text rather than an incremental delta. If the gateway treats that as a delta, it re-emits text it has already sent; treat a real delta as cumulative and it drops characters instead.

How to pin it down:

1. Send the same prompt twice, once streaming and once not, and compare the final text.
2. Non-streaming correct, streaming duplicated → the gateway's chunk handling is at fault.
3. Switch to a different upstream model. If only one model misbehaves, the fault is in that upstream's format adapter.

Verdict: a gateway implementation defect, not a model problem. Other models on the same site may be perfectly fine, so you only need to avoid the one that fails.

Failure two: broken multi-byte characters (mojibake or question marks)

Symptom: ? or the replacement character � in the output, or a multi-byte character split in half.

Cause: byte boundaries and character boundaries are not aligned. SSE chunks are cut on byte boundaries, and a UTF-8 Chinese character is 3 bytes while an emoji is 4, so a chunk boundary can easily land in the middle of one. If the gateway concatenates with byte-oriented logic, or calls substr somewhere along the way, it will cut multi-byte characters apart.

How to pin it down:

1. Ask the model for a mixture of prose, emoji and a code block.
2. Watch whether the corruption appears only at specific positions. Breakage tends to land in a fixed few places, because the chunking strategy is deterministic.
3. Use curl --no-buffer to look at the raw SSE stream. That tells you whether the gateway produced corrupted output or your client mangled it while parsing.

Verdict: a multi-byte handling bug in the gateway. It is a low-level implementation error, and its presence means the site's streaming path was never seriously tested.

Failure three: no end marker, or a missing [DONE]

Symptom: the client waits forever and never finishes, or reports that the stream ended unexpectedly.

Cause: one of two things. Either the gateway forwarded the data: lines but dropped data: [DONE], or the upstream uses a different end-of-stream convention (Anthropic emits a message_stop event) and the gateway never translated it into the client's [DONE].

How to pin it down: capture the raw stream with curl -N and look at the last chunk. Under the OpenAI protocol a well-formed stream must end with data: [DONE].

Verdict: an incomplete protocol conversion. If your client relies on the end marker to trigger follow-up work, saving automatically for instance, this breaks the feature outright.

Failure four: first-token latency far above the non-streaming average

Symptom: streaming turns out slower than non-streaming, and the first token takes a long time to arrive.

Cause: the gateway may be buffering the request, collecting the whole upstream response before forwarding anything, or performing auth, billing or routing lookups up front and doing them slowly.

How to pin it down:

1. Measure time to first token for streaming and non-streaming separately.
2. If streaming time-to-first-token is roughly the non-streaming total, the gateway is almost certainly buffering. That means "streaming" is only the shape of the endpoint, and nothing is actually being streamed.

Verdict: a functional gap, and a costly one for interactive uses such as code completion and conversation. The latency figures on our monitoring pages can help, but note that we measure homepage reachability, which is not the same as generation latency.

Failure five: silent mid-stream cutoff

Symptom: output stops halfway, no error is raised, the connection closes.

Cause: usually one of three.

• An upstream timeout: the gateway's connection timeout is shorter than the upstream's generation time.
• A per-request duration cap on the gateway that cuts the connection when it is reached.
• A gateway buffer limit tripped by long output.

How to pin it down:

1. Test with a long task, something that asks for 3,000 characters or more.
2. Record how long output had been running when it stopped. If the cut lands on a fixed duration, exactly 60 seconds for instance, the gateway's timeout configuration is almost certainly responsible.
3. Check your own client's read timeout. Many HTTP clients default to 30 seconds.

Verdict: rule out your own client timeout first, then look at the gateway's duration cap. Long-output workloads such as report generation and bulk code are sensitive to this one.

Failure six: claims to support streaming, returns a non-streaming format

Symptom: the request carries "stream": true, and the response is one complete JSON object.

Cause: the gateway ignored the parameter, or simply passed it through to an upstream that does not stream.

How to pin it down: check the response Content-Type. Streaming should be text/event-stream; non-streaming is application/json. Combine that with the timeout behaviour.

Verdict: a silently dropped parameter, and the most dangerous kind of all — nothing errors, the experience just gets worse, and the problem can go unnoticed for a long time.

A five-minute streaming check-up

Do these in order and you cover all six in one pass:

# 1) capture the raw stream to inspect format and end marker
curl -N -s https://<base_url>/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model":"<model>","stream":true,
       "messages":[{"role":"user",
       "content":"Produce an example with prose, an emoji and a short python snippet, at least 800 characters"}]}'

Look at four things:

1. Is every chunk a valid data: {...}?
2. Does it end with data: [DONE]?
3. Are multi-byte characters intact?
4. Output length and continuity — any repeats or jumps?

Then run the same prompt non-streaming and compare the final text.

Summary

Of the six streaming failures, only one can plausibly be the upstream's fault, and only in part: the buffering half of failure four. The rest are essentially all gateway implementation problems. So if a site streams badly, mark down its engineering and do not spend more time hunting for a reason.

Conversely, good streaming behaviour is a strong positive signal. It means the site at least tested its own protocol conversion layer seriously.