Evening peak jitter: upstream, network path, or your own concurrency

Separate four possibilities first

"Slower in the evening peak" has four possible layers of cause, and they call for completely different responses:

The order matters: start with the cheapest layer and the one most likely to yield a conclusion.

Step one: baseline with a minimal request, 2 minutes

Method: send a minimal request with max_tokens: 1 and record the elapsed time. Run it ten times.

Why a minimal request: its duration is determined almost entirely by network and queueing, independent of generated content length. What you are measuring is the channel baseline.

Criteria:

• Small requests are fast, under 2s, while your real workload is slow → the problem is not the channel but the generation stage: heavy upstream load, an over-long prompt, or an over-long output.
• Small requests are also slow → the problem is in the channel: queueing, the network path, or your concurrency.

This step alone cuts four possibilities down to two.

Step two: find the capacity knee with a concurrency ramp, 5 minutes

Method: run ten requests each at concurrency 1, 2, 4, 8 and 16, recording the median duration and failure rate for each group.

concurrency 1  -> median 1.2s   failures 0%
concurrency 2  -> median 1.5s   failures 0%
concurrency 4  -> median 2.8s   failures 0%
concurrency 8  -> median 9.1s   failures 12%   <- the knee
concurrency 16 -> median 22s    failures 40%

Criteria:

• A clear knee, between 4 and 8 above → this is the capacity ceiling. The cause may be upstream rate limiting, the relay's queue, or your own egress bandwidth.
• No knee, with latency rising linearly with concurrency → requests are being serialised in a queue. This is the classic "upstream or relay is queueing" pattern.
• Slow even at concurrency 1 → not a concurrency problem; the channel itself is the problem.

Key point: record the knee. It is the practical concurrency ceiling for this path. Production concurrency should sit at 50% to 70% of that value.

Note: this step consumes some quota, though a minimal request with max_tokens: 1 costs very little.

Step three: rule out the path with a second location, 3 minutes

If the first two steps both point at the channel, you need to separate the network path from the relay.

Method: probe the same endpoint simultaneously from two different network locations, such as your local machine and a cloud server.

Criteria:

• Both locations slow → the problem is the relay or the upstream.
• Only your location slow → the problem is your own egress, whether carrier congestion at peak hours or your proxy path.

Why this step matters: if the problem is your network path, changing relay sites is wasted effort.

An easier alternative: if you can reach the site's static assets, its homepage and documentation pages, measure their load time.

• Static assets slow too → a network path problem.
• Static assets fast, API slow → a relay or upstream problem.

This method needs no second machine and is very practical.

What to do about each cause

Upstream model service

Signature: every site on the same upstream slows at once; small requests are slow; slow from multiple locations.

Response: change site, moving to one on a different upstream, or shift your schedule. There is nothing else you can do, because it is an upstream capacity problem.

Network path

Signature: only certain network locations are slow; static assets are slow too; latency rises while success rate holds.

Response: change node, change route, or adjust your proxy configuration.

The relay site itself

Signature: the concurrency knee is very low, degrading badly past 4 for instance; different models on the same site perform consistently poorly; other sites are fine.

Response: reduce concurrency below the knee and add backoff retries, or change site.

Your own concurrency

Signature: packet captures or logs show a burst of requests in a short window; 429s are frequent.

Response:

1. Add a concurrency cap, enforced with a semaphore.
2. Add exponential backoff: wait 2s, 4s, 8s after a failure rather than retrying immediately.
3. Add jitter, so not every request retries at the same instant.

Item 3 is the most often overlooked. If ten requests fail together and retry together, you get a retry storm that turns mild rate limiting into total failure.

One factor easily overlooked: timeout settings

Peak-hour latency is genuinely higher. If you set your timeout to just about the off-peak latency, the evening peak will produce a wave of timeout errors — while the requests themselves may have succeeded, processed and billed upstream, with you simply not waiting long enough.

What to do: set the timeout to p99 latency times two, not to the average. Better to wait longer than to misjudge.

The practical impact here is large: many "evening peak outages" are really unreasonable timeout settings rather than a service that is genuinely unavailable.

A ten-minute diagnosis checklist

[ ] 1) Minimal request x10, check whether the channel baseline is normal
       - fast -> the problem is in generation (prompt/output too long, or upstream load)
       - slow -> continue to step 2
[ ] 2) Concurrency ramp 1->16, find the knee
       - knee found -> record it, that is your concurrency ceiling
[ ] 3) Static assets versus API comparison
       - static also slow -> network path problem
       - static normal -> relay or upstream
[ ] 4) Check whether your timeout is >= p99 x 2
[ ] 5) Check for a retry storm, meaning simultaneous retries

Work through those five and the cause is essentially localised to one layer. Nothing in the process needs specialist tools; curl and time are enough.

One last point

The "response latency" on our monitoring page is a measure of homepage reachability. It is not time to first generated token, and it does not reflect peak-hour queueing. The methodology is documented in the article on how availability rates are measured.

Its use is comparison, showing which site is faster overall. It cannot predict whether a given site will be slow tonight. That only your own probe can tell you; see the article on building your own probe.