A relay service starts erroring or slowing down: how to tell whose problem it is

Why guessing wastes money

When calls start failing, the instinct is to blame the relay service and switch providers. That is sometimes right and often wrong — and switching costs you the balance on the old account plus a round of migration work.

The failure can sit in four places. Each one has a different symptom pattern, and localising it takes a few minutes.

The four layers

Step 1: is it every model or one model

This single question splits the space in half.

Every model failing → look at your client first, then the relay service. A credential problem, a wrong base URL or a service-wide outage all look like this.

One model family failing → almost always upstream. The relay's route for that family is down while the rest of its capacity is fine.

Step 2: does the same request work against the vendor directly

If you have any vendor access at all, run the same prompt through it.

• Fails on the vendor too → the model provider is having a bad day. Nothing to fix locally; wait, or fall back to another model.
• Works on the vendor, fails on the relay → the problem is on the relay side or its upstream route.

This is the most decisive test available and it takes one call.

Step 3: read the status code properly

The code tells you which layer to look at.

The distinction between 429 as yours versus theirs: if you are well under your plan's limit and still getting 429, the cap is on their side or upstream.

Step 4: check whether it is latency or failure

These have different causes.

Latency only: time to first token rises while success rate stays high. Usually upstream congestion or the service routing you to a slower channel. Check whether it correlates with peak hours.

Failure only: requests come back as errors quickly. Usually a route or credential problem.

Both at once, across several models: this is the pattern that most often precedes a service going under. Worth watching, not worth panicking over on one occurrence.

Step 5: is it intermittent or continuous

Continuous failures on one model → the route is down. Report it; if it persists more than a day or two, start using your backup.

Intermittent failures with no pattern → most often the relay's capacity. Retries with exponential backoff absorb a lot of this. If your retry logic has no backoff, adding it may fix the symptom entirely.

What to do about it, in order

1. Check your own client first. Expired key, wrong base_url, request body shape. This is embarrassing to discover after blaming the provider, and it is common.
2. Retry with backoff. Transient 500s and 429s are normal on any relay and any direct vendor. Absorbing them costs nothing.
3. Have a backup configured. Two services, switchable by environment variable. This is the single most effective protection against a bad day.
4. Implement model fallback. If the flagship is unavailable, degrade to a mid-tier model rather than failing the request. Most workloads can survive this.
5. Then consider switching. Only after the checks above point at the service.

What not to do

• Do not conclude from one failed request. Single failures happen everywhere.
• Do not top up more to "unlock a better route" in the middle of an incident. That is exactly the moment the sales pitch is designed for.
• Do not hard-code the model name. Keep it in configuration so you can degrade without a deploy.

Related reading

• Give a relay service a health check: a 20-minute verification checklist (guides)
• How we measure uptime: our methodology and its limits (guides)
• Six signals that a relay service is about to disappear (guides)