Give a relay service a health check: a 20-minute verification checklist

Why a fixed order matters

Most people test a relay service in whatever order the problems occur to them, which means the cheapest and most decisive checks happen last — or never.

This list runs from the checks that take seconds to the ones that take minutes, and from those that eliminate a service outright to those that only raise a flag. If a service fails an early one, stop; there is no point continuing.

Total time is around twenty minutes, most of it waiting on responses.

Step 1: does the endpoint behave like a standard API (2 minutes)

What to do: point a standard OpenAI-compatible client at their base_url and call /v1/models.

What you are looking for:

• the endpoint returns a model list rather than an error
• the shape is standard OpenAI format, not something proprietary
• the models you actually need are in the list

Why it is first: it costs one request and it eliminates services whose endpoint is decorative. A service that fails here is not worth another minute.

Our service detail pages record the model list our own probe retrieved, and label services whose endpoint responded as confirmed. That is where to start if you want to know which services pass this step before you sign up.

Step 2: one basic call on the model you care about (2 minutes)

What to do: send a short prompt to the specific model you intend to use.

What you are looking for:

• a coherent answer at all
• the usage block is present and populated
• the response time is plausible for that model tier

Common failure: the service routes every request to one cheap backend regardless of the model name in the request. A single basic call on an expensive model will not prove anything on its own, but a call that returns in 300ms with a long answer on a flagship model is a flag.

Step 3: tool use — the check people skip and regret (5 minutes)

What to do: give the model a task that cannot be completed without calling a tool, and see whether a tool call comes back.

Why it matters most: this is the single most common reason a relay service is "connected but not usable". Many services proxy only the chat endpoint and do not forward tool calls. Everything looks fine until you run a coding client, which then sits there producing text and never editing a file.

If you are going to use Claude Code, Cline, Roo Code or any agent framework, this step is not optional. Test it before you commit, not after.

Step 4: token count comparison (3 minutes)

What to do: send the same fixed passage of about 2,000 characters to the vendor endpoint and to the service, asking for the text back unmodified, and compare usage.prompt_tokens.

How to read it: the same model should produce counts within a few percent of each other. A gap above 30% means the model or the tokeniser changed.

Why it is worth doing early: it is the cheapest hard evidence you will get about whether the model behind the name is the model you asked for.

Step 5: concurrency (5 minutes)

What to do: run ten concurrent requests and count the success rate.

How to read it:

• success rate above 95%: the tier is usable
• heavy 429s or timeouts: their capacity comes from a shared pool

Why it matters for cost: a cheap tier that collapses under modest concurrency is not actually cheap. The real cost is the tier you have to upgrade to.

Step 6: reconcile one bill (3 minutes)

What to do: make one call of known size, work out the expected charge from the vendor list price and the advertised rate, and compare it with what was deducted.

How to read it: a mismatch means the advertised rate is not what is being applied. This is the check that catches the most common form of quiet overcharging.

Step 7: long context, if you need it (5 minutes)

What to do: send a long document and ask about a detail near the end.

How to read it: if the model cannot answer about the back half, the usable context is smaller than advertised, or requests are being silently truncated.

Skip this one if your workload never goes long.

The condensed version

[ ] /v1/models responds in standard format
[ ] one basic call on the target model returns a usable answer
[ ] tool use actually comes back  ← do not skip this one
[ ] token counts within a few percent of the vendor
[ ] 10 concurrent requests: success rate > 95%
[ ] one bill reconciles against the advertised rate
[ ] long context (only if you need it)

What to do with the result

Any check that eliminates the service outright — endpoint shape, tool use, billing mismatch — stop there.

Flags that are worth noting but not disqualifying on their own: high latency, concurrency limits on the cheap tier, a long-context ceiling. Those affect which tier you buy, not whether to buy.

And whatever the checklist says, start with the smallest top-up. Twenty minutes of testing reduces the odds of a bad choice; it does not replace the discipline of not putting more on the account than you would be willing to lose.