How to check whether a relay service is quietly downgrading your model

Why downgrading is so hard to catch

Because the API response structure is standardised. The JSON that comes back has a model field. A fraudster writes claude-opus-5 into it and the job is done — your program never raises an error, because the format is exactly correct.

So detection cannot look at metadata. It has to look at behaviour.

Test one: compare token counts (fastest and strongest)

Different models use different tokenisers, so the same passage of text produces noticeably different token counts.

How:

1. Prepare a fixed passage of text, about 2,000 characters
2. Send it once to the vendor endpoint and once to the service under test, asking for the text back unmodified
3. Compare the usage.prompt_tokens values

How to read it: if it is the same model, the two numbers should be very close — a single-digit percentage apart. A gap of 30% or more effectively settles that the model or the tokeniser has changed.

This is the cheapest test with the hardest conclusion. Use it as your first filter.

Test two: a difficulty gradient

How: prepare three sets of questions, increasing in difficulty:

• Easy: general knowledge, simple arithmetic
• Medium: multi-step reasoning, finding bugs in code
• Hard: mathematics needing a long reasoning chain, complex refactoring

How to read it: if the easy set is perfect, the medium set starts to fall apart and the hard set fails entirely, the backend is a small model. The gap between a flagship and a lightweight model on hard questions is a cliff, not a slope — you will not see "roughly comparable".

One caveat: never conclude from a single question. Use eight to ten and look at the accuracy rate, so randomness does not decide it for you.

Test three: output style fingerprint

The same model handling the same prompt is highly consistent in style, including:

• whether it habitually uses subheadings
• whether lists use - or *
• whether code blocks carry a language tag
• whether answers open with a stock lead-in sentence

How: run the same prompt five times against the vendor and five times against the service under test, and compare those features. When a model is swapped, the habits usually shift visibly.

This is less reliable than the first two, but useful as supporting evidence.

Test four: concurrency and latency behaviour

How: measure behaviour at different concurrency levels (one versus ten), recording success rate and time to first token.

How to read it:

• Time to first token stuck around 200–400ms with very long output → suspicious (flagship models are usually slower)
• Heavy 429s or timeouts as soon as concurrency rises → their capacity comes from a shared pool, which usually points to resold subscriptions or reverse-engineered endpoints

Test five: is the long context real

How: send a document of around 50,000 tokens and ask about a detail near the end.

How to read it: cheaper models often have smaller context windows, or silently truncate (processing only the first tens of thousands of tokens). If it cannot answer about the second half of the document, the context capability does not match what was advertised.

A checklist you can copy

[ ] Token count comparison  — a gap over 30%: walk away
[ ] Difficulty gradient      — accuracy across easy / medium / hard
[ ] Output style fingerprint — compare against the vendor
[ ] Concurrency test         — success rate at 10 concurrent should be > 95%
[ ] Long context test        — a detail near the end of 50k tokens
[ ] Bill reconciliation      — does usage match what was deducted

How to reach a conclusion

One anomaly on its own could be network noise or sampling luck. Two or more, and you stop using the service.

It is also worth separating "misrepresentation" from "tiered routing". Some services openly distinguish a standard route from a premium one, and the standard route uses a constrained channel. As long as they tell you so, that is ordinary commercial tiering, not fraud. The real problem is the ones advertising "full-strength official model" while actually routing elsewhere.

How our data helps

We probe each listed service's /v1/models endpoint and record the model count and list that actually comes back, and services that pass are labelled "endpoint confirmed". That does not prove a model has not been downgraded, but it does filter out the services whose endpoint itself is fake.

If you find evidence of misrepresentation, send it through submit a correction. We verify and act.