We probed 194 sites: what this data can and cannot answer

Start by being clear about what the data is

Our data comes from three sources with entirely different natures:

We deliberately present these two categories separately, for reasons set out in the announcement on why advertised and measured values are kept apart. It bears repeating now: if they were mixed, you would assume the multipliers were measured too.

What it can answer, four kinds of question

One: comparison, which site is faster and more stable overall

This is what the data does best.

Because the 194 sites were measured under one methodology, from one probe location, in one time window, comparison across them is valid. Relative relationships are more reliable than absolute values.

Judgements it supports:

• Site A's availability is clearly higher than site B's.
• Site C's latency has been consistently lower than comparable sites over time.
• A given site's stability this month is noticeably worse than last month.

Two: trends, whether a site is improving or deteriorating

Availability and consecutive available days are time series, so trend judgements are valid.

This is where the practical value is greatest. A site's "pre-shutdown warning signs" usually show up as declining availability; see the article on the six signals. Our data gives you an objective observation point.

Three: coverage, the breadth of model supply

The "model count detected by probing" reflects each site's model coverage.

Judgements it supports:

• Site A's model list is richer than site B's.
• Which sites carry a particular niche model.

A judgement it does not support: whether every model in the list actually works, or how good it is.

Four: price baselines, judging whether a multiplier is reasonable

Our official price table at /model/lists is a third-party baseline independent of any relay.

Judgements it supports:

• What a site's advertised 0.3x multiplier amounts to in actual payment.
• Which band that price falls into relative to official pricing; 30% to 70% is the usual range for official resale, and below 30% warrants caution.

What it cannot answer, five kinds of question

One: whether a model is genuine

We do not verify models. We do not run model fingerprinting or parameter boundary tests.

Strictly speaking, "24 models detected by probing" means: /v1/models returned 24 entries. It does not mean all 24 work, still less that they are the models they claim to be.

So verification is something you must do yourself, or through a platform dedicated to it.

Two: how good the output quality is

We do not measure output quality. There is no quality score and no benchmark.

The reason is straightforward: quality depends heavily on your specific task. The same model may be excellent at summarisation and poor at code. Any universal quality score would mislead.

Three: time to first token

What we measure is site reachability and homepage response, not the time to the first generated token.

The two differ substantially. A site may have a fast homepage and slow generation, because of heavy queueing, or a slow homepage and fast generation.

So our latency figures cannot tell you whether a site feels fast in use. That you have to measure yourself; see the build your own probe article.

Four: capability support

We do not measure: streaming implementation quality, tool-call compatibility, structured output, multimodal, cache billing, speech, image generation.

All of these you have to verify yourself. Our tutorials on the six streaming failures, tool use compatibility, JSON mode and cache billing exist precisely so that you can.

Five: which site is best

We do not publish an overall ranking.

There are three reasons:

1. Different scenarios have different optimal answers; see the selection decision tree.
2. Rankings encourage a "look at the position, not the conditions" way of using them.
3. We have commercial interests in listings and advertising, and publishing a ranking would be a conflict of interest.

Three reminders about the methodology

1. A single probe location

Every probe goes out from one server. That means:

• results may differ from what you see reaching the site from your own network;
• if your users are spread across regions, our data cannot replace multi-node monitoring.

2. Limited probe content

We probe reachability and response time using one minimal request.

It cannot reflect peak-hour queueing, the handling of large requests, or differences between models.

3. How availability is defined

Availability is the proportion of successful requests, subject to a minimum sample size. Sites with too few samples are excluded from ranking, to avoid the distortion of "one success equals 100%".

The specific thresholds and windows are documented in the article on how availability rates are measured. If you are going to make a decision with that number, read that article first.

How to use this data

In one line: use it as a sieve, not as a conclusion.

Correct usage:

1. Use availability and latency for an initial filter, narrowing 194 sites to around 10.
2. Use model coverage and multipliers to narrow further to three to five.
3. Then verify among the candidates yourself: capability tests, small reconciliation, concurrency ramp, one business cycle of observation.

Incorrect usage:

• "100% availability means it must be good" — it may simply have few samples.
• "First place is the best" — the ordering rests on a single dimension.
• "It advertises 0.1x so it is a bargain" — that is a self-declared value, and a low price has a reason in the cost structure.

One last sentence

Our purpose in providing the data is to lower your filtering cost, not to decide for you. Layers three and four, verification and strategy, can only be done by you — and they are not expensive. Our tutorials together amount to about two hours of reading, which covers most of the ground.

The genuinely time-saving approach: use the data to shorten your candidate list, then spend the time you saved on verification rather than on comparing prices again.