Three traps in choosing a site from a leaderboard alone

Trap one: treating a single-dimension ranking as a composite judgement

How it shows up: seeing an "availability leaderboard" and concluding that the top site is best overall.

The problem: high availability only means it was not down at the moment it was probed. It may also be:

• very high latency, meaning available but slow;
• very expensive;
• missing the capabilities you need, such as caching, tool calling or structured output;
• narrow in model coverage.

Four dimensions are mutually independent: availability, performance, cost and capability. A ranking on any one of them cannot yield a composite conclusion.

Correct usage: be clear about which dimension you are looking at. If you want availability, look at the availability table; if you are choosing a supplier, assemble the four dimensions yourself. The act of assembling them is the selection process, and it cannot be outsourced to a leaderboard.

Trap two: ignoring sample size and timing

How it shows up: seeing "100% success rate" and concluding the site is reliable.

The problem: 100% out of how many measurements?

• 10 successes giving 100% → statistically almost meaningless.
• 1,000 successes giving 100% → meaningful.

How we handle it: our availability figures carry a minimum sample requirement, and sites below it are excluded from ranking. That design exists precisely to avoid the distortion of "one success equals 100.00%". An early version hit exactly that, with 59 sites showing 100.00% and 29 showing 0.00%, making the data useless.

Timing matters just as much:

• What time window is the data from? Last week or three months ago?
• Does it cover peak hours? Data measured only during quiet periods overstates stability.

Correct usage: for any number you see, ask three questions first. How large is the sample, how long is the window, and which periods does it cover?

Numbers that cannot answer those three have limited reference value.

Trap three: ignoring a leaderboard's commercial motive

How it shows up: assuming a leaderboard is objective by default.

The problem: a leaderboard may have several commercial motives:

• Listing or ranking fees, giving paid sites better positions;
• Affiliate or referral commissions, so recommendations favour the sites that pay best;
• Self-interest, where the operator also runs a relay business.

What actually happens in this industry: many relay sites appearing in "review articles" are there as promotions with signup links. Ordering in that kind of content follows commission rates, not technical metrics. There is a very simple way to spot it: check whether every recommended site has a signup link, and whether the links carry tracking parameters.

How to judge whether a leaderboard deserves attention:

Correct usage: treat a leaderboard as a source of leads, not a source of conclusions. A site you have never heard of appearing across several independent sources is worth investigating. But the bare fact of being first is not a reason to choose it.

A worked counter-example

Suppose you see two leaderboards:

• Board A ranks "stability" first.
• Board B ranks "cheapness" first.

The sites they recommend barely overlap.

That does not mean one of them is lying. It means the two use different weightings. And the weighting should be decided by your scenario, not the board.

So the real question is: in my scenario, how much weight does stability carry, and how much does cheapness?

Answer that and you no longer need the leaderboard at all. You need the raw data for each dimension, and then you apply your own weights.

Where a leaderboard belongs

In the selection process, as described in the decision tree article, a leaderboard belongs at layer two, as a filtering tool:

1) Establish purpose and hard constraints      <- yours to do
2) Filter down to a candidate list             <- a leaderboard helps here and saves time
3) Capability verification, reconciliation, concurrency test   <- yours alone
4) Operating strategy (top-up / backup / monitoring)           <- yours alone

A leaderboard takes you from "I have heard of 200 sites" to "I have five in mind". That is its value, and that is its boundary.

Step 3 cannot be skipped, because capability verification is the only way to rule out "unusable", while a leaderboard only tells you "others find it acceptable".

One last point

We provide data too, so everything above applies to us as well:

• We do not publish an overall ranking, for the reasons given in the article on what the data can and cannot answer.
• Our availability and latency figures are measured, but from a single methodology, one probe location and minimal requests.
• Our pages carry advertising space, and adverts are clearly labelled.

Evaluate any data source, ours included, by one standard: where the data comes from, what the methodology is, how large the sample is, and whether there is a commercial motive.

Get comfortable with that standard and you will avoid all three traps.