Reading a relay service's response latency: one measurement is not the experience

What that millisecond figure is

The response latency on our detail pages is the time taken to request a site's home page and receive the complete response, in milliseconds.

Three qualifiers:

1. It measures the home page, not the API endpoint
2. It measures the complete response, not the first byte
3. It is issued from our server, so it does not represent your location

Why the home page

A site can have a working API and a dead website, or the reverse. Measuring the home page is the pragmatic middle ground: it always exists, unlike an API endpoint that may return 401 to anonymous requests, and it passes through the site's full stack.

Limitation: home page latency and API latency are related but not equal. A site serving static pages can have a fast home page and a slow API.

Why not the first byte

For interactive workloads such as editor completion and chat, time to first byte matters more than total duration.

If a site streams, the first character arriving at 200ms and the stream finishing at 3 seconds still feels fast. If instead the site buffers the whole response, a 1.5 second total that waits 1.5 seconds for the first byte feels far worse.

We record total duration because it is automatable and comparable across sites. If perceived speed is what you care about, measure time to first byte yourself with curl -N.

A single measurement varies a lot

Latency depends on many things: the route from you to the site, the site's current load, intermediate CDN nodes, and fluctuations on international links.

So never draw a conclusion from one number. Instead:

• Look at the median of several measurements rather than a single run
• Compare sites measured in the same time window
• Watch the trend, because a sudden slowdown is a signal

The geographic bias

This is the most important point: our server measuring 100ms does not mean you will see 100ms. You might see 800ms.

Network latency is decided mainly by routing, not by geographic distance. The practical approach is to measure from your own environment:

for i in 1 2 3; do
  curl -s -o /dev/null -w "connect %{time_connect}s  first byte %{time_starttransfer}s  total %{time_total}s\n" \
    https://your-endpoint/
done

If your measured latency is far above our figure, the route from your network to that site is poor. In that case the site is a bad fit for you, even if it is fast for everyone else.

What counts as good latency

A rough scale, for API calls with network included:

Note that these figures include network transfer. The model's own inference time is separate, and a reasoning model taking ten-odd seconds is normal.

A distinction that is easy to miss

The site responding slowly and the model generating slowly are two different things.

How to tell them apart: send a very short task, something like reply OK. If the short task is also slow, it is the site. If short tasks are fast and long ones slow, that is simply how the model works.

Related reading

• How uptime is measured: our methodology and its limits
• Choosing for code completion: why latency beats the rate multiplier
• A relay service starts erroring or slowing down: how to tell whose problem it is