Choosing for code completion: why latency beats the rate multiplier

What makes completion different

Code completion and ask-and-answer usage are not the same thing at all:

High frequency plus extreme latency sensitivity - those two set the selection criteria.

Why latency matters more than money

The human interaction window is roughly 200-500 milliseconds. If a completion takes longer than 500 ms to appear, you lose focus without noticing.

And once you are interrupted, getting back into flow takes a dozen seconds or more. So the cost of latency is not "waiting half a second", it is a drop in throughput.

Work it out: if completion latency goes from 300 ms to 1,200 ms, and you get interrupted a few hundred times a day, the time lost vastly exceeds the API fees saved.

Which latency number to look at

Do not look at average latency. Look at time to first byte (TTFB).

The reason: the experience of completion depends on how fast the first character appears. One site might take 2 seconds in total but deliver the first byte in 200 ms (streaming); another might take 1.5 seconds in total with a first byte at 1.5 seconds (buffering the whole response). The second feels far worse.

How to measure: send a request with curl -N and stream:true, and time how long until the first chunk arrives.

Keeping the cost down

Completion volume looks small per request but the frequency is very high, so the total can be large.

Three levers:

One: use different models for different jobs.

Inline completion (a few words)  -> the cheapest model
Function-level generation        -> a mid-tier model
Cross-file refactoring           -> a strong model (triggered manually)

Running every completion through a strong model is absurdly expensive, and slower too.

Two: limit the context. Completion does not need the whole file. Restricting context to the region around the cursor cuts tokens substantially, and latency with it.

Three: rely on caching. Repeated completion requests in the same file share an enormous amount of context. In editor scenarios the value of caching is higher than anywhere else.

Priority order when choosing a site

1. Low time to first byte   <- affects every keystroke
2. prompt caching support   <- affects cost, by a factor of several
3. Protocol compatibility with your editor tools  <- incompatible means unusable
4. A high enough concurrency cap  <- editors issue concurrent requests
5. The rate                 <- limited effect, often cancelled out by 2 and 3

A practical configuration

Split editor traffic and production traffic across different sites:

• Editor: a low-latency site with caching support (price-insensitive, experience-sensitive)
• Production: a cheap, stable site (latency-insensitive)

The two traffic patterns are opposites, and mixing them interferes - a burst of editor requests can trip rate limiting and affect production.

Related reading

• Configuring Cursor and Cline to work with a relay service
• Prompt caching on a relay service: does it actually work
• How to read a relay service's response latency