RPM, TPM and concurrency: the four real shapes of relay rate limiting, and how to find the true ceiling

Why rate limiting is harder to diagnose than an error

An error at least tells you something went wrong. Rate limiting often does not — it has four shapes, and three of them return no error code at all.

Shape two is most often mistaken for "a slow network", shape three for "the model got dumber", and shape four for "a client bug".

All three mistakes have the same consequence: you optimise the wrong thing.

Three independent dimensions

Rate limiting is not one ceiling but three independent ceilings, and any one of them triggers:

• RPM (requests per minute): sensitive to short requests;
• TPM (tokens per minute): sensitive to long context — a single 100K-context request can consume a whole minute's quota;
• Concurrent connections: requests in flight at the same time, entirely independent of RPM. You can have low RPM and high concurrency, as long as every request is slow.

This is why "I am nowhere near my RPM limit but I still get throttled" happens so often — you are over TPM or concurrency, while the dashboard only shows RPM.

Step-load testing: the only way to find the real ceiling

Do not start by firing 50 concurrent requests. You will trigger every limit at once and get a curve you cannot interpret.

Instead, ramp in steps, 60 seconds per step, recording four numbers: success rate, P50 latency, P95 latency and time to first token.

import time, statistics, concurrent.futures as cf, urllib.request, json

BASE, KEY, MODEL = "https://<base_url>", "<key>", "<model>"

def one():
    body = json.dumps({"model": MODEL, "max_tokens": 64,
        "messages": [{"role": "user", "content": "Reply with the single word: ok"}]}).encode()
    req = urllib.request.Request(BASE + "/v1/chat/completions", data=body,
        headers={"Authorization": "Bearer " + KEY, "Content-Type": "application/json"})
    t0 = time.time()
    try:
        with urllib.request.urlopen(req, timeout=120) as r:
            r.read()
            return (time.time() - t0, r.status)
    except Exception as e:
        return (time.time() - t0, getattr(e, "code", 0))

for conc in (1, 2, 4, 8, 16):
    lat, codes = [], []
    with cf.ThreadPoolExecutor(max_workers=conc) as ex:
        deadline, futs = time.time() + 60, []
        while time.time() < deadline:
            futs.append(ex.submit(one))
            time.sleep(0.5)          # fixed arrival rate, so bursts do not distort results
        for f in cf.as_completed(futs):
            d, c = f.result(); lat.append(d); codes.append(c)
    ok = sum(1 for c in codes if c == 200) / len(codes)
    print(conc, "ok=%.0f%%" % (ok * 100), "p50=%.1fs" % statistics.median(lat),
          "p95=%.1fs" % sorted(lat)[max(0, int(len(lat) * 0.95) - 1)],
          "429=", codes.count(429))

Three points matter:

• A fixed arrival rate (one every 0.5 s above); otherwise you are measuring burst tolerance, not a sustained ceiling;
• Long enough steps (60 s). Anything under 30 s gets hidden by momentary quota;
• Measure a single-worker baseline first. If the baseline itself is unstable (P95 jitter beyond 2x), every later number is worthless.

Reading four curve shapes

Only the third row is friendly, because it is visible. The first two are where most people get caught.

How to pin down the tier rules

Relay limits are almost always per user tier: deposit level, identity verification and enterprise status directly determine RPM and concurrency.

Four questions, in this order:

1. Is the limit RPM, TPM or concurrency? Numbers for all three are best.
2. What comes back when you exceed it — 429 or a queue? If the answer is "we do not rate limit", read it as "you will be queued".
3. Is the limit per key or per account? Do multiple keys add up?
4. Is the quota at peak hours (20:00-23:00) the same as during the day?

Question four matters most. Many services are fine all day and produce 429s every evening — not because your volume grew, but because the shared pool is being split among more users.

A counter-intuitive conclusion

A low threshold is not the problem. An opaque threshold with silent throttling is.

A service that says "your tier is 60 RPM and you get a 429 above that" lets you write retries and queues. A service that says "we do not rate limit" and then queues your request for 40 seconds leaves you with no metric to monitor at all — because rising latency has infinitely many explanations.

So in any selection checklist, "can it state its rate-limit policy precisely" deserves its own line item, on par with availability.

Summary

1. Rate limiting has four shapes, and only hard 429 tells you it happened.
2. RPM, TPM and concurrency are independent, and the one you exceed is usually not the one you expected.
3. Step-load (1, 2, 4, 8, 16 workers, 60 s per step, fixed arrival rate) is the only reliable probe.
4. Measure a single-worker baseline first; an unstable baseline invalidates everything.
5. P50 rising linearly with no 429 is silent queueing — you are already limited.
6. In selection, treat "is the rate-limit policy transparent" as a standalone criterion, more important than the numbers themselves.