Customer support and chat: balancing reliability against concurrency

What makes this scenario different

Chat has two hard constraints:

1. Users are present - an interruption is noticed immediately and hits the experience
2. Concurrency is not yours to control - when users arrive and how many, you do not decide

That pushes the selection criteria towards "stable" rather than "cheap".

How to assess reliability

Do not look at a single "uptime" number. Separate three kinds of failure:

Partial failure is the worst of the three - it does not trigger your global switchover logic, but it makes the experience unpredictable.

How to assess: run sustained requests for a while and measure the failure rate, not just whether the endpoint is reachable.

How to estimate concurrency

peak concurrency ~ concurrent users x requests per user per minute / 60

For example: 100 users online at once, each sending two messages a minute:

100 x 2 / 60 ~ 3.3 concurrent (average)
Real peaks can be 3-5x the average -> 10-17 concurrent

The key point: the concurrency figure a site advertises usually means its overall capacity, not the quota for your single account. Per-account limits are often far lower - you have to ask.

One thing you must do: a fallback plan

A user-facing scenario cannot go without a fallback. Three forms:

One: switch the channel

PROVIDERS = ["https://primary/v1", "https://backup/v1"]

async def chat(msg):
    for base in PROVIDERS:
        try:
            return await call(base, msg)
        except Exception:
            continue
    return "The service is busy right now, please try again shortly"

Two: drop to a smaller model

When the main model is unavailable, switch to a smaller, steadier one. Quality drops, but at least you answer.

Three: a rule-based response

In the worst case, reply with a canned line ("an agent will contact you shortly"). That beats showing the user an error page.

Three easily missed details

One: your timeout must be shorter than the user's patience. People give up at around ten seconds. Set the client timeout to 15-20 seconds and route to the fallback on timeout rather than waiting longer.

Two: streaming hides latency. A three-second total with a first byte at 300 ms feels like "it started immediately". For chat, streaming is strongly recommended.

Three: watch content policy. Different sites differ a lot in what they allow. If a user-facing chat touches sensitive topics, confirm the site's boundaries in advance and filter in your own application layer - do not rely on the site to catch it for you.

Priority order when choosing a site

1. A low partial-failure rate   <- more important than "is it reachable"
2. Enough concurrency for one account
3. Streaming support
4. A working fallback channel
5. Cost

Related reading

• How to configure several API providers as mutual fallbacks
• After you have chosen: how to keep monitoring your API provider
• Common error reference: what 401/403/404/429/500 each mean