Configuring multiple API providers for failover

The goal is not never fail

Start by being precise about the goal: not to make the API never fail, which is impossible, but to reduce the impact of a failure from a business outage to a single retry.

With failover configured, when one site goes down your code switches to another and users never notice.

The minimum viable architecture

class Channel:
    def __init__(self, name, base_url, api_key, priority):
        self.name, self.base_url = name, base_url
        self.api_key, self.priority = api_key, priority

CHANNELS = [
    Channel("primary", "https://site-a/v1", key_a, 1),
    Channel("backup", "https://site-b/v1", key_b, 2),
    Channel("fallback", "https://official/v1", key_official, 3),
]

Calls try each channel in priority order:

async def call_with_failover(payload):
    errors = []
    for ch in sorted(CHANNELS, key=lambda c: c.priority):
        try:
            return await call(ch, payload), ch.name
        except RetryableError as e:
            errors.append(f"{ch.name}: {e}")
            continue
    raise AllChannelsFailed(errors)

Detail one: which errors should trigger a switch

Timeouts need separate treatment: if the request was already sent, switching may bill you twice. At minimum distinguish a connection timeout, where nothing was sent and switching is safe, from a response timeout, where a charge may already exist.

Detail two: do not turn failover into unlimited retries

# risky: 3 retries per site, 3 sites = 9 attempts
for ch in CHANNELS:
    for attempt in range(3):
        try: ...

# better: one attempt per site, rotate quickly
for ch in CHANNELS:
    try:
        return await call(ch, payload)
    except RetryableError:
        continue

The reasoning: when everything is down, as in an upstream outage, more retries change nothing. They just waste time and inflate the bill. Fail fast, rotate fast works better.

Detail three: log which channel was used

result, channel_used = await call_with_failover(payload)
logger.info("request completed", extra={"channel": channel_used})

Why it matters: when something goes wrong you need to know which site handled the request. Without this log, incident analysis becomes guesswork.

It also tells you how often the backup channels are being used. Frequent switching means the primary has a problem and needs attention.

Advanced: routing per model

ROUTES = {
    "claude-*": ["site-a", "site-c"],
    "gpt-*":    ["site-b", "site-c"],
    "*":        ["site-a", "site-b", "site-c"],
}

Each model gets its own primary and backup order, which raises overall availability.

Do not forget the supporting pieces

1. Map model names consistently. Sites name models differently, so the routing layer needs a mapping table.

2. Rehearse failover regularly. Configured does not mean working. Take the primary down manually now and then and confirm the backup really picks up.

3. Keep the backup funded. This is the most commonly overlooked point. A backup that is never used may run short of balance, and you only find out when you need it.

4. Monitor the switch rate. A sudden rise in switching is a failure signal and should trigger an alert.

Related reading

• After you have chosen: how to keep monitoring your API provider
• Common error codes explained: what 401, 403, 404, 429 and 500 each mean
• Customer support and chat: balancing reliability against concurrency