After you have chosen: how to keep monitoring your API provider

Why third-party monitoring is not enough

Our monitoring data is useful, but it has two limits:

1. The vantage point is not your location -- the route from your network to a site may be completely different
2. It does not test your traffic -- it measures reachability, not whether the model you actually call works

So building a lightweight probe of your own is worth it. It is not complicated, maybe a few dozen lines of code.

What the probe should cover

Do not just check whether the website opens. At minimum cover:

The real call is the most important one. A site can be reachable while every model is broken.

A minimal probe

import time, requests

def probe(name, base_url, key, model):
    t0 = time.time()
    try:
        r = requests.post(
            f"{base_url}/v1/chat/completions",
            headers={"Authorization": f"Bearer {key}"},
            json={"model": model, "messages": [{"role": "user", "content": "ping"}],
                  "max_tokens": 3},
            timeout=20,
        )
        return {"name": name, "ok": r.status_code == 200,
                "status": r.status_code, "ms": int((time.time() - t0) * 1000)}
    except Exception as e:
        return {"name": name, "ok": False, "status": 0,
                "ms": int((time.time() - t0) * 1000), "err": str(e)[:60]}

Write the results to a table, even if it is just a CSV. After a week you have reliability data that matches your own network environment.

How to set thresholds

Do not copy someone else's numbers. Derive them from your own history:

1. Run for a week to collect a baseline
2. Compute the P95 latency under normal conditions
3. Set the alert threshold at twice that P95
4. Treat the failure rate the same way

Thresholds derived this way will not cry wolf every day. A threshold picked out of the air, such as alert above 1 second, usually means the alert fires constantly and you start ignoring it.

When to change provider

Triggers for evaluating a switch:
- the failure rate is clearly above baseline for three days running
- the median latency doubles and stays there
- problems appear beyond stability, such as support going silent
- billing starts showing deviations you cannot explain

Billing deviations you cannot explain deserve particular attention. They may be a configuration problem, or something more serious.

Three supporting practices

1. Balance alerts. Running out of balance is the most common outage and it is entirely preventable. Set a threshold, such as alerting below three days of usage.

2. Record switching events. If you have multiple channels configured, log every switch. A rising switch rate is an early signal of declining provider quality.

3. Check by hand periodically. Ten minutes a month: glance at the usage trend, check for new announcements, and test the support response once. Automation finds problems, but judging quality still needs a person.

A metric that is easy to overlook

The trend of your effective unit cost, in currency per million tokens.

The calculation is this month's spend divided by this month's token consumption.

If that number is rising while your usage mix has not changed, it may mean the site raised prices, the billing rules changed so that caching no longer applies, or your token mix changed.

This metric catches changes that a total invoice figure hides.

Related reading

• Configuring multiple API providers for failover
• From list price to your invoice: a worked cost conversion
• How uptime is measured: our methodology and its limits