Build your own probe: twenty lines of script to monitor your API suppliers

The goal is knowing when it breaks

The first goal of monitoring is not optimisation, it is detecting failure. The minimum requirements for that are:

1. Probe on a schedule.
2. Have a baseline.
3. Notify you when something is off.

None of that needs a monitoring platform. Twenty lines of script will do.

Two levels of probe

Level one: reachability and latency

The simplest probe: send a minimal request and record success or failure plus the time taken.

#!/usr/bin/env bash
# probe.sh -- probe one site with a minimal request
BASE="https://<domain>/v1"
KEY="<KEY>"
MODEL="<model-name>"

START=$(date +%s%N)
HTTP=$(curl -s -o /tmp/probe_body -w '%{http_code}' \
  --max-time 20 \
  "$BASE/chat/completions" \
  -H "Authorization: Bearer $KEY" \
  -H 'Content-Type: application/json' \
  -d "{\"model\":\"$MODEL\",\"max_tokens\":1,
       \"messages\":[{\"role\":\"user\",\"content\":\"1\"}]}")
END=$(date +%s%N)
MS=$(( (END - START) / 1000000 ))

echo "$(date -Is)\t$HTTP\t${MS}ms"

Points:

• max_tokens: 1 keeps the probe request as cheap as possible. Probe cost must be near zero, or you will not be willing to run it often.
• --max-time 20 is mandatory, or the script can hang.
• Record both the HTTP status code and the elapsed time. Neither is optional.

Level two: authenticity

Reachability only tells you whether it is there. To catch "it got dumber" or "the backend changed", you need a probe with an assertion.

One cheap approach: use a question with a determinate answer and check the response.

For example: "What is 1+1? Reply with the digit only."

• Normal: returns 2.
• Swapped for a weaker model: may return something other than 2, or answer a different question.
• Silently downgraded: returns a long explanation, showing it did not obey "reply with the digit only".

Note: a single failure proves nothing, since models are stochastic. Only N consecutive failures are meaningful, with N equal to 3 being a reasonable choice.

Do not overbuild this level. Its value is detecting a change in behaviour, not assessing quality precisely. The simplest question with the most determinate answer is enough.

Choosing the interval and timeout

Why alert only after three consecutive failures: single failures are far too common, from network jitter to a momentary upstream 5xx. Alerting every time means you start ignoring alerts, and then the monitoring is worthless.

Why recovery also needs two in a row: otherwise one lucky success fires a recovery notice while the outage continues.

Three traps you must avoid

Trap one: probing so often you get rate limited

The consequence: your probe requests themselves trip rate limiting, producing false outages where "the probe says down but production is fine".

This happens easily. Ten sites probed every minute is ten requests a minute, which sounds trivial; but if several sites share one upstream account pool, or you are running production traffic at the same time, the total can cross the threshold.

What to do:

1. Keep the interval at 5 minutes or more.
2. Use a separate key for probing, if the site supports multiple keys.
3. Use a minimal max_tokens.

Trap two: counting business errors as availability failures

The problem: some failures are not the supplier's fault but the fault of your own request.

How to separate them:

The key point: 401 and 400 must be excluded from availability statistics, or changing a key will show you a wall of "outages".

Likewise, 429 deserves its own category. It reflects constrained capacity, which is a different thing from the service being down.

Trap three: drawing conclusions from a single measurement

The problem: a single probe returning 3000ms does not make the site slow. Network jitter, the first TLS handshake and DNS caching can all skew it.

What to do:

1. Record percentiles rather than averages. At minimum p50 and p95. An average of 500ms with 5% of requests stalling for 10 seconds is experienced as "occasionally hangs".
2. Do not let a single outlier fire an alert. Look at the trend.
3. If you can, probe from two different network locations, such as your server plus a cloud instance. That separates "the site's problem" from "your network's problem".

Storage and alerting

Storage: one row per probe is enough, with fields timestamp / site / status code / latency ms / success. SQLite or flat files both work.

Alerting: the simplest reliable route is a webhook to a chat bot. The content should include:

[API monitor] <site name> failed 3 times in a row
Recent: 500 / timeout / timeout
Last success: 2026-10-06 14:20

Include the last success time. It lets you tell at a glance whether this just broke or has been broken for a while.

One thing not to build yourself

Do not implement a complex monitoring system. The probe's goal is "know when it breaks", and twenty lines achieves that. Metric aggregation, visualisation and trend analysis, if you write them yourself, consume a lot of time for limited return.

If your team already runs a monitoring stack such as Prometheus, push probe results there and use the existing alerting and dashboards. That is the least effort by far.

One aside

The data on our availability monitoring page comes from the same kind of method, scheduled probing plus banded statistics, and the methodology is documented in the article on how availability rates are measured.

Note carefully, though: our probes run from the vantage point of a single server. Your network environment, the models you actually call and your concurrency pattern all differ from ours.

So our data is good for comparison and shortlisting, and not a replacement for your own probe. The two are complementary: we give you a broad view, and your probe gives you a deep view fitted to your own scenario.