Migrating suppliers gradually: replacing an API site without interrupting service

Why you cannot just change the base_url

Changing base_url from A to B looks like a one-line edit, but it changes several things at once:

• who actually provides the model, affecting output style, format compliance and tool-call behaviour;
• the latency profile, which affects your timeout settings;
• error semantics, since status codes and error wording may differ;
• the billing basis, which affects your cost model.

Switching in one go means all of those take effect simultaneously, and if something breaks you cannot tell which one caused it.

The point of a gradual rollout is to separate them.

Stage zero: prepare the abstraction layer, before switching

Precondition: all calls go through a single exit point. If your code has a dozen scattered references to api.openai.com or some site's domain, converge them first.

The minimum implementation is one config entry plus one client factory.

LLM_BASE_URL=...
LLM_API_KEY=...
LLM_MODEL=...

The value of this step extends beyond this migration. It is the foundation for every subsequent switch. Without it, every supplier change becomes a codebase-wide search and replace.

Stage one: shadow comparison, send to both, use one

Method: send the same batch of requests to both A and B at the same time, but use only A's results, recording B's for comparison.

Why both must be sent: only results from the same request at the same moment are comparable. If you run one batch on A and then a different batch on B, the inputs differ, and you cannot tell whether a difference comes from the supplier or from the input.

What to look at:

How long the shadow stage runs: at least one complete business cycle, usually one to two weeks, and it must include a peak period. Running the shadow only during quiet hours is not a test.

The cost of the shadow stage: sending to both doubles the spend. That is the necessary cost of migration, and it is far cheaper than the business interruption of a failed switch.

If cost is sensitive: sample the requests, sending only 10% to both for instance, but you must cover every request type, not just the easy ones.

Stage two: ramping by proportion

Method: move real traffic to B by proportion, in steps.

A suggested ramp: 1% → 5% → 20% → 50% → 100%, observing at least one complete business cycle at each level.

Why not a linear ramp: 1% to 5% is a factor of five, 5% to 20% is a factor of four, 20% to 50% is a factor of two and a half. Early steps can be larger, because the sample and the risk are small; later steps should be smaller, because the blast radius is large.

Choosing the split dimension:

• By user or tenant: suited to multi-tenant systems. One user's experience stays consistent and the blast radius is controllable.
• By request type: suited to systems with clear functional modules. You can move non-critical paths first.
• Random by percentage: the simplest, but one user may go to A sometimes and B other times, with an inconsistent experience.

Recommendation: prefer splitting by user or by request type. Avoid pure randomness.

What to observe at each level:

[ ] Error rate, compared with the A group
[ ] p95 latency
[ ] Business metrics, such as task completion rate and user action success rate
[ ] Cost, whether the actual charge matches expectations
[ ] New exception types in your own logs

Define the rollback triggers in advance, such as "error rate more than twice the A group" or "p95 latency degraded by 50%". Do not wait until something goes wrong to decide whether to roll back, because judgement at that moment is coloured by stress.

Stage three: full rollout and wrap-up

After going fully over, observe one more complete business cycle, then:

1. Do not delete A's configuration immediately. Keep it for at least a month as a rollback path.
2. Keep A's key but set a small cap on it, so it can sit there as a long-term backup without being drained when you forget about it.
3. Update documentation and alerts. Every place that mentions A needs changing, including the on-call handbook.

The rollback plan, which is really the most important part

Rollback must be a configuration change, not a code release.

Three requirements:

1. Configuration-driven, switchable by environment variable or a config service.
2. Both sides coexist long term. Do not delete A's credentials once you have switched to B.
3. Rehearsed at least once. Actually roll back and see, rather than assuming it works.

Item 3 is the one most often skipped. The value of a rollback path lies entirely in it actually working, and the only way to know is to have tried it.

A simplified version for small teams

If the full process is too heavy, here is a version a three-person team can run:

1. Converge base_url into one place — half a day.
2. Run the shadow stage for two days only, covering one peak period — one day.
3. Ramp in three steps, 10% → 50% → 100%, one day each — three days.
4. Keep both configurations for at least a month.

About a week in total. The parts that cannot be cut are item 1, the abstraction layer, and item 4, keeping the rollback path. The former decides whether you can switch at all; the latter decides whether you can come back if the switch goes wrong.

Everything else can be simplified.