Batch data processing: how to use a low-rate site correctly

Why batch work suits a low-rate site

The ability to retry and the ability to verify are what let you tolerate a lot more from the site.

No caching, occasional slowness - a batch job can live with all of that, provided it is cheap.

But there are two traps

Trap one: quality drops are harder to spot in batch

In interactive use you can tell at a glance when answers get worse. In batch processing the output goes straight downstream and nobody looks at it.

What to do: build automatic checks. Require JSON output and validate it against a schema; apply rule checks to key fields; sample for human review (five out of every hundred items, say).

Batch processing without validation means leaving quality to luck.

Trap two: run-away concurrency trips rate limits

It is easy to write a batch job that fires at full speed - and then hit rate limiting or a ban.

What to do: cap concurrency explicitly, and add retries with backoff.

import asyncio, random

sem = asyncio.Semaphore(5)

async def process(item, retries=3):
    for attempt in range(retries):
        async with sem:
            try:
                return await call_api(item)
            except RateLimitError:
                await asyncio.sleep((2 ** attempt) + random.random())
    return None

The jitter (random.random()) matters: without it every failed request retries at the same instant, and you trip the limit again.

A recommended rhythm

1. Run 20 items first        -> verify format and quality
2. Run 1,000, measure speed and cost -> extrapolate the total duration
3. Run in shards, with a gap between them -> lower the risk-control exposure
4. Record failures, rerun them separately at the end -> no unbounded retries in the main flow
5. Sample and check          -> confirm quality

The gap in step three does not need to be long; it is mainly there to avoid the machine-like signature of running flat out for hours.

Where else the cost can come down

Beyond choosing a low-rate site, there are two more directions:

One: compress the input. Batch prompts are usually highly templated - start by trimming the template. Shaving 100 tokens off the template, multiplied by hundreds of thousands of calls, is a significant number.

Two: use a smaller model. The bar for batch work is "is it accurate enough", not "is the answer elegant". Start with a small model and upgrade only for the categories that miss the bar.

When not to use a low-rate site

• Tasks whose output goes straight to users and cannot be checked automatically
• Judgement tasks with compliance or safety implications
• Batch processing of highly sensitive data

In those cases the money saved does not cover the cost of one serious mistake.

Related reading

• From list price to your invoice: a worked cost conversion
• Account throttled or banned: the common triggers for relay service risk controls
• Six common tricks in cheap relay services, and how to spot each one