Image input (vision) on relay sites: what actually works, and how to test it

Why vision has more traps than text

Text requests have a highly uniform shape: a messages array containing strings. Images come in several encodings, several sizes and several transports, and any mishandled link in the chain breaks things — and most of the failures raise no error at all.

The two ways to send an image

One: by URL

{"type": "image_url", "image_url": {"url": "https://example.com/a.png"}}

The key point: the image must be publicly reachable, because the upstream has to download it itself. Internal addresses, addresses behind a login, or expired signed links will all fail.

For a relay site this means it, and its upstream, must be able to reach that URL. If the upstream is overseas while the image sits in domestic object storage without public access enabled, the request fails silently.

Two: inline base64

{"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0..."}}

Key points:

1. The request body inflates sharply. A 1MB image becomes about 1.37MB as base64. That runs into the reverse proxy's client_max_body_size — nginx defaults are often 1MB — and you get a 413 rather than anything to do with images.
2. Token consumption is driven by image dimensions, not file size. The same photo saved as a heavily compressed JPG and as a PNG can differ tenfold in file size and consume almost identical tokens.
3. The format prefix is not optional. It must be data:image/png;base64,. Bare base64 is rejected.

Three typical failures

One: silent ignoring ★ the most dangerous

Symptom: the request returns 200, the model answers normally, and the answer makes no use of the image content at all. It invents a generic reply from the text prompt.

Cause: while converting protocols, the gateway hit an image_url content block it could not recognise and simply dropped it, rather than erroring.

How to detect it: send an image whose content contradicts the text prompt. A picture of a cat, with the question: "Is this a dog or a cat?"

• If the image really was processed → the answer is "cat".
• If it was silently ignored → the model picks one side of the either/or, dog or cat, or gives some vague reply that is not "I cannot see the image".

A stricter probe: write a random number into the image and ask "what number is in the image". Genuine processing answers correctly; silent ignoring answers wrongly or vaguely.

This is the only reliable criterion. Do not judge by "it answered a question about the image", because models are very good at inventing plausible answers.

Two: format conversion failure

Symptom: a 400 comes back, with an error message mentioning image format.

Cause: Anthropic's image block structure differs from OpenAI's (source.type / source.media_type / source.data). If the conversion layer does not handle all of it, some formats such as webp and gif fail while png and jpg work.

What to do: test once with PNG to establish the baseline, then test webp and gif separately to see how far coverage reaches.

Three: size and count limits

Symptom: 413 or 400, correlated with image size.

Cause:

• Reverse proxy body limits, the most commonly overlooked. Start probing at 1MB and step upward.
• Upstream image count ceilings. Most models accept several images per request, but some gateways forward only the first.
• Upstream dimension ceilings. Some implementations cap the longest edge and downsample when exceeded. No error, but detail is lost.

A five-minute self-test

Prepare two images:

• Image A: a random four-digit number in black text on a white background, 7382 for instance.
• Image B: an ordinary photo around 200KB.

Then:

Item 1 is the core. That item alone yields a valid conclusion.

What this means for selection

Vision capability comes in three grades:

• Not supported at all: errors, or silent ignoring.
• Works but limited: reads images, with constraints on format, dimensions and count.
• Fully supported: handles every format, multiple images and large images.

For most use cases "works but limited" is enough. Screenshot Q&A, receipt parsing and UI screenshot analysis all involve images under 1MB in PNG or JPG.

What genuinely deserves attention is item 1, silent ignoring. Nothing errors, so you may conclude that the model is bad at image questions when in fact the image never arrived.

One caution

If you use vision through a relay site, do not plan capacity for image and text traffic on one key. Image requests are far larger in body size and token consumption than text, and they will show up as sudden spikes in monitoring that trip the site's rate limiting.

Recommendation: keep a separate key for image calls, so you can watch them and cap them independently.