Embedding through a relay: the four traps RAG projects fall into

Start with what makes embeddings special

Embeddings differ from chat in one fundamental way: their output is not for a human to read. It is stored and used for arithmetic.

One consequence follows: an embedding problem does not error at call time. It shows up as degraded retrieval quality. And degraded retrieval quality is easily blamed on "a bad retrieval strategy", making it very hard to trace back to the embedding layer.

So all four traps below are the kind that only surface after launch.

Trap one: vector spaces cannot be mixed ★ the most serious

Embeddings from two suppliers are not comparable, even when the model name matches — both called text-embedding-3-small — and the dimensions are identical.

The reason is that the vector space is determined by the specific implementation. Even when two suppliers use the same open-source model, they can differ because of:

• a different weight version, where v1.5 and m1.5 occupy different spaces;
• a different tokenizer version;
• a different quantisation scheme, since fp16 and int8 produce different vectors;
• different normalisation.

Symptom: once vectors from both suppliers are in the same index, retrieval results can become effectively random. Documents that should be similar no longer are, and queries hit irrelevant content.

The most insidious part: nothing errors, and it does not necessarily look like a drop in recall. Sometimes it looks like "it returned a pile of documents that also seem related", because random vectors occasionally collide with semantically close results. So it reads as "mediocre" rather than "broken".

Prevention: one collection uses embeddings from exactly one supplier. If you must switch, rebuild the index in full.

Trap two: a dimension change invalidates the index

Different models output different dimensions — 768, 1024, 1536, 3072 are common. Most vector stores fix the collection dimension at creation time.

Symptom: writes fail after a model change with a dimension mismatch, or worse, some client libraries truncate or pad automatically and corrupt the vectors silently.

Prevention:

1. Record the dimensions and the model name in the collection's metadata, not only in code.
2. On a model change, create a new collection rather than overwriting in place. Keep the old index until the new one is verified.
3. Note that some models support a dimensions parameter, such as dimensions: 512. That is a different projection of the same model, not a model change, but adjusting dimensions still requires a rebuild.

Trap three: return order from the batch endpoint

Batch embedding endpoints usually accept an array of strings. Do not assume the return order matches the input order.

Most implementations do preserve order, but:

• some relay implementations shard and parallelise, so the return order can differ from the input order;
• a very small number of implementations deduplicate, returning one vector for repeated input, so the lengths do not match.

Symptom: vectors are misaligned with their documents. This bug is extremely hard to find, because retrieval returns documents that are "related but not the right one", which looks like a ranking problem.

Prevention:

1. When writing each record, associate by unique ID, never by array position.
2. Check that the returned array length matches the input, and raise an error rather than continuing when it does not.
3. After writing, self-check by retrieving each document using its own text. The correct hit rate should be 100%.

Item 3 is a very good self-check: search with your own text for yourself, and you must rank first. Misalignment fails this test immediately.

Trap four: embeddings come from a different channel

This one is specific to relay sites. A site may have excellent channels for chat models while its embeddings come from an entirely different source, cheaper or reverse-engineered.

How to judge:

1. Look at the embedding price on its own. If it is far below official pricing while the chat side is normal, it comes from elsewhere.
2. Test consistency: embed the same text today and again a week later, then compare the vectors. A significant difference, meaning cosine similarity clearly below 0.999, means the backend is changing.
3. Test semantic ranking quality: prepare a set of sentence pairs with clear semantic relationships — near-synonyms, opposites, unrelated pairs — and see whether the ranking matches expectations.

Why test consistency: if the site routes requests to different backends at different times, your index ends up holding vectors from several vector spaces at once. The effect is the same as trap one, but far harder to spot.

A pre-launch checklist

[ ] Single supplier: one collection uses embeddings from one source only
[ ] Dimensions recorded: model name and dimensions written into collection metadata
[ ] ID association: vectors tied to documents by ID, never by array position
[ ] Length validation: error out when the returned array length differs from the input
[ ] Self-check: retrieve each document with its own text, expect a 100% hit rate
[ ] Consistency: re-embed the same text a week later and compare vectors
[ ] Rebuild plan: a full rebuild is possible when switching suppliers, with a script ready

The last item is the most easily skipped, and it is the only insurance that covers the other six. If your index cannot be rebuilt quickly, you cannot safely change embedding suppliers, which means you have locked yourself in.

A note on cost

Embeddings are extremely cheap per unit, two to three orders of magnitude below chat, so economising here buys almost nothing. Rebuilding an index, by contrast, is not cheap — re-chunking, re-requesting, re-writing.

The pragmatic strategy follows: on embeddings, choose stability over cheapness. The money saved is worth far less than one index rebuild.