Tool use compatibility across relay sites, and how to self-test it

Why tool calling breaks most often

Differences in text generation quality are obvious at a glance. Tool calling is not like that, because its failures are usually silent:

• Arguments get mangled and the model raises nothing. It simply performs the wrong action.
• Context is lost across turns, which shows up as "the agent suddenly forgot what it just did".
• Tool calls are chunked incorrectly under streaming, which shows up as "the tool call stopped halfway".

Worse, none of these appear in a simple test. So if you are building agent-style applications, tool calling has to be verified on its own.

How the three APIs differ structurally

OpenAI

• Request: tools (an array whose elements are {"type":"function","function":{...}}) plus tool_choice
• Response: message.tool_calls, where function.arguments is a JSON string, not an object
• Returning a result: a message with role tool, carrying a tool_call_id

Anthropic

• Request: tools, whose elements are {"name","description","input_schema"} with no outer function wrapper
• Response: entries inside the content array shaped {"type":"tool_use","id","name","input"}, where input is an already-parsed object
• Returning a result: {"type":"tool_result","tool_use_id","content"} inside content

Gemini

• Request: tools[].functionDeclarations
• Response: the functionCall part

Put plainly, the differences are: whether arguments is a string or an object, how many wrapper layers there are, and which field links a result back to its call. Those three are exactly where conversions go wrong.

Three common failures

Failure one: double-encoded arguments

The most common bug of the lot. OpenAI's arguments is a string; Anthropic's input is an object.

If the gateway serialises arguments a second time even though it is already a string, or goes the other way and passes an object through as though it were a string, you get:

• arguments as an escaped string, "{\"city\":\"Beijing\"}" instead of {"city":"Beijing"}
• or a parse failure, and the tool receives empty arguments

Symptom: the tool really is called, but with the wrong arguments.

Self-test: design a trivial tool that cannot work without an argument, such as get_weather(city), and check whether the argument arrives intact.

Failure two: context lost across turns

The typical agent loop is "model requests a tool → you execute it → you return the result → the model continues". That requires the gateway to keep tool_use and tool_result (or tool_calls and role=tool) structurally consistent across the round trip.

If the gateway downgrades a tool result into an ordinary text message at some point, the model loses the semantic "this is the result of the call I just made", and you see:

• the same tool called over and over
• the model still saying "I have not been able to retrieve that" after receiving the result

Self-test: design a two-turn call, weather first and then a recommendation based on it, and see whether turn two behaves.

Failure three: chunked tool calls under streaming

Non-streaming gives you one complete JSON payload. Streaming splits it into many delta fragments, which the client or gateway has to reassemble by index.

If the reassembly logic is wrong, arguments arrive incomplete — city comes through and date is lost, for example.

Self-test: run the same tool call streaming and non-streaming, and compare the final arguments.

A ten-minute self-test

Prepare one tool with two arguments:

{
  "name": "book_room",
  "description": "Book a meeting room",
  "parameters": {
    "type": "object",
    "properties": {
      "room":   {"type": "string", "description": "Room name"},
      "people": {"type": "integer", "description": "Number of people"}
    },
    "required": ["room", "people"]
  }
}

Then run four cases in order:

Pass all four and the site's tool-calling path is usable. Fail any one and you will hit faults in a real agent that are very hard to localise.

What this means per use case

• Plain text chat: if you never use tools, you can ignore this article.
• Coding assistants (Claude Code, Cursor, Cline): this has to pass. Tool calling is the core mechanism of these tools; reading files, editing files and running commands all depend on it. One lesson the industry keeps relearning: when a client shows odd tool behaviour after being pointed at a relay site, suspect the relay's protocol conversion layer first, not the client.
• Agents and workflows (Dify, FastGPT, in-house): this has to pass, and multi-turn has to be tested.
• RAG applications: pure retrieval plus generation can skip it, but the moment you use a function to route retrieval, you have to test it.

One practical recommendation

If you are building agent-style products, put this self-test on your supplier onboarding checklist, at the same level as "can we connect at all".

A connectivity problem takes you five minutes to find. A tool-calling problem may take you two weeks, and you will suspect your own code first.