Skip to main content
Use casesAdvanced7 min read

SynthAPI: synthetic environments and scaling laws for training tool-use agents

i
iaLocale Team
•
SynthAPI: synthetic environments and scaling laws for training tool-use agents

A tool-use agent learns to use tools in two ways: training teaches it, or it learns on the fly from context. The first way, however, needs an environment to practice in: tools that respond, errors that come back, tasks that repeat. The most comfortable practice environment is your local server, but real tools — the actual APIs your agent will have to deal with in production — cost money, change, and sometimes shouldn't be touched.

The SynthAPI paper, published on arXiv, starts exactly here: build controlled synthetic environments in which to train agents that use tools, without touching the real APIs.

The problem: training an agent takes repetition

A model that has to do tool calling doesn't learn by watching examples: it learns by trying. Every task — "book a flight", "export the report to CSV" — produces a sequence of calls, responses, errors, corrections. To reach a reliable level on a task set, that cycle has to be repeated thousands of times, with variants: slightly different inputs, tools that fail in different ways, longer or shorter responses.

In a real environment, that repetition carries a triple price:

  • Cost: every agent call pushes context tokens (prompt, tool results, reasoning) through the model; with a frontier model via API, every repeated task is money leaving for an external provider.
  • Fragility: real APIs change, rate-limit, and modify their schema. A training task that relied on yesterday's behavior fails today because a field changed — not because of the agent.
  • Noise: real failures are unpredictable. If you're measuring how the agent improves, an unexpected backend 500 pollutes the metric.

The idea: a synthetic environment with failures on a dial

SynthAPI proposes replacing real tools with simulations: synthetic APIs that have the same schema as the real ones (same endpoints, same fields), but whose behavior is under your control. The key point is failure controllability: you can make the simulated tool fail with a probability you set, in ways you set.

Why that matters for training. An agent that has never seen an error doesn't learn to handle one: at the first 429 in production it stops. In a synthetic environment you can inject exactly the mix of failures you want — timeouts, missing fields, rate limits — and measure whether the agent's policy improves or degrades as that mix changes. It's the flight simulator for pilots: it's not the sky, but it's where the behavior that can't be tested in the sky gets trained.

The paper's scaling laws: how many tasks, and how they change

SynthAPI doesn't stop at "let's build a synthetic environment". It studies how synthetic data should be composed and how the required number of tasks scales with the target. The central result, stated in the paper as a logarithmic relationship: the number of synthetic tasks needed to reach a given success rate grows logarithmically with that rate.

In practice, translated into example numbers (the exact constants depend on the dataset and the model; the ratio is the paper's):

  • to reach a 90% success rate on a set of 200 tasks you need on the order of 3,500 synthetic tasks;
  • to reach 95% you need on the order of 7,100 synthetic tasks.

Read: from 90 to 95 percentage points the data cost doubles. That is the kind of fact that changes a project plan: if your use case doesn't need 95%, don't pay the doubling. The paper also shows that the quality of the synthetic data — task variety, failure distribution — weighs more than raw quantity: at equal task counts, an environment that covers the failure distribution well beats one that repeats the same schema.

The math: local synthetic vs frontier API

The point, however, is the where. The synthetic environment runs better — and costs less — if the model that generates the tasks and does the repetitions is yours. Let's compute for 10,000 synthetic tasks, each with roughly 5,000 context tokens (prompt + tool calls + responses), on a generation batch of about 150 GPU-hours of work:

PathCost assumption10,000 tasksCost per task
Your own 24 GB GPU (RTX 4090, ~0.25 kWh per 10k tasks, €0.25/kWh)energy: 6.25 kWh → ~€1.60~€1.60 (hardware amortization excluded)~€0.0002
Serverless GPU (€1.50/h)150 h → €225€225€0.0225
Frontier API ($3/M input tokens, ~20% output)50M input + 10M output ≈ $170~€158~€0.016

The number that matters: a training environment that costs ~€158 in frontier API for 10,000 tasks costs ~€1.60 in energy on hardware you already own — and the data never leaves your machine. This isn't a ten-thousand-euro argument, but it's the kind of multiplier that makes doing more iterations sensible: and in the paper, quality scales with task quantity, not with the elegance of the setup. API prices and energy are assumptions for a typical batch; re-measure them for your case.

And the limits?

Honesty, because the synthetic has a boundary the paper itself delineates:

  • Shifted distribution: the synthetic environment imitates the tool schema, not the real oddities — unexpected responses, error messages, varying latency. An agent over-trained on the synthetic can perform well in simulation and worse in production. The paper manages this by calibrating the failure rate; you manage it by validating the agent on a held-out battery of real calls before trusting anything.
  • Model ceiling: synthetic tasks improve a model's policy; they don't create capabilities that don't exist. A model that can't do tool calling at the base won't magically learn it from synthetic tasks.
  • Validation: the success rate measured in a synthetic environment is a proxy. The metric that counts is always the one on the target environment, even if small.

How you use it, in practice

If you're building a local agent that has to learn your tools, the pattern the paper suggests reduces to four steps:

  1. Write the schema of your tools (name, parameters, response types, failure modes) — it's the contract the synthetic environment must respect.
  2. Generate the synthetic tasks locally: a model on your hardware produces inputs, tool results, error scenarios. This is where you need a big context and repetition — and it's exactly the case where KV cache makes you feel the RAM you have, not the RAM you expected.
  3. Dial in the failure mix and measure the success rate on a validation subset that doesn't enter training.
  4. Validate in production with a battery of real calls: if the gap from synthetic to real is small, the setup worked.

If your agent stack already runs on Ollama, the engine comparison tells you where the bottleneck sits — because in a synthetic task generation pipeline the bottleneck is almost always inference, not data.

Sources

In summary

SynthAPI shows that a tool-use agent can be trained in a controlled synthetic environment, with the same structure as real tools but failures on a dial. The number of synthetic tasks grows logarithmically with the target success rate: from 90 to 95 percentage points the data cost doubles, and not every use case pays that jump. Running the generation batch on your own hardware — the math above shows ~€1.60 of energy versus ~€158 of API for 10,000 tasks — multiplies the number of possible iterations at equal budget, and keeps the training data in-house. The limit is fidelity to the real distribution: validation on real calls remains the final judge.

Article generated with AI

In these reading paths

Share:

Enjoyed this article?

Subscribe to the newsletter for new tutorials and guides.

After signing up you receive a confirmation email (double opt-in). Without confirmation we send nothing. No spam, we promise.

Related Articles