SynthAPI: synthetic environments and scaling laws for training tool-use agents

A tool-use agent learns to use tools in two ways: training teaches it, or it learns on the fly from context. The first way, however, needs an environment to practice in: tools that respond, errors that come back, tasks that repeat. The most comfortable practice environment is your local server, but real tools — the actual APIs your agent will have to deal with in production — cost money, change, and sometimes shouldn't be touched.
The SynthAPI paper, published on arXiv, starts exactly here: build controlled synthetic environments in which to train agents that use tools, without touching the real APIs.
The problem: training an agent takes repetition
A model that has to do tool calling doesn't learn by watching examples: it learns by trying. Every task — "book a flight", "export the report to CSV" — produces a sequence of calls, responses, errors, corrections. To reach a reliable level on a task set, that cycle has to be repeated thousands of times, with variants: slightly different inputs, tools that fail in different ways, longer or shorter responses.
In a real environment, that repetition carries a triple price:
- Cost: every agent call pushes context tokens (prompt, tool results, reasoning) through the model; with a frontier model via API, every repeated task is money leaving for an external provider.
- Fragility: real APIs change, rate-limit, and modify their schema. A training task that relied on yesterday's behavior fails today because a field changed — not because of the agent.
- Noise: real failures are unpredictable. If you're measuring how the agent improves, an unexpected backend 500 pollutes the metric.
The idea: a synthetic environment with failures on a dial
SynthAPI proposes replacing real tools with simulations: synthetic APIs that have the same schema as the real ones (same endpoints, same fields), but whose behavior is under your control. The key point is failure controllability: you can make the simulated tool fail with a probability you set, in ways you set.
Why that matters for training. An agent that has never seen an error doesn't learn to handle one: at the first 429 in production it stops. In a synthetic environment you can inject exactly the mix of failures you want — timeouts, missing fields, rate limits — and measure whether the agent's policy improves or degrades as that mix changes. It's the flight simulator for pilots: it's not the sky, but it's where the behavior that can't be tested in the sky gets trained.
The paper's scaling laws: how many tasks, and how they change
SynthAPI doesn't stop at "let's build a synthetic environment". It studies how synthetic data should be composed and how the required number of tasks scales with the target. The central result, stated in the paper as a logarithmic relationship: the number of synthetic tasks needed to reach a given success rate grows logarithmically with that rate.
In practice, translated into example numbers (the exact constants depend on the dataset and the model; the ratio is the paper's):
- to reach a 90% success rate on a set of 200 tasks you need on the order of 3,500 synthetic tasks;
- to reach 95% you need on the order of 7,100 synthetic tasks.
Read: from 90 to 95 percentage points the data cost doubles. That is the kind of fact that changes a project plan: if your use case doesn't need 95%, don't pay the doubling. The paper also shows that the quality of the synthetic data — task variety, failure distribution — weighs more than raw quantity: at equal task counts, an environment that covers the failure distribution well beats one that repeats the same schema.
The math: local synthetic vs frontier API
The point, however, is the where. The synthetic environment runs better — and costs less — if the model that generates the tasks and does the repetitions is yours. Let's compute for 10,000 synthetic tasks, each with roughly 5,000 context tokens (prompt + tool calls + responses), on a generation batch of about 150 GPU-hours of work:
| Path | Cost assumption | 10,000 tasks | Cost per task |
|---|---|---|---|
| Your own 24 GB GPU (RTX 4090, ~0.25 kWh per 10k tasks, €0.25/kWh) | energy: 6.25 kWh → ~€1.60 | ~€1.60 (hardware amortization excluded) | ~€0.0002 |
| Serverless GPU (€1.50/h) | 150 h → €225 | €225 | €0.0225 |
| Frontier API ($3/M input tokens, ~20% output) | 50M input + 10M output ≈ $170 | ~€158 | ~€0.016 |
The number that matters: a training environment that costs ~€158 in frontier API for 10,000 tasks costs ~€1.60 in energy on hardware you already own — and the data never leaves your machine. This isn't a ten-thousand-euro argument, but it's the kind of multiplier that makes doing more iterations sensible: and in the paper, quality scales with task quantity, not with the elegance of the setup. API prices and energy are assumptions for a typical batch; re-measure them for your case.
And the limits?
Honesty, because the synthetic has a boundary the paper itself delineates:
- Shifted distribution: the synthetic environment imitates the tool schema, not the real oddities — unexpected responses, error messages, varying latency. An agent over-trained on the synthetic can perform well in simulation and worse in production. The paper manages this by calibrating the failure rate; you manage it by validating the agent on a held-out battery of real calls before trusting anything.
- Model ceiling: synthetic tasks improve a model's policy; they don't create capabilities that don't exist. A model that can't do tool calling at the base won't magically learn it from synthetic tasks.
- Validation: the success rate measured in a synthetic environment is a proxy. The metric that counts is always the one on the target environment, even if small.
How you use it, in practice
If you're building a local agent that has to learn your tools, the pattern the paper suggests reduces to four steps:
- Write the schema of your tools (name, parameters, response types, failure modes) — it's the contract the synthetic environment must respect.
- Generate the synthetic tasks locally: a model on your hardware produces inputs, tool results, error scenarios. This is where you need a big context and repetition — and it's exactly the case where KV cache makes you feel the RAM you have, not the RAM you expected.
- Dial in the failure mix and measure the success rate on a validation subset that doesn't enter training.
- Validate in production with a battery of real calls: if the gap from synthetic to real is small, the setup worked.
If your agent stack already runs on Ollama, the engine comparison tells you where the bottleneck sits — because in a synthetic task generation pipeline the bottleneck is almost always inference, not data.
Sources
In summary
SynthAPI shows that a tool-use agent can be trained in a controlled synthetic environment, with the same structure as real tools but failures on a dial. The number of synthetic tasks grows logarithmically with the target success rate: from 90 to 95 percentage points the data cost doubles, and not every use case pays that jump. Running the generation batch on your own hardware — the math above shows ~€1.60 of energy versus ~€158 of API for 10,000 tasks — multiplies the number of possible iterations at equal budget, and keeps the training data in-house. The limit is fidelity to the real distribution: validation on real calls remains the final judge.
Article generated with AI
In these reading paths
Enjoyed this article?
Subscribe to the newsletter for new tutorials and guides.

