← All resources

Synthetic Data Generation for Agent Training

Guide · updated 2026-07-22 · Markdown variant

How to build agentic training corpora without human annotation at scale: the three generation patterns (distillation, self-play, environment rollout), open pipelines (distilabel, AgentInstruct, APIGen-MT, TOUCAN), quality filtering, and the model-collapse risk.

Frontier agents need training data that captures multi-step tool use, error recovery, and multi-turn reasoning — patterns that are expensive and slow to collect from humans at scale. Synthetic data generation fills this gap, but introduces its own failure modes. This guide covers the three principal patterns, the production-ready open pipelines, the quality controls that separate useful data from noise, and the collapse risk that shapes every design decision.

The three generation patterns

1. Teacher distillation. A stronger ("teacher") model generates instruction–response pairs used to post-train a smaller ("student") model. The student inherits the teacher's behavior at a fraction of the inference cost. Quality depends on the teacher's accuracy and the diversity of prompts. Distillation is the dominant method for instruction-following (SFT) data; frontier labs treat their human preference data as a moat precisely because distillation cannot fully replicate it.

2. Self-play / iterative refinement. The model generates a candidate, critiques it against a rubric or constitution, and refines it until a verifier (a reward model or rule-based checker) accepts the output. Anthropic's Constitutional AI is the earliest documented large-scale application. Reinforcement Learning from Verifiable Rewards (RLVR) extends this to tasks with ground-truth answers (math, code, tool-call correctness), where the verifier is deterministic rather than model-based.

3. Environment rollout. An agent executes actions in a real or simulated environment (a browser, a code interpreter, a set of MCP servers), and the resulting (observation, action, outcome) trajectories become training data. This produces the most agentic data but requires a live environment or a high-fidelity simulator, and quality depends heavily on the rollout policy and task diversity.

Production pipelines

distilabel (github.com/argilla-io/distilabel) is the most general-purpose open framework. It provides a unified pipeline abstraction over dozens of LLM providers, implements techniques from published papers (instruction following, dialogue generation, preference modeling), and supports distributed execution via Ray. Maintenance note (as of July 2026): the project's own README states the original Argilla authors have moved to other projects and a group of community collaborators has taken over maintenance and is preparing the next release — check current release cadence before depending on it for a production pipeline.

AgentInstruct generates both prompts and responses from raw seed documents using a multi-agent flow with reflection, avoiding the need for a pre-existing prompt set. A Mistral-7B post-trained on the full 25M-pair corpus improved 40% on AGIEval, 54% on GSM8K, and 45% on AlpacaEval over the untuned base (Microsoft Research, 2024). The 1M-pair public subset is at microsoft/orca-agentinstruct-1M-v1 on Hugging Face.

APIGen-MT (arXiv:2504.03601) targets multi-turn agentic tool use. Phase 1 produces task blueprints with ground-truth action sequences via an LLM-committee review; phase 2 instantiates them as full trajectories via simulated human–agent interplay. The resulting xLAM-2-70b-fc-r scores 56.2% on τ-bench and outperforms GPT-4o and Claude 3.5 on τ-bench and BFCL — and smaller xLAM-2 models often beat larger ones in multi-turn settings. 5K sample trajectories and the xLAM-2-fc-r models (1B–70B) are open-sourced.

TOUCAN (arXiv:2510.01179, IBM Research / Univ. of Washington) is the largest public tool-agentic dataset: 1.5M trajectories synthesized from ~500 real-world MCP server environments spanning 2,000+ tools. Five models generate task queries; three teacher models and two agentic frameworks construct trajectories; model-based filtering removes low-quality samples. Dataset: Agent-Ark/Toucan-1.5M (Hugging Face).

Quality filtering is not optional

Raw synthetic output degrades rapidly without filtering. The main failure modes:

• Hallucinated tool calls — a teacher invents tool names or argument schemas absent from the real API. Filter: execute the call against the real environment and reject any trajectory that produces a tool-not-found or schema-validation error.
• Reward hacking — self-play against a learned reward model can produce outputs that score well on the proxy but fail on held-out real tasks. Mitigation: prefer rule-based verifiers (unit tests, execution outcome, format checks) over model judges where the task allows.
• Low diversity / mode collapse — distillation from a single teacher with homogeneous prompts overrepresents the teacher's most probable outputs. Mitigation: seed from diverse real sources, deduplicate at the embedding level, and measure n-gram entropy of the corpus.

The model-collapse risk

Training successive model generations on an increasingly synthetic corpus — without retaining real human data — produces model collapse: progressive loss of rare, nuanced, and long-tail patterns that amplifies errors from prior generations. The research consensus practitioners act on:

• Never fully replace human data with synthetic data. Keep a substantial real-data share; treat the synthetic proportion as a ceiling, not a floor.
• Accumulate, do not substitute. Keeping the original human signal alongside synthetic augmentation prevents the recursive degradation loop.
• Track provenance. A growing share of scraped web text is itself AI-generated, so "human data" must be verified, not assumed.
• Use grounding signals. Data anchored to execution outcomes (code runs, tool calls succeed/fail, math answers check out) is far more collapse-resistant than purely generative text.

Related ChangeGamer resources

• When to fine-tune versus prompt or retrieve: /resources/fine-tuning-vs-rag
• Evaluation methodology for the resulting models: /resources/evaluating-ai-agents
• Testing fine-tuned agents in CI: /resources/testing-ai-agents
• RAG as a complement to fine-tuning for knowledge tasks: /resources/rag-retrieval-for-agents

Verified sources

• distilabel — Argilla synthetic-data framework (Apache 2.0), README confirms community-led maintenance transition as of July 2026: https://github.com/argilla-io/distilabel
• AgentInstruct — Microsoft Research blog: https://www.microsoft.com/en-us/research/blog/orca-agentinstruct-agentic-flows-can-be-effective-synthetic-data-generators/
• AgentInstruct paper (arXiv:2407.03502): https://arxiv.org/abs/2407.03502
• orca-agentinstruct-1M-v1 dataset (Hugging Face): https://huggingface.co/datasets/microsoft/orca-agentinstruct-1M-v1
• APIGen-MT paper (arXiv:2504.03601): https://arxiv.org/abs/2504.03601
• APIGen-MT project page: https://apigen-mt.github.io/
• TOUCAN paper (arXiv:2510.01179): https://arxiv.org/abs/2510.01179
• TOUCAN dataset — Agent-Ark/Toucan-1.5M (Hugging Face): https://huggingface.co/datasets/Agent-Ark/Toucan-1.5M
• TOUCAN code: https://github.com/TheAgentArk/Toucan
• RLHF Book — Synthetic Data chapter (Nathan Lambert): https://rlhfbook.com/c/12-synthetic-data

#fine-tuning #synthetic-data #training #data-pipelines #tool-calling #distillation #agents

Category: Guide