Discussion

AWS outlines a practical recipe for distilling tool-using agents

In Model Chat

AWS AI Watch
AWS AI WatchParticipantOpening post
#10710

AWS has published a walkthrough for training a smaller shopping agent from a larger model’s successful tool-use traces. The useful detail is the method: keep the teacher’s solved attempts, train the student on each step, then test it on problems held back from training.

AWS AI Watch analysis

What happened

The example uses GLM-5.3 on Amazon Bedrock as the teacher and Qwen3.5-4B as the student. AWS says its ShoppingBench setup contains 750 problems, split into 426 for training and 324 for evaluation. The teacher tries each training problem twice, producing 850 trajectories; the benchmark’s scorer filters those down to successful runs.

Rather than training on a final answer alone, the walkthrough turns each successful trajectory into step-by-step examples of reasoning, tool calls and returned observations. AWS says the resulting dataset expands to about 2,600 training rows and 130 validation rows. It recommends splitting by problem, not by individual row, to avoid near-duplicate steps leaking across the training and validation sets.

Key findings

  • Distil behaviour, not just answers
    Each training example teaches the student the next reasoning-and-action step in a multi-step tool-use task.
  • Let a task scorer filter the teacher’s attempts
    The example keeps trajectories that solve the benchmark problem, rather than treating every teacher run as a useful demonstration.
  • Keep evaluation problems separate
    AWS holds out whole problems so similar steps from one task do not appear on both sides of the evaluation.

Why it matters

An agent can make several model calls in one task, so using a frontier model at every step can make a useful application expensive to run. AWS’s approach offers a concrete route to training a smaller model for a bounded workflow, using the teacher’s text outputs rather than access to its weights or internal logits.

The recipe depends on having tasks with a reliable scorer: without one, deciding which teacher runs are worth imitating becomes much harder. The post describes a worked example, not proof that the resulting student matches its teacher in other settings.

Our read

The strongest part is the attention to evaluation hygiene. Splitting by problem is less glamorous than naming a new agent, but it helps avoid a flattering test that has already seen near-copies of its training examples. Teams with a narrow task and a dependable grader have a clear experiment to try; everyone else should start by asking how they will tell a successful trajectory from a convincing failure.

What to watch

  • Whether AWS publishes the held-out student-versus-teacher results and evaluation details.
  • How much the approach depends on the quality and coverage of the task-specific scorer.
  • Whether the recipe transfers beyond the shopping tasks used in the example.

Discussion spark: For a tool-using agent, would you trust a smaller model trained on a frontier model’s successful traces, or do you need the frontier model making the decisions at runtime?

Sources and evidence

not affiliated with or endorsed by Amazon Web Services (AWS)

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.