AWS tutorial: distill a shopping agent with sequence-level knowledge distillation
The AWS Machine Learning Blog shows how to build a lower-cost shopping agent on Amazon SageMaker AI using sequence-level knowledge distillation (SeqKD), generating multi-step trajectories with a frontier teacher on Bedrock, keeping only those that solved the task with ShoppingBench's scorer, and fine-tuning a smaller student on serverless infrastructure. The post also covers encoding trajectories as chat-formatted messages, tracking training with MLflow, gating quality with SageMaker AI model evaluation and a custom tool-selection scorer, and serving the student behind an endpoint that runs the tool loop. AWS reports the payoff was a 4B student that recovers essentially all of its frontier teacher's task quality.
It gives practitioners a concrete, portable recipe for cutting per-task frontier-model cost for agent loops, and works even when the teacher is a closed API-only model.
You built a shopping assistant on a frontier model, and it works. A shopper asks for “a black waist bag under twenty dollars from a shop with at least 4.5 stars,” and the agent searches the catalog, inspects candidates, checks the constraints, and commits to a recommendation. But what does that interaction actually cost? An agentic assistant is not one model call, it is a loop: search, read results, refine, verify, decide. A single completed task can be multiple calls to that frontier model, and you pay for every one, on every shopper. At production scale, that per-task cost compounds, erodes the project’s ROI, and stops the project from shipping.
One way to make an agentic application more efficient is to use a smaller model, which is faster to respond and more cost-effective to host. The catch is quality: a smaller model usually falls short on the task, which is why the larger one was chosen in the first place. Knowledge distillation closes that gap by transferring the behavior of the larger “teacher” model into a smaller “student”, training the student to reproduce the teacher’s outputs, so it keeps much of the quality at the smaller model’s speed and cost.
#Amazon SageMaker AI#Amazon Bedrock#MLflow#ShoppingBench#AWS Machine Learning Blog