Airbnb用单任务成本评估Bedrock编程代理
AWS Machine Learning Blog称,Airbnb每周有数千名工程师及工程相关岗位员工使用内部AirChat工具运行编程代理。这些代理通过部署在Amazon Bedrock上的内部LLM网关发送请求;Airbnb会先在自有任务上评估模型质量与单任务成本,再监控生产环境的延迟、限流、令牌消耗和支出。Airbnb还会利用评估结果寻找表现相当的替代模型,并在成本上升时重新分配重度用户。
该报道展示了企业如何结合模型评估、Amazon Bedrock可观测性和单任务成本指标管理生产环境中的编程代理;其中一个用户群体的模型周支出降至约七分之一,请求量则大致不变。
Amazon Bedrock provides access to foundation models from multiple providers through a common API, with AWS managing the model infrastructure. It integrates with AWS access controls and monitoring, and processes prompts, code, and model responses within AWS.
As coding agents move into production, changes in models and usage can affect latency, capacity, and cost. Amazon Bedrock runtime metrics in Amazon CloudWatch help teams investigate latency increases, throttling, and changes in token consumption. Evaluations, or evals, complement these operational metrics by assessing the quality of an agent’s work on a fixed set of representative tasks.
In this post, we show how Airbnb evaluates models for coding agents and monitors their use on Amazon Bedrock. We explain how cost per task guides model selection and how production data helps us revisit those choices as usage changes. We also describe the runtime signals used to investigate latency, throttling, and spend.
At Airbnb, thousands of engineers and engineering-adjacent people use AirChat, our internal tool for running coding agents, every week. The agents send their model requests through our large language model (LLM) gateway, which runs on Amazon Bedrock. Bedrock processes Airbnb’s code, prompts, and model responses within AWS. Eight new models came out in about five weeks. By mid-October, about two thirds of the requests our engineers sent through coding agents went to those new models.