Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Airbnb runs coding agents on Amazon Bedrock with cost-per-task evals

AI summary

AWS Machine Learning Blog reports that thousands of engineers and engineering-adjacent staff at Airbnb use the internal AirChat tool to run coding agents each week. The agents send requests through Airbnb's LLM gateway on Amazon Bedrock, where the company evaluates models using its own tasks and cost per task before rollout, then monitors production latency, throttling, token consumption, and spend. Airbnb also uses evaluation results to identify equivalent models and rebalance heavy users when costs rise.

In-depth insight

The report provides a concrete enterprise example of combining model evaluations with Bedrock observability and cost-per-task metrics to manage coding agents in production, including a case where one group's weekly model spend fell to about one-seventh with roughly the same request volume.

Amazon Bedrock provides access to foundation models from multiple providers through a common API, with AWS managing the model infrastructure. It integrates with AWS access controls and monitoring, and processes prompts, code, and model responses within AWS.

As coding agents move into production, changes in models and usage can affect latency, capacity, and cost. Amazon Bedrock runtime metrics in Amazon CloudWatch help teams investigate latency increases, throttling, and changes in token consumption. Evaluations, or evals, complement these operational metrics by assessing the quality of an agent’s work on a fixed set of representative tasks.

In this post, we show how Airbnb evaluates models for coding agents and monitors their use on Amazon Bedrock. We explain how cost per task guides model selection and how production data helps us revisit those choices as usage changes. We also describe the runtime signals used to investigate latency, throttling, and spend.

At Airbnb, thousands of engineers and engineering-adjacent people use AirChat, our internal tool for running coding agents, every week. The agents send their model requests through our large language model (LLM) gateway, which runs on Amazon Bedrock. Bedrock processes Airbnb’s code, prompts, and model responses within AWS. Eight new models came out in about five weeks. By mid-October, about two thirds of the requests our engineers sent through coding agents went to those new models.

#Airbnb#AirChat#Amazon Bedrock#Amazon CloudWatch#AWS

Original · 15 min readRead the original
Group chat

No comments yet. Start the conversation.