Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Testing Agent Tool Calls: Idempotency Under 504 Timeouts and Jitter

AI summary

A Juejin AI article argues that quality teams testing AI agents should stop treating tool calls as simple success/failure HTTP outcomes, proposing a three-state model: confirmed success, confirmed failure, and in-doubt/timeout where the backend may have executed nothing, executed the write, or completed asynchronously. It then lays out a fault-injection proxy setup (via tools such as WireMock, Mitmproxy or a lightweight Envoy gateway) covering four cases: server executed but response timed out, client abort while the server finishes asynchronously, argument tweaks that invalidate a parameter-hash idempotency key, and repeated failures requiring exponential backoff and a circuit-breaker retry cap. The proposed assertions include exactly one database record per business key, SDK-injected X-Idempotency-Key headers that never change on retries, and explicit user-facing reporting of an undetermined business state rather than fabricating success.

In-depth insight

It turns the common 504-then-retry failure into concrete, checkable test cases for production agent systems, including the assertion that only one write reaches the database and that idempotency keys are managed by the framework rather than left to the model.

给 Agent 配上外部工具(Tool Calling / Function Calling)之后,质量团队常把精力放在「模型能不能选对工具、参数填得对不对」上。但在生产环境下,真正导致业务故障的,往往是网络抖动引发的副效应失控。

最典型的现象是:Agent 发起一次带写操作的工具调用(例如扣减库存、创建工单、执行转账),请求已经打到后端并完成持久化,但网关由于偶发超时或链路抖动,向 Agent 返回了 504 Gateway Timeout。大模型接收到底层工具抛出的超时异常后,自愈机制通常会立即生成下一轮动作——重新调用该工具或调整入参再试一次。如果底层工具没有严格的幂等控制,重复扣款、重复建单、脏数据堆积就会随之产生。

本文给出一套针对 Agent 工具调用在超时、抖动场景下的幂等与补偿测试验证方案。

Agent工具调用超时幂等时序

#WireMock#Mitmproxy#Envoy#Kafka

Original · 6 min readRead the original
Group chat

No comments yet. Start the conversation.