Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Agent observability: tracing a wrong number across four handoffs

AI summary

A technical article uses a teaching scenario in which an agent cites the correct announcement but reports an incorrect number, with no API error and an all-green tracing page. It proposes four checkpoints: source to tool result, tool result to actual model request, request to generated answer, and generated answer to user-visible text. The article also draws on the authors’ own Langfuse observability records, covering a 256 KiB body-capture limit, a double-slash URL that redirected requests and caused a background export error, an isolated ClickHouse recovery drill, and de-duplication of cost records representing the same physical request. The opening scenario is explicitly a teaching exercise; possible causes such as missing units, omitted footnotes, incomplete input and model misunderstanding are branches rather than a confirmed root cause.

In-depth insight

The article argues that a successful call and green dashboard do not establish that an answer is correct. Locating the first deviation from the facts can guide whether to fix source selection, input assembly, model reasoning or delivery. Its engineering cases come from the authors’ own deployment and are not independently confirmed.

下面用一个教学任务开始:用户请 Agent 核对一份公告中的数字,生成带出处的报告。报告引用了正确的公告,数字却错了。接口没有报错,追踪页面也一片绿色。接下来应该换模型、改提示词,还是修读取工具?

只知道“调用成功”,还不足以作出这个选择。原文可能漏了单位,工具可能丢了脚注,模型可能没有收到完整结果,也可能读到了却理解错误。甚至模型生成的内容没错,用户看到的却是另一版正文。

有用的 Agent 可观测性,要让我们找到判断第一次偏离事实的位置,并知道手中的记录还缺什么。 本文沿这份报告说明怎样检查交接、怎样面对缺失,以及怎样验证观测工具自身。教学推演负责解释排查方法;文中另外标明的工程案例,则来自我们建设 Langfuse 观测链的实际记录。

一、先找到四次交接,再决定修哪里

追踪记录(Trace)可以把一项任务中的模型请求、工具调用和服务过程联系起来。但一条完整的调用树,仍不能代替内容核对。对于开头的报告,值得检查的是四次交接:

#Langfuse#OpenTelemetry#PostgreSQL#ClickHouse#Redis#MinIO#OTLP

Original · 13 min readRead the original
Group chat

No comments yet. Start the conversation.