Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Client-Side Concurrency Cuts DeepSeek Agent Tool Latency

AI summary

A developer reported that changing 12 tool calls in an internal code-inspection agent from serial to concurrent execution reduced elapsed time from 94 seconds to about 23 seconds. Their trace attributed 71 seconds of the original run to tool and API waits and only eight seconds to model inference. The Juejin AI report details client-side scheduling, concurrency limits, exception handling, result validation, streaming aggregation, and a two-stage context strategy that reduced input from 21,000 to 5,100 tokens.

In-depth insight

The first-party engineering case illustrates that Agent latency can be reduced substantially through client-side execution design, while also showing that concurrency requires safeguards against ordering errors, oversized context, failed calls, and excessive interface load.

我把 Agent 的 12 个工具调用改成并发后,耗时从 94 秒掉到 23 秒:DeepSeek 并行 function calling 的工程细节

上个月我们内部那个代码巡检 Agent 上线,产品经理试用第一句话是"能不能快点"。我拉了下 trace,一个 12 文件的巡检任务 94 秒,其中 71 秒是纯等待——等文件读出来、等 grep 跑完、等内部搜索接口返回。模型本身的推理只占了 8 秒。

这不是模型慢,是我的执行层太老实。这篇记录我把这批调用从串行改成并发时踩的坑,以及最后落地的一套并发策略,包含 8 个具体问题和可复用的代码骨架。

一、并行工具调用的边界在哪

先说清楚责任划分,这一点搞混了后面全是坑。

模型在一次响应里可以返回多个 tool_calls,这是"提议"。真正把这些调用同时跑起来的是客户端。也就是说,并行能力在你手里,不在模型手里。DeepSeek 的 chat completions 接口会老老实实把多个调用放在一个数组里返回,但如果你用 for 循环一个个 await,那就还是串行。

#DeepSeek

Original · 6 min readRead the original
Group chat

No comments yet. Start the conversation.