创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

长程编程智能体如何利用前缀缓存

AI 总结

Together AI 与 Ollama 介绍,编程智能体每轮都会重发完整对话,形成长输入、短工具调用的流量模式。双方建议保持提示词前缀稳定、让请求命中缓存所在 GPU,并控制工具结果与缓存内存;Together AI 还称其会将预填充与解码分置于不同集群,并通过 RDMA 传输缓存状态。

深度见解

这份指南为长程智能体会话中的提示词设计、缓存路由和推理基础设施提供了具体实践,但相关架构细节来自厂商本身。

Summary

Coding agents send the whole conversation again on every turn. Prefix caching keeps the inference engine from redoing that work, but only if the harness leaves the start of the prompt alone and the engine sends each turn back to the GPU that already has it cached. Together AI and Ollama work on opposite sides of that handoff, so we wrote down what each, the client side and the inference engine side, needs to do to successfully build agentic apps.

Coding agents have an odd traffic pattern. Every time the agent takes a step, the harness sends the model the entire conversation so far: the system prompt, every tool definition, your request, every file the agent has read, and every command it has run. What comes back is usually a tool call a few dozen tokens long. Then it happens again, with everything from last time plus a little more.

This large input sequence length (ISL) and small output sequence length (OSL) is characteristic of most agentic workloads. If the GPU had to work through that whole transcript from scratch on every turn, long sessions would get slower and more expensive with every step. Prefix caching is what saves most of that work. The problem is that if you’re not careful it's also very easy to break the prefix cache.

#Together AI#Ollama#GLM-5.3-Flash

原文约 13 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。