How Prefix Caching Keeps Long-Horizon Coding Agents Efficient
Together AI and Ollama explain how coding agents resend the full conversation each turn, creating long inputs and short tool-call outputs. They recommend stable prompt prefixes, cache-aware GPU routing, concise tool results, and sufficient cache memory; Together AI also says it separates prefill and decode across clusters and transfers cached state over RDMA.
The guide provides concrete harness and inference-engine practices for preserving prefix-cache efficiency in long-running agent sessions, though the architecture details come from the vendors involved.
Summary
Coding agents send the whole conversation again on every turn. Prefix caching keeps the inference engine from redoing that work, but only if the harness leaves the start of the prompt alone and the engine sends each turn back to the GPU that already has it cached. Together AI and Ollama work on opposite sides of that handoff, so we wrote down what each, the client side and the inference engine side, needs to do to successfully build agentic apps.
Coding agents have an odd traffic pattern. Every time the agent takes a step, the harness sends the model the entire conversation so far: the system prompt, every tool definition, your request, every file the agent has read, and every command it has run. What comes back is usually a tool call a few dozen tokens long. Then it happens again, with everything from last time plus a little more.
This large input sequence length (ISL) and small output sequence length (OSL) is characteristic of most agentic workloads. If the GPU had to work through that whole transcript from scratch on every turn, long sessions would get slower and more expensive with every step. Prefix caching is what saves most of that work. The problem is that if you’re not careful it's also very easy to break the prefix cache.