Key Takeaways

• •Building long-running agents takes constant context management, and making those decisions with a frontier LLM is slow and expensive. ScaleDown’s constrained decoding models (CDMs) make them faster and cheaper.
• •langchain-scaledown makes it easy to use ScaleDown models in your agent, and ships with middleware for summarization, tool result compression, and extraction.
• •ScaleDown is now supported in the LangSmith LLM Gateway, so its calls are traced and count toward your spend and rate limits.

Building long-running agents comes down to the art of context management: deciding which information goes into the context window, and when. That means summarizing old history, compacting large tool results, and extracting the right data from long inputs. Each of those steps is a decision. Historically we’ve made them with LLMs, which is slow and expensive at the volume a long-running agent needs.

To make those decisions faster and cheaper, we now have a langchain-scaledown integration and support in the LangSmith LLM Gateway for ScaleDown’s constrained decoding models (CDMs). Where a general-purpose LLM writes any string one token at a time, a CDM is trained for one task and only returns answers that task allows, like a label from your list or spans of the input. That lets it be much smaller, faster, and cheaper: ScaleDown reports its models are 200x faster and 467x cheaper than frontier models on these narrow tasks.