LangChain adds ScaleDown context middleware for long-running agents
LangChain's blog introduced the langchain-scaledown integration and said ScaleDown's constrained decoding models are now a direct model provider in the LangSmith LLM Gateway. The package ships with three middlewares for summarizing old history, compressing large tool results and extracting structured data, covering ScaleDown's four single-task models: Summarize, Compress, Extract and Classify. LangChain says each model accepts up to 1M tokens of input and that some handle images and scanned PDFs; ScaleDown's own benchmarks claim large speed and cost advantages over frontier models on narrow tasks, without independent verification in the post.
LangChain says long-running agents require constant context-management decisions, and it positions ScaleDown's narrow models as a cheaper, faster alternative to frontier LLMs for those steps.
Key Takeaways
- Building long-running agents takes constant context management, and making those decisions with a frontier LLM is slow and expensive. ScaleDown’s constrained decoding models (CDMs) make them faster and cheaper.
langchain-scaledownmakes it easy to use ScaleDown models in your agent, and ships with middleware for summarization, tool result compression, and extraction.- ScaleDown is now supported in the LangSmith LLM Gateway, so its calls are traced and count toward your spend and rate limits.
Building long-running agents comes down to the art of context management: deciding which information goes into the context window, and when. That means summarizing old history, compacting large tool results, and extracting the right data from long inputs. Each of those steps is a decision. Historically we’ve made them with LLMs, which is slow and expensive at the volume a long-running agent needs.
To make those decisions faster and cheaper, we now have a langchain-scaledown integration and support in the LangSmith LLM Gateway for ScaleDown’s constrained decoding models (CDMs). Where a general-purpose LLM writes any string one token at a time, a CDM is trained for one task and only returns answers that task allows, like a label from your list or spans of the input. That lets it be much smaller, faster, and cheaper: ScaleDown reports its models are 200x faster and 467x cheaper than frontier models on these narrow tasks.
#LangChain#ScaleDown#LangSmith LLM Gateway#Deep Agents#LangSmith#langchain-scaledown