ModelSphere details an open-source gateway for resuming interrupted LLM streams
ModelSphere presents an open-source gateway that sits between clients and inference engines such as SGLang and vLLM to continue interrupted streaming responses. The design uses at most two request legs, constructs continuation prompts from recovered reasoning and content, and aligns response fields, token budgets and usage estimates. ModelSphere says compatibility varies by model and engine and must be tested for each combination. The gateway does not support structured JSON output or continuation after a tool-call chunk has already been streamed, and the report says semantic continuity at the interruption point has no automated guarantee.
本文是 ModelSphere 核心技术硬核拆解系列第一篇。我们将讲清生产级流式推理里,「断流续写」如何在网关层落地:为什么整段重试不够、续写请求怎么构造、模型 × 引擎组合要踩哪些坑,以及哪些边界我们刻意不支持。
大模型的长回答,很少是一口气吐完的。更多时候,字是一段一段流出来的:你这边看着屏幕往下蹦,服务那边在算、在调度、在占着一张卡。
麻烦往往出在半路上。流到一半,负责生成的那台机器挂了、被挪走了,或者连接断了——屏幕上的字突然停住。刚才等了很久才出来的那一大段,常规做法几乎只有整段重试:推倒重来,用户再等一轮。
对用户,是重等;对平台,是已经烧出去的算力打了水漂。很多人把它当成「再试一次」的运维问题,其实卡点不在能不能重启服务,而在:已经生成出来的内容,能不能被接着用。