AI 编程代理加速 SGLang 接入 IBM Spyre
一个两人团队借助 AI 编程代理,在不到一周内将 SGLang 集成到 IBM Spyre 加速器,并基于现有 PyTorch 支持和 SGLang 扩展点完成约 1,500 行插件。PyTorch Blog 称,测试中 SGLang 的吞吐量约为 torch-spyre 参考 PyTorch 推理实现的 3.6 倍,在 Granite 3.3 8B 和 Gemma 4 26B A4B 工作负载上与 vLLM 持平,RadixCache 前缀共享也无需针对 Spyre 修改。该插件已随上游 SGLang 以开源形式提供。
该案例显示,借助现有框架抽象和 AI 编程代理,开发团队可以显著缩短将新型 AI 加速器接入服务栈的工作时间。
TL;DR
With help from AI coding agents, our two-person team integrated SGLang with IBM Spyre in less than a week, building a plugin of roughly 1,500 lines. Existing Spyre support in PyTorch and SGLang’s clear extension points made it straightforward to reuse the hardware code. SGLang delivered about 3.6× the throughput of the reference PyTorch inference implementation using torch-spyre. It was also on par with vLLM on the workloads measured for Granite 3.3 8B and Gemma 4 26B A4B. Prefix sharing through RadixCache worked without any Spyre-specific changes.
We work on IBM’s Spyre accelerator and wanted to make SGLang‘s serving stack, including its built-in prefix sharing, available to Spyre users.
In less than a week, the two of us used AI-assisted development to integrate SGLang into IBM Spyre with a plugin of just ~1,500 lines of code. A key factor was building on existing PyTorch hardware support and SGLang’s well-defined extension points. This allowed the two of us, alongside our AI coding agents, to integrate a new accelerator into SGLang in a remarkably short time. The AI coding agents used existing PyTorch implementations of Spyre operations as reference code and adapted the integration code to SGLang’s backend interfaces. Their work focused on connecting the existing hardware support to the new serving framework.
#SGLang#IBM Spyre#PyTorch#torch-spyre#vLLM#RadixCache#Granite 3.3 8B#Gemma 4 26B A4B