SGLang
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 19
- Tencent open-sources DSpine, a parallel drafter for speculative decoding
Tencent published inference code and open weights for DSpine as tencent/Qwen3-8B-DSpine-b16 on Hugging Face under Apache 2.0. DSpine is a parallel drafter for speculative decoding: at every draft layer each position receives the predicted feature of its predecessor, so the causal chain unfolds over network depth while all positions are still drafted in parallel. The model card says the paper was released on arXiv in September 2026 and the inference code in October 2026, with training code and a Qwen3.8-27B version still on the roadmap. In the team's comparison on Qwen3-8B against DSpark, DFlash2, DFlash and Domino (all retrained with the same configuration on Open-PerfectBlend), DSpine reports the best average acceptance length, 7.83 at T=0 and 6.98 at T=1, though these figures are self-reported by the releasing team.
腾讯混元 · Hugging Face · 🔥 10 - SGLang adds native support for Reflection's Beam, a 501B-parameter MoE
SGLang's team writes that Reflection's Beam is a 501B-parameter mixture-of-experts model with 23B active parameters per token, aimed at software engineering and long-horizon agentic tasks and slated for an Apache 2.0 release. SGLang now supports Beam natively and powers Reflection's production API, with BF16, NVFP4 and dynamic FP8 checkpoints plus DFlash2 speculative decoding, prefix caching/HiCache and chunked prefill. The reported figures: replaying an OpenHands multi-turn agentic coding workload with the NVFP4 checkpoint on four NVIDIA GB300 GPUs, SGLang reached 127.6K total tokens/sec per GPU at 185.87 tokens/sec per user (p50) with concurrency 32.
LMSYS Blog · 🔥 9 - ModelSphere open-source platform targets production-grade AI deployment
On October 16, Paradigm Intelligence (范式智能) and Beijing Ruisizhilian Technology (睿思智联) announced ModelSphere (模域), a one-stop production-grade AI deployment platform that is free and open source for enterprise users under Apache 2.0 and is already available on GitHub. The platform combines compute management, model deployment and runtime optimization in one workflow, offering a graphical console, preset deployments, automatic elastic scaling, prefill-decode disaggregation, three-tier KV caching, high-availability load balancing and automatic service-resumption after stream interruptions, plus AutoTune scheduling intended to speed up first-token response. The vendors claim about 285% higher peak throughput and roughly 4x performance versus a baseline, figures that are company-stated and not independently verified; the platform bundles the EngineX inference engine to schedule heterogeneous Chinese accelerators from Huawei Ascend, Iluvatar CoreX, Moore Threads and Hygon DCU, and says it has optimized support for models such as DeepSeek-V4/V4-Flash, Qwen, Kimi, MiniMax and GLM.
InfoQ 中文 AI&大模型 · 🔥 8 - Apple, CMU's SSR turns agent reasoning into multiple-choice, cutting per-turn latency 90%
Researchers at Apple and Carnegie Mellon University propose SSR (Selection-Based Structured Reasoning), which replaces each free-form thinking step in an agent loop with a choice from a library of six predefined reasoning options. Scoring reuses the model's own token likelihoods and prefills all options in parallel, so no new tokens are generated autoregressively, with no extra classification head and no SFT warm start. On Qwen3-VL 2B/4B models with H100 + SGLang across seven multimodal search benchmarks (MMSearch, InfoSeek, SimpleVQA, LiveVQA and others), the paper reports per-turn reasoning latency down more than 90% and total per-question latency down 28–54%, while success rates match free-form thinking (4B SSR+GRPO averaging 61.37%, 2B reaching 51.26%). The authors say SSR plugs into existing SFT and RL pipelines and is best understood as reorganizing the reasoning space from open-ended language into a compact, reusable strategy space, noting that open-ended math, theorem proving and code design still need free generation.
机器之心 · 🔥 6 - Reflection AI releases Beam, a 501B-parameter open-weight model
Reflection AI released Beam, its first open-weight model, with Apache 2.0-licensed weights on Hugging Face and BF16, FP8 and NVFP4 checkpoint variants. Beam is a sparse mixture-of-experts model with 501B total parameters, 23B active parameters per token, and a reported effective context length of 1 million tokens. It targets coding, agentic and long-horizon tasks; training used longer reinforcement-learning rollouts plus a length penalty to discourage unnecessary tokens, and users can tune a reasoning-effort parameter. Day-0 support is available through Baseten Model APIs and Dedicated Inference and via Amazon SageMaker JumpStart in US East (N. Virginia) and US East (Ohio), with multilingual and multimodal releases planned for later.
Hugging Face · 🔥 35 - SGLang and Miles add day-0 support for Muse Spark Open
An LMSYS blog post says SGLang and Miles have added day-0 support for Muse Spark Open, an open-weight multimodal mixture-of-experts model. Its text backbone has 816B parameters with roughly 42B active parameters per token, accepts text, image, video and audio inputs, supports a 1M-token context window, and ships NVFP4 and BF16 checkpoints. The SGLang integration adds sequence parallelism with NVFP4 MegaMoE for prefill, multimodal preprocessing optimizations and DFlash V2 speculative decoding; the team reports that on 8 NVIDIA B300 GPUs with the NVFP4 checkpoint, an OpenHands multi-turn agentic coding replay reaches 14.8K total tokens/sec per GPU at concurrency 1 and 80.3K at concurrency 128.
LMSYS Blog · 🔥 5 - nFox launches Blockhouse sandbox, plans to open-source it
AI governance startup nFOX Lab launched the nFOX Platform for managing self-hosted large language models and said it plans to release its Blockhouse security sandbox under the Apache 2.0 license. Blockhouse removes privileges by default and restores access through policies enforced at the Linux kernel level, routing communications through a monitored channel called Porthole and keeping runtime credentials and audit controls outside the model process. It supports the vLLM, SGLang and llama.cpp inference engines, while the platform is available now across cloud VMs, rented GPU hosts and on-premises Linux servers with Nvidia GPUs; Blockhouse is in beta for qualified enterprise evaluations. Founder and CEO Gordon Allott said the company is self-funded, has paying beta pilot customers but is not yet profitable, and intends to raise venture capital and later expand into agent protection.
SiliconANGLE AI · 🔥 5 - Sipeed unveils MaixStation, a $3,999 Chinese rival to Nvidia's DGX Spark
Sipeed announced the MaixStation, a mini PC built on Axera's new AX8860P: 16 Arm Cortex-A55 cores plus an in-house NPU rated at 256 TOPS FP4 without sparsity, with 128GB of LPDDR5X on a 256-bit bus at 273 GB/s, running Debian 13 and Linux 6.18. Sipeed told Tom's Hardware the 128GB model should cost about $3,999, with a Kickstarter campaign at the end of the year, shipping within three months of it closing, and an NPU toolchain due by the end of this month; the company claims vLLM, SGLang and PyTorch support. It says a single node runs Qwen 27B with 25% faster prefill and 30% faster decode than the DGX Spark, but in its email claims only about a 10% lead on both with FP4 on each machine and speculative decoding off, calling wider-margin runs unofficial. For comparison, the 128GB DGX Spark lists at $6,950 and the 128GB AMD Ryzen AI Halo at $3,999.99.
Tom’s Hardware · 🔥 5 - StepFun opens Step 5 Preview weights: 600B MoE, 1M-token context
StepFun released BF16 and FP8 weights for Step 5 Preview, its flagship long-horizon agent model, on Hugging Face and ModelScope, plus a technical blog and deployment guides; the sparse MoE model has 600B total and 27B active parameters, supports up to a 1M-token context with text and image input, and targets software engineering, professional knowledge work and finance. According to StepFun, the design uses a 92-layer narrow-deep architecture with 3:1 gated sliding-window to sparse global attention and Sparse GQA, where 8-token block indexing cuts index and top-k selection cost to one eighth, plus FP32 key states, fine-grained gradient clipping and long-horizon RL that trains context compression; it can be served via vLLM or SGLang with an OpenAI-compatible API, or run offline with Transformers. StepFun-cited data puts it at 44 on Artificial Analysis's Intelligence Index and second on OpenRouter's weekly and daily token charts (about 18.8T tokens in seven days), at roughly $0.71 per task versus $5.86 for Claude Opus 5, and reports it above Kimi K3 and GLM-5.3 on ALE-CLI, FrontierFinance and DRACO, behind GPT-6 Astra or Claude Opus 5.
阶跃星辰 · Hugging Face · 🔥 43 - Baidu open-sources Unlimited-OCR, a 3B long-horizon document parsing model
Baidu published Unlimited-OCR on Hugging Face, a 3B image-text-to-text model under an MIT license that the card frames as "one-shot long-horizon parsing," aiming to push beyond DeepSeek-OCR. It supports single images in two configs (gundam and base) plus multi-page and PDF parsing with a 32,768 max length and no-repeat n-gram constraints, and the card ships instructions for Transformers, vLLM and SGLang inference, ms-swift training, a ModelScope mirror, Baidu Cloud availability and a Hugging Face Spaces demo. The release timeline runs from June 22, 2026, the paper is listed as arXiv 2606.23050, and the authors credit DeepSeek-OCR, DeepSeek-OCR-2 and PaddleOCR; post-processing code for OmniDocBench evaluation is also provided.
百度文心 · Hugging Face · 🔥 0 - Liquid AI releases LFM2.5-24B-A2B for on-device agents
Liquid AI released LFM2.5-24B-A2B on Oct. 13, 2026, an open-weight reasoning-only model with 24B total and roughly 2B active parameters, built for on-device agents; base and post-trained checkpoints are on Hugging Face. It extends context from 32,768 to 131,072 tokens, doubles vocabulary to 128K, scales pretraining from 17T to 33T tokens, and adds RL training inside agent harnesses like OpenClaw, Hermes Agent, Pi and OpenCode. Liquid AI reports it beats or matches Qwen3.6-35B-A3B on five of eight agentic/tool-use benchmarks (LiveClaw 48.09 vs 38.19; AutomationBench 27.69 vs 8.72), while trailing on SWE-Bench Verified (51.00 vs 66.20) and knowledge tests; it also reports reward-hacking behavior during teacher specialization. Day-one support covers llama.cpp, MLX, vLLM, SGLang and ONNX, with 184 tokens/s on Apple M5 Max and API pricing of $0.10/$1.00 per 1M input/output tokens.
Liquid AI Blog · 🔥 0 - Modal: mixed-precision, mixed-accelerator samplers enable stable async RL
In a technical blog post, Modal reports that running reinforcement-learning samplers on mixed precisions and mixed accelerators can keep large-model RL stable while cutting cost. Six kinds of accelerators feed a single BF16 trainer on B200: B200 and B300 serve NVFP4 weights, H100 and H200 serve FP8 and BF16 weights, and A100 and RTX PRO 6000 serve BF16 weights, each running its own kernels. The run trains Qwen3.6-35B-A3B on agentic coding with up to 256K tokens of context, using the 2.7K repository tasks Xiaomi released with MiMo-V2.6, and evaluates on SWE-bench Pro V2 and its HARD-51 subset. The authors add masked importance sampling, score centering and top-p sampling with mask replay as off-policy corrections; uncorrected GRPO collapses soon after training starts on the mixed samplers, while SC+MIS with or without top-p trains stably, with pass@4 rising over roughly 100–140 steps and levelling around 65–75% on HARD-51 and rising from 89% to about 94% on the full set, at about 15% lower sampler cost per update for most of training.
Modal Blog · 🔥 0 - Axera launches AX8860 AI chip, claims 25% faster LLM inference than DGX Spark
Axera unveiled its AX8860 series AI chip at the 2026 China International Industry Fair, positioning it as a compute base for running large models at the edge and on-device. The chip pairs a 16-core Arm Cortex-A55 CPU with a self-developed NPU rated at 256 TOPS (native FP4, no sparsity), supports up to 128GB of LPDDR5X memory with 273 GB/s of bandwidth over a 256-bit bus, is built on a 6nm process and offers PCIe 5.0 connectivity. Form factors include a 40W module, a single-slot PCIe card and a single-slot four-chip design that Axera claims reaches 1024 TOPS at roughly 80% scaling efficiency, plus planned Mini PCs and Pocket PCs; software support covers vLLM, SGLang and PyTorch with an OpenAI-compatible API, alongside the newly announced Yuanxi A256S module. Axera and Sipeed claim the chip is 25% faster on prefill and 30% faster on decode than NVIDIA's DGX Spark on Qwen 27B, at close to half the price (DGX Spark lists at $4,999 for 64GB and $6,950 for 128GB) — claims that have not been independently verified.
Wccftech · 🔥 0 - Zhipu releases GLM-5.3 and multimodal GLM-5.3-Flash open-weight models
Zhipu (Z.ai) published GLM-5.3 and GLM-5.3-Flash on Hugging Face. GLM-5.3 is a 753B-parameter open-weight model that reuses the GLM-5.2 base model, with all gains coming from post-training: the team says it improves 50% over GLM-5.2 on its in-house Z.ai Code Bench and reaches open-source state of the art on public benchmarks including Terminal Bench 3.0 and Agents' Last Exam, while expanded post-training also produced unexpected cyber capability, leading on CyberGym vulnerability discovery and more than doubling GLM-5.2 on exploitation benchmarks. GLM-5.3-Flash is the first natively multimodal model in the GLM-5 series, with 320B total and 18B active parameters, hybrid sparse/linear attention and mHC; Zhipu says it outperforms GLM-5.2 at one-tenth the price and approaches Claude Opus 4.8 on coding and agentic benchmarks. Both support local deployment via SGLang and vLLM and expose a reasoning_effort parameter (low/high/max, default max).
智谱 GLM · Hugging Face · 🔥 12 - SGLang trends on GitHub's Python list
SGLang is an open-source inference framework hosted by LMSYS for large language, vision-language and diffusion models, optimized for agentic workloads, large-scale serving and RL rollouts; the repository also includes SGLang Diffusion, its built-in image and video generation engine. It supports a wide range of GPUs, TPUs, NPUs, CPUs and Apple Silicon, including NVIDIA B200/B300 and H100, AMD Instinct, Google TPU, Huawei Ascend, and ships via Docker images or a uv install. The project offers a Cookbook, docs and learning resources such as Mini-SGLang, and long-term contributors can apply for coding-agent sponsorship covering Cursor, Claude Code or OpenAI Codex.
GitHub Trending · Python · 🔥 0 - Hugging Face Transformers tops GitHub Trending
Hugging Face's Transformers library appeared on GitHub Trending's daily list. The project positions itself as the model-definition framework for state-of-the-art text, vision, audio, video and multimodal models, for both inference and training, and its documentation says there are over 1M Transformers model checkpoints on the Hugging Face Hub. It says the library interoperates with training frameworks such as Axolotl, Unsloth, DeepSpeed, FSDP and PyTorch-Lightning, and with inference engines and modeling libraries including vLLM, SGLang, TGI, llama.cpp and mlx, and requires Python 3.10+ and PyTorch 2.6+.
GitHub Trending（每日） · 🔥 0 - AI infra spending boom meets idle GPUs at Beijing salon
A closed-door salon on AI infrastructure hosted by Touzhongwang with Zhongguancun partners, republished by 36Kr, examined the gap between huge AI infrastructure spending and weak efficiency. It cites ByteDance's $29.6 billion syndicated loan, Alibaba's HK$80 billion placement, more than $80 billion raised earlier by Google and Amazon, and NVIDIA's $500 billion plan, while noting China's daily token calls topped 140 trillion in March, over a thousand times early-2024 levels. Participants cited a Carnegie Mellon test of 756 GPUs running 31 days with the cluster waiting nearly 20% of the time, claimed over half of OpenAI's inference energy goes to idle work, and said average GPU utilization at Chinese smart-computing centers is below 30%. Several investors called lowering token cost and raising utilization the new deciding factor; SGLang's RadixArk reportedly raised a $100 million seed at a $400 million valuation, while one dissenting view argued inference revenue will fall sharply as algorithms compress token use by orders of magnitude.
36氪 人工智能 · 🔥 0 - SGLang and Miles tuned for NVIDIA Vera Rubin, serving Kimi K3 NVFP4
The SGLang & Miles team says it received early access to two NVIDIA Vera Rubin nodes (8 GPUs) and brought both SGLang inference and Miles RL training up on the platform. For serving the Kimi K3 NVFP4 checkpoint, it retuned attention, MoE/communication and KDA kernels, reporting a 16% faster deeper MLA pipeline at batch 16 and 128k context, a 20% faster split-KV reduction, a 5.9% end-to-end gain from fusing the MoE tail, and 276 fewer kernel launches per decode step. On the training side, Miles uses SGLang for rollout and Megatron for training, and ran end-to-end RL out of the box on a single 4-GPU tray with Qwen3-30B-A3B and DeepSeek-V4-Flash, plus agentic RL with 64 sandboxes on the Arm-based Vera CPU on SWE-bench Verified. The team also cites Cognition as reporting up to a 4.8x increase in total token throughput on Vera Rubin NVL72 versus GB200 NVL72 using its own private SGLang fork.
LMSYS Blog · 🔥 0 - NVIDIA Dynamo adds session-aware routing and KV cache for agentic inference
A PyTorch blog post details how NVIDIA Dynamo turns request-level serving into program-aware serving using one session-level identifier, with child sessions carrying a parent_session_id. Dynamo recognizes the session headers that Claude Code, Codex and OpenCode already emit, while other harnesses opt in via X-Dynamo-Session-ID, and plugins exist for Hermes and OpenClaw. The post also covers session-aware routing and admission control (porting a scheduler from the ThunderAgent paper), a shared KV cache indexer, the Session-Prefix Indexer and proposed KvHint interface, plus DYN_REQUEST_TRACE=1 trace capture with AISimulate offline replay and AIPerf live benchmarking.
PyTorch Blog · 🔥 0
Experience and discussion from the community
Share my SGLang experienceAsk about SGLang
Nobody has shared their experience with SGLang yet.