FlashInfer

Headlines · 7

1. 5h
   k2-fsa releases OmniVoice, a zero-shot TTS model claiming 600+ languages

   The GitHub project k2-fsa/OmniVoice surfaced on GitHub Trending, describing itself as a massively multilingual zero-shot text-to-speech model covering more than 600 languages. According to its README, it uses a "diffusion language model-style" architecture and supports voice cloning plus voice design via speaker attributes such as gender, age, pitch, accent, dialect and whisper, along with inline tags like [laughter] and pinyin or phoneme pronunciation correction. The project also reports self-measured figures, including an RTF as low as 0.025 (about 40x faster than real time) and FlashInfer speedups over baseline on a single H100; these numbers are vendor-reported and no independent evaluation is cited.

   GitHub Trending · Python · 🔥 9
2. 3d
   Tencent releases DSpine, a parallel speculative-decoding drafter for Qwen3-8B

   Tencent published tencent/Qwen3-8B-DSpine-b16 on Hugging Face under the Apache 2.0 license. The model card describes DSpine as a parallel drafter for speculative decoding in which, at every draft layer, each position receives the predicted feature of its predecessor so the causal chain unfolds over network depth while all positions are still drafted in parallel; its News section says inference code was released in October 2026 and the paper appeared on arXiv in September 2026. The authors self-report acceptance-length results on Qwen3-8B, saying DSpine, DSpark and DFlash2 were all retrained on Open-PerfectBlend with the same configuration, with DSpine averaging 7.83 at T=0 versus 7.55 for DFlash2, while training code and a Qwen3.8-27B version remain unchecked roadmap items.

   腾讯混元 · Hugging Face · 🔥 0
3. 3d
   SGLang adds native support for Reflection's Beam, a 501B-parameter MoE

   SGLang's team writes that Reflection's Beam is a 501B-parameter mixture-of-experts model with 23B active parameters per token, aimed at software engineering and long-horizon agentic tasks and slated for an Apache 2.0 release. SGLang now supports Beam natively and powers Reflection's production API, with BF16, NVFP4 and dynamic FP8 checkpoints plus DFlash2 speculative decoding, prefix caching/HiCache and chunked prefill. The reported figures: replaying an OpenHands multi-turn agentic coding workload with the NVFP4 checkpoint on four NVIDIA GB300 GPUs, SGLang reached 127.6K total tokens/sec per GPU at 185.87 tokens/sec per user (p50) with concurrency 32.

   LMSYS Blog · 🔥 0
4. 4d
   vLLM adds day-0 support for Reflection's 501B Beam model with Blackwell optimizations

   vLLM published day-0 serving support for Reflection's Beam, a 501B-parameter mixture-of-experts model with 23B active parameters per token, optimized for NVIDIA Blackwell. The engine handles reasoning, tool calling and structured outputs out of the box and can run Beam with tensor, data or expert parallelism, with a Docker image and launch command provided. The post says coding agents autonomously tuned the implementation so batch-1 decode is 2.71× faster and prefill 1.87× faster than the first version, and that Beam's DFlash2 drafter reaches 708 tokens/s for a single request on TP8 with an acceptance length of 3.7. It also describes day-0 reinforcement-learning support built with the SkyRL team, reporting AIME25 average accuracy rising from about 61% to about 79% after 50 training steps.

   vLLM Blog · 🔥 0
5. 6d
   AWS details fused MoE kernel optimizations on SageMaker, up to 10% throughput gain

   An AWS Machine Learning Blog post describes several fused Mixture-of-Experts (MoE) kernel optimizations that AWS developed with the vLLM community and open sourced. A Swap-AB transposition that moves the large expert dimension into the leading WGMMA operand slot delivered a 2–4% end-to-end output throughput improvement on H200 and was upstreamed to vLLM 0.24.0 and later. Routing FP8 tensor-quantized MoE workloads to the Triton backend instead of FlashInfer+CUTLASS added about 10% throughput (available in the AWS LMI v28 container), while adding do_not_specialize annotations to variable-length parameters removed repeated Triton JIT recompilations and improved time to first token by 9% on H200 at 8-way tensor parallelism (vLLM 0.22.0 and later). AWS says the optimizations together yield up to 10% higher throughput and up to 9% faster TTFT.

   AWS Machine Learning Blog · 🔥 0
6. Oct 11
   SGLang trends on GitHub's Python list

   SGLang is an open-source inference framework hosted by LMSYS for large language, vision-language and diffusion models, optimized for agentic workloads, large-scale serving and RL rollouts; the repository also includes SGLang Diffusion, its built-in image and video generation engine. It supports a wide range of GPUs, TPUs, NPUs, CPUs and Apple Silicon, including NVIDIA B200/B300 and H100, AMD Instinct, Google TPU, Huawei Ascend, and ships via Docker images or a uv install. The project offers a Cookbook, docs and learning resources such as Mini-SGLang, and long-term contributors can apply for coding-agent sponsorship covering Cursor, Claude Code or OpenAI Codex.

   GitHub Trending · Python · 🔥 0
7. Oct 9
   vLLM adds day-0 support for NVIDIA Vera Rubin NVL72, claims 7.8x throughput over GB200

   vLLM says it now runs on NVIDIA's Vera Rubin NVL72, with nightly CUDA 13.4 and PyTorch 2.15 container images and support for models from DeepSeek, Moonshot AI's Kimi, Z.ai's GLM and MiniMax. Working with NVIDIA, Red Hat and Inferact, the project reports early results of 7.8x throughput per GPU versus GB200 NVL72 on AgentX at matched interactivity and up to 3.7x higher VLM throughput versus GB300 NVL72 in MLPerf; these are its own early figures. MiniMax separately said its M3 model is now running on Rubin with vLLM, crediting the same collaborators.

   vLLM Blog · 🔥 0

Experience and discussion from the community

Share my FlashInfer experienceAsk about FlashInfer