NIXL

Headlines · 2

1. 3d
   SGLang adds native support for Reflection's Beam, a 501B-parameter MoE

   SGLang's team writes that Reflection's Beam is a 501B-parameter mixture-of-experts model with 23B active parameters per token, aimed at software engineering and long-horizon agentic tasks and slated for an Apache 2.0 release. SGLang now supports Beam natively and powers Reflection's production API, with BF16, NVFP4 and dynamic FP8 checkpoints plus DFlash2 speculative decoding, prefix caching/HiCache and chunked prefill. The reported figures: replaying an OpenHands multi-turn agentic coding workload with the NVFP4 checkpoint on four NVIDIA GB300 GPUs, SGLang reached 127.6K total tokens/sec per GPU at 185.87 tokens/sec per user (p50) with concurrency 32.

   LMSYS Blog · 🔥 0
2. Oct 12
   vLLM adds bi-directional KV transfer to cut TTFT in multi-turn agent workloads

   The vLLM blog describes bi-directional KV cache transfer for prefill-decode (P-D) disaggregated serving: on later turns of a conversation, a decode node can return previously computed KV to the prefill node so shared context is not recomputed. In multi-turn benchmarks on a 1P1D deployment using Amazon EC2 P5en instances with AWS Elastic Fabric Adapter, the feature reduced TTFT by up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8, with results depending on workload and configuration. The capability was originally built and validated on AWS Trainium clusters and contributed to vLLM as an accelerator-agnostic feature.

   vLLM Blog · 🔥 0

Experience and discussion from the community

Share my NIXL experienceAsk about NIXL