Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

vLLM

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 195

  1. vLLM open-sources bitwise-consistent RL training kernels and an agent skill

    A vLLM blog post argues that inference-side optimizations in high-performance RL — low-precision compute, low-precision KV cache, kernel and communication tuning — shift the numerics, so the training engine assigns different probabilities to the same token than the engine that sampled it; on DeepSeek-V4-Flash the high-performance baseline's train-inference KL rose from 2e-3 to 29.9 as reward fell back. The proposed fix has the training forward call vLLM's inference kernels directly, modified so rollout and training produce bitwise-identical logprobs for every sampled token (diff = 0). The authors report diff of 0 at every one of 100 DeepSeek-V4-Flash steps at 91.3% of the baseline's rollout throughput, and say that on Nemotron-3.5 (a Mamba hybrid with NVFP4) the consistent run's SWE-bench Pro score held near 0.6 while the baseline's fell from 0.56 to 0.22. The workflow is packaged as an agent skill and released with recipes and code, and the team says DeepSeek-V4.1 has reached 0 diff with performance work still in progress.

    vLLM Blog · 🔥 0
  2. vLLM adds push-based NIXL KV transfer for disaggregated inference

    Posts on the vLLM Blog and the AWS Machine Learning Blog describe a push-based NIXL connector for prefill-decode (P-D) disaggregated inference in vLLM. In the default pull mode, the decode instance allocates destination blocks and issues an NIXL READ only after prefill finishes, so coordination and setup sit on the time-to-first-token critical path; in push mode the decode side registers blocks in advance and the prefill worker hands the transfer to a background thread that issues an NIXL WRITE. The authors say the feature was built and validated on AWS Trainium clusters with EFA and then contributed to vLLM as an accelerator-agnostic feature.

    vLLM Blog · 🔥 0
  3. Liquid AI releases LFM2.5-24B-A2B, a 24B open-weight reasoning model for on-device agents

    Liquid AI released LFM2.5-24B-A2B on Oct. 13, an open-weight reasoning model with 24B total parameters and roughly 2B active per token (Liquid AI elsewhere cites 2.45B), aimed at running agents locally on laptops and PCs. It is the largest and final member of the LFM2.5 family, with the context window extended from 32K to 131K tokens, a vocabulary doubled from 65K to 128K for better non-Latin tokenization, pretraining scaled from about 17T to 33T tokens, and a switch to reasoning-only output with an explicit chain of thought. Liquid AI says it matches the larger Qwen3.6-35B-A3B on five of eight agentic and tool-use benchmarks (for example 48.09 versus 38.19 on LiveClaw), though those benchmarks and the speed figures (184 tokens/s on an Apple M5 Max) are vendor-run. Base and post-trained weights are on Hugging Face with day-one support in llama.cpp, MLX, vLLM, SGLang and ONNX.

    Liquid AI Blog · 🔥 0
  4. AWS details fused MoE kernel optimizations upstreamed to vLLM

    An AWS Machine Learning Blog post describes several kernel optimizations AWS implemented for fused Mixture-of-Experts models running on GPUs. It says a swap-AB matrix transposition aligned with NVIDIA Hopper's WGMMA instruction layout produced a 2–4 percent end-to-end throughput improvement for large MoE models on H200, and was upstreamed to vLLM 0.24.0 or later. A second change added do_not_specialize annotations to eliminate repeated Triton JIT recompilation, which AWS says improved time to first token by 9 percent at 8-way tensor parallelism and is available in vLLM 0.22.0 or later. AWS says the optimizations are open sourced, applied in the LMI deep learning container on Amazon SageMaker AI, and together delivered up to 10 percent higher throughput and up to 9 percent faster TTFT; these figures are vendor-reported and not independently verified.

    AWS Machine Learning Blog · 🔥 0
  5. AWS shows multi-node inference with Ray LLM Deep Learning Containers

    An AWS Machine Learning Blog post walks through deploying one large language model across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Container, with separate prefill and decode nodes behind a single OpenAI-compatible endpoint. The container ships Ray Serve LLM, NVIDIA's NIXL library and the Elastic Fabric Adapter stack, and the example uses Qwen3.5-9B to show prefill/decode disaggregation configured entirely through YAML with no application code. The post also warns that NIXL silently falls back to a TCP-speed transport when it cannot use libfabric, so it recommends checking worker logs to confirm the backend.

    AWS Machine Learning Blog · 🔥 0
  6. vLLM adds bi-directional KV transfer to cut TTFT in multi-turn agents

    vLLM and the AWS Machine Learning Blog describe bi-directional KV cache transfer in vLLM: in prefill-decode disaggregated serving, a decode node can return previously computed KV blocks to the prefill node on later turns, avoiding recomputation of the conversation history. On a 1P1D deployment using Amazon EC2 P5en instances with AWS EFA, the authors report TTFT reductions of up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8, noting that results vary with workload and configuration. Using it requires vLLM v0.21.0 or later with NixlConnector, enabling bidirectional_kv_xfer on both prefill and decode servers, and a KV-cache-aware router, with decode-side block retention governed by decoder_kv_blocks_ttl (default 480 seconds).

    vLLM Blog · 🔥 0
  7. SGLang trends on GitHub's Python list

    SGLang is an open-source inference framework hosted by LMSYS for large language, vision-language and diffusion models, optimized for agentic workloads, large-scale serving and RL rollouts; the repository also includes SGLang Diffusion, its built-in image and video generation engine. It supports a wide range of GPUs, TPUs, NPUs, CPUs and Apple Silicon, including NVIDIA B200/B300 and H100, AMD Instinct, Google TPU, Huawei Ascend, and ships via Docker images or a uv install. The project offers a Cookbook, docs and learning resources such as Mini-SGLang, and long-term contributors can apply for coding-agent sponsorship covering Cursor, Claude Code or OpenAI Codex.

    GitHub Trending · Python · 🔥 0
  8. Hugging Face Transformers tops GitHub Trending

    Hugging Face's Transformers library appeared on GitHub Trending's daily list. The project positions itself as the model-definition framework for state-of-the-art text, vision, audio, video and multimodal models, for both inference and training, and its documentation says there are over 1M Transformers model checkpoints on the Hugging Face Hub. It says the library interoperates with training frameworks such as Axolotl, Unsloth, DeepSpeed, FSDP and PyTorch-Lightning, and with inference engines and modeling libraries including vLLM, SGLang, TGI, llama.cpp and mlx, and requires Python 3.10+ and PyTorch 2.6+.

    GitHub Trending(每日) · 🔥 0
  9. Tencent releases 5B omni-modal parsing model Youtu-Parsing-Omni

    Tencent Youtu Lab published Youtu-Parsing-Omni on Hugging Face, a compact 5B omni-modal parsing model that turns a document page, natural image, chart or flowchart, geometry figure, audio clip or audio-visual video into a single structured JSON output, with the output family selected by a task prompt. The model card says a single omni encoder handles image, audio and interleaved audio-visual video inputs, and that a vLLM plugin, pinned serving settings, task prompts and inference examples are included. Tencent reports 96.96 Overall on OmniDocBench v1.6, 75.08 Avg. on OmniParsingBench as the best open-weight model behind Gemini-3-Pro, and competitive results against specialized models on chemical-structure and music-score recognition; all of these benchmarks are vendor-reported and were not independently verified in the supplied material.

    腾讯混元 · Hugging Face · 🔥 0
  10. vLLM releases proto version 0.5.0

    The vLLM project published version 0.5.0 of vllm-proto on GitHub. The source provides only the release tag and version number, with no release notes or feature details.

    vLLM Releases · 🔥 0
  11. vLLM adds day-0 support for NVIDIA Vera Rubin NVL72, claims 7.8x throughput over GB200

    vLLM says it now runs on NVIDIA's Vera Rubin NVL72, with nightly CUDA 13.4 and PyTorch 2.15 container images and support for models from DeepSeek, Moonshot AI's Kimi, Z.ai's GLM and MiniMax. Working with NVIDIA, Red Hat and Inferact, the project reports early results of 7.8x throughput per GPU versus GB200 NVL72 on AgentX at matched interactivity and up to 3.7x higher VLM throughput versus GB300 NVL72 in MLPerf; these are its own early figures. MiniMax separately said its M3 model is now running on Rubin with vLLM, crediting the same collaborators.

    vLLM Blog · 🔥 0
  12. JetBrains releases Mellum2.1 coding model

    JetBrains launched Mellum2.1, keeping Mellum2's 12B mixture-of-experts architecture with 2.5B active parameters and the Apache 2.0 license, and focusing the upgrade on agentic coding. JetBrains says reinforcement learning was expanded from a short post-training step into the main training stage, letting the model explore codebases, edit files, identify root causes of failing tests and draft and verify fixes. The model is available on Hugging Face, with GGUF builds for llama.cpp, Ollama and LM Studio and a vLLM MTP speculative-decoding component promised later.

    IT之家 AI · 🔥 0
  13. NVIDIA Dynamo adds session-aware routing and KV cache for agentic inference

    A PyTorch blog post details how NVIDIA Dynamo turns request-level serving into program-aware serving using one session-level identifier, with child sessions carrying a parent_session_id. Dynamo recognizes the session headers that Claude Code, Codex and OpenCode already emit, while other harnesses opt in via X-Dynamo-Session-ID, and plugins exist for Hermes and OpenClaw. The post also covers session-aware routing and admission control (porting a scheduler from the ThunderAgent paper), a shared KV cache indexer, the Session-Prefix Indexer and proposed KvHint interface, plus DYN_REQUEST_TRACE=1 trace capture with AISimulate offline replay and AIPerf live benchmarking.

    PyTorch Blog · 🔥 0
  14. vLLM climbs GitHub Trending among Python projects

    vLLM is a high-throughput, memory-efficient inference and serving engine for large language models, originally developed in the Sky Computing Lab at UC Berkeley and now maintained by a community of more than 2,000 contributors. It manages attention key-value memory with PagedAttention and supports continuous batching, chunked prefill, prefix caching, speculative decoding and quantization formats including FP8, INT8/INT4, GPTQ/AWQ and GGUF, alongside an OpenAI-compatible API server. The engine runs on NVIDIA, AMD and Intel GPUs plus x86/ARM/PowerPC CPUs, and supports 200+ model architectures on Hugging Face.

    GitHub Trending · Python · 🔥 0
  15. vLLM v0.31.0 expands serving, speculation, and model support

    vLLM v0.31.0 adds broad serving and model-support updates, including DeepSeek-V4.1-Flash optimizations, Model Runner V2 speculative decoding, larger-scale expert parallelism, and improved multimodal support. It also introduces the `vllm preload` weight-cache daemon for faster engine restarts and experimental CRIU-based initialized-engine snapshots, alongside several security and breaking changes.

    vLLM Releases · 🔥 0

← Previous

Experience and discussion from the community

Share my vLLM experienceAsk about vLLM

torchtitan: A PyTorch-native platform for pretraining and post-training generative AI models at scale.

What problem it solves Training generative AI models at scale requires coordinating multiple parallelism strategies while supporting pretraining, fine-tuning, and reinforcement learning in one codebase. Large-scale training also needs reproducible configurations, checkpointing, debugging tools, and observability that can be adapted for research and model development. Key features Composable model …

github.comhttps://github.com/pytorch/torchtitan

vllm-ascend: A community-maintained hardware plugin for running vLLM on Huawei Ascend NPUs.

What problem it solves vLLM users need a way to run the framework on Huawei Ascend NPU hardware through a supported integration path. The plugin decouples Ascend NPU integration from vLLM and enables supported model types to run on the platform. Key features Hardware-pluggable vLLM integration — The plugin provides an interface for integrating Ascend NPUs with vLLM and follows the community's hard…

github.comhttps://github.com/vllm-project/vllm-ascend

vllm: A fast and easy-to-use library for LLM inference and serving.

What problem it solves Serving large language models requires efficient memory management, flexible model execution, and practical integration with existing models and APIs. vLLM addresses these needs with inference and serving capabilities spanning batching, caching, quantization, distributed execution, and multiple hardware platforms. Key features Efficient memory and serving — vLLM manages atte…

github.comhttps://github.com/vllm-project/vllm

semantic-router: An open, programmable decision layer for models and compute.

What problem it solves Agent harnesses need to coordinate models, tools, task state, and changing compute requirements without manually managing every backend. They also need a way to apply location constraints and priorities for quality, latency, and cost to each call. Key features Policy-driven model routing — Select or combine models for each call according to a configured policy while leaving …

github.comhttps://github.com/vllm-project/semantic-router

openworker: A desktop app and harness SDK for building and running AI coworkers that work across everyday tools.

What problem it solves People need a way to delegate research, writing, code review, recurring tasks and work across connected tools to AI agents without giving up control over actions and data. They also need a foundation for embedding agent loops, tool calls, approvals, streaming and durable sessions into their own applications. Key features Everyday AI coworker — The desktop app supports resear…

github.comhttps://github.com/andrewyng/openworker

Hy-MT2: A family of fast-thinking multilingual translation models for complex real-world translation scenarios.

What problem it solves Real-world translation involves more than converting text between languages: tasks can require terminology, style, personalization, delimiter preservation, and structured-data constraints. Hy-MT2 is designed to support translation among 33 languages and follow translation instructions in multiple languages for complex real-world scenarios. Key features Instruction-driven tas…

github.comhttps://github.com/Tencent-Hunyuan/Hy-MT2

G0DM0D3: A fully open-source, privacy-transparent, multi-model chat interface for red teaming, cognition research, and liberated AI interaction.

What problem it solves Researchers, red-teamers, and system tinkerers need a chat interface that can work with multiple hosted and local models while exposing evaluation, perturbation, and sampling controls. They also need clear visibility into data flows, storage, telemetry, and deployment boundaries so they can make informed choices about privacy and self-hosting. Key features Multi-model races …

github.comhttps://github.com/elder-plinius/G0DM0D3

memorilla: A memory system that gathers information from an entire document collection into query-dependent memory tokens for a frozen LLM.

What problem it solves A frozen LLM can only use information that fits in its context window, making it difficult to answer questions from large document collections or long interaction histories. Memorilla addresses this by reading a collection and supplying the LLM with a fixed number of query-dependent memory tokens. Key features Query-dependent memory compression — The system reads a complete …

github.comhttps://github.com/snap-stanford/memorilla