A vLLM blog post argues that inference-side optimizations in high-performance RL — low-precision compute, low-precision KV cache, kernel and communication tuning — shift the numerics, so the training engine assigns different probabilities to the same token than the engine that sampled it; on DeepSeek-V4-Flash the high-performance baseline's train-inference KL rose from 2e-3 to 29.9 as reward fell back. The proposed fix has the training forward call vLLM's inference kernels directly, modified so rollout and training produce bitwise-identical logprobs for every sampled token (diff = 0). The authors report diff of 0 at every one of 100 DeepSeek-V4-Flash steps at 91.3% of the baseline's rollout throughput, and say that on Nemotron-3.5 (a Mamba hybrid with NVFP4) the consistent run's SWE-bench Pro score held near 0.6 while the baseline's fell from 0.56 to 0.22. The workflow is packaged as an agent skill and released with recipes and code, and the team says DeepSeek-V4.1 has reached 0 diff with performance work still in progress.
Posts on the vLLM Blog and the AWS Machine Learning Blog describe a push-based NIXL connector for prefill-decode (P-D) disaggregated inference in vLLM. In the default pull mode, the decode instance allocates destination blocks and issues an NIXL READ only after prefill finishes, so coordination and setup sit on the time-to-first-token critical path; in push mode the decode side registers blocks in advance and the prefill worker hands the transfer to a background thread that issues an NIXL WRITE. The authors say the feature was built and validated on AWS Trainium clusters with EFA and then contributed to vLLM as an accelerator-agnostic feature.
Liquid AI released LFM2.5-24B-A2B on Oct. 13, an open-weight reasoning model with 24B total parameters and roughly 2B active per token (Liquid AI elsewhere cites 2.45B), aimed at running agents locally on laptops and PCs. It is the largest and final member of the LFM2.5 family, with the context window extended from 32K to 131K tokens, a vocabulary doubled from 65K to 128K for better non-Latin tokenization, pretraining scaled from about 17T to 33T tokens, and a switch to reasoning-only output with an explicit chain of thought. Liquid AI says it matches the larger Qwen3.6-35B-A3B on five of eight agentic and tool-use benchmarks (for example 48.09 versus 38.19 on LiveClaw), though those benchmarks and the speed figures (184 tokens/s on an Apple M5 Max) are vendor-run. Base and post-trained weights are on Hugging Face with day-one support in llama.cpp, MLX, vLLM, SGLang and ONNX.
An AWS Machine Learning Blog post describes several kernel optimizations AWS implemented for fused Mixture-of-Experts models running on GPUs. It says a swap-AB matrix transposition aligned with NVIDIA Hopper's WGMMA instruction layout produced a 2–4 percent end-to-end throughput improvement for large MoE models on H200, and was upstreamed to vLLM 0.24.0 or later. A second change added do_not_specialize annotations to eliminate repeated Triton JIT recompilation, which AWS says improved time to first token by 9 percent at 8-way tensor parallelism and is available in vLLM 0.22.0 or later. AWS says the optimizations are open sourced, applied in the LMI deep learning container on Amazon SageMaker AI, and together delivered up to 10 percent higher throughput and up to 9 percent faster TTFT; these figures are vendor-reported and not independently verified.
An AWS Machine Learning Blog post walks through deploying one large language model across two GPU nodes on Amazon EKS using the multi-node Ray LLM Deep Learning Container, with separate prefill and decode nodes behind a single OpenAI-compatible endpoint. The container ships Ray Serve LLM, NVIDIA's NIXL library and the Elastic Fabric Adapter stack, and the example uses Qwen3.5-9B to show prefill/decode disaggregation configured entirely through YAML with no application code. The post also warns that NIXL silently falls back to a TCP-speed transport when it cannot use libfabric, so it recommends checking worker logs to confirm the backend.
vLLM and the AWS Machine Learning Blog describe bi-directional KV cache transfer in vLLM: in prefill-decode disaggregated serving, a decode node can return previously computed KV blocks to the prefill node on later turns, avoiding recomputation of the conversation history. On a 1P1D deployment using Amazon EC2 P5en instances with AWS EFA, the authors report TTFT reductions of up to 1.6x for Qwen3-32B and up to 3.3x for zai-org/GLM-5-FP8, noting that results vary with workload and configuration. Using it requires vLLM v0.21.0 or later with NixlConnector, enabling bidirectional_kv_xfer on both prefill and decode servers, and a KV-cache-aware router, with decode-side block retention governed by decoder_kv_blocks_ttl (default 480 seconds).
SGLang is an open-source inference framework hosted by LMSYS for large language, vision-language and diffusion models, optimized for agentic workloads, large-scale serving and RL rollouts; the repository also includes SGLang Diffusion, its built-in image and video generation engine. It supports a wide range of GPUs, TPUs, NPUs, CPUs and Apple Silicon, including NVIDIA B200/B300 and H100, AMD Instinct, Google TPU, Huawei Ascend, and ships via Docker images or a uv install. The project offers a Cookbook, docs and learning resources such as Mini-SGLang, and long-term contributors can apply for coding-agent sponsorship covering Cursor, Claude Code or OpenAI Codex.
Hugging Face's Transformers library appeared on GitHub Trending's daily list. The project positions itself as the model-definition framework for state-of-the-art text, vision, audio, video and multimodal models, for both inference and training, and its documentation says there are over 1M Transformers model checkpoints on the Hugging Face Hub. It says the library interoperates with training frameworks such as Axolotl, Unsloth, DeepSpeed, FSDP and PyTorch-Lightning, and with inference engines and modeling libraries including vLLM, SGLang, TGI, llama.cpp and mlx, and requires Python 3.10+ and PyTorch 2.6+.
Tencent Youtu Lab published Youtu-Parsing-Omni on Hugging Face, a compact 5B omni-modal parsing model that turns a document page, natural image, chart or flowchart, geometry figure, audio clip or audio-visual video into a single structured JSON output, with the output family selected by a task prompt. The model card says a single omni encoder handles image, audio and interleaved audio-visual video inputs, and that a vLLM plugin, pinned serving settings, task prompts and inference examples are included. Tencent reports 96.96 Overall on OmniDocBench v1.6, 75.08 Avg. on OmniParsingBench as the best open-weight model behind Gemini-3-Pro, and competitive results against specialized models on chemical-structure and music-score recognition; all of these benchmarks are vendor-reported and were not independently verified in the supplied material.
The vLLM project published version 0.5.0 of vllm-proto on GitHub. The source provides only the release tag and version number, with no release notes or feature details.
vLLM says it now runs on NVIDIA's Vera Rubin NVL72, with nightly CUDA 13.4 and PyTorch 2.15 container images and support for models from DeepSeek, Moonshot AI's Kimi, Z.ai's GLM and MiniMax. Working with NVIDIA, Red Hat and Inferact, the project reports early results of 7.8x throughput per GPU versus GB200 NVL72 on AgentX at matched interactivity and up to 3.7x higher VLM throughput versus GB300 NVL72 in MLPerf; these are its own early figures. MiniMax separately said its M3 model is now running on Rubin with vLLM, crediting the same collaborators.
JetBrains launched Mellum2.1, keeping Mellum2's 12B mixture-of-experts architecture with 2.5B active parameters and the Apache 2.0 license, and focusing the upgrade on agentic coding. JetBrains says reinforcement learning was expanded from a short post-training step into the main training stage, letting the model explore codebases, edit files, identify root causes of failing tests and draft and verify fixes. The model is available on Hugging Face, with GGUF builds for llama.cpp, Ollama and LM Studio and a vLLM MTP speculative-decoding component promised later.
A PyTorch blog post details how NVIDIA Dynamo turns request-level serving into program-aware serving using one session-level identifier, with child sessions carrying a parent_session_id. Dynamo recognizes the session headers that Claude Code, Codex and OpenCode already emit, while other harnesses opt in via X-Dynamo-Session-ID, and plugins exist for Hermes and OpenClaw. The post also covers session-aware routing and admission control (porting a scheduler from the ThunderAgent paper), a shared KV cache indexer, the Session-Prefix Indexer and proposed KvHint interface, plus DYN_REQUEST_TRACE=1 trace capture with AISimulate offline replay and AIPerf live benchmarking.
vLLM is a high-throughput, memory-efficient inference and serving engine for large language models, originally developed in the Sky Computing Lab at UC Berkeley and now maintained by a community of more than 2,000 contributors. It manages attention key-value memory with PagedAttention and supports continuous batching, chunked prefill, prefix caching, speculative decoding and quantization formats including FP8, INT8/INT4, GPTQ/AWQ and GGUF, alongside an OpenAI-compatible API server. The engine runs on NVIDIA, AMD and Intel GPUs plus x86/ARM/PowerPC CPUs, and supports 200+ model architectures on Hugging Face.
vLLM v0.31.0 adds broad serving and model-support updates, including DeepSeek-V4.1-Flash optimizations, Model Runner V2 speculative decoding, larger-scale expert parallelism, and improved multimodal support. It also introduces the `vllm preload` weight-cache daemon for faster engine restarts and experimental CRIU-based initialized-engine snapshots, alongside several security and breaking changes.
What problem it solves
Training generative AI models at scale requires coordinating multiple parallelism strategies while supporting pretraining, fine-tuning, and reinforcement learning in one codebase. Large-scale training also needs reproducible configurations, checkpointing, debugging tools, and observability that can be adapted for research and model development.
Key features
Composable model …
What problem it solves
vLLM users need a way to run the framework on Huawei Ascend NPU hardware through a supported integration path. The plugin decouples Ascend NPU integration from vLLM and enables supported model types to run on the platform.
Key features
Hardware-pluggable vLLM integration — The plugin provides an interface for integrating Ascend NPUs with vLLM and follows the community's hard…
What problem it solves
Serving large language models requires efficient memory management, flexible model execution, and practical integration with existing models and APIs. vLLM addresses these needs with inference and serving capabilities spanning batching, caching, quantization, distributed execution, and multiple hardware platforms.
Key features
Efficient memory and serving — vLLM manages atte…
What problem it solves
Agent harnesses need to coordinate models, tools, task state, and changing compute requirements without manually managing every backend. They also need a way to apply location constraints and priorities for quality, latency, and cost to each call.
Key features
Policy-driven model routing — Select or combine models for each call according to a configured policy while leaving …
What problem it solves
People need a way to delegate research, writing, code review, recurring tasks and work across connected tools to AI agents without giving up control over actions and data. They also need a foundation for embedding agent loops, tool calls, approvals, streaming and durable sessions into their own applications.
Key features
Everyday AI coworker — The desktop app supports resear…
What problem it solves
Real-world translation involves more than converting text between languages: tasks can require terminology, style, personalization, delimiter preservation, and structured-data constraints. Hy-MT2 is designed to support translation among 33 languages and follow translation instructions in multiple languages for complex real-world scenarios.
Key features
Instruction-driven tas…
What problem it solves
Researchers, red-teamers, and system tinkerers need a chat interface that can work with multiple hosted and local models while exposing evaluation, perturbation, and sampling controls. They also need clear visibility into data flows, storage, telemetry, and deployment boundaries so they can make informed choices about privacy and self-hosting.
Key features
Multi-model races …
What problem it solves
A frozen LLM can only use information that fits in its context window, making it difficult to answer questions from large document collections or long interaction histories. Memorilla addresses this by reading a collection and supplying the LLM with a fixed number of query-dependent memory tokens.
Key features
Query-dependent memory compression — The system reads a complete …