Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

vLLM

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 196

  1. Phison showcases AI SSD with embedded NPU and 64GB memory

    At SC26 in Chicago, storage-controller vendor Phison demonstrated an SSD design for AI servers that integrates a neural processing unit (NPU) and 64GB of memory. The design is intended to run AI frameworks and models such as PyTorch, llama.cpp, vLLM and OpenVINO inside the drive, allowing some processing to occur before data leaves the storage device. Phison says that, for suitable workloads, the design could reduce data transfers 10 to 100 times and lower CPU load; its 64GB memory provides about 130GB/s of bandwidth. Available performance information is limited: Phison cited 36,800 DMIPS, a CPU integer benchmark that analysts believe may describe the SSD’s CPU cores rather than NPU performance, and did not disclose a TOPS metric. The company has not announced a formal product name, launch date or sampling plan, and many demonstrated features remain in development.

    cnBeta · 🔥 26
  2. AWS releases distributed KV caching agent skill for self-hosted LLMs

    AWS released a distributed KV caching skill for the Agent Toolkit for AWS that helps coding agents evaluate, configure, and troubleshoot caching for self-hosted LLM inference. The skill works with Kiro, Claude Code, OpenAI Codex, and other agents supporting Agent Skills, and can use KVTrace to estimate workload-specific savings before deployment. AWS says Amazon ElastiCache can reduce inference costs by up to 40% and time to first token by up to 80%, but these are workload-dependent vendor claims rather than universal guarantees.

    AWS Machine Learning Blog · 🔥 5
  3. Phison details AI SSD with built-in NPU and mixed-media design

    Phison presented two enterprise SSD concepts at Supercomputer 2026, Tom’s Hardware reported. One combines a built-in NPU with 64GB of memory offering 130 GB/s bandwidth and supports running PyTorch, vLLM, OpenVINO, and llama.cpp directly on the drive; the other combines pSLC cache with QLC capacity behind one controller. Phison says the first design can process resident data and return results with 10 to 100 times less transferred data for applicable workloads, but the host-access mechanism, memory type, and product timing remain undisclosed, while many features are still in development.

    Tom’s Hardware · 🔥 4
  4. TorchTitan unifies PyTorch-native generative AI training workflows

    TorchTitan is a PyTorch-native platform for pretraining and post-training generative AI models, spanning supervised fine-tuning and reinforcement learning. Its documentation says FSDP2, tensor, pipeline, context, and expert parallelism can be composed through DeviceMesh and DTensor, while TitanRL shares model definitions and kernels between the trainer and vLLM generation. The supplied report is a project-maintained repository snapshot and does not independently validate the stated performance.

    GitHub Trending · Python · 🔥 3
  5. Tracing Agent Latency Before Changing Models

    A 掘金 article argues that slow Agent responses often originate in tools, retrieval, sandboxing, or orchestration rather than the model itself. Its 30-second example attributes three seconds to model reasoning and 25 seconds to a search tool waiting on repeated website retries, making tracing the central diagnostic step. The article then surveys tool interfaces, process-aware evaluation, retrieval metrics, browser automation, sandboxing, prompt-injection defenses, RAG versus fine-tuning, graph-based workflows, and criteria for avoiding unnecessary Agent architectures.

    掘金 · 人工智能 · 🔥 1
  6. Tencent publishes NVFP4 WeMM-Video-35B-A3B checkpoint

    Tencent published the WeMM-Video-35BA3B-NVFP4 checkpoint, an NVFP4 W4A16 Safetensors variant of WeMM-Video-35B-A3B with 24.11 GiB of weight files. The model card says it supports Chinese and English audio-visual understanding, accepting text, images, audio, and video, and includes instructions for serving it with patched vLLM 0.31.0, ModelOpt Mixed Precision, and Marlin.

    腾讯混元 · Hugging Face · 🔥 3
  7. Tencent releases FP8 checkpoint for WeMM-Video-35B-A3B

    Tencent’s Hugging Face model repository lists a new FP8 Block128 Safetensors variant of WeMM-Video-35B-A3B under the ID tencent/WeMM-Video-35BA3B-FP8. The checkpoint is based on the quantized 35B-A3B model, uses Apache-2.0 licensing, and contains 36.09 GiB of weight files. Tencent says the bilingual model accepts text, images, audio, and video, with loading and generation checks reported on an NVIDIA L20 and H20.

    腾讯混元 · Hugging Face · 🔥 3
  8. Tencent releases AWQ-quantized WeMM-Video-35B-A3B

    Tencent published an AWQ W4A16 Safetensors checkpoint for WeMM-Video-35B-A3B, with 22.69 GiB of weight files and INT4 feed-forward weights. The bilingual audio-visual model accepts text, images, audio, and video; its documentation includes Transformers loading instructions, patched vLLM 0.18.0 serving guidance, and validation on NVIDIA L20 and H20 hardware.

    腾讯混元 · Hugging Face · 🔥 3
  9. Tencent releases WeMM-Video-9B multimodal checkpoint

    Tencent's model card describes WeMM-Video-9B as the dense 9B checkpoint in the bilingual WeMM-Video family for joint audio-visual understanding and reasoning. It accepts text, images, audio, and video and generates text responses. The release includes BF16 Safetensors, quantized variants, Transformers support, and serving instructions for patched vLLM and SGLang versions.

    腾讯混元 · Hugging Face · 🔥 3
  10. Tencent releases FP8 Block128 checkpoint for WeMM-Video-9B

    Tencent released the FP8 Block128 Safetensors variant of WeMM-Video-9B under the Apache 2.0 license, with 13.10 GiB of weight files available on Hugging Face. The Chinese-English bilingual model accepts text, images, audio, and video and generates text responses; its model card includes Transformers loading instructions and a patched vLLM 0.18.0 serving setup. Tencent reports successful loading and generation checks on NVIDIA L20 and H20 GPUs, but these validation details are self-reported.

    腾讯混元 · Hugging Face · 🔥 3
  11. Tencent releases NVFP4-quantized WeMM-Video-9B checkpoint

    Tencent published an NVFP4 W4A16 Safetensors variant of WeMM-Video-9B, with 13.07 GiB of weight files. The checkpoint accepts text, images, audio and video and generates Chinese or English responses, according to Tencent's Hugging Face repository; deployment uses patched vLLM 0.31.0 with ModelOpt Mixed Precision and Marlin.

    腾讯混元 · Hugging Face · 🔥 3
  12. Tencent releases WeMM-Video-9B AWQ W4A16 checkpoint

    Tencent’s Hugging Face repository released a 12.86 GiB AWQ W4A16 Safetensors checkpoint for WeMM-Video-9B. Tencent describes it as a Chinese-English bilingual model that jointly processes text, images, audio, and video, with support for Transformers and patched vLLM 0.18.0. The repository reports successful loading and generation checks on NVIDIA L20 and H20 hardware, but these validation details come from the publisher.

    腾讯混元 · Hugging Face · 🔥 3
  13. Tencent releases WeMM-Video-4B NVFP4 checkpoint

    Tencent’s model repository publishes WeMM-Video-4B-NVFP4, an Apache 2.0-licensed NVFP4 W4A16 Safetensors derivative of WeMM-Video-4B with 7.42 GiB of weights. The bilingual video-text-to-text model accepts text, images, audio, and video, and the repository provides patched vLLM 0.31.0 serving instructions using ModelOpt Mixed Precision and Marlin, plus limited generation checks.

    腾讯混元 · Hugging Face · 🔥 3
  14. WeMM-Video-4B model card details multimodal checkpoint

    The Hugging Face model card for tencent/WeMM-Video-4B describes the checkpoint as the dense 4B version in the bilingual WeMM-Video family. It says the model accepts text, images, audio, and video and generates Chinese or English responses. The listing provides BF16 Safetensors weights, four quantized variants, Apache License 2.0 terms, and examples for Transformers, vLLM, and SGLang deployment.

    腾讯混元 · Hugging Face · 🔥 3
  15. Tencent releases WeMM-Video-35B-A3B multimodal checkpoint

    Tencent’s WeMM-Video model card says WeMM-Video-35B-A3B is a bilingual Chinese-English mixture-of-experts checkpoint with 35B total parameters and 3B active parameters. It accepts text, images, audio, and video, and the Hugging Face repository publishes BF16 weights plus FP8, AWQ, GGUF, and NVFP4 variants under an Apache 2.0 license. Tencent claims the model leads open-source omni-modal video understanding, but that benchmark claim is vendor-reported.

    腾讯混元 · Hugging Face · 🔥 3
  16. Tencent Releases FP8 WeMM-Video-4B Checkpoint

    Tencent published an FP8 Block128 Safetensors variant of WeMM-Video-4B under the Apache 2.0 license. The model card documents loading and generation checks on one NVIDIA L20 with Transformers 5.2.0 and PyTorch 2.10.0+cu129, alongside serving instructions for patched vLLM 0.18.0 on FP8-capable GPUs. Tencent describes WeMM-Video as a Chinese-English bilingual family for joint audio-visual understanding and reasoning that accepts text, images, audio, and video.

    腾讯混元 · Hugging Face · 🔥 3
  17. Tencent releases WeMM-Video-4B AWQ W4A16 checkpoint

    Tencent published a 7.32 GiB AWQ W4A16 Safetensors checkpoint for WeMM-Video-4B. The language model’s feed-forward weights use group-128 INT4, while activations and attention weights remain BF16. The repository provides direct-loading instructions for Transformers 5.2.0 and a serving guide for patched vLLM 0.18.0 with the checkpoint’s compressed-tensors configuration; Transformers loading requires compressed-tensors 0.13.0. Tencent reports that the local checkpoint loaded and completed generation checks for text, images, audio, video frames and video with its soundtrack on one NVIDIA L20 using Transformers 5.2.0 and PyTorch 2.10.0+cu129.

    腾讯混元 · Hugging Face · 🔥 3
  18. Tencent releases WeMM-Video-2B-FP8 checkpoint

    Tencent released the WeMM-Video-2B-FP8 repository, an FP8 Block128 Safetensors variant of WeMM-Video-2B with 4.84 GiB of weights. The model supports text, image, audio, and video inputs and generates Chinese or English text responses; its listed capabilities come from Tencent’s model description. The repository documents successful loading and generation with Transformers 5.2.0 and patched vLLM 0.18.0 on an NVIDIA L20.

    腾讯混元 · Hugging Face · 🔥 3
  19. Tencent releases WeMM-Video-2B AWQ W4A16 checkpoint

    Tencent published an Apache 2.0-licensed repository for WeMM-Video-2B-AWQ, an AWQ W4A16 Safetensors checkpoint with 4.86 GiB of weight files. The model card says the 3B-parameter model accepts text, images, audio, and video and generates text responses in Chinese or English, while the quantization uses group-128 INT4 for feed-forward weights and BF16 for activations and attention weights. Tencent reports successful loading and generation checks on one NVIDIA L20 using Transformers 5.2.0 and a patched vLLM 0.18.0.

    腾讯混元 · Hugging Face · 🔥 3
  20. Tencent releases NVFP4 WeMM-Video-2B checkpoint

    Tencent published a Hugging Face repository for the NVFP4 W4A16 Safetensors variant of WeMM-Video-2B under the Apache 2.0 license. The checkpoint contains 4.90 GiB of weights and accepts text, images, audio, and video for Chinese or English text responses. Tencent documents deployment with patched vLLM 0.31.0 using ModelOpt Mixed Precision and Marlin.

    腾讯混元 · Hugging Face · 🔥 3
  21. Tencent releases Hy4-base, a 771B multimodal MoE model

    Tencent’s Hugging Face listing introduces Hy4-base as a 771B-parameter image-text-to-text model built on a 770B-parameter Mixture-of-Experts language backbone, with 49B parameters activated per token and a listed 1M-token context window. The model card says it supports text, image and video understanding, and provides Transformers support, GGUF weights, a finetuning pipeline, and deployment recipes for vLLM and SGLang. Tencent also reports benchmark and application improvements, but the supplied listing is the only source and its performance claims have not been independently verified.

    腾讯混元 · Hugging Face · 🔥 3
  22. Tencent releases Hy4 multimodal MoE model with 1M-token context

    The Tencent Hy Team introduced Hy4, a new-generation Mixture-of-Experts flagship model for text, image, and video understanding, according to its official model card. Its 770B-parameter language backbone activates 49B parameters per token and supports a 1M-token context, while a 3.3B DFly2 drafter is included for speculative decoding. Tencent also provides FP8 and GGUF artifacts, vLLM and SGLang deployment recipes, a finetuning pipeline, and documentation under the Tencent Hy License.

    腾讯混元 · Hugging Face · 🔥 3
  23. Tencent releases Hy4 multimodal MoE model

    Tencent Hy Team released Hy4 on Hugging Face as a 774B-class image-text-to-text model under the Tencent Hy License. The Mixture-of-Experts model supports text, image, and video understanding, activates 49B language parameters plus 1.23B vision parameters per token, and provides a 1,048,576-token context window. The release also includes deployment recipes for vLLM and SGLang, a 3.3B DFly2 drafter for speculative decoding, quantization resources, and finetuning documentation.

    腾讯混元 · Hugging Face · 🔥 3
  24. Tencent releases WeMM-Video-2B dense multimodal checkpoint

    Tencent released WeMM-Video-2B, a dense 2B checkpoint in its bilingual WeMM-Video family, under the Apache-2.0 license. The Hugging Face model card says it accepts text, images, audio, and video, generates text responses, and offers BF16 weights plus FP8, AWQ, GGUF, and NVFP4 variants.

    腾讯混元 · Hugging Face · 🔥 3
  25. AWS adds managed Dynamo and global routing to SageMaker Inference

    AWS Machine Learning Blog says Amazon SageMaker Inference added managed NVIDIA Dynamo, expanded HyperPod routing and multi-Region networking, and introduced Inference Gateway and Global Inference Gateway services. The latter provides one endpoint across clusters and Regions with failover in under two minutes, while managed Dynamo supports vLLM, SGLang, and TensorRT-LLM where HyperPod is available. AWS also said inference recommendations are available through SageMaker AI Studio, the SageMaker Python SDK, and coding agents.

    AWS Machine Learning Blog · 🔥 0
  26. Inkling 2 Mini launches with day-zero vLLM support

    Thinking Machines announced Inkling 2 Mini, a multimodal mixture-of-experts model with 26B total parameters, 3.5B active parameters and support for contexts up to 1M tokens. Its middle 12 layers run three times per token with shared weights, separate KV caches and gated messages between passes. vLLM Blog says vLLM provides day-zero support across NVIDIA Blackwell, Hopper and DGX Spark, plus serving support for the TML-d24-varloop-research checkpoint.

    vLLM Blog · 🔥 0
  27. Thinking Machines Lab releases Inkling 2 Mini open-weight model

    Thinking Machines Lab released Inkling 2 Mini, a 26.4B-parameter open-weight multimodal MoE model designed for local agentic workloads. Its 24 decoder layers include 12 middle layers that run three times per token, while the checkpoint has 3.4B active parameters and a reported 1M-token context window. Artificial Analysis independently placed it at 20 on its Intelligence Index, while Thinking Machines Lab reported leading results among open-weight models below 100B total parameters.

    TestingCatalog · 🔥 55
  28. SageMaker AI adds automatic inference endpoint optimization monitoring

    Amazon SageMaker AI now monitors generative AI inference endpoints using vLLM containers on GPU instances, detecting week-over-week latency drift, unused GPU capacity, and serving-configuration gaps. Findings are delivered through AWS Health and can be evaluated in Amazon SageMaker Studio using benchmarks that compare response time, throughput, and cost per million tokens without automatically changing production endpoints.

    AWS Machine Learning Blog · 🔥 0
  29. Stanford CS336 Lecture 10 on LLM Inference Optimization

    A Stanford CS336 Lecture 10 report explains why autoregressive LLM inference can be slow and how to evaluate performance using TTFT, TPOT, latency, throughput, and memory usage. It covers KV Cache sizing, GQA and MLA, quantization, speculative decoding, continuous batching, and PagedAttention. The report distinguishes Prefill’s generally compute-intensive characteristics from Decode’s bandwidth-sensitive bottlenecks and emphasizes that acceleration methods should be evaluated for model quality, stability, and cost. Its Llama 2 13B and H100 figures are idealized estimates derived from course formulas, not real-world service benchmarks.

    掘金 · 人工智能 · 🔥 0
  30. vLLM Ascend v0.30.0 released

    The vLLM Ascend project announced v0.30.0 as the latest stable version of its community-maintained hardware plugin for running vLLM on Huawei Ascend NPUs. The project directs users to its official guide and lists prerequisites including Linux, Python 3.12, CANN 9.2.1, PyTorch 2.10.0, TorchNPU 2.10.0.post7, and a matching vLLM version.

    GitHub Trending · Python · 🔥 0

Next →

Experience and discussion from the community

Share my vLLM experienceAsk about vLLM

torchtitan: A PyTorch-native platform for pretraining and post-training generative AI models at scale.

What problem it solves Training generative AI models at scale requires coordinating multiple parallelism strategies while supporting pretraining, fine-tuning, and reinforcement learning in one codebase. Large-scale training also needs reproducible configurations, checkpointing, debugging tools, and observability that can be adapted for research and model development. Key features Composable model …

github.comhttps://github.com/pytorch/torchtitan

vllm-ascend: A community-maintained hardware plugin for running vLLM on Huawei Ascend NPUs.

What problem it solves vLLM users need a way to run the framework on Huawei Ascend NPU hardware through a supported integration path. The plugin decouples Ascend NPU integration from vLLM and enables supported model types to run on the platform. Key features Hardware-pluggable vLLM integration — The plugin provides an interface for integrating Ascend NPUs with vLLM and follows the community's hard…

github.comhttps://github.com/vllm-project/vllm-ascend

vllm: A fast and easy-to-use library for LLM inference and serving.

What problem it solves Serving large language models requires efficient memory management, flexible model execution, and practical integration with existing models and APIs. vLLM addresses these needs with inference and serving capabilities spanning batching, caching, quantization, distributed execution, and multiple hardware platforms. Key features Efficient memory and serving — vLLM manages atte…

github.comhttps://github.com/vllm-project/vllm

semantic-router: An open, programmable decision layer for models and compute.

What problem it solves Agent harnesses need to coordinate models, tools, task state, and changing compute requirements without manually managing every backend. They also need a way to apply location constraints and priorities for quality, latency, and cost to each call. Key features Policy-driven model routing — Select or combine models for each call according to a configured policy while leaving …

github.comhttps://github.com/vllm-project/semantic-router

openworker: A desktop app and harness SDK for building and running AI coworkers that work across everyday tools.

What problem it solves People need a way to delegate research, writing, code review, recurring tasks and work across connected tools to AI agents without giving up control over actions and data. They also need a foundation for embedding agent loops, tool calls, approvals, streaming and durable sessions into their own applications. Key features Everyday AI coworker — The desktop app supports resear…

github.comhttps://github.com/andrewyng/openworker

Hy-MT2: A family of fast-thinking multilingual translation models for complex real-world translation scenarios.

What problem it solves Real-world translation involves more than converting text between languages: tasks can require terminology, style, personalization, delimiter preservation, and structured-data constraints. Hy-MT2 is designed to support translation among 33 languages and follow translation instructions in multiple languages for complex real-world scenarios. Key features Instruction-driven tas…

github.comhttps://github.com/Tencent-Hunyuan/Hy-MT2

G0DM0D3: A fully open-source, privacy-transparent, multi-model chat interface for red teaming, cognition research, and liberated AI interaction.

What problem it solves Researchers, red-teamers, and system tinkerers need a chat interface that can work with multiple hosted and local models while exposing evaluation, perturbation, and sampling controls. They also need clear visibility into data flows, storage, telemetry, and deployment boundaries so they can make informed choices about privacy and self-hosting. Key features Multi-model races …

github.comhttps://github.com/elder-plinius/G0DM0D3

memorilla: A memory system that gathers information from an entire document collection into query-dependent memory tokens for a frozen LLM.

What problem it solves A frozen LLM can only use information that fits in its context window, making it difficult to answer questions from large document collections or long interaction histories. Memorilla addresses this by reading a collection and supplying the LLM with a fixed number of query-dependent memory tokens. Key features Query-dependent memory compression — The system reads a complete …

github.comhttps://github.com/snap-stanford/memorilla