Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

vLLM

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 195

  1. Nvidia RTX Spark laptops run on Linux, but AC power cuts GPT-OSS-120B inference to 13.9 tok/s

    Hands-on testing found native Linux installs and runs on Nvidia RTX Spark laptops such as the MSI Prestige N16 Flip AI Plus, with GPU acceleration via Nvidia's open kernel driver, a full desktop, 4K 120Hz output and working audio and input devices after configuration. The catch is power management: when plugged into wall power the GPU is locked to roughly 780 MHz on Linux versus about 2.2 GHz on Windows, and on GPT-OSS-120B Windows reached 65.4 tokens per second, WSL 65.1, an Nvidia DGX Spark 61, while native Linux managed only 13.9. The report attributes the gap to an encrypted firmware power table and a Windows-only component called Nvidia Platform Controllers and Framework, which has no Linux equivalent, and says a fix requires Nvidia to port that power-management logic into its Linux driver. On battery, Linux clocks and throughput approach Windows levels, and WSL is presented as the practical middle ground since it keeps full GPU boost while retaining tools such as vLLM.

    Hacker News 首页(100+ 分) · 🔥 0
  2. Local LLM inference tools multiply as users weigh speed against quality

    A Reddit user says the local LLM inference ecosystem has expanded rapidly since March 2025, from established tools such as llama.cpp, Ollama, vLLM and TabbyAPI to numerous newer projects. The author reports that llama.cpp continues to improve quickly, while newer engines may prioritize speed or lower resource use at the expense of output quality; their specific model and hardware claims are not independently verified here.

    r/LocalLLaMA · 🔥 0
  3. vLLM adds day-0 support for Kimi K3.1 Pro

    vLLM announced day-0 support for Moonshot AI's Kimi K3.1 Pro across NVIDIA and AMD GPUs, Google TPUs, and Alibaba T-Head Zhenwu PPUs. The implementation adds sparse-attention support, a unified KV cache for MLA latent data, indexer keys and KDA state, and specialized decode and prefill paths. vLLM also reported long-context, throughput and reinforcement-learning benchmark gains, but these measurements have not been independently corroborated in the supplied report.

    vLLM Blog · 🔥 0
  4. Buoyant launches open-source Kinference for vLLM

    Buoyant announced Kinference, an Apache 2.0-licensed open-source control plane for building inference platforms with vLLM and Kubernetes. It offers token rate limiting, tenant usage and cost tracking, inference-specific load balancing, and policy-based routing through an HTTP endpoint compatible with OpenAI and Anthropic clients. In Buoyant's public benchmarks, Kinference reported lower median time-to-first-token and end-to-end latency than standard Kubernetes load balancing on one workload, along with 7.4 times the goodput on a shared-prefix workload and about 3.5 times on a multiturn workload.

    vLLM Blog · 🔥 0
  5. Olla v0.1.0 adds TEI support and reliability improvements

    The Olla project released v0.1.0, its 25th release. The update adds Hugging Face Text Embeddings Inference support for embeddings, reranking and classification; configurable retry and failover behavior; a reworked dashboard; improved model-capability detection; and fixes for concurrency, request statistics, rate limiting, routing, backend discovery and configuration validation. Olla is a Go-based open-source proxy and load balancer that sits between applications and local inference servers. It supports multiple runtimes, including Ollama, vLLM, SGLang, llama.cpp and LM Studio, and provides OpenAI- and Anthropic-compatible interfaces with translation where required.

    r/LocalLLaMA · 🔥 0
  6. FreeToken targets local MoE inference on consumer hardware

    A technical report describes FreeToken as an edge-native MoE serving engine that combines GPU compute, CPU compute and host memory in a unified inference pool. The report says the project can run 290B-parameter open-weight models at interactive speeds on gaming PCs and supports features including semantic checkpoints, expert caching and dynamic VRAM allocation. It also reports support for Windows and Linux, Anthropic- and OpenAI-compatible APIs, several MoE models and NVIDIA RTX 30-, 40- and 50-series GPUs.

    掘金 · 人工智能 · 🔥 0
  7. LlamaIndex reports nearly 10x faster token generation for OCRia-1 on one B200

    LlamaIndex reports that its optimizations raised OCRia-1 token generation from about 285 to 2,800 tokens per second on a single NVIDIA B200 GPU, a vendor-reported result not independently verified here. The stack combines workload-aware mixed-precision quantization, DFlash2 speculative decoding, ReplaySSM, custom kernels and 17 additional optimizations. LlamaIndex says OCRia-1 is available through LlamaParse across all tiers and plans to contribute generally applicable improvements to vLLM.

    LlamaIndex Blog · 🔥 0
  8. Egzakta AI Lab releases KVARK Sparkle 1.0 open-weight 27B model

    Egzakta AI Lab released KVARK Sparkle 1.0, its first open-weight model, a 27B reasoning model based on Qwen3.8-27B. The developer team says it uses up to 67% fewer thinking tokens and responds roughly twice as fast as the original Qwen3.8-27B in its own testing, with a median response time of 13.7s versus 28.1s, while maintaining comparable accuracy across five standard benchmarks (GPQA Diamond, AIME, LiveCodeBench, IFEval and MMLU-Pro), with none of the accuracy differences statistically significant in that testing. The team also reports faster results on practical tasks such as dashboard generation (1.6x), landing page generation (1.4x) and an SVG illustration task (3.2x). Open weights are released under the Apache 2.0 license in BF16, FP8 and GGUF formats, supporting llama.cpp, vLLM, Transformers and SGLang, and the model is published on Hugging Face.

    r/LocalLLaMA · 🔥 0
  9. DeepTutor connects tutoring, research, quizzes, and shared memory

    A Juejin AI article describes DeepTutor as an open-source, agent-native tutoring workspace that links Chat, Quiz, Research, and Visualize through one runtime. It reports that the system shares session context and memory across workflows, supports several RAG engines, and uses a file-backed three-layer memory system for inspectable traces, summaries, and synthesis. The article also identifies Next.js and FastAPI as its frontend and backend technologies and says Codex and Claude Code can operate as subagents.

    掘金 · 人工智能 · 🔥 0
  10. A Practical Framework for Profiling and Tuning vLLM Inference

    This tutorial explains how to evaluate LLM inference services by balancing throughput, request latency, and memory utilization, with latency further divided into TTFT and TPOT. It recommends profiling bottlenecks before tuning vLLM parameters, and maps options such as batched-token limits, concurrency, prefix caching, chunked prefill, and speculative decoding to different workload goals. The author also warns that some listed vLLM commands, environment variables, and metric names are invalid, and that NVIDIA-specific tooling must be replaced when running on Ascend NPUs.

    掘金 · 人工智能 · 🔥 0
  11. JetBrains Adds MTP Head to Mellum2.1 Thinking

    A JetBrains model card reposted on Reddit presents Mellum2.1 Thinking with a multi-token prediction (MTP) head, an additional transformer block trained to predict the token following the sampled one. The model card says inference engines can use it for speculative drafting while verifying every token, claiming 1.4–1.8x faster decoding and about 90% draft acceptance; support is listed for vLLM 0.33+, Transformers 5.20+, and llama.cpp.

    r/LocalLLaMA · 🔥 0
  12. Sparse-KV extension runs eight 250K-context coding agents on two RTX 3090s

    A Reddit author reports a sparse-attention extension for HyperQwen that keeps full int4 KV caches in system RAM while retrieving roughly 1% of tokens through a rank-64 key sketch. On a 3090 Ti and 3090, the author measured eight concurrent Qwen3.8-27B-W4A16 coding agents with 250K-token contexts at about 545 tokens per second on explain-style output, 31/32 recall, and roughly five-second follow-up starts. The author also reports that approximate attention can occasionally confuse near-duplicate code lines.

    r/LocalLLaMA · 🔥 0
  13. A hands-on tour of popular AI open-source tools

    This Juejin tutorial organizes AI open-source tools around three common needs: local model chat, agent orchestration, and multimodal understanding. It demonstrates Ollama with Qwen3 4B, including installation, model download, local chat, and API access, before showing a vLLM server for Qwen2-VL-2B-Instruct. The article also recommends consulting each project’s official GitHub repository and README, and suggests lightweight models when VRAM is limited.

    掘金 · 人工智能 · 🔥 0
  14. Reddit user reports Strata underperformed llama.cpp on dual R9700

    A Reddit user testing the Strata inference engine with Qwen Next Flash on two Radeon R9700 cards said they reached only Q3 quantization and encountered hangs when experts were offloaded to RAM. The user reported about 60 tokens per second on paper, but HumanEval+ results of 93.9% for 3-bit Flash-Next versus 95.1% for Qwen3.8-27B Q8 under llama.cpp, with nearly twice as many tokens needed. They also questioned whether favorable reports about model-specific inference engines reflect hype or edge cases.

    r/LocalLLaMA · 🔥 0
  15. vLLM v0.33.0 tag published

    The supplied GitHub report shows a vLLM v0.33.0 tag associated with commit 4ba787e. It states that khluu tagged it on November 1 at 09:21. The captured page contains repeated loading errors and does not provide changelog details, so the release’s specific changes could not be verified.

    vLLM Releases · 🔥 0
  16. Reddit user seeks advice on Threadripper PRO 3995WX LLM server

    A Reddit user asked whether to buy a used workstation listed in Iraq with an AMD Ryzen Threadripper PRO 3995WX, 224 GB of DDR4-3200 RAM, and an ASUS Pro-series motherboard. The poster wants to evaluate it as a local LLM inference server with four to eight RTX 3090 GPUs, but the configuration, exact motherboard model, price, and upgrade feasibility remain unverified.

    r/LocalLLaMA · 🔥 0
  17. DeepSeek-V4.1-Flash benchmarked on dual DGX Spark hardware

    Nick Mermiges of PLI Labs benchmarked DeepSeek-V4.1-Flash-1017 at 2.75 bpw with EXL3 and vLLM across two NVIDIA DGX Spark systems. An EPYC system with two RTX 3090s handled Engram tables and speculative drafting. In the author's tests, reported as medians of three runs, prefill reached 2,618 tokens per second with a 256K prompt, while structured JSON output reached 322 tokens per second across four streams. The report published the code, launch scripts, benchmark tool and raw results, while noting lower prose throughput and slower decoding than some cited deployments. Planned 1M-token context, llama-benchy, BFCL and head-to-head KLD tests were not yet available, and the prefix-cache code was still pending publication.

    r/LocalLLaMA · 🔥 0
  18. QEngine 1.3.1 hits 504 tok/s in community Flash-Next test

    A r/LocalLLaMA user reports that QEngine 1.3.1 served Qwen3.8-Flash-Next at 406–504 tok/s in single-stream decoding on two RTX 5090 GPUs. The test also found QEngine roughly level with Strata on agentic prompt latency and faster at 10K-context prefill, while Strata led at 90K. The author cautions that repeated prompts and QEngine's adaptive expert cache put the decode result near best-case conditions, and the RAM-versus-VRAM comparison used only one boot and pass per mode.

    r/LocalLLaMA · 🔥 0
  19. NVIDIA AICR validates Kubernetes runtimes for Dynamo inference

    NVIDIA Technical Blog details how NVIDIA AI Cluster Runtime (AICR) packages NVIDIA Dynamo and its supporting Kubernetes infrastructure into versioned, validated recipes. The Dynamo recipe pins the dynamo-platform chart to version 1.4.2 and defines component versions, compatibility constraints, deployment checks, and conformance checks covering GPUs, scheduling, observability, and ingress. The walkthrough also compares Dynamo Frontend and Gateway API/EPP routing and explains the planned native Rust EPP path in Dynamo 1.5.

    NVIDIA Technical Blog · 🔥 0
  20. Community post reports PLE disk offload support in vLLM nightly

    A community post reports that vLLM nightly has added PLE disk offload support. The supplied material contains no official vLLM announcement, release notes, or technical details about the feature, its requirements, limitations, or availability.

    r/LocalLLaMA · 🔥 0
  21. vLLM v0.33.0rc2 fixes AITER GEMM issue on ROCm gfx950

    vLLM v0.33.0rc2 includes a ROCm bugfix that skips AITER tuned GEMM when M*N is at least 2^31 on gfx950. The release note says the change was cherry-picked from commit 6d01893.

    vLLM Releases · 🔥 0
  22. Transformers v5.21.1 Restores vLLM Remote Code Loading

    Transformers released patch version 5.21.1 to fix the rope_scaling setter for configurations that assign it during initialization. The release maintainers also said it restores remote-code loading in vLLM, which had been broken by an earlier Transformers release.

    Transformers Releases · 🔥 0
  23. vLLM adds day-0 support for Mistral Large 4

    vLLM announced day-0 support for Mistral Large 4, Mistral’s native multimodal mixture-of-experts model, through the dedicated vllm/vllm-openai:mistral-large-4 Docker image. The implementation supports FP8 and NVFP4 checkpoints, image inputs, built-in DSpark speculative decoding, long-context serving and AMD Instinct MI350 deployment with ROCm. It also includes generic improvements to context parallelism, NIXL-based disaggregated serving and HiSparse host-memory KV offload. Performance and quality figures reported by vLLM were not independently verified in the supplied material.

    vLLM Blog · 🔥 0
  24. Baseten Reports 7x Throughput Boost for Qwen3-ASR Batch Transcription

    Baseten reports that it optimized a Qwen3-ASR 1.7B deployment on a single NVIDIA RTX PRO 6000 using native audio decoding, FP8 KV-cache quantization, and fewer host-to-device synchronizations. At high concurrency, the optimized stack processed up to 7 times more audio per second than out-of-the-box vLLM. Baseten also reports sustaining more than 3,000 times real time on hour-long files, at an estimated cost of about $0.0013 per audio hour, with no meaningful change in word error rate. It describes Qwen3-ASR 1.7B as the top-ranking open-source model on Hugging Face's Open ASR Leaderboard, with a 5.4% average word error rate. These deployment performance and cost figures are vendor-reported.

    Baseten Blog · 🔥 0
  25. Google adds local-model support to Antigravity SDK

    InfoQ reported that Google added local-model support to the Antigravity SDK, allowing developers to run agent workflows on their machines while using cloud models for higher-level tasks. In Google's hybrid Architect-Builder pattern, the Architect plans in the cloud using filenames and task descriptions without accessing source code, while local Builders reproduce vulnerabilities, propose fixes, review patches and run regression tests. In a vendor demonstration involving three vulnerable Python modules, Google reported that 95 of roughly 3,400 tokens were used for cloud planning and 97.2% were consumed locally. A community participant questioned whether token share alone shows whether local models run faster or more cheaply than cloud models.

    InfoQ 中文 AI&大模型 · 🔥 0
  26. Local GPU server benchmarks 12 models from 4B to 35B parameters

    A Reddit developer says they built a Linux server with two RTX 5060 Ti cards providing 32 GB of total VRAM, running vLLM in Docker. The developer reports testing 12 models from 4B to 35B parameters on complete datasets including IFEval, IFBench, ARC AGI 1, SimpleQA, SimpleQA Verified and SciCode, both with thinking enabled and disabled. The system also records wall-clock duration and token throughput to produce a time-efficiency score, although the methodology and results are self-reported.

    r/LocalLLaMA · 🔥 0
  27. Oumi Investigates Jev’s Likely Architecture and Training

    Oumi researchers used behavioral probes, benchmark results and serving profiles to estimate which model TypeSafe’s Jev may be based on and how it may have been trained and served. Without insider knowledge, they say Jev is likely built on Qwen 3.5 or 3.6 in a 35B-A3B configuration, with a tokenizer merging Qwen and GPT-OSS. They also speculate that its RLCD method uses soft-label cross-entropy on synthetic data rather than PPO or GRPO. For serving, they propose tree batching and either a form of sparse attention or deployment on two GPUs with the prefill split between them.

    r/LocalLLaMA · 🔥 0
  28. Six Inference Engines Benchmarked on RTX 5090

    A TensorSharp author benchmarked six inference engines running Qwen3.8 27B on one RTX 5090 32 GB, with and without DFlash2 speculative decoding. NInfer led speculative and aggregate concurrent throughput, while llama.cpp recorded the highest plain single-request decode rate; the tests also covered 32K prompts, image input, and a four-turn tool loop. The report attributes results to complete configurations because checkpoints, drafters, and KV formats differed, and its author disclosed being TensorSharp’s creator.

    r/LocalLLaMA · 🔥 0
  29. Calibration-Free Top-k Selection Reduces MoE Expert Reads

    A paper by Geunhyoung Kim and Sung Ju Hwang proposes top-k channel selection for Mixture-of-Experts decoding, avoiding the offline calibration data used by threshold-based selectors. The authors report that top-k matched or exceeded threshold variants in 40 of 43 comparisons at 13% and 18% read budgets, including all 23 comparisons on generation tasks, and avoided repetition-loop collapse observed with calibrated thresholds on Gemma 4. Their separate reader retained 98.1% of full-model accuracy while reading 17.8% of routed-expert weights across three post-trained MoE models. The authors also report that its Triton kernels ran MoE blocks 1.60–1.90 times faster than optimized vLLM at batch size one and up to 2.69 times faster at larger batch sizes.

    Hugging Face · Papers · 🔥 0
  30. GitHub lists vLLM v0.33.0rc0 release candidate

    GitHub lists a new vLLM release-candidate tag, v0.33.0rc0, associated with commit 4135590. The commit was created on GitHub and carries GitHub’s verified signature, with the signing key identified as GPG key ID B5690EEEBB952194. The report does not provide release notes or list the version’s changes.

    vLLM Releases · 🔥 0

← PreviousNext →

Experience and discussion from the community

Share my vLLM experienceAsk about vLLM

torchtitan: A PyTorch-native platform for pretraining and post-training generative AI models at scale.

What problem it solves Training generative AI models at scale requires coordinating multiple parallelism strategies while supporting pretraining, fine-tuning, and reinforcement learning in one codebase. Large-scale training also needs reproducible configurations, checkpointing, debugging tools, and observability that can be adapted for research and model development. Key features Composable model …

github.comhttps://github.com/pytorch/torchtitan

vllm-ascend: A community-maintained hardware plugin for running vLLM on Huawei Ascend NPUs.

What problem it solves vLLM users need a way to run the framework on Huawei Ascend NPU hardware through a supported integration path. The plugin decouples Ascend NPU integration from vLLM and enables supported model types to run on the platform. Key features Hardware-pluggable vLLM integration — The plugin provides an interface for integrating Ascend NPUs with vLLM and follows the community's hard…

github.comhttps://github.com/vllm-project/vllm-ascend

vllm: A fast and easy-to-use library for LLM inference and serving.

What problem it solves Serving large language models requires efficient memory management, flexible model execution, and practical integration with existing models and APIs. vLLM addresses these needs with inference and serving capabilities spanning batching, caching, quantization, distributed execution, and multiple hardware platforms. Key features Efficient memory and serving — vLLM manages atte…

github.comhttps://github.com/vllm-project/vllm

semantic-router: An open, programmable decision layer for models and compute.

What problem it solves Agent harnesses need to coordinate models, tools, task state, and changing compute requirements without manually managing every backend. They also need a way to apply location constraints and priorities for quality, latency, and cost to each call. Key features Policy-driven model routing — Select or combine models for each call according to a configured policy while leaving …

github.comhttps://github.com/vllm-project/semantic-router

openworker: A desktop app and harness SDK for building and running AI coworkers that work across everyday tools.

What problem it solves People need a way to delegate research, writing, code review, recurring tasks and work across connected tools to AI agents without giving up control over actions and data. They also need a foundation for embedding agent loops, tool calls, approvals, streaming and durable sessions into their own applications. Key features Everyday AI coworker — The desktop app supports resear…

github.comhttps://github.com/andrewyng/openworker

Hy-MT2: A family of fast-thinking multilingual translation models for complex real-world translation scenarios.

What problem it solves Real-world translation involves more than converting text between languages: tasks can require terminology, style, personalization, delimiter preservation, and structured-data constraints. Hy-MT2 is designed to support translation among 33 languages and follow translation instructions in multiple languages for complex real-world scenarios. Key features Instruction-driven tas…

github.comhttps://github.com/Tencent-Hunyuan/Hy-MT2

G0DM0D3: A fully open-source, privacy-transparent, multi-model chat interface for red teaming, cognition research, and liberated AI interaction.

What problem it solves Researchers, red-teamers, and system tinkerers need a chat interface that can work with multiple hosted and local models while exposing evaluation, perturbation, and sampling controls. They also need clear visibility into data flows, storage, telemetry, and deployment boundaries so they can make informed choices about privacy and self-hosting. Key features Multi-model races …

github.comhttps://github.com/elder-plinius/G0DM0D3

memorilla: A memory system that gathers information from an entire document collection into query-dependent memory tokens for a frozen LLM.

What problem it solves A frozen LLM can only use information that fits in its context window, making it difficult to answer questions from large document collections or long interaction histories. Memorilla addresses this by reading a collection and supplying the LLM with a fixed number of query-dependent memory tokens. Key features Query-dependent memory compression — The system reads a complete …

github.comhttps://github.com/snap-stanford/memorilla