Reson8 released the 0.5B-parameter Rumble-2-Flash speech recognition model under the Apache 2.0 license with open weights. It supports more than 40 languages, automatic language detection and configurable transcription output, with native integration for Transformers and vLLM. Reson8 claims the model can process an hour of audio in about half a second and performs on par with NVIDIA's 30B Nemotron 3.5 Transcribe on a 302-hour noisy call-centre benchmark.
Microsoft opened the Windows Server 2028 Build 29706 public preview for developers and IT administrators through the Windows Server Insider program. The preview enables native NVMe by default, supports ReFS-formatted system partitions and an early-preview NVMe over Fabrics client, and introduces infrastructure for local AI inference. New Server Core installations also receive stricter security defaults, which TechRepublic says may require compatibility checks for legacy workloads.
Kestrel_VLLM decoupled CUDA batched-prefill block width from mdl_mmax, allowing adaptive trailing blocks and making the feature default to 16 when VLLM_CUDA_Q35_MM is unset. The report says logits remained bitwise equivalent in its tests on x86-64 with an NVIDIA RTX 5090 and Qwen3.5-27B q4 hybrid quantization. It also documents a variable-shadowing access violation and a timing result inflated by diagnostic stderr output, which was corrected from 57.41 ms to 26.75 ms.
Tarek Ziadé reported a Transformers implementation that streams quantized MoE experts from SSD into a bounded device cache, enabling the 180B-parameter Qwen3.8-Flash-Next model to run on a MacBook M5 with 32 GB of unified memory. The author measured 13.9 tokens per second after optimization and said this remained about 6% behind Strata, with per-layer synchronization identified as the remaining gap. The implementation is available through an open pull request, but its API was still under discussion with maintainers.
A vLLM blog post describes integrating FlyDSL MonoKernels into vLLM for interactive, low-concurrency LLM serving on AMD Instinct GPUs. The approach combines dependent operator stages within a single kernel launch using a persistent GPU schedule, reducing repeated kernel startup, drain and scheduling overhead. The blog reports same-node concurrency-1 interactivity gains of 1.20× to 2.26× over stock vLLM, while noting that results vary by model, workload and system configuration.
AWS Machine Learning Blog says Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo for multi-node model serving, disaggregated prefill and decode, and KV-aware routing. The managed add-on installs Dynamo components and provides endpoint configuration, GPU scheduling, node recovery, and autoscaling, with deployment examples using GLM-5.3 on vLLM.
AWS Machine Learning Blog reports that Amazon SageMaker AI real-time endpoints now support KV cache-aware routing, a new RoutingStrategy option for LLM serving. The feature uses cache state reported by the model container to send requests to instances holding matching cached context while avoiding overly busy instances. Adoption requires the SM_VLLM_KV_EVENTS_CONFIG environment variable, the KV_AWARE routing strategy, and a supported container.
SenseNova-Loop-9B has been released under the Apache 2.0 license. According to the poster, it retains the Preview design, with 32 layers stored and 24 of them run twice, producing 56 layers of forward-pass work for a model of about 9 billion parameters. The SenseNova README reportedly says the full release outperformed the Preview on all seven listed benchmarks, including AutomationBench rising from 14.33 to 36.67, Terminal-Bench 2.1 from 40.45 to 59.55 and GAIA from 37.86 to 69.90. The Reddit poster said they had not run the model themselves.
The second installment of the personal learn-vllm-paged-kvcache project implements and explains physical block allocation, block tables, slot mapping, reference counting, and prefix-cache reuse for paged KV Cache management. Written in dependency-free C++17, it uses deterministic integer simulation instead of GPU tensors; its author reports 61 unit tests, two stress tests, and CI across Linux/GCC, Windows/MSVC, and ThreadSanitizer, without independently verified results.
The Clover 2 team says it has released the 27.8B base model, seven specialized LoRA adapters, a trained router, and a vLLM plugin. Rather than merging the adapters, the router selects one or more experts for each token and can fall back to the unmodified base model. The developer also reports stronger GSM8K, MATH-500, GSM-Hard, and LongBench-v2 results, but those benchmark gains have not been independently validated.
The PyTorch Foundation announced the winners and nominees of its fourth annual Contributor Awards, presented at PyTorch Conference 2026 in San Jose, California. The program recognized 15 winners and 11 nominees across nine categories, including three new categories for knowledge sharing, bug resolution and developer workflow contributions. Contributions covered PyTorch, vLLM, Ray and DeepSpeed.
The PyTorch Foundation published a November 2026 roundup covering ambassador-led meetups, university sessions, technical talks, workshops, articles, and learning resources. Highlights included a Kampala workshop on retrieval-augmented generation and semantic search, a Dhaka demonstration of running native PyTorch on Google TPUs through Colab, a Nepal session on speculative decoding for LLM inference, and a Kuala Lumpur talk on training-inference consistency in reinforcement-learning post-training.
A personal C++17 learning repository teaches vLLM-style continuous batching and paged KV-cache scheduling without a GPU or third-party dependencies. The implementation covers request admission, preemption with recomputation, background-thread concurrency, copy-on-write, prefix caching, invariant checks, and differential testing. The author reports 61 unit tests and two stress tests, with CI configurations for Linux/GCC, Windows/MSVC, and ThreadSanitizer. The project is described as a personal learning reproduction unrelated to the official vLLM project and contains none of its source code.
The MMFlash authors released code and draft-model artifacts for speeding up speculative decoding in vision-language models, with support for Qwen3.5-4B, Qwen3.5-9B, Qwen3.8-27B and Gemma 4 E4B-it targets. They report that MMFlash adaptively prioritizes visually dependent tokens and uses sparse attention for long-video inputs, achieving lossless decoding speedups of up to 3.58× across multimodal benchmarks. These benchmark results come from the authors and have not been independently validated by the Reddit report.
vLLM v0.34.0 introduces 10 models, including Mistral Large 4, MiniMax-M3.1-Flash and NVIDIA Nemotron-3.5-Transcribe, and makes extensible KV caching the default. The release removes Model Runner V1, requires huggingface_hub 2.0.0 or later, changes the default block size to 64 and drops ROCm 7.2 artifacts. vLLM reports up to 1.95× faster decoding, 1.6× higher throughput and 1.24–1.40× faster long-prompt TTFT on Blackwell and MI355X hardware.
A community benchmark on four 16 GB Tesla V100 GPUs found that 1Cat-vLLM delivered higher single-stream decode rates than Strata across all tested prompt lengths, including 124.9 versus 74.8 tokens per second at 2,654 tokens. It also led most prefill and concurrent-request tests, but Strata performed better on prefill near 100K tokens and generally achieved higher multi-token-prediction acceptance. The author says 1Cat-vLLM is a Tesla V100-focused vLLM fork with original expert-caching work.
A troubleshooting account describes serving Qwen/Qwen2.5-7B-Instruct with vLLM on an AutoDL instance equipped with a 24 GB NVIDIA RTX 4090. After encountering an out-of-memory error, the author recommends terminating residual vLLM and EngineCore processes, setting GPU-memory utilization to 0.85, and limiting the maximum model length to 4,096 to reduce KV Cache allocation. The launch command also enables ModelScope for model downloads. The account further documents network and authentication problems involving Hugging Face, including a 401 error associated with Xet, and notes that the ModelScope Python package must be installed separately. The source title refers to “QWeb7B,” while its launch command and final API request identify Qwen/Qwen2.5-7B-Instruct.
A Reddit user reports completing a stable local AI rig built around eight matching Gigabyte Gaming OC RTX 3090 GPUs, an ASRock ROMED8-2T motherboard, PCIe 4.0 risers and CPU water cooling. The builder says the GPUs were bought by early 2026 for roughly £600–£700 each, with six operating at PCIe 4.0 x16 and two sharing a slot at PCIe 4.0 x8 each. They also report GPU peer-to-peer transfers of 22 GB/s, excluding the two x8 GPUs, and Qwen Flash generation of about 60 tokens per second for one agent and 370 tokens per second in aggregate for 10 parallel agents. The hardware, cost and performance figures are self-reported and have not been independently verified; the exact Qwen Flash model is not specified.
vLLM Releases reports that the v0.34.0rc4 release candidate was tagged on 14 Nov at 10:15, pointing to commit 62d4bc3. The tag references a truncated test-runner change titled “[Release][Tests] Give the #67296 warmup-gate test runner num_speculat…”.
The vLLM Releases listing identifies the v0.34.0rc3 tag with commit 9950ce3 and includes a KVConnector change described as allocating an extensible KV cache that can be exported. The supplied listing is truncated and contains loading errors, so it does not explain the implementation, supported export paths, or provide independent confirmation.
NVIDIA Technical Blog shows how to serve the quantized 753-billion-parameter GLM-5.3 locally with vLLM across two connected NVIDIA DGX Stations. Tensor parallelism splits weights and computation between the GB300 GPUs, keeping the roughly 464 GB checkpoint in GPU memory behind one OpenAI-compatible API. In NVIDIA’s SPEED-Bench test, throughput peaked at 607.8 tokens per second at concurrency 16 before falling to 385.0 tokens per second at concurrency 32.
Helion now analyzes kernel intermediate representation and dataflow at compile time to select configurations without autotuning. Its heuristics classify kernels as matmul, reduction, or pointwise and choose block sizes, warp counts, and pipeline stages. According to PyTorch Blog, the project reported gains over handwritten implementations in linear-attention and vLLM reduction benchmarks, with the feature enabled by default for H100 and B200 on the Triton backend.
An unofficial r/OpenAI post reports that Row-Bot 5.3.3 can now run in a browser, with a Railway flow creating a signed-in Row-Bot in the user's account. The release adds 12 API providers, local backends including LM Studio, vLLM and llama.cpp, backup-model failover, broader media-generation support, and a low-severity security fix for tools_config.json; these vendor claims have not been independently confirmed.
Salesforce and Missionforce Sovereign AI released SOV-Transcribe-1, an open-weight, speech-augmented language model for non-streaming automatic speech recognition. Artificial Analysis reported that it ranked first among open-weight speech-to-text models with a 2.3% AA-WER score and seventh among 44 models overall, while a Salesforce description cited by r/LocalLLaMA placed it second of 23 systems on VoiceCodeBench for entity accuracy and task success. The model supports seven languages, natural-language transcription controls, context biasing, and deployment through vLLM or Hugging Face Transformers.
A two-person team used AI coding agents to integrate SGLang with IBM's Spyre accelerator in less than a week, producing an approximately 1,500-line plugin based on existing PyTorch support and SGLang extension points. The PyTorch Blog reports that SGLang reached about 3.6 times the throughput of a reference PyTorch inference implementation using torch-spyre, matched vLLM on the tested workloads for Granite 3.3 8B and Gemma 4 26B A4B, and retained RadixCache prefix sharing without Spyre-specific changes. The plugin is available as open source alongside upstream SGLang.
The vLLM Releases page lists the v0.34.0rc2 tag, corresponding to commit 8eda490. The visible change concerns documentation and tests that stop setting gpu_memory_utilization in most extensible cases. The source provides no further release notes and displays loading errors.
InfoQ’s 2026 trends report summarizes its editorial team’s and podcast guests’ discussions of emerging AI, ML and data engineering developments and directions to watch over the coming 12 months. It organizes technologies and topics on an adoption curve, including innovator, early-adopter, early-majority and late-majority stages. Topics include agent communication protocols, inference engines, harness engineering, physical AI, AI agents, model and agent evaluation, context engineering, and AI-assisted software development. The report also presents forecasts for 2027.
According to vLLM Blog, vLLM now supports Thinking Machines’ Inkling 2 Small, a multimodal mixture-of-experts model with 594B total parameters, 17B active parameters and a context window of up to 1M tokens. The integration supports mixed-precision serving on a single node, LoRA, reinforcement learning through SkyRL and a native DSpark speculator that drafts seven tokens per step. vLLM reports optimizations that reached about 630 tokens per second for a single user at TP4 on GB200, with an acceptance length of 3.5. Its accuracy evaluations on NIAH, MMMU-Pro and HLE reportedly matched runs without speculative decoding. The performance and accuracy figures are the project’s own results in the supplied report.
vLLM says its serving stack achieved 1.7 million tokens per second per GPU on one NVIDIA Vera Rubin NVL72 rack while serving each user at 387 tokens per second, a claimed 58× result versus GB300 NVL72. The project attributed the results to Rubin-tuned kernels, a specialized tensor-parallel decode path, rack-scale KV-cache and Engram memory pools, and workload-aware routing. It also reported peak throughput of 2.52 million tokens per second and 1,810 tokens per second for a single user. The figures came from project-led testing with the SemiAnalysis AgentX benchmark, and the report’s HBM extension of the Mooncake KV pool remained experimental.
The vLLM releases report shows that the v0.34.0rc1 tag points to commit 52aeb58 and was tagged on November 11 at 19:37. Its visible release note lists an XPU speculative-decoding bugfix that changes greedy_argmax to treat NaN as negative infinity, although the title is truncated and the page shows loading errors.
What problem it solves
Training generative AI models at scale requires coordinating multiple parallelism strategies while supporting pretraining, fine-tuning, and reinforcement learning in one codebase. Large-scale training also needs reproducible configurations, checkpointing, debugging tools, and observability that can be adapted for research and model development.
Key features
Composable model …
What problem it solves
vLLM users need a way to run the framework on Huawei Ascend NPU hardware through a supported integration path. The plugin decouples Ascend NPU integration from vLLM and enables supported model types to run on the platform.
Key features
Hardware-pluggable vLLM integration — The plugin provides an interface for integrating Ascend NPUs with vLLM and follows the community's hard…
What problem it solves
Serving large language models requires efficient memory management, flexible model execution, and practical integration with existing models and APIs. vLLM addresses these needs with inference and serving capabilities spanning batching, caching, quantization, distributed execution, and multiple hardware platforms.
Key features
Efficient memory and serving — vLLM manages atte…
What problem it solves
Agent harnesses need to coordinate models, tools, task state, and changing compute requirements without manually managing every backend. They also need a way to apply location constraints and priorities for quality, latency, and cost to each call.
Key features
Policy-driven model routing — Select or combine models for each call according to a configured policy while leaving …
What problem it solves
People need a way to delegate research, writing, code review, recurring tasks and work across connected tools to AI agents without giving up control over actions and data. They also need a foundation for embedding agent loops, tool calls, approvals, streaming and durable sessions into their own applications.
Key features
Everyday AI coworker — The desktop app supports resear…
What problem it solves
Real-world translation involves more than converting text between languages: tasks can require terminology, style, personalization, delimiter preservation, and structured-data constraints. Hy-MT2 is designed to support translation among 33 languages and follow translation instructions in multiple languages for complex real-world scenarios.
Key features
Instruction-driven tas…
What problem it solves
Researchers, red-teamers, and system tinkerers need a chat interface that can work with multiple hosted and local models while exposing evaluation, perturbation, and sampling controls. They also need clear visibility into data flows, storage, telemetry, and deployment boundaries so they can make informed choices about privacy and self-hosting.
Key features
Multi-model races …
What problem it solves
A frozen LLM can only use information that fits in its context window, making it difficult to answer questions from large document collections or long interaction histories. Memorilla addresses this by reading a collection and supplying the LLM with a fixed number of query-dependent memory tokens.
Key features
Query-dependent memory compression — The system reads a complete …