A Reddit user reports using Qwen 3.8 Flash Next locally through OpenCode with a vLLM backend to manage an MSc dissertation workflow involving student records, supervisors, projects, examiners and grades. The sandboxed system ingested differently formatted CSV exports, built a relational database and generated a project booklet while keeping sensitive records on local disk. The user says the model performed this specific task as well as a frontier cloud model and ran faster than Codex, but this is an unverified personal assessment rather than a general benchmark.
Ubicloud engineers reported that serving Kimi K2.6 with vLLM 0.28 produced valid tool-call schemas in 92% of 2,000 K2VV ToolCall benchmark requests. After enabling constrained decoding in vLLM 0.31, they said schema accuracy reached 100% for newer Kimi and DeepSeek models before deploying the change in production. The post also warns that constrained decoding can introduce off-course generation, grammar-translation errors, request failures, and conflicts with speculative decoding, so vLLM did not enable it by default.
MiniMax released MiniMax-M2.7, a 229B-parameter text-generation model with open weights. The company says the model supports Agent Teams, complex Skills and dynamic tool search, and can be served with SGLang, vLLM, Transformers, ModelScope or NVIDIA NIM. MiniMax also reports that an internal version updated its memory, built skills for reinforcement-learning experiments and optimized a programming scaffold over more than 100 rounds, improving its performance by 30%. The report provides company-reported benchmark results for software engineering and professional work.
MiniMax released MiniMax-M2.1 under a Modified-MIT license, enabling local deployment and use through supported inference frameworks. The model is also available through the MiniMax Open Platform, while MiniMax Code, built on M2.1, is publicly available. MiniMax says the model targets coding, tool use, instruction following, and long-horizon planning, with benchmark improvements based on its own evaluations.
MiniMax released MiniMax-M2, a mixture-of-experts model with 230 billion total parameters and 10 billion active parameters, and made its weights available for local deployment. The model is designed for coding and agentic workflows. MiniMax recommends SGLang, vLLM, MLX and Transformers for serving it, while explicitly citing day-zero support for SGLang and vLLM. The company also reported benchmark and efficiency results, but these claims remain vendor-reported. The model card lists a modified MIT license, while the release is described as open-source.
DeepSeek open-sourced DeepInfer, the inference engine behind its own service, along with related kernels. In project-run tests using DeepInfer’s benchmarks and the same hardware, SGLang reached up to 1.04x prefill throughput on four GB300 or four B300 GPUs and cut decode step time by as much as 7.4% on 64 GB300 GPUs, according to the SGLang report. A 36氪 report said vLLM also approached or partly exceeded DeepInfer under its disclosed configurations. The comparisons used project-reported results and differing workloads and measurement methods.
A retrospective empirical systems paper submitted to arXiv examines whether small open-weight models can complete real coding work rather than merely generate coherent text. Based on 4,165 gateway turns, 623 debug jobs, and three delegation studies, it reports major differences in tool-call formatting, session continuity, artifact validation, and verification across models and runtimes. The author explicitly says the studies were not pooled and do not rank models or runtimes.
A Reddit post identified pull request 22 in the vllm-xpu-kernels repository as related to Qwen4-27B. Another post said the page showed a 404 error and interpreted the change as a four-way parallel residual-stream architecture within a fixed 27-billion-parameter budget. The post speculated that this could improve multi-step reasoning, coding and consistency, but the supplied reports do not independently confirm the architecture or establish the PR's current contents.
Developer Zack Ku announced ModelAnalysis.si, a free dashboard for comparing LLM benchmarks, API token pricing, VRAM requirements, and self-hosting economics. The tool supports side-by-side and radar-chart comparisons, API spend simulation, cloud-GPU break-even calculations, and deployment commands for Ollama and vLLM.
According to Ku, the site draws on benchmark and pricing data covering more than 250 models, while a later post says it covers over 280 models. These coverage and accuracy claims have not been independently verified.
A Reddit user reports that Zaggita’s GPU performance unlock works on Chinese RTX 3080 20GB cards. In the poster’s vLLM test on two GPUs, prefill throughput rose about 20.8%–28.6% while decode steps per second showed no apparent change; the characterization of the unlock as removing an NVIDIA AI-compute restriction was not independently confirmed.
Bottlecap AI released ThinkingCap-gemma-4-26B-A4B-it, a drop-in replacement designed to reduce the amount of thinking performed by Google's Gemma 4 26B-A4B-it. In its own evaluation across 12 benchmarks, the model used 38.9% fewer thinking tokens on average while average accuracy fell by 0.15 percentage points, although AA-LCR accuracy dropped by 4 points. Bottlecap also reports lower looping and published the model with GGUF builds.
Nvidia’s new 64GB DGX Spark arrives Oct. 23 with a starting price of $4,999 and is available through manufacturing partners Acer, ASUS, Dell, Gigabyte, HP, and MSI. It retains the GB10 Grace Blackwell Superchip, DGX OS, and Nvidia’s AI software ecosystem, including Ollama, vLLM, PyTorch with CUDA, Nemotron models, and the Nvidia Agent Toolkit. Nvidia says the system supports models with up to 100 billion parameters, but TechRepublic notes that this vendor claim is not a guarantee for every model or workload. The 64GB configuration has not been independently benchmarked for the report, and its reduced memory may constrain model size, context length, and concurrent workloads. Buyers should confirm final configuration and regional pricing with the manufacturer.
StartLux has open-sourced StartLux-Prism, an inference engine designed for StartLux-Decision. According to StartLux’s tests on one NVIDIA H200, Prism ran 1.2 to 5.3 times faster than the original vLLM setup depending on the workload, while supporting combined tensor and data parallelism and deterministic output in supported configurations. The reported performance figures are vendor-published and vary with request pattern, shared-context reuse, input length, concurrency and hardware configuration.
In a discussion published by Lightcone AI and authorized by TMTPost, NVIDIA's Lai Junjie and Lightcone AI founder Wang Yisu discuss how enterprises can measure AI investment beyond GPU and data-center inputs. NVIDIA frames token economics around value, demand, supply and monetization. The speakers describe three layers of measurement: business outcomes such as task completion and business returns, process efficiency such as automation and rework, and infrastructure metrics such as cost per million tokens, latency and throughput. The article also cites rising daily token-call figures attributed to China's National Data Administration and an unnamed supermarket-chain case. The supermarket example is presented by the article as a 3% cost reduction, fourfold return on investment and a 35-day extension of payment terms.
MiniMax AI announced a partnership with Inferact to advance production inference for its models and contribute engineering improvements to the vLLM project. Inferact said its platform served MiniMax M3.1 Flash in production from the model’s launch day.
MiniMax released the open weights for M3.1 Flash, a multimodal mixture-of-experts model with 428B total parameters and 23B activated per token, according to IT之家 AI and MiniMax’s Hugging Face model card. It accepts text, image, and video input with a context window of up to 1M tokens, while MiniMax says sparse attention, quantization-aware training, and power-aware scheduling nearly tripled inference speed and reduced agent-task costs by 70% versus M3. The weights are available through Hugging Face and ModelScope, with Tenstorrent Cloud, vLLM, and SGLang among the platforms and frameworks cited as supporting deployment.
In a vLLM-published benchmark, DeepSeek V4.1 Flash reached 954,000 tokens per second per GPU on a single NVIDIA GB300 NVL72 rack using P/D disaggregation and Wide-EP, with P90 throughput of 196 tokens per second per user. On GB200, vLLM’s decode throughput was within 6% of DeepInfer’s published results across batch sizes 8–256. A specialized decode path achieved 826 P90 tokens per second for one user on four B300 GPUs. In an AgentX prefill test at concurrency 256, DEP2 delivered 64,400 tokens per second per GPU, compared with 53,300 for DEP4. The report said the configurations differed and did not characterize the difference as an engine-to-engine speedup; replay tests attributed a 13.6% DEP4 throughput loss to data-parallel imbalance.
vLLM Blog reports day-0 support for MiniMax-M3.1-Flash on NVIDIA Blackwell GPUs, NVIDIA Rubin GPUs with dedicated kernels, and AMD MI350-series GPUs. Prebuilt NVIDIA and AMD images are available. The implementation uses MiniMax Sparse Attention, an NVFP4 KV cache, NVFP4-quantized MoE weights and a DSpark drafter. vLLM reports gains in kernel and serving performance, including optimized agentic serving on GB300, Rubin and MI355X hardware. It also reports day-0 end-to-end reinforcement-learning support with veRL, FSDP2 and LoRA, demonstrated on 16 GB300 GPUs.
vLLM Blog reports a GLM-5.3 serving deployment that uses 16 NVIDIA H200 GPUs, divided into separate prefill and decode nodes. Prefill Context Parallelism splits prompts by token, Decode Context Parallelism shards the KV cache, and Hybrid HiSparse keeps decode KV data in host memory. On AgentX, the tested PCP8 × DCP8 configuration with Hybrid HiSparse reported 1.7–3.2× per-user speed at equal throughput to data-parallel serving and was the only tested configuration that served 1M-token contexts; however, these results came from the project's own contributors.
A Reddit user who has been running models on Huawei Atlas 300I Duo cards says a Silver-rated Huawei partner gave them new information on the latest Atlas lineup, which they turned into an English comparison overview. The user says the Atlas 150 is brand new and uses 950 Ascend cores, and that the material on it existed only in Chinese and had to be translated; no further specifications or independent confirmation are offered. The same post describes getting QFN working across one to three 300I Duo cards in a vLLM fork, exploring GLM5.3-flash and DeepSeek models, and running a REAP pass to cut experts so Meta Muse would fit on the three-card setup, with the user saying Muse and GLM are not yet fast enough there.
MiniMax said on X that Inferact is an API launch partner for its M3.1 Flash model, crediting the partner's vLLM expertise for the serving collaboration and saying the two will keep building serving infrastructure for complex agent workloads. Inferact said it is working with MiniMax on production inference for the model, which it describes as built for multi-step automation, real-codebase engineering and visual reasoning. The partner-stated text also cites a score of 29.6 on Agents' Last Exam (ALE); no independent measurement was provided.
PyTorch reports that TorchTitan, a PyTorch-native platform for generative AI pretraining and post-training, now supports Muse Spark Open, Muse Glimmer, Qwen3-VL, Qwen3.5, Kimi K2.7, Kimi K3 and DeepSeek-V4 through full training-stack integrations. The update adds the TitanRL reinforcement-learning stack, asynchronous rollouts, FlexShard and Dist-MoE integrations, and distributed checkpointing improvements. PyTorch also reports internal benchmarks for these features, but the announcement is a single vendor source and does not provide an independent comparison with other training frameworks.
PyTorch Blog reports that AMD Instinct MI350-series GPU users can access FlyDSL kernels through existing PyTorch operator APIs. The backend covers dense and grouped GEMM, MXFP8 and MXFP4 scaled GEMM, attention, RMSNorm and TopK. GEMMs use the optional backend through torch.compile, eligible RMSNorm and TopK calls can dispatch automatically in eager mode, and unsupported inputs retain existing PyTorch implementations. FlyDSL attention kernels are included in AOTriton 0.14.2b or later. Official tests on MI355X reported a 1.10x geometric-mean speedup for the BF16 dense-GEMM suite, while end-to-end vLLM A/B tests reached whole-request speedups of 1.13x for BF16, 2.04x for MXFP8 and 3.01x for MXFP4. These results depend on the workload, model and concurrency, and were not independently verified in the supplied reports.
At Nvidia's GTC event in Berlin, Red Hat and Nvidia expanded their collaboration to help enterprises secure autonomous AI agents, govern access to AI models and deploy workloads on Nvidia's upcoming Vera Rubin data center infrastructure. The updates center on Red Hat AI Factory with Nvidia, combining agent runtime controls, model inference software and hardware support across hybrid clouds, and Red Hat said it is working to bring Nvidia's OpenShell sandbox runtime into its AI platforms as a native capability. The companies aim to place security controls outside the agent and, where appropriate, outside the host operating system, pairing OpenShell with Nvidia's Sentry framework on BlueField data processing units, where DOCA Argus monitors workloads from a separate hardware security domain. A joint reference design adds Model-as-a-Service access controls and an AI Gateway alongside vLLM, llm-d, Nvidia Dynamo and NIM, and Red Hat said its Enterprise Linux for Nvidia now supports Vera CPUs and is prepared for Vera Rubin.
IBM released Granite Speech 5.0 2B and revised Granite Speech 5.0 Turbo CTC and Granite Speech 4.1 2B Plus, all under Apache 2.0 with native Transformers support. According to IBM, the new 2B model adds multilingual transcription and translation, language-ID prompting and keyword biasing for up to 2,000 terms, while Turbo CTC gains multilingual transcription and native punctuation; 4.1 2B Plus improves speaker attribution, timestamps and long-speech performance. IBM also reports a 5.06% average English WER for Granite Speech 5.0 2B and 19,870 RTFx for Turbo CTC, though these results are vendor-provided and the former ranking is described as unofficial.
OvisOCR3 has been released as a 0.8B OCR model built on Qwen3.5-0.8B and published under Apache 2.0 with vLLM inference examples, according to a Reddit post in r/LocalLLaMA. Beyond OvisOCR2's Markdown parsing, the poster says it adds structured JSON output with bounding boxes, element categories and content plus a layout-only mode, key information extraction from specified fields or a supplied JSON template, figure descriptions with Mermaid output for flowcharts, and cross-page table reconstruction that parses pages and then merges table fragments with the same model. The post cites model-card numbers of 96.74 on OmniDocBench v1.6, 88.80 on MDPBench and 82.31 on PureDocBench, with the largest reported gains over OvisOCR2 on MDPBench (69.00 to 88.80) and PureDocBench (75.10 to 82.31); the poster notes these are the team's own reported figures rather than independent measurements and asks how the structure holds up on messy scans and long financial tables.
Ant Group and the vLLM community have open-sourced Humming, a general-purpose kernel library for quantized inference, hosted at github.com/vllm-project/humming. The project says it supports NVIDIA GPU architectures from Turing onward plus CUDA-compatible devices such as T-Head's Zhenwu 810E, and uses a unified quantization representation covering integer and floating-point weight/activation formats, bit widths and scaling schemes for dense and MoE workloads. The authors report that Humming matches or beats vLLM Marlin, FlashInfer and DeepGEMM on many workloads in their own GEMM microbenchmarks, and say it exploits two undocumented Blackwell Tensor Core formats, float4e0m3 and float8e3m4, that are not exposed in the PTX type system.
vLLM said it supports Recursive's Flint from day 0, with both the NVFP4 and BF16 checkpoints running on a single GPU. Flint is a 40B-parameter mixture-of-experts model that activates about 2.5B parameters per token, uses sliding-window attention in most of its 49 layers and carries a 262K-token context. vLLM reports that a single request decodes at roughly 500 tokens/s on one GB200 even with 100K tokens of context, and that its Flint-specific kernels cut NVFP4 decode from 4.81 ms to 1.78 ms and reduced a forward pass from 1,096 kernels to 346.
SGLang launched SGLang-TorchTPU, an open-source native PyTorch backend that brings its serving stack to Google Cloud TPU while retaining the same core codebase used for GPUs. The project, developed by RadixArk’s SGLang team with Google and Meta, includes real-TPU CI tests and support for multiple language and diffusion models. SGLang and Meta personnel said the backend is running production inference at Meta, while Tom’s Hardware also reported the open-source release of a matching PyTorch-native vLLM version.
AWS Machine Learning Blog reports that supervised fine-tuning raised GPT-OSS-20B’s reasoning-language accuracy from 22% to 100% on a 50-case test covering English, French, German, Spanish and Italian. Applying GRPO after SFT increased answer-language accuracy from 62% to 70%, while a Qwen2.5-7B control experiment added 6 points on multilingual reasoning but produced no gain on GSM8K.
What problem it solves
Training generative AI models at scale requires coordinating multiple parallelism strategies while supporting pretraining, fine-tuning, and reinforcement learning in one codebase. Large-scale training also needs reproducible configurations, checkpointing, debugging tools, and observability that can be adapted for research and model development.
Key features
Composable model …
What problem it solves
vLLM users need a way to run the framework on Huawei Ascend NPU hardware through a supported integration path. The plugin decouples Ascend NPU integration from vLLM and enables supported model types to run on the platform.
Key features
Hardware-pluggable vLLM integration — The plugin provides an interface for integrating Ascend NPUs with vLLM and follows the community's hard…
What problem it solves
Serving large language models requires efficient memory management, flexible model execution, and practical integration with existing models and APIs. vLLM addresses these needs with inference and serving capabilities spanning batching, caching, quantization, distributed execution, and multiple hardware platforms.
Key features
Efficient memory and serving — vLLM manages atte…
What problem it solves
Agent harnesses need to coordinate models, tools, task state, and changing compute requirements without manually managing every backend. They also need a way to apply location constraints and priorities for quality, latency, and cost to each call.
Key features
Policy-driven model routing — Select or combine models for each call according to a configured policy while leaving …
What problem it solves
People need a way to delegate research, writing, code review, recurring tasks and work across connected tools to AI agents without giving up control over actions and data. They also need a foundation for embedding agent loops, tool calls, approvals, streaming and durable sessions into their own applications.
Key features
Everyday AI coworker — The desktop app supports resear…
What problem it solves
Real-world translation involves more than converting text between languages: tasks can require terminology, style, personalization, delimiter preservation, and structured-data constraints. Hy-MT2 is designed to support translation among 33 languages and follow translation instructions in multiple languages for complex real-world scenarios.
Key features
Instruction-driven tas…
What problem it solves
Researchers, red-teamers, and system tinkerers need a chat interface that can work with multiple hosted and local models while exposing evaluation, perturbation, and sampling controls. They also need clear visibility into data flows, storage, telemetry, and deployment boundaries so they can make informed choices about privacy and self-hosting.
Key features
Multi-model races …
What problem it solves
A frozen LLM can only use information that fits in its context window, making it difficult to answer questions from large document collections or long interaction histories. Memorilla addresses this by reading a collection and supplying the LLM with a fixed number of query-dependent memory tokens.
Key features
Query-dependent memory compression — The system reads a complete …