vLLM 0.33.0

Headlines · 2

1. 6d
   Whitepaper runs Qwen2.5-7B on an 8GB RTX 4060 with vLLM

   A version 0.3 whitepaper documents an author-run deployment of Qwen2.5-7B-Instruct-AWQ and Qwen2.5-7B-Instruct-GPTQ-Int4 with vLLM 0.33.0 on a Windows 11 laptop containing an RTX 4060 Laptop GPU with 8GB of memory. Using 4-bit quantization, an 8,192-token context limit, 0.85 GPU-memory utilization and eager mode, the test used about 7.3GB of GPU memory and reported roughly 53-54 tokens per second for a single 256-token request. The paper also reproduces three vLLM issues and publishes logs, scripts and patches, although its results cover one Windows machine and cannot provide same-system comparisons for bf16 or GGUF.

   掘金 · 人工智能 · 🔥 0
2. Oct 31
   vLLM 0.33.0 adds new models and CUDA 13.0-only builds

   vLLM released version 0.33.0 with 927 commits from 389 contributors, including 122 new contributors. The release adds support for models such as Reflection Beam-1.0, AliceAI-Foundation-80B-A3B, Recursive Flint, Step-5 Preview, Alpamayo 2 Super, Nemotron Parse 3.5 and MiniCPM-OCR 1.0, while upgrading to PyTorch 2.14.0. It also introduces speculative-decoding changes and publishes release artifacts for CUDA 13.0 only; project benchmarks cited in the release notes report throughput gains, but no independent measurements are provided.

   vLLM Releases · 🔥 0

Experience and discussion from the community

Share my vLLM 0.33.0 experienceAsk about vLLM 0.33.0