Qwen2.5-7B-Instruct-AWQ

Headlines · 1

1. 6d
   Whitepaper runs Qwen2.5-7B on an 8GB RTX 4060 with vLLM

   A version 0.3 whitepaper documents an author-run deployment of Qwen2.5-7B-Instruct-AWQ and Qwen2.5-7B-Instruct-GPTQ-Int4 with vLLM 0.33.0 on a Windows 11 laptop containing an RTX 4060 Laptop GPU with 8GB of memory. Using 4-bit quantization, an 8,192-token context limit, 0.85 GPU-memory utilization and eager mode, the test used about 7.3GB of GPU memory and reported roughly 53-54 tokens per second for a single 256-token request. The paper also reproduces three vLLM issues and publishes logs, scripts and patches, although its results cover one Windows machine and cannot provide same-system comparisons for bf16 or GGUF.

   掘金 · 人工智能 · 🔥 0

Experience and discussion from the community

Share my Qwen2.5-7B-Instruct-AWQ experienceAsk about Qwen2.5-7B-Instruct-AWQ