Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Running Qwen2.5-7B-Instruct with vLLM on an AutoDL RTX 4090

AI summary

A troubleshooting account describes serving Qwen/Qwen2.5-7B-Instruct with vLLM on an AutoDL instance equipped with a 24 GB NVIDIA RTX 4090. After encountering an out-of-memory error, the author recommends terminating residual vLLM and EngineCore processes, setting GPU-memory utilization to 0.85, and limiting the maximum model length to 4,096 to reduce KV Cache allocation. The launch command also enables ModelScope for model downloads. The account further documents network and authentication problems involving Hugging Face, including a 401 error associated with Xet, and notes that the ModelScope Python package must be installed separately. The source title refers to “QWeb7B,” while its launch command and final API request identify Qwen/Qwen2.5-7B-Instruct.

RTX4090 的显存是 24G,不够用

  • 模型是Qweb-7B,需要15G显存。那么需要使用 --gpu-memory-utilization 0.85 让它只占用一部分。

1. 杀掉残留的僵尸进程,把显存腾出来

pkill -9 -f vllm

pkill -9 -f EngineCore

2. 确认显存是不是真的降下来了

watch -n 1 nvidia-smi

#Qwen2.5-7B-Instruct#vLLM#ModelScope#AutoDL#NVIDIA GeForce RTX 4090

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.