Qwen3.5-9B微调方案求助
一名Reddit用户就使用QLoRA微调Qwen3.5-9B寻求建议，训练数据约500GB，包含图片、扫描文档、PDF、Word和Excel文件。他称自己使用配备128GB内存和8块NVIDIA T4的ASUS ESC4000 G3服务器，通过独立的llama.cpp实例和轮询nginx负载均衡器运行Q5_K_XL版本。其内部测试工具报告单用户平均速度为35–40 tokens/s，首词延迟为3–5秒，但帖子没有给出完整微调流程或经验证的答案。
该帖提供了Qwen3.5-9B本地部署的自述配置与性能快照，但微调方案仍未得到解决。
Hi All:
I done overthought and burned money after consulting with AI chats (from chatgpt to grok to deepseek and kimi/etc). I had to learn.
I have in my possession:
- 1 Asus ESC4000 G3 with 128 GB DDR4 RDIMM and in it 8x T4s deployment. This server hosts the Qwen 3.5-9B Q5_K_XL at 87k context per slot and 44 slots. Each T4 has its own llama.cpp instance and the average single user gets 35-40 tps. TTFT is about 3-5 seconds depending on the size of the context. Using a round robin nginx load balancer for requests. Extensive real time testing inside my own harness (built strictly for us, non coding use), it actually isn’t bad at all. (Comparing to gpt-4o-mini quality).
- 2x V620s and 2 Mi50s (32GBs) and 1 3060 12 GB with another 3060 12 GB on the way (marketplace $250 cant complain tbh).
All this ran me $650 for Asus server, 350 per V620 and 480 shipped per Mi50