Reddit user seeks Qwen3.5-9B QLoRA fine-tuning advice
A Reddit user is seeking advice on fine-tuning Qwen3.5-9B with QLoRA using about 500 GB of mixed images, scanned documents, PDFs, Word files, and Excel files. They report running a Q5_K_XL build on an ASUS ESC4000 G3 with 128 GB of RAM and eight NVIDIA T4 GPUs, using separate llama.cpp instances behind a round-robin nginx load balancer. Their internal harness reportedly measured 35–40 tokens per second for an average single user and a 3–5 second time to first token, but the post does not provide a completed fine-tuning workflow or validated answer.
The post provides a detailed, self-reported local inference setup and performance snapshot for Qwen3.5-9B, but its fine-tuning questions remain unresolved.
Hi All:
I done overthought and burned money after consulting with AI chats (from chatgpt to grok to deepseek and kimi/etc). I had to learn.
I have in my possession:
- 1 Asus ESC4000 G3 with 128 GB DDR4 RDIMM and in it 8x T4s deployment. This server hosts the Qwen 3.5-9B Q5_K_XL at 87k context per slot and 44 slots. Each T4 has its own llama.cpp instance and the average single user gets 35-40 tps. TTFT is about 3-5 seconds depending on the size of the context. Using a round robin nginx load balancer for requests. Extensive real time testing inside my own harness (built strictly for us, non coding use), it actually isn’t bad at all. (Comparing to gpt-4o-mini quality).
- 2x V620s and 2 Mi50s (32GBs) and 1 3060 12 GB with another 3060 12 GB on the way (marketplace $250 cant complain tbh).
All this ran me $650 for Asus server, 350 per V620 and 480 shipped per Mi50