RustingStrata Brings Pure-Rust Qwen Inference to CUDA and Vulkan
Developer Serious_Scarcity7818 introduced RustingStrata, a pure-Rust inference engine for Qwen GGUF models without ggml, supporting CUDA and Vulkan as well as hybrid CPU-GPU execution for MoE experts that exceed VRAM capacity. On the developer’s RTX 3060 12 GB system, it averaged about 36 tokens per second decoding Qwen3.8-Flash-Next, while a third-party tester reported roughly 209 tokens per second for Qwen3.6-35B-A3B on an RX 7900 XT.
RustingStrata offers a new open development route for running Qwen models locally across CUDA and Vulkan, including large MoE models through CPU-GPU offloading, though most performance results remain developer-reported.
I've been working on this for a while: RustingStrata. It's an inference engine for Qwen GGUF models, written in pure Rust, no ggml. The CUDA kernels get compiled at startup with NVRTC, and the same kernels also run on Vulkan.
It runs a lot of Qwen models by now:
- Qwen3.5 / 3.6: 2B up to 35B-A3B
- Qwen3: 4B dense, 30B-A3B
- Qwen3.8-27B dense
- Qwen3-Next-80B-A3B
#RustingStrata#Qwen#Qwen3.8-Flash-Next#Qwen3.6-35B-A3B#Qwen3-Next-80B-A3B#Strata#llama.cpp#CUDA#Vulkan#NVRTC#NVIDIA RTX 3060#AMD Radeon RX 7900 XT