Developer releases Qwen3-0.6B MoE conversion with GGUF build
Developer bayliner1980 released Qwen3-0.6B-MoE-A0.5B, a mixture-of-experts conversion of Qwen3-0.6B, together with a GGUF version. The developer adapted the process from Qwen 2.5's qwen2moe to Qwen3's qwen3moe and used a pseudo-shared expert to recover some quality lost because Qwen3 lacks a shared expert. The developer's evaluation found it worse than the dense model on every reported quality metric, with no GPU speed improvement at this size, while the converted model's code was broken and its thinking mode was not trained.
This is a concrete community release with GGUF artifacts and self-reported benchmarks, but its reported quality regression and lack of independent validation limit its immediate practical value.
This is another update about my MoE-fication project, the first post can be found here.
Foreword
Since my last model conversion, I’ve been experimenting a lot with new methods, though sadly, most attempts haven't helped much. Compatibility has been the main limiting factor. I’ve made GGUF compatibility a priority to ensure as many people as possible can use my models, but this means sacrificing some potential improvements.
Still, my perfectionism has finally let me publish another model: Qwen3-0.6B, my first chat-style conversion. This brought two new challenges: preserving its chat capabilities during conversion and adapting my methods from Qwen 2.5’s qwen2moe to Qwen3’s qwen3moe.
The latter proved particularly difficult. Unlike the previous architecture, Qwen3 has no shared expert, which helped preserve quality in my earlier conversions by keeping important neurons always active. Without it, quality dropped considerably.
To mitigate this, I experimented with a pseudo-shared expert, basically an expert that the router always picks. It doesn’t work quite as well as a true shared expert due to differences in weighting, but it does recover some of the lost quality.
#Qwen3-0.6B-MoE-A0.5B#Qwen3-0.6B#Qwen#bayliner1980#GGUF#qwen3moe