Qwen3-0.6B

Headlines · 3

1. Oct 23
   Developer releases Qwen3-0.6B MoE conversion with GGUF build

   Developer bayliner1980 released Qwen3-0.6B-MoE-A0.5B, a mixture-of-experts conversion of Qwen3-0.6B, together with a GGUF version. The developer adapted the process from Qwen 2.5's qwen2moe to Qwen3's qwen3moe and used a pseudo-shared expert to recover some quality lost because Qwen3 lacks a shared expert. The developer's evaluation found it worse than the dense model on every reported quality metric, with no GPU speed improvement at this size, while the converted model's code was broken and its thinking mode was not trained.

   r/LocalLLaMA · 🔥 0
2. Oct 23
   OASIS adds null routing to attention residuals

   A team from Northwestern University, the Illinois Institute of Technology, Rutgers University, and other institutions proposed OASIS, whose token- and depth-level null options let Attention Residuals abstain from assigning full probability mass to real branches. According to Xinzhiyuan’s report on the paper, the method was accepted to NeurIPS 2026 and reduced average activation kurtosis by 95.90% versus Vanilla AttnResidual across LLaMA-3.2-1B, Qwen3-0.6B, and Phi-4. The reported experiments also found a 13.21-point Qwen3-0.6B W4A4 GSM8K Pass@1 advantage over Vanilla AttnResidual and recovered 77.3%–91.8% of selected RULER and LongBench performance losses at 12K-token contexts.

   新智元 · 🔥 0
3. Oct 9
   ByteDance Seed finds DeepSeek-V4 phase sensitivity in long-context retrieval

   ByteDance's Seed team reports in a paper that adding a few irrelevant characters ahead of the same prompt makes DeepSeek-V4's answers flip on a 4-token cycle: on a code-completion task, the wrong answer 32 averaged 71.3% probability at some fill lengths while the correct answer 8 rose to 91.5% at others. In a 128K-token needle-in-a-haystack test where the keys, question and total length were held fixed and only the target's position changed, DeepSeek-V4-Flash-Base showed a maximum accuracy gap of 40.2 percentage points and V4-Pro-Base 34.8 points. The team ties the effect to the models' chunked KV cache compression, with the period matching the compression stride, and reproduces it in Qwen3-0.6B-based models trained with several compression schemes, while an uncompressed full-attention baseline did not show the same periodicity. Post-training and newer versions narrow the gap (19.1, 14.8 and 6.1 points for V4-Flash-0731, V4-Pro-0813 and V4.1-Flash-0910 respectively), but it persists.

   36氪 人工智能 · 🔥 0

Experience and discussion from the community

Share my Qwen3-0.6B experienceAsk about Qwen3-0.6B