35个Qwen MoE领域LoRA公开,结果喜忧参半
一名Reddit作者以Apache-2.0许可证公开了面向Qwen3.6-35B-A3B的35个领域LoRA适配器,并发布数据集、领域配置、MiniLM路由器和MLX训练脚本。这些适配器在一台512GB Mac Studio M3 Ultra上以BF16训练;作者称8位融合会输出不可用内容,而运行时BF16热切换每次约需两秒。其自测中,路由准确率约为top-1 46%、top-3 70%,除HTML/CSS外,多数专用适配器未超过基础模型。
这次开源提供了可复现的MoE领域LoRA材料和真实部署经验,也显示在该设置下,路由准确率与适配器质量可能比专业化本身更关键。
Over the last months I trained 35 domain-expert LoRA adapters on Qwen3.6-35B-A3B (native MoE, 256 experts, 3B active) with MLX on a Mac Studio M3 Ultra 512 GB, BF16, no quantization during training. Everything is public: adapters, datasets, per-domain configs, router, training scripts.
Things I learned the hard way:
- Fusing a MoE LoRA into 8-bit produced garbage; only runtime BF16 adapter hot-swap was usable. Serving = one base loaded once + adapters swapped per request (~2 s).
- A small MiniLM router over 34 domains tops out around top-1 46 %, top-3 70 % on my eval. Routing is the weak link, not the adapters.
- On an independent audit bench, the base 35B model beat most of my specialist adapters. Only one domain (HTML/CSS) clearly gained. I now route almost everything to the base. Publishing a negative result felt more useful than hiding it.
- `mx.random.normal` is not bit-exact across Apple GPU generations (M1 vs M3+). Reported upstream, docs PR merged in ml-explore/mlx last week.
Repos: github.com/hypneum-lab/micro-kiki (adapters + router), github.com/ailiance/ailiance-mac-tuner (training stack, archived). HF: huggingface.co/hypneum-lab
Question for the sub: has anyone gotten domain LoRAs to beat a strong MoE base at the 30B scale without full-parameter training? I'd like to compare routers.