创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

35个Qwen MoE领域LoRA公开,结果喜忧参半

AI 总结

一名Reddit作者以Apache-2.0许可证公开了面向Qwen3.6-35B-A3B的35个领域LoRA适配器,并发布数据集、领域配置、MiniLM路由器和MLX训练脚本。这些适配器在一台512GB Mac Studio M3 Ultra上以BF16训练;作者称8位融合会输出不可用内容,而运行时BF16热切换每次约需两秒。其自测中,路由准确率约为top-1 46%、top-3 70%,除HTML/CSS外,多数专用适配器未超过基础模型。

深度见解

这次开源提供了可复现的MoE领域LoRA材料和真实部署经验,也显示在该设置下,路由准确率与适配器质量可能比专业化本身更关键。

Over the last months I trained 35 domain-expert LoRA adapters on Qwen3.6-35B-A3B (native MoE, 256 experts, 3B active) with MLX on a Mac Studio M3 Ultra 512 GB, BF16, no quantization during training. Everything is public: adapters, datasets, per-domain configs, router, training scripts.

Things I learned the hard way:

- Fusing a MoE LoRA into 8-bit produced garbage; only runtime BF16 adapter hot-swap was usable. Serving = one base loaded once + adapters swapped per request (~2 s).

- A small MiniLM router over 34 domains tops out around top-1 46 %, top-3 70 % on my eval. Routing is the weak link, not the adapters.

- On an independent audit bench, the base 35B model beat most of my specialist adapters. Only one domain (HTML/CSS) clearly gained. I now route almost everything to the base. Publishing a negative result felt more useful than hiding it.

- `mx.random.normal` is not bit-exact across Apple GPU generations (M1 vs M3+). Reported upstream, docs PR merged in ml-explore/mlx last week.

Repos: github.com/hypneum-lab/micro-kiki (adapters + router), github.com/ailiance/ailiance-mac-tuner (training stack, archived). HF: huggingface.co/hypneum-lab

Question for the sub: has anyone gotten domain LoRAs to beat a strong MoE base at the 30B scale without full-parameter training? I'd like to compare routers.

#Qwen3.6-35B-A3B#MLX#MiniLM#Mac Studio M3 Ultra

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。