Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

35 Qwen MoE domain LoRAs released with mixed results

AI summary

A Reddit author released 35 domain-expert LoRA adapters for Qwen3.6-35B-A3B under Apache-2.0, along with datasets, domain configurations, a MiniLM router, and MLX training scripts. The adapters were trained in BF16 on one 512GB Mac Studio M3 Ultra; 8-bit fusion produced unusable output, while runtime BF16 hot-swapping reportedly took about two seconds per request. On the author's evaluations, routing reached about 46% top-1 and 70% top-3 accuracy, and the base model outperformed most specialist adapters except in HTML/CSS.

In-depth insight

The Apache-2.0 release provides reproducible artifacts and candid deployment findings for domain LoRAs on a MoE model, while showing that routing and adapter quality can outweigh specialization in this setup.

Over the last months I trained 35 domain-expert LoRA adapters on Qwen3.6-35B-A3B (native MoE, 256 experts, 3B active) with MLX on a Mac Studio M3 Ultra 512 GB, BF16, no quantization during training. Everything is public: adapters, datasets, per-domain configs, router, training scripts.

Things I learned the hard way:

- Fusing a MoE LoRA into 8-bit produced garbage; only runtime BF16 adapter hot-swap was usable. Serving = one base loaded once + adapters swapped per request (~2 s).

- A small MiniLM router over 34 domains tops out around top-1 46 %, top-3 70 % on my eval. Routing is the weak link, not the adapters.

- On an independent audit bench, the base 35B model beat most of my specialist adapters. Only one domain (HTML/CSS) clearly gained. I now route almost everything to the base. Publishing a negative result felt more useful than hiding it.

- `mx.random.normal` is not bit-exact across Apple GPU generations (M1 vs M3+). Reported upstream, docs PR merged in ml-explore/mlx last week.

Repos: github.com/hypneum-lab/micro-kiki (adapters + router), github.com/ailiance/ailiance-mac-tuner (training stack, archived). HF: huggingface.co/hypneum-lab

Question for the sub: has anyone gotten domain LoRAs to beat a strong MoE base at the 30B scale without full-parameter training? I'd like to compare routers.

#Qwen3.6-35B-A3B#MLX#MiniLM#Mac Studio M3 Ultra

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.