Developer prunes Gemma 4 26B-A4B to 15% for on-device use
A Reddit developer reported pruning the MoE model Gemma 4 26B-A4B to 15% of its original size by retaining eight experts per layer. Because the model normally selects eight experts per token, this makes routing deterministic and produces a dense 3.81B-parameter model, or 2.75 GB in Q4_K_M. The pruned model achieved 92% agreement with the original on one narrow classification task, compared with 90% agreement between the 12B and 26B models on the same samples. On the developer's Mac, it ran at 79 tokens per second using 3.8 GB of memory, roughly twice the speed and half the memory of the 12B model. The developer said it is not a general-purpose model and that knowledge-dependent tasks suffer.
The self-reported demonstration shows that expert pruning can substantially reduce an MoE model's footprint and increase local inference speed for an app that runs an on-device LLM around the clock. The results are limited to the developer's Mac and a narrow classification task, and the developer said the pruned model is not intended for general use or knowledge-dependent tasks.
| I pruned Gemma 4 26B-A4B, an MoE model, down to 15% of its original size, and the model still does great on one narrow task. It gets 92% agreement with the original on a classification task. That’s close to how often the 12B agrees with the 26B on the same ticks, which is 90%. This isn’t a model for general tasks, and tasks that require knowledge suffer because the knowledge is just gone. It was made for my app that runs an on-device LLM 24/7, so that makes memory footprint and energy consumption extremely important. On my Mac it hits 79 tokens/s, which is roughly double the speed of the 12B, using 3.8 GB of memory (roughly half). I selected a set of 8 experts per layer and used them for everything. Since the model selects 8 experts per token by default and there are only 8, the router no longer chooses which experts to use, and the model becomes dense: 3.81B parameters, 2.75 GB at Q4_K_M. What I want is for each block to do the same computation and produce the same result, given the same input. With block-local training, I’m not training on the answer. I’m training on the thought process. I’m saying, do the same thinking, not get the same answer. This is all trained off my own data. The method is efficient enough that it could be trained on a user’s own device, on their own data, in about 5 GB of memory, to mitigate issues stemming from deployment having different data than was in the training dataset. I haven’t tested that on a real user’s device yet. If you are curious about why I decided that fine tuning was a bad idea or how I designed this, take a look at the post I wrote going into the details of how I made this and why I felt this route was right for my application. This is the first time posting publicly like this. I would love feedback or thoughts, and to anyone who knows of better methods for choosing which experts to keep, please lmk. Post: https://tolby.app/blog/frankensteining-gemmas-experts submitted by /u/Accomplished_Ad_56 [link] [comments] |