开发者将 Gemma 4 26B-A4B 剪枝至原体积 15% 用于端侧运行
AI 总结
一位 Reddit 开发者称，通过每层保留 8 个专家，将 MoE 模型 Gemma 4 26B-A4B 剪枝至原体积的 15%。由于该模型原本每个词元会选择 8 个专家，剪枝后路由变为确定性，并成为 3.81B 参数的稠密模型，Q4_K_M 量化后体积为 2.75 GB。剪枝模型在一个狭窄分类任务上与原模型达到 92% 一致，而 12B 与 26B 模型在相同样本上的一致率为 90%。在开发者的 Mac 上，该模型以 79 tokens/s 的速度运行，占用 3.8 GB 内存，速度约为 12B 模型的两倍，内存约为其一半。开发者强调，该模型不适用于通用任务，依赖知识的任务会受到影响。
深度见解
这项开发者自测表明，对于全天候运行端侧 LLM 的应用，专家剪枝可以显著缩小 MoE 模型的内存占用并提升本地推理速度。但测试仅在开发者的 Mac 和一个狭窄分类任务上进行；开发者也明确表示，该模型不适合通用用途或依赖知识的任务。
| I pruned Gemma 4 26B-A4B, an MoE model, down to 15% of its original size, and the model still does great on one narrow task. It gets 92% agreement with the original on a classification task. That’s close to how often the 12B agrees with the 26B on the same ticks, which is 90%. This isn’t a model for general tasks, and tasks that require knowledge suffer because the knowledge is just gone. It was made for my app that runs an on-device LLM 24/7, so that makes memory footprint and energy consumption extremely important. On my Mac it hits 79 tokens/s, which is roughly double the speed of the 12B, using 3.8 GB of memory (roughly half). I selected a set of 8 experts per layer and used them for everything. Since the model selects 8 experts per token by default and there are only 8, the router no longer chooses which experts to use, and the model becomes dense: 3.81B parameters, 2.75 GB at Q4_K_M. What I want is for each block to do the same computation and produce the same result, given the same input. With block-local training, I’m not training on the answer. I’m training on the thought process. I’m saying, do the same thinking, not get the same answer. This is all trained off my own data. The method is efficient enough that it could be trained on a user’s own device, on their own data, in about 5 GB of memory, to mitigate issues stemming from deployment having different data than was in the training dataset. I haven’t tested that on a real user’s device yet. If you are curious about why I decided that fine tuning was a bad idea or how I designed this, take a look at the post I wrote going into the details of how I made this and why I felt this route was right for my application. This is the first time posting publicly like this. I would love feedback or thoughts, and to anyone who knows of better methods for choosing which experts to keep, please lmk. Post: https://tolby.app/blog/frankensteining-gemmas-experts submitted by /u/Accomplished_Ad_56 [link] [comments] |