RTX 3090与内存盘加速Optane运行Kimi K3
一位爱好者在第二代Intel Optane系统中加入两张RTX 3090,用于运行量化后的2.8万亿参数Kimi K3模型。据其测量,将94个分片中的45个放入tmpfs内存盘后,生成速度达到每秒3.11个token,提示评估速度为每秒16.8个token;仅使用Optane时分别为2.20和13.3。Tom’s Hardware指出,持久内存Optane温度仍然很高,需要额外安装风扇。
前后对比数据表明,增加GPU并将近一半模型权重放入RAM可显著提升大型MoE模型的推理速度,但结果仅来自一位爱好者的单套机器。
A homelabber detailed improvements to the second-gen Optane AI build in a follow-up post on Reddit, showing the addition of two RTX 3090s to run the 2.8-trillion-parameter Moonshot Kimi K3 model at 3.11 tokens per second. Transferring almost half the model onto a RAM drive added a significant 41% to generation speed and 26% to prompt evaluation. The extra fans are still needed, though, as the persistent Optane memory gets very hot.
Follow-up: Kimi K3 Q2_K on second-gen Optane + 2× RTX 3090 — 3.11 t/s generation, 16.8 t/s prompt eval from r/homelab
That speed is not the most impressive, but it comes with 16.8 tokens per second for prompt evaluation, with 45 of 94 shards, or 414 GB out of 865 GB, living on a RAM drive. Such an arrangement is not new to enthusiasts who would set aside system memory for extra-fast performance in a pseudo-non-volatile manner — it’s still volatile, but appears like a disk — although not everyone has that much RAM sitting around free these days. With both GPUs but no RAM drive, the build ran at 2.20 tokens per second for generation and 13.3 for prompt evaluation.