创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

RTX 3090与内存盘加速Optane运行Kimi K3

AI 总结

一位爱好者在第二代Intel Optane系统中加入两张RTX 3090,用于运行量化后的2.8万亿参数Kimi K3模型。据其测量,将94个分片中的45个放入tmpfs内存盘后,生成速度达到每秒3.11个token,提示评估速度为每秒16.8个token;仅使用Optane时分别为2.20和13.3。Tom’s Hardware指出,持久内存Optane温度仍然很高,需要额外安装风扇。

深度见解

前后对比数据表明,增加GPU并将近一半模型权重放入RAM可显著提升大型MoE模型的推理速度,但结果仅来自一位爱好者的单套机器。

A homelabber detailed improvements to the second-gen Optane AI build in a follow-up post on Reddit, showing the addition of two RTX 3090s to run the 2.8-trillion-parameter Moonshot Kimi K3 model at 3.11 tokens per second. Transferring almost half the model onto a RAM drive added a significant 41% to generation speed and 26% to prompt evaluation. The extra fans are still needed, though, as the persistent Optane memory gets very hot.

Follow-up: Kimi K3 Q2_K on second-gen Optane + 2× RTX 3090 — 3.11 t/s generation, 16.8 t/s prompt eval from r/homelab

That speed is not the most impressive, but it comes with 16.8 tokens per second for prompt evaluation, with 45 of 94 shards, or 414 GB out of 865 GB, living on a RAM drive. Such an arrangement is not new to enthusiasts who would set aside system memory for extra-fast performance in a pseudo-non-volatile manner — it’s still volatile, but appears like a disk — although not everyone has that much RAM sitting around free these days. With both GPUs but no RAM drive, the build ran at 2.20 tokens per second for generation and 13.3 for prompt evaluation.

#Kimi K3#Moonshot AI#RTX 3090#Intel Optane#llama.cpp

原文约 4 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。