Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

RTX 3090s and RAM drive boost Kimi K3 on second-gen Optane

AI summary

A homelabber added two RTX 3090s to a second-generation Intel Optane system running the quantized 2.8-trillion-parameter Kimi K3 model. The builder reported 3.11 tokens per second for generation and 16.8 tokens per second for prompt evaluation with 45 of 94 shards in a tmpfs RAM drive, versus 2.20 and 13.3 with Optane alone. Tom’s Hardware noted that persistent Optane memory still required additional cooling fans because it became very hot.

In-depth insight

The measured comparison shows that adding GPUs and placing nearly half of the model in RAM can materially improve a large MoE model’s inference speed, although the results come from one enthusiast’s setup.

A homelabber detailed improvements to the second-gen Optane AI build in a follow-up post on Reddit, showing the addition of two RTX 3090s to run the 2.8-trillion-parameter Moonshot Kimi K3 model at 3.11 tokens per second. Transferring almost half the model onto a RAM drive added a significant 41% to generation speed and 26% to prompt evaluation. The extra fans are still needed, though, as the persistent Optane memory gets very hot.

Follow-up: Kimi K3 Q2_K on second-gen Optane + 2× RTX 3090 — 3.11 t/s generation, 16.8 t/s prompt eval from r/homelab

That speed is not the most impressive, but it comes with 16.8 tokens per second for prompt evaluation, with 45 of 94 shards, or 414 GB out of 865 GB, living on a RAM drive. Such an arrangement is not new to enthusiasts who would set aside system memory for extra-fast performance in a pseudo-non-volatile manner — it’s still volatile, but appears like a disk — although not everyone has that much RAM sitting around free these days. With both GPUs but no RAM drive, the build ran at 2.20 tokens per second for generation and 13.3 for prompt evaluation.

#Kimi K3#Moonshot AI#RTX 3090#Intel Optane#llama.cpp

Original · 4 min readRead the original
Group chat

No comments yet. Start the conversation.