Reddit User Runs Kimi K3 Without a GPU Using Intel Optane Memory
Reddit user u/IndividualSurvival reported running the 2.8-trillion-parameter Kimi K3 model with llama.cpp on a 40-core third-generation Intel Xeon Scalable system using discontinued Intel Optane persistent memory, without a graphics card. The system used App Direct mode to present DRAM and Optane as separate address spaces. With 1-bit quantization, generation increased from about 0.85 to 1.50 tokens per second after DAX was disabled, allowing frequently used data to benefit from DRAM caching. A 3-bit test achieved 0.75 tokens per second and took about 18.5 hours to generate 50,000 tokens.
The report provides a quantified example of running a very large model through CPU and persistent memory without a graphics card. The test also documents the tradeoff between quantization and generation speed, although the reported output rates were low.