DenseCore

Headlines · 1

1. 3d
   Rust framework DenseCore claims bit-exact PyTorch match on nanoGPT

   A computer science student posted on r/MachineLearning about DenseCore, a small Rust deep learning framework with a PyTorch-like Python API that is roughly 2 MB versus about 580 MB for PyTorch. The author says that running Karpathy's nanoGPT unchanged against it produces identical initial weights from manual_seed, loss, gradients and updated weights bit for bit across all 50 tested steps, including with dropout enabled. Getting there required matching undocumented PyTorch behaviour such as the CPU flash attention kernel and its dropout path, rounding differences in Adam's tail loop, and clip_grad_norm_ using a reciprocal multiply; the match currently holds only on Apple Silicon Macs.

   r/MachineLearning · 🔥 0

Experience and discussion from the community

Share my DenseCore experienceAsk about DenseCore