NanoGPT

Headlines · 2

1. 6d
   KS3 benchmark reports faster convergence than MLP at similar GPU latency

   Independent researcher u/1ncehost compared the third version of K-Splanifolds, KS3, with multilayer perceptrons and earlier KS versions while fitting a 4D Mandelbulb through gradient descent on random samples. The author reports that KS3 converged faster and had nearly identical GPU latency to an MLP. The author also says KS3 ranged from considerably faster convergence than an MLP in areas including noise removal, geometric fitting and financial prediction to roughly equal performance for large language models. These are self-reported results that have not been independently verified.

   r/MachineLearning · 🔥 0
2. Oct 16
   Rust framework DenseCore claims bit-exact PyTorch match on nanoGPT

   A computer science student posted on r/MachineLearning about DenseCore, a small Rust deep learning framework with a PyTorch-like Python API that is roughly 2 MB versus about 580 MB for PyTorch. The author says that running Karpathy's nanoGPT unchanged against it produces identical initial weights from manual_seed, loss, gradients and updated weights bit for bit across all 50 tested steps, including with dropout enabled. Getting there required matching undocumented PyTorch behaviour such as the CPU flash attention kernel and its dropout path, rounding differences in Adam's tail loop, and clip_grad_norm_ using a reciprocal multiply; the match currently holds only on Apple Silicon Macs.

   r/MachineLearning · 🔥 0

Experience and discussion from the community

Share my NanoGPT experienceAsk about NanoGPT