Rust framework DenseCore claims bit-exact PyTorch match on nanoGPT
A computer science student posted on r/MachineLearning about DenseCore, a small Rust deep learning framework with a PyTorch-like Python API that is roughly 2 MB versus about 580 MB for PyTorch. The author says that running Karpathy's nanoGPT unchanged against it produces identical initial weights from manual_seed, loss, gradients and updated weights bit for bit across all 50 tested steps, including with dropout enabled. Getting there required matching undocumented PyTorch behaviour such as the CPU flash attention kernel and its dropout path, rounding differences in Adam's tail loop, and clip_grad_norm_ using a reciprocal multiply; the match currently holds only on Apple Silicon Macs.
The author reports that only 1 of 26 tested PyTorch ops produced the same bits between Mac and Linux x86, indicating that reproducing PyTorch results across machines already varies and that bit-exact parity requires copying many undocumented kernel behaviours.
hey, CS student here. for the past few days I've been building DenseCore, a small deep learning framework in rust with a pytorch-like python API.
the thing I'm kind of proud of: if you point `import torch` at it and run karpathy's nanoGPT unchanged, you get the exact same numbers as pytorch. same initial weights from manual_seed, same loss, same gradients, same weights, bit for bit, for all 50 steps I tested, even with dropout on. not "close enough", literally identical.
getting there was a rabbit hole. pytorch's CPU attention isn't the textbook formula, it's a flash attention kernel, and it switches to a different path when dropout is on. adam rounds differently on the last few elements of each thread's chunk because of how the compiler fuses the tail loop. clip_grad_norm_ secretly does reciprocal() * max_norm instead of a division. stuff like that everywhere.
also found out pytorch doesn't match itself across machines. same version, mac vs linux x86, only 1 of 26 ops I tried gave the same bits. so mine only matches pytorch on apple silicon macs.
it's tiny though, about 2 MB vs ~580 MB for pytorch.
anyway the honest question: is this useful to anyone, or is it just a cool learning project? if I keep going I'm thinking either keep copying pytorch until most of it works, or make it a lightweight drop-in for small cpu stuff like serverless/CI, or turn it into a tool that tells you why two pytorch runs give different results. would any of those be useful to you? or should I move on? brutal answers are fine.
repo: https://github.com/lncch/DenseCore
(`python scripts/demo_bit_exact.py` runs the nanoGPT comparison if you want to check)