Tsinghua open-sources Puro-2B: 2B pretraining on RTX 5090s for about $4,400
A Tsinghua University team led by professors Chen Wenguang and Lyu Kaifeng released Puro-2B, a fully open pretraining recipe that publishes the training data, framework, configuration and model weights. The roughly 2B-parameter model was trained from scratch on a cluster of consumer RTX 5090 GPUs using up to 1.4 trillion tokens; on a 15-task evaluation, a version with about $4,400 of pretraining compute beat Qwen2-1.5B on average, while the $6,900 best version came close to Qwen2.5-1.5B on average. The team also fit a Puro Cost Scaling Law to estimate achievable quality at a given budget, combining hardware choice, FP8 training, the MuonH optimizer, Curriculum Model Averaging and data-slice evaluation.
It open-sources a full small-model pretraining recipe plus a cost curve, letting budget-constrained labs reproduce and study pretraining methods themselves.