Q Labs introduces Dust for Transformer pretraining without backpropagation
Q Labs introduced Dust, a zeroth-order optimization method that estimates update directions from random, token-level activation perturbations instead of standard backpropagation. The company reported training Transformers on FineWeb up to 243M parameters and 20M tokens, while a SOTAAZ Lab reproduction found basic reproducibility but no clear advantage over backpropagation. Dust also remained far more computationally expensive than backpropagation.
Dust demonstrates that Transformers can be pretrained without standard backpropagation, offering a route to models containing non-differentiable modules or external programs, but the reported evidence does not establish a practical replacement for backpropagation.
原创 机器之心 2026-10-26 14:13 北京
对「苦涩的教训」的又一次致敬?
编辑|张倩
训练大模型,能不能不用反向传播?