Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Q Labs introduces Dust for Transformer pretraining without backpropagation

AI summary

Q Labs introduced Dust, a zeroth-order optimization method that estimates update directions from random, token-level activation perturbations instead of standard backpropagation. The company reported training Transformers on FineWeb up to 243M parameters and 20M tokens, while a SOTAAZ Lab reproduction found basic reproducibility but no clear advantage over backpropagation. Dust also remained far more computationally expensive than backpropagation.

In-depth insight

Dust demonstrates that Transformers can be pretrained without standard backpropagation, offering a route to models containing non-differentiable modules or external programs, but the reported evidence does not establish a practical replacement for backpropagation.

原创 机器之心 2026-10-26 14:13 北京

对「苦涩的教训」的又一次致敬?

编辑|张倩

训练大模型,能不能不用反向传播?

#Q Labs#Dust#Transformer#FineWeb#EGGROLL#SOTAAZ Lab

Original · 11 min readRead the original
Group chat

No comments yet. Start the conversation.