FineWeb
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 3
- OpenEBFM releases code, weights and training stack
The OpenEBFM authors released the full training and evaluation stack, pretrained checkpoints, training recipes and an EBFM-based chatbot. In their experiments, EBFMs achieved lower validation bits per byte than matched GPT baselines during data scaling, with the gap generally increasing as the training-token budget grew. From roughly 268 million to 1 billion parameters, EBFMs improved faster than corresponding GPT baselines during parameter scaling. On DCLM, the largest EBFM reached an average zero-shot question-answering accuracy of 44.71%, compared with 39.79% for the corresponding GPT baseline in the authors’ setup. The authors cautioned that these self-reported results do not establish that energy-based models are universally better than autoregressive models, and noted that the experiments remain at a scale of about 1 billion parameters.
r/LocalLLaMA · 🔥 0 - Q Labs introduces Dust for Transformer pretraining without backpropagation
Q Labs introduced Dust, a zeroth-order optimization method that estimates update directions from random, token-level activation perturbations instead of standard backpropagation. The company reported training Transformers on FineWeb up to 243M parameters and 20M tokens, while a SOTAAZ Lab reproduction found basic reproducibility but no clear advantage over backpropagation. Dust also remained far more computationally expensive than backpropagation.
机器之心 · 🔥 0 - PyTorch's Zephon brings elastically deterministic data loading to TorchTitan
A PyTorch blog post introduces Zephon, a data loader for TorchTitan that aims to produce identical global batches even when the number of GPUs changes. In experiments training a 1B-parameter model from scratch on 20B tokens across 8 to 64 GPUs, a standard loader let DCLM Core v1 scores vary by up to 0.82 points and FineWeb scores by 0.64 points — roughly four times the thresholds those benchmarks' authors used — while Zephon's spread was 0.05 and 0.011 points respectively. Zephon tokenizes, packs, shuffles and mixes data on the fly inside the training job, using lanes, single-threaded accumulators and periodic flush sentinels to reproduce state, with checkpoints of a few MiB that can be resumed on a different number of data-parallel ranks; the authors say it keeps pace with Google Grain on a vision-language pipeline.
PyTorch Blog · 🔥 0
Experience and discussion from the community
Share my FineWeb experienceAsk about FineWeb
Nobody has shared their experience with FineWeb yet.