Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

FineWeb

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 3

  1. OpenEBFM releases code, weights and training stack

    The OpenEBFM authors released the full training and evaluation stack, pretrained checkpoints, training recipes and an EBFM-based chatbot. In their experiments, EBFMs achieved lower validation bits per byte than matched GPT baselines during data scaling, with the gap generally increasing as the training-token budget grew. From roughly 268 million to 1 billion parameters, EBFMs improved faster than corresponding GPT baselines during parameter scaling. On DCLM, the largest EBFM reached an average zero-shot question-answering accuracy of 44.71%, compared with 39.79% for the corresponding GPT baseline in the authors’ setup. The authors cautioned that these self-reported results do not establish that energy-based models are universally better than autoregressive models, and noted that the experiments remain at a scale of about 1 billion parameters.

    r/LocalLLaMA · 🔥 0
  2. Q Labs introduces Dust for Transformer pretraining without backpropagation

    Q Labs introduced Dust, a zeroth-order optimization method that estimates update directions from random, token-level activation perturbations instead of standard backpropagation. The company reported training Transformers on FineWeb up to 243M parameters and 20M tokens, while a SOTAAZ Lab reproduction found basic reproducibility but no clear advantage over backpropagation. Dust also remained far more computationally expensive than backpropagation.

    机器之心 · 🔥 0
  3. PyTorch's Zephon brings elastically deterministic data loading to TorchTitan

    A PyTorch blog post introduces Zephon, a data loader for TorchTitan that aims to produce identical global batches even when the number of GPUs changes. In experiments training a 1B-parameter model from scratch on 20B tokens across 8 to 64 GPUs, a standard loader let DCLM Core v1 scores vary by up to 0.82 points and FineWeb scores by 0.64 points — roughly four times the thresholds those benchmarks' authors used — while Zephon's spread was 0.05 and 0.011 points respectively. Zephon tokenizes, packs, shuffles and mixes data on the fly inside the training job, using lanes, single-threaded accumulators and periodic flush sentinels to reproduce state, with checkpoints of a few MiB that can be resumed on a different number of data-parallel ranks; the authors say it keeps pace with Google Grain on a vision-language pipeline.

    PyTorch Blog · 🔥 0

Experience and discussion from the community

Share my FineWeb experienceAsk about FineWeb

Nobody has shared their experience with FineWeb yet.