FineWeb-Edu
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 2
- Researcher shares PHSMLP and Pilot neural architecture experiments
Reddit user /u/WAMFT shared code and results for two small neural-network architecture experiments: PHSMLP, which reuses one MLP across Transformer layers with learned head pooling, and Pilot, which combines FFT-computed input history with nonlinear recurrent matrix memory. In the reported tests, shared pooled PHSMLP used 13.4% fewer parameters, while parameter-matched Pilot reported 16.7% lower test perplexity than a four-head Transformer, though Pilot performed worse on memory probes and was about 60 times slower to train.
r/MachineLearning · 🔥 0 - DeepMem Debuts Multiscale Byte-Level Language Modeling Architecture
Afshin Khadangi's arXiv paper introduces DeepMem, a causal multiscale architecture that separates semantic compression from exact byte preservation. It combines a learned invertible subband transform, a lossless carrier, explicit detail sidebands, fast weight memory blocks, and an attention-based super-resolution decoder. The paper reports stronger language modeling than a matched byte-level Transformer on FineWeb-Edu and PG19, while claiming approximately 7,600 times fewer counted FLOPs for bottleneck refinement; these results have not been independently verified.
Hugging Face · Papers · 🔥 0
Experience and discussion from the community
Share my FineWeb-Edu experienceAsk about FineWeb-Edu
Nobody has shared their experience with FineWeb-Edu yet.