Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Researcher shares PHSMLP and Pilot neural architecture experiments

AI summary

Reddit user /u/WAMFT shared code and results for two small neural-network architecture experiments: PHSMLP, which reuses one MLP across Transformer layers with learned head pooling, and Pilot, which combines FFT-computed input history with nonlinear recurrent matrix memory. In the reported tests, shared pooled PHSMLP used 13.4% fewer parameters, while parameter-matched Pilot reported 16.7% lower test perplexity than a four-head Transformer, though Pilot performed worse on memory probes and was about 60 times slower to train.

In-depth insight

The experiments provide code-backed evidence for parameter-efficient Transformer sharing and FFT-compatible recurrent memory, but their single-seed, small-scale results and weak memory-task performance make them exploratory rather than established improvements.

#PHSMLP#Pilot#FineWeb-Edu#Transformer#FFT#T4

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.