Causal Grassmann layer
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- Paper proposes attention-free sequence modeling with Grassmann flows
An arXiv paper proposes an attention-free sequence architecture based on Grassmann flows. Its Causal Grassmann layer linearly reduces token states, encodes local token pairs as two-dimensional subspaces on a Grassmann manifold through Plücker coordinates, and fuses the geometric features back into hidden states through gated mixing. The authors report that pure Grassmann models with 13 million to 18 million parameters achieve WikiText-2 validation perplexities within about 10% to 15% of size-matched Transformers. On SNLI, a Grassmann-Plücker head on DistilBERT achieves reported best validation and test accuracies of 0.8550 and 0.8538, compared with 0.8545 and 0.8511 for a Transformer head. The supplied report does not include an independent reproduction of these experiments.
Hacker News 首页(100+ 分) · 🔥 0
Experience and discussion from the community
Share my Causal Grassmann layer experienceAsk about Causal Grassmann layer
Nobody has shared their experience with Causal Grassmann layer yet.