创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

论文提出基于Grassmann流的无注意力序列模型

AI 总结

一篇arXiv论文提出基于Grassmann流的无注意力序列架构。其Causal Grassmann层先线性压缩词元状态,再通过Plücker坐标将局部词元对编码为Grassmann流形上的二维子空间,最后使用门控混合将这些几何特征融合回隐藏状态。 作者报告称,1300万至1800万参数的纯Grassmann模型在WikiText-2验证集上的困惑度与同规模Transformer相差约10%至15%。在SNLI任务中,基于DistilBERT的Grassmann-Plücker头取得最高验证与测试准确率0.8550和0.8538,而Transformer头分别为0.8545和0.8511。所给报告未包含对这些实验的独立复现。

深度见解

该论文为显式自注意力提供了一种结构化替代方案,并报告了具有竞争力的语言建模与自然语言推理结果。论文还分析了固定秩Grassmann混合随序列长度线性扩展的特性。

View PDFHTML (experimental)

Abstract:We revisit a basic question in sequence modeling: is explicit self-attention actually necessary for strong performance and reasoning? We argue that standard multi-head attention is best seen as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining this lifted tensor through gradient descent. This mechanism is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants. To explore an alternative, we propose an attention-free architecture based on Grassmann flows. Instead of forming an L by L attention matrix, our Causal Grassmann layer (i) linearly reduces token states, (ii) encodes local token pairs as two-dimensional subspaces on a Grassmann manifold via Plucker coordinates, and (iii) fuses these geometric features back into the hidden states through gated mixing. Information therefore propagates by controlled deformations of low-rank subspaces over multi-scale local windows, so the core computation lives on a finite-dimensional manifold rather than in an unstructured tensor space. On the Wikitext-2 language modeling benchmark, purely Grassmann-based models with 13 to 18 million parameters achieve validation perplexities within about 10 to 15 percent of size-matched Transformers. On the SNLI natural language inference task, a Grassmann-Plucker head on top of DistilBERT slightly outperforms a Transformer head, with best validation and test accuracies of 0.8550 and 0.8538 compared to 0.8545 and 0.8511. We analyze the complexity of Grassmann mixing, show linear scaling in sequence length for fixed rank, and argue that such manifold-based designs offer a more structured route toward geometric and invariant-based interpretations of neural reasoning.

Subjects:Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Algebraic Geometry (math.AG) Cite as:arXiv:2512.19428 [cs.LG] (or arXiv:2512.19428v1 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2512.19428

arXiv-issued DOI via DataCite

Submission history

From: Chong Zhang [view email] [v1] Mon, 22 Dec 2025 14:29:18 UTC (16 KB)

#Grassmann flow#Causal Grassmann layer#Grassmann-Plücker head#DistilBERT#Transformer

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。