Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

DistilBERT

AI overview

Sign in and the AI will write an overview from our coverage.

Headlines · 1

  1. Paper proposes attention-free sequence modeling with Grassmann flows

    An arXiv paper proposes an attention-free sequence architecture based on Grassmann flows. Its Causal Grassmann layer linearly reduces token states, encodes local token pairs as two-dimensional subspaces on a Grassmann manifold through Plücker coordinates, and fuses the geometric features back into hidden states through gated mixing. The authors report that pure Grassmann models with 13 million to 18 million parameters achieve WikiText-2 validation perplexities within about 10% to 15% of size-matched Transformers. On SNLI, a Grassmann-Plücker head on DistilBERT achieves reported best validation and test accuracies of 0.8550 and 0.8538, compared with 0.8545 and 0.8511 for a Transformer head. The supplied report does not include an independent reproduction of these experiments.

    Hacker News 首页(100+ 分) · 🔥 0

Experience and discussion from the community

Share my DistilBERT experienceAsk about DistilBERT

Nobody has shared their experience with DistilBERT yet.