RLT：回环 Transformer 提升长序列泛化

Hugging Face · Papers10-05 20:00Yifan Zhang, Jichen Feng, Shihan Qin

AI 总结

该论文提出 Recurrent Looped Transformer（RLT），把 8 层拆分为并行因果编码器与循环解码器，每个 token 的解码器会把编码器输出与上一 token 的最终解码器状态合并，使计算路径随序列长度增长而单 token 成本固定。在六个算法任务上，仅用最多 40 比特训练的两种 RLT 分层方案把 parity 泛化到 256 比特并在所有随机种子下达到 100% 准确率，而八层普通 Transformer 始终停留在随机水平；S5 置换追踪在 8 倍训练长度下达到 97%（基线不足 1%），模运算最高达 93%（基线 33%）。消融显示收益依赖反馈：去掉反馈后 parity 与 S5 全部跌回随机水平；把反馈改为每 4 个 token 更新一次可保持 64 比特 parity 99%，但长度 64 的 S5 从 100% 降至 20%。

为什么重要：它给 Transformer 的深度—序列长度权衡提供了一个简单可行的架构思路，但证据目前只来自小规模合成算法任务。

Recurrent Looped Transformer

来源原文Hugging Face · Papers · 约 3 分钟读完

Computer Science > Computation and Language

arXiv:2610.07591 (cs)

[Submitted on 6 Oct 2026]

Title:Recurrent Looped Transformer

Authors:Yifan Zhang, Jichen Feng, Shihan Qin

View a PDF of the paper titled Recurrent Looped Transformer, by Yifan Zhang and 2 other authors

View PDF HTML (experimental)

Abstract:State tracking requires an update at every input, but the depth a Transformer applies to each token is fixed regardless of sequence length. We introduce the Recurrent Looped Transformer (RLT), which splits its layers between a parallel causal encoder and a recurrent decoder. At each token, the decoder merges the encoder output with the previous token's final decoder state, so the computation path grows with sequence length at a fixed per-token cost. On six algorithmic tasks, we compare five splits of eight layers with an eight-layer Transformer over three seeds. Trained on at most 40 bits, two RLT splits generalize parity to 256 bits with 100% accuracy in every seed, while the Transformer stays at chance. On swap-based $S_5$ permutation tracking at eight times the training length, RLT reaches 97% final-state accuracy versus under 1% for the Transformer, and accuracy increases with decoder depth. On modular arithmetic beyond the training lengths, RLT reaches up to 93% versus 33% for the Transformer. Ablations show that these gains depend on the feedback: removing it drops parity and swap-based $S_5$ to chance at every split. Updating the feedback once per four-token chunk lets known tokens in a chunk run in parallel and keeps 64-bit parity at 99%, while permutation tracking depends on per-token feedback: chunking lowers length-64 swap-based $S_5$ from 100% to 20%.

Comments:	Project Page: this https URL
Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Cite as:	arXiv:2610.07591 [cs.CL]
	(or arXiv:2610.07591v1 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2610.07591 arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Yifan Zhang [view email]
[v1] Tue, 6 Oct 2026 01:29:46 UTC (1,042 KB)

Full-text links:

Access Paper:

View a PDF of the paper titled Recurrent Looped Transformer, by Yifan Zhang and 2 other authors

View PDF
HTML (experimental)
TeX Source

view license

Additional Features

Audio Summary

Current browse context:

cs.CL

< prev | next >

new | recent | 2026-10

Change to browse by:

cs
cs.AI
cs.LG

References & Citations

BibTeX formatted citation

Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle

Bibliographic Explorer (What is the Explorer?)

Connected Papers Toggle

Connected Papers (What is Connected Papers?)

Litmaps Toggle

Litmaps (What is Litmaps?)

scite.ai Toggle

scite Smart Citations (What are Smart Citations?)

Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle

alphaXiv (What is alphaXiv?)

Links to Code Toggle

CatalyzeX Code Finder for Papers (What is CatalyzeX?)

DagsHub Toggle

DagsHub (What is DagsHub?)

GotitPub Toggle

Gotit.pub (What is GotitPub?)

Huggingface Toggle

Hugging Face (What is Huggingface?)

ScienceCast Toggle

ScienceCast (What is ScienceCast?)

Demos

Replicate Toggle

Replicate (What is Replicate?)

Spaces Toggle

Hugging Face Spaces (What is Spaces?)

Spaces Toggle

TXYZ.AI (What is TXYZ.AI?)

Recommenders and Search Tools

Link to Influence Flower

Influence Flower (What are Influence Flowers?)

Core recommender toggle

CORE Recommender (What is CORE?)

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)

阅读原始来源 →

登录 ReadmeX

资讯