Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

TPHC Combines Tensor Parallelism With Hyper-Connections Streams

AI summary

A new arXiv paper proposes TPHC, which combines tensor-parallel shards with Hyper-Connections streams. It routes a separate mixture of residual streams to each shard and keeps the shard outputs separate during write-back, allowing each stream to receive a different mixture. The authors report that the design adds no parameters or FLOPs to the attention or FFN sublayers. On a 1.1B model trained with 100B tokens, TPHC reduced validation loss by 0.7–1.0% and raised average zero-shot accuracy across six commonsense reasoning benchmarks by 0.3–0.7 percentage points versus manifold-constrained Hyper-Connections at matched stream counts.

In-depth insight

The paper reports that TPHC makes residual-stream updates more expressive without adding parameters or FLOPs to the attention or FFN sublayers, offering a way to improve Hyper-Connections under tensor parallelism.

Computer Science > Computation and Language

arXiv:2611.23026 (cs)

[Submitted on 17 Nov 2026]

Title:TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates

Authors:Xinying Hu, You Wu, Haoyi Wu, Jianchao Tan, Kefeng Zhang, Yuchen Xie, Kewei Tu

View a PDF of the paper titled TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates, by Xinying Hu and 6 other authors

#TPHC#Hyper-Connections#manifold-constrained Hyper-Connections

Original · 3 min readRead the original
Group chat

No comments yet. Start the conversation.