TPHC Combines Tensor Parallelism With Hyper-Connections Streams
A new arXiv paper proposes TPHC, which combines tensor-parallel shards with Hyper-Connections streams. It routes a separate mixture of residual streams to each shard and keeps the shard outputs separate during write-back, allowing each stream to receive a different mixture. The authors report that the design adds no parameters or FLOPs to the attention or FFN sublayers. On a 1.1B model trained with 100B tokens, TPHC reduced validation loss by 0.7–1.0% and raised average zero-shot accuracy across six commonsense reasoning benchmarks by 0.3–0.7 percentage points versus manifold-constrained Hyper-Connections at matched stream counts.
The paper reports that TPHC makes residual-stream updates more expressive without adding parameters or FLOPs to the attention or FFN sublayers, offering a way to improve Hyper-Connections under tensor parallelism.
Computer Science > Computation and Language
arXiv:2611.23026 (cs)
[Submitted on 17 Nov 2026]
Title:TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates
Authors:Xinying Hu, You Wu, Haoyi Wu, Jianchao Tan, Kefeng Zhang, Yuchen Xie, Kewei Tu
View a PDF of the paper titled TPHC: Improving Hyper-Connections with Tensor-Parallel Grouped Updates, by Xinying Hu and 6 other authors
#TPHC#Hyper-Connections#manifold-constrained Hyper-Connections