Hyper-Connections
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 1
- TPHC Combines Tensor Parallelism With Hyper-Connections Streams
A new arXiv paper proposes TPHC, which combines tensor-parallel shards with Hyper-Connections streams. It routes a separate mixture of residual streams to each shard and keeps the shard outputs separate during write-back, allowing each stream to receive a different mixture. The authors report that the design adds no parameters or FLOPs to the attention or FFN sublayers. On a 1.1B model trained with 100B tokens, TPHC reduced validation loss by 0.7–1.0% and raised average zero-shot accuracy across six commonsense reasoning benchmarks by 0.3–0.7 percentage points versus manifold-constrained Hyper-Connections at matched stream counts.
Hugging Face · Papers · 🔥 0
Experience and discussion from the community
Share my Hyper-Connections experienceAsk about Hyper-Connections
Nobody has shared their experience with Hyper-Connections yet.