Manifold-Constrained Hyper-Connections
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 3
- Maxwell-V1-Flash brings 256K-context sparse attention to a 35B MoE
The Maxwell Team introduced Maxwell-V1-Flash, a Mixture-of-Experts language model with 35B total parameters, 3B active parameters per token, and support for contexts up to 256K tokens. It uses Maxwell Block Sparse Attention, which combines fixed prefix and local blocks with remotely selected blocks, while Manifold-Constrained Hyper-Connections widen information flow across residual streams. The developer says its MBSA kernel is 6.7 times faster than dense GQA at 256K context length, but no methodology or independent measurement was provided.
r/LocalLLaMA · 🔥 0 - TPHC Combines Tensor Parallelism With Hyper-Connections Streams
A new arXiv paper proposes TPHC, which combines tensor-parallel shards with Hyper-Connections streams. It routes a separate mixture of residual streams to each shard and keeps the shard outputs separate during write-back, allowing each stream to receive a different mixture. The authors report that the design adds no parameters or FLOPs to the attention or FFN sublayers. On a 1.1B model trained with 100B tokens, TPHC reduced validation loss by 0.7–1.0% and raised average zero-shot accuracy across six commonsense reasoning benchmarks by 0.3–0.7 percentage points versus manifold-constrained Hyper-Connections at matched stream counts.
Hugging Face · Papers · 🔥 0 - CPU-only replication examines Kimi K3 Attention Residuals
The second installment of a CPU-only Attention Residuals replication associated with Kimi K3 examines the scoring kernel, Full and Block variants, and the AttnResState lifecycle. It also introduces a two-stage online-softmax algorithm and explains production-oriented SGLang fused-kernel concepts. Components without open-source versions are replaced by approximations, and the paper's 25-configuration architecture experiment was not reproduced.
掘金 · 人工智能 · 🔥 0
Experience and discussion from the community
Share my Manifold-Constrained Hyper-Connections experienceAsk about Manifold-Constrained Hyper-Connections
Nobody has shared their experience with Manifold-Constrained Hyper-Connections yet.