Maxwell-V1-Flash brings 256K-context sparse attention to a 35B MoE
The Maxwell Team introduced Maxwell-V1-Flash, a Mixture-of-Experts language model with 35B total parameters, 3B active parameters per token, and support for contexts up to 256K tokens. It uses Maxwell Block Sparse Attention, which combines fixed prefix and local blocks with remotely selected blocks, while Manifold-Constrained Hyper-Connections widen information flow across residual streams. The developer says its MBSA kernel is 6.7 times faster than dense GQA at 256K context length, but no methodology or independent measurement was provided.
Maxwell-V1-Flash combines high long-context capacity with a sparsely activated architecture and attention mechanisms aimed at reducing inference cost, though its performance and speed claims still require independent validation.
| Model Introduction Maxwell-V1.0-Flash is a next-generation, high-performance Mixture-of-Experts (MoE) language model developed by the Maxwell Team, with 35B total parameters and 3B active parameters per token (both excluding embeddings) and support for context lengths up to 256K tokens. It is designed for strong reasoning, agentic, and long-context capabilities at a practical inference cost. Key Features Pre-training. Approximately 26T tokens of multilingual data, with the Muon optimizer on most hidden weight matrices. Training starts dense; block-sparse attention enters at 128K mid-training, once the indexer is warmed up against dense full-history attention, then trains jointly with the backbone. Tokenizer and Long Context. A script-aware multilingual tokenizer stays compact across languages, especially low-resource Southeast Asian ones such as Thai, Lao, Myanmar, Khmer, and Shan. Long-context training then extends the sequence length in stages from 4K to 256K. Post-training. Two-stage all-domain SFT builds general, reasoning, and agentic capabilities, then targets harder reasoning problems and agent trajectories. Independent RL branches cover general tasks, verifiable reasoning (mathematics, science, logic, code), and agentic tasks (search, terminal use, function calling, software engineering). Multi-teacher on-policy distillation then merges them into one model by supervising student-generated responses. submitted by /u/pmttyji [link] [comments] |
#Maxwell-V1-Flash#Maxwell Team#Maxwell Block Sparse Attention#Manifold-Constrained Hyper-Connections