Maxwell-V1-Flash发布:35B稀疏模型支持256K上下文
AI 总结
Maxwell Team推出Maxwell-V1-Flash,这是一款总参数35B、每Token激活3B的MoE语言模型,支持最长256K上下文。模型采用Maxwell Block Sparse Attention,将固定前缀块、本地块和学习索引器选择的远程块结合,并通过Manifold-Constrained Hyper-Connections增强残差流间的信息流动。开发方称其MBSA内核在256K上下文下比密集GQA快6.7倍,但未提供测试方法或独立测量数据。
深度见解
Maxwell-V1-Flash将长上下文支持、稀疏激活和降低推理成本的技术结合在一起,但其实际性能与速度主张仍需独立验证。
| Model Introduction Maxwell-V1.0-Flash is a next-generation, high-performance Mixture-of-Experts (MoE) language model developed by the Maxwell Team, with 35B total parameters and 3B active parameters per token (both excluding embeddings) and support for context lengths up to 256K tokens. It is designed for strong reasoning, agentic, and long-context capabilities at a practical inference cost. Key Features Pre-training. Approximately 26T tokens of multilingual data, with the Muon optimizer on most hidden weight matrices. Training starts dense; block-sparse attention enters at 128K mid-training, once the indexer is warmed up against dense full-history attention, then trains jointly with the backbone. Tokenizer and Long Context. A script-aware multilingual tokenizer stays compact across languages, especially low-resource Southeast Asian ones such as Thai, Lao, Myanmar, Khmer, and Shan. Long-context training then extends the sequence length in stages from 4K to 256K. Post-training. Two-stage all-domain SFT builds general, reasoning, and agentic capabilities, then targets harder reasoning problems and agent trajectories. Independent RL branches cover general tasks, verifiable reasoning (mathematics, science, logic, code), and agentic tasks (search, terminal use, function calling, software engineering). Multi-teacher on-policy distillation then merges them into one model by supervising student-generated responses. submitted by /u/pmttyji [link] [comments] |
#Maxwell-V1-Flash#Maxwell Team#Maxwell Block Sparse Attention#Manifold-Constrained Hyper-Connections