ConeML 1.2B实验MoE开放复现
独立开发者Significant-Serve-58发布了ConeML 1.2B实验稀疏混合专家模型、GGUF量化版本及评测集。该模型有11.7亿个存储参数,每个token激活约3.48亿参数,从头训练约110亿个token。在开发者的本地测试中,模型在深度5的打乱事实链任务中答对512题中的438题,得分为85.5%。开发者强调,这些探索性结果并非受控训练对比,该模型也不是通用助手。
此次发布提供了权重和评测材料,便于独立测试一种小型稀疏MoE,但其结论范围较窄,而且模型在链中成对关系比较、HumanEval和多轮追问上的表现较弱。
I've been developing a small sparse MoE independently, and I've got a result I'd like strangers to attack.
The model has 1.17B stored parameters, ~348M active per token, trained from scratch on approximately 11B tokens including instruction tuning. For context, Qwen2.5 reports up to 18T training tokens across its family, while SmolLM2-1.7B reports 11T. These aren't controlled training comparisons.
It's not a general assistant, and I'm not claiming it is.
THE TASK
A chain of relational facts ("A is taller than B", "B is taller than C", etc.), with the facts shuffled and the model asked to identify an endpoint.
Depth is chain length. Each result is out of 512 questions across four entity families. All models received the same questions, using their native chat templates, greedy decoding, and a 16-token answer budget.
#ConeML 1.2B experimental#Qwen2.5-1.5B#SmolLM2-1.7B#Llama-3.2-1B#Gemma-3-1B