创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

ConeML 1.2B实验MoE开放复现

AI 总结

独立开发者Significant-Serve-58发布了ConeML 1.2B实验稀疏混合专家模型、GGUF量化版本及评测集。该模型有11.7亿个存储参数,每个token激活约3.48亿参数,从头训练约110亿个token。在开发者的本地测试中,模型在深度5的打乱事实链任务中答对512题中的438题,得分为85.5%。开发者强调,这些探索性结果并非受控训练对比,该模型也不是通用助手。

深度见解

此次发布提供了权重和评测材料,便于独立测试一种小型稀疏MoE,但其结论范围较窄,而且模型在链中成对关系比较、HumanEval和多轮追问上的表现较弱。

I've been developing a small sparse MoE independently, and I've got a result I'd like strangers to attack.

The model has 1.17B stored parameters, ~348M active per token, trained from scratch on approximately 11B tokens including instruction tuning. For context, Qwen2.5 reports up to 18T training tokens across its family, while SmolLM2-1.7B reports 11T. These aren't controlled training comparisons.

It's not a general assistant, and I'm not claiming it is.

THE TASK

A chain of relational facts ("A is taller than B", "B is taller than C", etc.), with the facts shuffled and the model asked to identify an endpoint.

Depth is chain length. Each result is out of 512 questions across four entity families. All models received the same questions, using their native chat templates, greedy decoding, and a 16-token answer budget.

#ConeML 1.2B experimental#Qwen2.5-1.5B#SmolLM2-1.7B#Llama-3.2-1B#Gemma-3-1B

原文约 2 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。