Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

ConeML 1.2B experimental MoE opens for reproduction

AI summary

Solo developer Significant-Serve-58 released the ConeML 1.2B experimental sparse mixture-of-experts model, its GGUF quantization, and an evaluation set. The model has 1.17B stored parameters, activates about 348M per token, and was trained from scratch on roughly 11B tokens; in the developer’s local test, it answered 438 of 512 shuffled fact-chain questions at depth 5, or 85.5%. The developer explicitly said these exploratory results are not controlled training comparisons and that the model is not a general assistant.

In-depth insight

The release provides weights and evaluation materials for independent testing of an unusual small sparse MoE, but its narrow benchmark claim and weak results on interior pairwise comparisons, HumanEval, and multi-turn follow-ups limit what the reported score demonstrates.

I've been developing a small sparse MoE independently, and I've got a result I'd like strangers to attack.

The model has 1.17B stored parameters, ~348M active per token, trained from scratch on approximately 11B tokens including instruction tuning. For context, Qwen2.5 reports up to 18T training tokens across its family, while SmolLM2-1.7B reports 11T. These aren't controlled training comparisons.

It's not a general assistant, and I'm not claiming it is.

THE TASK

A chain of relational facts ("A is taller than B", "B is taller than C", etc.), with the facts shuffled and the model asked to identify an endpoint.

Depth is chain length. Each result is out of 512 questions across four entity families. All models received the same questions, using their native chat templates, greedy decoding, and a 16-token answer budget.

#ConeML 1.2B experimental#Qwen2.5-1.5B#SmolLM2-1.7B#Llama-3.2-1B#Gemma-3-1B

Original · 2 min readRead the original
Group chat

No comments yet. Start the conversation.