ConeML 1.2B experimental MoE opens for reproduction
Solo developer Significant-Serve-58 released the ConeML 1.2B experimental sparse mixture-of-experts model, its GGUF quantization, and an evaluation set. The model has 1.17B stored parameters, activates about 348M per token, and was trained from scratch on roughly 11B tokens; in the developer’s local test, it answered 438 of 512 shuffled fact-chain questions at depth 5, or 85.5%. The developer explicitly said these exploratory results are not controlled training comparisons and that the model is not a general assistant.
The release provides weights and evaluation materials for independent testing of an unusual small sparse MoE, but its narrow benchmark claim and weak results on interior pairwise comparisons, HumanEval, and multi-turn follow-ups limit what the reported score demonstrates.
I've been developing a small sparse MoE independently, and I've got a result I'd like strangers to attack.
The model has 1.17B stored parameters, ~348M active per token, trained from scratch on approximately 11B tokens including instruction tuning. For context, Qwen2.5 reports up to 18T training tokens across its family, while SmolLM2-1.7B reports 11T. These aren't controlled training comparisons.
It's not a general assistant, and I'm not claiming it is.
THE TASK
A chain of relational facts ("A is taller than B", "B is taller than C", etc.), with the facts shuffled and the model asked to identify an endpoint.
Depth is chain length. Each result is out of 512 questions across four entity families. All models received the same questions, using their native chat templates, greedy decoding, and a 16-token answer budget.
#ConeML 1.2B experimental#Qwen2.5-1.5B#SmolLM2-1.7B#Llama-3.2-1B#Gemma-3-1B