Oumi调查Jev可能的架构与训练方式
AI 总结
Oumi研究人员利用行为探测、基准测试结果和服务特征，推断TypeSafe的Jev可能基于哪个模型，以及可能如何训练和部署。在没有内部消息的情况下，他们认为Jev很可能基于Qwen 3.5或3.6，并采用35B-A3B配置，其分词器可能是Qwen与GPT-OSS的合并版本。他们还推测，RLCD可能使用合成数据上的软标签交叉熵，而非PPO或GRPO。在服务方面，他们提出可能采用树批处理，以及某种形式的稀疏注意力，或部署在两块GPU上并拆分预填充任务。
深度见解
这项分析为识别本地决策模型的具体组成提供了一种公开的实验方法。不过，Oumi明确表示，其关于基础模型、架构、分词器、训练和服务的结论均为基于测试结果的最佳猜测，而非TypeSafe确认的细节。
| Full disclosure: I'm one of the authors of this blog, and work at Oumi. I thought folks here would find our investigation interesting, especially anyone interested in building local decision models. We ran several experiments to investigate which model Jev may be based on and how it might be trained and served. The TypeSafe team has publicly mentioned that they are not interested in pretraining a model from scratch, so Jev is almost certainly built from a recently-released publicly available model -- we were curious if we could identify exactly which one! We don't have any insider knowledge, so these are our best guesses based on the results: - Jev is likely built on Qwen 3.5 or 3.6, based on obscure / famous knowledge comparisons, name probing, and sensitive question responses - Jev is likely a 35B-A3B model, based on the benchmark performance and serving profiles we ran (see in particular the Rare Facts analysis and curves) - Its tokenizer is likely a merge of Qwen and GPT-OSS - Its training methodology (RLCD) is likely standard soft-label cross entropy from synthetic data. It's likely being framed as RL because of a parallel with RLHF (for which the human feedback part was the unique part), but it likely isn't using PPO or GRPO or anything like that. - This synthetic data is a mix of teacher-taught from reasoning models and programmatic answers to code-generated questions. - Serving is likely done with tree batching (pack all questions + 1 state into single request with per-question attention mask) rather than the standard serving path on vLLM - Based on long document performance, they're likely using a form of sparse attention or serving on two GPUs and splitting the prefill between them. Most likely on B200 GPUs or above. Our nitty gritty analysis is in the article for those curious. Hope you folks enjoy! submitted by /u/Taenin [link] [comments] |
#Jev#TypeSafe#Oumi#Qwen 3.5#Qwen 3.6#GPT-OSS#RLCD#vLLM#NVIDIA B200