We have noticed recent comparisons published by Unsloth that do not align with our results for DeepSeek-V4.1-Flash-1017. So just if it is not obvious: we did of course not modify any hyperparameters to favor our models. We used standard settings appropriate for a model of this scale (763B parameters) and its 1M context length capability: a 128K context window and a reasoning effort set to 50.

To the best of our knowledge, the discrepancy in performance stems from a difference in benchmark constraints - not in the model being worse. Unsloth's comparisons has a strict 32K token reasoning budget, whereas our benchmarks allow up to 128K output tokens. This significant difference in compute explains the variance in results; it is not due to any hidden adjustments on our part. Furthermore, imposing such a low reasoning budget (32K) appears inconsistent for a model designed with a 1M context window and 763B params.

Our primary goal is open collaboration and transparency. All of our code is available on GitHub, and our papers are on arXiv. We are genuinely happy that other quantizations are performing well, as this was a key motivation for publishing these models in the first place. We invite anyone with the compute resources to reproduce our results using our exact settings (128K output budget, reasoning effort 50) for independent verification. Our previous models were benchmarked by third parties like Benjamin Marie (The Kaitchup), who do great work for the community. We welcome rigorous testing of our claims. Furthermore, we are currently running the comparisons on the models within our setup and will post results soon.

Thanks and stay tuned!

submitted by /u/Loginhe [link] [comments]