ISTA disputes Unsloth's DeepSeek-V4.1-Flash-1017 benchmark comparison
ISTA said its DeepSeek-V4.1-Flash-1017 benchmarks used a 128K context window, reasoning effort of 50 and an output budget of up to 128K tokens. It attributes the discrepancy with Unsloth's published comparisons to different constraints, saying Unsloth imposed a strict 32K reasoning budget, and denies using hidden hyperparameter adjustments to favor its models. ISTA said its code is available on GitHub, its papers are on arXiv and additional comparisons were underway. It invited independent reproduction using its stated settings.
ISTA's explanation makes the performance discrepancy testable, but it remains an unverified claim. The differing output and reasoning constraints must be accounted for when interpreting the benchmark results.
We have noticed recent comparisons published by Unsloth that do not align with our results for DeepSeek-V4.1-Flash-1017. So just if it is not obvious: we did of course not modify any hyperparameters to favor our models. We used standard settings appropriate for a model of this scale (763B parameters) and its 1M context length capability: a 128K context window and a reasoning effort set to 50.
To the best of our knowledge, the discrepancy in performance stems from a difference in benchmark constraints - not in the model being worse. Unsloth's comparisons has a strict 32K token reasoning budget, whereas our benchmarks allow up to 128K output tokens. This significant difference in compute explains the variance in results; it is not due to any hidden adjustments on our part. Furthermore, imposing such a low reasoning budget (32K) appears inconsistent for a model designed with a 1M context window and 763B params.
Our primary goal is open collaboration and transparency. All of our code is available on GitHub, and our papers are on arXiv. We are genuinely happy that other quantizations are performing well, as this was a key motivation for publishing these models in the first place. We invite anyone with the compute resources to reproduce our results using our exact settings (128K output budget, reasoning effort 50) for independent verification. Our previous models were benchmarked by third parties like Benjamin Marie (The Kaitchup), who do great work for the community. We welcome rigorous testing of our claims. Furthermore, we are currently running the comparisons on the models within our setup and will post results soon.
Thanks and stay tuned!
submitted by /u/Loginhe [link] [comments]