Unsloth的DeepSeek GGUF基准成绩引发质疑
Unsloth发布了新版DeepSeek-V4.1-Flash-1017 GGUF,并称其Dynamic V3.1量化版本在512K上下文下多项基准中领先;其中1-bit版本在DeepSWE和Terminal-Bench上分别取得68.1和72.2。r/LocalLLaMA的一份分析质疑,四种Unsloth量化版本及Q8基线为何在LiveCodeBench v6上同步提升约2至3个百分点。Unsloth称早期图表来自一次运行,之后在设置不变的情况下进行了更多评测。该分析依赖统计假设,尚未独立复现Unsloth的结果;报道也未提供底层原始数据。
We benchmarked DeepSeek-V4.1-Flash-1017 GGUFs on Terminal Bench, DeepSWE & more on 512K context. Unsloth ranks highest overall.
1-bit Unsloth GGUF scores 68.1 on DeepSWE, 72.2 on Terminal-Bench, 52.7 on AutomationBench. >10% ahead of other quants.
GGUF: https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF https://x.com/UnslothAI/status/2117266365929365525/photo/1 https://twitter.com/UnslothAI/status/2116156587404562491

@UnslothAI: We’re releasing new DeepSeek-V4.1-Flash-1017 GGUFs, ranking highest across benchmarks. Dynamic V3.1 leads across nearly all benchmarks: GPQA Diamond, AIME25, LiveCodeBench v6, MMLU-Pro, Aider & more. Run on 64GB+ RAM. GGUF: https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF Blog: https://unsloth.ai/docs/models/deepseek-v4.1-flash https://x.com/UnslothAI/status/2116156587404562491/photo/1 https://twitter.com/deepseek_ai/status/2111375640717033777 https://x.com/UnslothAI/status/2116156587404562491
#Unsloth#DeepSeek-V4.1-Flash-1017#GSQ-RCO#DeepSWE#Terminal-Bench#AutomationBench#LiveCodeBench v6