Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Unsloth DeepSeek GGUF Benchmark Results Face Scrutiny

AI summary

Unsloth released new DeepSeek-V4.1-Flash-1017 GGUF builds and reported that its Dynamic V3.1 quantizations led across several benchmarks at 512K context. The 1-bit build scored 68.1 on DeepSWE and 72.2 on Terminal-Bench. A r/LocalLLaMA analysis questioned why all four Unsloth quantizations and the Q8 baseline improved by roughly 2–3 percentage points on LiveCodeBench v6. Unsloth said the earlier chart represented one run, followed by additional evaluations under unchanged settings. The analysis depends on statistical assumptions and has not independently reproduced Unsloth's results; the reports did not provide the underlying raw data.

In-depth insight

The release provides locally runnable quantized DeepSeek builds for machines with 64GB or more RAM. However, the unresolved questions about the LiveCodeBench evaluation mean Unsloth's claimed overall benchmark leadership has not been independently verified.

We benchmarked DeepSeek-V4.1-Flash-1017 GGUFs on Terminal Bench, DeepSWE & more on 512K context. Unsloth ranks highest overall.

1-bit Unsloth GGUF scores 68.1 on DeepSWE, 72.2 on Terminal-Bench, 52.7 on AutomationBench. >10% ahead of other quants.

GGUF: https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF https://x.com/UnslothAI/status/2117266365929365525/photo/1 https://twitter.com/UnslothAI/status/2116156587404562491

@UnslothAI: We’re releasing new DeepSeek-V4.1-Flash-1017 GGUFs, ranking highest across benchmarks. Dynamic V3.1 leads across nearly all benchmarks: GPQA Diamond, AIME25, LiveCodeBench v6, MMLU-Pro, Aider & more. Run on 64GB+ RAM. GGUF: https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF Blog: https://unsloth.ai/docs/models/deepseek-v4.1-flash https://x.com/UnslothAI/status/2116156587404562491/photo/1 https://twitter.com/deepseek_ai/status/2111375640717033777 https://x.com/UnslothAI/status/2116156587404562491

#Unsloth#DeepSeek-V4.1-Flash-1017#GSQ-RCO#DeepSWE#Terminal-Bench#AutomationBench#LiveCodeBench v6

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.