Community test compares 2.25bpw DeepSeek-V4.1-Flash with official API
A r/LocalLLaMA author compared ISTA-DASLab's DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF 2.25bpw quantization with the official DeepSeek API on a 100-question, text-only subset of Humanities Last Exam. Among 74 questions the API completed, semantic accuracy was 19/74 for the local quantization and 21/74 for the API; the author called this a preliminary smoke test because differing token limits, non-completion, and substantial run-to-run variance limited the comparison.
The community test provides practical evidence that the 2.25bpw quantization remains usable, while its documented limitations mean it cannot establish how closely the quantized model matches the full model.
I've been running ISTA-DASLab's 2.25bpw quant of DSv4.1-1017 for a few days, and it's holding up well. But the fact that it's only 2.25bpw average did make me wonder how good the quant actually is against the full model.
Local setup: Project-maya v1.1.27. ISTA-DASLab's 2.25bpw quant. Quality-relevant settings: "DS4_INDEX_TC": "1". "MAYA_DS4_PROMPT_SKIP": "2".
Test: 100 question text-only subset of Humanities Last Exam.
Benchmarking models isn't easy, and these results are nothing but a smoke test. There are lots of issues with these results:
1) I ran the questions with a 32768 token limit per question, because anything larger becomes prohibitively time consuming. It already took my local instance around 12 hours to complete the test, and that's at an average of 40.4 toks/s.
2) When approaching the token limit, Deepseek's API and project-maya work differently. Project-maya will reduce thinking_budget to around 3/4 of the token limit so it always gives an answer, but as a result it never used more than around 24768 tokens on a question. The Deepseek API will instead just use all 32768 tokens thinking if it needs and then output nothing.
#DeepSeek#ISTA-DASLab#DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF 2.25bpw#DeepSeek API#Project-maya#Humanities Last Exam