社区实测2.25bpw DeepSeek-V4.1-Flash
一名r/LocalLLaMA作者将ISTA-DASLab的DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF 2.25bpw量化版与DeepSeek官方API进行了比较，测试使用Humanities Last Exam的100道纯文本子集题。在API完成回答的74道题中，本地量化版的语义正确率为19/74，API为21/74；作者称这只是初步冒烟测试，因为令牌限制、未完成回答和显著的运行波动影响了比较。
这项社区测试为2.25bpw量化版的实用性提供了初步证据，但其测试缺陷也意味着尚不能据此确定量化模型与完整模型的接近程度。
I've been running ISTA-DASLab's 2.25bpw quant of DSv4.1-1017 for a few days, and it's holding up well. But the fact that it's only 2.25bpw average did make me wonder how good the quant actually is against the full model.
Local setup: Project-maya v1.1.27. ISTA-DASLab's 2.25bpw quant. Quality-relevant settings: "DS4_INDEX_TC": "1". "MAYA_DS4_PROMPT_SKIP": "2".
Test: 100 question text-only subset of Humanities Last Exam.
Benchmarking models isn't easy, and these results are nothing but a smoke test. There are lots of issues with these results:
1) I ran the questions with a 32768 token limit per question, because anything larger becomes prohibitively time consuming. It already took my local instance around 12 hours to complete the test, and that's at an average of 40.4 toks/s.
2) When approaching the token limit, Deepseek's API and project-maya work differently. Project-maya will reduce thinking_budget to around 3/4 of the token limit so it always gives an answer, but as a result it never used more than around 24768 tokens on a question. The Deepseek API will instead just use all 32768 tokens thinking if it needs and then output nothing.
#DeepSeek#ISTA-DASLab#DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF 2.25bpw#DeepSeek API#Project-maya#Humanities Last Exam