Qwen3.8 27B在用户韩语RAG测试中与Opus 5.5持平
一名Reddit用户使用Allganize的韩语RAG数据集评测Qwen3.8 27B和Opus 5.5，该数据集包含300道题和64份PDF，两个模型使用相同提示词及前五个检索页面。开启思考模式后，Qwen3.8 27B得分为77.3%，Opus 5.5为76.6%；由Opus 5.5评判时，两者分别得80.6%和80.3%。作者报告的McNemar检验结果为p=0.82。作者同时提醒，这只是温度为0的单次运行，换用不同硬件后有11项判定发生变化；另外增加的20道不可回答问题过于简单，所有模型都选择拒答。测试中一分之差不能被视为确定结论。
报告显示，在所测试的韩语文档RAG任务中，Qwen3.8 27B的成绩与Opus 5.5非常接近，但单次运行的评测方式和硬件敏感性限制了结论的确定性。
So I’ve been running local RAG over Korean PDFs for a while and my only evaluation was “yeah the answers look ok”. Finally sat down and used Allganize’s Korean RAG set since every question has a known answer and the page it’s on (300 questions, 64 PDFs, MIT). Fed every model the same top 5 pages and same prompt. I used qwen3.6 27B as a judge but figured it might go soft on its own family, so Opus 5.5 judged everything too.
qwen3.8 27B with thinking on got 77.3%, Opus 5.5 got 76.6%. p=0.82 on McNemar so that’s a tie. Opus judging gave 80.6 vs 80.3, same story. Thinking off drops qwen about 6 points though, so don’t skip that.
What surprised me more was looking at where Opus failed. 70 misses total, and 22 of those were pages that are just scanned images with no text layer, 23 the retriever never pulled the right page, and only 25 were actually the model screwing up. so honestly the model isn’t the bottleneck anymore for me, my PDF parsing is. There’s a 2024 commercial system (Alli + Claude 3.5 Sonnet) with results on this same set and it only beats my setup on those scanned pages, 19 to 1. I don’t do OCR. lol.
#Qwen3.8 27B#Opus 5.5#Allganize Korean RAG dataset#NVIDIA RTX 3090