Qwen3.8 27B Ties Opus 5.5 in User-Run Korean RAG Test
A Reddit user evaluated Qwen3.8 27B and Opus 5.5 on Allganize’s Korean RAG dataset, which contains 300 questions and 64 PDFs, giving both models the same prompt and top five retrieved pages. With thinking enabled, Qwen3.8 27B scored 77.3%, versus 76.6% for Opus 5.5. When Opus 5.5 served as judge, the scores were 80.6% and 80.3%, respectively. The author reported a McNemar test result of p=0.82. The author also cautioned that this was a single run at temperature 0, that 11 verdicts changed on different hardware, and that 20 additional unanswerable questions were too easy because every model refused them. The one-point gap therefore was not treated as conclusive.
The report shows Qwen3.8 27B closely matching Opus 5.5 on the tested Korean document RAG task, but its single-run methodology and hardware sensitivity limit the strength of the conclusion.
So I’ve been running local RAG over Korean PDFs for a while and my only evaluation was “yeah the answers look ok”. Finally sat down and used Allganize’s Korean RAG set since every question has a known answer and the page it’s on (300 questions, 64 PDFs, MIT). Fed every model the same top 5 pages and same prompt. I used qwen3.6 27B as a judge but figured it might go soft on its own family, so Opus 5.5 judged everything too.
qwen3.8 27B with thinking on got 77.3%, Opus 5.5 got 76.6%. p=0.82 on McNemar so that’s a tie. Opus judging gave 80.6 vs 80.3, same story. Thinking off drops qwen about 6 points though, so don’t skip that.
What surprised me more was looking at where Opus failed. 70 misses total, and 22 of those were pages that are just scanned images with no text layer, 23 the retriever never pulled the right page, and only 25 were actually the model screwing up. so honestly the model isn’t the bottleneck anymore for me, my PDF parsing is. There’s a 2024 commercial system (Alli + Claude 3.5 Sonnet) with results on this same set and it only beats my setup on those scanned pages, 19 to 1. I don’t do OCR. lol.
#Qwen3.8 27B#Opus 5.5#Allganize Korean RAG dataset#NVIDIA RTX 3090