Qwen3.8 Scores 43% on 13-Language GitHub Coding Benchmark
AI summary
A team said it built a benchmark of 202 real GitHub tasks across 13 programming languages that rarely appear in coding benchmarks. The team tested Qwen3.6 27B, Qwen3.8 27B and GLM-5.3 Flash, reporting that Qwen3.8 solved 43% of the tasks versus 17% for Qwen3.6. Its leaderboard also includes results for Kimi K3, DeepSeek, GLM and closed models, with a per-language breakdown. The author disclosed that Qwen3.8 was used to select difficult tasks, which may bias its reported score downward.
| Hi! My team and I built a benchmark of 202 real GitHub tasks in 13 languages that rarely show up in coding benchmarks: Elixir, OCaml, Haskell, Clojure, Kotlin, Swift, Julia, R and others. I think a lot of people do local development in these languages too. We didn’t have a huge compute budget, so I specifically added Qwen3.6 27B and Qwen3.8 27B to compare them. Qwen3.8 solves 43% of the tasks, up from 17% for Qwen3.6. You can compare them with Kimi K3, DeepSeek, GLM and closed models on the leaderboard, including a per-language breakdown. One caveat: Qwen3.8 was used to pick hard tasks, so its score is a bit biased down. If you’ve been running Qwen3.8 locally on any of these languages, share how it went. submitted by /u/Fabulous_Pollution10 [link] [comments] |