Qwen3.8在13语言GitHub编程基准上得分43%
AI 总结
一个团队称，其构建了一个包含202个真实GitHub任务的基准，涵盖13种很少出现在编程基准中的语言。团队测试了Qwen3.6 27B、Qwen3.8 27B和GLM-5.3 Flash，并报告Qwen3.8解决了43%的任务，而Qwen3.6为17%。排行榜还列出了Kimi K3、DeepSeek、GLM和闭源模型的结果，并提供分语言细目。作者披露，Qwen3.8参与了困难任务的筛选，因此其报告得分可能受到向下偏差影响。
| Hi! My team and I built a benchmark of 202 real GitHub tasks in 13 languages that rarely show up in coding benchmarks: Elixir, OCaml, Haskell, Clojure, Kotlin, Swift, Julia, R and others. I think a lot of people do local development in these languages too. We didn’t have a huge compute budget, so I specifically added Qwen3.6 27B and Qwen3.8 27B to compare them. Qwen3.8 solves 43% of the tasks, up from 17% for Qwen3.6. You can compare them with Kimi K3, DeepSeek, GLM and closed models on the leaderboard, including a per-language breakdown. One caveat: Qwen3.8 was used to pick hard tasks, so its score is a bit biased down. If you’ve been running Qwen3.8 locally on any of these languages, share how it went. submitted by /u/Fabulous_Pollution10 [link] [comments] |