Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

Artificial Analysis changes make benchmark scores harder to compare

AI summary

Artificial Analysis revised its benchmark methodology several times from v4.1 through v4.3.2, changing test weights, replacing or adding evaluations, retiring GPQA Diamond, and recalibrating GDPval Elo scores, according to its official changelog and methodology. Huxiu AI notes that Kimi K3 fell from 57 at its July evaluation to 44 on the current model page, while Kimi K3.1 Pro received 55 and was expected to lead the open-model ranking once weights were equalized. Because the service did not publish a crosswalk between old and new scores, results from different versions or capture dates should not be compared directly.

In-depth insight

The methodology changes make version labels essential when reading recent leaderboard rankings, particularly for open models such as MiMo-V2.6-Pro, Kimi K3.1 Pro and DeepSeek V4 Pro.

9月22日,小米开源MiMo-V2.6-Pro,46分,登顶Artificial Analysis开源榜。一个月后,Kimi K3.1 Pro发布,55分——等权重一放,将成为新的开源第一。一个月,两个"开源第一"。这不是中国模型突然行了,也不是突然不行了。是尺子又动了。AA这张榜,问题不在中国模型分低,在尺......

本文来自微信公众号: 穷奇商业 ,作者:穷奇观察

9月22日,小米开源MiMo-V2.6-Pro,46分,登顶Artificial Analysis开源榜。一个月后,Kimi K3.1 Pro发布,55分——等权重一放,将成为新的开源第一。

一个月,两个"开源第一"。

这不是中国模型突然行了,也不是突然不行了。是尺子又动了。

#Artificial Analysis#Xiaomi#MiMo-V2.6-Pro#Moonshot AI#Kimi K3.1 Pro#Kimi K3#DeepSeek V4 Pro#OpenAI#GDPval

Original · 8 min readRead the original
Group chat

No comments yet. Start the conversation.