TinyLeague 移除 Artificial Analysis,新增 KV 缓存效率指标
TinyLeague 维护者 nunodonato 宣布,该小型模型排名项目将停止追踪 Artificial Analysis 评分。维护者表示,许多小模型没有此类评分;他认为 Artificial Analysis 正转向封闭基准,这可能使小模型和开放权重模型获得的评分减少。TinyLeague 新增 KV 缓存效率指标,按 128K 上下文估算内存占用(GB),并以 0–100% 的比例对模型评分。维护者称,该指标面向显存有限的本地模型运行者。独立 TinyLeague-Index 的工作仍在继续,但维护者表示,由于不同模型使用不同基准,尚未找到公平的方法论。
新指标让显存有限的本地模型用户可以直接比较模型在 128K 上下文下的预计 KV 缓存内存占用。该公告未提供对这一指标的独立验证。
Hi folks,
Today I made a significant update to Tiny League.
Based on the recent developments of AA (and leaving conspiracy theories aside), I decided it's no longer a viable metric to track and use. Mostly because many small models don't get a score, and because AA is moving toward closed benchmarks, we will see fewer scores for small/open-weight models in the future.
Second, I added a new metric that I think is useful for everyone running local models but is VRAM-poor: KV cache efficiency! I calculate an estimated GB value for a 128K context size, and then rate the models on 0-100% scale.
Still working out a way to have our own TinyLeague-Index, but haven't reached a fair methodology yet due to so many models using different benchmarks. More soon!
Let me know if you have any suggestions for improvements :)
thanks
submitted by /u/nunodonato [link] [comments]