创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

TinyLeague 移除 Artificial Analysis,新增 KV 缓存效率指标

AI 总结

TinyLeague 维护者 nunodonato 宣布,该小型模型排名项目将停止追踪 Artificial Analysis 评分。维护者表示,许多小模型没有此类评分;他认为 Artificial Analysis 正转向封闭基准,这可能使小模型和开放权重模型获得的评分减少。TinyLeague 新增 KV 缓存效率指标,按 128K 上下文估算内存占用(GB),并以 0–100% 的比例对模型评分。维护者称,该指标面向显存有限的本地模型运行者。独立 TinyLeague-Index 的工作仍在继续,但维护者表示,由于不同模型使用不同基准,尚未找到公平的方法论。

深度见解

新指标让显存有限的本地模型用户可以直接比较模型在 128K 上下文下的预计 KV 缓存内存占用。该公告未提供对这一指标的独立验证。

Hi folks,

Today I made a significant update to Tiny League.

Based on the recent developments of AA (and leaving conspiracy theories aside), I decided it's no longer a viable metric to track and use. Mostly because many small models don't get a score, and because AA is moving toward closed benchmarks, we will see fewer scores for small/open-weight models in the future.

Second, I added a new metric that I think is useful for everyone running local models but is VRAM-poor: KV cache efficiency! I calculate an estimated GB value for a 128K context size, and then rate the models on 0-100% scale.

Still working out a way to have our own TinyLeague-Index, but haven't reached a fair methodology yet due to so many models using different benchmarks. More soon!

Let me know if you have any suggestions for improvements :)

thanks

submitted by /u/nunodonato [link] [comments]

#TinyLeague#Artificial Analysis

原文约 1 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。