Create

Sign in to ReadmeX

or

How we handle your data: Privacy Policy

ReadmeX

Your AI reading companion

What would you like to catch up on?

Pick a briefing, or ask your own question.

Your community and people briefings stay personal to you.

Story

TinyLeague removes Artificial Analysis, adds KV cache efficiency metric

AI summary

TinyLeague maintainer nunodonato announced that the small-model ranking project will stop tracking Artificial Analysis scores. The maintainer said many small models lack such scores and that, in his view, Artificial Analysis is moving toward closed benchmarks, which could result in fewer scores for small and open-weight models. TinyLeague added a KV cache efficiency metric that estimates memory use in gigabytes for a 128K context and rates models on a 0–100% scale. The maintainer said the metric is intended for people running local models with limited VRAM. Work on a separate TinyLeague-Index continues, but the maintainer said a fair methodology has not yet been found because models use different benchmarks.

In-depth insight

The new metric gives users running local models with limited VRAM a direct way to compare estimated KV cache memory use at a 128K context. The announcement does not include independent validation of the metric.

Hi folks,

Today I made a significant update to Tiny League.

Based on the recent developments of AA (and leaving conspiracy theories aside), I decided it's no longer a viable metric to track and use. Mostly because many small models don't get a score, and because AA is moving toward closed benchmarks, we will see fewer scores for small/open-weight models in the future.

Second, I added a new metric that I think is useful for everyone running local models but is VRAM-poor: KV cache efficiency! I calculate an estimated GB value for a 128K context size, and then rate the models on 0-100% scale.

Still working out a way to have our own TinyLeague-Index, but haven't reached a fair methodology yet due to so many models using different benchmarks. More soon!

Let me know if you have any suggestions for improvements :)

thanks

submitted by /u/nunodonato [link] [comments]

#TinyLeague#Artificial Analysis

Original · 1 min readRead the original
Group chat

No comments yet. Start the conversation.