Reddit user proposes private quality benchmarks for open-weight LLMs
In a post on r/LocalLLaMA, one member argued that the community should make its own private benchmarks focused on model quality, especially across quantization levels, rather than throughput (tok/s). The author asked participants to keep their benchmarks private and share the chart results with one another, and said benchmark competition is getting out of hand in the Hugging Face open-weight LLM space.
Proposal: A community revolution for private quality benchmarks (and why endless chasing tok/s is rotting our brains
honestly we in r/LocalLLaMA need a revolution of people making their own private benchmarks(not tok/s) i mean benchmarking the quality(especially across quantisation). keep yours private and share the chart results to each other. i feel this is what this sub was made for it seems nowaday every few hours someone is posting about tok/s and also Benchmaxxing is getting out of hand in the huggingface openweight LLM space.