Community coding benchmark V2 tests five algorithmic tasks on 16GB VRAM
A Reddit user published version 2 of a community benchmark evaluating models across five algorithmic tasks on hardware with 16GB VRAM and 32GB RAM. The update tightened the tasks, changed the model lineup, and added a GitHub repository with results so others can run the benchmark. The author said CyberTiel and Ornith performed surprisingly well, while LFM and Mellum performed poorly across the tasks and real workloads tested.
The update provides a reproducible community evaluation of local models under a constrained hardware setup, though its findings come from one author's methodology and test conditions.
Posted a bech I did here before: https://www.reddit.com/r/LocalLLaMA/comments/1x8qlvw/nine_models_five_coding_tasks_one_16gb_gpu/
Now I've done V2, tightened up the tasks a bit and a different lineup, plus a github repo now for anyone wanting to run this as well.
https://github.com/Tako88/coding-bench/blob/main/A/RESULTS-baseline.md
Tried posting the tables here directly, but they're very hard to read, so linking the results file instead.
CyberTiel and Ornith really surprised me this time.
LFM and Mellum just suck at anything I throw at them, real workloads too.