OpenRouter详解应用型LLM评测方法
AI 总结
OpenRouter Blog介绍了面向应用任务的LLM评测方法,包括脚本判分、重复试验、账单核验和请求时间统计。在其60项合成任务测试中,GPT-6.1 Sol三轮共通过全部180次试验;DeepSeek响应成本最低,但仅通过157次。测试结果与公开模型排名的相关性较弱,合并结果的Spearman相关系数为0.109。
深度见解
该报告提供了复现应用型模型对比的方法,也表明公开排名未必能反映模型在特定工作负载上的表现。
How to Benchmark LLMs
OpenRouter ·11/12/2026


Choose models using evidence from your application’s inputs and output formats. Run identical tasks through OpenRouter and compare graded accuracy, billed cost, and request time. On our 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials across three repeats under the tested contracts and settings.
TL;DR
- Code checks measure correctness and output compliance.
- Sol passed 180/180 trials on our synthetic set; DeepSeek had the lowest response cost, with 157/180 passes.
- Agreement with the public ranking was weak, with a Spearman correlation of 0.109.
- Before you trust a score, check how many times each task ran, how many answers came back, and whether each bill was verified.