创建

登录 ReadmeX

或

了解我们如何处理你的信息: 隐私政策

ReadmeX

陪你读懂 AI 动态

今天,想先了解什么?

选一份动态总结,或直接聊聊你的问题。

社区和关注动态总结只属于你自己。

资讯

OpenRouter详解应用型LLM评测方法

AI 总结

OpenRouter Blog介绍了面向应用任务的LLM评测方法,包括脚本判分、重复试验、账单核验和请求时间统计。在其60项合成任务测试中,GPT-6.1 Sol三轮共通过全部180次试验;DeepSeek响应成本最低,但仅通过157次。测试结果与公开模型排名的相关性较弱,合并结果的Spearman相关系数为0.109。

深度见解

该报告提供了复现应用型模型对比的方法,也表明公开排名未必能反映模型在特定工作负载上的表现。

How to Benchmark LLMs

OpenRouter ·11/12/2026

How to Benchmark LLMs
How to Benchmark LLMs

Choose models using evidence from your application’s inputs and output formats. Run identical tasks through OpenRouter and compare graded accuracy, billed cost, and request time. On our 60 synthetic tasks, GPT-6.1 Sol passed 180/180 trials across three repeats under the tested contracts and settings.

TL;DR

  • Code checks measure correctness and output compliance.
  • Sol passed 180/180 trials on our synthetic set; DeepSeek had the lowest response cost, with 157/180 passes.
  • Agreement with the public ranking was weak, with a Spearman correlation of 0.109.
  • Before you trust a score, check how many times each task ran, how many answers came back, and whether each bill was verified.

#OpenRouter#GPT-6.1 Sol#DeepSeek

原文约 14 分钟读完阅读原文
群聊

还没有评论,来说说你的看法。