社区基准测试比较七种AI决策模型
AI 总结
一份提交至r/LocalLLaMA的报告,使用来自GitHub、Firefox Bugzilla和CVE数据库的2,826个软件工程决策,比较七种决策模型。报告称,Liquid d1-ultra、Nace Drex Max v1.1和Perplexity Decider V1.2在平衡准确率上并列领先,而Perplexity Decider V1.2因准确率、成本和许可的综合表现被列为总体首选。GPT-6 Luna Decisions则被报告为准确率最低、成本最高且概率过度自信最严重。
深度见解
这项社区测量为结构化工程任务中的专用决策模型提供了涵盖准确率、成本、延迟、校准、许可和数据污染控制的对比参考。
| This report evaluates 7 specialized decision models tested across 2,826 real-world decisions sourced from GitHub, Firefox (Bugzilla), and the CVE database. Unlike open-ended generative language models, decision models evaluate structured inputs and produce discrete probability distributions across predefined categories in a single pass. Data was benchmarked through November 6, 2026, using inputs created after September 15, 2026, to avoid data leakage from pre-training sets. 1. Executive Summary & Model Overview Perplexity Decider V1.2 ranks as the strongest overall recommendation based on a balance of top-tier accuracy, lowest cost among frontrunners, and open-source licensing. Nace Drex Max v1.1 serves as a close runner-up, leading in domain-specific triage tasks. GPT-6 Luna Decisions registered the lowest accuracy, highest cost, and greatest degree of probability overconfidence. https://preview.redd.it/mxhvsi4qtl0i1.png?width=582&format=png&auto=webp&s=2778cfbeed370cf1c2dee93133cc8f640559fe05 2. Benchmark Design & Dataset Breakdown The evaluation evaluated 6 representative engineering tasks against human ground truth determined by maintainers. To ensure neutral evaluation: https://preview.redd.it/45zrsncutl0i1.png?width=563&format=png&auto=webp&s=7fcbd1d76870bcafceaf446b86f100d3b7e700d9 3. Task-by-Task Balanced Accuracy In overall balanced accuracy, Liquid d1-ultra (66.1%), Nace Drex Max v1.1 (65.6%), and Perplexity Decider V1.2 (65.6%) form a statistically tied top tier. All three significantly outperform Liquid d1.1 (63.7%) and GPT-6 Luna Decisions (62.8%). https://preview.redd.it/vt2wcq8ytl0i1.png?width=560&format=png&auto=webp&s=9cfa886c3e618ffb5dcffaae8482b0af6ba6729a 4. Latency, Repeatability, and Confidence Calibration Operational feasibility depends heavily on predictable inference speeds and probability calibration. A model is only operational if its stated confidence accurately reflects true probability. https://preview.redd.it/9iizqkt0ul0i1.png?width=561&format=png&auto=webp&s=fc876dd47088f5076ba9d537c136cfdd4c8770d8 5. Pricing, Openness, and Data Governance Model licensing and enterprise data policies differ markedly across providers. Output tokens are free for all tested decision models; costs reflect input processing fees. https://preview.redd.it/5ggo2u64ul0i1.png?width=559&format=png&auto=webp&s=bd92d0fd1a92b29f4d97acd52f161c5b27b8f387 6. Overall Composite Rankings Models were rated across five pre-weighted pillars: Quality (45%), Efficiency (20%), Trust (20%), Reviews (10%), and Signup Bonus (5%). Scores are normalized out of 100. https://preview.redd.it/6pv1ef28ul0i1.png?width=566&format=png&auto=webp&s=1a8196dc456d2835943f0f523ab1c6510c508311 7. Benchmark Constraints & Observations submitted by /u/m4rkGS [link] [comments] |
#Perplexity Decider V1.2#Nace Drex Max v1.1#GPT-6 Luna Decisions#Liquid d1-ultra#Liquid d1.1#OpenRouter