Artificial Analysis
AI overview
Sign in and the AI will write an overview from our coverage.
Headlines · 119
- Reddit user tests MKA benchmark across six AI workflows
A Reddit submitter tested an Education/History task from Hugging Face against Google and Artificial Analysis's Multimodal Knowledge Agent (MKA) benchmark rubric. The six AI workflows included GPT, Gemini, Claude, and NotebookLM; some produced polished presentations but struggled with numerical accuracy. The submitter published the results, actual answers, and observations in a separate blog.
r/OpenAI · 🔥 0 - Free Chinese AI Reportedly Capable of Hacking Companies
Ariel Zeitlin, co-founder and CTO of Israeli security startup Tenzai, said freely downloadable Chinese AI has become capable of breaking into corporate networks. Micah Hill-Smith, CEO of AI benchmarking firm Artificial Analysis, said the most advanced cybersecurity AI is becoming harder and more costly to improve. He said this means Chinese models are unlikely to fall far behind and that no single developer is likely to pull far ahead of rivals.
The Information · 🔥 0 - Gemini 4 Hydrogen Rolls Out Globally in Google Search
Google began rolling out Gemini 4 Hydrogen globally in Google Search for all users, making it the default model for Search AI Mode and using it for more complex AI Overviews. Google says the model improves intent understanding, query fan-out and visual or structured answers, while other reports say it is also available through the Gemini app, AI Studio, Gemini API and Antigravity. Artificial Analysis measured Hydrogen at 51 points versus 50 for Gemini 4 Argon, but broader comparisons show it trailing Claude Sonnet 5.5 on Terminal-Bench 4.1, 57% versus 63.9%.
Google AI · The Keyword · 🔥 0 - Salesforce releases SOV-Transcribe-1 open-weight speech-to-text model
Salesforce and Missionforce Sovereign AI released SOV-Transcribe-1, an open-weight, speech-augmented language model for non-streaming automatic speech recognition. Artificial Analysis reported that it ranked first among open-weight speech-to-text models with a 2.3% AA-WER score and seventh among 44 models overall, while a Salesforce description cited by r/LocalLLaMA placed it second of 23 systems on VoiceCodeBench for entity accuracy and task success. The model supports seven languages, natural-language transcription controls, context biasing, and deployment through vLLM or Hugging Face Transformers.
ArtificialAnlys (X) · 🔥 0 - ElevenLabs launches Scribe v3 speech-to-text models
ElevenLabs released Scribe v3 for prerecorded audio and Scribe v3 Live for low-latency real-time transcription, with both models available through ElevenAPI and ElevenAgents. Artificial Analysis ranked Scribe v3 among the most accurate transcription models but found it slower and more expensive than several rivals; its benchmark did not test real-time performance, speaker diarization, or rare-language accuracy.
The Decoder · 🔥 0 - Claude Fable 5.2 Leads AA-Robotics v0.1 With a Score of 75
A report says Claude Fable 5.2 is the new leader on the AA-Robotics v0.1 benchmark, scoring 75 ahead of GPT-6 Astra at 69 and GPT-6.1 Sol at 65. It is also the first model to score on the measure-box task, using video alone and using the tape measure’s markings as a scale to estimate the box’s size.
ArtificialAnlys (X) · 🔥 0 - ACE Music v1 Tops AA's Vocal Arena Ahead of Suno v6
ACE Music v1, ACE Studio's new music-generation model, debuted at No. 1 on Artificial Analysis' AA-Music-Vocal v1.1 leaderboard with 1160 Elo, ahead of Suno v6 at 1144. Artificial Analysis found that it performed best on Reggae/Dancehall and Classical/Orchestral prompts, while a later hands-on report highlighted its symbolic-score generation workflow for more granular editing. ACE Music can create vocal or instrumental tracks from descriptions or user-provided lyrics through the ACE Music app, whose free tier includes 20 credits per day; no API is currently available.
ArtificialAnlys (X) · 🔥 3 - Baseten creates optimized DeepSeek V4.1 Flash 1017 Fast variant
Baseten says it used an internal harness to optimize more than 80 kernels and bottlenecks in DeepSeek V4.1 Flash 1017, creating a Fast variant. The reported techniques include kernel fusion, Programmatic Dependent Launch and host-loop minimization. Baseten says the variant appeared on Artificial Analysis with a rate of 991 output tokens per second and the lowest time per task on the cited Pareto frontier.
Baseten Blog · 🔥 0 - Google and Artificial Analysis launch MKA agent benchmark
Google DeepMind and Artificial Analysis launched MKA, a benchmark for testing agents on visually complex, real-world knowledge work such as blueprints, charts, reports, and spreadsheets. The evaluation set contains 200 private tasks developed with experts across 10 industries and 83 occupations, while 10 sample tasks are public on Hugging Face. Google reported that Gemini 4 Carbon ranked second in quality, with an average task cost about one-sixth that of the leading model.
IT之家 AI · 🔥 0 - Google DeepMind and Artificial Analysis launch MKA benchmark
Google DeepMind and Artificial Analysis launched the Multimodal Knowledge Agent (MKA) evaluation, which tests agents on 200 private visual tasks drawn from 10 industries, including architectural drawings, engineering schematics, and clinical scans. Tasks are scored against expert rubrics averaging about 70 weighted criteria, with Climb WSR as the headline metric; Artificial Analysis runs them in its Stirrup harness with code execution, web access, and image-viewing tools. Artificial Analysis reports Claude Fable 5.2 leading with a 57% Climb WSR, followed by Gemini 4 Carbon at 54% and Claude Opus 5.5 at 53%, alongside substantial differences in cost per task.
ArtificialAnlys (X) · 🔥 0 - MiniMax releases open-weight H3-Image for native 4K generation
MiniMax released H3-Image with open weights, enabling text-to-image generation, image editing, and the use of up to 16 reference images. The vendor says its high-speed version can generate native 4096×4096, 16-megapixel images in about 10 seconds, while a local RTX 5090 test took roughly one minute. Artificial Analysis ranked H3-Image first among open-weight models for text-to-image generation and image editing, and the model is available through MiniMax's API, MiniMax Code, and Hailuo AI.
智东西 · 🔥 0 - Fable 5.2 vs Opus 5.5: Reddit user identifies a reasoning-effort sweet spot
A Reddit user says Artificial Analysis ranks Fable 5.2 above Claude Opus 5.5, with performance rising smoothly from High to XHigh and jumping more noticeably at Max. The author recommends Opus 5.5 XHigh, Fable 5.2 High, or Fable 5.2 Max, describing Fable 5.2 High as the preferred switching point.
r/ClaudeAI · 🔥 0 - Anthropic launches Claude Fable 5.2 and Mythos 5.2
Anthropic released Claude Fable 5.2 and Mythos 5.2, describing both as cheaper and faster upgrades, with Fable 5.2 priced at $8 per million input tokens and $40 per million output tokens. Artificial Analysis placed Fable 5.2 first on its Intelligence Index, five points ahead of the unreleased Gemini 4 Carbon, while Mythos 5.2 is limited to vetted security and life-sciences groups. Anthropic says Fable 5.2 is 20% cheaper and up to 30% faster than version 5.1, though it remains twice as expensive as Opus 5.5.
The Rundown AI · 🔥 0 - Artificial Analysis announces first image and video AI event
Artificial Analysis announced its first image and video event, scheduled for November 17 in San Francisco. The event will focus on subjective evaluation of generative media, with a panel spanning image, video, and other media generation, plus live answers to selected attendee questions. The organization also claimed that AI-generated media has become commercially valuable and can sometimes be indistinguishable from reality.
ArtificialAnlys (X) · 🔥 0 - Anthropic Releases Claude Fable 5.2 and Mythos 5.2
Anthropic released Claude Fable 5.2 and Mythos 5.2 for complex work including coding, reasoning, scientific research and cybersecurity. Fable 5.2 is available to all users at $8 per million input tokens and $40 per million output tokens. Access to Mythos 5.2 requires approval through Anthropic’s Life Sciences and Cyber Verification Programs. Anthropic says both models produce output up to 30% faster than their predecessors and have token prices about 20% below the previous versions.
Simon Willison's Weblog · 🔥 0 - Meta launches Muse Ember for cost-efficient agent tasks
Meta launched Muse Ember, a lightweight model designed for complex reasoning and repeated agent work such as tool calls, coding subtasks, routing, extraction, and long-document parsing. It supports a one-million-token context window and native text, image, audio, and video understanding, and is available through Meta Model API and Muse Code. Meta says Artificial Analysis results show it matching Claude Haiku 5.5’s best score at roughly one-third the cost and outperforming every GPT-6 Luna reasoning setting, though the comparative figures are vendor-reported.
Hacker News 首页(100+ 分) · 🔥 0 - GPT-6.2 Sol reportedly appears in a limited ChatGPT rollout
新智元 and 36Kr reported that some ChatGPT users were quietly routed to GPT-6.2 Sol, although the supplied material contains no official OpenAI launch announcement. Community demonstrations reportedly produced browser-based 3D scenes from a single prompt without Blender or external assets. The reports also cited Artificial Analysis results giving GPT-6.2 Sol Max a 60 intelligence score, an average task time of 18.7 minutes and a task cost of $1.19.
36氪 人工智能 · 🔥 0 - Artificial Analysis releases Decision Index v2
Artificial Analysis released Decision Index v2, whose new Decision Score combines answer correctness with confidence calibration. Private datasets now make up 45% of the index, up from 25% in v1. The update adds the 1,500-document private AA-Paperclip benchmark and the AA-Symmetry consistency test, adds an implementation of the ONET benchmark, removes WildGuardTest because of high saturation, and reduces When2Call’s weighting from 25% to 10%. Claude Haiku 5.5 (Max) and GPT-6 Luna (Max) scored 77 and 76, respectively, while Drex Max v1.1 and GLiDE v1.2 tied at 73 among specialist decision models. GPT-6 Luna cost $1,530 per million tasks and took 11.34 seconds per task, compared with $102 and 275 milliseconds for Drex Max v1.1. GLiDE v1.2 cost $780 per million tasks, nearly eight times Drex Max v1.1’s cost, because it bills the document input for every question.
ArtificialAnlys (X) · 🔥 0 - Artificial Analysis v5.0 evaluates real-world AI tasks
A Juejin commentary describes Artificial Analysis Intelligence Index v5.0 as a framework for evaluating end-to-end workplace tasks rather than isolated question-answering. It says the index contains nine evaluations grouped into four equally weighted categories—knowledge work, coding, science and general capabilities—while the individual evaluations have different weights. The article highlights constrained workflows, deliverable quality, tool use, safety guardrails and first-attempt success, and notes limitations including an English focus, private test sets, high difficulty and limited multimodal coverage. These details reflect the commentary's interpretation and are not independently corroborated in the supplied material.
掘金 · 人工智能 · 🔥 0 - Artificial Analysis launches VIALS-AA visual-reasoning leaderboard
Artificial Analysis launched VIALS-AA, a leaderboard for Handshake AI’s VIALS benchmark, with 161 tasks across eight life-science and biotech domains. Models must interpret scientific artifacts—including gel blots, flow cytometry plots, plasmid maps, and phylogenies—and produce specific conclusions, with each task run three times in the Stirrup agent harness. Gemini 4 Carbon led with a 64% pass^3 score, followed by GPT-6 Astra at 63% and Claude Opus 5.5 at 61%, while Kimi K3.1 Pro led the open-weight models with 45%.
ArtificialAnlys (X) · 🔥 0 - Gemini 4 Carbon Nears Wider Availability
The Batch reports that Google’s Gemini 4 Carbon is approaching wider availability. It also says the model put Google back on top of Artificial Analysis’ rankings, although the report does not specify the rollout timeline or availability regions.
The Batch · DeepLearning.AI · 🔥 0 - Nace AI details Drex training and updates Drex Max v1.1
Nace AI published a technical report covering the architecture, training, evaluation, calibration and local inference of four open-weight Drex decision models. The models use full-parameter supervised fine-tuning on 1.15 million verified decision records, followed by consistency training and reinforcement learning from adjusted feedback; Nace AI also released Drex Max v1.1. Nace AI claimed that Max v1.1 reached the state of the art and Pareto frontier for decision models, while Artificial Analysis reportedly ranked it first, but these performance claims remain vendor-attributed and the v1.1 weights were described as still forthcoming.
r/LocalLLaMA · 🔥 0 - Artificial Analysis Endpoint Accuracy Index v1.1 expands coverage
Artificial Analysis launched Endpoint Accuracy Index v1.1, expanding coverage to six open-weight models, including GLM-5.3 Flash and Muse Spark Open. The index compares provider endpoints with a self-hosted reference using tool calling, scientific reasoning and long-context recall; its updated AA-LCR v1.1 uses 25 repeats and narrows the average 95% confidence interval from ±4.7 to ±2.6 points.
ArtificialAnlys (X) · 🔥 0 - Thinking Machines launches Inkling 2 Small
Thinking Machines Lab launched Inkling 2 Small, a 594B-total-parameter, 17B-active mixture-of-experts reasoning model available for inference and fine-tuning through Tinker and Tinker Playground. Artificial Analysis measured an Intelligence Index score of 31, up 18 points from Inkling Small, while AutomationBench-AA rose from 5% to 46% and FrontierCode 2.0 from 5% to 25%. The model initially has closed weights, with Thinking Machines planning an Apache 2.0 release after external safety testing and red-teaming.
ArtificialAnlys (X) · 🔥 0 - FLUX 3 Image Klein Ranks Second Among Open-Weight Image Models
Black Forest Labs released the 5.6B-parameter FLUX 3 Image Klein on October 15 under the FLUX Kommunity License, allowing non-commercial use and commercial output use by individuals and companies with annual revenue below $5 million. Artificial Analysis ranked it second among open-weight models on AA-Image-T2I v2.1 and third on AA-Image-Editing v2.1, based on 1K-resolution tests through the BFL API with its prompt upsampler. The model is available through that API and the BFL Playground, with open weights on Hugging Face, at $0.009 per 1K image.
ArtificialAnlys (X) · 🔥 0 - Gemini 4 Carbon reportedly tops Artificial Analysis Intelligence Index v5
A secondary report citing Artificial Analysis says Gemini 4 Carbon placed first in the Intelligence Index v5 with 59 points, leading four of nine evaluations: HLE-Diamond, Terminal-Bench-Science, AA-Omniscience and AutomationBench-AA. Carbon also scored 59 on the coding-agent index, three points behind Claude Opus 5.5, while its reported hallucination rate rose from Argon’s 4% to 11%. The report says Carbon remains limited to selected users, with no public availability timetable.
掘金 · 人工智能 · 🔥 0 - Artificial Analysis launches Progress Index with Intelligence Index v5
Artificial Analysis published the full presentation from its Intelligence Index v5 launch event in San Francisco. The company says it introduced the Progress Index as its biggest upgrade to the Index since v4, and shared what comes next for evaluation across 3D reasoning, personality, cyber defense, fast classifier models, enterprise evaluations, and robotics. The posts come from a single Artificial Analysis account, with the second duplicating the first, and give no technical details, timelines, scoring methodology, or results for the planned evaluations.
ArtificialAnlys (X) · 🔥 0 - MiMo-V2.6-Pro leads models in question-ended responses
Artificial Analysis found that MiMo-V2.6-Pro ended 34% of its AA-PersonalityBench responses with a question, the highest share among the models measured. GPT-6 Astra followed at 32%, while GPT-6.1 Sol, Muse Glimmer, and Muse Spark Open recorded shares of 29%, 28%, and 26%, respectively.
ArtificialAnlys (X) · 🔥 0 - Tencent Hunyuan Hy4 benchmarks strongly as inference capacity expands
Tencent Hunyuan Hy4 scored 35 on the Artificial Analysis Intelligence Index v5, ranking second among models from China-based labs, according to ArtificialAnlys. It posted 31% on FrontierCode 2.0 and 37% on Terminal-Bench 4.0, while its one-million-token context window is priced at $0.834 per million input tokens and $2.501 per million output tokens. Tencent also said it expanded Hy4's inference cluster tenfold and increased free user allowances following launch-day quota pressure, though the capacity figures were not independently verified.
量子位(原生 RSS) · 🔥 0 - Artificial Analysis makes Time per Task its headline API metric
Artificial Analysis announced that Time per Task is now the headline metric in its API performance benchmark, rather than streaming speed. The update applies to roughly 1,000 serverless endpoints. End-to-end output speed is measured from request submission to the final token, including prefill, hidden reasoning and generation. Every measurement now includes short, document-length and long-context prompts, plus multi-turn coding and knowledge-work sessions; headline figures are five-day averages. The benchmark tracks each provider’s API default effort alongside highest effort for key models, because effort labels are not comparable across labs.
ArtificialAnlys (X) · 🔥 0
Experience and discussion from the community
Share my Artificial Analysis experienceAsk about Artificial Analysis
Nobody has shared their experience with Artificial Analysis yet.