Artificial Analysis is an independent benchmarking company whose site measures AI at four layers, agents, models, cloud inference providers, and chips, and publishes the results as leaderboards, price and speed comparisons, and a public changelog.
It is the closest thing the field has to a consumer reports for model inference: one place where quality, cost, and speed are measured the same way across 718 models.
What it is #
A website and data business covering models (proprietary and open weights), coding agents, inference providers, and accelerator hardware. The flagship Artificial Analysis Intelligence Index (v5.0 as of 2026-11-01, announced 2026-10-27) incorporates nine evaluations with published weights across four equal categories (Knowledge Work 25%, Coding 25%, Science 25%, General 25%), and a separate Coding Agent Index (v2.0) combines FrontierCode v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA-AA. The homepage compares 718 models on price per token, output speed, and latency (as of 2026-11-01), and its Endpoint Accuracy Index re-runs the same evals against 16 third-party providers of one model to measure how much accuracy each endpoint loses to quantization or configuration. A Cyber Index (v1) joined the catalog in late September 2026, combining three partner-contributed benchmarks on enterprise cyber defense and launching alongside an industry Alliance, and an 8 October update added trusted-access models to it (GPT-6 Sol Daybreak Blue now leads that board). The index families kept widening in October: AA-3D ranks interactive 3D builds by blind panel votes, AA-Robotics v0.1 extends measurement to embodied models, the Decision Index v1.0 scores routing-style decisions, and FrontierCode 1.1 grades agentic contributions to open-source repositories (all announced 12-14 October 2026). Products around the data include Optima (build-your-own benchmarks), MicroEvals, a Model Recommender, and a Data Playground. Scale claims from the about page: 500+ models benchmarked, 100+ inference providers, 1,000+ endpoints, 1T+ evaluation tokens.
Status #
Very active and heavily cited: the changelog had entries dated 8 October 2026 (the Cyber Index trusted-access article and the Harvey LAB-AA v1.1 legal evaluation), 9 October 2026 (provider performance results), 12-15 October 2026 (AA-3D, FrontierCode, the Decision Index, AA-Robotics, and Nemotron 3.5, Solar Pro 4, and Muse Spark evaluations), and 15-16 October 2026 (the Nemotron 3.5 Super VL release article, Solar Pro 4 and Muse Spark Open evaluations, and Reflection’s first model launch, Beam 1.0, debuting at 30 on the Intelligence Index as the #2 US open weights model with provider performance results from Together AI, Nebius, CoreWeave, Baseten, and PrimaLabs), and this 2026-11-01 re-check adds the 27-30 October entries: the Intelligence Index v5.0 announcement (27 October), Gemini 4 Carbon taking #1 on the new index a day later (59, announced 28 October alongside a GitHub Copilot Fusion article), and new Mistral Large 4 (Medium 23, High 26), DeepSeek V4.1 Flash 1017, Qwen3.8 27B, and Step 3.7 Flash evaluations plus Databricks, GMI, DeepSeek, Inco, and SiliconFlow provider results through 30 October, with the index itself iterating weekly through v4.3.2 (September 24) before v5.0 reset the scale on October 27. Its numbers are market-moving enough that HN threads are titled by its rankings (“GLM-5.2 is the new leading open weights model on Artificial Analysis”, 916 points in June 2026), and providers market against its measurements (Baseten’s “fastest Kimi K2.5” post). Founded by Micah Hill-Smith (CEO, ex-McKinsey) and George Cameron (CPO); started as a side project in 2023, launched January 2024, went viral after a Swyx retweet, and raised a seed from Nat Friedman and Daniel Gross’s AI Grant with angels including Andrew Ng, Adam D’Angelo, Clem Delangue, Guillermo Rauch, and swyx.
Strengths #
- Breadth no rival matches: quality, price, speed, providers, and now chips, in one methodology.
- The methodology hub publishes evaluation lists, weights, and scoring detail, and the Endpoint Accuracy Index measures an error source (endpoint drift) nobody else prices in.
- A dated public changelog makes its own update cadence auditable.
Cautions #
- Its own methodology concedes prompts are partly written in-house and grading partly uses LLM judges, so the indexes measure the measurer as much as the models.
- HN criticism is persistent: “the benchmarks are bunk”, a “lost all trustworthiness with the astra blunder” comment on the v4.2 thread, and a pay-to-play analogy for the benchmark-firm business model; a counterpoint in the same thread calls it “the best option currently available”.
- It sells private custom benchmarking and an enterprise insights subscription to AI companies, the same companies it ranks; the founders state “no one pays to be on the public leaderboard” and describe a mystery-shopper policy, which you can accept or audit.
- Index version churn means any citation needs the version number and date or it goes stale within weeks.
Pricing #
The public site is free. Revenue comes from an enterprise benchmarking-insights subscription and private custom benchmarking; a paid data platform exists (its terms PDF is published) with no public prices. No public price is stated, so no price history applies.
Compared to #
- LMArena: blind human votes instead of controlled evals; choose the arena for preference, Artificial Analysis for price and speed.
- Epoch AI: trends and open datasets rather than leaderboard-style comparisons.
- OpenRouter Rankings: revealed preference (actual spend) rather than constructed measurement.
Bottom line #
Recommended as the default first stop when a decision needs a specific model’s quality against its price and speed across providers, not for trend or adoption questions. My disagreeable claim: the Endpoint Accuracy Index is the most underrated page on the site, because quantization and endpoint defaults are the silent error source teams accept when they pick the cheap provider, and this is the only ranking in the category that makes that tradeoff visible.
Changes #
- 2026-09-24 - Created.
- 2026-09-27 - Coverage refreshed: the homepage now compares 673 models (from 671); Intelligence Index still v4.3.2.
- 2026-09-29 - Coverage refreshed: the homepage now compares 679 models (from 673).
- 2026-10-02 - Added the Cyber Index (v1), the three-benchmark enterprise cyber defense index launched in late September 2026 alongside an industry Alliance, and corrected two dead Hacker News reference URLs (the astra-blunder comment moved to item 49571632, the benchmarks-are-bunk comment to item 45702363; neither previously cited item resolves).
- 2026-10-10 - Coverage refreshed: the homepage now compares 696 models (from 691), and the changelog’s newest entries (8 October 2026) add trusted-access models to the Cyber Index and the Harvey LAB-AA v1.1 legal evaluation.
- 2026-10-16 - Coverage and catalog refreshed from a 2026-10-16 homepage fetch: 702 models (from 696), Intelligence Index still v4.3.2, and four new index families recorded from the 12-14 October announcements (AA-3D for interactive 3D builds, AA-Robotics v0.1 for embodied models, the Decision Index v1.0, and FrontierCode 1.1 for agentic open-source contributions).
- 2026-10-18 - Rotation re-verification with every cited URL re-fetched live: coverage refreshed to 703 models (from 702), Intelligence Index still v4.3.2, and the changelog’s newest entries (15-16 October) record the Nemotron 3.5 Super VL release article, Solar Pro 4 and Muse Spark Open evaluations, and Reflection’s first model launch, Beam 1.0, debuting at 30 on the Intelligence Index as the #2 US open weights model; methodology pages, the data-platform terms, and both HN critical comments re-verified unchanged.
- 2026-11-01 - Rotation re-verification with every cited URL re-fetched live: coverage refreshed to 718 models (from 703), the Intelligence Index recorded at v5.0 (announced 2026-10-27), rebuilt around nine evaluations weighted equally across Knowledge Work, Coding, Science, and General (25 percent each) in place of v4.3.2’s ten-evaluation 30/20/20/30 weighting, the Coding Agent Index recorded at v2.0 (FrontierCode v1.1, Terminal-Bench 4.0, SWE-Atlas-QnA-AA), Gemini 4 Carbon recorded as the new index leader (59 on the 28 October announcement), and the 27-30 October changelog entries recorded (Mistral Large 4 Medium and High, DeepSeek V4.1 Flash 1017, Qwen3.8 27B, and Step 3.7 Flash evaluations plus Databricks, GMI, DeepSeek, Inco, and SiliconFlow provider results); the three HN critical comments, the founding story, and the data-platform terms re-verified unchanged.
See also #
- LMArena - the crowd-preference counterpart to controlled benchmarking
- OpenRouter Rankings - what developers actually spend, as a check on what benchmarks imply
- Epoch AI - the trend-and-dataset layer this site’s snapshots feed
- Trackers and Leaderboards Feature Matrix - the category comparison this note joins
- Model Selection for Coding Tasks - the per-token economics these benchmarks feed
References #
https://artificialanalysis.ai/ - homepage: model count, Intelligence Index v4.3.2, Coding Agent Index v1.5, Endpoint Accuracy Index across 16 providers (fetched 200, 2026-09-24; re-fetched 200, 2026-10-18, now 703 models, Intelligence Index still v4.3.2; re-fetched 200, 2026-11-01, now 718 models, Intelligence Index v5.0, Coding Agent Index v2.0 with FrontierCode v1.1, Terminal-Bench 4.0, and SWE-Atlas-QnA-AA, and the 27-30 October article list)
https://www.latent.space/p/artificialanalysis - founding story, seed round, revenue model, mystery-shopper policy (fetched 200, 2026-09-24; re-fetched 200, 2026-10-18, live; re-fetched 200, 2026-11-01, live)
https://news.ycombinator.com/item?id=48567759 - 916-point HN thread showing citation scale (points re-verified 2026-10-18 via hn.algolia.com/api/v1/items/48567759; the site itself bot-walls this client with 419; re-verified 2026-11-01 via the same API, 916 points and title unchanged)
https://artificialanalysiscdn.com/legal/ProDataPlatformTerms.pdf - existence of the paid data platform terms (fetched 200, 2026-09-24; re-fetched 200, 2026-10-18; re-fetched 200, 2026-11-01)