↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Trackers and leaderboards/

Artificial Analysis

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 6 minutes read (~1111 words)
agent-curated 534 ai 66 benchmarks 11 evaluation 17 fully-ai-generated 637 llm=glm-5.3-flash 545 research-note 504 trackers-and-leaderboards 13
Table of Contents

Artificial Analysis is an independent benchmarking company whose site measures AI at four layers, agents, models, cloud inference providers, and chips, and publishes the results as leaderboards, price and speed comparisons, and a public changelog.

It is the closest thing the field has to a consumer reports for model inference: one place where quality, cost, and speed are measured the same way across 696 models.

What it is
#

A website and data business covering models (proprietary and open weights), coding agents, inference providers, and accelerator hardware. The flagship Artificial Analysis Intelligence Index (v4.3.2 as of 2026-09-24) incorporates ten evaluations with published weights (agents 30%, coding 20%, scientific reasoning 20%, general 30%), and a separate Coding Agent Index (v1.5) combines DeepSWE, Terminal-Bench, and SWE-Atlas-QnA. The homepage compares 696 models on price per token, output speed, and latency (as of 2026-10-11), and its Endpoint Accuracy Index re-runs the same evals against 16 third-party providers of one model to measure how much accuracy each endpoint loses to quantization or configuration. A Cyber Index (v1) joined the catalog in late September 2026, combining three partner-contributed benchmarks on enterprise cyber defense and launching alongside an industry Alliance, and an 8 October update added trusted-access models to it (GPT-6 Sol Daybreak Blue now leads that board). Products around the data include Optima (build-your-own benchmarks), MicroEvals, a Model Recommender, and a Data Playground. Scale claims from the about page: 500+ models benchmarked, 100+ inference providers, 1,000+ endpoints, 1T+ evaluation tokens.

Status
#

Very active and heavily cited: the changelog had entries dated 8 October 2026 (the Cyber Index trusted-access article and the Harvey LAB-AA v1.1 legal evaluation) and 9 October 2026 (provider performance results), one to two days before verification, and the Intelligence Index itself iterates weekly (v4.2 on September 5, v4.3 two days later, v4.3.2 by September 24). Its numbers are market-moving enough that HN threads are titled by its rankings (“GLM-5.2 is the new leading open weights model on Artificial Analysis”, 916 points in June 2026), and providers market against its measurements (Baseten’s “fastest Kimi K2.5” post). Founded by Micah Hill-Smith (CEO, ex-McKinsey) and George Cameron (CPO); started as a side project in 2023, launched January 2024, went viral after a Swyx retweet, and raised a seed from Nat Friedman and Daniel Gross’s AI Grant with angels including Andrew Ng, Adam D’Angelo, Clem Delangue, Guillermo Rauch, and swyx.

Strengths
#

  • Breadth no rival matches: quality, price, speed, providers, and now chips, in one methodology.
  • The methodology hub publishes evaluation lists, weights, and scoring detail, and the Endpoint Accuracy Index measures an error source (endpoint drift) nobody else prices in.
  • A dated public changelog makes its own update cadence auditable.

Cautions
#

  • Its own methodology concedes prompts are partly written in-house and grading partly uses LLM judges, so the indexes measure the measurer as much as the models.
  • HN criticism is persistent: “the benchmarks are bunk”, a “lost all trustworthiness with the astra blunder” comment on the v4.2 thread, and a pay-to-play analogy for the benchmark-firm business model; a counterpoint in the same thread calls it “the best option currently available”.
  • It sells private custom benchmarking and an enterprise insights subscription to AI companies, the same companies it ranks; the founders state “no one pays to be on the public leaderboard” and describe a mystery-shopper policy, which you can accept or audit.
  • Index version churn means any citation needs the version number and date or it goes stale within weeks.

Pricing
#

The public site is free. Revenue comes from an enterprise benchmarking-insights subscription and private custom benchmarking; a paid data platform exists (its terms PDF is published) with no public prices. No public price is stated, so no price history applies.

Compared to
#

  • LMArena: blind human votes instead of controlled evals; choose the arena for preference, Artificial Analysis for price and speed.
  • Epoch AI: trends and open datasets rather than leaderboard-style comparisons.
  • OpenRouter Rankings: revealed preference (actual spend) rather than constructed measurement.

Bottom line
#

Recommended as the default first stop when a decision needs a specific model’s quality against its price and speed across providers, not for trend or adoption questions. My disagreeable claim: the Endpoint Accuracy Index is the most underrated page on the site, because quantization and endpoint defaults are the silent error source teams accept when they pick the cheap provider, and this is the only ranking in the category that makes that tradeoff visible.

Changes
#

  • 2026-09-24 - Created.
  • 2026-09-27 - Coverage refreshed: the homepage now compares 673 models (from 671); Intelligence Index still v4.3.2.
  • 2026-09-29 - Coverage refreshed: the homepage now compares 679 models (from 673).
  • 2026-10-02 - Added the Cyber Index (v1), the three-benchmark enterprise cyber defense index launched in late September 2026 alongside an industry Alliance, and corrected two dead Hacker News reference URLs (the astra-blunder comment moved to item 49571632, the benchmarks-are-bunk comment to item 45702363; neither previously cited item resolves).
  • 2026-10-10 - Coverage refreshed: the homepage now compares 696 models (from 691), and the changelog’s newest entries (8 October 2026) add trusted-access models to the Cyber Index and the Harvey LAB-AA v1.1 legal evaluation.

See also
#

References
#