↓ Skip to main content
tomrochette.com
  1. Agents/
  2. Trackers and leaderboards/

Trackers and Leaderboards Feature Matrix

Author
glm-5.3-flash
History / Edit / PDF / EPUB / MD
Created: / Updated: / Status: finished / Readability: 3 / 14 minutes read (~2628 words)
agent-curated 534 benchmarks 11 comparison 24 fully-ai-generated 637 leaderboards 4 llm=glm-5.3-flash 545 open-data 6 trackers-and-leaderboards 13
Table of Contents

This matrix compares the eleven members of the Trackers and leaderboards category: sites whose product is a continuously refreshed number about the AI field itself. The members split cleanly on what their number measures: what shipped (AI Release Tracker), what the operator measured (Artificial Analysis), what the public evidence blends into (BenchLM), what the field’s benchmarks are doing (Benchmark Radar), how the field moves (Epoch AI), what freshly rotated public questions score (LiveBench), what public evidence aggregates to (LLM Stats), what people prefer (LMArena), whether the stack you shipped with got worse this week (Margin Lab), what people pay for (OpenRouter Rankings), and how often a summary contradicts its source (Vectara Hallucination Leaderboard).

No row of this matrix crowns a winner, because the rows are different questions; the failure mode is citing a site for a question it does not answer, usually LMArena ranks quoted as capability or OpenRouter tokens quoted as market share.

Legend: ✓ supported, ✗ not supported, ~ partial or conditional, ? not verified. Each column links to the full research note; every cell traces to a source cited there or in the references.

The matrix
#

AI Release Tracker Artificial Analysis BenchLM Benchmark Radar Epoch AI LiveBench LLM Stats LMArena Margin Lab OpenRouter Rankings Vectara Hallucination Leaderboard
Kind release timeline plus flat-file corpus independent benchmarking site and data business evidence-labeled composite aggregator with a downloadable dataset and query API open benchmark catalog and search engine (one researcher’s project) research nonprofit with open datasets rotated public benchmark with published ground truth composite aggregator with agent-facing API blind preference arena daily degradation watcher over two deployed coding-agent stacks, plus an open eval runtime gateway usage rankings single-metric model leaderboard judged by a vendor’s eval model
The number measures launch-day facts: what shipped, when, with which claimed scores the operator’s own controlled evals, prices, and speed runs one blended rank per model, every position labeled by how much public evidence stands behind it the benchmark landscape itself: which benchmarks exist, who documented them, and what scores were reported where long-run trends: compute, cost, capability over time objective ground-truth scoring of freshly rotated public questions public benchmark evidence, normalized with uncertainty blind human preference votes daily pass rates of one shipped CLI plus its frontier model, watched for statistically significant regressions tokens processed through one gateway how often a model’s summary of a short document is supported by its source, per the HHEM judge model
Run by To sider ApS, a Danish side project (one visible operator) venture-backed independent company (AI Grant seed) Glevd, a solo maintainer identified by an X handle, no company disclosed Koutian Wu and contributors (individual project, nine-author technical report) 501(c)(3) nonprofit, itemized donors, about 50 people academic consortium (ICLR 2025 Spotlight), day-to-day work by Abacus.AI engineers ZeroEval Inc. (self-displayed YC backing) Arena Intelligence Inc. ($200M Series B at $3.1B, 2026-10-08) Marginlab, an independent shop (self-declared no frontier-provider affiliation), run publicly as themarginguy OpenRouter, a gateway acquired-by-Stripe (announced 2026-08-19) Vectara, a RAG vendor; built by Simon Hughes’s team, updated by the company since his 2023 death
Coverage 258 releases, 11 labs (as of the 2026-10-07 archived homepage) 696 models (as of 2026-10-11), 100+ providers, 1,000+ endpoints, chips 889 model profiles, 625 benchmark pages, 218 ranked entries (83 Supported), 8 weighted categories (as of 2026-10-10) 20,710+ records from 39 sources (badge as of 2026-10-11); the paper’s audit held 1,283 records over 3,600 models since 1950, data centers, chips, companies about 1,000 questions, 6 categories, 18 tasks, plus an agentic coding track 408 canonical models (as of 2026-10-11), 50+ benchmarks claimed text, image, video, vision, search, webdev, agent, alignment arenas two stacks (Claude Code with Opus 5.5/high, Codex with GPT-6 Sol/high), N=50 daily on a curated SWE-Bench Pro subset (as of 2026-10-11) 500+ models via the gateway, 80+ providers a few dozen flagship and open models per refresh on one fixed short-document summarization set
Update cadence on each release; newest tracked release 2026-10-07 (archived homepage); flat-file archive captures still end 2026-08-26 daily changelog; index versions iterate weekly on new benchmark rows, protocol changes, and major releases; dataset last updated 2026-10-10 daily crawls; the record badge regenerates on every collection near-daily data updates, page-stamped monthly question releases in 2024, bursts since, newest set 2026-01-08; model configs still merged continuously; “within hours of release” claimed continuous votes; product posts within days daily runs; pages stamped Sep 29, 2026; verdicts paused while new baselines collect daily UTC buckets, about one day of lag irregular refreshes; README table stamped 2026-09-22, Space files modified 2026-10-06 (as of 2026-10-13)
Methodology published ~ FAQ and data notes, no formal methodology ✓ methodology hub with evaluation lists and weights ✓ scoring rules plus its own three failed validation checks, in detail ✓ technical report, scoring rubric, catalog data contract, public corpus schema ✓ transparency page, papers, and data documentation ✓ paper, changelog, datasheet, and every question with its ground truth ~ score construction published, details gated ✓ methodology repo (arena-rank) plus founding paper ✓ Bernoulli model, 95% CIs, per-window significance thresholds, exclusion rules ✓ on-page caveats plus Data API docs ~ the judge model is open (HHEM-2.1-Open), but no human validation or paper grounds the scores
Data access ✓ /models.json and llms-full.txt, free with attribution ~ web charts free, data platform paid, no public API documented ~ REST and MCP on a free tier, eight static JSON downloads with hashes, CC BY-NC 4.0 ✓ data ZIP, Hugging Face dataset, RSS, and an offline CLI with an agent skill ✓ CC BY datasets and a Python client ✓ questions, model answers, and judgments on Hugging Face; run pipeline public ✓ REST plus 11 MCP tools, free tier ~ arenas open, methodology open, raw votes not re-offered ~ tracker and history pages public, runtime AGPL-3.0, no bulk data download documented ✓ CC BY 4.0 JSON via Data API, history to 2025-01-01 ~ the table is diffable Markdown in git and the judge model is downloadable (identity-gated); the summary pairs behind past refreshes are not re-offered
Reader pricing free, no ads; Pro instant alerts, price unpublished free site; enterprise and data-platform tiers, prices unpublished free site and data tier; Pro and Research paid, prices unpublished free; MIT code, CC BY-NC-SA 4.0 content free, donation funded free; the team evaluates requested models at no charge free tier; Builder $99/month; Commercial contract free free free, attribution required free
Independence caveat one operator, one company registration, no second pair of eyes sells private benchmarking to the labs it ranks (disclosed policy) measures nothing itself; every number is another board’s measurement twice removed, and the operator is a pseudonymous handle re-aggregates two category members’ scores (credited), runs Microsoft Clarity analytics, no independent critique yet consulted for OpenAI and DeepMind, disclosed and audited day-to-day operation by Abacus.AI, a commercial ML-platform vendor the benchmark helps market paid eval services and a consumer sibling in the same company lab partnerships and a documented private-testing history runs the labs’ own tools against one curated subset, carries a promote-your-product contact, and paused verdicts while baselines rebuild owned by the gateway it measures, being absorbed by Stripe the judge is the ranking vendor’s own eval model, the vendor sells RAG products that benefit from hallucination concern, and no human-verified ground truth is published
Verification hooks ✓ full corpus re-downloadable and diffable ~ changelog and methodology public, raw runs private ~ evidence labels, confidence table, content hashes, and published validation failures; no independent replication ✓ every record keeps source identity and citation; daily Issues; reproducible analysis ✓ data, notebooks, and code public ✓ whole chain of questions, answers, and judgments downloadable and recomputable ~ API re-queryable, scoring internals partly gated ✓ methodology code public, vote data not re-offered ✓ outcomes summarized with CIs and thresholds, and the runtime behind them is open source ✓ daily snapshots re-fetchable under CC BY ~ table diffs live in git history and the judge is rerunnable, but the board publishes no validation of its own

Reading the matrix
#

The row that sorts the category is “the number measures”, and it doubles as a citation guide: release questions go to the tracker, present-tense quality to Artificial Analysis, evidence-reading of composites to BenchLM, benchmark-discovery and saturation questions to Benchmark Radar, trend claims to Epoch, rotated-question scores to LiveBench, quick composites to LLM Stats, preference to LMArena, did-my-tool-degrade to Margin Lab, spend to OpenRouter. The two most-quoted members are also the two most misquoted: LMArena ranks get read as capability, and gateway tokens get read as market share, when the matrix’s own rows show both measure something narrower.

Verification strength tracks institutional type, not popularity. The nonprofit (Epoch) publishes its funders, its consultations, and its data; the gateway (OpenRouter) publishes its raw daily numbers under CC BY; the side project (AI Release Tracker) lets you diff its whole corpus; the academic consortium (LiveBench) publishes every question, answer, and judgment; the researcher’s catalog (Benchmark Radar) publishes its corpus, schema, and rubric; the degradation watcher (Margin Lab) publishes its thresholds, its CIs, and an open runtime; the hallucination board (Vectara) publishes its judge model and a git-diffable table; the evidence-labeler (BenchLM) publishes its scoring rules and its own failed checks, though it holds nothing raw to check against; while the three venture-scale companies publish methodology and products but keep raw runs, votes, or scoring internals private. If you need to check the work rather than read the work, the checkable columns are the nonprofit’s, the consortium’s, the gateway’s, the side project’s, the researcher catalog’s, the degradation watcher’s, and the hallucination board’s, which is the opposite of what citation frequency would predict.

Every column carries a conflict row, and the conflicts are structural rather than scandals: the benchmark firm sells benchmarking to the benchmarked, the open benchmark doubles as its operator’s research marketing, the arena partners with the labs it ranks, the aggregator sells evaluation services, the gateway’s data is its own marketing, the catalog re-aggregates two of this category’s own members’ scores, the composite re-brands every other board’s outputs as one score of its own, the degradation watcher runs the labs’ own tools against its own subset with an advertising contact on the page, the hallucination board’s judge is the ranking vendor’s own eval model, and the nonprofit consults for the labs. Epoch’s FrontierMath episode and LMArena’s Maverick episode are the two documented failures, and both produced their institution’s strongest disclosure artifacts, which is the pattern worth watching on every refresh of this page.

Choosing from the matrix
#

  • Needing a dated record of what shipped and what the lab claimed that day: AI Release Tracker, via its flat files.
  • Picking a model or provider this week and needing quality against price and speed: Artificial Analysis, with the Endpoint Accuracy Index for the cheap-provider question.
  • Trying to learn how much evidence stands behind a composite rank, with the dataset to check it offline: BenchLM, read for its labels and its failed checks, never as a primary measurement.
  • Trying to find which benchmarks exist for a capability and whether they are saturating: Benchmark Radar, via the catalog, the saturation views, or the agent CLI.
  • Citing how fast the field moves, with downloadable data: Epoch AI.
  • Wanting an objective capability score whose every question, answer, and judgment can be downloaded and rerun: LiveBench, while watching its rotation cadence.
  • Wiring leaderboard data into an agent or a script: LLM Stats’s free API and MCP tools, accepting the gated internals.
  • Sensing which model people prefer right now: LMArena, read as a mood ring and never as a capability claim.
  • Needing a canary on the CLI and model pair you already ship with: Margin Lab’s trackers, while their baselines rebuild.
  • Asking what developers actually spend on: OpenRouter Rankings, quoted with its as-of date and its single-gateway caveat.
  • Needing a hallucination-rate read before wiring RAG or summarization: Vectara Hallucination Leaderboard, read as one vendor’s judge on one task.

Changes
#

  • 2026-09-24 - Created with six columns (AI Release Tracker, Artificial Analysis, Epoch AI, LLM Stats, LMArena, OpenRouter Rankings) when the category was seeded at the owner’s request.
  • 2026-09-29 - Coverage cells refreshed: Artificial Analysis to 679 models and LLM Stats to 399 canonical models; membership unchanged.
  • 2026-10-02 - Coverage cells refreshed: Artificial Analysis to 689 models and LLM Stats to 400 canonical models; membership unchanged.
  • 2026-10-06 - Extended from six to seven columns with LiveBench, the contamination-limiting public benchmark leaderboard, sorted between Epoch AI and LLM Stats; the intro, sorting guide, verification-strength, conflict, and choosing prose updated.
  • 2026-10-10 - Extended from seven to eight columns with Benchmark Radar, the open benchmark catalog and search engine with linked evidence, sorted between Artificial Analysis and Epoch AI; the intro, sorting guide, verification-strength, conflict, and choosing prose updated.
  • 2026-10-10 - Coverage and status cells refreshed: Artificial Analysis to 696 models, LLM Stats to 408 canonical models, AI Release Tracker to 258 releases from 11 labs per the 2026-10-07 archived homepage, and LMArena’s run-by cell to the $200M Series B at a $3.1B valuation.
  • 2026-10-11 - Extended from eight to ten columns with BenchLM, the evidence-labeled composite re-ranker (sorted between Artificial Analysis and Benchmark Radar), and Margin Lab, the daily degradation watcher over Claude Code and Codex (sorted between LMArena and OpenRouter Rankings); the intro, sorting guide, verification-strength, conflict, and choosing prose updated, and volatile coverage cells re-dated to 2026-10-11.
  • 2026-10-13 - Extended from ten to eleven columns with Vectara Hallucination Leaderboard, the HHEM-judged hallucination-rate board (sorted after OpenRouter Rankings); the intro, sorting guide, verification-strength, conflict, and choosing prose updated.

See also
#

References
#