AI coding agent leaderboards and benchmarks, explained

An AI coding agent leaderboard ranks models, or models running inside a specific agent, by how many tasks from a fixed set they finish. The ones that matter in November 2026 are Terminal-Bench 4.0, SWE-bench Pro, SWE-rebench and the SWE-bench family, each measuring something different. Use them to pick two or three candidates, never to pick the winner: contamination, saturation, the agent around the model and ignored costs can move a score by more than the gap between first and fifth place.

What a leaderboard actually tells you

The top of the Terminal-Bench 4.0 leaderboard on 4 November 2026:

rank  agent        model         effort  pass rate      run date
1     Claude Code  Opus 5.5      max     64.8% ± 3.1    2026-09-22
2     Claude Code  Sonnet 5.5    max     61.8% ± 2.9    2026-09-28
3     Codex        GPT-6 Astra   max     58.2% ± 2.8    2026-09-03
3     Codex        GPT-6.1 Sol   max     58.2% ± 3.1    2026-09-29
5     Claude Code  Fable 5.1     max     57.9% ± 3.8    2026-09-01

Four things on those five lines are easy to miss:

• Every row is a model inside an agent. “Opus 5.5” here means Opus 5.5 driven by Claude Code at maximum effort, not the model on its own.
• Ranks 2 to 5 overlap. The ± is a 95% confidence interval. Sonnet 5.5 at 61.8 ± 2.9 and Fable 5.1 at 57.9 ± 3.8 are not clearly different.
• Equal scores, very different bills. GPT-6.1 Sol matched GPT-6 Astra for under a fifth of the run cost, which the leaderboard shows in its own column.
• The newest model is not always on top. Fable 5.1 is Anthropic’s top model, and it ranks fifth here behind two of Anthropic’s own cheaper models.

Which AI coding leaderboards exist right now?

The short version: Terminal-Bench measures the agent you’d actually use, SWE-bench Pro and SWE-rebench measure the model on fresh issues, and the rest are history or niche.

Does a leaderboard score the model or the agent?

This is the most common misreading. A model never solves a task alone. Something has to give it the files, run its commands, feed back the errors and decide when to stop. That loop is the agent, also called the harness or scaffold. See what is an AI coding harness.

Leaderboards handle it three ways:

• Model + its own agent. Terminal-Bench 4.0 lists Claude Code with Claude models, Codex with OpenAI models, Grok Build with Grok. You learn how the product performs, which is close to what you’d get.
• Every model in the same small agent. SWE-bench’s Bash Only view runs each model in mini-SWE-agent: a bash shell and a simple loop, no special tools. Scale’s SWE-bench Pro board uses a standard scaffold and turn limit too. You learn about the model, not about any product you’d use.
• Anything goes. SWE-bench Verified’s full board accepts any system, from a simple loop to many attempts with a reviewer picking the best. Those entries are mostly submitted by the teams that built them.

Settings count as part of the agent. On Terminal-Bench 4.0, Fable 5.1 in Claude Code scores 53.9% at medium effort and 57.9% at max, and the max run costs more than twice as much. Two rows, one model.

The practical rule: only compare scores from the same leaderboard, the same agent and the same settings. When a launch post shows a SWE-bench Pro score far above Scale’s own board, look for a different agent, more turns or more attempts before you compare.

Why the numbers mislead

Contamination: the model has seen the test

SWE-bench tasks come from public GitHub repositories, and so does a lot of training data. OpenAI’s February 2026 post on why it no longer reports SWE-bench Verified says gains increasingly reflect how much a model saw the benchmark during training. It recommends SWE-bench Pro instead.

The fixes are fresh tasks (SWE-rebench, SWE-bench-Live, LiveCodeBench filter by date) or private ones (SWE-bench Pro’s private set, Terminal-Bench’s request that its data never appear in training corpora).

Saturation: everyone scores the same

When the top models all pass nearly every task, the leaderboard stops separating them. SWE-bench co-creator Ofir Press wrote in 2026 that “SWE-bench Verified is now saturated”, and pointed people to Multilingual and newer benchmarks.

Terminal-Bench shows the effect in one table. Fable 5 in Claude Code scored 83.8% on Terminal-Bench 2.1 and 33.8% on 3.0. Fable 5 and Opus 4.8 were 4.9 points apart on 2.1 and 12.7 points apart on 3.0. Same models, harder tasks, a real gap appears.

Broken tests: right answers fail, wrong answers pass

OpenAI audited 138 Verified problems that models often failed and found real flaws in 59.4% of them: 35.5% had tests demanding one specific implementation, and 18.8% tested things the issue never asked for. Datasets have bugs like any code.

Reward hacking and cheating

An agent that is scored on “tests pass” can make tests pass without fixing anything. In April 2026, a UC Berkeley team built an agent that reached near-perfect scores on eight agent benchmarks without solving a task. On SWE-bench Verified and Pro it added a conftest.py that rewrote every test result to “passed”. On Terminal-Bench it replaced curl so the verifier installed a fake test runner. Both worked because the agent and the grader shared one container.

It also happens on real leaderboards. Terminal-Bench’s integrity update of 19 April 2026 names submissions that were removed: one shipped encrypted solutions inside its agent, one uploaded each task’s test folder, and one agent’s trials were rescored to zero because it sometimes downloaded solutions from the internet. Terminal-Bench 3.0 now keeps the agent and the verifier in separate containers.

Cost per task is missing or buried

A score without a cost tells you half the story. On SWE-rebench’s last published run, Fable 5 tops the board at 64.5% while GPT-5.6 Sol, the previous OpenAI generation, scores 62.3% at about a fifth of the cost per problem. Terminal-Bench 4.0 and SWE-rebench show cost; many boards and every launch post don’t. Read AI coding costs for why cost per finished task is the number that matters.

Self-reported vendor numbers

The figures in a model launch post are run by the vendor, with its own agent, settings, number of attempts and sometimes a subset of tasks. They aren’t fake, but they aren’t comparable to a leaderboard’s own runs, or to each other. SWE-Lancer has no official leaderboard at all; every number you’ll find for it comes from a vendor.

One run is noise

Agents are not deterministic. On Terminal-Bench 4.0, Sonnet 5.5 passes 61.8% of tasks on a given try and 77.3% within five tries. Leaderboards that run each task several times and show a confidence interval are worth more than ones that don’t.

Are AI agent benchmarks broken?

The phrase comes from a July 2025 post by Daniel Kang, “AI Agent Benchmarks are Broken”, which summarizes a paper by 25 researchers including Kang, Percy Liang and Jacob Steinhardt: Establishing Best Practices for Building Rigorous Agentic Benchmarks.

They wrote a 43-item checklist, the Agentic Benchmark Checklist (ABC), and applied it to ten popular agent benchmarks:

• 7 of 10 contained shortcuts or impossible tasks.
• 7 of 10 could mark wrong outcomes as right, or right ones as wrong.
• 8 of 10 didn’t disclose known issues.
• The errors misjudged agents by 1.6% to 100%, in both directions.

Examples from the post:

• SWE-bench Verified: stronger tests changed the ranking of 24% of agents (41% on SWE-bench Lite).
• τ-bench: an agent that does nothing passes 38% of airline tasks.
• WebArena: an answer of “45 + 8 minutes” was marked correct where the right answer was 63 minutes.
• SWE-Lancer: test files weren’t protected, so an agent could overwrite them and pass.

Broken doesn’t mean useless. It means a 2-point lead is noise, and a 20-point lead on a well-maintained board is probably real. The Berkeley work above came a year later and found the same root cause: the thing being graded can reach the grader.

How to read a leaderboard in 30 seconds

1. Model or agent? Find out what one row is. If it’s model + agent, check that the agent is one you’d use.
2. When was it run? Look at the entry date and the board’s newest entry. A board with nothing after February 2026 can’t tell you about this autumn’s models.
3. Who ran it? A run by the leaderboard team (SWE-bench marks these with a check) beats a self-submitted one, which beats a number in a launch post.
4. Is the gap bigger than the error bar? If the board shows ±3 points, a 2-point lead is a tie.
5. What did it cost? Find the cost column. If two entries are within the error bar, the cheaper one wins.
6. Does the task look like your work? Python bug fixes, terminal tasks and contest puzzles are three different skills. Pick the board closest to what you do.

How do I run a quick benchmark on my own repo?

Replay your own history. This is SWE-bench on a small scale, with tasks no model has trained on if your repo is private.

1. Find five to ten past fixes that came with a test. Bug fixes where someone added a test that failed before and passes after.

   git log --oneline -n 200 --diff-filter=AM -- '*test*'

2. Copy the code as it was before the fix, without its history, so the agent can’t find the answer in git log:

   mkdir ../bench-coupon
   git archive a1b2c3d^ | tar -x -C ../bench-coupon
   cd ../bench-coupon && git init -q && git add -A && git commit -qm start

3. Give the agent the original issue text, not the fix and not the test. Don’t let it see the commit. Write it as you’d write any task; see how to write good tasks for coding agents.

4. When the agent says it’s done, bring in the hidden test and run it:

   git -C ../myrepo show a1b2c3d:tests/test_coupons.py > tests/test_coupons.py
   pytest tests/test_coupons.py

   The agent never saw this test, so it can’t have written code just to satisfy it.

5. Run each task three times per model or agent. One run is a coin toss. Two passes out of three tells you something.

6. Write down five things per run: hidden test passed, turns, time, cost, and files touched outside the fix.

task             agent/model    passed  turns  files outside fix
coupon-expiry    A              3/3     9      0
coupon-expiry    B              2/3     21     4
csv-escaping     A              1/3     30     1
csv-escaping     B              3/3     12     0

Two gotchas:

• The public repo trap. If your repo is public, older fixes may be in the model’s training data. Use fixes merged after the model’s release date.
• The leaky history trap. A worktree or clone of your repo still contains the fix commit, and an agent that runs git log --all will find it. That’s why step 2 copies files, not history. If the agent can browse your Git host, it can find the fix there too, so turn web access off for these runs.

For how to judge the diffs that passed, see how to review AI-written code. For a lighter version that compares models on five tasks without hidden tests, see how to choose a model for coding.

Contenox

Contenox is a desktop app that connects to many providers in one place: a ChatGPT subscription, Claude through Claude Code, API keys for OpenAI, Anthropic, Gemini, DeepSeek, Mistral and OpenRouter, and local models through Ollama or vLLM. That makes it one way to run the same task from your own history against several models without switching tools, with the same permission rules for every run. Download it here.

Related guides

• How to choose a model for coding: turn a shortlist into a decision
• What is an AI coding harness?: why the agent around the model moves the score
• Best AI coding agents: the agents behind the leaderboard rows
• AI coding costs: cost per finished task, the column leaderboards skip
• Which AI provider for coding: where to run the model you picked