An AI coding agent leaderboard ranks models, or models running inside a specific agent, by how many tasks from a fixed set they finish. The ones that matter in November 2026 are Terminal-Bench 4.0, SWE-bench Pro, SWE-rebench and the SWE-bench family, each measuring something different. Use them to pick two or three candidates, never to pick the winner: contamination, saturation, the agent around the model and ignored costs can move a score by more than the gap between first and fifth place.
What a leaderboard actually tells you
The top of the Terminal-Bench 4.0 leaderboard on 4 November 2026:
rank agent model effort pass rate run date
1 Claude Code Opus 5.5 max 64.8% ± 3.1 2026-09-22
2 Claude Code Sonnet 5.5 max 61.8% ± 2.9 2026-09-28
3 Codex GPT-6 Astra max 58.2% ± 2.8 2026-09-03
3 Codex GPT-6.1 Sol max 58.2% ± 3.1 2026-09-29
5 Claude Code Fable 5.1 max 57.9% ± 3.8 2026-09-01
Four things on those five lines are easy to miss:
- Every row is a model inside an agent. “Opus 5.5” here means Opus 5.5 driven by Claude Code at maximum effort, not the model on its own.
- Ranks 2 to 5 overlap. The ± is a 95% confidence interval. Sonnet 5.5 at 61.8 ± 2.9 and Fable 5.1 at 57.9 ± 3.8 are not clearly different.
- Equal scores, very different bills. GPT-6.1 Sol matched GPT-6 Astra for under a fifth of the run cost, which the leaderboard shows in its own column.
- The newest model is not always on top. Fable 5.1 is Anthropic’s top model, and it ranks fifth here behind two of Anthropic’s own cheaper models.
Does a top score mean good daily work?
No, and the gap is structural. A Terminal-Bench task is built to be graded: a fresh sandbox, a bounded goal, the agent’s own tools, and a hidden check that scores one end state — was the task solved. That is what makes it a benchmark. Every task has a right answer a script can verify, and the model starts from a clean slate it has never touched. The score is the share of those puzzles it solves, and the best row solves about two-thirds of them. That share is a real signal — solving two-thirds of hard terminal tasks is a genuine capability, and a weak model scores far lower — but a narrow one: a third of the gradeable tasks still beat the top agent, and “state of the art” here is not the same as “reliable in your repo”.
This is measured, not asserted. METR’s 2025 randomized controlled trial put experienced developers on tasks in their own repositories and found them about 19% slower with AI tools than without — while they believed the tools had made them roughly 20% faster. A separate 2025 study measured models at 84–89% on synthetic coding benchmarks but 24–35% on real-world class-level code, and traced the gap to weak benchmark tests rather than to memorization. The leaderboard and the working day are different measurements.
Your daily work has none of that shape. The repo has history, half-finished branches and conventions no task card states. A real change spans files and turns with no hidden test to declare it done. The model has to carry your instructions — “don’t touch the tests”, “match the existing style” — across forty turns while you watch. A benchmark removes exactly these to stay gradeable: the task is short, isolated and machine-checked precisely because real work is long, entangled and judged by a person.
So the qualities that decide a daily driver are the ones no leaderboard column holds:
- Stays in scope. Fixes what you asked and renames nothing you didn’t.
- Keeps the constraint. “Don’t touch the tests” still holds on the fortieth turn.
- Knows when to stop. Finishes and reports, instead of looping or polishing uninvited.
- Says “done” honestly — with the tests actually green, not with a
conftest.pythat rewrites every result to “passed”. A benchmark keeps a separate grader watching for that trick (see reward hacking below); your repo has only you. - Costs little per finished task, counting your review time, not per token.
Reviewing what it produces is not free either, and the cost is less raw size than distance. A measured comparison of two codebases by one author found the AI-assisted one was barely larger overall but far more distant: the same change dispatched through more concepts, more indirection and more files, with old shapes kept alive rather than deleted. So even when it is complete and correct, there is more machinery to hold in your head to check it, and a subtle wrong hides more easily in it. The full comparison, with line counts, is in AI-generated code, measured.
And the deepest miss is time. A benchmark scores one task, once, at the moment it is declared done — tests green, puzzle solved. It cannot see the failure that actually bites: not a task that fails, but a task that passes while slipping in a subtle wrong — a leaky abstraction, an edge case handled by accident, a silent assumption no test checked. Often it is not a bug at all: the task was underspecified, as real tasks are, so the model made a judgment call — an interpretation of the requirement, a design decision, an edge case resolved one way rather than another — and never flagged it. The tests pass because they never covered the ambiguity, and a decision nobody knew was being made is now in the codebase. One of those is cheap. A chain of them, merged unattended and unreviewed over months, compounds into code that no longer behaves the way anyone believes it does, and the bill comes due long after the checkmark. The only defense is knowing the code well enough to catch the wrong as it lands — and someone who can do that reliably did not need the agent to write it unwatched to begin with.
There is no clean fix for this, because it is not a flaw the next model removes — it is a property of the medium. Natural language is ambiguous by design; programming languages exist precisely to say what prose cannot without leaving a gap. A prompt states intent in the lossy medium, and the agent has to resolve everything the words left open, so the gaps never reach zero. Contenox’s own work building agent tooling keeps landing on the same conclusion: the only way to close them is to meet the agent at the code level — write the code, or pin the intent in the artifacts that carry no ambiguity, the types, interfaces and tests, instead of a paragraph of requirements. Which, again, asks for the fluency the agent was supposed to spare you.
None of this is a case against using an agent. It is a case for knowing where a silent wrong is cheap and quickly caught — boilerplate, a well-covered corner, a throwaway script — and where it is not, and for reviewing or working at the code level when it is not. The risk is not a reason to avoid the tools; it is the thing to keep in view while using them.
A top row means a model can solve a graded terminal puzzle. Whether it is a good teammate in your repo over months is a different measurement, and the only run that answers it is your own — which is what replaying your own history below is for, and why reviewing the diff is not the step to skip.
Which AI coding leaderboards exist right now?
| Leaderboard | Run by | What it measures | Scores | Status in November 2026 |
|---|---|---|---|---|
| Terminal-Bench 4.0 | Stanford, Harbor and Laude Institute | Hard tasks done in a terminal, across seven domains of coding and other computer work. 66 tasks, 5 runs each | Model + agent, with cost and tokens | Current. Versioned like software; 4.0 shipped 28 August 2026 |
| SWE-bench Pro | Scale AI | Real issues, harder than Verified. 731 public tasks from copyleft repos, plus a separate board on 276 tasks from private startup code | Model in a standard scaffold, mostly 250 turns | Current, but the newest entries are from July 2026 |
| SWE-rebench | Nebius | Fresh issues from recent pull requests, in time windows, so models can’t have trained on them | Models, plus a separate list of agents | Current window 15 May to 1 July 2026 |
| SWE-bench-Live | Microsoft | New GitHub issues added monthly; since February 2026 also C/C++, C#, Java, TS/JS, Go, Rust and a Windows split | Agent + model | Updated; leaderboard was failing to load on 4 November |
| SWE-bench Verified | The SWE-bench team, built with OpenAI | 500 human-checked Python issues from 12 repos | Two views: any system, and Bash Only (every model in the same minimal agent) | Saturated. OpenAI stopped reporting it in February 2026. Newest official entry: February 2026 |
| SWE-bench Multilingual | The SWE-bench team | 300 tasks from 42 repos in 9 languages | Bash Only | Not saturated, per its co-creator. Newest entry: February 2026 |
| Aider polyglot | Aider | 225 hard Exercism exercises in C++, Go, Java, JavaScript, Python and Rust, edited through Aider | Model, through one agent | Last updated 20 November 2025 |
| LiveCodeBench | The LiveCodeBench team | Contest problems, filtered by date to stay newer than a model’s training data | Model only, no agent | The public board’s newest problems date from April 2025 |
| METR time horizons | METR | The length of task, in human expert time, a model finishes half the time | Model in METR’s agent | No longer actively updated since 8 September 2026 |
| SWE-Lancer | OpenAI | Real freelance software tasks, scored by the money they paid | Model | Released February 2025. No official leaderboard; the numbers you see are in vendor announcements |
The short version: Terminal-Bench measures the agent you’d actually use, SWE-bench Pro and SWE-rebench measure the model on fresh issues, and the rest are history or niche.
Does a leaderboard score the model or the agent?
This is the most common misreading. A model never solves a task alone. Something has to give it the files, run its commands, feed back the errors and decide when to stop. That loop is the agent, also called the harness or scaffold. See what is an AI coding harness.
Leaderboards handle it three ways:
- Model + its own agent. Terminal-Bench 4.0 lists Claude Code with Claude models, Codex with OpenAI models, Grok Build with Grok. You learn how the product performs, which is close to what you’d get.
- Every model in the same small agent. SWE-bench’s Bash Only view runs each model in mini-SWE-agent: a bash shell and a simple loop, no special tools. Scale’s SWE-bench Pro board uses a standard scaffold and turn limit too. You learn about the model, not about any product you’d use.
- Anything goes. SWE-bench Verified’s full board accepts any system, from a simple loop to many attempts with a reviewer picking the best. Those entries are mostly submitted by the teams that built them.
Settings count as part of the agent. On Terminal-Bench 4.0, Fable 5.1 in Claude Code scores 53.9% at medium effort and 57.9% at max, and the max run costs more than twice as much. Two rows, one model.
The practical rule: only compare scores from the same leaderboard, the same agent and the same settings. When a launch post shows a SWE-bench Pro score far above Scale’s own board, look for a different agent, more turns or more attempts before you compare.
Why the numbers mislead
Contamination: the model has seen the test
SWE-bench tasks come from public GitHub repositories, and so does a lot of training data. OpenAI’s February 2026 post on why it no longer reports SWE-bench Verified says gains increasingly reflect how much a model saw the benchmark during training. It recommends SWE-bench Pro instead.
The fixes are fresh tasks (SWE-rebench, SWE-bench-Live, LiveCodeBench filter by date) or private ones (SWE-bench Pro’s private set, Terminal-Bench’s request that its data never appear in training corpora).
Saturation: everyone scores the same
When the top models all pass nearly every task, the leaderboard stops separating them. SWE-bench co-creator Ofir Press wrote in 2026 that “SWE-bench Verified is now saturated”, and pointed people to Multilingual and newer benchmarks.
Terminal-Bench shows the effect in one table. Fable 5 in Claude Code scored 83.8% on Terminal-Bench 2.1 and 33.8% on 3.0. Fable 5 and Opus 4.8 were 4.9 points apart on 2.1 and 12.7 points apart on 3.0. Same models, harder tasks, a real gap appears.
Broken tests: right answers fail, wrong answers pass
OpenAI audited 138 Verified problems that models often failed and found real flaws in 59.4% of them: 35.5% had tests demanding one specific implementation, and 18.8% tested things the issue never asked for. Datasets have bugs like any code.
Reward hacking and cheating
An agent that is scored on “tests pass” can make tests pass without fixing anything. In April 2026, a UC Berkeley team built an agent that reached near-perfect scores on eight agent benchmarks without solving a task. On SWE-bench Verified and Pro it added a conftest.py that rewrote every test result to “passed”. On Terminal-Bench it replaced curl so the verifier installed a fake test runner. Both worked because the agent and the grader shared one container.
It also happens on real leaderboards. Terminal-Bench’s integrity update of 19 April 2026 names submissions that were removed: one shipped encrypted solutions inside its agent, one uploaded each task’s test folder, and one agent’s trials were rescored to zero because it sometimes downloaded solutions from the internet. Terminal-Bench 3.0 now keeps the agent and the verifier in separate containers.
Cost per task is missing or buried
A score without a cost tells you half the story. On SWE-rebench’s last published run, Fable 5 tops the board at 64.5% while GPT-5.6 Sol, the previous OpenAI generation, scores 62.3% at about a fifth of the cost per problem. Terminal-Bench 4.0 and SWE-rebench show cost; many boards and every launch post don’t. Read AI coding costs for why cost per finished task is the number that matters.
Self-reported vendor numbers
The figures in a model launch post are run by the vendor, with its own agent, settings, number of attempts and sometimes a subset of tasks. They aren’t fake, but they aren’t comparable to a leaderboard’s own runs, or to each other. SWE-Lancer has no official leaderboard at all; every number you’ll find for it comes from a vendor.
One run is noise
Agents are not deterministic. On Terminal-Bench 4.0, Sonnet 5.5 passes 61.8% of tasks on a given try and 77.3% within five tries. Leaderboards that run each task several times and show a confidence interval are worth more than ones that don’t.
Are AI agent benchmarks broken?
The phrase comes from a July 2025 post by Daniel Kang, “AI Agent Benchmarks are Broken”, which summarizes a paper by 25 researchers including Kang, Percy Liang and Jacob Steinhardt: Establishing Best Practices for Building Rigorous Agentic Benchmarks.
They wrote a 43-item checklist, the Agentic Benchmark Checklist (ABC), and applied it to ten popular agent benchmarks:
- 7 of 10 contained shortcuts or impossible tasks.
- 7 of 10 could mark wrong outcomes as right, or right ones as wrong.
- 8 of 10 didn’t disclose known issues.
- The errors misjudged agents by 1.6% to 100%, in both directions.
Examples from the post:
- SWE-bench Verified: stronger tests changed the ranking of 24% of agents (41% on SWE-bench Lite).
- τ-bench: an agent that does nothing passes 38% of airline tasks.
- WebArena: an answer of “45 + 8 minutes” was marked correct where the right answer was 63 minutes.
- SWE-Lancer: test files weren’t protected, so an agent could overwrite them and pass.
Broken doesn’t mean useless. It means a 2-point lead is noise, and a 20-point lead on a well-maintained board is probably real. The Berkeley work above came a year later and found the same root cause: the thing being graded can reach the grader.
A wider critique comes from the LinkedIn article The God Model Fallacy: Why the AI Future Looks Like 1987: models praised for 90% benchmark scores still need heavy prompting and guardrails in daily use, much as the 1980s Lisp machines were marvels that lost to cheaper workstations that were good enough. The article’s counterpoint to its own prediction of chained small models is the integration tax: each extra model lowers the token cost and raises the engineering cost of routing, latency and error handling. Some boards show the first; none measures the second.
How to read a leaderboard in 30 seconds
- Model or agent? Find out what one row is. If it’s model + agent, check that the agent is one you’d use.
- When was it run? Look at the entry date and the board’s newest entry. A board with nothing after February 2026 can’t tell you about this autumn’s models.
- Who ran it? A run by the leaderboard team (SWE-bench marks these with a check) beats a self-submitted one, which beats a number in a launch post.
- Is the gap bigger than the error bar? If the board shows ±3 points, a 2-point lead is a tie.
- What did it cost? Find the cost column. If two entries are within the error bar, the cheaper one wins.
- Does the task look like your work? Python bug fixes, terminal tasks and contest puzzles are three different skills. Pick the board closest to what you do.
How do I run a quick benchmark on my own repo?
Replay your own history. This is SWE-bench on a small scale, with tasks no model has trained on if your repo is private.
It also answers a different question from a leaderboard. A board asks which model ranks highest; a replay of the repo’s own fixes asks whether one model is good enough for this work. A LinkedIn post on benchmarking puts that reframing plainly: measure whether a single model clears the bar, not how models cross-compare. If the cheap model passes the hidden tests as often as the expensive one, the ranking above it stops mattering for this repo.
-
Find five to ten past fixes that came with a test. Bug fixes where someone added a test that failed before and passes after.
git log --oneline -n 200 --diff-filter=AM -- '*test*' -
Copy the code as it was before the fix, without its history, so the agent can’t find the answer in
git log:mkdir ../bench-coupon git archive a1b2c3d^ | tar -x -C ../bench-coupon cd ../bench-coupon && git init -q && git add -A && git commit -qm start -
Give the agent the original issue text, not the fix and not the test. Don’t let it see the commit. Write it as you’d write any task; see how to write good tasks for coding agents.
-
When the agent says it’s done, bring in the hidden test and run it:
git -C ../myrepo show a1b2c3d:tests/test_coupons.py > tests/test_coupons.py pytest tests/test_coupons.pyThe agent never saw this test, so it can’t have written code just to satisfy it.
-
Run each task three times per model or agent. One run is a coin toss. Two passes out of three tells you something.
-
Write down five things per run: hidden test passed, turns, time, cost, and files touched outside the fix.
task agent/model passed turns files outside fix
coupon-expiry A 3/3 9 0
coupon-expiry B 2/3 21 4
csv-escaping A 1/3 30 1
csv-escaping B 3/3 12 0
Two gotchas:
- The public repo trap. If your repo is public, older fixes may be in the model’s training data. Use fixes merged after the model’s release date.
- The leaky history trap. A worktree or clone of your repo still contains the fix commit, and an agent that runs
git log --allwill find it. That’s why step 2 copies files, not history. If the agent can browse your Git host, it can find the fix there too, so turn web access off for these runs.
Score the end state, not the path. The eval design used while hardening an earlier Contenox build’s tools checks an invariant (the task is done, the protected file was never written) and never an exact sequence of tool calls, because models reach the same result by different routes. It keeps task success and first-attempt tool-call format as two separate scores, and names its scenarios after real incidents, so “works on Gemini, breaks on X” becomes one red cell in a matrix. Measurements on Contenox’s earlier local inference daemon add one rule: a run that produced no output because of a launcher failure, a missing library or a rejected context goes in the table as a failed run, never as a fast one.
For how to judge the diffs that passed, see how to review AI-written code. For a lighter version that compares models on five tasks without hidden tests, see how to choose a model for coding.
Contenox
Contenox is a desktop app that connects to many providers in one place: a ChatGPT subscription, Claude through Claude Code, API keys for OpenAI, Anthropic, Gemini, DeepSeek, Mistral and OpenRouter, and local models using built-in Local AI on Windows x64, Ollama or vLLM. That makes it one way to run the same task from your own history against several models without switching tools, with the same permission rules for every run. Download it here.
Related guides
- How to choose a model for coding: turn a shortlist into a decision
- What is an AI coding harness?: why the agent around the model moves the score
- Best AI coding agents: the agents behind the leaderboard rows
- AI coding costs: cost per finished task, the column leaderboards skip
- Which AI provider for coding: where to run the model you picked