🔍 Read the full analysis: FrontierCode: What AI Contributors Should Know About Maintainer Reviews on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added the FrontierCode benchmark on October 12, 2026, testing 23 AI models on repository tasks graded against maintainer review criteria. Claude Opus 5.5 led with a 52.2% score and cleared all blocking checks in 57.7% of attempts, while results and costs varied widely across models.
Artificial Analysis added FrontierCode to its model evaluations on October 12, testing 23 AI models on coding tasks graded against maintainers’ criteria for accepting a change. Claude Opus 5.5 ranked first with a score of 52.2%, but cleared every blocking check in just 57.7% of attempts, suggesting that even the leading model’s patches still need human review.
FrontierCode’s results do not measure only whether code passes tests. The benchmark checks patches across six areas: hidden tests, command checks, whether the agent’s own tests fail on the original code, adaptive tests for complex behavior, limits on file scope and diff size, and an LLM-based code review against maintainer criteria. A failed blocking check makes an attempt score zero; passing all blocking checks produces a score between 70% and 100%, depending on quality checks. The benchmark also assigns zero to attempts flagged by Cognition’s trajectory scanner for looking up solutions.
After Opus, GPT-6.1 Sol scored 49.1%, and Claude Fable 5.1 scored 49.0%. Nine models from six labs were between 43% and 52%. Results fell off below that group: Grok 4.7 scored 33.8%, MiniMax-M3 23.0%, Inkling 15.8%, Muse Glimmer 13.9%, and Nemotron 3 Ultra 10.5%. The report says nine of the 100 Main-set tasks were not solved by any model on any attempt.
Artificial Analysis ran the models through the same open-source agent harness, Stirrup, allowing up to 250 turns per attempt and three attempts per task. GPT-6 Luna (medium) served as the judge for the review step. The reported cost per task ranged from $0.09 to $12. GPT-6.1 Sol scored 49.1% at $0.81 per task, while Opus scored 52.2% at $10.15, a 3.1-point difference at substantially higher cost.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
FrontierCode focuses on a practical question for teams using AI coding agents: would a maintainer accept the patch? A change can pass ordinary tests yet still be too broad, poorly tested, regressive, or inconsistent with the repository’s expectations. By making some of these issues blocking, the benchmark highlights failure modes that can create review work even when an agent produces functioning code.
The results also caution against treating a high score on a general model leaderboard as evidence of strong repository work. In the comparison reported by Artificial Analysis, Claude Sonnet 5.5 ranked second among the listed models on the Intelligence Index but fifth on FrontierCode. DeepSeek V4.1 Flash ranked higher on FrontierCode than its general intelligence result might suggest. Model choice can depend on the task-specific evaluation, and the benchmark’s cost figures show that higher scores may come at a steep price.
These findings do not establish how often an AI-generated patch would be accepted in every real-world project. They do show that, under this benchmark’s rules, the top model failed to clear all blocking checks in more than four attempts out of ten. For teams, that makes human review and clear ownership relevant even when agents can complete coding tasks autonomously.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Tests Repository Work
FrontierCode was built by Cognition and open-source maintainers, and announced in June 2026. Version 1.1 followed in July with rules intended to prevent models from looking up solutions online. More than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy, and Mattermost. The source report says maintainers spent more than 40 hours on each task.
The full benchmark contains 150 private tasks; Artificial Analysis ran its 100-task Main set. Keeping tasks private is intended to reduce the chance that models have encountered the answers during training. The reported scores therefore reflect performance on this particular task set and evaluation setup, not a guarantee of behavior across all codebases.
Unlike a basic test-pass leaderboard, FrontierCode penalizes excessive diffs, edits outside relevant files, and unhelpful tests. Tomer Nosrati, a Celery maintainer, described the approach in Cognition’s announcement as: “Where others grade like a CI, FrontierCode grades like a tech lead.” That framing captures the benchmark’s emphasis on review quality as well as correctness.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, as quoted in Cognition’s announcement
software developer code review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the Published Results
The results cover Artificial Analysis’s 100-task Main set, using a specified harness, attempt limit, and review judge. The source material does not establish how directly scores translate to acceptance rates in live repositories, where maintainers, project rules, and task complexity differ. Nor does it provide enough information to infer a universal success rate for AI-generated pull requests from the benchmark’s attempt results.
Several models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full, and Mistral Large 4 Preview. Their absence means the published ranking is not a complete comparison of every prominent model. The report also does not specify when those models might be evaluated.
As an affiliate, we earn on qualifying purchases.
Further Model Results to Watch
The next visible development is whether additional models, including those absent from the October 13 results, are evaluated on FrontierCode. Results could change the ranking as the tested field expands. Readers comparing models should also watch for changes to the benchmark, task set, or evaluation method, since those can affect how scores compare over time.
For now, the published data gives teams a starting point for comparing review-oriented coding performance and per-task cost, not a basis for removing human review. Any practical decision should account for the repository, the consequences of a faulty patch, and the team’s own acceptance process.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates AI-generated code changes using hidden tests, command checks, test quality, scope limits, adaptive tests, and an LLM code review based on maintainer criteria. Some checks are blocking, so failing one makes the attempt score zero.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led the reported results with a 52.2% FrontierCode score. It cleared all blocking checks in 57.7% of attempts.
Does a 52.2% score mean the model’s patches are accepted 52.2% of the time?
No. The score is FrontierCode’s benchmark measure, and the source does not equate it with real-world maintainer acceptance. The separately reported 57.7% pass rate is the share of attempts that cleared every blocking check under the benchmark setup.
How much did the leading models cost per task?
Artificial Analysis reported $10.15 per task for Claude Opus 5.5 and $0.81 for GPT-6.1 Sol. Costs are specific to the reported evaluation and should not be treated as a universal price for other workloads.
Are all major coding models included?
No. As of October 13, 2026, the source report listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full, and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Columbus Day / Indigenous Peoples' Day Picks
long weekend sales
As an affiliate, we earn on qualifying purchases.
