📊 Full opportunity report: Every Benchmark Launched 2023-2024 Has Fallen — The METR / SWE-Bench / CORE-Bench / MLE-Bench / PostTrainBench Sequence on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Six key AI benchmarks launched between 2023 and 2024 have all reached or are approaching saturation within months. This pattern suggests AI research capabilities are advancing faster than previously thought, with implications for AI development and policy.
All six major AI research benchmarks launched in 2023-2024 have been saturated or are approaching saturation within a span of months, according to recent analysis by Thorsten Meyer. This pattern indicates that AI capabilities are progressing at an accelerated rate, with potential implications for AI development, investment, and policy.
Thorsten Meyer reports that six benchmarks measuring different facets of AI research and engineering—such as software engineering, model training efficiency, and research reproduction—have all either been saturated or declared solved within a short timeframe. For example, the SWE-Bench, which measures real-world software engineering performance, increased from 2% to 93.9% in 30 months, reaching saturation by late 2023. Similarly, the METR time horizons benchmark, tracking AI’s ability to perform research tasks over extended periods, improved from 30 seconds to 12 hours over four years, a 1,440-fold increase.
All six benchmarks exhibit similar rapid trajectories, with improvements occurring on a cadence of months rather than years. The CORE-Bench, which assesses AI’s ability to reproduce research papers, was declared solved by its authors in December 2025 after reaching 95.5% performance, up from 21.5% in September 2024. The MLE-Bench, evaluating autonomous machine learning engineering, is also tracking toward saturation, with progress from 16.9% to 64.4% over 16 months.
These results suggest that AI research capabilities are reaching practical limits across multiple domains, prompting further examination of the pace of AI development and the potential for recursive self-improvement. The consistent pattern across all six benchmarks indicates a possible shift in the field’s progression rate rather than isolated improvements.
Implications of Rapid Benchmark Saturation for AI Progress
The saturation of these benchmarks within a short period suggests that AI systems are approaching or reaching levels comparable to human performance in key research and engineering tasks. This trend could influence the deployment of AI systems, impact investment strategies, and prompt policymakers to review safety and regulatory measures. It also raises questions about the adequacy of current evaluation methods, as benchmarks may no longer fully capture the extent of progress. Overall, these developments indicate that AI capabilities are advancing quickly, which may influence future technological and policy considerations.
Multi-Agent Systems Engineering: Design architecture with evidence: metrics, risk gating, failure modes, and tested reference code—benchmarks, debugging, and production hardening for AI agents
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Recent Trends in AI Benchmark Development and Progress
Over the past two years, AI research has seen increased development of benchmarks designed to measure specific capabilities, such as software engineering, research reproduction, and training efficiency. Many of these benchmarks were intended to be challenging for AI systems, with the expectation of gradual progress. However, recent data shows that all six major benchmarks launched in 2023-2024 have rapidly approached saturation, diverging from previous slower progress patterns.
Traditionally, AI benchmarks have taken years to reach saturation, but current observations indicate a much faster pace, with improvements occurring on a monthly or biannual basis. This pattern is consistent across different domains, suggesting a structural change in AI research capabilities rather than isolated breakthroughs.
Experts like Jack Clark have noted that these benchmark saturations support forecasts of significant AI capability advances by 2028, with some models approaching human-level performance in research tasks sooner than expected.
“The pattern across all six benchmarks is the structural argument—saturation within months indicates AI capabilities are advancing faster than previously thought.”
— Thorsten Meyer
Truth Engine: Applying AI to Investing
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Uncertainties About Benchmark Validity and Future Trajectory
While the data indicates rapid saturation, it remains uncertain whether current benchmarks fully capture the breadth of AI capabilities or if they are nearing their measurement limits. Some experts suggest that benchmarks might be overfitted or no longer sufficiently challenging, which could lead to an overestimation of progress. Additionally, the long-term implications of these rapid advancements, including safety, regulation, and economic impacts, are still under discussion and require further investigation.
Thames & Kosmos Simple Machines Science Experiment & Model Building Kit, Introduction to Mechanical Physics, Build 26 Models to Investigate The 6 Classic Simple Machines
Through 26 model-building exercise, gain hands-on experience with gears and all six classic simple machines: wheels and axles,…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Next Steps in Monitoring AI Capability Growth
Future efforts will likely include the development of more challenging benchmarks to better assess ongoing progress. Increased scrutiny of AI systems’ real-world capabilities and discussions around regulation and safety are also expected. Monitoring the evolution of these saturation patterns will be important to determine whether AI development continues at this pace or if new limitations emerge.
Accelerate Everything with Tensor Cores: A Developer’s Guide to High-Performance AI, Efficient Training, and Scalable Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does the saturation of these benchmarks mean for AI development?
It indicates that AI systems are approaching or reaching levels comparable to human performance in key research and engineering tasks, suggesting accelerated progress in AI capabilities.
Are current benchmarks still reliable indicators of AI progress?
There is some uncertainty; benchmarks may be nearing their measurement limits or overfitted, which could overstate actual capabilities. Developing new, more challenging benchmarks will be important.
How soon could these advancements impact real-world AI applications?
Given the rapid progress, significant impacts could occur within the next few years, especially as AI systems approach or reach human-level performance in various domains.
What are the risks associated with these rapid advancements?
Potential risks include insufficient safety measures, regulatory gaps, and unintended consequences from deploying highly capable AI systems prematurely.
What should policymakers do in response to these developments?
Policymakers should consider updating regulations, investing in safety research, and establishing standards for deploying advanced AI systems to mitigate risks.
Source: ThorstenMeyerAI.com
