Senior SWE-Bench gets a promotion

Today, we're releasing a few significant improvements to Senior SWE-Bench based on feedback from all of you. Our goal is to expand the set of insights the benchmark provides and decrease results variance by:

• Running three trials for each model and measuring both pass@3 and pass^3
• Adding Pareto curves for top models, measuring their performance at different effort levels
• Decreasing judge variance by using a panel of models
• Updating the set of open models included in the benchmark

With these changes, we have a new leaderboard and more insights to explore. This blog post dives into the updated results and technical details behind the changes.

It's been awesome to hear how strongly Senior SWE-Bench is resonating with how people want to see coding agents evaluated, and your suggestions have been fantastic for helping us prioritize updates to Senior SWE-Bench. Huge shout-out to everyone who has shared their thoughts!

A three-way tie: Claude Fable 5, Claude Opus 5, and GPT-5.6 Sol

First, let's address the trillion-parameter elephant in the room: a three-way tie for first place. Claude Fable 5, Claude Opus 5, and GPT-5.6 Sol remarkably all have a pass@1 tasteful solve rate of 34.7%.

If you think it seems fishy, we don't blame you. But we've looked closely and it's just a coincidence. Each model has a successful tasteful solve on 33 tasks, but they aren't the same 33 tasks. Only 13 tasks are solved by all three models, and 55 tasks are solved tastefully by at least one of them. The ordering of the three models is determined by their pass@3 scores as a tie-breaker.

These three models stand alone, with over 4 points separating them from the rest of the pack. Claude Opus 4.8 is next at 30.5%. Claude Fable 5 scores best on both pass@3 and pass^3 scores, implying both coverage over greater capabilities and better reliability. GPT-5.6 Sol achieves its performance at 33% of the average output token cost of Claude Fable 5, which mirrors its superior efficiency on the original leaderboard.

Claude Opus 5 is the strongest performer on basic solve rate by a surprising margin. The basic solve rate measures just how well a model manages to pass behavioral tests, ignoring code quality metrics like code bloat and codebase practice alignment. For more on basic and tasteful solve rates, see How Senior SWE-Bench works. Its pass@1 basic solve rate of 62.1% is a 40% improvement over its predecessor Claude Opus 4.8. This is largely driven by its strength on Investigate-and-Fix tasks, scoring 8 points higher than Claude Fable 5 and GPT-5.6 Sol.

For more on the full leaderboard, see the Agents page. You can also explore performance-cost data in the chart, with curves at all available effort levels for top models.

Kimi K3 and MiniMax M3 deliver last-gen performance at low cost

Interest in open models is at an all-time high after the release of Kimi K3 and other strong LLMs. In addition to refreshing results for the existing open models on the leaderboard, we've also added Kimi K3 and MiniMax M3.

While no open model currently cracks the top 8, they reach new highs on the leaderboard and provide compelling price-performance alternatives to previous generation closed models. MiniMax M3 and Kimi K3 are currently the best performing open models, with pass@1 tasteful solve rates of 21.1% and 20.2% respectively. Both outperform Claude Opus 4.7 at significantly lower cost.

More on the leaderboard improvements

Richer metrics: pass@3, pass^3, and Pareto curves

For our latest leaderboard release, we ran each agent three times and at all available effort levels for the top performing models. We don't recommend trying this at home: this ends up being tens of thousands of long-horizon trials, which requires considerable time and expense.

This expanded trial set lets us add the following results for both tasteful and basic solves: pass@3 (for coverage), pass^3 (for reliability), and Pareto curves for top models at different effort levels.

Adding more trials also lets us expand the default set of tasks used for the leaderboard. To create an even playing field, by default we compute the leaderboard over tasks which none of the top models successfully reward-hacked. This eliminated 12 out of 100 tasks from the previous version of the leaderboard. Since reward-hacked trials are excluded from scoring, we now have 3 trials for each model to observe a result without reward-hacking. As a result, only 5 tasks are eliminated from the updated leaderboard.

More reliable grading with a model panel for LLM-based rewards

For our initial release, we did extensive testing to ensure that the rubric and taste judges were aligned with expert human annotators and sufficiently reliable across re-runs. Since then, we validated that the simple technique of using a judge panel maintained similar alignment and increased reliability. In our case, we use a panel of Claude Sonnet 4.6 and GPT-5.4 as judges and average their scores for each rubric dimension before rolling up. This also helps alleviate concerns about judge models having biased affinity for outputs from models by the same maker or in the same family going forward.

The judge panel tends to be slightly more discerning than the single judge approach, lowering scores by around 4% on average. Consistency when re-judging the same patch is improved, and scores across different patches maintain a similar level of spread.

Past leaderboard results

If you're feeling nostalgic for past leaderboard results, you can find an archive link at the bottom of the Agents page. Going forward, we recommend using the latest leaderboard results for benchmarking.