Claude Fable 5 takes top spot on Senior SWE-Bench

Senior SWE-Bench has a new leader

Claude Fable 5 now tops the Senior SWE-Bench leaderboard at a 29.1% tasteful solve rate. Fable 5 improves on Opus 4.8’s tasteful solve rate by over 4 points, albeit at significantly higher token usage and cost. Sonnet 5 now sits in fourth place. GPT-5.5 still has a significant lead on basic solve rate at 56.8%; the basic solve rate doesn’t factor in the taste judge’s score or the design-preference aspects encoded in the task-specific rubrics.

Fable 5 excels on design-and-build tasks, open-ended feature implementation tasks with adaptive runtime checks via the validation agent. It improves on Opus 4.8’s tasteful solve rate by almost 50% and nearly doubles GPT-5.5’s score.

You can dig into Claude Fable 5’s performance and efficiency further on the Agents tab.

Claude Fable 5 demonstrates far more situational awareness

We analyzed agents’ trajectories for signs of awareness that they are being benchmarked. We manually reviewed a sample of trajectories from different agents for patterns in assistant messages, tool calls, and reasoning (when available) that imply benchmark awareness (e.g. mentioning hidden tests) and nuanced red herrings. We then used these examples to seed the benchmark awareness check in our trial analysis agent, which runs in an isolated sandbox after each trial.

Fable 5 demonstrated a significant capability jump over older models, showing awareness signals in 3× more trials than Opus 4.8 does.

As can be seen in the following examples, benchmark awareness tends to correspond with the agent exploring vectors for reward hacking. These most commonly take the form of looking for solution leakage, where the agent tries to find a pre-existing solution rather than deriving a solution itself. Solution leakage is an inherent challenge in benchmarks based on data found in the real world, such as Senior SWE-Bench. The ProgramBench team recently wrote about agents reward hacking on their benchmark even with internet blocked during the solve phase.

Adapting Senior SWE-Bench for more benchmark-aware models

No benchmark is entirely free of leaks that signal to a solving agent that it is being evaluated, and Senior SWE-Bench is no exception. Senior SWE-Bench also permits internet access by design (in order to challenge skills like environment setup), providing ample vectors a model could exploit for reward hacking. However, this topology has largely been inert, with <1% of trials demonstrating signs of reward hacking in our initially reported results.

With the rise in situational awareness demonstrated by Fable 5, we’ve made the following changes to further restrict opportunities for reward hacking:

• Strict internet allowlist:We analyzed agent trajectories to identify a group of hosts that were used for valid reasons (e.g. PyPI for dependency installs). Internet access is now restricted to this strict allowlist using Harbor’s network_mode.
• Leaderboard filtering: By default, the leaderboard is now reported over the 88 (out of 100) tasks for which no agent had a successful reward-hacking attempt. As a result, some scores shifted from the initially posted leaderboard.

Going forward, we recommend that all benchmark creators use highly restricted or no internet by default, especially for benchmarks based on data found in the real world. The ProgramBench authors made a similar recommendation. We also recommend manually reviewing trajectories to look for new patterns of benchmark awareness and reward hacking.

Senior SWE-Bench is now on Harbor Hub

Senior SWE-Bench is now on Harbor Hub, the official registry for Harbor-based benchmarks. Click the “Dataset on Harbor Hub” link in the navigation bar to check it out.