Senior SWE-Bench
We treat agents like senior engineers, so why evaluate them like junior engineers?
Senior engineers build features without over-specified requirements
Senior SWE-Bench feature tasks have realistic instructions that read like natural language messages rather than over-specified requirements. To reliably evaluate these tasks, we introduce a validation agent which uses expert-designed recipes to write behavioral tests that adapt to submitted solutions.
Senior engineers solve bugs that require runtime investigation from behavioral reports
Senior SWE-Bench bug tasks reflect tricky user reports and focus on investigation, from starting services to debugging subtle runtime issues. They are sourced from PRs that needed significant runtime investigation to solve (e.g. logs, profiling data, reproduction steps).
Senior engineers ship the right code without being told to
Senior SWE-Bench scores tasteful solves by combining runtime correctness tests with several quality metrics based on observed codebase practices. In addition, verifiers and validation can test against load-bearing codebase practices that go unstated in instructions.
Leaderboard
- 1Claude Fable 5Mini-SWE-Agent · high34.7%@1
- 2Claude Opus 5Mini-SWE-Agent · high34.7%@1
- 3GPT-5.6 SolMini-SWE-Agent · xhigh34.7%@1
- 4Claude Opus 4.8Mini-SWE-Agent · max30.5%@1
- 5GPT-5.5Mini-SWE-Agent · xhigh29.5%@1
- 6Claude Sonnet 5Mini-SWE-Agent · max28.0%@1
- 7GPT-5.6 TerraMini-SWE-Agent · high27.4%@1
- 8Grok 4.5Mini-SWE-Agent · medium23.2%@1
- 9MiniMax M3Mini-SWE-Agent · adaptive21.1%@1
- 10Kimi K3Mini-SWE-Agent · max20.2%@1
| # | Model | Effort | |
|---|---|---|---|
| 1 | Claude Fable 5 | high | 34.7% |
| 2 | Claude Opus 5 | high | 34.7% |
| 3 | GPT-5.6 Sol | xhigh | 34.7% |
| 4 | Claude Opus 4.8 | max | 30.5% |
| 5 | GPT-5.5 | xhigh | 29.5% |
| 6 | Claude Sonnet 5 | max | 28.0% |
| 7 | GPT-5.6 Terra | high | 27.4% |
| 8 | Grok 4.5 | medium | 23.2% |
| 9 | MiniMax M3 | adaptive | 21.1% |
| 10 | Kimi K3 | max | 20.2% |
The top-performing frontier models fail to complete tasks with senior-level correctness and taste over 65% of the time.
Tasks
Senior SWE-Bench tasks are sourced from PRs in repos spanning libraries to multi-service applications, authored by engineers with hundreds of commits in their respective repos. We focus on multi-phase, multi-stack feature PRs and bug/performance PRs with significant runtime investigation. For more on task design, read the blog post
More naturally under-specified instructions
Senior SWE-Bench tasks reflect natural communication with agents, with a median instruction length 31% that of SWE-Bench Pro.
More diverse task scope
Senior SWE-Bench feature tasks can span across multiple services, with an average of 11 files touched per feature task.
Longer task horizon
Senior SWE-Bench tasks are designed to be long-horizon, requiring hundreds of steps for even the strongest agents.
More naturally under-specified instructions
Senior SWE-Bench tasks reflect natural communication with agents, with a median instruction length 31% that of SWE-Bench Pro.
More diverse task scope
Senior SWE-Bench feature tasks can span across multiple services, with an average of 11 files touched per feature task.
Longer task horizon
Senior SWE-Bench tasks are designed to be long-horizon, requiring hundreds of steps for even the strongest agents.
Reference-solution SLOC & files are measured identically across all three benchmarks. Instruction length excludes harness boilerplate. Token and step counts for other benchmarks are based on their self-reported metrics.
Citing Senior SWE-Bench
@misc{seniorswebench2026,
title = {Senior SWE-bench: evaluating agents on their ability to act as senior engineers},
author = {Henry R. Ehrenberg and Vincent S. Chen and Austin W. Hanjie and Gabriel Orlanski and Frederic Sala and Karthik R. Narasimhan},
year = {2026},
url = {https://senior-swe-bench.snorkel.ai},
}