The best coding agent scored 38.8% on real code

The email hit my inbox in September 2026, announcing Real-SWE, a new benchmark for coding agents. My eye immediately went to the headline number: 38.8%. That's what the best agent, Fable 5.1, achieved on real enterprise codebases, as detailed in the Real-SWE benchmark report.

For months, engineering leaders have been asking me, "Which coding agent should we buy?" They usually bring up SWE-bench scores, which often sit in the 80% range for top models. So, seeing a 38.8% for a frontier model on actual production code, not simplified public datasets, felt like a splash of cold water. It's a wake-up call for anyone thinking an AI agent is about to replace a significant chunk of their engineering team.

The Real-SWE benchmark didn't just test models. It tested eight model-and-harness combinations, running ten real-world tasks taken verbatim from production systems. These weren't toy problems. They required changes across a median of 11 files, with instruction prompts averaging 1,742 characters. This is a far cry from the six files typically seen in other benchmarks, and it reflects the kind of complex, interconnected work we do every day.

The leaderboard shows a deeper truth

Here's how the top contenders stacked up:

This table isn't just about the models. Look at the "Harness" column. GLM 5.3, running in Claude Code, scored better than GPT-5.6 Sol running in Codex CLI. This tells me that the orchestration layer, the "how" the model interacts with the codebase and tools, is as critical as the model itself. Most "model X vs model Y" comparisons we've seen this year are underspecified if they don't account for the harness. It's like comparing two engines without mentioning the car they're in.

Missed requirements are a specification problem

The most striking finding from Real-SWE wasn't the low scores, but the type of failure. Across every model, the dominant failure mode was "MISSED REQUIREMENTS". Agents weren't failing because their code was broken, or because tests didn't pass. They failed because they didn't do all of what was asked.

This is a specification problem, not purely a reasoning problem. It suggests that even the best models struggle with the nuances of a complex request, especially when that request is in natural language and implicitly relies on context from a large codebase. We're used to writing tickets for humans, who can ask clarifying questions, infer context, or spot unstated assumptions. Agents don't do that effectively yet.

What does this mean for how you write tickets for agents? It means being excruciatingly precise. You need to break down tasks into atomic steps, explicitly state every constraint, and define every edge case. We're talking about a level of detail that would feel micromanaging if you wrote it for an experienced human engineer. For an agent, it's a necessity.

And here's another kicker: higher API cost didn't correlate with better performance. Paying more for a "smarter" model didn't buy more accuracy. The report also found that duration didn't help either. 71.4% of rollouts under ten minutes failed versus 73.4% of longer ones. That's essentially flat. More time and more money didn't fix the fundamental issue of missing requirements.

Around the same time Real-SWE was published, GitHub released a research preview of Project HydraFusion in Copilot CLI. InfoQ reported on it, detailing its multi-model orchestration approach: Single, Cascade, and Critique patterns. GitHub claimed a 4.9 percentage-point quality gain on TerminalBench 2.1 at 67% lower estimated cost versus Claude Opus 5, and a 0.1pp tie on its internal CheckpointBench at 65% lower estimated cost. These are GitHub's own reported figures, and the costs are "estimated," as seen in their community discussion. The industry is clearly optimising for routing and cost efficiency, which makes sense, but Real-SWE shows that the ceiling on actual task completion for real-world enterprise code is still surprisingly low, even for a model like OpenAI's GPT-6 Astra.

Build your own benchmark

Given these findings, relying solely on public benchmarks to choose a coding agent feels naive. The best way to evaluate an agent is to test it on your own codebase, with your own specific problems.

Here's a simple, concrete recipe for an internal evaluation:

1. Pick your tasks

   Go through your Git history and find ten merged pull requests that involved mid-sized feature work or bug fixes, not just trivial changes. These should be changes you've already shipped.

2. Strip the diffs

   For each chosen PR, grab the original ticket text, bug report, or feature request. Crucially, remove the actual code diff from the PR. You want the problem statement, not the solution.

3. Run the rollouts

   Set up the agent you're evaluating. For each of your ten tasks, run the agent eight separate times. Each run is a "rollout," giving you 80 attempts in total, just like Real-SWE. This helps account for non-deterministic behavior.

4. Score on requirements met

   Don't just check if the code compiles or if a few tests pass. Compare the agent's generated solution against the original requirements of the ticket. Did it address every point? Did it introduce new issues? Score it on how many of the requirements it actually met, not just whether the code is "valid".

This approach gives you a realistic understanding of how an agent performs in your environment, with your code, and your style of task specification. It's messy, it takes effort, but it's the only way to get a real signal on whether an agent can actually help your team.

The 38.8% number from Real-SWE isn't the whole story. The real takeaway is that agents are still struggling with the fundamental act of understanding complex human instructions and translating them into comprehensive, requirement-fulfilling code. It's not about how "smart" the model is in isolation, but how well it integrates into our messy, real-world development process, driven by the quality of our prompts and the robustness of its harness. If we want better results, we need to focus on refining our inputs and our tooling, not just waiting for the next bigger model.