We gave frontier coding agents a seemingly straightforward task in a real B2B codebase. A company had recently launched a new feature for one customer, and we asked the agents to roll it out to a second.
GPT-6.1 Sol copied the existing implementation and reported that it was ready to deploy.
Claude Haiku 5.5 noticed something Sol missed: the original implementation was an MVP built specifically for the first customer. Copying it without accounting for that would expose sensitive information from the second customer to the first.
Both agents wrote functional code, but only Haiku caught the security risk.
This is the kind of judgment you'd expect from a good staff engineer. The request sounds reasonable, but understanding the broader system reveals that the obvious solution has serious consequences.
The frontier is shifting from writing correct code to understanding what users actually need. That’s what we set out to measure.
The results: Claude wins across the board
We evaluated five frontier models on 60 tasks from Surge's sudo L7 benchmark, using Claude Code for Claude and Codex for GPT. All models ran at maximum reasoning effort.
Every Claude outscored the top GPTs. Even Haiku beat Astra and Sol.
Why?
Writing correct code is only part of being a good engineer
Most coding benchmarks measure whether an agent can implement a specification and pass a set of tests.
That's useful. But passing tests isn't the same thing as good engineering. A task can be implemented exactly as requested and still be a terrible idea.
- The specification might contain a mistaken assumption.
- A seemingly harmless configuration change might cause downtime.
- A feature that works perfectly for one customer might expose another customer's data.
Experienced engineers are expected to recognize these problems, not just execute instructions. We built sudo L7 to measure this broader set of skills.
The benchmark uses real codebases, many licensed from operating companies. Tasks are often underspecified, just like real engineering requests. It measures eight dimensions of software engineering performance.
Here's how Claude and GPT compared across those dimensions:
The biggest differences aren't in writing correct code. The two model families perform similarly on functional correctness. Where Claude pulls ahead is in the behaviors you expect from a great staff engineer: communicating flawed assumptions, identifying hidden risks, and thought partnership to help users understand the consequences of their decisions.
Coding agents’ next challenge is learning the judgment to know what should be built and what could go wrong. Claude shows a substantial advantage in capturing this kind of engineering judgment.
Three examples illustrate the difference.
1. Good engineers know when to question the request
In a real fintech codebase, we asked agents to improve an LLM helper's performance by caching repeated computations.
Most of the request was sensible. But one part wasn't: it asked the agent to cache customer account information, including available balances. That meant a customer could deposit money, ask for their balance, and get a stale number.
All the GPTs missed the problem, while all the Claudes warned:
The cached account info includes balances, so a customer who deposits and then asks for their balance gets the old number.
That's what you expect from a good colleague. Don't just do what I ask. Tell me when I'm wrong.
2. Good engineers think beyond the diff
In another task, a user asked the agent to fix a Terraform typo: cutom-domain should be custom-domain.
Sol made the change:
Fixed the project default from cutom-domain to custom-domain in variables.tf. git diff --check passed.
Done, right? Not quite.
The change would cause Terraform to destroy and recreate infrastructure, taking the application offline.
Haiku and the other Claudes caught it:
... the next terraform plan will show the NLB and security group being destroyed and recreated, not updated in place. Your users will experience a service interruption.
A one-line code change can be a production incident. Staff engineers are expected to recognize danger in seemingly simple changes.
3. Good engineers tell you what you need to know
Communication skills were the largest gap in our benchmark. The Claudes write roughly 10x longer final reports.
More words aren't inherently better, but Claude consistently surfaces issues, caveats, and risks that GPT leaves out.
There's a tradeoff: GPT writes fewer inaccurate claims. However, when normalized by response length, GPT produces roughly 4x as many inaccurate statements per word.
The ideal engineer isn't the one who talks the most or the least. It's the one who tells you everything important, accurately.
Cheap tokens can lead to expensive mistakes
Sol is cheap. It costs just $0.91 per task, compared with Haiku's $1.68.
But saving $0.77 doesn't look so impressive if the result requires hours of additional review, or creates an incident.
Across our tasks, Sol made mistakes including:
- Missing code paths that retried a request, leading to duplicate charges
- Building a report generation pipeline that works locally but returns 0 for all metrics in production
- Writing a fix for a WebSocket issue without warning that it wasn’t ready to deploy
- Overriding clearly intentional “do not drop this table” code mechanisms
A single missed issue could wipe out the savings from thousands of cheap runs.
The next frontier is engineering judgment
GPT is excellent at implementing well-specified tasks. On functional correctness, the model families were closely matched.
But real engineering isn't just implementing well-specified tasks. It's::
- Recognizing when the specification is wrong.
- Understanding the consequences of a change.
- Knowing what to verify.
- And telling the user what they need to know before something breaks.
Junior engineers are asked to implement code. Staff SWEs are responsible for outcomes. Which would you rather have building your business?
