We asked coding agents to make a small change to an event registration app: add a Zoom link and passcode to guest confirmation emails.
One of the agent setups we tested was equipped with a harness containing 37 additional skills. Surely all that machinery would help? It decided the change deserved a thorough review, and spawned 17 agents:
- Five reviewed the plan for things like feasibility, scope, and security.
- Five looked for ways to simplify the implementation.
- Seven reviewed the resulting code.
But somehow, none of them noticed the most important problem: the confirmation email went out when someone registered, before they'd paid. Following the user's instructions would give unpaid registrants the Zoom passcode!
Seventeen agents, lots of review, and the central product issue went unnoticed. How much do coding-agent add-ons actually help?
There's a Cambrian explosion of harnesses, skills, plugins, and elaborate coding workflows. Some people use vanilla Claude Code. Others have assembled small bureaucracies of specialized agents, reviewers, planners, and supervisors.
A lot of these setups feel sophisticated. But do they actually produce better software? We decided to measure it.
The results
We tested several popular coding-agent setups using Opus 5.5 in Claude Code, evaluating them on software engineering tasks from Surge's sudo L7 benchmark.
- Vanilla Claude Code
- Superpowers: A skills plugin that forces plan-and-test discipline before coding, with 311K+ Github stars.
- Compound Engineering: A plan/work/review workflows with specialized reviewer agents and persistent learnings, with 25K+ Github stars.
- Claude Code's built-in /goal and /code-review plugins.
We also tested a simpler approach: leaving the harness unchanged and increasing the model's thinking effort.
Here's what we found:
The simplest approach won!
- Thinking effort had the biggest impact. Running vanilla Claude Code at xhigh effort improved its score by 11.3 points, compared with 2–3 points for the most effective harness additions.
- Some lightweight additions helped. Superpowers improved performance by 2.4 points while increasing cost by just 10%. Claude Code's /goal and /code-review combination improved performance by 3.3 points at 1.8× cost.
- More orchestration didn't necessarily mean better results. Compound Engineering cost 8× as much as vanilla, without a statistically clear improvement in our experiment.
Vanilla Claude Code at xhigh effort improved by 11.3 points over medium effort. That was much larger than the gains from any individual harness configuration in this experiment.
If you're spending an afternoon engineering a complicated agent setup, it's worth asking whether you've tried the simpler lever first.
Testing engineering judgment, not just correctness
We built sudo L7 to evaluate coding agents on the broader judgment expected of staff engineers.
Tasks take place in real codebases, many licensed from operating companies. Prompts are often underspecified, and expert-written rubrics evaluate functional correctness alongside verification, practical judgment, communication, and other engineering skills.
A passing test suite doesn't necessarily mean the agent did a good job. It might have misunderstood the user's needs, overlooked a security risk, or failed to investigate the existing system. sudo L7 goes beyond functional correctness, measuring eight dimensions of software engineering behavior, from verification to practical judgement.
We held the underlying model fixed at Opus 5.5, with medium thinking effort for the harness comparisons and xhigh for the additional vanilla run.
Our overall question: when the model stays the same, how much improvement comes from changing the workflow around it?
What did extra thinking actually buy us?
At xhigh effort, the model often did more the work to make its answer trustworthy. Compared with medium effort, xhigh improved on the sudo L7 rubric dimensions by:
- 14.9 points on Verification & Thoroughness
- 9.8 points on Persistence
- 7.0 points on Thought Partnership
Here's an example.
In a video-editing application, a code path that spliced two clips together would crash if the first clip was missing. The user asked the agent to make that clip optional.
- At medium effort, the agent fixed the immediate cause of the crash and reported success.
- At xhigh effort, the agent kept investigating. It discovered that the initial fix would send execution down a fallback path that also assumed the first clip existed. The application would still crash.
The medium-effort agent fixed the explicit bug and stopped, while the xhigh agent traced the rest of the code path and caught a second one.
On this task, an extra 29 cents made the difference between a broken fix and a working solution:
More agents don't mean better judgment
The Compound Engineering example from the opening illustrates a different problem.
Five agents examined the plan, including security and product concerns. Seven reviewed the code. Yet none noticed that adding the Zoom passcode to the confirmation email would give unpaid users access to the event.
All that review still missed the central business requirement.
It's one example, and multiagent review can certainly help on other tasks. But it captures a weakness we're particularly interested in measuring: an agent can execute a request thoroughly while misunderstanding what the user actually needs.
Additional reviewers won't necessarily help if they all inherit the same mistaken assumption.
What coding agent setup should you actually use?
These results reflect particular configurations of one model on one distribution of engineering tasks. Your own workloads may behave differently. But they suggest a few practical recommendations.
- For a simple, low-cost setup: Vanilla Claude Code is a strong baseline. There's plenty you can accomplish without maintaining a custom harness.
- For an inexpensive improvement: Superpowers gained 2.4 points at roughly 1.1× the cost.
- For more thorough completion and review: /goal followed by /code-review gained 3.3 points at 1.8× cost.
- When quality matters more than cost: Try increasing thinking effort. Xhigh delivered the largest improvement we measured, at 3.5× the cost of medium-effort vanilla.
- Before adopting an elaborate multiagent harness: Benchmark it on your own tasks. Our Compound Engineering configuration was substantially more expensive without a clearly established performance benefit.
There are some interesting experiments we haven't run yet. Would combining xhigh effort with /goal and /code-review produce even better results? How would the setups compare under identical dollar or time budgets?
But for now, more thinking seems to beat more complex configurations.
Think smarter
It's easy to spend hours tweaking a coding-agent setup based on a few impressive-looking runs. More planning, more reviewers, and more elaborate workflows can all feel like improvements.
Our results suggest being a little more skeptical.
Some harness additions genuinely helped. Increasing thinking effort helped considerably more.
If you're building or customizing an agent workflow, test it on real tasks. Measure whether the changes improve the actual engineering work.
Otherwise, you might spend eight times as much money and still email the Zoom passcode to people who haven't paid.
