Table of contents
Case Study Llama
Analysis of Individual writings
Appendix

An investor was about to put $20 million into a startup. Gemini 4 Carbon was tasked with reviewing the company's records and calculating what ownership stake the investment would buy.

The agent uncovered a critical discrepancy: 350,000 shares held by a former executive were missing from the company's ownership spreadsheet but still appeared on its official stock ledger. The board had approved a buyback, but there was no proof it had actually happened.

The agent spotted the problem. Then it ignored it.

It calculated a precise price of $3.3526 per share, assuming the disputed shares had been cancelled. If they hadn't, the investor's ownership stake and the price they should pay could change.

The model caught a potentially multimillion-dollar issue, then produced an answer that failed to account for it.

That's the challenge at the heart of DAYJOB: Law. Can AI agents move beyond finding the right facts to exercising the judgment required when millions of dollars and real clients are on the line?

Introducing DAYJOB: Law

DAYJOB: Law evaluates AI agents on assignments that require factual investigation, legal reasoning, and professional judgment..

The benchmark features 71 assignments created by practicing lawyers, spanning corporate transactions, litigation, criminal defense, employment disputes, intellectual property, estate matters, and more.

Agents receive realistic client requests, messy case files, and little hand-holding. They're expected to investigate the facts, apply the law, and produce work a lawyer could actually use.

The prompts can be surprisingly brief:

Can you draft the sentencing exposure memo for Whitfield?

That brevity is the point. Real lawyers aren't handed step-by-step checklists. They're expected to know which questions to ask, which evidence to investigate, and which legal issues matter.

DAYJOB: Law tests whether AI agents can do the same: thinking independently and exercising legal judgment, not just following instructions.

“Like code, law will benefit from an abundance of evaluations that provide complementary perspectives on agent performance across a diverse and nuanced domain. DAYJOB: Law provides a rigorous new lens to agent performance assessing whether agents can identify legally salient information and convert it into the correct next steps with minimal direction. We’re excited by Surge’s recent work and welcome more benchmarks that helps raise the bar for legal AI” – Julio Pereyra, Head of Legal Research, Harvey

Results: Leaderboard

GPT 6 Astra leads with a 36.6% pass rate on the DAYJOB: Law Leaderboard, followed by GPT 6.1 Sol at 31.6% and Claude Sonnet 5.5 at 26.8%.

An attempt passes when it satisfies at least 95% of its task's rubric criteria.

Complex environments. Messy evidence.

DAYJOB: Law contains 2,638 synthetic files spanning 903 MB, with up to 160 files per assignment. The workspaces include more than 2,000 PDFs, alongside hundreds of contracts, Word documents, Excel spreadsheets, emails, court filings, and images.

One assignment involves a handwritten will, a donation deed, and conflicting accounts of the donor's relationship to an artist. The agent must determine whether the donor actually owned the rights she purported to transfer.

In another, a beneficiary suspects a dog was replaced to preserve payments from a pet trust. A suspicious text message and an unnamed cremation receipt raise questions, but prove nothing on their own.

The files also contain outdated records, irrelevant documents, and misleading clues. Agents must distinguish genuine risks from distractions, reconcile contradictory evidence, and recognize when critical information is missing.

The challenge is knowing what matters, what can be trusted, and what remains unproven.

The lawyers behind the benchmark

DAYJOB: Law assignments were written by experienced lawyers to reflect substantive legal work that require agents to reconcile evidence, apply the law, and prepare legal work products.

Our legal expert roster includes attorneys with backgrounds such as:

  • Intellectual property and corporate transactions: An attorney with experience at Kirkland & Ellis advising businesses on intellectual property, licensing, technology-transfer agreements, and commercial contracts. Their work included managing patent, trademark, and copyright portfolios, conducting IP due diligence for corporate transactions, and negotiating commercial agreements. They earned a JD from Harvard Law School.
  • Immigration and worker rights: A Massachusetts-barred attorney and Skadden Fellow at the Central West Justice Center, representing immigrant child workers and providing rights education. Their experience includes Yale Law School's Worker and Immigrant Rights Advocacy Clinic and the ACLU's Immigrants' Rights Project. They earned a JD from Yale Law School.
  • Criminal defense and appellate litigation: An Illinois-barred attorney with experience as an Assistant Appellate Defender and Assistant Public Defender, handling criminal appeals, post-conviction proceedings, and trials involving matters ranging from misdemeanors to first-degree murder. They earned a JD from Northwestern University Pritzker School of Law.

Getting the law right and the sentence wrong

In one DAYJOB: Law task, a client faced charges from several shoplifting incidents, including one where he struggled with a store employee while carrying stolen goods. Claude Sonnet 5.5 was tasked with preparing a memorandum estimating his potential prison sentence.

The model correctly challenged the prosecution's attempt to combine the thefts into a more serious charge and recognized that the struggle could constitute robbery.

But then it concluded that the theft and robbery sentences could be punished consecutively, producing a scenario of up to 11 years in prison.

The problem? The same theft was already part of the robbery. Under the supplied law, the sentences could not simply be added together without changing how the charges were constructed.

The model knew the law but misapplied it, overstating the client's potential prison time.

Knowing the law isn't enough

As agents improve, knowing black letter law is increasingly not the bottleneck to their ability to perform real-world work. More often, they know the law but fail to see how it may apply to the sparse instructions and diverse data required to execute legal tasks. DAYJOB: Law provides a novel lens on this problem, emphasizing an agent's ability to exercise judgment needed to make the many right calls needed to deliver legal work product.

That means knowing which evidence matters, when the facts are insufficient, and how to turn legal analysis into advice a client can trust.

Even frontier models struggle. An agent can spot 350,000 disputed shares, then ignore them when pricing a $20 million investment. It can correctly identify criminal charges, then overstate a client's potential prison sentence.

Passing the bar is one thing. Practicing law is another.

DAYJOB: Law tests the gap between knowing the law and knowing what to do with it.

Special Acknowledgment: We previewed DAYJOB: Law with the team at Harvey and are grateful for their review of the scenarios’ realism and their feedback on our early results.

To learn more or request access to the benchmark, contact benchmarks@surgehq.ai.

‍

/surge-ai
@hellosurgeai

Read what
frontier labs read

Subscription confirmed

You'll get updates when we post
Oops! Something went wrong while submitting the form.

More Posts

The Smallest Claude is a Better Software Engineer Than the Biggest GPTThe Smallest Claude is a Better Software Engineer Than the Biggest GPT
Surge RL Pushes NVIDIA Nemotron 3.5 Super VL Toward Frontier-Level PerformanceSurge RL Pushes NVIDIA Nemotron 3.5 Super VL Toward Frontier-Level Performance
We Benchmarked Coding Agent Harnesses. Thinking Harder Beat All of Them.We Benchmarked Coding Agent Harnesses. Thinking Harder Beat All of Them.
Six Surge Benchmarks Join the Fireworks Specialized Intelligence IndexSix Surge Benchmarks Join the Fireworks Specialized Intelligence Index
DAYJOB: Tax — Benchmarking AI Agents on Professional Tax WorkDAYJOB: Tax — Benchmarking AI Agents on Professional Tax Work
sudo L7: Can Coding Agents Act Like Staff Engineers?sudo L7: Can Coding Agents Act Like Staff Engineers?
Appendix