Can AI survive the 9-to-5?
Frontier models can solve increasingly difficult problems. But how well can they handle the demands of a real job?
Can they make the judgment calls of a staff engineer? Reconcile conflicting financial records? Work through a healthcare case with multiple files? Know when technically correct code is still the wrong thing to ship?
These are the questions behind Surge benchmarks. And today, six of them are joining the Fireworks Specialized Intelligence Index (SII).
The SII brings together practitioner-built evaluations to show how open, closed, and specialized models perform against real work. Surge leaderboards are built for the same purpose: testing models on messy environments, ambiguous instructions, and the judgment required to finish the job.
What we test
Real work rarely arrives as a perfectly specified prompt. A finance analyst might have to find the right number in a spreadsheet and notice when two sources disagree. An engineer might get asked to “make this faster” and have to determine what should change, what could break, and whether the request is a good idea in the first place.
Surge benchmarks test whether models can handle that ambiguity and responsibility.
Judgment over execution. Models are increasingly good at well-specified tasks: write the patch, run the calculation, answer the question. We test the decisions around that execution: what matters, what could go wrong, and when a technically correct answer is professionally wrong.
Messy, real environments. Agents work inside production codebases with years of history, or across dozens of files where sources go stale or disagree, on assignments that can take a professional days. The prompts sound like something a colleague would actually send.
Practitioners write the tests. Our benchmarks are built by people who have done the work themselves, from staff engineers and portfolio managers to physicians and accountants.
Failures you can learn from. Expert-authored rubrics measure correctness, verification, judgment, communication, and other behaviors that a single pass/fail score can hide.
The SII now includes six Surge benchmarks spanning professional knowledge work and software engineering:
Each asks a version of the same question: Would you trust this model to get through the workday?
Built by people who do the job
Every Surge benchmark task is written by a practitioner drawing on work they’ve actually done. They define the assignment, build the environment, and write the rubric that captures what good work looks like. Every task then goes through multiple rounds of review for correctness, completeness, and solvability.
Our experts also study how models fail, turn that judgment into measurable signals, and help define what models need to learn next. The experts behind these six benchmarks include:
- A senior portfolio manager at BlackRock
- A pediatric endocrinologist and epidemiologist
- An astrophysicist with an Oxford MSc and PhD
- A principal software engineer at NASA JPL
By the numbers
DAYJOB: Environments average 22–36 input files per task, with some exceeding 300. Forty-four percent of tasks are estimated to require 20+ hours of professional work, and nearly half require 5–10 years of experience.
sudo L7: Roughly 70% of tasks come from privately licensed production codebases across 75+ repositories and 15+ organizations. Tasks touch systems including identity and access control, payments, ledgers, payroll, and webhooks.
Measuring specialized intelligence
The best model depends on the job.
A system that excels at coding may struggle with financial analysis. A model that performs well on clean, self-contained problems may behave very differently when the relevant information is buried across dozens of files. Models with similar headline scores can also differ substantially in judgment, reliability, and their ability to finish an assignment end to end.
The Specialized Intelligence Index gives teams a way to see those differences across the work they need models to perform.
Explore the Specialized Intelligence Index.
