Designing Evaluations That Actually Tell You Something

Start with the business decisions and workflows the AI system is expected to support.

The goal is to identify the parts of the product or process where model performance actually matters. That might include high-volume workflows, tasks that are especially important to customers, new capabilities you are considering launching, or areas where mistakes would have meaningful consequences.

It can help to group candidate tasks into categories such as:

• Core workflows the system handles every day
• High-value or high-risk use cases
• Tasks that require more reasoning or judgment
• Workflows where the model already appears to struggle
• New capabilities you want to test before deployment

You do not need to evaluate every possible task equally. The eval set should concentrate on the workflows that matter most to the business and the decisions you want the evaluation to inform.

An eval dataset is a structured collection of test cases, along with the information needed to run and score each one.

A single eval example might include:

• The user request or task
• Any documents, data, or context the model should receive
• The tools or actions available to the model
• The expected outcome
• A reference answer, when one exists
• A rubric or set of grading criteria
• Metadata such as task type, difficulty, customer segment, or risk level

For simple tasks, an example might contain only a prompt and a correct answer. For an agentic workflow, it might include an entire simulated environment, available tools, and criteria for judging the final state.

The structure of the dataset should make it possible to run the same task consistently across different models, prompts, or system versions and compare the results fairly.

Once you have chosen the workflows you care about, the next question is whether the mix of examples inside the eval set gives you enough coverage of those workflows.

Representativeness does not always mean copying production traffic exactly. A rare case that creates significant financial, legal, operational, or reputational risk may deserve more weight in an evaluation than its raw frequency would suggest.

Teams should look at the distribution of examples and ask questions such as:

• Are the important task categories represented in the right proportions?
• Are there enough examples across different levels of difficulty?
• Are relevant customer, user, or workflow segments covered?
• Are high-risk scenarios represented even if they are uncommon?
• Are known failure modes included?
• Is any one type of example overrepresented simply because it was easier to collect?

A representative eval set should give you confidence that the results reflect the parts of the system you actually care about, rather than the quirks of the particular examples you happened to include.

Happy-path examples show whether the system can handle straightforward cases. They do not tell you much about how it behaves when conditions are messy.

Real users provide incomplete instructions, unusual requests, contradictory information, ambiguous inputs, and unexpected combinations of tasks. Production systems also encounter missing data, tool failures, edge cases, and scenarios that were not anticipated during development.

Including harder cases helps expose differences between models that may look similar on easy examples.

For enterprise applications, it is especially important to test situations where the correct behavior is to ask for clarification, refuse an unsupported conclusion, escalate to a person, or acknowledge uncertainty.

An edge case is an unusual or difficult situation that sits outside the most common workflow.

An adversarial case is designed specifically to test whether the system can be pushed into making a mistake.

For example, an edge case might involve an unusual contract structure or an incomplete customer record. An adversarial case might deliberately include misleading information, conflicting instructions, or a prompt designed to bypass a policy.

Both are useful because model failures often appear first at the boundaries of normal behavior.

The right balance depends on the product. A customer support assistant may need broad coverage of unusual customer requests, while a system handling sensitive data may require more deliberate adversarial testing.

Evaluator calibration is the process of making sure different graders apply the same standards in a similar way.

Human evaluators are typically given shared examples, scoring guidance, and feedback before grading a larger dataset. The team can then compare how different evaluators score the same responses and resolve areas of disagreement.

Calibration is also useful for model graders. A model judge can be tested against a set of trusted human judgments to determine whether it is applying the rubric as intended.

Good calibration reduces noise and makes evaluation results easier to trust. Our expert workforce is trained extensively on all kinds of evaluation tasks and is always calibrated to your specific guidelines.

Inter-rater agreement measures how often different evaluators reach the same conclusion when judging the same output.

High agreement suggests that the evaluation criteria are clear and can be applied consistently. Low agreement can indicate that the rubric is ambiguous, the task is highly subjective, or the evaluators need more calibration.

Perfect agreement is not always realistic, especially for subjective tasks. The important question is whether disagreements are frequent enough or significant enough to affect the decisions you are making.

When disagreement is high, teams should inspect the disputed examples rather than simply averaging the scores. Surge can help you better understand the situations that lead to disagreements across your evaluations.

Quality control checks whether the evaluation process itself is working correctly.

For human evaluation, this can include qualification tasks, calibration exercises, hidden quality checks, duplicate grading, expert review, and adjudication of disagreements.

For automated or model-based grading, quality control can include testing against human-reviewed examples, inspecting false positives and false negatives, and periodically revalidating the grader as the task changes.

The level of quality control should reflect the importance of the decision being made. An informal internal experiment may need less oversight than an evaluation used to approve a high-stakes system for production.

Not every mistake has the same impact.

A system that produces a slightly awkward sentence and a system that gives materially incorrect legal advice may both receive one failed example, but those failures should not carry the same weight.

Teams can account for this by assigning severity levels, weighting certain task categories more heavily, or reporting critical failures separately from the overall score.

This is especially important when evaluating enterprise systems where a small number of severe mistakes can matter more than a large number of minor ones.

A useful evaluation should reflect the business impact of failure, not only how frequently failures occur.

Production failures are one of the best sources of new evaluation examples.

When a meaningful failure appears, teams can capture the underlying task, identify why the system failed, define the correct behavior, and add a representative version of the case to the eval set.

You can learn more about how to add production failures to your eval sets here.

“Overfitting” is the idea that repeatedly using the same evaluation to guide development results in gradually optimizing for that specific set of examples, rather than the underlying behavior.

A few practices can help reduce this risk:

• Keep some evaluation examples hidden from the development process
• Regularly add new examples
• Maintain separate development and final evaluation sets
• Test the same capability using different prompts and scenarios
• Look at production performance alongside offline eval scores

The goal is to measure whether the system learned the underlying capability, rather than whether it learned how to succeed on a fixed collection of tests. Surge helps you build eval sets that are robust to overfitting and can identify when a set has become overly saturated (and therefore less meaningful).

An evaluation is useful when it gives you enough confidence to answer the question you started with.

That threshold depends on the decision. Comparing two early prototypes may only require a relatively small evaluation. Selecting a model for a production workflow may require broader coverage, stronger grader validation, and closer analysis of high-severity failures.

Before acting on the result, teams should understand:

• What the evaluation covers
• What it does not cover
• How reliable the grading is
• How large the observed performance difference is
• Which failure modes remain
• Whether those failures are acceptable for the intended use

A single score should rarely be the only input. The distribution and severity of failures often matter just as much as the average result.