Comparing and Choosing Models

Start with the requirements of the workflow you are trying to support.

For each candidate model, we can help you measure performance on a representative eval set and compare the results alongside practical considerations such as cost, latency, context window, tool use, deployment options, and reliability.

It is also useful to compare models by task type rather than relying only on one aggregate score. One model may be stronger at analysis, another at extraction, and another at tool use.

A good comparison should help answer a practical question: which model gives you the best performance for the constraints of this application?

Quality is usually the most important metric, but it is rarely the only one.

Enterprise teams may also care about:

• Cost per request or task
• Latency
• Throughput
• Reliability and error rates
• Context window
• Tool-use performance
• Structured output reliability
• Deployment and data residency requirements
• Availability and rate limits
• Model stability over time

The right set of metrics depends on how the system will be used. A customer-facing assistant may care heavily about latency, while a research workflow may tolerate slower responses in exchange for higher quality.

The Pareto frontier is a way to identify models that offer the best tradeoffs across multiple metrics.

Imagine comparing models on quality and cost. A model is on the Pareto frontier if there is no other model that is both better and cheaper.

Models below the frontier are dominated because another option performs at least as well while also costing less, or performs better at the same cost.

The same idea can be applied to quality and latency, quality and throughput, or any other pair of metrics that matter to the business.

The Pareto frontier is useful because model selection often involves tradeoffs rather than a single winner.

A Pareto frontier chart usually plots one metric on each axis.

For example:

• The horizontal axis might represent cost
• The vertical axis might represent quality

Models farther up have higher quality. Models farther to the left cost less.

The most attractive options tend to sit along the upper-left boundary of the chart. Those models offer the best available quality for a given level of cost.

A model that sits below and to the right of another model is generally less attractive because it costs more while performing worse.

The frontier does not tell you which model to choose. It helps narrow the decision to the options that offer meaningful tradeoffs.

Not always.

A model that ranks highly overall on public benchmarks may perform poorly on your documents, tools, policies, or workflow. A smaller model may also perform nearly as well on a constrained task while being significantly cheaper and faster.

This is why custom evaluations are important for enterprise model selection. They let you compare models on the work they will actually perform.

The right choice depends on the level of performance, control, cost, and operational responsibility your application requires.

Closed models are typically accessed through an API operated by the model provider. They often provide access to the strongest frontier capabilities with relatively little infrastructure work, but give you less control over the underlying model and how it is deployed.

Open-weight models make their model weights available for others to run or adapt, subject to their license terms. They can offer more flexibility for customization, deployment, and post-training, though performance and operational requirements vary widely between models. These can either be self-hosted (see below) or accessed via a third-party hosting service such as OpenRouter.

Self-hosting means running the model in infrastructure you control, whether in your own environment or through a cloud provider. Self-hosting is most commonly associated with open-weight models, but it is a deployment choice rather than a separate class of model.

For enterprise teams, the decision usually comes down to whether the benefits of greater control outweigh the additional infrastructure and operational complexity.

Closed models are often a strong choice when you want access to frontier performance without operating the underlying model infrastructure yourself.

They can be especially attractive when:

• The provider's models perform best on your evaluations
• Fast deployment matters
• Your application does not require access to model weights
• API-based security and data handling meet your requirements
• You want the provider to manage inference infrastructure, scaling, and model updates

The tradeoff is less control. You may have limited ability to modify the underlying model, control exactly when it changes, or customize how inference is performed.

Teams should evaluate the model itself alongside the provider's security, reliability, pricing, deployment, and data-handling options.

Open-weight models can be useful when greater control or customization matters.

They may be a good fit when you need to:

• Post-train a model for specialized capabilities
• Control the exact model version running in production
• Optimize inference for a particular workload
• Deploy within a specific infrastructure environment
• Avoid dependence on a single model API
• Operate at a scale where hosting economics become attractive

Open-weight models also give teams more freedom to experiment with training methods, inference settings, and model architecture.

That flexibility comes with additional responsibility. Teams may need to manage serving infrastructure, optimization, security, monitoring, upgrades, and model quality themselves or through an infrastructure partner.

Licensing also matters. Models described as "open" can come with very different permissions and restrictions, so enterprises should review the specific license rather than assuming all open-weight models can be used in the same way.

Self-hosting can make sense when you need tighter control over infrastructure, data handling, model versions, or inference behavior.

Common reasons include:

• Strict security or data residency requirements
• A need to run inside a private environment
• High-volume workloads where infrastructure economics favor self-hosting
• Custom post-training or inference optimizations
• A need to control model upgrades and availability
• Requirements that cannot be met through a provider's API

Self-hosting also introduces meaningful operational work. The team becomes responsible for areas such as infrastructure, scaling, uptime, model serving, security, monitoring, and optimization.

For many enterprises, the right question is not simply whether self-hosting is possible, but whether the added control is valuable enough to justify that complexity.

Often, no.

Model routing is the process of choosing which model should handle a particular request.

The decision might be based on task type, complexity, customer tier, latency requirements, cost constraints, or confidence in the output.

For example, a system might send straightforward requests to a smaller model and route more difficult cases to a stronger model.

Routing can improve efficiency, but it introduces another component that needs to be evaluated. A routing system should be tested to make sure it sends difficult requests to models that can handle them and does not unnecessarily escalate easy tasks to expensive models.

Some models offer multiple reasoning levels or inference settings that trade additional computation for better performance.

These settings should be treated as separate options during evaluation.

A higher reasoning setting may improve performance on difficult tasks while increasing cost and latency. On simpler tasks, the improvement may be small enough that the lower setting is a better choice.

The most useful comparison is usually based on the same eval set across all settings, with quality, cost, and latency measured together.

This can reveal where additional reasoning actually improves outcomes and where it does not.

A higher score does not always mean the difference is large enough to matter.

Small differences may come from random variation in the eval set or grading process. They may also be too small to justify a large increase in cost or latency.

Teams should look at:

• The size of the performance gap
• The number of examples evaluated
• The consistency of the difference across task categories
• The severity of the failures each model makes
• The uncertainty in the measurement
• The business impact of the improvement

A two-point gain may be valuable if it eliminates a critical failure mode. A five-point gain may matter less if it comes entirely from low-priority tasks.

Surge can help you better understand what a score difference practically means and whether the gap warrants a decision one way or another.

Model cost should be measured in the context of the workflow, not just the provider's price per token.

The total cost can depend on:

• Input and output length
• Reasoning or inference settings
• Number of model calls per task
• Retries
• Tool calls
• Routing strategy
• Volume
• Human review required after the model responds

A more expensive model can sometimes lower overall costs if it reduces retries or manual review. A cheaper model can become expensive if it frequently fails and requires additional processing.

The most useful unit is often cost per successfully completed task.

Model selection should be revisited periodically.

The model landscape changes quickly, and new releases may offer better quality, lower cost, or stronger performance on specific workflows. Your own product may also change in ways that alter what you need from the model.

Common reasons to rerun model selection include:

• A major new model release
• A meaningful pricing change
• Changes to your product or workflow
• New customer requirements
• New failure modes
• A shift in traffic or usage patterns
• Improvements to your evaluation set

Keeping a stable evaluation suite makes it much easier to compare new options without rebuilding the decision process each time.