An oncologist reads a survival curve. A meteorologist reads a wind rose. Across medicine, engineering, finance, manufacturing, and science, charts carry the evidence that decisions depend on.
We built Chartography to measure whether frontier models can interpret those graphics with the accuracy and judgment professional work demands. The benchmark spans Sankey diagrams, candlestick charts, contour maps, atmospheric back-trajectories, Bode plots, growth curves, wind roses, and other domain-specific formats.
Every task was written by a professional who works with these charts for a living, and independently reviewed by multiple domain experts. Questions require models to estimate unlabeled values, trace overlapping features, interpret complex geometry, apply domain conventions, and combine multiple visual readings into a final answer.
The best frontier model only reaches 45%.
The name combines chart and cartography. A chart maps a body of data, and reading that map accurately is a professional skill. Chartography measures how well frontier models have learned it.
Professionals work from charts
When an oncologist evaluates a treatment option, the evidence arrives as a Kaplan-Meier curve.
When a structural engineer signs off on a building, the governing wind speed comes from a contour map.
When a trader sizes a position, the price history is a wall of candlesticks.
In each case, the working source of information is a chart, not a structured spreadsheet. Reading the chart is a core skill of the job: a professional needs to interpolate between contour lines, follow a narrow flow through a crowded Sankey diagram, or project a value from a three-dimensional surface.
AI agents working in these fields face the same visual interface. They must interpret a curve they can see but never query, and make calculations from values that were never printed explicitly.
Chartography measures that ability.
Why another chart benchmark?
Existing benchmarks are nearing saturation. Frontier models now perform well on direct extraction from familiar bar, line, pie, and scatter charts. ChartQA was largely solved by the Claude 3.5 Sonnet generation; frontier models now score around 90% on CharXiv, while multiple models exceed 80% on ChartMuseum. Many of the remaining misses appear to come from errors in the benchmark ground truth itself.
Look inside those datasets and the ceiling makes sense:
- Their charts are usually simple: bars, lines, pies, scatterplots. These formats cover only a small fraction of the graphical systems used in technical and professional work. Cardiologists, process engineers, atmospheric scientists, and controls engineers routinely rely on chart types that rarely appear in general-purpose benchmarks.
- Their questions rely on short reasoning chains. A typical task asks the model to find a value, maybe add two numbers. But professional chart reading often requires chaining several inferences together: identify the correct curve, estimate its position between gridlines, read another value elsewhere in the figure, apply a domain convention, and only then begin the arithmetic.
- Their answer formats constrain the difficulty further. Multiple choice and true-or-false questions shrink the option space enough that a model can score well while barely looking at the chart.
- Their grading doesn't reflect visual precision. Answers are either taken directly from labels or accepted within a blanket percentage margin, regardless of how precisely the underlying chart can actually be read.
Professional chart reading requires fluency in the graphical system itself. A clinician must know which percentile curve to follow, and an engineer must know when extrapolating beyond a plotted range is inappropriate.
Earlier benchmarks have measured extraction from familiar graphics. Chartography focuses on domain-native formats, professional conventions, visual estimation, and multi-step reasoning.
Introducing Chartography
We use Chartography to describe the ability to navigate a chart: knowing where to look, how to interpret it, which convention applies, and how much confidence the visual evidence supports.
Four principles define the benchmark.
1. Charts from real professional domains
Every task begins with a chart used in real work.
Chartography spans healthcare, finance, engineering, manufacturing, atmospheric science, process control, and other technical fields. The benchmark includes:
- Kaplan–Meier curves and pediatric growth charts
- Candlestick and financial market charts
- Contour maps and wind roses
- Sankey and material-flow diagrams
- Bode plots and engineering response curves
- Atmospheric back-trajectories
- Failure-criterion and design charts
- Three-dimensional surface plots
2. Questions written by real professionals
Every task was authored by a professional who reads these charts for a living, including clinicians, engineers, scientists, and financial professionals. Many questions depend on knowledge an expert applies automatically. All questions reflect their real day-to-day work:
- Determining the design wind speed for a building
- Calculating a child’s height percentile
- Sizing an outlet-protection apron
- Identifying out-of-control process readings
- Tracing material flows through a production system
- Measuring a plotted value against a failure criterion
3. Visual reasoning beyond label lookup
Reading printed labels is largely solved. Chartography targets the operations that remain difficult for frontier models:
- Estimating values along sparsely labeled axes
- Interpolating between contours and curves
- Tracing thin or overlapping flows
- Associating labels with the correct visual elements
- Interpreting projected geometry
- Measuring the gap between a point and a criterion curve
- Combining several visual observations in a multi-step calculation
A single task may require the model to inspect several regions of the chart before performing any arithmetic.
The observations also often depend on one another, like in our ComplexConstraints benchmark. A small error in the first visual read can propagate through an otherwise correct calculation.
4. Expert grading calibrated to the chart
Visual estimation requires chart-specific precision.
A contour map may support an answer near 150 mph with an acceptable range of 140 to 150 mph. A pediatric growth chart may support a much narrower interval. A universal percentage tolerance can't capture that difference, since it depends on the density of the grid, the thickness of the curves, and the resolution of the figure.
For every task, our expert authors provide:
- The intended answer
- A complete walkthrough of the visual and computational steps
- An expert-determined acceptable range
This approach rewards accurate chart reading while accounting for the natural limits of visual estimation.
Each task then passes through additional expert review. Reviewers verify the intended reading, revise unclear wording, and check the acceptable answer range.
Results
The best-performing model only reaches 45%.
The same frontier models that score between 80% and 90% on several earlier chart benchmarks remain below 50% on Chartography. GPT-5.6 Sol (Max reasoning) leads the leaderboard, while most evaluated models score between approximately 10% and 40%.
More inference-time computation does not reliably solve the problem. Some models generate substantially more tokens, or cost several times more per trial, without producing better chart readings. Longer reasoning traces often preserve an incorrect visual observation rather than correct it.
Where models break
Failures span the leaderboard, but most fall into a few recurring categories.
1. They miss visual elements
Thin, faint, or overlapping features disappear from the model's representation of a chart. In one Sankey task, every model identified the largest flow into a sector and missed the three thinner flows beside it. Every model failed the task.
2. They follow the right feature and read the wrong value
Models frequently pick the correct curve, contour, or trace, then misjudge its position.
Example errors include:
- Reading a peak two gridlines too low
- Assigning a location to the neighboring contour band
- Following the correct percentile curve but misreading its intersection
- Confusing one nearby axis label in an otherwise correct calculation.
3. They struggle with complex geometry
Three-dimensional surface plots produced some of the lowest performance in the benchmark. No model used the projected gridlines that make these plots readable to humans, and answers scattered around the truth accordingly.
4. They apply domain conventions inconsistently
Professional charts encode rules that are never stated in the prompt.
In one engineering task, models had to size a riprap apron for a flow below the plotted range for the relevant pipe diameter. Standard practice is to use the curve’s lowest plotted value. Most models extrapolated past the chart instead. One invented an unrelated formula. The chart encoded a rule about how its numbers may be used, and every model missed the rule.
5. Errors compound across otherwise-correct reasoning
Again and again, a model identifies the correct curve, chooses the correct formula, describes the correct calculation, misreads one visual input, and lands on the wrong answer. Premature rounding compounds it: several models rounded each intermediate read even when the prompt demanded precision.
The reasoning trace can appear persuasive because most of it is correct. In professional settings, a single faulty visual observation can invalidate the result.
Three example tasks
Example 1: Engineering design chart
Question:
For an outlet pipe with a diameter of 48 inches and a flow of 73 cubic feet per second, what is the minimum downstream width of the riprap apron?
Answer: 30 feet
Accepted range: 29–31 feet
Example 2: Pediatric growth curve
Question:
What height corresponds to the 50th percentile for a four-year-old girl? Round to the nearest 0.5 centimeters.
Answer: 100.5 cm
Accepted range: 100–101 cm
Example 3: Wind rose
Question:
In which three directions did winds above 8.5 m/s occur for the longest total duration?
Answer: S, SSW, W
From demo to deployment
Professional agents will have to make decisions based on the same visual evidence professionals use every day: clinical curves, engineering plots, financial charts, process dashboards, atmospheric maps, and technical diagrams.
These graphics encode critical information through position, geometry, visual convention, and domain-specific judgment.
Frontier models score around 90% on familiar academic chart tasks and below 50% on Chartography. That difference is the boundary between impressive demos and dependable professional systems.
Chartography makes that boundary measurable.
Check out the leaderboard here. Link to GitHub repo with all tasks and scripts can be found here. The standalone dataset on Hugging Face can be found here. Reach out to benchmarks@surgehq.ai if you're interested in learning more!
