Evaluation
v0.2Updated
How well do agents actually do analyst work — and what does it cost per task?
A curated comparison of current frontier models on independent analytics and agentic benchmarks, weighted into a single index and set against the measured cost of running each one.
The index combines 4 evaluations: Vals EMB (Excel Modeling), LiveBench Data Analysis, Terminal-Bench 2.1, and LiveBench Agentic Coding.
Claude Fable 5.1 leads the index at 77.8, ahead of GPT-6 Astra at 76.0 — a 1.8-point gap.
Ranked results
Higher is better
Analytics Agent Index vs Cost per Task
| Model | Provider | Analytics Agent Index | Cost per Task | Provenance |
|---|---|---|---|---|
| Claude Fable 5.1 | Anthropic | 77.8 | $15.93 | public-benchmark |
| GPT-6 Astra | OpenAI | 76.0 | $5.83 | public-benchmark |
| GPT-5.6 Sol | OpenAI | 74.8 | $6.01 | public-benchmark |
| Claude Opus 5 | Anthropic | 74.8 | $6.07 | public-benchmark |
| Kimi K3 | Moonshot AI | 72.4 | $3.48 | public-benchmark |
| Claude Sonnet 5 | Anthropic | 68.5 | $15.44 | public-benchmark |
| Grok 4.6 | xAI | 68.3 | $3.06 | public-benchmark |
| Claude Opus 4.8 | Anthropic | 66.0 | $12.06 | public-benchmark |
| GLM 5.2 | Zhipu AI | 65.0 | $4.47 | public-benchmark |
| GLM 5.3 | Zhipu AI | 64.2 | $3.79 | public-benchmark |
| DeepSeek V4 | DeepSeek | 51.1 | $0.80 | public-benchmark |
Figures are curated from public sources and vendor announcements, not SyneHQ measurements.
Leaderboard · v0.2
Sort by any column. Filter by provider or model name.
Every number on this page is read from an independent third-party leaderboard, never from a vendor's own announcement. These are not SyneHQ measurements. A dash (—) means the figure is not published, not zero.
Score by benchmark
The index score is a weighted average of the component benchmarks below. Every component is a public leaderboard or paper result, so a model only appears on a benchmark its authors actually published — a model missing from a panel was never run on it, and is not scored zero.
0–100 scale · 35.0% of the index
0–100 scale · 30.0% of the index
0–100 scale · 20.0% of the index
0–100 scale · 15.0% of the index
Components and weights
The agent is handed a real financial-modelling task in a spreadsheet and must build or extend the model itself. Template tasks are graded by exact cell match, Scratch tasks against a partial-credit rubric. This is the closest public benchmark to the analyst work an agent actually gets asked to do.
LiveBench's data-analysis category: table reformatting, join prediction and column-type inference over tables drawn from recent Kaggle and Socrata datasets. Questions are refreshed from sources published after the models trained, so a score here is hard to reach by memorisation.
89 end-to-end terminal tasks run under the Terminus 2 harness, scored pass@1 with the model required to pass every pytest for a task. Measures whether an agent can actually drive tooling to completion rather than describe how it would.
LiveBench's agentic coding category, where the model works a task through a tool loop instead of emitting one shot of code. Included at a low weight because warehouse work is mostly code the agent has to run, check and correct.
Index score = 35.0% × Vals EMB (Excel Modeling) + 30.0% × LiveBench Data Analysis + 20.0% × Terminal-Bench 2.1 + 15.0% × LiveBench Agentic Coding. Raw weights (0.35 : 0.3 : 0.2 : 0.15) are shown as percentages of their total, so the 4 components add up to 100% of the index.
Figures are curated public data — vendor announcements and public benchmark leaderboards. They are not SyneHQ measurements.
Reading the index
The index is a weighted average of four independent benchmarks: Vals AI's Excel Modeling Benchmark at 35%, LiveBench Data Analysis at 30%, Vals AI's Terminal-Bench 2.1 at 20%, and LiveBench Agentic Coding at 15%. The two analytics components carry 65% between them because that is what this page is about; the agentic components are included because warehouse work is mostly code an agent has to run and correct, not code it writes once.
Only third-party measurements are used. Vendor-reported figures were collected and deliberately excluded — a model's own announcement is not evidence on the same footing as a leaderboard the vendor does not control. Where a vendor published a conflicting figure for the same benchmark, that is noted rather than averaged.
Where a model is missing a component, the index is renormalised over the components it does have, and the row says so. A missing benchmark renders as an em dash, never as zero. DeepSeek V4 is the one model here scored on two components rather than four, so its index is not directly comparable to the rest.
Cost per task is the average USD the agent actually spent completing one Excel Modeling Benchmark task, as measured by Vals AI — not a token-price estimate. It is the real cost of the run, including whatever reasoning the model chose to do. Three other boards publish cost figures over different task sets; they are not interchangeable, so only one is used here.
There is no text-to-SQL component, and that is a finding rather than an omission. Spider 2.0, BIRD-CRITIC, DABstep, LiveSQLBench and the BIRD Data Intelligence Index were all checked: not one of them lists a single model on this page. Those boards take months to validate a submission and DABstep has closed validation entirely, so they currently stop a generation or more behind the models people are actually deploying. The Text-to-SQL Leaderboard page carries that evidence for the models that do appear on it.
Model names are printed as the board prints them. Provider names are normalised to the vendor's own name, so the Vals label 'SpaceXAI' appears here as xAI. Release dates come from each provider's own documentation; where a provider publishes none, the field is left blank rather than inferred.
List prices are the published base tier. Long-context requests are billed higher by OpenAI, Google, xAI and Alibaba once a threshold is crossed, and an analytics agent working a wide schema will cross it routinely — so treat every price here as a floor. DeepSeek figures are peak-rate; off-peak is exactly half.
Because none of the public text-to-SQL leaderboards list any of these models. Spider 2.0, BIRD-CRITIC, LiveSQLBench and the BIRD Data Intelligence Index were each read in full; their newest entries are a generation or more behind, and DABstep has closed validation pending a v2. Publishing a SQL column here would have meant inventing numbers or silently mixing model generations. The Text-to-SQL Leaderboard page shows what those boards do measure.
No. Every figure is read from an independent public leaderboard and cited per row. SyneHQ ran none of these evaluations and has no submission on any of these boards.
They were collected — 163 cited vendor figures across Anthropic, OpenAI, DeepSeek, Moonshot and Zhipu — and then left out of the index on purpose. Vendors choose which benchmarks to report and under which harness, and several published mutually inconsistent figures for their own models across their own pages. A third-party board the vendor does not control is stronger evidence, and mixing the two into one ranking would imply they are equally rigorous.
Not directly. These benchmarks use public spreadsheets, tables and terminal tasks, not your schema, your dialect or your data volumes. The index is a prior, not a prediction. The clearest warning on the page is GPT-6 Astra: first on Terminal-Bench and first on LiveBench Data Analysis, sixth on the Excel modelling benchmark.
Because it is measured, not estimated. It is the actual spend to complete one benchmark task, including whatever reasoning the model chose to do. That is why Claude Sonnet 5 costs more per task than Claude Opus 5 despite a list price less than half as high — Opus 5 finishes tasks in fewer tokens.
It is regenerated from the source boards rather than hand-edited, so it moves when they do. The Vals boards were read on 2026-09-05 and the LiveBench release used here is dated 2026-06-25. Leaderboards restate scores as evaluation suites are corrected, so treat this as a snapshot with the date attached.
Different task sets, different harnesses, different scoring. GLM 5.2 beats GLM 5.3 on both data-analysis components and loses on both agentic ones — neither board is wrong; they measure different things. That disagreement is a large part of why the index blends four boards rather than trusting one.
Latency and output-token counts, which none of these boards publish per task in a form worth reproducing. Also any model too new or too small to have been picked up by an independent evaluator — absence here means unmeasured, not bad.
Every figure on this page traces back to one of these published references.
Provenance
Every number on this page is read from an independent third-party leaderboard, never from a vendor's own announcement. These are not SyneHQ measurements.
SyneHQ Analytics Agent Index v0.2 · last updated 2026-09-09. All evaluations
