Spider 2.0-Snow
6 of 8 published- Claude Sonnet 4.590.5
- Claude Opus 4.686.3
- GPT-565.6
- o362.9
- GLM-550.3
- Claude Sonnet 425.8
0–100 scale · 25.0% of the index
Evaluation
Text-to-SQL Leaderboard
v0.1Updated
What the public SQL benchmarks actually measure, and which models they have got to.
A curated view of the public text-to-SQL and SQL-debugging leaderboards — Spider 2.0, BIRD-CRITIC and DABstep — covering the models those boards have validated.
The index combines 5 evaluations: Spider 2.0-Snow, Spider 2.0-Lite, Spider 2.0-DBT, DABstep (Hard split), and BIRD-CRITIC 1.0-Open.
Claude Opus 4.6 leads the index at 66.1, ahead of o3 at 59.0 — a 7.0-point gap.
- Models compared
- 8
- Component benchmarks
- 5
Ranked results
Every model, ranked
Higher is better
- 1Claude Opus 4.6 (QUVI-3 (DAQUV))66.1
- 2o3 (ReFoRCE (Hao AI Lab x Snowflake))59.0
- 3Claude Sonnet 4.5 (Prism Swarm with Deepthink (Paytm))54.1
- 4GPT-5 (SSDAT)52.3
- 5GLM-5 (PGV-Agent)41.3
- 6Claude Sonnet 4 (Spider-Agent)24.8
- 7Claude 3.7 Sonnet14.1
- 8GPT-4.111.7
Leaderboard · v0.1
Full results table
Sort by any column. Filter by provider or model name.
| # | Source | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 4.6Anthropic | QUVI-3 (DAQUV) | 66.1 | 86.3% | — | — | — | 40.9% | public-benchmarkSpider 2.0 leaderboardBIRD-CRITIC leaderboard |
| 2 | o3OpenAI | ReFoRCE (Hao AI Lab x Snowflake) | 59.0 | 62.9% | 55.2% | — | — | — | public-benchmarkSpider 2.0 leaderboard |
| 3 | Claude Sonnet 4.5Anthropic | Prism Swarm with Deepthink (Paytm) | 54.1 | 90.5% | 41.9% | 35.3% | — | 33.3% | public-benchmarkSpider 2.0 leaderboardBIRD-CRITIC leaderboard |
| 4 | GPT-5OpenAI | SSDAT | 52.3 | 65.6% | 54.7% | 41.2% | 57.7% | 33.0% | public-benchmarkSpider 2.0 leaderboardDABstep leaderboard (Adyen x Hugging Face), Validated tabBIRD-CRITIC leaderboard |
| 5 | GLM-5Zhipu AI | PGV-Agent | 41.3 | 50.3% | 32.4% | — | — | — | public-benchmarkSpider 2.0 leaderboard |
| 6 | Claude Sonnet 4Anthropic | Spider-Agent | 24.8 | 25.8% | 27.8% | — | 19.8% | — | public-benchmarkSpider 2.0 leaderboardDABstep leaderboard (Adyen x Hugging Face), Validated tab |
| 7 | Claude 3.7 SonnetAnthropic | — | 14.1 | — | — | 14.7% | 13.8% | — | public-benchmarkSpider 2.0 leaderboardDABstep leaderboard (Adyen x Hugging Face), Validated tab |
| 8 | GPT-4.1OpenAI | — | 11.7 | — | — | 10.3% | 12.4% | — | public-benchmarkSpider 2.0 leaderboardDABstep leaderboard (Adyen x Hugging Face), Validated tab |
Every number is read from a public benchmark leaderboard, never from a vendor's own announcement. These are not SyneHQ measurements. A dash (—) means the figure is not published, not zero.
Score by benchmark
What the Text-to-SQL Leaderboard is made of
The index score is a weighted average of the component benchmarks below. Every component is a public leaderboard or paper result, so a model only appears on a benchmark its authors actually published — a model missing from a panel was never run on it, and is not scored zero.
Spider 2.0-Lite
5 of 8 published- o355.2
- GPT-554.7
- Claude Sonnet 4.541.9
- GLM-532.4
- Claude Sonnet 427.8
0–100 scale · 25.0% of the index
Spider 2.0-DBT
4 of 8 published- GPT-541.2
- Claude Sonnet 4.535.3
- Claude 3.7 Sonnet14.7
- GPT-4.110.3
0–100 scale · 10.0% of the index
DABstep (Hard split)
4 of 8 published- GPT-557.7
- Claude Sonnet 419.8
- Claude 3.7 Sonnet13.8
- GPT-4.112.4
0–100 scale · 20.0% of the index
BIRD-CRITIC 1.0-Open
3 of 8 published- Claude Opus 4.640.9
- Claude Sonnet 4.533.3
- GPT-533.0
0–100 scale · 20.0% of the index
Components and weights
Spider 2.0-Snow
25.0% of the indexThe agent is dropped into a real Snowflake warehouse with prepared metadata and asked to answer an enterprise analytics question end to end. Tables routinely carry over 1,000 columns and correct answers often need SQL longer than 100 lines. Scored on execution accuracy.
- Tasks
- 547
- Weight
- 0.25
Spider 2.0-Lite
25.0% of the indexThe same enterprise text-to-SQL tasks spread across BigQuery, Snowflake and SQLite, so a model cannot succeed by memorising one dialect. The agent explores the schema itself and returns a result set, not just a query string.
- Tasks
- 547
- Weight
- 0.25
Spider 2.0-DBT
10.0% of the indexRepository-level tasks where the agent reads an existing dbt project and writes or edits transformation code against DuckDB, rather than answering a single question. Scored on success rate. This split replaced the original full agentic setting on 2025-05-22.
- Tasks
- 68
- Weight
- 0.1
DABstep (Hard split)
20.0% of the indexAdyen's data-agent benchmark: the agent gets raw payments files plus a domain manual and must chain several analysis steps to one factoid answer. No hard-split task is solvable in a single shot of code. Validation is now closed pending a v2, so this board will not gain newer models.
- Tasks
- 378
- Weight
- 0.2
BIRD-CRITIC 1.0-Open
20.0% of the indexInstead of writing SQL from scratch, the model is handed a real user-reported SQL bug across four dialects and must fix it so the query runs and returns the right rows. Scored as success rate against a held-out test suite.
- Tasks
- 570
- Weight
- 0.2
Index score = 25.0% × Spider 2.0-Snow + 25.0% × Spider 2.0-Lite + 10.0% × Spider 2.0-DBT + 20.0% × DABstep (Hard split) + 20.0% × BIRD-CRITIC 1.0-Open. Raw weights (0.25 : 0.25 : 0.1 : 0.2 : 0.2) are shown as percentages of their total, so the 5 components add up to 100% of the index.
Figures are curated public data — vendor announcements and public benchmark leaderboards. They are not SyneHQ measurements.
Reading the index
Methodology
The index is a weighted average of five components: Spider 2.0-Snow at 25%, Spider 2.0-Lite at 25%, DABstep hard split at 20%, BIRD-CRITIC 1.0-Open at 20% and Spider 2.0-DBT at 10%. Weighting favours the enterprise warehouse settings over the smaller dbt split.
Rows that name no model are excluded. A large share of the current Spider 2.0 top ten are vendor agents that publish no underlying LLM — Genloop at 96.7 and Native mini at 96.53 both outrank every named model on Snow. They are real results, but this page ranks models, and a row with no model cannot carry a provider or a release date. One further row was dropped for naming a Sonnet version it did not specify, and one for filing an agent named 'gpt5' under the model family 'Doubao'.
Where a model appears several times under different harnesses, the best published score is used and the harness is named on the row. GPT-5 appears three times on Spider 2.0-DBT alone. This follows the boards' own convention but means a score reflects a model-plus-scaffold, not a bare model.
That distinction matters most on DABstep, where the leaders are scaffolds rather than models: the top row is NVIDIA's agent toolkit driving Claude Haiku 4.5 at 89.95, while the bare-model baselines on the same board sit between 8 and 20. Every row here names its harness where the board published one.
A model needs at least two published components to be ranked. Where a model is missing some, the index is renormalised over the components it does have and the row says so. A missing benchmark renders as an em dash, never as zero. The two-component floor exists because a weighted average over one benchmark is not an index — it is that benchmark wearing a composite's name, and it would rank a model measured once above a model measured five times.
These boards are slower than the release cycle, and that is the main thing to take from this page. None of them lists Claude Opus 5, Claude Sonnet 5, Claude Fable 5.1, GPT-6 Astra, GPT-5.6 Sol, Grok 4.6, Kimi K3, DeepSeek V4 Pro or GLM 5.3. DABstep has closed validation pending a v2 and will not gain them. The Analytics Agent Index page covers those models on the benchmarks that do measure them.
Per-row dates published on the Spider 2.0 board are visibly unreliable — one row carries 'Dev 2, 2025' and a Claude Opus 4.5 submission is dated January 2025, before that model existed — so they are not reproduced here. Cost and latency are absent because none of these three boards publish them.
Excluded by that floor, with their single published result stated so nothing is hidden: Gemini 3 Pro (preview) at 94.15 on Spider 2.0-Snow; Claude Haiku 4.5 at 89.95 on DABstep (Hard split); Qwen3 at 87.57 on DABstep (Hard split); GLM 5.2 at 76.23 on Spider 2.0-Lite; DeepSeek V3.2 at 70.38 on Spider 2.0-Snow; Claude Opus 4.5 at 65.81 on Spider 2.0-Lite; gpt-oss-120b at 65.45 on Spider 2.0-Snow; GPT-5.4-mini at 50.46 on Spider 2.0-Snow; Gemini 3.1 Pro (preview) at 45.96 on BIRD-CRITIC 1.0-Open; Gemini 2.5 Pro at 45.24 on DABstep (Hard split); Qwen3-Max at 45.15 on Spider 2.0-Snow; DeepSeek V4 Flash at 42.4 on Spider 2.0-Lite; GPT-5.4 at 39.71 on Spider 2.0-DBT; Qwen3-Coder-Plus at 37.8 on Spider 2.0-Snow; GPT-5 (high effort) at 34.96 on BIRD-CRITIC 1.0-Open; Grok 4 at 33.68 on BIRD-CRITIC 1.0-Open; DeepSeek R1 at 33.51 on BIRD-CRITIC 1.0-Open; Kimi K2.5 at 32.63 on BIRD-CRITIC 1.0-Open; Qwen3-Coder-480B at 32.63 on BIRD-CRITIC 1.0-Open; GLM-4.7 at 32.46 on BIRD-CRITIC 1.0-Open; Claude 3.5 Sonnet at 28.04 on DABstep (Hard split); MiniMax M2.1 at 26.84 on BIRD-CRITIC 1.0-Open; DeepSeek V3 at 16.4 on DABstep (Hard split); o4-mini at 14.55 on DABstep (Hard split); Llama 4 Maverick at 8.73 on DABstep (Hard split). Each is a real, cited score — it is the composite built on top of one number that would have been fiction.
Frequently asked
Why does the Spider 2.0 leader not appear on this page?
Because it names no model. The top Spider 2.0-Snow rows are vendor agents — Genloop's Sentinel Agent at 96.7, Native mini at 96.53 — that disclose no underlying LLM. They beat every named model on the board. This page ranks models, so they are excluded here, but the exclusion is worth knowing about: a purpose-built system currently outperforms any general model on that benchmark.
Why are the newest models missing?
They have not been submitted or validated yet. Public SQL leaderboards take months to verify a submission, and DABstep has closed validation entirely pending a v2. Every SQL board we checked — Spider 2.0, BIRD-CRITIC, DABstep, LiveSQLBench and the BIRD Data Intelligence Index — stops at least one model generation short of what is currently shipping.
Is a score here a bare model or a model in an agent?
Almost always a model inside an agent. Every row on all three Spider 2.0 boards is an agent submission, and DABstep's leaders are scaffolds. The harness is named on each row where the board published it. Two models under different harnesses are not directly comparable, which is why the component columns matter more than the index.
Are these SyneHQ's own measurements?
No. Every figure is read from a public leaderboard and cited per row. SyneHQ ran none of these evaluations and has no submission on any of these boards.
Why is Claude Sonnet 4 still listed if it is retired?
Anthropic retired it on 2026-06-15, but it remains the reference point that many published comparisons are anchored to, and it appears on three of these five components. It is here for continuity, not as a recommendation.
How reliable are these numbers?
The boards themselves warn that scores shift as evaluation suites are corrected, so this is a dated snapshot. One further caution from collecting it: fetching the BIRD-CRITIC page as raw HTML returned a materially stale table that disagreed with the live page. The figures here were read from the rendered boards on 2026-09-09.
Sources
Every figure on this page traces back to one of these published references.
Provenance
Every number is read from a public benchmark leaderboard, never from a vendor's own announcement. These are not SyneHQ measurements.
Text-to-SQL Leaderboard v0.1 · last updated 2026-09-09. All evaluations