Evaluation methodology
A score should come
with a way to check it.
Our existing evaluation pages curate externally reported results. This kit is a separate, small SQL correctness harness. We have not run a model benchmark with it and publish no model scores from its self-test.
What the kit checks
- Paid revenue grouping without pending orders.
- Distinct customers and NULL handling.
- Customers with no orders, including NULL join keys.
- Half-open monthly date boundaries.
- Zero versus missing revenue and refund rows.
Five tasks each run against three synthetic fixtures, including empty data. Answers must match column names, row order, and values. SQLite execution denies writes and attachment operations and uses a one-second progress deadline and 1,000-row result cap.
Run it locally
Download the Python harnesspython3 sql-harness.py --self-test python3 sql-harness.py --tasks python3 sql-harness.py --answers answers.json
The standard library is sufficient. No API keys, database credentials, or customer data are needed. Supply SQL answers as a JSON object keyed by task ID. The reference SQL is public, so this is a regression kit, not a contamination-resistant benchmark.
Before publishing model results
Record the exact model identifier and version, date, prompt, schema, tools, temperature, seed when available, token budget, retry policy, and harness commit. Keep first-attempt and retry-assisted success separate. Report median and tail latency, token usage, the dated price schedule, and measured cost. Missing measurements stay missing.
This kit cannot establish production reliability, dialect coverage, safe tool use, long-horizon planning, or performance on private schemas. Agno/Kole changes additionally require the product’s human-approval matrix and golden-set tests. Compare models on held-out tasks and repeated runs before making product or marketing claims.
Read the externally sourced evaluations