The Independent Analytics Agent Benchmark

Season 2026-S2 · question bank seed-v1 · judge claude-opus-4-8-r4 · k=1

Every configuration — an agent product, the LLM it ships with by default, and a semantic layer carrying equivalent content — answers the same 300 questions against the same deliberately messy simulated e-commerce warehouse, graded against ground truth that is never published. No vendor pays to appear here, none can pay to be removed, and none saw the questions or the results early.

Live board, receipts and per-configuration detail on benchouse.ai →

#	Configuration	Accuracy	Completeness	Restraint	$/question
1	Snowflake Cortex Agentsclaude-opus-4-8 · snowflake semantic view	83.7%246/294	99.7%	100.0%	$0.59
2	Hexgpt-5.6 (Hex) · hex semantic model (dbt)	74.4%195/262	100.0%	100.0%	$0.35
3	naoclaude-sonnet-5 · nao table docs	70.0%203/290	100.0%	100.0%	$0.95
4	SupersimpleLLM not disclosed · supersimple semantic model	69.7%202/290	98.3%	100.0%	—
5	Databricks GenieLLM not disclosed · genie space instructions	65.7%119/181	68.0%	80.0%	—
6	Snowflake Cortex Analystclaude-sonnet-4-6 · snowflake semantic view	54.8%131/239	81.0%	100.0%	$0.13
7	Lightdash AIclaude-sonnet-4-6 · lightdash semantic model	53.6%134/250	95.4%	100.0%	$0.31

What the three numbers mean

Accuracy — of the questions it answered, how many were right.
Completeness — of the questions it had the data to answer, how many it answered at all. Without this, an agent that declines everything hard looks excellent.
Restraint — a seeded subset cannot be answered from the data. The only correct response is to say so; answering anyway is how confidently wrong output gets counted.

What these numbers do not support

The judge's agreement with human graders is not yet measured for this judge version. Earlier versions were calibrated; this one has not been re-calibrated, and the board says so rather than borrowing an old number.
k=1. Each question is asked once, so run-to-run flakiness is invisible.
Products, not models. Each configuration runs its vendor-default LLM, so part of any gap belongs to the model underneath.
Warehouse reach differed between arms; it is disclosed per arm rather than equalised after the fact.
Answer-level grading only. The SQL each agent ran is recorded but not separately graded.
Scores never compare across seasons — each season regenerates the universe and rotates the traps, so rank is the only signal that carries.

Why there is no dataset here

The question bank and its truth ledger are withheld deliberately. The warehouse is simulated from a seed so that no agent has trained on it; publishing the questions would put them into scraped training corpora and every future season would measure recall instead of analysis. How it is measured →