📌 Key takeaways
- 1. The field separates on challenging and hard questions: Spotter answered 70.6% of the challenging band, while the lowest engine managed 46.8%.
- 2. Spotter scored 82.8% on BIRD (across all the difficulty bands), the highest of the five engines tested, with a warehouse-native assistant 0.8 points behind.
- 3. Accuracy slope, not headline accuracy, is what predicts performance on your data.
- 4. Swapping in a newer model added 3.4 points overall with no change to the agent harness underneath it.
- 5. Full protocol, confidence intervals, and per-question results are available on request.
Analytics vendors claim their AI answers questions accurately, but almost none of them will show you how they checked.
The standard move is a percentage with no denominator: "90%+ accuracy on internal benchmarks." No dataset you can download. No scoring method you can inspect. No competitor runs under the same conditions. You're asked to trust the grade without ever seeing the exam1
We ran the exam in public terms instead. This summer we put Spotter and four competing agentic analytics engines through BIRD, the most widely cited public benchmark for natural-language questions over realistic databases: 1,533 questions, 11 databases, one scoring method for everyone, one attempt per question.
The results: Spotter answered 82.8% correctly, ahead of the warehouse-native assistant at 82.0%, the frontier LLM at 78.4%, the lakehouse agent at 74.9%, and the coding agent at 72.5%.
But it's necessary to understand these numbers in the knowledge work space. BARC's BI & Analytics Survey puts active BI adoption at roughly a quarter of employees, a share that has barely moved in seven years.
For example: In a 10,000-person enterprise, that is 2,500 analytics users; at five data questions a week each, about 650,000 questions a year. One accuracy point is 6,500 answers a year arriving right instead of wrong. The 10.3-point gap to the coding agent needs no caveat: 67,000 wrong answers a year.
Figure 1. Overall execution accuracy, five engines, BIRD development split dev_20251106. Pass@1, single attempt.
What Does "Simple to Challenging" Mean When the Data Is Yours?
BIRD grades questions in three bands: simple (859 questions), moderate (443), challenging (231). The labels track the structure of the required SQL, not the length of the question.
Five engines sat the same exam over the same source data. We report them as categories rather than product names, and the reproduction pack names every one:
Spotter, working from a governed semantic model
A warehouse-native assistant, reading the raw schema inside its own platform
A lakehouse AI/BI agent, working from its catalog
A frontier LLM writing SQL directly
A coding agent discovering the schema at question time
Take sales ops at a B2B software company. The warehouse under a RevOps analyst is never one clean table. It's an opportunity table joined to accounts, owners, and territories. A stage-history table where every deal appears once per stage change, so counting rows counts nothing.
Three amount fields: amount, ARR, TCV, that agree on good days. A fiscal calendar that starts when finance says it starts. And a "segment" that isn't a column at all; it's a rule about employee count and spend that lives in a doc somewhere.
Now climb the ladder with one team's questions.
Simple: "How many open opportunities do we have in EMEA right now?" One table, two filters. The only work is mapping open onto the right stage values. BIRD's version of this tier is counting trading cards by border color, and every engine we tested cleared 80% here.
Moderate: "Which enterprise accounts have open deals created this fiscal year, and who owns them?" Now the answer spans three tables, enterprise resolves to a rule rather than a column, and this fiscal year is not January. BIRD's version is joining a school-finance table to a directory to find charter schools by a Y/N flag and a date stored as text. This is the difficulty level where an engine without a map of your schema starts guessing join paths.
Challenging: "Among the deals sourced by our top SDR, what share closed above $100k?" Top SDR is a superlative that has to become a subquery, before you even get to the ratio – and the ratio has a right denominator and several wrong ones. This is structurally the same question as BIRD's challenging tier, where the field separated hard: Spotter answered 70.6% of that band, the coding agent 46.8%.
The hard questions aren't hard because of SQL syntax. They're hard because of your business: which definition of win rate, whose fiscal calendar, which of the three amount fields is true.
For the RevOps analyst, the visible win is skipping the SQL. For the CRO, the win that matters is whether the coverage number that reaches the board deck is right.
Why Did Five Engines See It Differently?
Every engine answered the same questions over the same source data. No two engines saw the same data model. The coding agent and the frontier LLM discover the schema at question time: list the tables, read the column names, infer the joins, and hope the names tell the truth.
The warehouse-native assistant reads the raw schema inside its own platform with retrieval on top. The lakehouse agent works from its catalog. Spotter works from a governed semantic model, where the join topology, metric definitions, and business vocabulary are encoded once, ahead of every question.
That difference is invisible on the simple tier, and it decides the challenging one.
One honest wrinkle: BIRD ships an evidence hint with every question, a one-line note like “black border card refers to borderColor = 'black'”. Every engine got the hints; the protocol was identical.
But notice what the hint is. It's a human handing the engine exactly the business definition it can't infer from the schema.
In production, nobody types an evidence hint into a chat box at 8 a.m. before a forecast call. The place those definitions live permanently is the semantic model.
The benchmark hands every engine a sticky note; Spotter is the only one that keeps the filing cabinet.
Where Does the Lead Live?
Figure 2. Accuracy by BIRD difficulty band, pass@1. The slope from simple to challenging separates the field.
Read the margins honestly. On simple and moderate questions, Spotter's lead over the warehouse-native assistant is 0.1 and 0.4 points, one and two questions. Call that a tie.
The durable gap is at the challenging difficulty level: 70.6%, a full 3.9 points and nine questions clear of the nearest competitor, on the tier where "top SDR" has to become a subquery.
For calibration: a 231-question band carries a sampling interval of about ±6 points, so we publish that margin with its interval attached, and we're running a paired per-question analysis before calling it definitive.
They are the multi-join, multi-condition, business-logic questions, and they are the closest thing the exam has to your enterprise data. Here is what the players did on them.
| Engine | Challenging (231) | Simple to Challenging |
|---|---|---|
| Spotter | 70.6% | -16.4 pts |
| Warehouse-native assistant | 66.7% | -20.2 |
| Lakehouse AI/BI agent | 58.4% | -23.0 |
| Frontier LLM + SQL | 56.7% | -26.8 |
| Coding agent + SQL | 46.8% | -33.5 |
Start with the bottom of that table, because the collapse is the story. The coding agent posts a respectable 72.5% overall and then answers fewer than half the hard questions correctly.
That is 55 questions of daylight between it and Spotter, on the only band anyone should care about. The lakehouse agent gives up 28 questions and the frontier LLM 32.
Those three gaps are large; they sit well outside the sampling intervals, and they hold at p < 0.05 or better.
The slope matters more than any single point. From simple to challenging, Spotter loses 16.4 points of accuracy. The warehouse-native assistant loses 20.2, the lakehouse agent 23.0, the frontier LLM 26.8, and the coding agent 33.5.
Every engine looks smart on easy questions. The slope tells you which one to trust with hard ones – and the coding agent's respectable 72.5% overall conceals a fall to 46.8% exactly where questions start resembling your pipeline review. Barely one in two.
That matters because moderate and challenging are not the exam's edge case. It’s the enterprise workload: multi-table joins, definitions that live in no column, fiscal calendars that start when finance says so, while single-table questions are the demo.
What Does a Model Upgrade Buy?
Between Spotter's two runs, exactly one variable changed: Claude Opus 4.8 became Claude Opus 5. Semantic models, prompts, and protocol were identical. Accuracy rose 3.4 points overall and 4.8 on challenging questions. All 11 databases improved, with the biggest gains landing on the most join-heavy schemas. Nothing regressed, and tool calls per question stayed flat at 2.1, so the gain is reasoning quality, not extra work.
Here is what that experiment actually measures. The model is one component inside an analyst harness: the governed semantic model underneath it, the coached definitions and reference questions it accumulates in Spotter Memory, and the verification layer that checks generated queries before results ship.
All of that context was built once and carried over untouched between the two runs. The upgrade was a component swap, not a rebuild.
That is also why the gain compounds. Teams that wire a frontier LLM straight to the warehouse rebuild their prompt scaffolding every model release, and the fixes they accumulated die with the old prompts.
A harness keeps everything the deployment has already learned, every learned definition, every verified join path, and each new model lands on top of it.
The context compounds; the model swaps. Frontier models improve every few months, and a platform that absorbs them cleanly converts each release into an accuracy dividend on day one. Swap the model. Keep the enterprise knowledge.
Opus 4.8 → Opus 5: What the Upgrade Bought
| Measure | Opus 4.8 | Opus 5 | Change |
|---|---|---|---|
| Accuracy | 79.4% | 82.8% | +3.4 pts |
| Challenging questions | 65.8% | 70.6% | +4.8 pts |
| Tool calls per question | 2.12 | 2.13 | unchanged |
| Data models improved | — | 11 of 11 | no regressions |
What Does This Mean for You?
Every engine we tested aced the easy questions; that headline could have been any vendor's. Spotter held 70.6% there while the other players fell as low as 46.8%, and the slope, not the headline number, is what tells you which engine to trust with those questions.
The definitions and join paths those questions turn on live in the semantic model and in Spotter Memory, built once, compounding with every question taught and every model release absorbed on top.
Think of it this way: The model is the commodity, the data model is the moat, and the harness is the compounding asset.
Want to try Spotter on your own data? Request your live demo.
Appendix:
Protocol and scoring. BIRD development split dev_20251106; every number is pass@1: one attempt, no retries, no consensus sampling, production configurations, identical prompting. Scoring is binary against BIRD's gold answers: a deterministic pre-pass auto-passes exact matches under numeric, NULL, date, case, and scale tolerances (it can only pass, never fail), then a rubric judge scores the remainder, seeing the question, expected output, and answer data, never the SQL. BIRD's official scorer could not run end-to-end because it executes SQL against SQLite, and not every engine emits portable SQL; Spotter answers through its semantic layer. The judge is [model], a different model family from any engine's generator, validated against a human-audited sample of [N] questions ([X]% agreement) and against the official execution match on the subset where both could run.
Statistics. Overall results carry a 95% confidence interval of roughly ±2 points; the challenging band is about ±6. Margins we call ties are not significant at these sample sizes. The 3.9-point challenging margin has not yet cleared a paired per-question test; we will update this post with the result either way. The gaps over the lakehouse agent, the frontier LLM, and the coding agent, and the Opus 4.8 to Opus 5 paired delta, are significant at p < 0.05 or better.
Known flaws in the answer key. The audit above found annotation problems in 52.8% of reviewed BIRD Mini-Dev examples and showed corrections can move scores by double digits and reorder closely spaced systems. All five engines used the identical key, so the error is symmetric, but it is a second uncertainty source beyond sampling. It is also independent evidence for the position this post takes: a benchmark score is where scrutiny starts, not where it ends. Within-system deltas, including the model upgrade, use the same key on both runs, so key errors cancel out of the difference. [If the re-score lands: we re-scored all five engines against the community-corrected annotations; the result appears here, whichever way it came out.]
Names and reproduction. Competitors appear as categories; the repro pack names exact products, versions, and configurations, with per-question results and judge transcripts, available to any customer or analyst on request.
How we scored it? Every number in this post is pass@1: one attempt per question, no retries, no consensus sampling, production configurations only, identical prompting for every engine. Scoring was binary pass/fail against BIRD's gold answers, in two stages. A deterministic pre-pass auto-passes exact matches under numeric, NULL, date, case, and scale tolerances – by construction, it can only pass an answer, never fail one. A rubric judge scores the remainder and sees three things: the question, the expected output, and the answer data. Never the SQL. An engine that reaches the right rows through an ugly query passes; an elegant query that returns the wrong rows fails. Which is exactly how your CFO grades it.
