Open source & research
In development
Analytics Agent Reliability Harness
An open-source harness for evaluating whether AI analytics agents return correct business answers, not merely syntactically valid SQL.
The problem
Modern analytics agents can generate SQL that executes cleanly and still returns the wrong business answer. The query runs, the chart renders, and the number is wrong in a way that is easy to miss and expensive to trust.
That failure mode does not show up in most evaluation approaches, which stop at whether a query is valid, not whether it is true.
What it evaluates
01
Business-truth accuracy
Does the answer match the actual business fact, not just a plausible-looking one.
02
Golden question and answer evaluation
A fixed set of known-correct question/answer pairs to evaluate against, so results are comparable over time.
03
Semantic correctness
Whether the agent respects how a metric is actually defined, not a nearby approximation.
04
Repeatability
Whether the same question produces the same answer, run after run.
05
Abstention behavior
Whether the agent says it does not know when it does not, instead of guessing with confidence.
06
Query correctness and cost
Whether the underlying query is efficient and correct, not just eventually right.
Why this exists
This is a genuine open-source and research contribution first: public benchmark work that demonstrates real technical depth in an area most teams have not rigorously measured.
It is not, today, a product or a company. It may eventually create demand for a focused Analytics Agent Reliability Audit as a commercial offer, but that is not the primary way to engage Data Outfit right now. The Terrain Survey is.
The repository will be linked here once it is public.
Looking for the commercial work?
This is research and open source, not the primary way to engage me. Start with the Terrain Survey instead.