Aardbench

Evaluating AI capabilities with official data.

Aardbench evaluates the capabilities of AI tools to use real-world data from official sources. The work breaks into three steps:

i)discover the right data
ii)extract it cleanly
iii)analyse and reason over what comes back

Aardbench runs as three suites: Aardbench Economics, Aardbench People, and Aardbench Development. Each suite shares a common framework measured across discovery, extraction, and analysis.

How Aardbench is organised

Aardbench is built on two concepts: functions and dimensions.

Functions are the goals a user pursues against official data. There are three:

1.Discovery — find a source of data that aligns with the use case or intent.
2.Extraction — bring the correct subset of data from the source into the workspace.
3.Analysis — analyse and reason over what comes back.

Each function decomposes into named skills and workflow steps, the specific competencies a capable system must master. Functions describe what the system has to do.

Dimensions are four orthogonal axes along which test cases vary:

1.Source prevalence — from common to regional sources. LLMs perform better with the most-cited sources (World Bank, OECD, Eurostat, IMF) than with regional ones (e.g. ABS, Stats NZ, SPC).
2.Data input complexity — from single-source questions (e.g. “What was the population of Australia in 2016?”) to multi-source questions (e.g. examining how changes in PISA scores by country correlate with education funding from national accounts data).
3.Concreteness — from concrete metrics (e.g. population, GDP) to abstract concepts (e.g. country risk).
4.Cognitive burden — from simple computation (e.g. averages, ratios) to deep reasoning (e.g. examining the impact of policies on a joint shift in economic and environmental indicators).

Every test case sits somewhere in this four-dimensional space. Dimensions describe the ways in which a use case can be hard, independent of which function it exercises.

A high-performing AI tool should succeed across every dimension within every function.

Dimensions of difficulty

Every test case sits somewhere in this four-dimensional space. Handling the easy corner of every dimension is not the same as handling the hard corner of any of them.

Source provenance

Common (UN, IMF, OECD, World Bank, ECB, BIS, Eurostat) Regional (SPC, STATEC, Fiji Bureau of Statistics, NSO Cambodia)

The same phrasing that resolves cleanly at a common provider often fails at a country or regional office.

Dependencies

Single datasource Multiple datasources

Multi-source questions force codelist harmonisation, unit reconciliation, and temporal alignment.

Definitional abstraction

Concrete metric (“CPI all items, Japan, monthly”) Abstract concept (“is the economy overheating?”)

Abstract questions have many candidate mappings. The model must do definitional work before any data is touched.

Computational abstraction

Simple (a ratio, a single read) Complex (index construction, standardisation, novel transformations, harmonising disparate definitions)

The retrieval is the easy half. The reasoning that turns numbers into an answer is where most systems break.

Functions and skills

Aardbench has three functions: Discovery, Extraction, and Analysis. Each one decomposes into a set of skills a capable system must master. Each function below carries one illustrative example. The examples are created for this page only and are representative of the private suite.

1. Discovery.

Given a question, find the right data.

1.1Dataset identification

Map plain-language phrasing to the correct dataflow. The space is tens of thousands of dataflows across more than 100 agencies.

1.2Dataset refinement

Narrow to the correct slice. Slices vary along five components: measure, geography, time window, breakdown, and frequency.

1.3Disambiguation

Some requests are ambiguous. “Show me debt” could mean government, household, corporate, or external. The system should ask which one, or state the assumption explicitly.

1.4Entity resolution

Map informal names to authoritative codelist values. Three examples. Eswatini and Swaziland refer to the same country. EA17, EA19, and EA20 are different vintages of the euro area. Georgia is both a country and a US state.

1.5Metadata awareness

Surface revision status, methodology, base year, and last-updated date. Recognise when a series is dormant, has been superseded, or has just been revised.

1.6Graceful degradation

Recognise when exact data does not exist. Two sub-cases:

1.6.1Unanswerability: Refuse cleanly rather than fabricate a dataflow.
1.6.2Proxy suggestion: Propose a defensible substitute and flag the limitation.

1.7Entity-based discovery

Handle open queries with no specific indicator named. Examples: “what do we know about Suriname?”, or “what data do we have on five-year-olds in Southeast Asia?”.

Illustrative example

Take “What’s the latest household debt level in Australia?” This should resolve to a household-credit stock series. Two reasonable candidates: RBA Statistical Table E2 (Selected Household Liabilities), or BIS Total Credit Statistics, households segment. It should not resolve to general government debt, household final consumption, or net household worth.

Now take “What’s the share of Australian household wealth held in crypto-assets?” This should be refused. No harmonised SDMX dataflow tracks household crypto holdings. The system should explain that, and then surface the closest proxies. Reasonable candidates: RBA Survey of Consumer Expectations, ABS Survey of Income and Housing, or aggregate exchange-reported data. The system should not fabricate a dataflow.

2. Extraction.

Turn a known target into a correct request.

2.1Generate API call: Produce a well-formed SDMX request. Four requirements: correct dimension key, correct codelist values, correct time filter, and no over-broad wildcards.
2.2Format return: Respect the requested output shape. Supported shapes: CSV, JSON, SDMX-ML, table, plot, or raw API URL.

Illustrative example

The target is the Eurostat EU-27 unemployment rate, seasonally adjusted, monthly, from 2015 onward, in CSV. A correct extraction produces a request with the following parts. Provider: ESTAT. Dataflow: une_rt_m. Dimensions: AGE=TOTAL, SEX=T, UNIT=PC_ACT, S_ADJ=SA, GEO=EU27_2020, FREQ=M. Time filter: startPeriod=2015-01. Format: CSV. A naïve system pulls the whole dataflow and leaves the user to filter. A correct one returns the exact slice.

3. Analysis.

Analyse and reason over the retrieved data.

3.1Conceptual reasoning: Work out what the question is actually asking. Then determine which indicators, transformations, and aggregations apply.
3.2Temporal resolution: Convert informal time expressions to precise start and end periods. Examples: “since Brexit”, “the Lost Decade”, or “YTD”. Account for non-Gregorian calendars where sources publish in them, such as the Buddhist calendar (Thailand) or Japanese era years.
3.3State management: Carry conversational context across turns. On a follow-up, modify only what the new information changes about the question.
3.4Structural break identification: Flag when a series crosses a methodology change. Examples: the transition in European System of accounts from 1995 to 2010, a base-year revision, or a classification change.
3.5Identifying disparate methodologies: Recognise when sources nominally measuring the same concept use incompatible definitions. Money supply across central banks is a typical case. The system should reconcile the difference, or flag it.
3.6Ontological reasoning over metrics: Distinguish flows from stocks, levels from growth rates, nominal from real, and gross from net.
3.7Ontological reasoning over other concepts: Reason about how categorical concepts compose, overlap, and decompose across sources. Categorical concepts include regions, sectors, age groups, and methodologies.

Illustrative example

Take “Has UK real wage growth recovered to its pre-Brexit trend?” This question exercises five skills.

First, temporal resolution. “Pre-Brexit” maps to the period before the 2016-Q2 referendum. The trend window is roughly 2010 to 2016.

Second, conceptual reasoning. Real wages are nominal earnings deflated by a price index. The trend is the compound growth rate over the reference window.

Third, retrieval. The system needs ONS Average Weekly Earnings, plus an ONS price index.

Fourth, ontological reasoning over metrics. The system has to distinguish levels from growth rates, and nominal from real.

Fifth, structural-break awareness. CPIH replaced CPI as the UK headline measure in 2017. ONS AWE classifications were revised in 2017.

Scoring

Scoring is skill-specific. Different skills demand different scoring rules.

Pass / fail. Identification and disambiguation. Either the system resolves the right dataflow, or it does not.

Dimension by dimension. Refinement and extraction. Scored against the expected slice across measure, geography, time window, breakdown, and frequency.

Rubric. Analytical skills. Indicator selection, transformation, and synthesis are scored against a written rubric.

Presence or absence. Robustness conditions. Includes unanswerability, proxy suggestion, structural-break warning, and follow-up coherence.

Contribute

Get in touch

If you have questions about the benchmark, would like to understand more, or would like to contribute, we’d love to hear from you.

benchmark@aard.ai