# Data-quality flags: per-source lookup

Flags are **not** standardised across agencies, and they shift over time within an agency. Resolve them per source and per vintage. The same character means different things in different systems — `E` is "estimated" in SDMX `CL_OBS_STATUS` but "use with caution, high CV" at Statistics Canada. Never map a flag between systems without checking both tables.

**This file owns the publication conventions, not the SDMX code lists.** Eurostat's `:c`, ABS's `n.p.` and Statistics Canada's CV grades are symbols in rendered tables — they appear in no DSD, so they are curated here and are stable. SDMX `OBS_STATUS`/`CONF_STATUS` are the opposite: the DSD names the code list, providers use their own, and those lists change. Those are resolved at runtime — see the SDMX section below — and the last column of the table opposite therefore says "resolve per DSD" rather than guessing.

**This file is the single source of flag semantics for everything it does own.** Two different consumers read it for two different purposes and should not maintain their own copies: this skill reads it to render a flag against a reported value; `reasoning-linkage-confidence` and `guarding-derived-indicators` read it to detect a break, a provisional status or a confidentiality suppression that changes whether an operation is admissible. One table, two questions.

Contents: [cross-agency table](#cross-agency-comparison) · [Eurostat](#eurostat) · [ABS](#abs) · [Statistics Canada](#statistics-canada) · [SDMX — resolve at runtime](#sdmx-obs_status-and-conf_status--resolve-these-at-runtime) · [rendering rules](#rendering-rules)

## Cross-agency comparison

| Concept | Eurostat | ABS | Statistics Canada | SDMX (`OBS_STATUS` / `CONF_STATUS`) |
|---|---|---|---|---|
| Not available | `:` (never "NA"/"ND") | `n.a.` | `.` (any period) / `..` (specific period) | resolve per DSD |
| Not applicable | `–` or `z` | `. .` | `...` | `z` is a DSD sentinel; otherwise resolve per DSD |
| Real zero | `0` | `—` nil or rounded to zero | `0`; `0s` where meaningful | — |
| Provisional | `p` | `p` (older publications) | `p` | resolve per DSD |
| Estimated | `e` | — | — | resolve per DSD |
| Forecast | `f` | — | — | resolve per DSD |
| Break in series | `b` or `|` | — | — | resolve per DSD |
| Confidential | `c` (as `:c`) | `n.p.` not for publication | `x` (Statistics Act) | resolve per DSD (`CONF_STATUS`) |
| Low reliability | `u` | see RSE markers below | `E` use with caution; `F` too unreliable | resolve per DSD |
| Definition differs | `d` (see metadata) | — | — | — |
| Not significant | `n` (as `:n` or `0n`) | — | — | — |
| Imputed | — | — | — | resolve per DSD |
| Revised | — | — | `r` | resolve per DSD — **`R` is not "revised"** in `CL_OBS_STATUS` 2.3 |

## Eurostat

Coding rules (https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Tutorial:Symbols_and_abbreviations):

- `0n` = less than half of the final digit shown and greater than real zero.
- `c`, `z`, `n` are meaningful **only in combination with `:`** — i.e. `:c`, `:z`, `:n`.
- `e`, `f`, `p` attach only to a statistical value.
- `b`, `d`, `u` attach to either.
- Not-available and confidential flags must **always** be included in output, never dropped.

**Vintage break:** from **27 January 2025** Eurostat dispatches flags across two SDMX-aligned code lists, `Obs_status` (observation status) and `Conf_status` (confidentiality status). `N` (not for publication) and `P` (information under non-statistical secrecy arrangements) were newly introduced, and three earlier flags were recoded. Data extracted before and after this date carries different flag semantics — check the extraction date before interpreting.

EU-LFS specifics: `a` = below reliability limit (suppress, publish a dot); `b`-flagged values shown in brackets; `u` = low reliability.

Flag propagation to aggregates is non-trivial in Eurostat's own documentation — do not assume an aggregate inherits a clean flag from flagged components.

## ABS

**Relative standard error markers (current convention):**

| Marker | RSE band | Interpretation |
|---|---|---|
| `^` | 10–25% | Use with caution |
| `*` | 25–50% | Use with caution |
| `**` | > 50% | Too unreliable for general use |

Older ABS publications use a different convention: `*` = high standard errors, `**` = RSE > 50%. Check the publication vintage before interpreting an asterisk.

Full ABS abbreviation set also includes `n.e.c.` (not elsewhere classified), `n.e.i.` (not elsewhere included), `n.e.s.` (not elsewhere specified), `n.f.d.` (not further defined), `n.p.` (not for publication), `n.y.a.` (not yet available).

Source: https://www.abs.gov.au/Ausstats/abs@.nsf/0/F41289256D682C7CCA2570DD00827251

## Statistics Canada

Quality is graded A–F by coefficient of variation for sample surveys. Published thresholds:

| CV | Treatment |
|---|---|
| ≤ 16.5% | Presented without restriction |
| > 16.5% and ≤ 33.3% | Interpret with caution; `E` symbol appears in the data cell |
| > 33.3%, or sample size < 10 | Suppressed; `F` symbol appears in the data cell |

`A`–`D` denote levels of "acceptable or better". A yellow-triangle `C` denotes a high-level correction.

When reproducing a Statistics Canada figure carrying `E`, state the CV band in the note — the letter alone does not tell the reader what it means.

Sources: https://www.statcan.gc.ca/en/concepts/definitions/guide-symbol and 82-221-x.

## SDMX `OBS_STATUS` and `CONF_STATUS` — resolve these at runtime

**There is no table for these here, deliberately.** `OBS_STATUS` and
`CONF_STATUS` are DSD-declared attributes: the data structure definition names
the code list that decodes them, and providers routinely point at their own
rather than the SDMX cross-domain list. Eurostat maintains its own `Obs_status`
and re-coded three flags on 27 January 2025. A table shipped in this file cannot
know which list a given dataflow uses, and cannot know when one changes — but it
will answer anyway, which is worse than not answering.

That is not hypothetical. A cross-domain copy carried here until 2026-08-19
glossed `R` as "revised". In `CL_OBS_STATUS` 2.3 `R` means **"excludes one or
more subcategories"** — a statement about which subpopulations the number
covers, not about whether it was revised. Anything built on that gloss was
wrong about what the observation counted.

### How to resolve one

1. `inspect(agency_id, dataflow_id)` and read the code list reference on the
   `OBS_STATUS` (or `CONF_STATUS`) attribute — agency, id and version.
2. Retrieve that code list.
3. Decode the code against **that** list, and carry the list's identity with
   the meaning.

If you are working from the `citing-official-statistics` bundle, its
`sdmx_cite.py` helper does step 3 for you — `flag --code <CODE> --codelist
<file>` accepts either `{"agency_id", "id", "version", "codes": {...}}` or an
SDMX-JSON structure message unchanged, and prints the meaning prefixed with the
list it came from, e.g. `ESTAT:OBS_STATUS(1.0): p = Provisional`. The other
skills that read this file do not ship it; for them steps 1-3 are the whole
procedure.

Quote that provenance when you publish the flag. "Provisional per
`ESTAT:OBS_STATUS(1.0)`" is checkable; "provisional" alone is not.

### When you cannot resolve it

Report the raw code and the code list it belongs to, and **publish no meaning
for it**. `sdmx_cite.py` refuses rather than falling back, and so should you. A
plausible gloss that turns out to be another list's is indistinguishable from a
correct one at the point a reader relies on it.

### Two traps worth knowing even so

- **A single character is never enough.** Eleven codes — `A D E F G M N O P S X`
  — appear in both the SDMX cross-domain `CL_OBS_STATUS` and `CL_CONF_STATUS`
  with unrelated meanings. `D` is "definition differs" against one and
  "secondary confidentiality" against the other; `N` is "not significant"
  versus "not for publication"; `S` is "strike" versus "secondary
  confidentiality set by the receiver". You must know which attribute the value
  came from, not just the letter.
- **The list is heterogeneous by design.** It mixes quality status and
  missingness in one dimension, and SDMX recommends a single-flag hierarchy
  where only one flag per value survives. A value's true status may be
  under-reported by that hierarchy regardless of how well you decode it.

Related structural metadata that must be read alongside flags:

- `PRE_BREAK_VALUE` — retains the pre-break value where a series break is flagged. Preserve it when charting across a break.
- `UNIT_MEASURE`, `UNIT_MULT`, `DECIMALS` — see the pre-emit checks in SKILL.md.

## Rendering rules

These generalise across all sources:

1. **Codes go in a separate column or a legend**, never as inline superscripts against a value. Inline superscripts collide with footnote markers and with digits, and corrupt machine-readable output.
2. **Never silently drop a flagged observation.** Missing and confidential cells must be visible as such.
3. **Flags survive derivation.** An aggregate over a series containing a break, a provisional value or a confidential suppression must carry a note saying so.
4. **State the threshold with the marker.** An asterisk or a letter grade without its RSE or CV band is uninterpretable to the reader.
5. **Presentation symbols are not source flags.** The en dash / two dots / `n/a` convention used in a typeset table is a display layer over the source's own flag; keep the source flag in the underlying data and metadata.