Export

data-dict can render a data-dict.yaml file to JSON, resolving everything the validator already computes internally — parsed types, constraint flags, joins, assertions, definitions — so downstream tools don’t need to re-implement any of the spec’s parsing or inference rules.

Two levels

Export has two levels, mirroring validation:

• export-spec renders the dictionary itself: every table, column, relationship, and glossary entry, fully resolved. It reads only the data-dict.yaml file, never the data. Internally this runs the same validate-spec pass and serializes the resulting model, so export-spec fails with the same S## diagnostics as validate-spec if the file is invalid.

• export-data does everything export-spec does, and also profiles each table’s source data: distinct/missing counts, sample values, and a histogram (numeric and temporal columns) or top values (string, boolean, and enum columns) per column — the same profiling describe performs on a single Parquet file, run here across every table via each one’s source. This is the only level that reads the data, so it can be expensive. It does not validate the data against the dictionary — that’s validate-meta/validate-data’s job: a source that’s missing (M04) or unreadable (M05) is reported as a warning and that table’s profiles are omitted, and a declared column the data doesn’t have simply gets no profile, so a partially-sourced dictionary still exports everything it can.

Both levels emit the same JSON document shape; export-spec just never populates the profile fields.

For a ready-made example of the export-spec output, each dictionary on the examples page offers its JSON export alongside the raw YAML.

Output shape

A key with nothing to say is omitted rather than serialized as null or []: keys marked ? below may be absent, meaning the value wasn’t declared (or, for a profile statistic, couldn’t be established). Zeroes and falses are real data and always appear. Consumers should read absent and null interchangeably — jq, JavaScript property access, and optional-aware decoders already do.

Prose fields — every description, details, and todo, and each glossary definition — are written as Markdown in the dictionary and arrive here rendered to HTML (any raw HTML in the source is escaped rather than passed through), so consumers can place them straight into a page without a Markdown implementation of their own.

todo is carried wherever the spec allows it — the dataset, a table, a column or struct field, a relationship — so a consumer can show a dictionary’s unfinished work in place rather than only in a validator’s output, where validate-spec reports each one as an S31 warning. It is often a list of several tasks, since that’s how draft writes it.

Top level

{
  "$version": "0.1.0",           // version of the export document format itself
  "name?": "string",
  "label?": "string",
  "description?": "string",
  "details?": "string",
  "todo?": "string",
  "origin?": "string",
  "learn_more?": "string",
  "version?": { "number": "1.2.3" } | { "date": "2024-01-31" } | { "hash": "..." },
  "tables": [ Table ],
  "relationships?": [ Relationship ],
  "glossary?": [ { "term": "string", "definition": "string" } ]
}

Table

{
  "name": "string",
  "label?": "string",
  "description?": "string",
  "details?": "string",
  "todo?": "string",
  "origin?": "string",
  "source?": { "parquet": "path/relative/to/dictionary" },
  "rows?": 123,                  // the source data's row count; export-data only,
                                 // absent when the table's source wasn't profiled
  "columns": [ Column ],
  "constraints?": [ Assertion ], // table-level `assert` entries
  "definitions?": [ Definition ] // the table's `definitions`, resolved
}

Column

A Column is recursive: fields holds child Columns for struct and list(struct); a field uses the same shape, minus the keys the spec doesn’t allow on fields (label, display, constraints). A column or field with no declared type makes no claims and is omitted from the export entirely, including from COLUMNS(...) expansions:

{
  "name": "string",
  "label?": "string",
  "description?": "string",
  "details?": "string",
  "todo?": "string",
  "display?": "restricted",
  "type": "string",              // canonical type, e.g. "list(number(quantity))"
  "units?": "string",
  "time_zone?": "string",
  "constraints?": ["primary_key" | "foreign_key" | "unique" | "required", ...],
  // constraints as declared PLUS those implied by other constraints
  // (e.g. primary_key implies both unique and required)
  "references?": { "table": "string", "column": "string" },
  // present when this column is a `foreign_key`: the primary-key column it points at
  "referenced_by?": [ { "table": "string", "column": "string" } ],
  // present when this column is a `primary_key` that foreign-key columns
  // elsewhere reference: each of those columns
  "values?": [Scalar],            // enum
  "value_labels?": { "value": "what it means", ... },
  // present when `values` was written as a map, giving each value a label
  "range?": { "min": Scalar, "max": Scalar },
  "examples?": [Scalar],
  "fields?": [ Column ],
  "assertions?": [ Assertion ],   // column-level `assert` entries
  "profile?": Profile             // export-data only; never present under export-spec
}

values lists what an enum column may hold, in the order the dictionary declares them. Writing them as a map ({M: Male, F: Female}) labels each one; the values themselves still appear in values, and value_labels maps each to its label. A column whose values were written as a plain list has no value_labels — there, the values speak for themselves.

Both references and referenced_by are derived from relationships, not read directly off the column — they’re absent for a column that isn’t part of any relationship, even if it’s marked foreign_key/primary_key in isolation (which validate-spec’s S01 already treats as an error).

Assertion

{
  "expression": "string",         // the `assert` text, as the author wrote it
  "language?": "r",               // absent when written in the data-dict language
  "canonical?": "string",         // the same expression in the data-dict language;
                                  // present exactly when `language` is
  "fidelity?": "divergent",       // how faithfully it was read; absent when exact
  "notes?": ["string", ...],      // absent when `fidelity` is
  "description?": "string",
  "columns": ["string", ...],     // every column (and struct field, dotted) the expression references
  "definitions?": ["string", ...], // every definition of the same table it references
  "translations?": [ Translation ]
}

columns lists the columns the expression reads, in first-appearance order, with a COLUMNS(...) selection expanded to the columns it matches.

expression is what the author wrote, verbatim, in whatever language they wrote it. It is the string a diagnostic quotes and the string to search the dictionary for. When that language isn’t the dictionary’s own, language names it and canonical gives the same rule in the data-dict language, which is the form every other field on this object is about: columns, translations, and the validator all work from the one reading. fidelity and notes grade that reading on the same scale a Translation uses, present under the same rule. A reading is never "guarded" or "unsupported": a reader adds no code of its own, and an expression with no reading doesn’t validate, so it never reaches an export.

Translation

A Translation is the expression rendered as a bare expression in one target language — the same translation translate produces, so a consumer can embed it directly in that language’s pipeline rather than re-parsing expression. For an assertion the rendering is a predicate; for a definition it computes the definition’s value, so a metric’s translation is an aggregate expression, embeddable only where the target aggregates (a SELECT list, a summarise()). A reference to another definition renders as a bare name, as if it were a column — substituting the referenced definition’s own translation (named by definitions) is the consumer’s job. One entry per target the exporter can produce; a target that can’t express the expression still appears, carrying the reason instead of code:

{
  "target": "SQL(duckdb)" | "R(tidyverse)",
  "code?": "string",              // the predicate, absent when the target refused
  "error?": "string",             // why the target refused, absent when it produced code
  "notes?": ["string", ...]       // edge cases where the translation diverges from the
                                  // expression's own semantics; absent when it agrees exactly
}

Definition

A Definition is one of a table’s definitions — a named metric, filter, or derived value — with its expression’s kind, value type, and references resolved:

{
  "name": "string",
  "label?": "string",
  "description?": "string",
  "details?": "string",
  "expression": "string",          // the `expr` text, as the author wrote it
  "language?": "r",                // absent when written in the data-dict language
  "canonical?": "string",          // the same expression in the data-dict language;
                                   // present exactly when `language` is
  "fidelity?": "divergent",        // how faithfully it was read; absent when exact
  "notes?": ["string", ...],       // absent when `fidelity` is
  "kind": "filter" | "metric" | "derived",
  // read off the expression's type and shape, as the spec defines: a boolean
  // row-level expression is a filter, an aggregate or constant a metric,
  // anything else a derived value
  "type?": "string",               // the expression's value type:
                                   // "number" | "string" | "boolean" | "date" | "datetime" | "interval"
  "columns": ["string", ...],      // every column (and struct field, dotted) the expression reads
  "definitions?": ["string", ...], // other definitions of the same table it builds on
  "translations?": [ Translation ]
}

columns and definitions list direct references only, in first-appearance order — a definition that builds on another definition names it under definitions, and the columns that definition reads appear on its own entry.

language, canonical, fidelity and notes carry what an Assertion’s do, under the same presence rules.

Relationship

A Relationship is normalized so cardinality is always read left-to-right as “many-to-one” (a declared one-to-many has each pair’s left/right swapped so left is always the “many” side and right the “one” side; one-to-one is unaffected):

{
  "description?": "string",
  "todo?": "string",
  "cardinality": "one-to-one" | "many-to-one",
  "declared_cardinality": "one-to-one" | "one-to-many" | "many-to-one",
  // the cardinality as written — the orientation the `join` text documents
  "pairs": [ { "left": ColumnRef, "right": ColumnRef } ],
  // ColumnRef = { "table": "string", "column": "string" }
  "join": "string",               // original join text, unnormalized
  "aliases?": [ { "name": "string", "table": "string" } ],
  "conflicts?": ["string", ...]
}

pairs records which column matches which, one pair per join conjunct. Pair tables are real table names — an alias in the join is resolved through aliases, which is preserved so the original join text can still be read. A range join like a.date >= b.start AND a.date <= b.end yields two pairs: a.date ↔︎ b.start and a.date ↔︎ b.end.

Profile

A Profile (export-data only) accompanies one column, carrying the same statistics as describe. Its shape follows the column’s type, so a key that could never apply to that type doesn’t appear at all. (Strictly, the shape follows the data’s type — the two agree whenever the data conforms to the dictionary, which validate-meta can confirm.)

Numeric and temporal columns (number, number(...), date, datetime) summarize on a scale:

{
  "distinct?": { "count": 123, "approximate": false },
  // absent for a continuous (float) column, where per-value equality is
  // misleading — its shape is the histogram
  "missing?": 4,
  "range?": { "min": Scalar, "max": Scalar },   // the observed extremes
  "sample_values?": [Scalar, ...],
  "histogram?": {
    "bins": [
      { "min": 0, "max": 10, "count": 5, "closed": "right" | "both" },
      ...
    ],
    // float values with no place on the number line, counted apart from the
    // bins; each appears only when nonzero
    "nan_count?": 1,
    "negative_infinity_count?": 1,
    "positive_infinity_count?": 1
  }
}

The three non-finite counts are values, not missing data. They are kept out of the bins and the observed extremes because neither has a place on the number line, but they are what IS_NAN and IS_INFINITE test for in an assertion, and an assertion folds them into an aggregate rather than skipping them.

String, boolean, and enum columns summarize by value:

{
  "distinct?": { "count": 123, "approximate": false },
  "missing?": 4,
  "sample_values?": [Scalar, ...],
  "common_values?": {
    "approximate": false,
    "values": [ { "value": Scalar, "count": 42 }, ... ]
  }
}

sample_values are up to 20 representative values, spread along the column’s sorted distinct values rather than drawn from its start, and reported exactly — never rounded, since how much precision a value carries is part of what it tells you. They are omitted for a column that declares its values: that declaration is exhaustive, so it already gives the reader every value the column can hold, in full rather than as a sample.

A list column reports only its containers — { "missing": 4 }, the null-list count — never its elements.

A display: restricted column’s profile carries counts only — missing and distinct — never a value the data held: sample_values, common_values, range, and histogram are always omitted, so consumers can safely embed the export in user-facing output. (The column’s declared examples and range still appear; the spec requires those to be plausible fakes, written by the author rather than read from the data.)

Each histogram bin’s closed says which of its boundary values it includes: every bin is "right" ((min, max]) except the first, which is "both" ([min, max]) so the column minimum has a home; bins are otherwise contiguous.

Nested columns profile as far as the data allows:

• A struct column carries no profile — its fields carry their own instead. A field reached through a list layer (list(struct)) is profiled per element, so its counts are over elements rather than rows.
• A list-typed field inside a struct carries no profile at all (its container nulls aren’t countable below the top level).
• A column whose Parquet type can’t be summarised (uuid, decimal, json, …) gets the list shape — missing alone, when the file’s footer supplies it — or no profile at all.

Scalar

A Scalar is a literal JSON value: a number, string, boolean, or null, following the same rendering range/examples/values already use elsewhere. An infinite range bound (.inf), which JSON can’t spell, renders as null — that end of the range is open. A NaN never appears: it is legal neither as a bound nor as an example (S12), and the profile counts non-finite values separately rather than reporting them as extremes.