Design

Inspirations

data-dict draws inspiration from three lineages of work: data dictionaries, semantic models and data contracts.

Data dictionaries

Data dictionaries come from the sciences, particularly the social sciences, where documenting datasets for reuse and secondary analysis has a long history. Survey research gave us standards like DDI, and fields with large shared datasets, like neuroimaging’s BIDS, developed their own conventions. More recently, the Frictionless and Data Package standards brought the same ideas to machine-readable tabular data.

• Data management in large-scale education research
• Frictionless data
• Data Package Standard
• Brain Imaging Data Structure
• Data Documentation Initiative

Semantic models

Semantic models come from the data warehousing and business intelligence community. They describe data in business terms — dimensions, metrics, and the joins between tables — so that analysts and BI tools can write queries that return consistent answers, without needing to know the underlying schema.

• Hex’s semantic modelling
• Snowflake’s semantic views

It’s worth noting that while semantic models influenced the design of data-dict, it is not a semantic model. It doesn’t model dimensions or metrics, because that distinction reflects intended use, not the data itself.

Additionally, while terminology is still evolving, the “semantic” in semantic models is typically interpreted narrowly, focussing on structural semantics — what’s needed for queries to return consistent values — rather than what the data actually means.

Data contracts

Data contracts come from the data engineering community, where pipelines move data between teams. A contract is an agreement between producers and consumers about the shape and quality of a dataset, enforced automatically so that breaking changes are caught in CI rather than in downstream dashboards.

• Soda’s contract language
• dbt tests