Lineage YAML Reference

Field-level specification for Lineage YAML definitions (lineage/*.yaml).

File Convention

• Location: lineage/*.yaml
• entity_id must match the filename stem (e.g. cases.yaml requires entity_id: cases)
• Describes the full Bronze-Silver-Gold pipeline for one entity
• Source system keys in source_files are validated against the pack manifest’s source_systems list
• Validator: python3 scripts/lineagecheck.py
• Compiler: python3 scripts/lineagecompile.py

Top-Level Fields

Field	Type	Required	Description
entity_id	string	Yes	Must match filename stem
version	string	Yes	Semver (e.g. "1.0.0")
created_at	string	No	ISO date
modified_at	string	No	ISO date
display_name	i18n mapping	Yes	Human-readable name (en/de/fr required)
description	i18n mapping	Yes	Entity description (en/de/fr required)
source_type	string	No	source (default) or seed. See Source Types
seed_file	string	Conditional	Required when source_type: seed. Seed CSV filename
source_files	mapping	Yes	See Source Files
bronze	mapping	Yes	See Layer Blocks
silver	mapping	Yes	See Silver Block
gold	mapping	Yes	See Gold Block
consumers	list of strings	No	Signal IDs or model names that consume this entity

All i18n fields are mappings with keys en, de, fr (all three required).

Source Types

Type	Description
source	CSV-based entity from source systems (default)
seed	dbt seed entity (e.g. taxonomy data). Requires seed_file

Source Files

Mapping of source system name to CSV filename. Keys must match the pack manifest’s source_systems list.

For source_type: source:

source_files:
  opale: cases.csv
  sap: aufk.csv
  navision: service_order.csv

For source_type: seed, all values must be null:

source_files:
  opale: null
  sap: null
  navision: null

Layer Blocks

Bronze

Field	Type	Required	Description
description	i18n mapping	Yes	What Bronze ingestion does (en/de/fr required)

Silver Block

Field	Type	Required	Description
description	i18n mapping	Yes	What Silver validation does (en/de/fr required)
validations	list	Yes	Non-empty list of validation rules. See Validations

Validations

Each entry documents one Silver validation rule.

Field	Type	Required	Description
invalid_reason	string	Yes	The invalid_reason value set when this rule fails
description	i18n mapping	Yes	Human-readable explanation (en/de/fr required)

Gold Block

Field	Type	Required	Description
description	i18n mapping	Yes	What Gold filtering/transformation does (en/de/fr required)
seed_rows	list of mappings	No	Inline seed data for source_type: seed entities. Each entry is a column-value mapping

Example — Source Entity

entity_id: carriers
version: "1.0.0"
created_at: "2026-03-17"
modified_at: "2026-03-17"

display_name:
  en: "Carriers"
  de: "Frachtfuehrer"
  fr: "Transporteurs"

description:
  en: "Logistics carriers with name, transport modes, and service regions."
  de: "Logistikspediteure mit Name, Transportmodi und Serviceregionen."
  fr: "Transporteurs logistiques avec nom, modes de transport et regions de service."

source_files:
  sap_tm: frachtfuehrer.csv
  wms_manhattan: carriers.csv

bronze:
  description:
    en: "Structural ingestion from source CSV. Maps source-system fields to canonical schema."
    de: "Strukturelle Erfassung aus Quell-CSV. Bildet Quellsystemfelder auf kanonisches Schema ab."
    fr: "Ingestion structurelle depuis le CSV source. Mappe les champs vers le schema canonique."

silver:
  description:
    en: "Domain validation: carrier_id and carrier_name required. Surrogate key via dbt_utils."
    de: "Domaenenvalidierung: carrier_id und carrier_name erforderlich."
    fr: "Validation metier : carrier_id et carrier_name requis."
  validations:
    - invalid_reason: "missing carrier_id"
      description:
        en: "Every carrier must have a unique identifier"
        de: "Jeder Frachtfuehrer muss eine eindeutige ID haben"
        fr: "Chaque transporteur doit avoir un identifiant unique"
    - invalid_reason: "missing carrier_name"
      description:
        en: "Every carrier must have a name"
        de: "Jeder Frachtfuehrer muss einen Namen haben"
        fr: "Chaque transporteur doit avoir un nom"

gold:
  description:
    en: "Clean view: only rows that passed all Silver validations."
    de: "Bereinigte Ansicht: nur Zeilen, die alle Silver-Validierungen bestanden haben."
    fr: "Vue nettoyee : uniquement les lignes validees."

consumers:
  - probe_carrier_performance

Example — Seed Entity

entity_id: taxonomy_nodes
version: "1.0.0"
created_at: "2026-03-10"
modified_at: "2026-03-10"

display_name:
  en: "Taxonomy Nodes"
  de: "Taxonomie-Knoten"
  fr: "Noeuds de taxonomie"

description:
  en: "Hierarchical classification nodes for cost centers, departments, and material groups."
  de: "Hierarchische Klassifikationsknoten fuer Kostenstellen, Abteilungen und Materialgruppen."
  fr: "Noeuds de classification hierarchiques pour centres de couts, departements et groupes de materiels."

source_type: seed
seed_file: taxonomy_nodes.csv

source_files:
  opale: null
  sap: null
  navision: null

bronze:
  description:
    en: "Direct seed ingestion. No source-system dispatch needed."
    de: "Direkte Seed-Erfassung. Kein Source-System-Dispatch noetig."
    fr: "Ingestion directe du seed. Pas de dispatch source-systeme necessaire."

silver:
  description:
    en: "Validates node_id uniqueness and parent references."
    de: "Validiert node_id-Eindeutigkeit und Elternreferenzen."
    fr: "Valide l'unicite de node_id et les references parentes."
  validations:
    - invalid_reason: "missing node_id"
      description:
        en: "Every taxonomy node must have a unique identifier"
        de: "Jeder Taxonomie-Knoten muss eine eindeutige ID haben"
        fr: "Chaque noeud de taxonomie doit avoir un identifiant unique"
    - invalid_reason: "orphan parent_id"
      description:
        en: "Parent node must exist in the taxonomy"
        de: "Elternknoten muss in der Taxonomie existieren"
        fr: "Le noeud parent doit exister dans la taxonomie"

gold:
  description:
    en: "Recursive CTE adds depth, path, and is_leaf. Only valid nodes."
    de: "Rekursive CTE fuegt Tiefe, Pfad und is_leaf hinzu. Nur gueltige Knoten."
    fr: "CTE recursive ajoute profondeur, chemin et is_leaf. Uniquement les noeuds valides."