Aide-mémoire data engineer

Astuce

Imprimez cette page (Cmd+P) pour une référence de bureau.

Pipeline de build

jinflow make                         # build default tenant
jinflow make millesime.domaine_zufferey   # specific tenant
jinflow make millesime                    # all tenants in pack
jinflow make --all                   # all tenants, all packs
jinflow make --clean                 # drop + rebuild from scratch
jinflow make --sync                  # copy CSVs from DLZ first
jinflow make --extract               # XLSX → CSV → sync → build
jinflow make --snapshot post-audit   # freeze KLS after build

Phases de build

Phase	Ce qui se passe
0a	Extraction XLSX → CSV (si --extract)
0b	Sync DLZ → raw/ (si --sync)
1	Validation + enrichissement CSV
2	Compilation des instruments (Signals, Theses, Verdicts, SMEbits, Reports, Entities, Lineage)
3a	dbt build : Bronze → Silver → Gold
3b	dbt build : Signals → Perspectives → Theses → Verdicts → SMEbits
3c	dbt build : Lineage
3d	dbt build : Reports
4	Pipeline graph, rapports PDF, calibration
5	Estampillage métadonnées, archive AFS, création SIS

Couches Medallion

Couche	Usage	Matérialisation	Principe clé
Bronze	Ingestion structurelle	TABLE	Dispatch système source, ajoute source_file + row_number
Silver	Validation métier	TABLE	is_valid + invalid_reason sur chaque ligne
Gold	Contract de consommation	VIEW	Uniquement is_valid = true, agnostique système source
Platform	Inter-tenant	VIEW	UNION ALL avec tenant_id

Compilation des instruments

# Validate (check YAML against contracts)
python3 scripts/probecheck.py
python3 scripts/hypothesischeck.py
python3 scripts/diagnosischeck.py
python3 scripts/smebitcheck.py
python3 scripts/reportcheck.py

# Compile (YAML → dbt SQL)
python3 scripts/probecompile.py          # --check for dry-run
python3 scripts/hypothesiscompile.py
python3 scripts/diagnosiscompile.py
python3 scripts/smebitcompile.py
python3 scripts/reportcompile.py

Variables d’environnement

Variable	Usage
JINFLOW_LIVE	Override du live root
JINFLOW_TENANT	Override du tenant par défaut
JINFLOW_DB_PATH	Chemin KLS explicite (Explorer)
JINFLOW_AFS_ROOT	Racine AFS (Explorer)
JINFLOW_SYSTEM_DB_PATH	Chemin DB système
ANTHROPIC_API_KEY	Clé API Claude (evolve)