jinflow make

The make command is the build pipeline. It compiles YAML instruments, runs dbt, and produces the KLS.

Pre-make gathers inputs. Make is a pure function.

The pipeline has two phases. pre-make gathers all inputs from the real world (extracts source data, fetches notes and bookmarks from the SIS) and commits them to the AFS. make then reads only from the AFS and writes only to the KLS — no SIS mutation, no external fetches, no network calls.

This split is the security boundary: every build is reproducible from the AFS commit it was built on. See Operating Modes for the full architecture.

Basic Usage

jin make                                  # build current tenant
jin make --tenant millesime.domaine_zufferey            # build a specific tenant
jin make --pack alptrack                  # build all tenants in a pack
jin make --all                            # build all tenants across all packs

Pre-Make

pre-make is the preparation phase. It has two channels:

1. Source data — extracts XLSX → CSV, validates against the extraction contract, writes to raw/
2. User content — fetches notes and bookmarks from the SIS, writes them to afs/notebook/*.yaml and afs/bookmarks/*.yaml

After pre-make, make has everything it needs and runs as a pure function.

jinflow pre-make                              # extract + publish SIS content (default tenant)
jinflow pre-make millesime.domaine_zufferey   # specific tenant
jinflow pre-make --notes-only                 # just publish SIS content, skip extraction
jinflow pre-make --extract-only               # just extract sources, skip SIS

Pre-make commits to git when it’s done (pre-make: sync sources + publish SIS content), which makes every build auditable: the AFS commit history shows exactly what changed between runs.

Where the SIS lives depends on the operating mode. In local mode pre-make reads a file; in cloud modes it calls the Explorer API. The developer doesn’t care.

Data Sync (—sync)

--sync copies CSVs from the Data Landing Zone (DLZ) into the tenant’s raw/ directory before building. Only changed files are copied (SHA-256 comparison against the previous manifest).

jin make --sync                           # sync DLZ → raw → build
jin make --all --sync                     # all tenants, only copy changed CSVs
jin make --tenant millesime.domaine_zufferey --sync     # single tenant with sync

Extraction (—extract)

--extract runs source-system-specific extraction scripts (e.g. Excel → CSV) in the DLZ before syncing. Implies --sync.

jin make --extract                        # extract Excel → CSV → sync → build
jin make --all --extract                  # extract + sync + build all tenants

Extraction is governed by the extraction contract (pipeline.yml at the tenant AFS root), which pins every source file by SHA-256, declares its expected schema, and fails hard on any mismatch. Read the Extraction Guide for the full trust-boundary model, the six source types, and the jin extract --contract inspection command.

Not all source systems have extraction pipelines. Tenants without one skip extraction and proceed to sync.

Operating Modes

make adapts to the operating mode without changing invocation:

• Local — make runs on your machine, reads and writes local files.
• Proxy (P2P2P) — make runs on the data owner’s machine; the proxy serves the resulting KLS via tunnel.
• Cloud (R2) — make runs locally, then jinflow cloud sync uploads the KLS to R2.
• Semi-Cloud — make runs locally; the SIS lives in the cloud and pre-make fetches from it remotely.
• Make-as-a-Service — make runs in the cloud on push or cron; no local invocation.

Clean (—clean)

--clean clears intermediary build artifacts (compiled SQL, enriched CSVs) and forces a full rebuild. Does not touch the AFS or raw data.

jin make --clean                          # clean rebuild
jin make --tenant millesime.domaine_zufferey --clean    # reset single tenant

Snapshots (—snapshot)

--snapshot freezes the KLS after a successful build. The snapshot is immutable — make will refuse to overwrite it.

jin make --snapshot                       # auto-tagged: YYYYMMDD-HHMM
jin make --snapshot post-audit            # named snapshot

Flags

Flag	Effect
--sync	Copy changed CSVs from DLZ → raw/
--prune	Remove files from raw/ that no longer exist in DLZ (with —sync)
--extract	Run extraction scripts, conditional on Excel file changes (implies —sync)
--clean	Clean rebuild, clear intermediaries
--snapshot [tag]	Freeze KLS after build
--all	Build all tenants across all packs
--pack <name>	Build all tenants in one pack
--tenant <id>	Build a specific tenant (pack.tenant)
--continue-on-error	Don’t stop on compiler/dbt errors
--quiet	Suppress dbt output
--skip-enrich	Skip CSV enrichment phase
--skip-validate	Skip CSV schema validation
--metadata-only	Only rebake metadata (logo, branding, about) — skip all build phases

Build Phases

0a  Extract          (--extract, source-system-specific)
0b  Sync from DLZ    (--sync, copies changed CSVs to raw/)
1a  Validate CSVs    (schema check against source system definition)
1b  Enrich CSVs      (taxonomy generation, backfill, normalization)
2   Compile          (YAML → dbt SQL: signals, theses, verdicts, ...)
3   dbt build        (Bronze → Silver → Gold → Signals → Theses → ...)
4   Post-build       (pipeline graph, audit reports, PDF generation)
5   Stamp + package  (metadata, AFS archive, snapshot)

Conditional Behavior

• —extract hashes Excel files against _extract_manifest.json — skips extraction if unchanged
• —sync only copies CSVs whose SHA-256 hash differs from the previous sync manifest
• —prune removes files from raw/ that no longer exist in DLZ (default: warn only)
• No-extraction systems are skipped with an info message (no xslx/ directory → no extract pipeline)
• Metadata-only changes auto-detected: if only tenant.yml, about.yaml, logo, or jinflow.yml changed since the last build, rebakes metadata without running dbt. Also available as --metadata-only
• Incremental builds reuse the existing KLS — dbt skips models that haven’t changed
• Failed builds are detected on retry (missing KLS stamp → automatic fresh start)

Examples

# Daily routine: sync new data, rebuild
jin make --all --sync

# After editing signals/theses: rebuild without re-syncing data
jin make

# Fresh start after schema changes
jin make --clean

# Just update logo/branding (no rebuild)
jin make --metadata-only

# Freeze a good state before making changes
jin make --snapshot before-refactor

# Build only the millesime pack
jin make --pack millesime --sync

# Extract new Excel files, sync, build
jin make --tenant millesime.domaine_zufferey --extract