Sense 19.2: The Incremental Scenario

Sense 19.2 · Forming · Last touched 2026-08-20

• last_verified: 2026-07-27

Synced from docs/design/sense_19_2_the_incremental_scenario.md in the engine repo — that’s the source; this page is a build-time mirror.

A scenario whose intervention only nudges a signal threshold should not pay the cost of re-reading every CSV. Bronze, Silver, and Gold are bit-identical to production in that case; only what the intervention touched, and what depends on it, must rebuild.

Status: proposed Date: 2026-05-25 Depends on: Sense 19 — The Simulation, Sense 19.1 — The Simulation Prerequisites Author: the owner + Claude (conversation, 2026-05-25)

---

The observation

When Sense 19 — The Simulation shipped Phase 5 on 2026-05-24, the build path for a scenario was a full jin make against the scenario branch. That’s correct, deterministic, and slow.

For our rmc and inspire Phase 5 fixtures the intervention is a single yaml_edit on a signal’s tolerance_pct. Nothing upstream of that signal changes — Bronze stays the same bytes, Silver stays the same bytes, Gold stays the same bytes. Only signal_findings__<signal> and the thesis / verdict / perspective tables that reference it actually need to be rebuilt. For a 60-second rmc make, ~55 seconds of that is wasted compute.

This is the most common shape an intervention will take. Sense 19 Phase 5 named it the “Gold-and-downstream sweet spot”: local YAML edit, fast rebuild, unambiguous interpretation. The architecture should reward authoring interventions at that level by making the rebuild proportional to the scope of the change.

The mistake to avoid (narrative correction from Phase 5)

The scenario KLS is a new sibling file at <pack>_<tenant>_scenario_<id>_kls.duckdb. The working KLS at <pack>_<tenant>_kls.duckdb is never touched by a scenario build. DuckDB’s per-file lock does not cross. Phase 4’s Compare UI was designed around exactly this isolation.

The optimization below preserves that property — it copies the working KLS to the scenario path before mutating anything. The working KLS remains exactly what the user is browsing in JinDesk.

The proposal

For an intervention whose blast radius is bounded (we can identify it mechanically — see below), jin scenario build should:

1. Copy the working KLS at <pack>_<tenant>_kls.duckdb to the scenario path <pack>_<tenant>_scenario_<id>_kls.duckdb. Bronze, Silver, Gold tables come along for free.
2. Recompile the affected YAML against the scenario AFS branch. This is the existing signalcompile.py / thesiscompile.py / etc. step, scoped to the changed artifacts. Output: new SQL files for the modified models.
3. Selective dbt run against the scenario KLS: dbt build --defer --state <prev_manifest> --select state:modified+.
   • --defer: for any model not in the selection, read from the state environment’s already-built tables. Implementations of --defer resolve {{ ref('upstream_model') }} against the deferred catalog — exactly what we need when the upstream bronze/silver/gold tables in the scenario KLS are byte-identical copies.
   • --state <prev_manifest>: a baseline manifest from the working KLS’s last successful build. dbt compares the freshly-parsed manifest against this baseline to determine which models changed.
   • --select state:modified+: the + is “and everything downstream of.” Together: “rebuild every model whose compiled SQL differs from the baseline, plus every model that depends on those.”
4. Restamp scenario metadata in the new KLS:
   • Replace _jinflow_afs_archive with the scenario branch’s AFS blob (not the working branch’s — see below).
   • Update _jinflow_snapshot: branch = scenario/<id>, commit = scenario branch HEAD, dirty = false.
   • Rebake _<tenant>.scenario_metadata (Phase 4 already does this).
   • Rebake _<tenant>.suggestion_registry and _<tenant>.intervention_registry from the scenario branch’s AFS — they may have changed if the intervention touched suggestions/ or interventions/ (rare, but the contract allows it).
5. Done. The Compare UI’s existing scenario-mode picks the new KLS up automatically — same name pattern, same metadata.

The correctness gate

The optimization is sound if and only if the bytes we don’t rebuild genuinely don’t need to be rebuilt. This is a mechanical check, not a judgment call:

• The intervention declares target_file (an AFS-relative path).
• Each target_file maps to a known dbt model node (or to a YAML that compiles to one — for signal/thesis/verdict YAMLs the mapping is 1:1).
• The dbt manifest carries the full model dependency DAG.
• Downstream-of-target = everything dbt’s state:modified+ selector would rebuild given that model as the seed of the change set.

When an intervention targets a model in Bronze, Silver, or Gold, the cascade fans out to most of the pipeline and the savings disappear — the optimization should bail to a full make.

(There is no cross-tenant scope to worry about: a KLS is strictly per-tenant. The model DAG never crosses tenant boundaries, so state:modified+ is contained to the scenario’s own tenant by construction.)

The gate, then: apply the incremental path only when

1. target_file resolves to a leaf or near-leaf instrument (signal / thesis / verdict / perspective / smebit / suggestion / intervention YAML), AND
2. state:modified+ selection does not pull in any Bronze / Silver / Gold model.

Anything outside the gate falls back to a full make. The optimization is opt-in by artifact shape, not by user flag.

Where this fits the upstream-vs-downstream framing

The further upstream an intervention reaches, the fewer bytes the optimization saves, until it saves nothing at all. That’s a feature, not a limitation:

Intervention layer	Layers that copy	Layers that rebuild	Savings
Signal / thesis / verdict / perspective YAML	bronze + silver + gold	signal_findings__X + downstream	~90%
Smebit / suggestion / intervention YAML	everything model-level	registry bakes only	~99% (metadata-only equivalent)
Silver model SQL or YAML	bronze	silver + gold + signals + downstream	~30%
Bronze dispatch macro / source-system change	nothing	full pipeline	0% — bails to full make
Pack config.yml / tenant config.yml change	depends — config touches everything that reads it	depends	conservatively bail to full make
Seed CSV (calibration / reference data)	nothing	full pipeline	0% — bails to full make

The optimization is most generous to the interventions Phase 5 makes easiest to author. That alignment is on purpose.

What this does NOT do

• No new correctness guarantees over what dbt provides. --defer --state state:modified+ is dbt-native machinery, used at scale by dbt-cloud and dozens of teams that ship multi-environment builds. We are not inventing dependency resolution; we are using it.
• No reach across tenants. A scenario lives in one tenant’s AFS branch and produces one tenant’s KLS. Cross-tenant simulation is out of scope (per the original Sense 19 doc) and the per-tenant data architecture enforces it automatically — no special gate required.
• No incremental update of the existing scenario KLS on re-run. Re-running jin scenario build <id> after the user edits the intervention should still copy fresh from the working KLS and rebuild — the scenario branch can have moved arbitrarily. Always re-copy; never patch a previous scenario KLS in place.
• No partial-failure semantics. If dbt build state:modified+ fails midway, the scenario KLS is in a half-state. The build wraps the copy → recompile → dbt step in a temp-path-then-rename so the final scenario KLS file either exists fully-formed or doesn’t exist at all. Same convention jin make already uses for the working KLS.

Implementation deltas from today’s jin scenario build

Today’s verb is a thin wrapper around jin make --branch scenario/<id>, which is a full make. The incremental path needs:

• A new module, e.g. jinflow/scenarios/build_incremental.py, that orchestrates the five steps above.
• A gating function that takes the intervention(s) and the current dbt manifest and returns True/False/why for whether the incremental path is safe.
• A baseline manifest source — the dbt target/manifest.json from the working KLS’s last build needs to be persistable. Two options: (a) bake it into the working KLS as _<tenant>.dbt_manifest_blob next to the AFS archive; (b) keep it on disk in store/state/manifest.json (current dbt convention, fine for local but breaks if the user moves a KLS).
• A scenario branch checkout for the recompile step (Phase 2’s create_scenario already creates the branch; the build step needs to read from it without disturbing the working tree).
• An AFS-archive rebake that grabs the scenario branch’s AFS, not the working branch’s.

jin scenario build should accept a --full flag that bypasses the gate and forces a full make — useful for verifying that the incremental and full paths produce equivalent KLSes (regression gate for the optimization itself).

How this manifests in jinflow

Not yet manifest in the engine. Status: proposed. See the implementation phases / open questions above for the path from this paper to running code.

---

Open questions

1. Where to persist the baseline dbt manifest — KLS blob or disk path. Tradeoff: in-KLS makes the optimization self-contained (the KLS carries everything needed to drive an incremental build from it); on-disk is simpler and matches dbt’s own convention.
2. What happens if the user moves the working KLS (e.g. snapshot tagging) between build and scenario create. The baseline manifest needs to be findable. In-KLS would solve this trivially.
3. AFS archive correctness check. The KLS-copy step bypasses the normal _jinflow_afs_archive bake. We need to verify the restamping step truly replaces (not appends to) the blob. Trivial but easy to get wrong.
4. Does dbt build state:modified+ handle macro changes correctly? A change to a dbt macro that’s used by many models should mark all of those models as modified. Test this against the current dbt-duckdb 1.10.0 manifest format before committing to the gate.
5. Determinism under parallel execution. dbt run schedules models in parallel where the DAG allows. For an incremental scenario where most upstream models are deferred, parallelism is limited but still present. Confirm no race condition between --defer resolution and writes to the scenario KLS.

Relation to Phase 5b

This is not a prerequisite for Phase 5b (JinDesk-triggered build). Phase 5b can ship against today’s full-make jin scenario build and feel slow but correct. The incremental path makes the Phase 5b UX good — sub-3-second feedback from “click What if?” to “open Compare” — instead of “click What if?, walk to coffee machine, come back, open Compare.”

The order to ship matters:

• First: Phase 5b with the existing full-make build. Get the end-to-end loop working in the browser.
• Then: Sense 19.2 (this doc) as a transparent speedup. No UX change; the user experiences “the build got faster.” The gating function ensures correctness is never traded for speed.

Shipping Sense 19.2 before Phase 5b means optimizing a code path that’s only ever invoked from the CLI — measurable benefit but no UX leverage.

Related

• Sense 19 — The Simulation — the parent design, four-phase plan, vocabulary of Suggestion / Intervention / Scenario / Simulation.
• Sense 19.1 — The Simulation Prerequisites — three things that must exist before Sense 19’s UX lands cleanly.
• Sense 36 — The Lineage — .pack-init.yml and AFS provenance. The scenario KLS’s _jinflow_snapshot metadata draws on the same self-describing-snapshot machinery.
• The Operating Modes doc’s “make is a pure function” claim. Sense 19.2 preserves the claim by gating the optimization to provably-equivalent rebuilds.

---

Numerical neighbors: ← Sense 19.1: The Simulation Prerequisites · Sense 20: The Seam — Where Pack and Tenant Meet →