Sense 61: The Strata

Sense 61 · In bloom · Last touched 2026-07-27

last_verified: 2026-07-27

Synced from docs/design/sense_61_the_strata.md in the engine repo — that’s the source; this page is a build-time mirror.

Status: proposed — sketch / position paper, on the agenda (not specced to build) Date: 2026-07-15 Working name: The Strata — provisional. The name, the Sense-number, and even whether this is a Sense or a plain persistence-architecture note are Mig’s to set. (The tenant store below is called the Reservoir as a placeholder — likewise his to name.) Origin: a discussion about the future of R2 in the persistence model — seeded by Sparky’s report AFS Detached Trunks on a Cloud Filesystem Substrate (etc/Sparky/…v0.9.md, JuiceFS/R2/GitHub) — that turned on a correction from Mig: the value isn’t “don’t depend on the laptop” (cloud deploys handle that) but “we love git, and we don’t want to depend on git.” That reframing, plus Mig’s proposal of a dedicated tenant database for objects independent of the data flow, exposed a deeper thing: jinflow has three kinds of persistence with three different contracts, and today it conflates them. Companion Sense docs: Sense 59 The Double Circle (River/Garden — this sits beneath both) · Sense 16 P2P2P / Operating Modes (where each stratum physically lives varies by mode) · Sense 25 Identity + Sense 46 Roster (the poster-child operational objects — sister-3’s lane; this intersects it and needs her) · Sparky’s report (harvested, not adopted wholesale)

Claim

Persistence should be organized by the character of what is stored — not by the pipeline that produced it, and not by the tool that happens to version it.

jinflow stores three kinds of thing, and they have three genuinely different persistence contracts:

Stratum	What it is	Persistence contract	Today’s (conflated) home
Authored	the framework humans write and review — packs, signals, theses, schemas, docs, the `note:` log	source of record; versioned, diffable, reviewed; reproducible by being the source	git / AFS
Built	the data-flow output — River → Gold → Garden findings + baked registries	derived; reproducible from (authored + source-data); immutable once published; disposable/rebuildable	the KLS
Operational	tenant objects independent of the data flow — identity, principals, roster, config, bookmarks, notebook, invites, stage buffer, build journal	primary state of record; NOT derived, NOT rebuildable; evolves by events, must be backed up	scattered: KLS `_<tenant>` meta schema + SIS + the engine-level sys DB

The confusion in the current design is that the Operational stratum has no home of its own, so it is forced to masquerade as one of the other two:

as Authored — by publishing operational state into git (SIS flush → afs/notebook/*.yaml → a make commit). This is the git-dependence that chafes.
as Built — by baking operational overlays (principal_overlay, bookmarks) into the KLS _<tenant> meta schema, so they get dragged through a full data rebuild they don’t need.

Name the third stratum, give it its own store and its own contract, and both masquerades dissolve. That is the whole idea.

Part 1 — The git boundary: love git, don’t depend on git

We love git for what git is genuinely great at: authored, reviewed, source-controlled content. Packs, signals, theses, schemas, docs, the note: log. Keep all of that in git, forever.

We don’t want git as the mandatory runtime and operational substrate, because the Operational stratum isn’t authored content and forcing it through commits hurts:

SIS-published notes/bookmarks/invites round-trip a commit at make Phase 1d.
make itself commits input phases (today’s audit boundary) — so make depends on git.
Cloud workers pushing git (make-in-cloud, Sense 55/56) means credentials, push access, conflict handling — friction the bucket-sync path already had to route around.

The resolution is Sparky’s origin-split (§13), promoted to the load-bearing idea: trunks are git-backed (Authored) or native (Operational/Built) or hybrid. Git is a source, not the entire runtime (his Principle 6). The honest core:

Operational state is primary, not derived. A roster, an invitation, a bookmark movement, a build-journal row are events and state — not derivations. Trying to make them reproducible-from-a-commit is the source of the friction. Give them a database, back it up like any database, and stop pretending they rebuild.

This is why “don’t depend on git” and “a dedicated tenant database” are the same insight seen from two sides.

Part 2 — The Reservoir: a dedicated tenant database

Mig’s proposal: a per-tenant store holding the tenant objects that are independent of the data flow — no River/Garden material. Identity, principals, roster, config, bookmarks, notebook, invites, the stage buffer, the operational/build log.

It is where SIS and the sys DB converge. SIS is today’s bolt-on for the mutable slice of the Operational stratum; the sys DB is today’s engine-level scrap of the same stratum. Both are the Reservoir, seen partially.

Why it simplifies and speeds things up:

Admin/identity/roster operations stop needing a data build. Managing a roster, accepting an invite, moving a bookmark — none of these should require the AFS→KLS trip. They read/write the Reservoir directly. (The Double Circle freed authoring from re-ingest; the Reservoir frees operations from the rebuild entirely.)
The KLS gets lighter and more honest. It becomes purely the Built stratum — River → Gold → Garden — reproducible from (AFS + source). The operational overlays (principal_overlay, bookmarks) leave the _<tenant> meta schema for the Reservoir. Rebuildable registries (signal_registry, thesis_registry, baked from the AFS) stay in the KLS, because they are Built.
The workspace→publish pattern survives — but internal to the Reservoir, not as a separate SIS file that publishes into git. (Sparky §11/§17: mutable workspace → validate → publish → immutable revision. Still the right shape; now it lives inside the operational store, not across a git boundary.)

The reproducibility bifurcation this forces — stated plainly: today’s implicit model is everything is derived (KLS from AFS+source; operational overlays pushed into git so they too become reproducible-from-commit). Naming the Operational stratum breaks that:

	Built (KLS)	Operational (Reservoir)
Origin	derived	primary / of-record
On loss	rebuild it from AFS + source	restore it from backup — it cannot be rebuilt
Versioning	immutable published revisions	event history / audit log
Git’s role	stamped-by (SHA in `_jinflow_snapshot`), not stored-in	none

This is not a regression from “reproducible from one commit” — it is the recognition that some things were never derivable, and pretending otherwise is what made them awkward.

The KLS containment rule — what the Built stratum may hold — adopted 2026-07-23 (Mig)

The bifurcation above is usually read from the Operational side (“give operational state a home”). Read from the Built side it yields a rule you can apply to a single table — sharper than a philosophy:

The KLS carries only what a rebuild reproduces. The actual data, everything derived from it, the metadata that describes this build, and the reference inputs processing consumes — and nothing else. If an item cannot be regenerated from Authored + Delivered — if it carries state whose authority lives elsewhere — it does not belong in the KLS.

The discriminator is rebuildability, and it is the exact twin of the Operational stratum’s recovery contract (Reservoir P13): Operational state is what you restore because a rebuild can’t reproduce it; therefore the KLS is what a rebuild can reproduce, and only that. One seam, stated from both sides.

The test is one question asked of a table — delete the KLS, run make: does this reappear, identical in meaning, from AFS + sources?

Yes → it is Built; it belongs. (Gold, findings, signal_registry, the AFS-baked role/group tree, _jinflow_snapshot’s build description.)
No → it carries operational state and lives in the Reservoir. (Roster assignments, a note a user wrote, run records, a person’s tour progress.)

Things this rule decides — not just describes:

No Reservoir cache in the KLS. A one-behind copy of operational state, baked “for convenience,” is neither rebuildable nor processing-needed — out. An online cloud tenant reads its live roster from the authority (Clerk), not a stale KLS copy; an offline tenant with no Reservoir is the sealed island (P12) and a degraded roster is the honest result. This retires the KLS principal_registry assignment bake and the clerk_user_id pin.
Operational bakes already in the KLS are debt. run_registry’s KLS bake (already “one-behind + blind to red runs”) and the notes-closure bake (“deliberate redundancy”) are operational state squatting in Built; the rule names them for unwinding.
New artifacts sort on sight. The tour splits cleanly: a pack-authored first-visit tour is rebuildable → KLS; a person’s tour progress, and any owner-composed tour, are not → Reservoir.

Rebuildability is necessary, not sufficient — the second axis is scope — added 2026-07-23 (Mig)

The containment rule says what the KLS may never hold. It does not say everything rebuildable belongs in the KLS. Among the rebuildable things, a second question decides the home:

Build-scoped — computed on, or describing, this build (Gold, findings, signal_registry, which operates on this build’s Gold). → KLS.
Tenant-scoped — stable across every build the tenant produces. → OPS.

The identity tree (roles, groups) is the case that forced this axis. It is rebuildable from AFS — so the containment rule permits it in the KLS — yet it is identical across a tenant’s base KLS, its generations, and its variations: it is of the tenant, not of any build. Baking it into each KLS is redundant and asks “which build owns identity?”, a question that shouldn’t exist. So the tree stays authored (AFS YAML, git-reviewed) but bakes into the OPS, whose scope it matches — tenant-scoped, outlives every KLS (Reservoir P11), and promotable to a future org-OPS (Part 6). The role/group bake therefore moves KLS→OPS; only the signal/thesis/verdict registries stay KLS-baked, because they are build-scoped.

This corrects an earlier draft of this section that offered the tree as a “belongs in the KLS” example — it does not; it is the poster child for rebuildable-but-tenant-scoped → OPS. Once the tree moves, only the assignments were ever operational, and now the definitions join them in the OPS too — the tree by scope, the assignments by character. Kind decides eligibility; scope decides placement.

The OPS therefore holds two disciplines of table: authored-baked (the tree — overwritten from AFS each bake, like the KLS) and operational-merged (roster, notes, runs — restore-not-rebuild, merged on sync). That is Reservoir P17’s overwrite-vs-merge distinction turned inward; each OPS table is marked with its kind.

Part 3 — R2 as the runtime substrate (and JuiceFS, deferred)

For both the Built revisions and the Reservoir, the non-git home is R2 as a content-addressed object store, fronted by a minimal versioning registry — not a POSIX filesystem.

Built stratum → content-addressed immutable KLS revisions in R2 (this is cloud sync grown into a proper revision store; cheap tenant cloning comes from content-addressed dedup, not a filesystem CoW).
Reservoir → an R2-backed operational DB (so identity/roster survive independent of any one machine — the cloud-not-laptop property, applied to operational state).
Registry → a tiny control plane: IDs → objects, movable/fixed bookmarks, lineage. A manifest in R2 or a small DuckDB index. This is what provides identity + the atomic “current” pointer once git isn’t doing it (Sparky §15/§16.2, sized way down).

JuiceFS stays deferred. “Don’t depend on git” needs object operations + a small registry, not a POSIX filesystem + a stateful metadata engine (Redis/TiKV/Postgres). JuiceFS would trade “one contained file” for “a distributed system,” which attacks the very value — simple, contained, one store — that this Sense protects. Containedness and git-independence are orthogonal: R2-objects-plus-registry preserves the single-blob KLS; JuiceFS is what would dissolve it into blocks. Revisit JuiceFS only if a genuine concurrent-POSIX, many-writers-on-one-shared-tree need appears — which jinflow’s coarse, per-tenant, read-mostly, single-make-writer shape does not have today. Trigger to reopen: a fleet of cloud workers needing simultaneous POSIX writes to one shared AFS tree.

Part 4 — The future of the sys database

The engine-level state/system.duckdb (schema_version, instrument_states, build_journal, compilation_log, config) is, in this frame, simply the Operational stratum with no tenant scope and no proper home — already .gitignored, already best-effort, already a DuckDB file that nobody rebuilds.

Direction: it does not survive in its current form. Its tenant-scoped contents (instrument_states, build_journal, compilation_log per tenant) fold into each tenant’s Reservoir, where they belong beside the other operational objects. Whether any genuinely cross-tenant, engine-level sliver remains (a global build ledger?) is open — but the default is: retire the standalone sys DB; its data is Reservoir data.

Part 5 — Where this sits relative to the Double Circle

The Strata sit beneath Sense 59, and they clarify it:

Authored = the source of both circles (River models + Garden instruments, all in the AFS).
Built = the output of both circles (Gold + findings) — the contract-versioned KLS.
Operational = neither River nor Garden. This is exactly Mig’s “no garden/river stuff”: the tenant’s ground, not its water. The Double Circle is about the flow; the Reservoir is what stays put while the flow rebuilds.

So the Double Circle answered “how does make split?”; the Strata answer “what persists, where, and under what contract?” — the layer below.

Part 6 — Above the tenant? (named, two flight levels below)

Once identity, principals, and roster live in a Reservoir rather than baked per-KLS, a question surfaces that the Double Circle already raised from the other direction. Sense 59 reframed the tenant downward — one River, many Gardens. This reframes it upward: is a tenant really the top unit, or is there an organization above it that owns several tenants (several rivers), with people and policy spanning them?

If an org’s people span its tenants, some Operational objects (identity, cross-tenant roster) may want an org-level Reservoir above the tenant ones. Deliberately unhurried — two flight levels below where we are. Named here only so the Reservoir design doesn’t accidentally foreclose it (keep operational identity addressable, not hard-welded to one tenant).

Part 7 — The fourth stratum? (Delivered data) — Mig’s question, 2026-07-21

The Claim lists three strata, but its own Built row reads “reproducible from (authored + source-data).” That equation names source-data as a first-class co-input — yet it gets no stratum row. It rides in as an unnamed input. Mig asked the obvious question: isn’t there a fourth — a “D”, for the delivered data?

Apply the paper’s own test — how do you recover it when it’s gone? — and four distinct answers fall out:

Stratum	Recovery character	Home
Authored	re-author it (or restore from git) — you wrote it	AFS
Built	rebuild it — derived from Delivered + Authored	KLS
Operational	restore from backup — event history, neither rebuildable nor re-authorable	Reservoir
Delivered (source)	re-request from the source system, or restore the immutable pinned copy — externally-owned ground truth, neither rebuildable nor re-authorable	`source/` + the DLZ

Delivered’s recovery story — “ask the source system again” — is shared by none of A, B, O. Externally delivered, immutable-as-delivered, the trust boundary. By the very character-test the strata rest on, it qualifies.

And jinflow already models it — the paper just never named it a stratum. The DLZ (Data Landing Zone), the immutable source/ directory, and the extraction contract (pipeline.yml — SHA-256 pinned, append-only audit, “the configuration IS the contract”) are the platform treating delivered data as a distinct, governed, first-class concern. The stratum exists operationally; Part 0 of this paper simply omitted it — most plausibly because source/ already had a clean home and was never part of the homeless-operational problem the paper set out to solve. Out of scope, not out of existence.

The honest counter. A purist can say Delivered is the boundary, not a store jinflow owns: the strata are jinflow’s own persistence, and source is ground truth from outside that jinflow merely holds a pinned copy of. Fair. But source/ is persisted (immutably, by us), and the reproducibility equation already treats it as a co-input — so the lean here is that it is a genuine fourth stratum, not merely an input.

What adopting it would (and wouldn’t) change. Mostly framing, not code: source/, the DLZ, and the extraction contract already exist with exactly this character — nothing moves, nothing is built. It sharpens the model (the reproducibility equation becomes Built = f(Delivered, Authored) with both inputs named as strata) and it makes the extraction contract’s “trust boundary” language click into place as the Delivered stratum’s persistence contract. The four then read as a clean pipeline: Delivered + Authored → Built, with Operational alongside — D · A · B · O.

For Mig (names + Sense-numbers are yours):

Adopt it as a fourth stratum, or keep it as a named input to Built?
If adopted, the name: “Data” is generic (the KLS and Reservoir are also data); the character is sharper — externally delivered, immutable, ground truth — so Delivered / Received / Sourced may name it better than Data, though D-for-Data is the clean mnemonic.
Does this stay inside Sense 61, or is the Delivered stratum its own Sense?

What we keep / harvest / defer

Keep	Harvest (from Sparky, adapted)	Defer
git for the Authored stratum	origin-split (git-backed vs native vs hybrid)	JuiceFS (+ trigger to reopen)
the contained, self-describing KLS	identity-before-location; publish-creates-immutability; movable/fixed bookmarks; layered consistency	a full trunk/node/bookmark control plane (build the minimal registry only)
snapshots as immutable revisions	R2 as a content-addressed revision store, not just KLS-sync	org-above-tenant
DuckDB-over-httpfs direct reads	a minimal versioning registry as git’s replacement for runtime versioning	`make`’s audit-boundary substitution (pending Mig’s scope answer)

Non-goals (for now)

Not removing git. Git keeps the Authored stratum — we love it there.
Not adopting JuiceFS, a metadata engine, or a full control plane.
Not dissolving the contained KLS into a filesystem of blocks.
Not specced to build. This is a position paper to align the strategy; each slice (Reservoir, R2 revision store, sys-DB retirement) gets its own spec-first pass.

Open questions (these go back to Mig)

A fourth stratum? (Part 7): is Delivered data (source/ + the DLZ) a stratum in its own right, or an unnamed input to Built? If a stratum, what name (Delivered / Received / Sourced / Data), and is it Sense 61 or its own Sense?
Scope of git-independence: the cloud authoring loop only (Vera/evolve/SIS publishing without commits — a contained slice), or also make itself (substitute git’s audit boundary with content-addressed revision immutability — the deeper re-platforming)?
The Reservoir boundary: which objects are “independent of the data flow”? Clear: identity/principals/roster, bookmarks, notebook, invites, stage, build/operational log. Borderline — do narratives, bell, config-resolution belong in the Reservoir (operational) or stay Built (rebuilt from AFS)?
sys DB: retire fully into per-tenant Reservoirs, or keep a thin engine-level cross-tenant ledger?
Reservoir location by mode: R2-backed (survives independent of any machine) even in local/P2P2P modes, or local-with-sync? (Operating-mode-dependent, like the KLS.)
Does the Reservoir replace SIS, with the workspace→publish pattern becoming internal to it? (My lean: yes.)
First slice: I’d start with SIS-published content → Reservoir (R2-backed), no git commit — it proves git-independence on the loop that hurts most and de-risks the cloud authoring loop, needing none of the heavy machinery. Agree?
Coordination: the Operational stratum contains identity/principals/roster — sister-3’s Sense 25/46 territory. This can’t be a solo structural move; she should co-own the Reservoir’s identity portion.

Vocabulary this Sense proposes

The Strata — the three persistence layers, each with its own contract: Authored (git, source-of-record), Built (KLS, derived/rebuildable), Operational (the Reservoir, primary-state-of-record). Proposed.
The Reservoir (placeholder name) — the per-tenant operational store for objects independent of the data flow; where SIS and the sys DB converge. Proposed.
Origin-split — a trunk/artifact is git-backed (authored), native (operational/built), or hybrid; git is a source, not the whole runtime. Adopted from Sparky.

Numerical neighbors: ← Sense 61, Part 2 realized: The Reservoir — the spec · Sense 63 — The Satellites: connected windows →

jinflow is a jazzisnow product

v0.64.7 · built 2026-09-20 19:48 UTC 