Sense 47: The Log — the system's write-once memory

Sense 47 · In bloom · Last touched 2026-08-20

• last_verified: 2026-06-05

Synced from docs/design/sense_47_the_log.md in the engine repo — that’s the source; this page is a build-time mirror.

Every executed action leaves a row. The row is append-only, attributable, queryable, replayable. The Log is the substrate; the Senses above it consume it.

Status: proposed Author: the owner + Claude (conversation, 2026-05-28)

---

What pulled us in

jinflow has six places that record “what happened.” Each was added under a different pressure, with a different schema, in a different storage technology. None of them are wrong; none of them know they are siblings:

Surface	Storage	What it records
afs/state/extract_log.jsonl	AFS JSONL	Every extractor run (Sense 31 Beat 0)
afs/log/build_log.jsonl	AFS JSONL	Every jin make (Sense 21 Beat 1)
_<tenant>.navigation_journal	SIS DuckDB	Page visits in JinDesk (Beat 2)
state/system.duckdb → build_journal	systemdb	Per-compiler builds
state/system.duckdb → compilation_log	systemdb	Per-artifact compile events
AFS git log	git	Authored YAML changes + branch sync commits

Plus three more designed but not yet built — publish_log.jsonl (Beat 3), notification_journal (Beat 5), reconcile_log.jsonl (Beat 6) — and an aspirational 2026-03 design at authorization_and_journal.md that specified a hash-chained tamper-resistant audit log for authorization decisions, never implemented.

The naming is also inconsistent. Half the surfaces are called log, half journal. The word “journal” appears in jinflow with at least three meanings: filesystem-style buffer (wrong here), runtime audit record (the surfaces above), and editorial chronicle (the human- written docs/journal/ retrospectives — Sense 45’s chronicle axis).

This Sense names the discipline that all the runtime surfaces share, gives it one word — Log — and frees the word journal to mean editorial prose, the way everyday English already uses it.

The two-word split

Word	Meaning in jinflow	Examples
Log	Append-only system event record. Write-once. Machine-written. Per-Beat. One row per executed action.	build_log.jsonl, extract_log.jsonl, navigation_log (renamed), _<tenant>.event_log (baked)
Journal	Editorial chronicle. Curated prose. Human-written. Reads chronologically.	docs/journal/W17.md, monthly retrospectives, hrcentral Genesis posts

Logs are written by code. Journals are written by humans. They share the chronicle axis from Sense 45 (The Loom), but they answer different questions: what did the system do? vs what did we learn?

The word “log” matches what the industry already settled on (Kafka’s “the log”, Git’s git log, Postgres WAL, application logs everywhere). Half of jinflow’s current files already use it. This Sense finishes the alignment.

Properties of a Log

Every Log surface in jinflow shares these properties:

1. Append-only. Rows are added, never updated or deleted. Mutation of an existing row is a bug, not a feature. (Retention is handled by archival — see §Retention below.)
2. Attributable. Every row names who (Sense 25 identity) caused it, even if “who” is system for automated beats.
3. Time-stamped. Server time, ISO 8601 UTC. Client clocks are recorded separately when relevant (forensics, dedupe) but never trusted for ordering.
4. Outcome-tagged. Every row records whether the action succeeded, failed, or was denied. A missing row for a protected action is itself evidence of a bug or bypass.
5. Content-typed. A kind field names the action class (e.g. extract, build, visit, authz). The schema is uniform across all kinds; per-kind details live in a context JSON field.
6. Optionally hash-chained. When tamper resistance matters (authorization decisions, billing-relevant events), each row includes the SHA-256 of the previous row’s serialised form, making after-the-fact mutation detectable. Not all Logs need this — the build_log doesn’t.

A surface that lacks any of properties 1–4 is not a Log; it’s something else (a cache, a counter, a queue).

The tiered storage model

The Log lives in three tiers, each chosen for fit, not theology. jinflow already does this — this Sense names the pattern.

Hot tier — SIS

Used for: high-volume, per-user, ephemeral telemetry — page visits, query exec, JinDesk interactions.

Storage: SIS DuckDB table _<tenant>.event_log (renamed from navigation_journal).

Why SIS: writes are frequent (every page load), per-machine, not yet shared. Git would be absurd at this rate; DuckDB takes millions of rows comfortably.

Lifecycle: dedupes within a short window (current navigation pattern: 3s). Published to AFS by Beat 3 on cadence (current jin sis publish, future periodic).

Durable tier — AFS

Used for: authorial, cross-machine, audit-grade events — extracts, builds, publishes, authorization decisions, reconcile events, identity changes.

Storage: JSONL files under afs/log/*.jsonl and afs/state/*.jsonl, git-tracked. Each file’s name is the Log kind (build_log.jsonl, extract_log.jsonl, publish_log.jsonl, authz_log.jsonl, …).

Why AFS: git’s content-addressable commits give true immutability for free. Cross-machine sync uses the same plumbing as everything else in AFS. Hash-chaining is optional — git already provides it at the commit level.

Lifecycle: appended on every action that the kind covers. jin make commits the new entries to the AFS git repo at the end of its run, so the AFS history is a verifiable record of every build.

Baked tier — KLS

Used for: read-only consumer view — the Pulse page, the Bell, analytics queries that want to JOIN log entries with signal findings.

Storage: _<tenant>.event_log table (and per-kind sibling views) baked by jin make from the AFS JSONL files.

Why KLS: queries need SQL, not file IO. Baking happens once per build; the table is read-only between builds (matching KLS’s general contract).

Lifecycle: rebuilt from scratch on every jin make that runs the AFS-mirror bakes. The trigger is engine-version mismatch — not AFS drift — between the running engine and the per-tenant _<tenant>.metadata_bake_stamp. A jin make cycle that produces no data changes AND uses the same engine version that last stamped the KLS is a true no-op (no pulse re-bake). When the engine ships a new release, the first make per tenant — even an otherwise idle cycle — refreshes the pulse tables once and writes the new stamp. The AFS files remain the source of truth; KLS is the indexed mirror. See idempotent_make.md § Engine-Version Bake Stamp for the full pattern.

Unified schema

Every Log entry, regardless of tier or kind, has the same shape:

event_log (
    event_id        TEXT PRIMARY KEY,     -- ULID, time-ordered, globally unique
    timestamp       TEXT NOT NULL,        -- ISO 8601 UTC, server time
    kind            TEXT NOT NULL,        -- 'extract' | 'build' | 'publish' | 'visit'
                                          -- | 'authz' | 'notify' | 'reconcile' | ...
    tenant_id       TEXT NOT NULL,        -- which tenant this concerns
    actor_id        TEXT NOT NULL,        -- Sense 25 identity, or 'system'
    capability      TEXT,                 -- WHAT (Sense 25), nullable for system events
    resource        TEXT,                 -- target ('signal:revenue_leakage', 'note:nb_rmc_x')
    outcome         TEXT NOT NULL,        -- 'success' | 'fail' | 'denied' | 'noop'
    context         TEXT,                 -- JSON: kind-specific details
    prev_hash       TEXT,                 -- SHA-256 of previous row (nullable: not all kinds chain)
    -- forensic fields (optional)
    source          TEXT,                 -- 'server' | 'client'
    client_timestamp TEXT,                -- client wall-clock when relevant
    device_id       TEXT,
    ip_address      TEXT
);

A handful of conventions:

• Per-kind details go in context JSON, not in extra columns. This is how event sourcing systems stay extensible — adding a new kind doesn’t migrate the schema.
• actor_id is mandatory. 'system' is a legitimate value (e.g. the cron that fires a periodic build), but the field is never null. This is how we keep the “attributable” property unconditional.
• prev_hash is per-chain. Logs that need tamper resistance (authz_log, billing-relevant) maintain their own chain. Logs that don’t (build_log — git already provides it) leave prev_hash null.

What consumes the Log

The Log is substrate. Several Senses ride on top:

Sense	How it consumes the Log
21 — Heartbeat	Every Beat writes an entry. The Pulse page renders the recent rows (/system/pulse reads the Log directly).
25 — Identity & Passes	Every capability check writes an authz row ((actor, capability, resource, outcome)). Hash-chained for compliance.
33 — The Bell	Polls the Log tail for entries whose kind + outcome map to a notification class. Surfaces them in the bell-icon UI.
19 — Progress	Suggestions, interventions, scenarios all emit kind: evolution entries when their state changes.
45 — The Loom	The Log is the chronicle axis. Renders Log entries as a temporal stream (warp); the Sense’s curation axis (weft) groups them by editorial intent.

Each consumer reads — never writes — the Log via the shape above. Writes go through the per-kind ingestion path (file append for AFS, DB insert for SIS).

Retention

Logs grow. Retention answers two questions: when do we move old rows out of the hot path? and when, if ever, do we delete them?

Hot tier (SIS):

• event_log rows older than 90 days are eligible for prune (current jin sis prune extends to this; today it only prunes navigation_journal).
• Pruned rows are not lost — they were already published to the durable tier by Beat 3 before becoming eligible.

Durable tier (AFS):

• Never deleted. Append-only is real. AFS JSONL files grow forever; git history grows forever.
• Annual rollups produce compressed afs/log/archive/<year>/*.jsonl.gz archives. The hot files (build_log.jsonl, extract_log.jsonl) are truncated after archival but the archive is git-tracked.
• The archive policy is configurable per-Log-kind. Some kinds (authz, billing-relevant) keep their hot file forever; others (extract — high volume in active extraction) archive yearly.

Baked tier (KLS):

• Always full mirror of the durable tier at jin make time. No retention question — it’s a snapshot.

The River reads the Log

Sense 21’s Pulse page (renamed to the River on 2026-06-07 as part of Sense 42’s “Landscape” rewrite — see sense_42_the_landscape.md) at /[tenant]/river (formerly /system/pulse, shipped 2026-05-27) is the first consumer of the Log substrate. Today it reads two surfaces directly:

• afs/state/extract_log.jsonl (Beat 0 entries)
• afs/log/build_log.jsonl (Beat 1 entries)

It does not yet read the SIS hot tier (no per-user telemetry on the River yet), nor the KLS baked tier (no SQL JOINs yet). Both are extensions that this Sense unblocks.

Technical names stay: the underlying baked table is _<tenant>.pulse_event_log, the bake function is bake_pulse_data, the API endpoint is /api/pulse/latest-build. “Pulse” is the technical name for the heartbeat substrate; “River” is the navigational name for the surface that visualises it.

Relationship to the 2026-03 design

docs/design/authorization_and_journal.md (aspirational, last verified 2026-03-21) defined a capability-bound, hash-chained, tamper-resistant journal table for authorization decisions. It was written before the Senses framework and before Sense 25 (Identity & Passes), so it owned both the authorization model (now Sense 25’s territory) and the journal substrate (now this Sense). The schema and hash-chain design from §5 of that document carry forward into this Sense — they were correct, they just needed to be split from the authz-model that wrapped them.

The 2026-03 doc remains useful as prior art and as the source for Sense 25’s audit-grade authorization detail. The schema in §5.2 of that doc is the ancestor of the unified schema above; the only real changes are:

• journal → event_log (the rename this Sense is for)
• user_id → actor_id (Sense 25 vocabulary)
• Added kind (the doc assumed authz-only; this Sense is general)

Implementation Roadmap

Phase 0 — This paper

Name the discipline. Make the existing six surfaces visible as siblings. Free the word “journal” for editorial use.

Phase 1 — Unify the storage paths

Scope: Mode 1 (local), single tenant.

Deliverables:

• New AFS Log files: afs/log/publish_log.jsonl (existing path), afs/log/authz_log.jsonl (Sense 25), afs/log/reconcile_log.jsonl (future Beat 6) — appended by their respective code paths using one shared appendLog(kind, entry) helper in jinflow/cli/commands/log.py (Python) and $lib/server/log.ts (TypeScript).
• Rename SIS table navigation_journal → event_log (or navigation_log to keep granularity per-kind — TBD; see Open Questions).
• Pulse page extended to read whichever Log files exist, not just the two it currently reads.

Exit criteria: every Beat in Sense 21 has a corresponding Log file; the appendLog() helper is the only writer; old surfaces that previously wrote ad-hoc still work (legacy reads pass through).

Landed so far (2026-06-05):

• jinflow/log.py — append_log(), discover_log_kinds(), read_recent() (15 unit tests).
• _bake_pulse_data() in baking.py bakes _<tenant>.pulse_event_log
  • _<tenant>.pulse_pipeline_yml so Pulse works through the cloud proxy (Sense 16) with no AFS mount.
• Auto-discovery in the bake — the hardcoded [("build", …), ("extract", …)] list is gone; discover_log_kinds() drives the loop, so any new <kind>_log.jsonl shows up in pulse_event_log with zero code changes (4 unit tests).
• First real append_log() caller — _afs_git_commit (Sense 38) now writes an afs_commit row on every commit attempt: success or fail, Scribe identity in actor_id, commit SHA in context. Two end-to-end tests verify the round-trip.
• Pulse bake escapes the engine-version idempotency gate (2026-06-08). The build_log and afs_commit_log JSONLs grow on every cycle even when dbt skips; the engine-version stamp alone left pulse_event_log stale on those runs. _bake_pulse_data now runs unconditionally when the KLS exists — it’s a cheap DROP+CREATE of two small tables, no measurable cycle-time cost, and pulse now reflects the latest commits on the next make.

Still in Phase 1 scope:

• TypeScript appendLog() helper in $lib/server/log.ts.
• Rename SIS navigation_journal → event_log (or navigation_log).
• Migrate _append_build_log + extraction’s two writers onto the shared helper (existing JSONL paths preserved — the helper already writes there).
• afs/log/publish_log.jsonl, afs/log/authz_log.jsonl callers (once Sense 25 capability checks and the publish-back beat are ready to emit).

Phase 2 — Unified KLS bake

Scope: All tenants.

Deliverables:

• jin make reads every afs/log/*.jsonl and afs/state/*.jsonl at bake time, unions them into a single _<tenant>.event_log table with the unified schema above.
• Sibling per-kind views (build_log, extract_log, …) provided for backward compatibility with the Pulse page and other readers.
• Index on (timestamp DESC) and (kind, timestamp DESC) for the common query patterns.

Exit criteria: SELECT kind, count(*) FROM _<tenant>.event_log GROUP BY kind returns one row per Log kind with a non-zero count for any tenant that has ever been touched.

Phase 3 — Hash chains where they matter

Scope: Tenants that need tamper resistance (compliance).

Deliverables:

• The appendLog() helper accepts a chain: boolean flag per call.
• When chain=true, the helper computes prev_hash from the previous row in that kind’s chain and writes it.
• jin log verify --kind authz walks the chain and reports the first broken link, if any.
• Periodic checkpoints (daily) export the chain head to an external append-only sink (R2 with Object-Lock, or a separate git repo).

Exit criteria: the authz_log can be fully verified end-to-end on demand; tampering with any row breaks the chain detectably.

Phase 4 — Archival rollups

Scope: Tenants whose hot Log files exceed a threshold (TBD; ~10 MB?).

Deliverables:

• jin log archive --year 2026 rolls up the year’s entries into afs/log/archive/2026/*.jsonl.gz, truncates the hot file, commits.
• Per-kind retention policy in afs/jinflow.yml:

  log:
    archive_after_days: { extract: 365, build: 365, authz: never, navigation: 90 }

Exit criteria: a long-running tenant can prune hot files without losing history; the archive is queryable from the same event_log view.

---

Open Questions

1. Per-kind file vs single file in AFS. Today the convention is one file per kind (build_log.jsonl, extract_log.jsonl). The unified KLS table merges them. Should we keep the per-kind file layout (simpler diffs, smaller files), or move to a single afs/log/event_log.jsonl? Lean: keep per-kind files, merge only at KLS-bake time.

2. navigation_log vs event_log as the SIS table name. Today it’s _<tenant>.navigation_journal. Phase 1 renames it. To navigation_log (per-kind, parallel to AFS files) or to a single event_log (unified at SIS level)? Lean: unified event_log in SIS — the SIS tier is high-volume, one table simplifies the indexing.

3. Hash chain per kind, or one global chain per tenant? The 2026-03 design was “one chain per tenant.” Per-kind chains let readers verify just the kind they care about, but mean N chains to manage. Lean: per-kind, opt-in. Most kinds don’t need it.

4. Who triggers Beat 3 (SIS→AFS publish) for the Log? Today it’s on-demand (jin sis publish). Once the Log lives in SIS, the publish has to happen reliably or rows accumulate in SIS forever. Lean: post-pre-make hook fires the publish automatically.

5. PII in context. Some Logs may carry PII in the context field (e.g. data:query could contain SQL with patient IDs). The 2026-03 design says “Never log the SQL itself” for that capability. Do we hash query bodies, store them encrypted, or omit them entirely? Lean: per-capability audit_fields allowlist; anything not in the list is dropped at write time. The 2026-03 doc has the canonical list.

6. Cross-tenant Log entries. Some operations span tenants (engine-level identity changes, super-admin actions). Where do they go? Lean: a sibling afs/log/engine_log.jsonl at the engine level, outside any tenant AFS. Sense 24 (Steward) territory.

7. Retention conflict with append-only. §Retention says AFS archives are “never deleted” but archival truncation of the hot file is a deletion (the archive still has the rows). Does that violate the append-only property? Lean: no, because the row exists in the archive; the hot file is just an index window. But this needs to be explicit in the contract.

---

Connections

• Sense 21 (Heartbeat) — every Beat writes a Log entry. This Sense is the substrate Heartbeat’s per-beat journals collapse into.
• Sense 25 (Identity & Passes) — every authorization decision is a Log row. The 2026-03 audit design lives on inside Sense 25, rebased onto this Sense’s substrate.
• Sense 33 (The Bell) — reads the Log tail for notify-worthy rows.
• Sense 19 (Progress) — suggestion/intervention/scenario state changes are Log entries.
• Sense 45 (The Loom) — the Log is the canonical chronicle axis; the Loom’s two-axis pattern is how you navigate one.
• Sense 24 (Steward) — engine-level Log entries (operations the Steward performs) live outside any tenant AFS.
• Per-Artifact Engagement (Phase 2) — extends this Sense with two new kind values (compile, fire) that record what happened to each signal / thesis / verdict individually, not just the run as a whole. Sense 42 Pass 7’s engaged-into-Understanding gate moves from a build-time approximation to a Log-driven snapshot.
• extractor_discipline.md — Beat 0’s append-only contract is the same shape as this Sense; extractor discipline is the vocabulary for one Log kind.
• docs/design/authorization_and_journal.md — the prior-art ancestor. Its §5 schema becomes this Sense’s §Unified schema, with Sense 25 inheriting the authorization-model parts.

---

The Elevator Pitch

jinflow already writes six different “what happened” surfaces, each with a different schema, in a different storage technology, called a different word. The Log is the discipline that unifies them.

Three tiers — hot SIS for telemetry, durable AFS for audit, baked KLS for query. One uniform schema. Append-only is real. Every Beat in Sense 21 writes a row; every Sense above reads the rows it cares about.

The word journal is freed for what it actually means in English — a writer’s chronicle. Logs are what code writes; journals are what humans write. The two complement each other on Sense 45’s chronicle axis but they’re not the same thing.

---

Numerical neighbors: ← Sense 46: The Roster — Clerk authors, AFS materialises · Sense 48: The Stage — dev/prod plane separation →