Sense 56: The Run
Sense 56 · Steady · Last touched 2026-08-20
- last_verified: 2026-07-27
Synced from
docs/design/sense_56_the_run.mdin the engine repo — that’s the source; this page is a build-time mirror.
Status: proposed — Phase 1 (Run identity + manifest) shipped 2026-07-12
Date: 2026-07-11
Origin: the first fly machines run — an execution we could observe but not reference. There was nothing named to point at, query, cancel, or later analyse.
Companion Sense docs: Sense 55 The Broadcast (what a Run emits) · Sense 42 The Landscape (river-processing category)
Related concept: ProcessCore’s Process Definitions as First-Class Citizens — same north star, different mechanics
Every execution of a jinflow Verb is a Run. A Run is a first-class noun in jinflow — persistent, identified, queryable, referenceable. It has a lifecycle, produces evidence, and outlives the process that emitted it.
Runs are not commands that fire and vanish. They are named things you can point at long after the process has exited.
The vocabulary
Section titled “The vocabulary”- Verb — the abstract kind of operation.
make,ship,snapshot,backfill-clerk-metadata,sync-principals,evolve,explore, and so on. Every jinflow CLI command IS a Verb. Verbs have names, implementations, and versions (they live in the engine code). - Run — a specific execution of a Verb. Has identity, lifecycle, state, produces artefacts.
Every Run has a Verb. A Verb has many Runs. In OO terms: Verb is closer to a class name; a Run is closer to an object — but the analogy is imperfect (see Why not “Definition/Instance” below).
The story that made this Sense necessary
Section titled “The story that made this Sense necessary”At 17:15 UTC on 2026-07-11, we fired jin make riverflow.delta from a laptop against a Fly Machine in Frankfurt. The Machine pulled AFS + source from R2, ran make, pushed the KLS. We watched via fly logs.
Then we tried to answer three simple questions:
-
Which execution was that? We didn’t have a name for it. “The 17:15 one” only worked while we remembered it. Two Runs later, we couldn’t tell them apart.
-
Can we cancel it? The
fly machine destroyroute worked but was crude — it killed the container, not the Verb it was running. If we had wanted to pause it, we couldn’t. -
Can we query it later? “Show me all the makes of delta in the last 24 hours” was impossible. The events had scrolled off
fly logs; the KLS was overwritten by the next Run; there was no persistent handle to the specific execution.
The failure was named. We had no first-class citizen for the execution itself. Every jinflow noun was accounted for — signals, theses, verdicts, tenants, KLSes, snapshots, principals, passes — but the VERBS themselves, and the acts of running them, were second-class. Ephemeral. Un-referenceable.
This Sense fixes that.
What a Run is
Section titled “What a Run is”A Run is a persistent record of one execution of one Verb against one target. It exists in three places over its life:
- Live (during execution): a directory on disk carrying its identity, DAG, event stream, and control signal file.
- Recorded (post-completion): a row in the tenant’s
_<tenant>.run_registrytable inside the KLS, plus its events.jsonl archived in the store bucket. - Aggregated (jinCommander, when it exists): a row in the fleet-wide registry that spans tenants, verbs, environments.
The shape
Section titled “The shape”A Run’s identity and state are captured by the following schema. This is the shape carried in every persistent tier:
run_id: run_20260711T171531_riverflow.delta_a3f
verb: name: make # which Verb version: 0.56.8 # engine version at execution time
target: tenant: riverflow.delta # what the Verb operated against args: # exact CLI arguments (for reproducibility) - --afs-bucket=s3://riverflow-delta-afs/ - --source-bucket=s3://riverflow-delta-source/ - --build-bucket=s3://riverflow-delta-build/ - --kls-bucket=s3://riverflow-delta-store/
trigger: # who/what caused this Run kind: cli | cron | webhook | signal | jincommander | evolve actor: mig | schedule_id | signal_id | ... # for cron: schedule_id + fire_time # for signal: originating_run_id + signal_name # for webhook: source_url
lifecycle: requested_at: 2026-07-11T17:15:29Z # user or trigger asked for the Run started_at: 2026-07-11T17:15:31Z # execution actually began finished_at: 2026-07-11T17:20:12Z status: success # queued | running | paused | success | failed | cancelled cancelled_by: null # actor id if cancelled cancelled_reason: null
worker: # where it ran kind: local | fly_machine | docker id: 185432ef979168 # Fly machine ID, local hostname, etc. region: fra # for cloud runs
inputs: # what it read afs_commit: 4c658b3 # git SHA of the AFS at execution time afs_dirty: false # was the AFS dirty at execution time source_hash: sha256-9f8a... # composite hash of source folder build_incremental: false # did we restore build/ from previous Run? previous_run_id: run_20260711T170000_... # if incremental, which Run's state was restored
outputs: # what it produced - kind: kls uri: s3://riverflow-delta-store/riverflow_delta_kls.duckdb sha256: sha256-a3f... size_bytes: 153800000 - kind: snapshot uri: s3://riverflow-delta-store/20260711-1720_kls.duckdb
broadcast: # links to the Sense 55 event stream events_uri: s3://riverflow-delta-store/runs/run_.../events.jsonl dag_uri: s3://riverflow-delta-store/runs/run_.../dag.json # final DAG snapshot
parent_run_id: null # for sub-Runs (e.g. jin ship spawning jin make)child_run_ids: [] # if this Run spawned sub-RunsThis is what a Run looks like written down. Not everything is known at every stage — lifecycle.finished_at and outputs only appear post-execution — but the schema is unified across live, recorded, and aggregated tiers.
Lifecycle
Section titled “Lifecycle”A Run passes through a small set of states:
┌──────────┐ trigger ┌──────────┐ Verb starts ┌──────────┐ │ none │ ───────────► │ queued │ ──────────────► │ running │ └──────────┘ └──────────┘ └────┬─────┘ │ pause signal │ ┌────────────────────────────┐ │ ▼ │ │ ┌──────────┐ resume signal │ │ │ paused │ ─────────────────────┘ │ └──────────┘ │ │ ┌────────────────────────────────────┤ │ │ ▼ ▼ ┌────────────┐ ┌────────────┐ │ cancelled │ │ success │ └────────────┘ │ failed │ └────────────┘- queued — a trigger has fired; the Run exists as an intent; the Verb has not yet started
- running — the Verb is executing
- paused — a pause signal was observed; execution is on hold at a node boundary
- success / failed / cancelled — terminal states; the Run is done
Every state transition emits an event on the Broadcast (Sense 55). The run.finished event is always the last event.
Data-first: the DAG is emergent
Section titled “Data-first: the DAG is emergent”This is the philosophical claim that separates Runs from what workflow-first tools like BPMN call an “instance.”
A Run’s shape is not dictated by its Verb’s code alone. It is the code walking the data. The number of dbt models depends on the tenant’s entities. The number of signal compilations depends on the AFS. The number of extractors that run depends on the pipeline.yml. The exact shape of a jin make riverflow.delta execution today is different from a jin make riverflow.delta execution next week — even with the same Verb, same code version, same tenant — because the AFS or source may have changed.
The DAG emerges from execution reading data. It is broadcast in two forms:
- A coarse skeleton at Run start (the phases that will always exist for this Verb)
- Expansion events as execution reveals structure (dbt’s model list, compile scripts’ children, etc.)
By the time the Run reaches run.finished, the full DAG is known and can be persisted alongside the events. Before then, it is inherently partial.
This is consistent with the rest of jinflow: signals are data-driven, theses are data-driven, verdicts are data-driven. Runs are data-driven too. They are not instances of a rigid definition; they are executions whose shape emerges from the data they read.
The trigger axis
Section titled “The trigger axis”A Run does not necessarily begin because a user typed a command. Triggers are first-class metadata on every Run:
| Trigger kind | Meaning | Example |
|---|---|---|
cli | A user invoked the Verb from a shell or JinDesk CLI bottombar | actor: mig |
cron | A scheduled trigger fired | actor: schedule_nightly_delta |
webhook | An external system posted to jinCommander | actor: cloudflare_bucket_event_... |
signal | A Sense 14 signal changed severity and had an associated action | actor: signal_stale_material_fired |
jincommander | A fleet-management operation | actor: rerun_last_failed_batch |
evolve | The evolve panel asked to run a Verb on the user’s behalf | actor: mig via evolve |
The trigger tells you WHY this Run exists. Post-mortem analytics rely on it. jinCommander’s dispatch relies on it. The Bell’s routing relies on it.
Persistence
Section titled “Persistence”Three tiers, corresponding to three lifespans:
Tier 1: Live (during execution)
Section titled “Tier 1: Live (during execution)”A directory on the runner’s filesystem:
<live>/<pack>/<tenant>/runs/<run_id>/ manifest.json # the Run schema above, current state events.jsonl # the Broadcast (Sense 55) dag.json # current known DAG signal.json # control channel (created by senders of cancel/pause) build/ # intermediary state (mirrored to build bucket if configured)The manifest is updated on every state transition. When the Run reaches a terminal state, the manifest is finalised and the directory can be uploaded.
Tier 2: Recorded (in the KLS)
Section titled “Tier 2: Recorded (in the KLS)”Every make bakes a _<tenant>.run_registry table containing every Run that has produced or contributed to this KLS. Schema mirrors the manifest above, one row per Run.
The events.jsonl and dag.json are archived to the store bucket at:
s3://<pack>-<tenant>-store/runs/<run_id>/events.jsonls3://<pack>-<tenant>-store/runs/<run_id>/dag.jsons3://<pack>-<tenant>-store/runs/<run_id>/manifest.jsonJinDesk can query the registry table for “recent Runs of this tenant” without needing to hit the bucket. When the user drills into a specific Run for detailed event view, the bucket path is fetched.
Tier 3: Aggregated (jinCommander, when it exists)
Section titled “Tier 3: Aggregated (jinCommander, when it exists)”jinCommander maintains its own DuckDB with a fleet-wide runs table aggregating Run records from every worker. This lets you query across tenants, verbs, environments, and time:
SELECT verb, tenant, status, duration_msFROM runsWHERE requested_at >= now() - INTERVAL 24 HOUR AND status = 'failed'ORDER BY requested_at DESC;jinCommander is future work. The Run schema is designed so it slots in without change when it arrives.
Relationships to other jinflow nouns
Section titled “Relationships to other jinflow nouns”A Run is a first-class citizen among jinflow’s other nouns. Relationships:
| From | To | Relationship |
|---|---|---|
| Run | Verb | verb.name — every Run is of exactly one Verb |
| Run | Tenant | target.tenant — Runs are usually scoped to a tenant (fleet ops may span) |
| Run | KLS | outputs[].kind: kls — a Run may produce a KLS |
| Run | Snapshot | outputs[].kind: snapshot — a Run may produce a snapshot |
| Run | Signal | trigger.kind: signal, actor: signal_id — a signal firing may trigger a Run |
| Run | Observation | Observations may reference a Run via observed_during: run_id (Sense 15) |
| Run | Bell | The Bell fires on run.finished (Sense 33) |
| Run | Run | parent_run_id, child_run_ids — Runs can spawn sub-Runs (e.g. jin ship spawns jin make) |
| Run | Pass | The mila pass sees all Runs; a tenant admin’s Pass sees their tenant’s Runs (Sense 25) |
Why not “Definition/Instance”
Section titled “Why not “Definition/Instance””An earlier draft of this Sense used the vocabulary of “Definition/Instance” from ProcessCore. It was wrong for jinflow.
In BPMN and workflow-engine thinking, a Definition is a stable, versioned schema of nodes and transitions — a musical score. An Instance is a specific play-through of that score. The score dictates the shape; the Instance follows it. This is workflow-first thinking.
Jinflow is data-first. The shape of any given Run is not dictated by its Verb’s code — it emerges from the code walking the actual data. Two Runs of the same Verb against the same tenant, days apart, may have wildly different DAGs. Not because the Verb changed, but because the data did.
Adopting Definition/Instance would have said: “the score is stable, the performance is not.” That is not what happens in jinflow. The CODE is stable (the Verb implementation lives at an engine version), but the shape of any Run is data-first, not schema-first.
So we use Verb for the abstract action (“the class of things called make”) and Run for the concrete emergent execution. Verb is not a Definition. Run is not an Instance. The relationship is looser and more honest.
What this Sense does not do
Section titled “What this Sense does not do”- Not a workflow orchestrator. This Sense names one execution and how it persists. Chaining, scheduling, and cross-Run coordination live in jinCommander.
- Not a distributed transaction system. A Run is atomic-ish (it succeeds or fails as a whole) but does not carry ACID guarantees across sub-Runs. If a
jin shipsucceeds and its childjin makefails, the parent Run is what the user sees; the child’s failure is its own record. - Not process definition. Verbs are defined in engine code, not in YAML. This Sense does not propose a DSL for declaring new Verbs — that’s a much bigger conversation adjacent to Sparky’s ProcessCore vision, deliberately deferred.
- Not a queueing system. Runs can queue (state =
queued), but jinflow itself does not maintain queues. jinCommander might, in the future. For now,queuedmeans “requested but not yet started”; no ordering guarantee beyond that.
Implementation phases
Section titled “Implementation phases”-
Run identity + manifest. ✅ Shipped 2026-07-12. Every
jin makeinvocation opens a<live>/.../runs/<run_id>/manifest.jsonat start and updates it on every state transition. Therun_idis derivable (timestamp + tenant + short hash of args). Lives in thejinflow.runmodule (Run,mint_run_id,detect_worker); the manifest records verb/version, target/args, trigger, lifecycle, worker (Fly-aware), inputs, and outputs (the produced KLS/snapshot). Best-effort — a Run that cannot persist degrades to an in-memory record and never aborts make. (Bakingrun_idinto every existing bake step is still open — see Phase 3.) -
Event stream (Sense 55). Manifest updates go hand-in-hand with Broadcast events. Adding one to
jin makeinherently adds the other. -
KLS-baked run_registry table. A new bake step at end-of-make writes the Run’s manifest as a row in
_<tenant>.run_registry. JinDesk gains a “Recent Runs” page for each tenant. -
Bucket archival. At end-of-Run,
manifest.json,dag.json, andevents.jsonlare uploaded to the store bucket underruns/<run_id>/. -
CLI status commands.
jin runs list <tenant>,jin runs show <run_id>,jin runs cancel <run_id>— thin wrappers around the manifest and signal channel. -
JinDesk Recent Runs page. SvelteKit page per tenant showing the run_registry table, with drill-down to a Run detail view that reads the events.jsonl.
-
Sub-Run linkage. When
jin shipspawnsjin make, the child’s manifest carries the parent’srun_id. Displayed as a tree in JinDesk. -
jinCommander aggregation. When it exists.
Phases 1-4 unlock most of the visible value and can ship this month. Phases 5-7 are JinDesk-side. Phase 8 waits.
Related senses
Section titled “Related senses”- Sense 55 The Broadcast. The event stream a Run emits. Broadcast is the wire; Run is what walks on the wire.
- Sense 42 The Landscape. Category 2 (river-processing) is where Runs live. The Landscape names the topology; the Run names one execution within it.
- Sense 33 The Bell. The Bell fires on Run outcomes.
- Sense 25 Identity and Passes. Passes gate Run visibility and control.
- Sense 15 The Observation. Observations may link to Runs — “the make of 2026-07-11 revealed that trout-spawning intensity dropped.” Observation cites Run.
- Sense 14 The Signal. Signals may trigger Runs (e.g. stale-material signal fires → automatic re-make).
Open questions
Section titled “Open questions”- Cross-tenant Runs. Some Verbs span tenants (
jin sync-principalsoperates on the whole fleet). Do those Runs live in a tenant’s run_registry, or is there a system-level run_registry in a separate schema? Suggest: system-level Runs live insystem.run_registry(unqualified), pointed at by tenant-level references. - Retention. How long do old Runs live in
_<tenant>.run_registry? Successful Runs might be pruned to a rolling window (e.g. last 90 days); failed Runs kept indefinitely. Configurable via analytical_defaults cascade. - Reproducibility. Given a
run_id, can I re-run the exact same Verb against the exact same inputs? In principle yes —manifest.target.args+manifest.inputs.afs_commit+manifest.inputs.source_hashshould be sufficient. In practice, source may have moved andafs_commitmay not be reachable if the AFS remote’s history was rewritten. Best-effort, not a guarantee. - Cancellation of sub-Runs. If a parent Run is cancelled, should its children also be cancelled? Suggest: yes, by default; the cancel signal to a Run also writes signal files for every child in
child_run_ids. Configurable via trigger kind.
Summary
Section titled “Summary”A Run is a first-class jinflow noun. Every execution of a Verb — make, ship, snapshot, whatever — becomes a persistent, identified, queryable Run. Its shape emerges from data, not from a rigid definition. It lives in three tiers (live, recorded, aggregated). Everything else in jinflow — the Broadcast, the Bell, evolve, jinCommander, JinDesk CLI bottombar — treats it as a citizen it can reference and reason about.
Together with Sense 55 The Broadcast, this Sense gives jinflow a proper substrate for the “process” side of the diamond — one that respects jinflow’s data-first ethos while making executions observable, controllable, and legible.
Numerical neighbors: ← Sense 55: The Broadcast · Sense 57: The Rooms →