Aller au contenu

Sense 56: The Run

Ce contenu n’est pas encore disponible dans votre langue.

Sense 56 · Steady · Last touched 2026-08-20

  • last_verified: 2026-07-27

Synced from docs/design/sense_56_the_run.md in the engine repo — that’s the source; this page is a build-time mirror.

Status: proposed — Phase 1 (Run identity + manifest) shipped 2026-07-12 Date: 2026-07-11 Origin: the first fly machines run — an execution we could observe but not reference. There was nothing named to point at, query, cancel, or later analyse. Companion Sense docs: Sense 55 The Broadcast (what a Run emits) · Sense 42 The Landscape (river-processing category) Related concept: ProcessCore’s Process Definitions as First-Class Citizens — same north star, different mechanics

Every execution of a jinflow Verb is a Run. A Run is a first-class noun in jinflow — persistent, identified, queryable, referenceable. It has a lifecycle, produces evidence, and outlives the process that emitted it.

Runs are not commands that fire and vanish. They are named things you can point at long after the process has exited.

  • Verb — the abstract kind of operation. make, ship, snapshot, backfill-clerk-metadata, sync-principals, evolve, explore, and so on. Every jinflow CLI command IS a Verb. Verbs have names, implementations, and versions (they live in the engine code).
  • Run — a specific execution of a Verb. Has identity, lifecycle, state, produces artefacts.

Every Run has a Verb. A Verb has many Runs. In OO terms: Verb is closer to a class name; a Run is closer to an object — but the analogy is imperfect (see Why not “Definition/Instance” below).

At 17:15 UTC on 2026-07-11, we fired jin make riverflow.delta from a laptop against a Fly Machine in Frankfurt. The Machine pulled AFS + source from R2, ran make, pushed the KLS. We watched via fly logs.

Then we tried to answer three simple questions:

  1. Which execution was that? We didn’t have a name for it. “The 17:15 one” only worked while we remembered it. Two Runs later, we couldn’t tell them apart.

  2. Can we cancel it? The fly machine destroy route worked but was crude — it killed the container, not the Verb it was running. If we had wanted to pause it, we couldn’t.

  3. Can we query it later? “Show me all the makes of delta in the last 24 hours” was impossible. The events had scrolled off fly logs; the KLS was overwritten by the next Run; there was no persistent handle to the specific execution.

The failure was named. We had no first-class citizen for the execution itself. Every jinflow noun was accounted for — signals, theses, verdicts, tenants, KLSes, snapshots, principals, passes — but the VERBS themselves, and the acts of running them, were second-class. Ephemeral. Un-referenceable.

This Sense fixes that.

A Run is a persistent record of one execution of one Verb against one target. It exists in three places over its life:

  • Live (during execution): a directory on disk carrying its identity, DAG, event stream, and control signal file.
  • Recorded (post-completion): a row in the tenant’s _<tenant>.run_registry table inside the KLS, plus its events.jsonl archived in the store bucket.
  • Aggregated (jinCommander, when it exists): a row in the fleet-wide registry that spans tenants, verbs, environments.

A Run’s identity and state are captured by the following schema. This is the shape carried in every persistent tier:

run_id: run_20260711T171531_riverflow.delta_a3f
verb:
name: make # which Verb
version: 0.56.8 # engine version at execution time
target:
tenant: riverflow.delta # what the Verb operated against
args: # exact CLI arguments (for reproducibility)
- --afs-bucket=s3://riverflow-delta-afs/
- --source-bucket=s3://riverflow-delta-source/
- --build-bucket=s3://riverflow-delta-build/
- --kls-bucket=s3://riverflow-delta-store/
trigger: # who/what caused this Run
kind: cli | cron | webhook | signal | jincommander | evolve
actor: mig | schedule_id | signal_id | ...
# for cron: schedule_id + fire_time
# for signal: originating_run_id + signal_name
# for webhook: source_url
lifecycle:
requested_at: 2026-07-11T17:15:29Z # user or trigger asked for the Run
started_at: 2026-07-11T17:15:31Z # execution actually began
finished_at: 2026-07-11T17:20:12Z
status: success # queued | running | paused | success | failed | cancelled
cancelled_by: null # actor id if cancelled
cancelled_reason: null
worker: # where it ran
kind: local | fly_machine | docker
id: 185432ef979168 # Fly machine ID, local hostname, etc.
region: fra # for cloud runs
inputs: # what it read
afs_commit: 4c658b3 # git SHA of the AFS at execution time
afs_dirty: false # was the AFS dirty at execution time
source_hash: sha256-9f8a... # composite hash of source folder
build_incremental: false # did we restore build/ from previous Run?
previous_run_id: run_20260711T170000_... # if incremental, which Run's state was restored
outputs: # what it produced
- kind: kls
uri: s3://riverflow-delta-store/riverflow_delta_kls.duckdb
sha256: sha256-a3f...
size_bytes: 153800000
- kind: snapshot
uri: s3://riverflow-delta-store/20260711-1720_kls.duckdb
broadcast: # links to the Sense 55 event stream
events_uri: s3://riverflow-delta-store/runs/run_.../events.jsonl
dag_uri: s3://riverflow-delta-store/runs/run_.../dag.json # final DAG snapshot
parent_run_id: null # for sub-Runs (e.g. jin ship spawning jin make)
child_run_ids: [] # if this Run spawned sub-Runs

This is what a Run looks like written down. Not everything is known at every stage — lifecycle.finished_at and outputs only appear post-execution — but the schema is unified across live, recorded, and aggregated tiers.

A Run passes through a small set of states:

┌──────────┐ trigger ┌──────────┐ Verb starts ┌──────────┐
│ none │ ───────────► │ queued │ ──────────────► │ running │
└──────────┘ └──────────┘ └────┬─────┘
│
pause signal │
┌────────────────────────────┐ │
▼ │ │
┌──────────┐ resume signal │ │
│ paused │ ─────────────────────┘ │
└──────────┘ │
│
┌────────────────────────────────────┤
│ │
▼ ▼
┌────────────┐ ┌────────────┐
│ cancelled │ │ success │
└────────────┘ │ failed │
└────────────┘
  • queued — a trigger has fired; the Run exists as an intent; the Verb has not yet started
  • running — the Verb is executing
  • paused — a pause signal was observed; execution is on hold at a node boundary
  • success / failed / cancelled — terminal states; the Run is done

Every state transition emits an event on the Broadcast (Sense 55). The run.finished event is always the last event.

This is the philosophical claim that separates Runs from what workflow-first tools like BPMN call an “instance.”

A Run’s shape is not dictated by its Verb’s code alone. It is the code walking the data. The number of dbt models depends on the tenant’s entities. The number of signal compilations depends on the AFS. The number of extractors that run depends on the pipeline.yml. The exact shape of a jin make riverflow.delta execution today is different from a jin make riverflow.delta execution next week — even with the same Verb, same code version, same tenant — because the AFS or source may have changed.

The DAG emerges from execution reading data. It is broadcast in two forms:

  • A coarse skeleton at Run start (the phases that will always exist for this Verb)
  • Expansion events as execution reveals structure (dbt’s model list, compile scripts’ children, etc.)

By the time the Run reaches run.finished, the full DAG is known and can be persisted alongside the events. Before then, it is inherently partial.

This is consistent with the rest of jinflow: signals are data-driven, theses are data-driven, verdicts are data-driven. Runs are data-driven too. They are not instances of a rigid definition; they are executions whose shape emerges from the data they read.

A Run does not necessarily begin because a user typed a command. Triggers are first-class metadata on every Run:

Trigger kindMeaningExample
cliA user invoked the Verb from a shell or JinDesk CLI bottombaractor: mig
cronA scheduled trigger firedactor: schedule_nightly_delta
webhookAn external system posted to jinCommanderactor: cloudflare_bucket_event_...
signalA Sense 14 signal changed severity and had an associated actionactor: signal_stale_material_fired
jincommanderA fleet-management operationactor: rerun_last_failed_batch
evolveThe evolve panel asked to run a Verb on the user’s behalfactor: mig via evolve

The trigger tells you WHY this Run exists. Post-mortem analytics rely on it. jinCommander’s dispatch relies on it. The Bell’s routing relies on it.

Three tiers, corresponding to three lifespans:

A directory on the runner’s filesystem:

<live>/<pack>/<tenant>/runs/<run_id>/
manifest.json # the Run schema above, current state
events.jsonl # the Broadcast (Sense 55)
dag.json # current known DAG
signal.json # control channel (created by senders of cancel/pause)
build/ # intermediary state (mirrored to build bucket if configured)

The manifest is updated on every state transition. When the Run reaches a terminal state, the manifest is finalised and the directory can be uploaded.

Every make bakes a _<tenant>.run_registry table containing every Run that has produced or contributed to this KLS. Schema mirrors the manifest above, one row per Run.

The events.jsonl and dag.json are archived to the store bucket at:

s3://<pack>-<tenant>-store/runs/<run_id>/events.jsonl
s3://<pack>-<tenant>-store/runs/<run_id>/dag.json
s3://<pack>-<tenant>-store/runs/<run_id>/manifest.json

JinDesk can query the registry table for “recent Runs of this tenant” without needing to hit the bucket. When the user drills into a specific Run for detailed event view, the bucket path is fetched.

Tier 3: Aggregated (jinCommander, when it exists)

Section titled “Tier 3: Aggregated (jinCommander, when it exists)”

jinCommander maintains its own DuckDB with a fleet-wide runs table aggregating Run records from every worker. This lets you query across tenants, verbs, environments, and time:

SELECT verb, tenant, status, duration_ms
FROM runs
WHERE requested_at >= now() - INTERVAL 24 HOUR
AND status = 'failed'
ORDER BY requested_at DESC;

jinCommander is future work. The Run schema is designed so it slots in without change when it arrives.

A Run is a first-class citizen among jinflow’s other nouns. Relationships:

FromToRelationship
RunVerbverb.name — every Run is of exactly one Verb
RunTenanttarget.tenant — Runs are usually scoped to a tenant (fleet ops may span)
RunKLSoutputs[].kind: kls — a Run may produce a KLS
RunSnapshotoutputs[].kind: snapshot — a Run may produce a snapshot
RunSignaltrigger.kind: signal, actor: signal_id — a signal firing may trigger a Run
RunObservationObservations may reference a Run via observed_during: run_id (Sense 15)
RunBellThe Bell fires on run.finished (Sense 33)
RunRunparent_run_id, child_run_ids — Runs can spawn sub-Runs (e.g. jin ship spawns jin make)
RunPassThe mila pass sees all Runs; a tenant admin’s Pass sees their tenant’s Runs (Sense 25)

An earlier draft of this Sense used the vocabulary of “Definition/Instance” from ProcessCore. It was wrong for jinflow.

In BPMN and workflow-engine thinking, a Definition is a stable, versioned schema of nodes and transitions — a musical score. An Instance is a specific play-through of that score. The score dictates the shape; the Instance follows it. This is workflow-first thinking.

Jinflow is data-first. The shape of any given Run is not dictated by its Verb’s code — it emerges from the code walking the actual data. Two Runs of the same Verb against the same tenant, days apart, may have wildly different DAGs. Not because the Verb changed, but because the data did.

Adopting Definition/Instance would have said: “the score is stable, the performance is not.” That is not what happens in jinflow. The CODE is stable (the Verb implementation lives at an engine version), but the shape of any Run is data-first, not schema-first.

So we use Verb for the abstract action (“the class of things called make”) and Run for the concrete emergent execution. Verb is not a Definition. Run is not an Instance. The relationship is looser and more honest.

  • Not a workflow orchestrator. This Sense names one execution and how it persists. Chaining, scheduling, and cross-Run coordination live in jinCommander.
  • Not a distributed transaction system. A Run is atomic-ish (it succeeds or fails as a whole) but does not carry ACID guarantees across sub-Runs. If a jin ship succeeds and its child jin make fails, the parent Run is what the user sees; the child’s failure is its own record.
  • Not process definition. Verbs are defined in engine code, not in YAML. This Sense does not propose a DSL for declaring new Verbs — that’s a much bigger conversation adjacent to Sparky’s ProcessCore vision, deliberately deferred.
  • Not a queueing system. Runs can queue (state = queued), but jinflow itself does not maintain queues. jinCommander might, in the future. For now, queued means “requested but not yet started”; no ordering guarantee beyond that.
  1. Run identity + manifest. ✅ Shipped 2026-07-12. Every jin make invocation opens a <live>/.../runs/<run_id>/manifest.json at start and updates it on every state transition. The run_id is derivable (timestamp + tenant + short hash of args). Lives in the jinflow.run module (Run, mint_run_id, detect_worker); the manifest records verb/version, target/args, trigger, lifecycle, worker (Fly-aware), inputs, and outputs (the produced KLS/snapshot). Best-effort — a Run that cannot persist degrades to an in-memory record and never aborts make. (Baking run_id into every existing bake step is still open — see Phase 3.)

  2. Event stream (Sense 55). Manifest updates go hand-in-hand with Broadcast events. Adding one to jin make inherently adds the other.

  3. KLS-baked run_registry table. A new bake step at end-of-make writes the Run’s manifest as a row in _<tenant>.run_registry. JinDesk gains a “Recent Runs” page for each tenant.

  4. Bucket archival. At end-of-Run, manifest.json, dag.json, and events.jsonl are uploaded to the store bucket under runs/<run_id>/.

  5. CLI status commands. jin runs list <tenant>, jin runs show <run_id>, jin runs cancel <run_id> — thin wrappers around the manifest and signal channel.

  6. JinDesk Recent Runs page. SvelteKit page per tenant showing the run_registry table, with drill-down to a Run detail view that reads the events.jsonl.

  7. Sub-Run linkage. When jin ship spawns jin make, the child’s manifest carries the parent’s run_id. Displayed as a tree in JinDesk.

  8. jinCommander aggregation. When it exists.

Phases 1-4 unlock most of the visible value and can ship this month. Phases 5-7 are JinDesk-side. Phase 8 waits.

  • Cross-tenant Runs. Some Verbs span tenants (jin sync-principals operates on the whole fleet). Do those Runs live in a tenant’s run_registry, or is there a system-level run_registry in a separate schema? Suggest: system-level Runs live in system.run_registry (unqualified), pointed at by tenant-level references.
  • Retention. How long do old Runs live in _<tenant>.run_registry? Successful Runs might be pruned to a rolling window (e.g. last 90 days); failed Runs kept indefinitely. Configurable via analytical_defaults cascade.
  • Reproducibility. Given a run_id, can I re-run the exact same Verb against the exact same inputs? In principle yes — manifest.target.args + manifest.inputs.afs_commit + manifest.inputs.source_hash should be sufficient. In practice, source may have moved and afs_commit may not be reachable if the AFS remote’s history was rewritten. Best-effort, not a guarantee.
  • Cancellation of sub-Runs. If a parent Run is cancelled, should its children also be cancelled? Suggest: yes, by default; the cancel signal to a Run also writes signal files for every child in child_run_ids. Configurable via trigger kind.

A Run is a first-class jinflow noun. Every execution of a Verb — make, ship, snapshot, whatever — becomes a persistent, identified, queryable Run. Its shape emerges from data, not from a rigid definition. It lives in three tiers (live, recorded, aggregated). Everything else in jinflow — the Broadcast, the Bell, evolve, jinCommander, JinDesk CLI bottombar — treats it as a citizen it can reference and reason about.

Together with Sense 55 The Broadcast, this Sense gives jinflow a proper substrate for the “process” side of the diamond — one that respects jinflow’s data-first ethos while making executions observable, controllable, and legible.


Numerical neighbors: ← Sense 55: The Broadcast · Sense 57: The Rooms →

jazzisnow jinflow is a jazzisnow product
v0.65.0 · built 2026-11-13 10:20 UTC