Aller au contenu

Sense 55: The Broadcast

Ce contenu n’est pas encore disponible dans votre langue.

Sense 55 · Steady · Last touched 2026-08-20

  • last_verified: 2026-07-27

Synced from docs/design/sense_55_the_broadcast.md in the engine repo — that’s the source; this page is a build-time mirror.

Status: proposed — Phases 1, 2, 4 shipped 2026-07-12 (JSONL broadcast, signal control, SSE daemon) Date: 2026-07-11 Origin: the first fly machines run of jinflow-worker — we watched a live cloud make from our terminal via fly logs and realized every surface that wants to observe a running jinflow verb needs the same structured event stream. Companion Sense docs: Sense 56 The Run (the entity that broadcasts) · Sense 42 The Landscape (river-processing category) Related concept: ProcessCore’s Runtime Responsibilities — north star for observability of long-running processes

Every long-running jinflow verb — jin make, jin ship, jin snapshot, whatever comes next — should broadcast structured, data-driven events as it proceeds. Every surface that wants to observe the verb — JinDesk’s CLI panel, the evolve chat, jinCommander, the Bell, a future desktop status app — should consume from the same stream in the same shape.

There is one wire. Everything talks over it.

Late in the day on 2026-07-11 we fired the first cloud invocation of jinflow-worker — a Fly Machine in Frankfurt, four R2 buckets in EU, jin make riverflow.delta running to completion inside a container we could not attach to.

To know what was happening we tailed fly logs in a terminal. It worked. But we could not tell — from those logs alone — how far along the make was, which phase was current, whether we were stuck or making progress. dbt printed model-by-model output; the compile scripts printed one line each; the bake steps printed nothing. The signal was there. The shape was not.

Then the surrounding conversation crystallised:

  • You want a CLI bottombar in JinDesk — the horizontal counterpart to the evolve vertical — where a user types jin make hrcentral.vai and watches it run right there.
  • jinCommander will need to schedule and monitor many verbs across many tenants. It needs a stream, not a log.
  • The Bell (Sense 33) should ring when a run finishes — but only if it knows which run, in which state.
  • Evolve should be able to reason about runs — “what happened in the last three makes of numetrix.rmc?” — which requires runs to be queryable AND their state to be legible.

Every one of these surfaces needs the same thing: a structured, well-defined event stream describing what a running verb is doing. Right now we have unstructured stdout. That’s not a substrate. This Sense names the substrate.

Two words that appear throughout this doc and its companion Sense 56 The Run:

  • Verb — the abstract kind of operation (make, ship, snapshot, backfill-clerk-metadata, …). Every jinflow CLI command is a Verb.
  • Run — a specific execution of a Verb. Has identity, lifecycle, state, produced artefacts. See Sense 56 for the full noun.

A Run broadcasts. A Verb is what a Run is a Run OF.

Every Run emits a sequence of structured events during its life. The sequence is the Broadcast.

Every event has the same shape:

{
"specversion": "1.0",
"id": "ev_20260711T171531_a3f_0047",
"run_id": "run_20260711T171531_riverflow.delta_a3f",
"verb": "make",
"tenant": "riverflow.delta",
"worker_id": "185432ef979168",
"sequence": 47,
"time": "2026-07-11T17:15:31.234Z",
"type": "dbt.model.finished",
"data": { ... event-specific payload ... }
}

We adopt CloudEvents 1.0 as the envelope specification. CloudEvents is the CNCF standard for structured events; every event bus (NATS, Kafka, Redis Streams, Cloudflare Queues, AWS EventBridge) speaks it natively. We get a well-defined shape for id, time, type, and data for free.

The fields specific to jinflow — run_id, verb, tenant, worker_id, sequence — live under CloudEvents’ extension mechanism, which permits arbitrary top-level fields.

The most important event a Run emits is not a phase marker — it is the DAG that describes what the Run is going to do.

Because jinflow is data-first, the DAG cannot be fully known in advance. A Run’s shape is not dictated by its Verb’s code alone; it is the code walking the actual data. The number of dbt models depends on the tenant’s entities. The number of signal compilations depends on the AFS. The number of extracted files depends on what arrived in the DLZ. The DAG is emergent.

Therefore the DAG is broadcast in two forms:

  1. run.dag — emitted once at Run start with the coarse skeleton known before execution reads data. For jin make, that’s the phase nodes (phase_0, phase_0b, phase_1a, phase_1b, phase_2, phase_3, phase_4) and their dependencies.

  2. run.dag.expanded — emitted whenever the Run learns more of its own shape. When dbt has parsed the project, an expansion event enumerates every model as a child of phase_3. When compile scripts run, they announce their compile-step children of phase_2.

Every subsequent event references the DAG by node ID:

{ "type": "run.node.started", "data": { "node_id": "phase_1a" } }
{ "type": "run.node.finished", "data": { "node_id": "phase_1a", "status": "success", "duration_ms": 1234 } }

Consumers don’t need per-Verb knowledge. They render whatever DAG they receive. A CLI bottombar renders a live tree. jinCommander shows a Gantt chart. The Bell counts pending nodes. Every surface, same wire.

Twelve event types cover everything a Run emits during its life:

TypeWhen
run.startedVerb execution has begun (very first event of a Run)
run.dagThe coarse DAG skeleton is now known
run.dag.expandedAdditional nodes are known (emergent shape)
run.node.startedA DAG node begins executing
run.node.finishedA DAG node completes (status + duration in payload)
run.node.skippedA DAG node was skipped (e.g. incremental, no changes)
run.logFree-text stdout capture (throttled; escape hatch)
run.warningStructured diagnostic (jinflow W-code + context)
run.errorStructured error (jinflow E-code + context)
run.pausedRuntime observed a pause signal and is waiting
run.resumedPause cleared, execution continues
run.finishedTerminal event (`status: success

That’s the whole vocabulary. Not two. Not fifty. Twelve — small enough to enumerate on a page, large enough to describe every shape a Run can take.

The Broadcast is transport-agnostic. Consumers connect over whichever transport fits their situation. The initial two:

The jin make process writes each event as one JSON line to:

<live>/<pack>/<tenant>/runs/<run_id>/events.jsonl

Also written to stdout so fly logs, terminal tails, and docker run viewers pick it up.

At Run finish, the file is uploaded to the store bucket alongside the KLS:

s3://<pack>-<tenant>-store/runs/<run_id>/events.jsonl

Consumers that don’t need real-time (post-mortem analysis, jinCommander archival, evolve-panel history queries) read from the bucket.

Zero infrastructure required. Files, everywhere.

The Phase 2 jinflow-worker daemon variant hosts an HTTP endpoint:

GET /runs/{run_id}/events → text/event-stream (SSE)

Each SSE frame is one CloudEvents-encoded event. Consumers (JinDesk’s CLI bottombar, jinCommander’s live view) subscribe once and receive events as they occur. Reconnects use the SSE Last-Event-ID header for resumption.

SSE is a W3C standard. Browsers speak it natively via EventSource. Zero library needed on JinDesk side.

Both transports emit the same event shape. Both are legitimate. Choose per situation.

Broadcasting alone is not enough. Consumers need to be able to intervene — cancel a Run, pause it, request a status update. That flows the other direction, so it is a companion mechanism, not part of the Broadcast itself.

Between DAG nodes, the running Verb checks:

<live>/<pack>/<tenant>/runs/<run_id>/signal.json

Content is a JSON directive:

{ "action": "cancel", "requested_by": "cli/mig", "requested_at": "2026-07-11T17:20:00Z" }

Recognised actions:

ActionEffect
cancelVerb exits cleanly at next node boundary. Emits run.finished with status: cancelled.
pauseVerb loops with 1-second sleep, re-reading the signal file until action clears or changes. Emits run.paused on entry, run.resumed on exit.
statusVerb emits a synthetic run.log with a snapshot of current node state (no execution effect).

jin gains a companion CLI verb: jin signal <run_id> <action> writes to the signal file.

The daemon variant hosts a companion endpoint:

POST /runs/{run_id}/signal
Content-Type: application/json
{ "action": "cancel" }

Same shape, just written over the wire instead of to disk. JinDesk’s CLI bottombar issues these on the user’s “cancel” button.

  • No mid-node cancellation. A Verb only checks the signal at node boundaries. If a dbt model is halfway through, it will finish. This is a deliberate trade — Verbs are not required to poll during long-running child processes.
  • No adjustment. Changing target sets, adding tenants, or altering args mid-run is not supported. The Run is what it is; if you need something different, cancel and start a fresh Run.
  • No sub-Run control. If a Run has spawned sub-Runs (see Sense 56), each has its own control channel.

Cover 95% of what “intervene in a running Run” actually means. Leave the other 5% for future refinement.

Named surfaces the Broadcast enables:

JinDesk CLI bottombar (in-browser terminal). The horizontal counterpart to the evolve vertical. User types jin make riverflow.delta in the panel. JinDesk server invokes the Verb (locally or via daemon), streams the Broadcast to the browser over SSE, renders the DAG as a live tree, and offers a cancel button that writes to the signal channel. Two-way, real-time, no polling.

Evolve panel (conversational). Queries against the run registry: “what happened in the last three makes of numetrix.rmc?” The panel reads Run records (see Sense 56) and their event streams from the store bucket. Reasons about durations, failure modes, trends.

jinCommander (planned). Aggregates event streams from many workers into a fleet view. Answers “which Runs are running right now?” “Which failed at Phase 1b this week?” “What’s our success rate?” Built on the same wire.

The Bell (Sense 33). Rings on run.finished with status: failed for any tenant a subscriber owns. Rings on cross-tenant fleet events (e.g. “make failed on 3 tenants in the same 10-minute window”).

Terminal tails. jinflow tail <run_id> reads events from the JSONL file (local Run) or SSE stream (daemon Run) and renders them in the same style as fly logs — real-time, colored, structured. Reuses whichever code JinDesk’s bottombar uses.

Post-mortem analysis. When a Run fails and 3 hours later someone asks “why?”, the events.jsonl is in the store bucket. Load it, replay it, understand it.

  • Not workflow orchestration. The Broadcast describes what a running Verb is doing. It does not schedule Verbs, chain them, or manage dependencies between them. That is jinCommander’s job.
  • Not sub-tick observability. Events fire at node boundaries — not every second. If a dbt model runs for 60s, you get one run.node.started and one run.node.finished bracketing it. If you need continuous progress inside the model, that is dbt’s problem to solve; we do not proxy it.
  • Not authentication or authorization. Consumers of the Broadcast must have already been granted access to the Run (via the Pass system, Sense 25). This Sense assumes the audience is authorized.
  • Not persistence of the Run itself. The Broadcast is the events; the Run is the entity that emits them. See Sense 56 for the Run’s identity, lifecycle, and persistence model.
  1. Broadcast to JSONL. ✅ Shipped 2026-07-12. jin make emits CloudEvents to <live>/.../runs/<run_id>/events.jsonl and (sentinel-prefixed) to stdout. Lives in the jinflow.broadcast module (Broadcast); all 12 event types, emergent DAG persisted to dag.json. jin make broadcasts the coarse 7-phase skeleton at start and brackets each phase with run.node.started/finished (real durations), terminating with run.finished on every exit path. stdout mirroring defaults on for non-TTY (cloud fly logs) and off for interactive terminals; JINFLOW_BROADCAST_STDOUT=1|0 overrides. Best-effort — an emit failure never aborts the build. (Emergent dag.expanded for per-model dbt nodes is scaffolded but not yet wired.)

  2. Signal file control. ✅ Shipped 2026-07-12. jin make checks runs/<run_id>/signal.json at each node boundary (Broadcast.check_signal) — cancel fires run.finished(cancelled) and unwinds cleanly, pause/resume loop on run.paused/run.resumed, status emits a node snapshot. The jin signal <run_id> cancel|pause|resume|status CLI verb writes the file.

  3. Bucket sync of events. At end-of-Run, upload events.jsonl to the store bucket alongside the KLS. Enables post-mortem access without local filesystem.

  4. SSE endpoint on the daemon variant. ✅ Shipped 2026-07-12. The jinflow worker serve daemon (jinflow.worker_server, docs/design/cloud_processing.md) hosts GET /runs/{run_id}/events (SSE, Last-Event-ID resumption) + POST /runs/{run_id}/signal, plus POST /run, GET /runs, GET /runs/{id}, GET /healthz. Bearer-token auth; command whitelist; subprocess execution reusing the CLI Broadcast.

  5. JinDesk CLI bottombar. SvelteKit component that subscribes to the SSE stream, renders the DAG live, shows a cancel button. Companion to the evolve vertical.

  6. jinCommander event aggregation. When it exists.

Phase 1-3 are cheap and can ship this month. Phases 4-6 wait for their respective substrates.

  • Sense 56 The Run. The persistent noun this Sense broadcasts events for. Where Broadcast is the wire, the Run is the thing that walks on the wire.
  • Sense 42 The Landscape. Category 2 (river-processing) is what broadcasts; the Broadcast is how you observe Category 2 in flight.
  • Sense 33 The Bell. Consumes run.finished events to fire notifications.
  • Sense 25 Identity and Passes. Gates access to the Broadcast — mila sees all Runs’ streams, tenant admin sees their tenant’s Runs.
  • Sense 15 The Observation. An Observation can reference a Run (observed_during: run_...) to link human notice to machine event.
  • Retention. How long does events.jsonl live in the store bucket? Forever seems wasteful for successful Runs; failure Runs deserve long retention for post-mortem. Suggest a lifecycle policy per bucket (<tenant>-store/runs/*/events.jsonl → 30 days for success, 1 year for failed/cancelled).
  • Sampling. For very high-cadence Runs (continuous processing every 60s), broadcasting every DAG node event may generate a lot of data. Do we sample? Aggregate? Not addressed by this Sense; defer until we feel the pain.
  • Cross-Verb Run correlation. When a jin ship spawns a jin make sub-Run, the events for both should be reference-linked. Sense 56 handles this by making sub-Runs first-class children with parent_run_id. The Broadcast just carries the field through.

There is one wire. Every long-running jinflow verb broadcasts on it, in the same shape, describing what it is doing. Consumers subscribe and render, without needing per-Verb knowledge. Control flows through a companion signal file (local) or POST endpoint (daemon). The whole substrate lives at the intersection of two well-adopted specs (CloudEvents + SSE), inventing nothing that already exists.

This is the substrate the CLI bottombar sits on. The jinCommander sits on. The evolve-panel’s history queries sit on. The Bell sits on. Everything observability-adjacent in jinflow, from this point forward, sits on this wire.


Numerical neighbors: ← Sense 54 — The Typology · Sense 56: The Run →

jazzisnow jinflow is a jazzisnow product
v0.65.0 · built 2026-11-13 10:20 UTC