Sense 55: The Broadcast
Ce contenu n’est pas encore disponible dans votre langue.
Sense 55 · Steady · Last touched 2026-08-20
- last_verified: 2026-07-27
Synced from
docs/design/sense_55_the_broadcast.mdin the engine repo — that’s the source; this page is a build-time mirror.
Status: proposed — Phases 1, 2, 4 shipped 2026-07-12 (JSONL broadcast, signal control, SSE daemon)
Date: 2026-07-11
Origin: the first fly machines run of jinflow-worker — we watched a live cloud make from our terminal via fly logs and realized every surface that wants to observe a running jinflow verb needs the same structured event stream.
Companion Sense docs: Sense 56 The Run (the entity that broadcasts) · Sense 42 The Landscape (river-processing category)
Related concept: ProcessCore’s Runtime Responsibilities — north star for observability of long-running processes
Every long-running jinflow verb — jin make, jin ship, jin snapshot, whatever comes next — should broadcast structured, data-driven events as it proceeds. Every surface that wants to observe the verb — JinDesk’s CLI panel, the evolve chat, jinCommander, the Bell, a future desktop status app — should consume from the same stream in the same shape.
There is one wire. Everything talks over it.
The story that made this Sense necessary
Section titled “The story that made this Sense necessary”Late in the day on 2026-07-11 we fired the first cloud invocation of jinflow-worker — a Fly Machine in Frankfurt, four R2 buckets in EU, jin make riverflow.delta running to completion inside a container we could not attach to.
To know what was happening we tailed fly logs in a terminal. It worked. But we could not tell — from those logs alone — how far along the make was, which phase was current, whether we were stuck or making progress. dbt printed model-by-model output; the compile scripts printed one line each; the bake steps printed nothing. The signal was there. The shape was not.
Then the surrounding conversation crystallised:
- You want a CLI bottombar in JinDesk — the horizontal counterpart to the evolve vertical — where a user types
jin make hrcentral.vaiand watches it run right there. - jinCommander will need to schedule and monitor many verbs across many tenants. It needs a stream, not a log.
- The Bell (Sense 33) should ring when a run finishes — but only if it knows which run, in which state.
- Evolve should be able to reason about runs — “what happened in the last three makes of numetrix.rmc?” — which requires runs to be queryable AND their state to be legible.
Every one of these surfaces needs the same thing: a structured, well-defined event stream describing what a running verb is doing. Right now we have unstructured stdout. That’s not a substrate. This Sense names the substrate.
The vocabulary
Section titled “The vocabulary”Two words that appear throughout this doc and its companion Sense 56 The Run:
- Verb — the abstract kind of operation (
make,ship,snapshot,backfill-clerk-metadata, …). Every jinflow CLI command is a Verb. - Run — a specific execution of a Verb. Has identity, lifecycle, state, produced artefacts. See Sense 56 for the full noun.
A Run broadcasts. A Verb is what a Run is a Run OF.
The Broadcast
Section titled “The Broadcast”Every Run emits a sequence of structured events during its life. The sequence is the Broadcast.
The envelope
Section titled “The envelope”Every event has the same shape:
{ "specversion": "1.0", "id": "ev_20260711T171531_a3f_0047", "run_id": "run_20260711T171531_riverflow.delta_a3f", "verb": "make", "tenant": "riverflow.delta", "worker_id": "185432ef979168", "sequence": 47, "time": "2026-07-11T17:15:31.234Z", "type": "dbt.model.finished", "data": { ... event-specific payload ... }}We adopt CloudEvents 1.0 as the envelope specification. CloudEvents is the CNCF standard for structured events; every event bus (NATS, Kafka, Redis Streams, Cloudflare Queues, AWS EventBridge) speaks it natively. We get a well-defined shape for id, time, type, and data for free.
The fields specific to jinflow — run_id, verb, tenant, worker_id, sequence — live under CloudEvents’ extension mechanism, which permits arbitrary top-level fields.
The DAG contract
Section titled “The DAG contract”The most important event a Run emits is not a phase marker — it is the DAG that describes what the Run is going to do.
Because jinflow is data-first, the DAG cannot be fully known in advance. A Run’s shape is not dictated by its Verb’s code alone; it is the code walking the actual data. The number of dbt models depends on the tenant’s entities. The number of signal compilations depends on the AFS. The number of extracted files depends on what arrived in the DLZ. The DAG is emergent.
Therefore the DAG is broadcast in two forms:
-
run.dag— emitted once at Run start with the coarse skeleton known before execution reads data. Forjin make, that’s the phase nodes (phase_0,phase_0b,phase_1a,phase_1b,phase_2,phase_3,phase_4) and their dependencies. -
run.dag.expanded— emitted whenever the Run learns more of its own shape. When dbt has parsed the project, an expansion event enumerates every model as a child ofphase_3. When compile scripts run, they announce their compile-step children ofphase_2.
Every subsequent event references the DAG by node ID:
{ "type": "run.node.started", "data": { "node_id": "phase_1a" } }{ "type": "run.node.finished", "data": { "node_id": "phase_1a", "status": "success", "duration_ms": 1234 } }Consumers don’t need per-Verb knowledge. They render whatever DAG they receive. A CLI bottombar renders a live tree. jinCommander shows a Gantt chart. The Bell counts pending nodes. Every surface, same wire.
The event vocabulary
Section titled “The event vocabulary”Twelve event types cover everything a Run emits during its life:
| Type | When |
|---|---|
run.started | Verb execution has begun (very first event of a Run) |
run.dag | The coarse DAG skeleton is now known |
run.dag.expanded | Additional nodes are known (emergent shape) |
run.node.started | A DAG node begins executing |
run.node.finished | A DAG node completes (status + duration in payload) |
run.node.skipped | A DAG node was skipped (e.g. incremental, no changes) |
run.log | Free-text stdout capture (throttled; escape hatch) |
run.warning | Structured diagnostic (jinflow W-code + context) |
run.error | Structured error (jinflow E-code + context) |
run.paused | Runtime observed a pause signal and is waiting |
run.resumed | Pause cleared, execution continues |
run.finished | Terminal event (`status: success |
That’s the whole vocabulary. Not two. Not fifty. Twelve — small enough to enumerate on a page, large enough to describe every shape a Run can take.
Transports
Section titled “Transports”The Broadcast is transport-agnostic. Consumers connect over whichever transport fits their situation. The initial two:
JSONL file (one-shot mode’s home)
Section titled “JSONL file (one-shot mode’s home)”The jin make process writes each event as one JSON line to:
<live>/<pack>/<tenant>/runs/<run_id>/events.jsonlAlso written to stdout so fly logs, terminal tails, and docker run viewers pick it up.
At Run finish, the file is uploaded to the store bucket alongside the KLS:
s3://<pack>-<tenant>-store/runs/<run_id>/events.jsonlConsumers that don’t need real-time (post-mortem analysis, jinCommander archival, evolve-panel history queries) read from the bucket.
Zero infrastructure required. Files, everywhere.
HTTP SSE stream (daemon mode’s home)
Section titled “HTTP SSE stream (daemon mode’s home)”The Phase 2 jinflow-worker daemon variant hosts an HTTP endpoint:
GET /runs/{run_id}/events → text/event-stream (SSE)Each SSE frame is one CloudEvents-encoded event. Consumers (JinDesk’s CLI bottombar, jinCommander’s live view) subscribe once and receive events as they occur. Reconnects use the SSE Last-Event-ID header for resumption.
SSE is a W3C standard. Browsers speak it natively via EventSource. Zero library needed on JinDesk side.
Both transports emit the same event shape. Both are legitimate. Choose per situation.
The control channel
Section titled “The control channel”Broadcasting alone is not enough. Consumers need to be able to intervene — cancel a Run, pause it, request a status update. That flows the other direction, so it is a companion mechanism, not part of the Broadcast itself.
Signal file (one-shot mode’s control)
Section titled “Signal file (one-shot mode’s control)”Between DAG nodes, the running Verb checks:
<live>/<pack>/<tenant>/runs/<run_id>/signal.jsonContent is a JSON directive:
{ "action": "cancel", "requested_by": "cli/mig", "requested_at": "2026-07-11T17:20:00Z" }Recognised actions:
| Action | Effect |
|---|---|
cancel | Verb exits cleanly at next node boundary. Emits run.finished with status: cancelled. |
pause | Verb loops with 1-second sleep, re-reading the signal file until action clears or changes. Emits run.paused on entry, run.resumed on exit. |
status | Verb emits a synthetic run.log with a snapshot of current node state (no execution effect). |
jin gains a companion CLI verb: jin signal <run_id> <action> writes to the signal file.
HTTP POST (daemon mode’s control)
Section titled “HTTP POST (daemon mode’s control)”The daemon variant hosts a companion endpoint:
POST /runs/{run_id}/signalContent-Type: application/json
{ "action": "cancel" }Same shape, just written over the wire instead of to disk. JinDesk’s CLI bottombar issues these on the user’s “cancel” button.
What the control channel does not do
Section titled “What the control channel does not do”- No mid-node cancellation. A Verb only checks the signal at node boundaries. If a dbt model is halfway through, it will finish. This is a deliberate trade — Verbs are not required to poll during long-running child processes.
- No adjustment. Changing target sets, adding tenants, or altering args mid-run is not supported. The Run is what it is; if you need something different, cancel and start a fresh Run.
- No sub-Run control. If a Run has spawned sub-Runs (see Sense 56), each has its own control channel.
Cover 95% of what “intervene in a running Run” actually means. Leave the other 5% for future refinement.
Consumers
Section titled “Consumers”Named surfaces the Broadcast enables:
JinDesk CLI bottombar (in-browser terminal).
The horizontal counterpart to the evolve vertical. User types jin make riverflow.delta in the panel. JinDesk server invokes the Verb (locally or via daemon), streams the Broadcast to the browser over SSE, renders the DAG as a live tree, and offers a cancel button that writes to the signal channel. Two-way, real-time, no polling.
Evolve panel (conversational). Queries against the run registry: “what happened in the last three makes of numetrix.rmc?” The panel reads Run records (see Sense 56) and their event streams from the store bucket. Reasons about durations, failure modes, trends.
jinCommander (planned). Aggregates event streams from many workers into a fleet view. Answers “which Runs are running right now?” “Which failed at Phase 1b this week?” “What’s our success rate?” Built on the same wire.
The Bell (Sense 33).
Rings on run.finished with status: failed for any tenant a subscriber owns. Rings on cross-tenant fleet events (e.g. “make failed on 3 tenants in the same 10-minute window”).
Terminal tails.
jinflow tail <run_id> reads events from the JSONL file (local Run) or SSE stream (daemon Run) and renders them in the same style as fly logs — real-time, colored, structured. Reuses whichever code JinDesk’s bottombar uses.
Post-mortem analysis. When a Run fails and 3 hours later someone asks “why?”, the events.jsonl is in the store bucket. Load it, replay it, understand it.
What this Sense does not do
Section titled “What this Sense does not do”- Not workflow orchestration. The Broadcast describes what a running Verb is doing. It does not schedule Verbs, chain them, or manage dependencies between them. That is jinCommander’s job.
- Not sub-tick observability. Events fire at node boundaries — not every second. If a dbt model runs for 60s, you get one
run.node.startedand onerun.node.finishedbracketing it. If you need continuous progress inside the model, that is dbt’s problem to solve; we do not proxy it. - Not authentication or authorization. Consumers of the Broadcast must have already been granted access to the Run (via the Pass system, Sense 25). This Sense assumes the audience is authorized.
- Not persistence of the Run itself. The Broadcast is the events; the Run is the entity that emits them. See Sense 56 for the Run’s identity, lifecycle, and persistence model.
Implementation phases
Section titled “Implementation phases”-
Broadcast to JSONL. ✅ Shipped 2026-07-12.
jin makeemits CloudEvents to<live>/.../runs/<run_id>/events.jsonland (sentinel-prefixed) to stdout. Lives in thejinflow.broadcastmodule (Broadcast); all 12 event types, emergent DAG persisted todag.json.jin makebroadcasts the coarse 7-phase skeleton at start and brackets each phase withrun.node.started/finished(real durations), terminating withrun.finishedon every exit path. stdout mirroring defaults on for non-TTY (cloudfly logs) and off for interactive terminals;JINFLOW_BROADCAST_STDOUT=1|0overrides. Best-effort — an emit failure never aborts the build. (Emergentdag.expandedfor per-model dbt nodes is scaffolded but not yet wired.) -
Signal file control. ✅ Shipped 2026-07-12.
jin makechecksruns/<run_id>/signal.jsonat each node boundary (Broadcast.check_signal) —cancelfiresrun.finished(cancelled)and unwinds cleanly,pause/resumeloop onrun.paused/run.resumed,statusemits a node snapshot. Thejin signal <run_id> cancel|pause|resume|statusCLI verb writes the file. -
Bucket sync of events. At end-of-Run, upload
events.jsonlto the store bucket alongside the KLS. Enables post-mortem access without local filesystem. -
SSE endpoint on the daemon variant. ✅ Shipped 2026-07-12. The
jinflow worker servedaemon (jinflow.worker_server,docs/design/cloud_processing.md) hostsGET /runs/{run_id}/events(SSE,Last-Event-IDresumption) +POST /runs/{run_id}/signal, plusPOST /run,GET /runs,GET /runs/{id},GET /healthz. Bearer-token auth; command whitelist; subprocess execution reusing the CLI Broadcast. -
JinDesk CLI bottombar. SvelteKit component that subscribes to the SSE stream, renders the DAG live, shows a cancel button. Companion to the evolve vertical.
-
jinCommander event aggregation. When it exists.
Phase 1-3 are cheap and can ship this month. Phases 4-6 wait for their respective substrates.
Related senses
Section titled “Related senses”- Sense 56 The Run. The persistent noun this Sense broadcasts events for. Where Broadcast is the wire, the Run is the thing that walks on the wire.
- Sense 42 The Landscape. Category 2 (river-processing) is what broadcasts; the Broadcast is how you observe Category 2 in flight.
- Sense 33 The Bell. Consumes
run.finishedevents to fire notifications. - Sense 25 Identity and Passes. Gates access to the Broadcast — mila sees all Runs’ streams, tenant admin sees their tenant’s Runs.
- Sense 15 The Observation. An Observation can reference a Run (
observed_during: run_...) to link human notice to machine event.
Open questions
Section titled “Open questions”- Retention. How long does
events.jsonllive in the store bucket? Forever seems wasteful for successful Runs; failure Runs deserve long retention for post-mortem. Suggest a lifecycle policy per bucket (<tenant>-store/runs/*/events.jsonl→ 30 days for success, 1 year for failed/cancelled). - Sampling. For very high-cadence Runs (continuous processing every 60s), broadcasting every DAG node event may generate a lot of data. Do we sample? Aggregate? Not addressed by this Sense; defer until we feel the pain.
- Cross-Verb Run correlation. When a
jin shipspawns ajin makesub-Run, the events for both should be reference-linked. Sense 56 handles this by making sub-Runs first-class children withparent_run_id. The Broadcast just carries the field through.
Summary
Section titled “Summary”There is one wire. Every long-running jinflow verb broadcasts on it, in the same shape, describing what it is doing. Consumers subscribe and render, without needing per-Verb knowledge. Control flows through a companion signal file (local) or POST endpoint (daemon). The whole substrate lives at the intersection of two well-adopted specs (CloudEvents + SSE), inventing nothing that already exists.
This is the substrate the CLI bottombar sits on. The jinCommander sits on. The evolve-panel’s history queries sit on. The Bell sits on. Everything observability-adjacent in jinflow, from this point forward, sits on this wire.
Numerical neighbors: ← Sense 54 — The Typology · Sense 56: The Run →