Skip to content

Sense 42: The Landscape

Sense 42 · In bloom · Last touched 2026-08-20

  • last_verified: 2026-06-07

Synced from docs/design/sense_42_the_landscape.md in the engine repo — that’s the source; this page is a build-time mirror.

The analyst stands on a hill at dawn. The Tuscan light reveals the shape of what’s there: a garden on the slope, a river in the valley, paths between them.

The Landscape is the vantage. Walk it, return at a different hour, see what the same data wears today.

Status: in_bloom — dissolves into Sense 57 The Rooms (2026-07-13): the Garden becomes the Analysis room’s canvas, the River the Flow room’s. The metaphor survives; it splits across two rooms rather than sitting as one “Landscape” view. Date: 2026-05-22 (original title was “The Pipes”); rewritten 2026-06-07 as “The Landscape” Author: jazzisnow Architect + Claude Title history:

  • 2026-05-22 → 2026-06-06: The Pipes. The original framing rendered the diamond surface as a navigable 2D pipeline diagram.
  • 2026-06-07: renamed The Landscape. The same Sense; the scope was broadened. “Pipeline” misnamed the centre-less, multi-view surface as a one-directional conveyor. The Landscape carries the meta-container honestly: a vantage where multiple views of the system coexist.

Adjacent:

  • Sense 13: The Canvas — Canvas is the analyst’s workbench; Landscape is the system’s vantage. Siblings, both composable.
  • Sense 14: The Signal — defines the signal → perspective → thesis → verdict chain. The Garden (the structural view within the Landscape) paints it.
  • Sense 17: The Cathedralretired. The Landscape’s Garden replaces it. (See “What it replaces” below.)
  • Sense 36: The Lineage — Landscape shares the topology Lineage tracks, but Lineage is about provenance over time; the Garden is about shape right now.
  • Sense 47: The Log — the substrate that the River reads from.
  • Sense 14.2: Typed Signals — once units propagate to signals, the Garden’s edges can be coloured by impact dimension.

The Landscape is the meta-surface the analyst stands on. Within it live multiple views, each named for a natural landscape feature that matches its character:

ViewLandscape nounWhat it showsTechnical nameRead from
StructuralThe GardenThe arrangement of every artifact in the system, laid out as a navigable two-axis diamond (layer × area + the concept axis). Cultivated, deliberate, walkable.The Diamond<tenant>.pipeline_graph
TemporalThe RiverThe flow of events through the system over time — builds, extracts, signal fires, scenario runs. Watched, not arranged. Eddies and currents.The Pulse_<tenant>.pulse_event_log
(future)The Chapel(placeholder — a small focused view, e.g. a guided narrative tour)
(future)The Workshop(placeholder — where artifacts are crafted, possibly the Atelier surface)

The separation is semantic. The Garden answers “how is the system arranged?” — structural, durable, contemplated. The River answers “what is the system doing right now?” — temporal, dynamic, sampled. They’re complementary: the same data wears different faces depending on which question the analyst brings.

The technical names stay. The Diamond is the geometric arrangement Sense 42 has been about since the start; the Pulse is the heartbeat substrate of Sense 47. Internal references (pipeline_graph table, bake_pulse_data function, pulse_event_log schema) keep their precision. The landscape vocabulary (Garden, River, Landscape) is the navigational layer — what the analyst sees in the sidebar, how the surface is spoken about, what URLs read.

In JinDesk:

  • /landscape — the vantage page. Shows the schema of the landscape, links into each feature.
  • /garden — the Garden view (the diamond diagram, formerly /system/pipeline).
  • /river — the River view (the pulse stream, formerly /system/pulse).

The note embed for a saved Garden view is [[garden:<base64>]] (formerly [[pipes:<base64>]]). The PipelineView contract in lib/pipeline/view.ts keeps its name (it captures a Garden snapshot); the user-facing share button says “Share view”, the link reads “open in the garden ↗”.


River and Garden — the operational meaning

Section titled “River and Garden — the operational meaning”

The vocabulary section above defines the River and the Garden navigationally: two views the analyst uses to see the system. That is the original Sense 42 meaning, and it stays intact. But the same nouns carry a second, operational meaning that turns out to be load-bearing for how we talk about updates, cadence, and the seam between data and analytics. Both meanings coexist deliberately — the elegance of the metaphor is that the place you look at and the place work happens are the same place.

Operationally, the River is where data moves and gets processed. The Garden is where insights are cultivated by tending — with tools, reference material, human markers, and everyday user activity.

When something in a jinflow tenant is updated, the update lands in one of six categories. Each has a natural cadence, and each maps to a specific part of the landscape metaphor.

#MetaphorWhat actually changesWhere it livesNatural cadence
1Water flowingData content — new / updated source rowsRiver (contents)Continuous, source-driven
2River-processingPipeline logic — Bronze / Silver / Gold rules, extractors, contractsRiver (how it processes)Rare, deliberate
3Garden toolsData-instruments — signals, perspectives, theses, verdictsGarden (what shapes plants)Occasional, curator-driven
4Gardener’s field guideKnowledge — Subject Matter (Statement / Check), DossiersGarden (reference material)Rare, pack-sourced
5Garden-tendingSystem-instruments — notes, bookmarks, recents, Bell entriesGarden (supports human use)Continuous, user-driven
6Person-planted markersObservations — signed human notices about this tenant (Sense 15)Garden (attributed entries)Event-driven, human-paced

The plants growing in the Garden are the insights: findings, verdicts, thesis outputs, perspective scores. They are what we foster and what the Garden yields. The six categories above are the conditions and interventions that shape which plants grow.

How the categories map to the three-inputs-to-Understanding

Section titled “How the categories map to the three-inputs-to-Understanding”

The three feeders in The jinflow — Observations, Knowledge, Data — correspond exactly to three of the operational categories:

  • Data stream = categories 1 + 2 + 3 (water flowing + river-processing + garden tools). The River delivers Gold; garden tools read Gold and produce insight-plants.
  • Knowledge stream = category 4 (gardener’s field guide). Not a tool that produces things — reference the gardener consults while using tools.
  • Observations stream = category 6 (person-planted markers). Not machine output — attributed, dated, bounded human notices.

Category 5 (garden-tending) is orthogonal. Notes, bookmarks, recents, and Bell entries support human use of the Garden — remembering where you were, marking what to come back to, being told when something has fired. They don’t contribute to insight generation directly; they support the humans who do.

This is the vocabulary that carries Sense 54 — The Rhythm (cadence as a first-class axis). Each of the six categories has its own natural update frequency:

  • Data content updates on the source’s rhythm — could be twice a year, could be by the minute.
  • River-processing updates rarely — a curator deliberately reshaping the pipeline.
  • Garden tools evolve occasionally — a new signal drafted, a thesis refined.
  • The gardener’s field guide grows over years — accumulated domain expertise codified into Subject Matter.
  • Garden-tending accumulates continuously — every note taken, every bookmark placed.
  • Person-planted markers appear at their own pace — whenever someone signs an observation.

Treating all six as one “make” cycle is the current status quo and it works. But naming them separately makes it possible to schedule them separately — refresh categories 1 + 5 hourly, categories 3 + 4 + 6 daily, category 2 as-needed. That is the concrete deliverable of Sense 54.

An earlier draft of this section spoke of “riverbed interventions” for category 2. That’s wrong: the riverbed is passive terrain, but pipeline logic is active machinery. When you tune a Silver join or add a Gold column, you’re changing how the river processes what flows through it, not the shape of its banks. The right frame is river-processing (the works along the river), not riverbed (the ground it runs over).


Sense 17 — The Cathedral — proposed an animated 3D pipeline tour: a guided camera rail through a virtual nave, eight scripted stops, progressive reveal, warm atmosphere. It was retired in May 2026 because:

  • 3D in a browser is heavy. Each tenant load paid a meaningful first-paint cost just to render a few hundred boxes.
  • The scripted tour was a one-way ride. Users wanted to navigate, not be driven.
  • The 3D-ness itself didn’t carry insight. Once you’d seen the nave once, the camera rail had nothing new to say about your second tenant.

The lesson: the system is worth showing, but show it as a place to walk, not a ride to take. A 2D diagram a user can pan, zoom, prune, drag, and save is the right shape — that’s the Garden. The retired Cathedral is preserved as a reference (docs/design/sense_17_the_cathedral.md); the idea of a guided tour could return as a future Landscape feature (the Chapel placeholder above), but only if it earns its place alongside the Garden and the River.


jinflow’s analytical surface has a canonical chain:

Signal → Perspective → Thesis → Verdict → Lens
(observation) (composite) (claim) (conclusion) (consumption)

Each step transforms what came before. A signal observes; a perspective composites several signals into a single reading; a thesis turns that reading into a claim about the institution; a verdict turns the claim into a recommended action; a lens reframes one part of the picture for a specific audience.

Around that chain orbits the world the analytics are about: the CSVs flow into Bronze, Bronze refines to Silver, Silver lifts to Gold. And around the chain are the human artifacts that wrap it: Subject Matter, Dossiers, Notebooks, Reports.

The Garden makes that whole arrangement visible at once. A user can read the institution’s analytical posture by looking at the diagram. They can isolate the gold layer to see what entities exist. They can isolate the signal layer to see what’s being watched. They can fold everything together and trace one signal’s lineage from CSV to verdict.

The diagram is the institution’s analytical anatomy, drawn so a non-author can read it.


The five-step diamond gets a distinct hue family per step, so the eye can read the chain at a glance:

StepHueHexShapeWhy
Signalblue#3b82f6diamondcalm, observational, single-point
Perspectivecyan#06b6d4octagoncomposite — more sides than a diamond
Thesispurple#a855f7hexagonclaim — geometric, settled
Verdictamber#d97706starconclusion — warm, “star treatment”
Lenspink#ec4899round-diamondconsumption — playful, framed

The cool-to-warm gradient mirrors the chain’s direction of commitment. A signal is information. A verdict is a decision. The colour temperature carries that.

This replaced a previous palette of four shades of blue-purple — #6b9fd4 → #8b8fd4 → #a78bce → #b99ad4 — that read as one colour at zoomed-out distances. The geometry alone (diamond/octagon/hexagon/star) was doing all the work; the eye had to focus to tell the steps apart. The 2026-05-22 palette refresh fixed that.

The non-diamond layers (CSV, Bronze, Silver, Gold, Subject Matter, Dossier, Notebook, Report) keep their own colours, distinct from the chain. The whole-garden view stays readable because no two adjacent layers in the canonical order share a hue family.


Edges aren’t decoration. There are five kinds of relationship, and the diagram names each one:

TypeStyleStory
refsolid slatedbt model uses another model
ingestiondashed tealraw CSV feeds a bronze table
compiles_todotted pinkYAML compiles into SQL
generatesdotted rosePython script writes a CSV
servessolid amberbackend serves a frontend route

A user who reads the diagram learns five facts at once: not just that artifacts connect, but how. A dashed teal line between a CSV and a Bronze table tells a different story from a dotted pink line between a YAML and a SQL model. The first is “data crossed a trust boundary”; the second is “an authored declaration was rendered into executable form”.

The diagram surfaces this through:

  • Distinct visual styles for each edge type (different colour, different stroke pattern).
  • A sidebar legend that names each style so the user can map symbol to meaning without guessing.
  • A hover tooltip that names the edge type, its plain-English meaning, and the source/target nodes — no need to memorise.

Before this, every edge was solid grey. Four meanings collapsed into one symbol.


Layouts that respect the shape of what’s visible

Section titled “Layouts that respect the shape of what’s visible”

The diagram has one global layout problem and several local layout problems. The Garden solves them per situation:

Custom layered-block layout. Each visible layer becomes its own rectangular tile, with nodes packed into a grid sized to make the block roughly square. Tiles stack vertically by layer order (CSV at top, Frontend at bottom). Edges connect across tiers naturally.

Why not cytoscape’s breadthfirst? Because each tier has dozens of nodes, and breadthfirst puts every tier on a single horizontal row. The result is a 6-row × 600-node pancake that no human can read. Tiles let each tier breathe in a rectangular block, and the eye gets to see both the tier’s shape and the cross-tier flow.

(For example: Gold, where Persons sits in the middle and most other entities reference it.)

cose (force-directed) for the connected component, plus a tidy grid for the orphans below. Connected nodes cluster organically around their attractors; disconnected nodes sit in a separate, label-aware grid that doesn’t overlap.

Pure cose. Let the force-direction draw the topology.

Pure grid, width-aware so labels don’t collide.

For any view with fewer than ~50 visible nodes, the layout post-step clamps the zoom level to ≥1.0. Otherwise cy.fit() zooms out far enough to leave every node tiny. Below 50 nodes, readability matters more than fitting the whole canvas.


There are three places a layout can come from. The Garden resolves them in priority order:

  1. User-saved layout (localStorage, keyed per (tenant, visible-layer set)). The user dragged nodes into an arrangement that tells the story they want and clicked Save. The saved positions survive page reloads and have the strongest claim.
  2. In-memory cache (module-scope Map). Computed by the most recent auto-layout, valid only if the visible-layer set still matches the cache’s filter key. Avoids paying the layout cost a second time for the same view.
  3. Fresh auto-layout (the strategies above). Used when neither saved nor cached positions match.

The cache key includes the filter signature on purpose. Earlier, switching from a one-layer view to an all-layers view would silently restore positions computed for the one-layer view, leaving hidden nodes scattered at wherever they last sat. The filter signature kills that bug.

The localStorage key includes the filter state on purpose too. A user can curate a Gold view and an All view independently — they don’t overwrite each other.


The diagram is interactive, and the gestures are composable:

  • Click a node — replaces selection, shows upstream and downstream lineage in coloured haloes.
  • Shift-click a node — adds to selection without triggering lineage. Used when building a multi-selection.
  • ⌘-drag / Ctrl-drag — marquee selection box.
  • Drag any selected node — moves the whole group together.
  • Click empty canvas — clears everything.

Multi-selection is the gesture that lets a user curate. Grab a cluster, move it, rotate the institutional story.


The Landscape is the pipeline-shape view — the analytical machinery seen from above. JinDesk’s Landscape page renders Bronze → Silver → Gold → signals → theses → verdicts as an interactive map, with each node carrying its current state (row counts, last build, latest changes). Renamed from “The Pipes” in May 2026.

Status: in bloom. First views are live; deeper drill-down + provenance integration with Sense 18 The Ledger is forming.


Things the Garden does not yet do, and might want to:

Today, every node is its own atom. A user wanting to see “all signals about workforce visibility” has to mentally filter. Could the diagram cluster nodes by signal_category (or theis_id, or dossier_id) into compound nodes that expand on click? cytoscape supports compound nodes; the data model would need a clustering rule per layer.

A saved layout lives in localStorage today. The natural next step is to make it transportable:

MechanismLiftShareable acrossBest for
Clipboard copy/pastetinySlack DM, email”Hey check this view I made”
URL fragmenttinyAnyone who can open the URLPermanent reference in docs
Embed in a NotemediumAnyone who pulls the tenant AFSCurated views with prose: “the Pernet succession map”

The Note path is the jinflow-grade answer. A note carrying a garden_layout: block becomes a curated lens onto the Garden, versioned with the tenant. The Notebook becomes a place where institutional knowledge accumulates alongside the maps that tell its stories. Sense 42 considers this in scope; implementation is for a future iteration.

Edge labels on the canvas (already opt-in)

Section titled “Edge labels on the canvas (already opt-in)”

The current toggle shows edge type as text on the line. It’s right for single-layer views and wrong for the whole-garden view (too cluttered). The next question: can the diagram auto-decide when labels help — say, when the visible edge count is below some threshold?

The signal layer in particular benefits from grouping. workforce-visibility signals belong together; financial signals belong together. Today they’re alphabetical inside the layer-block. A secondary-sort by category would group them visually. Cheap, and the YAMLs already carry the category.

When a single layer is isolated, the canvas currently shows just the nodes. A banner naming the layer (and its node count, and a one-line description from layerConfig) would orient the reader. Lift: small.

A non-3D successor to the Cathedral: animate the camera (i.e., pan/zoom transitions) along the upstream/downstream lineage of a selected node. Pause briefly at each ancestor. Could be triggered by a “Walk the lineage” button on the selected-node panel. Not a guided tour — a self-directed lineage walk.

When the analyst is comparing two tenants’ gardens (numetrix.rmc vs numetrix.inspire, say), it would be useful to see one diagram with both stacks side by side and a delta colour. Out of scope for v1.


  • The 3D Cathedral (retired — see Sense 17 for archeology).
  • The Lens runtime (Sense 13.1) — though Lenses appear on the diagram, how a lens compiles to SQL is a different paper.
  • The analytical chain itself (signal → … → lens) — that’s Sense 14 (The Signal) and Sense 15 (The Strategy). The Garden paints them but doesn’t define them.
  • The instrument-graph (/[tenant]/instruments) — a related but distinct view. The Garden is artifact-shaped; the instrument graph is symbol-shaped. They could converge or stay split.

DateWhat
2026-04-12Pipeline Cathedral (3D) lands as Sense 17.
2026-04-??Cathedral retired — heavy first-paint, one-way tour, no analytical payoff.
2026-05-??2D pipeline page (/[tenant]/system/pipeline) lives with default cytoscape look.
2026-05-22The Pipes — palette refresh, layout strategies, persistence, multi-select, edge legend.
2026-05-27Pass 3 — jinflow_name regrouping; Hide/Dim toggle; Understanding overlay; Strategy/Progress surfaced as nodes; Identity & Governance compound summaries.
2026-05-29Pass 4 — sidebar reduced to 8 collapsible areas; node-count rollup; OPERATIONS section; platform layer retired.
2026-05-29Pass 5 — compound-parent area clouds; diamond-radial layout (Analysis north, Strategy west, Understanding centre, Expertise east, Progress south); dynamic sizing/centroids.

Future iterations will keep adding to this list. The Sense is the worldview these iterations should be consistent with.


Pass 6 spec — Edge bundling (zoom-aware area edges, 2026-05-31)

Section titled “Pass 6 spec — Edge bundling (zoom-aware area edges, 2026-05-31)”

Status: Spec. Code follows on review.

The pain. The diamond now reads cleanly at the area level (clouds + labels), but the canvas at default zoom is full of arrows. A single Analysis → Progress path between two big areas can be tens of edges (gold → signal → perspective → thesis → verdict), and similar bundles run every cross-area direction. The eye can’t track which area-pair is busy and which is quiet — every line looks equally important.

What we want. At the diamond-reading zoom, the user should see one summary edge per (sourceArea, targetArea, edgeType), with thickness encoding count. When they zoom in past a threshold, the summary edges fade out and the individual ones fade in — the same data, looked at from closer.

area_edge_key = (source_area, target_area, edge_type)
  • source_area = parent(edge.source) ?? '_floating'. Nodes that have no area parent (the Understanding marker, any floating Subject-Matter nodes) collapse to a sentinel _floating area; their area edges still render but link to/from the relevant individual node, not a cluster.
  • target_area likewise.
  • edge_type ∈ {ref, ingestion, compiles_to, generates, serves} — already on edge.data.type.
  • Self-loops (source_area === target_area) are omitted from area edges. Intra-area connectivity is only meaningful in detail view; at the diamond zoom, “Analysis → Analysis” is noise. Hides the gold→signal→thesis chain inside Analysis until the user zooms in.
type AreaEdge = {
id: string; // `area-edge:${sourceArea}:${targetArea}:${type}`
source: string; // 'area:analysis' (or a free-floating node id)
target: string; // 'area:evolution'
type: EdgeType;
count: number; // how many individual edges this represents
sampleEdgeIds: string[]; // first N for hover/click drill-in
};

Computed once per (filter-state, hidden-layers) snapshot — no per-frame work.

Linear gets crushed at the top end; log keeps small bundles visible without the big bundles dominating the canvas.

width = clamp(2 + Math.log2(count) * 1.5, 2, 14) // px
  • 1 edge → 2 px (a real line, not invisible)
  • 4 edges → 5 px
  • 16 edges → 8 px
  • 64 edges → 11 px
  • 256 edges → 14 px (capped)

Edge colour matches the per-type palette already in the Sense (“Five edges, five stories”). Label = count at low zoom only.

State machine, hysteretic to prevent flicker at the threshold:

area edges individual edges
zoom < 0.6 ▶▶▶ visible hidden
0.6 ≤ zoom < 1.2 transition (both visible, fading inversely)
zoom ≥ 1.2 ▶▶▶ hidden visible

Implementation: cytoscape mappers on opacity driven by cy.zoom(), with a debounced listener on cy.on('zoom', …) (16 ms) so we don’t restyle on every wheel tick.

Modes:

  • auto (default) — zoom drives the swap.
  • area_only — pin area edges; individual edges stay hidden at any zoom. For screenshots, conversations.
  • individual_only — the pre-Pass-6 behaviour. For users who want the old view back.

Mode persists in localStorage alongside hiddenLayers, expandedAreas, etc. (existing storage key jinflow:pipeline:config, bump DEFAULTS_VERSION to 4).

Edge-type toggles still apply. If ref is filtered off, no ref area edges appear and no ref individual edges appear — the filter cuts before aggregation. Same for the existing layer-hidden / node-hidden machinery: aggregation runs on the visible subset.

Trace-mode highlighting (upstream / downstream from a selected node) only applies in individual_only or above-the-threshold zoom — area edges don’t carry lineage semantics. A node’s “lineage” through area edges is a fundamentally different abstraction we’re not committing to in Pass 6.

  • Hover area edge: tooltip → "<count> <type> edges from <Source Area> to <Target Area>".
  • Click area edge: highlight the individual edges underneath it (briefly switch to individual view, lineage-highlight those sampleEdgeIds, then revert). Cheap drill-in without a full zoom.

Aggregation runs at initGraph time and after any of: hiddenLayers change, hiddenNodes change, traceMode change. The full graph on rmc is ~300 nodes / ~500 edges, so aggregation is O(E) over a few hundred — sub-millisecond. We do not re-aggregate on zoom; zoom only swaps visibility.

Sample edge cap: store up to 8 sample IDs per area edge for hover/click; beyond that, we drop. Keeps the JSON small for any future copy/share use.

FileChange
+page.svelteNew computeAreaEdges(pipelineData, hiddenLayers, hiddenNodes) function. New edgeBundlingMode state. New cytoscape style block for edge[kind="area"] with opacity mappers. Hover/click handlers reuse the existing edge-tooltip + lineage-highlight code.
Cytoscape elementsThe element list now includes area edges alongside individual edges (parent compound nodes from Pass 5 already provide the source/target endpoints).
STORAGE_KEYDEFAULTS_VERSION bump 3 → 4; new key edgeBundlingMode.

Estimated diff: ~150 lines added, ~10 lines modified.

  • Geometric bundling (curve control points so parallel individual edges route together) — the option B from yesterday’s menu. Pass 6 doesn’t change individual-edge geometry; it just hides them until zoomed.
  • Hierarchical Edge Bundling (HEB, D3-style) — option E. Heavy lift; not native to cytoscape.
  • Area-level lineage (“show all upstream areas from Verdicts”) — different abstraction; not in Pass 6.
  • Edge labels on area edges other than the count — type colour + width carries the rest; label clutter risk.
  • Persisting the count history (sparklines on area edges showing “more activity this week”) — fun but speculative; needs a separate data source.

When the user zooms partway (e.g. zoom = 0.9), both area edges and individual edges are visible with crossfaded opacities. Is that a feature (smooth transition) or a bug (visual confusion)? Lean: feature for now, watch in use; if it feels muddled, narrow the transition band to ±0.1 around a single hard switch.


Pass 7 spec — Engaged artifacts move into Understanding (2026-06-01)

Section titled “Pass 7 spec — Engaged artifacts move into Understanding (2026-06-01)”

Status: Spec. Code follows on review.

The pain. The Understanding area is currently empty. No LAYER_CONFIG entry has jinflow_name: 'insight', so the centre cloud on the diamond is a topological landmark with no inhabitants. The sidebar shows Understanding 0. Meanwhile, the Understanding overlay (Pass 3d, yellow border) highlights finding + verdict nodes that live in Analysis — a visual contradiction: “these are Understanding” said by the overlay, “but they’re in Analysis” said by the cloud.

The cut the owner named in conversation (2026-06-01):

Active engaged signals belong to Understanding. They are manifesting the result of analytical work (which may consist of creating such signals).

Operationalised: the apparatus (defined signals/perspectives/theses/verdicts/their YAMLs and SQL) lives in Analysis. Their firings (the rows their findings tables hold) constitute Understanding. Same artifact, different state, different home.

Analysis (the apparatus)Understanding (the firings)
signal YAML defines revenue_leakagesignal_findings__signal_revenue_leakage table has ≥ 1 row
thesis YAML defines a business questionthesis_verdicts has a row at status ∈ {confirmed, plausible}
verdict YAML defines a root-cause ruleverdict_findings has ≥ 1 row (rule matched)
perspective YAML defines a lenssignal_findings__perspective_X has ≥ 1 row
gold_* tables (substrate)(stays Analysis)
signal_registry, thesis_registry, etc. (catalogs)(stays Analysis — metadata about the apparatus, not findings)

Diamond reading becomes:

  • Analysis = “what we built to look for”
  • Understanding = “what we found”

An unfired signal sits as a quiet diamond in Analysis. The same signal once it engages becomes a diamond in Understanding. Same color, same shape, different parent — the visual chain reads the change.

Per artifact type, what makes a node “engaged”:

Node typeEngagement check
signal (signal_findings__signal_*)SELECT count(*) FROM <table> > 0
perspective (signal_findings__perspective_*)same — any rows
thesis (thesis_verdicts filtered by thesis_id)row exists AND status IN ('confirmed', 'plausible'). not_observed and insufficient don’t count — the apparatus tried but didn’t engage.
verdict (verdict_findings filtered by verdict_id)any row — a row means the rule matched at least one finding
smebit (Level 1 — smebit_verdicts__*)any row at status IN ('confirmed', 'violated'). no_data doesn’t count.
gold, registry, silver, bronze, csv, all operational layersnever engaged — they’re substrate / catalog / wiring

The pipeline_graph node payload gains one field:

{
"id": "signal_findings__signal_revenue_leakage",
"layer": "signal",
"label": "RevenueLeakage",
"engaged": true // ← new
}

engaged defaults to false. The set of layers where engagement applies is fixed (see table above); all other layers carry engaged: false and the value is ignored.

Static LAYER_CONFIG[layer].jinflow_name stops being the final word. The graph generator computes the effective area per node:

def area_for(node):
base = LAYER_CONFIG[node.layer]["jinflow_name"]
if node.engaged and node.layer in ENGAGEMENT_LAYERS:
return "insight"
return base

ENGAGEMENT_LAYERS = {"signal", "perspective", "thesis", "verdict", "smebit"}.

JinDesk continues to read node.data('jinflow_name') as the single source of truth — it doesn’t need to know about engagement, just where the node belongs.

generate_pipeline_graph.py runs at the end of jin make, after the KLS is built. The KLS holds the findings tables. So the generator can query DuckDB for row counts per artifact at bake time. Each tenant’s _<tenant>.pipeline_graph table carries the per-tenant engagement state — so different tenants can have the same signal in different places (one tenant’s revenue_leakage is engaged → Understanding; another tenant’s hasn’t fired → Analysis). That’s correct and desired.

rmc cost estimate: ~42 signals + ~7 perspectives + ~13 theses + ~11 verdicts + ~15 smebits = ~88 tables to scan. Each is a single COUNT(*) against a small-to-medium table. Total bake-time cost: well under 2 seconds. Acceptable.

What happens to the Understanding overlay (Pass 3d)

Section titled “What happens to the Understanding overlay (Pass 3d)”

Currently the overlay puts a yellow border on signal-finding + verdict nodes regardless of where they live. After Pass 7, those nodes are in the Understanding cloud by area. The overlay becomes structurally redundant for them.

Decision: retire the Understanding overlay button. The diamond’s area membership now expresses Understanding; the overlay was a workaround for the cloud being empty. Removing the toggle simplifies the sidebar.

If a future need surfaces (“show me the lineage from Understanding back to its inputs”), that’s a different feature — a lineage trace, not a highlight overlay. Out of scope for Pass 7.

  • Same shape + color, different cloud. Engaged signals stay blue diamonds; they just live inside the Understanding cloud. The eye reads “this is a signal, and it’s in Understanding” naturally.
  • Understanding cloud now non-empty. Its size adapts via Pass 5.1’s dynamic centroid math — Understanding grows to wrap its inhabitants and pushes neighbours apart.
  • Cardinal vs centre. Pre-Pass-7 Understanding sat at canvas centre with no children. Post-Pass-7 Understanding has children, so its sizing follows the same computeAreaSize rules as the four cardinal areas. The diamond’s centre is still Understanding; the marker remains.
  • Engine: LAYER_CONFIG unchanged. area_for(node) is new. Graph generator queries the KLS for engagement. Existing per-tenant _<tenant>.pipeline_graph tables get re-baked on next jin make — node payloads gain engaged, parent compound IDs change for engaged nodes.
  • JinDesk: nothing needs to change. The parent field on each node already drives Pass-5 cloud membership; this just resolves to area:insight for engaged nodes instead of area:analysis.
  • Sidebar: Understanding stops being the empty 0 entry — it now has a real count. Strategy / Expertise / Progress / Analysis counts adjust accordingly (Analysis loses its engaged children to Understanding).
  • Understanding overlay: removed from sidebar + cytoscape style block + saveConfig key. Migration noted in DEFAULTS_VERSION → 5; older configs with insightOverlay: true silently drop the key.
FileChange
scripts/generate_pipeline_graph.pyAdd area_for(node) helper. Add compute_engagement(tenant_meta_schema, kls_path) that queries finding tables. Wire into the per-tenant bake loop. Add engaged to the node payload.
explorer/src/routes/[tenant]/garden/+page.svelte (was system/pipeline/+page.svelte)Remove Understanding overlay button + state + style block + saveConfig key. Understanding area’s tier stays 'name', treatment in sidebar unchanged (now shows a real count).
docs/design/sense_42_the_pipes.mdThis Pass 7 spec; a history-row entry; remove “Understanding overlay” from the open-questions / features list since it’s retired.
LAYER_CONFIG (generate_pipeline_graph.py)Unchanged. The base layer→area mapping stays; engagement is the override.

Estimated diff: ~120 lines in the graph generator (queries + area_for + tests), ~30 lines of UI removal in +page.svelte.

  • Per-severity gating (“only severity ≥ critical counts as engaged”). V1 is binary: any row counts. If a user finds the Understanding cloud cluttered with low-severity signal noise, severity filters are a follow-up — and the sidebar layer toggles already let them hide whole layers if they want.
  • Time-windowed engagement (“engaged in the last 30 days”). V1 is point-in-time: as of this jin make, does it have findings? Out of scope; needs a temporal column on findings tables.
  • Visual differentiation between barely-engaged and heavily-engaged. V1: in Understanding or in Analysis, binary. A future pass could weight by finding count (bigger node, brighter colour) — speculative.
  • Lineage trace from Understanding back to Analysis inputs. A future Pass — interesting but not what this one is doing.
  1. Perspective engagement criterion. A perspective with rows but all rows are noop / null severity — does it count? Lean: yes for V1 (any row = it produced output). Re-examine if noise.
  2. Smebit Level 0 (Statement). Statements have no check, no verdict — never engageable. Stays in Analysis as smebit layer? Or treated differently from Level 1 Checks? Lean: keep all smebit in Analysis unless engaged (which only Level 1 Checks can be).
  3. Bitbundles (Dossiers). Curated narrative groupings. Never “fire” — they don’t have a SQL behind them in the analytical sense. Stay in Expertise unchanged.

Pass 8 spec — Orthogonal layer × area visibility (2026-06-02)

Section titled “Pass 8 spec — Orthogonal layer × area visibility (2026-06-02)”

Status: Spec. Code follows on review.

The pain. After Pass 7, a single layer name (e.g. signal) can have nodes in two areas at once — engaged signals in Understanding, defined-but-quiet ones in Analysis. The sidebar reflects that accurately (signal: 35 under Understanding, signal: 6 under Analysis). But the visibility toggles still live on a single axis: hiddenLayers: Set<string> keyed by layer name. Toggling signal off hides every signal everywhere — there’s no way to say “hide quiet signals, keep engaged ones”, or “hide Analysis entirely, leave the diamond’s centre visible.”

The owner named the fix in conversation (2026-06-02): make layer and area visibility completely orthogonal. A node’s layer is one independent axis; its area is another. The visible set is the intersection of “layer on” AND “area on”.

Replace the single hiddenLayers axis with two independent sets:

let hiddenLayers: Set<string> = $state(new Set()); // e.g. {'signal', 'gold'}
let hiddenAreas: Set<string> = $state(new Set()); // NEW: e.g. {'analysis', 'expertise'}

The visibility rule:

function isNodeVisible(node) {
const layer = node.data('layer');
const area = node.data('jinflow_name') ?? layerConfig[layer]?.jinflow_name;
return !hiddenLayers.has(layer) && !hiddenAreas.has(area);
}

Edges inherit from their endpoints (existing logic): an edge is hidden when either endpoint is hidden, regardless of which axis hid it.

  • “Hide Analysis” — one click on the Analysis area toggle. All Gold, all unengaged signals/theses/etc. disappear. Engaged ones in Understanding stay. Strategy / Expertise / Progress / Understanding / Ingestion / Infrastructure / Identity stay.
  • “Hide signal” — one click on the signal layer toggle. Every signal-shaped node disappears, regardless of which area it lives in. Other Analysis nodes (gold, theses, verdicts, perspectives) stay.
  • “Show only Understanding” — toggle every other area off. The centre cloud remains.
  • “Show signals only, anywhere they live” — turn off every other layer. Engaged signals in Understanding + unengaged signals in Analysis stay; nothing else.

The Pass 4 sidebar already has two visual rows that fit perfectly:

  • Area row (e.g. “ANALYSIS 130”) — has the chevron + label + count. Today the label click toggles all layers in that area; the spec moves it to toggle the area axis.
  • Layer row inside an expanded area (e.g. “Gold 51”) — toggles the layer axis.
Click targetWhat it toggles
Area label (“ANALYSIS”)hiddenAreas membership for analysis
Layer row inside the expansion (“Gold 51”)hiddenLayers membership for gold
Chevron next to area labelExpansion state only (unchanged)
Master row — “All Areas: Select / Deselect”Flips the area axis only
Master row — “All Layers: Select / Deselect”Flips the layer axis only

Per the owner’s review (2026-06-02): offer both master toggles, not one. Each axis gets its own Select All / Deselect All. Two small rows at the top of the sidebar’s Areas section, visually paired but independently operated. Users who want “show everything” need both buttons; one click each. “Hide everything via areas” and “hide everything via layers” produce visually identical empty canvases but are semantically distinct — restoring from each takes one click on the matching master.

The two axes are visually separate: each row’s on/off reflects only its own axis. A layer can be “on” by its own button but still hidden because its area is off. That’s fine — the combined state is what produces the canvas; the per-row UI reflects only what that row controls.

Decided per the owner’s review: layer toggle visuals reflect only the layer’s own state, not the intersection. Keeps the on/off concept simple. Users who want to know “is this layer actually visible right now?” read the canvas.

Visual cue when a layer is on but its area is off

Section titled “Visual cue when a layer is on but its area is off”

A layer toggle that’s “on” while its area is “off” looks identical to “on” today (full opacity in the sidebar). That’s deliberate — the layer button represents the layer state, not the combined visibility. The canvas tells the truth.

If a future need surfaces (“highlight layers whose effective visibility is suppressed by area state”), that’s a follow-up. Out of scope for Pass 8.

  • hiddenAreas defaults to empty — every area starts visible. Same as today’s behaviour.
  • hiddenLayers keeps its existing DEFAULT_HIDDEN set — same as Pass 5.3 + Pass 7 list (platform retired, debug/lineage/metrics/frontend + Sense-25 identity layers hidden by default).
  • No new DEFAULT_HIDDEN_AREAS constant — leave the area axis open.

Saved config gains one new field; bump DEFAULTS_VERSION 5 → 6:

localStorage[STORAGE_KEY] = {
defaultsVersion: 6,
hiddenLayers: [...hiddenLayers],
hiddenAreas: [...hiddenAreas], // ← new
expandedAreas: [...expandedAreas],
edgeBundlingMode,
// ...
};

Migration from v5: a missing hiddenAreas field defaults to empty (no areas hidden) — pure additive change, no user state lost. v5 → v6 doesn’t trigger the DEFAULT_HIDDEN merge pass (no new layer defaults).

FileChange
+page.svelteAdd hiddenAreas state. Add toggleArea(name) function. Modify applyLayerVisibility to check BOTH axes when deciding .layer-hidden. Modify isGroupOn to read hiddenAreas instead of “every layer in group is hidden”. Modify the area-master-toggle in the sidebar (AREAS row) to call toggleArea(name) instead of toggleGroup(group). Keep the per-layer rows inside the expansion calling toggleLayer(layer) — unchanged.
saveConfig / loadConfigRead/write hiddenAreas. Bump DEFAULTS_VERSION to 6.
Master toggles (the “Select All / Deselect All” at top)Two pairs now: keep toggleAllLayers (flips layer axis) AND add toggleAllAreas (flips area axis). Two small rows visually paired, each axis independent.

Estimated diff: ~80 lines, mostly state + one new function + UI wiring on the existing area-row click handler.

  • Engine: nothing changes. pipeline_graph payload is unchanged.
  • JinDesk: existing saved configs (v5) load with hiddenAreas = new Set() (all areas visible). On the next sidebar click that changes either set, the v6 schema is written. No user state is lost; no defaults change.
  • The Pass 7 demonstration “Understanding has 73 nodes” remains true on first paint; users now also have the option to hide entire areas if they want a focused view.
  • Per-area-per-layer hiding (e.g. “hide engaged signals but keep unengaged ones”). With orthogonal axes you can hide all signals or hide all of Analysis — but you can’t hide “engaged signals only” without also turning off the Understanding area. That’s a corner case; the canvas already encodes engagement structurally, so the use case is rare. Composite keys would be the future shape if it arises.
  • DEFAULT_HIDDEN_AREAS — speculative; not needed at V1. The current DEFAULT_HIDDEN layer set already produces a sensible first paint.
  • Visual cue for layers suppressed by area state — see “Visual cue” subsection above; deferred until a real need surfaces.
  • Master toggle for the OPERATIONS section — same shape as AREAS; could be added if asked. V1 keeps the existing “Select All / Deselect All” only on AREAS.
  1. Should toggleAllLayers be renamed? Resolved 2026-06-02 — offer both master toggles, not one. toggleAllLayers keeps its name + behaviour (flips layer axis); new toggleAllAreas flips the area axis. Two small paired rows at the top of the AREAS section.
  2. Should area-master toggles auto-hide their expansion? Resolved 2026-06-02 — no auto-collapse. Expansion is independent UI state; hiding an area doesn’t change whether you’ve twirled it open.
  3. Edge bundling under orthogonal hiding. Pass 6’s aggregation already cuts on .layer-hidden. When an area is hidden, all its children get .layer-hidden. Area edges to/from the hidden area drop naturally. Verified by inspection of computeAreaEdges; no Pass-6 change needed.

Pass 9 spec — Semantic clustering (2026-06-05)

Section titled “Pass 9 spec — Semantic clustering (2026-06-05)”

Status: Spec, open questions walked + resolved 2026-06-05. Code follows.

The pain. After Passes 5–8 the diamond reads cleanly on the two axes the pack author lays out: layer (what kind of artifact) and area (where it sits in the five-name model). But many of the analyst’s mental questions cut across both axes:

  • “Show me everything that touches material_id.”
  • “Show me the whole case chain — bronze through verdict.”
  • “Which artifacts contribute to the financial-anomaly thesis?”
  • “This SMEbit is anchored to four things; light them up.”

These aren’t layer questions or area questions — they’re concept questions. The same node can belong to several concept chains at once (a gold view typically touches three or four). Trying to answer them today means scanning the canvas by name, or building ad-hoc filter sets that don’t persist.

A new third axis: concepts. Each node optionally carries a list of concept IDs it touches:

// added to the existing PipelineNode payload
concepts?: string[]; // e.g. ['material_id', 'case', 'fiscal_year']

A new sidebar state mirrors the existing layer / area axes:

let visibleConcepts: Set<string> = $state(new Set()); // empty = all concepts shown
let conceptComposeMode: 'union' | 'intersection' = $state('union');

When visibleConcepts is empty, the page behaves exactly as today (the concept axis is “off”). When one or more concepts are selected, the visibility rule composes with the existing two-axis intersection:

function isNodeVisible(node) {
const layer = node.data('layer');
const area = node.data('jinflow_name') ?? layerConfig[layer]?.jinflow_name;
const concepts = node.data('concepts') ?? [];
if (hiddenLayers.has(layer)) return false;
if (hiddenAreas.has(area)) return false;
if (visibleConcepts.size === 0) return true; // axis is off
if (conceptComposeMode === 'union') {
return concepts.some((c) => visibleConcepts.has(c));
} else {
return [...visibleConcepts].every((c) => concepts.includes(c));
}
}

Three axes, all independent, all set-typed, all URL-driven via the share-view encoding (Pass 8 + thread 4).

Concept discovery — the diagram teaches the author

Section titled “Concept discovery — the diagram teaches the author”

Concepts are auto-discovered by generate_pipeline_graph.py — not hand-declared. The pack author opens the Garden and sees what’s actually flowing through it, instead of having to know what to declare upfront. Mig’s framing (2026-06-05, when the surface was still called “the pipes”): “the author should LEARN from the pipes.”

The discovery walk:

  1. Scan every node’s underlying content for column / field references:

    • SQL models: column references in SELECT lists, WHERE / JOIN clauses, GROUP BY.
    • Signals / theses / verdicts / perspectives: entity_field references in the YAML.
    • SMEbits handled separately (see “SUBJECT MATTER section” below).
  2. For each unique reference, count how many distinct nodes touch it.

  3. Apply two filters to cut noise:

    • Stop list — ubiquitous columns that aren’t interesting as flow concepts: id, created_at, updated_at, _loaded_at, _dbt_inserted_at, surrogate-key fragments from dbt_utils.generate_surrogate_key. Engine ships a default; packs can extend via concepts/stop_list.yaml.
    • Minimum threshold — a reference must appear in at least N distinct nodes to surface as a concept (default N=3; tunable per pack). A column read once is not a chain.
  4. The survivors are the auto-discovered concepts. Each one’s membership is the set of nodes that reference it.

Optional pack annotations. Auto-discovered concepts use their raw column name as the display label by default. A pack can OPTIONALLY decorate any concept with a friendlier presentation:

# <pack>/concepts/material_id.yaml — entirely optional
display_name:
en: Material chain
de: Materialkette
fr: Chaîne de matériel
description:
en: The flow of a single material code through bronze, silver, gold,
pricing, signals, and the verdicts that depend on it.

No annotation = the concept still surfaces, labelled with its raw name (material_id). The annotation is a polish layer on top of auto-discovery, not a gate that controls visibility.

Output lands in the node’s concepts: string[] field, baked into pipeline_graph alongside the existing topology. The concept registry (display name + member count + any annotations) lands in a sibling field pipelineData.concepts: ConceptDescriptor[].

Three modes the analyst can compose:

FilterReads as
?chains=material_id”the material chain” — every node that touches a material code, top to bottom of the diamond
?chains=material_id,case_id&mode=intersection”where material flow meets case context” — the join surface only
?chains=material_id + hiddenAreas=evolution”the material chain up to but not including the suggestion engine”

Beyond filtering, the chain becomes a navigable path:

  • A “Trace chain” button on each cluster pans + zooms to the cluster’s bounding box.
  • The cluster’s nodes can be highlighted with the concept’s declared colour (a thin coloured ring around each), independent of the layer palette.
  • On hover of a single node, a chip surfaces the concepts that node carries — “this one touches material_id + case_id + fiscal_year” — answering the second-most-common analyst question without leaving the canvas.

Two new sections above the existing AREAS / OPERATIONS, in this order:

  1. CHAINS — auto-discovered column-flow concepts. First row is a compose-mode toggle (union ⇄ intersection); each subsequent row is one concept with its display name (annotated or raw column name) + member count. A single running visible-count line under the section header (e.g. “43 of 142 visible”) updates as the selection composes.

  2. SUBJECT MATTER — SMEbit anchors: blocks as hand-authored clusters, surfaced separately because they’re a different mental model (an explanation cluster — “everything this SMEbit explains” — not a flow cluster). Same interaction model: click to toggle, contributes to the same compose mode. Each row shows the SMEbit’s display name + member count.

Both sections share the same visibleConcepts state and the same compose mode toggle — a node visible under the union of (selected CHAINS) ∪ (selected SUBJECT MATTER) is shown, others dim to 30%.

Visual treatment on the canvas: highlight + dim only. Cluster nodes keep their layer colour at full opacity; non-cluster nodes drop to 30% opacity. No coloured rings, no layer-colour replacement at V1 — the layer palette keeps its meaning, cluster identity rides entirely on the sidebar swatch + the dim/highlight cut. Coloured-ring overlays remain a possible Phase 2 refinement if the dim/highlight cut feels too subtle in practice.

Estimated diff (JinDesk side): ~180 lines split across +page.svelte (state + visibility rule + two sidebar sections + running count) and a small concepts.ts helper for the descriptor + auto-discovery merge.

Estimated diff (engine side): one new file (scripts/concepts.py) for the auto-discovery walk + stop-list resolution, plus a ~50-line addition to generate_pipeline_graph.py for the concepts: string[] field on every node payload and the pipelineData.concepts registry on the graph_json output.

?chains=<concept_id>[,<concept_id>...]
&chain_mode=union|intersection

Composed into the existing share-view encoding (PipelineView) so a snapshot embedded in a note carries the cluster filter too. Backward-compatible: views authored before Pass 9 omit visibleConcepts and conceptComposeMode, decoding as empty + ‘union’ (the no-op default).

  • Concept hierarchies (a concept that contains sub-concepts, like case → case_admission → case_discharge). Possible later; not needed at V1 — overlapping flat clusters cover the common cases.
  • Cross-tenant concepts in the AFS cascade (a tenant overriding a pack concept by ID). The cascade rule already exists at Sense 43 level; once L4 cascade extends to concepts, this falls out. V1: pack-level only.
  • Live concept extraction at request time. The membership is baked at make time and read at render time — no parsing in the browser, no Python sidecar at request time.
  • Concept-to-concept relationships (e.g. “the material_id chain feeds the pricing chain”). Could become a meta-graph later; for V1 each concept is independent.
  1. Auto-detect vs hand-only at V1? Auto-discover both concepts AND members. Mig flagged the inverted premise — hand-only assumes the author already has the mental model the diagram should be building for them. The page teaches the author what’s flowing; auto-discovery is the foundation, optional pack annotations are polish on top. Noise is managed by a stop list + minimum-threshold filter, not by hand-authored exclusion. (Discovery walk + filters spec’d above.)
  2. What’s the canonical concept set per pack? Dissolved by Q1’s resolution. There is no canonical set. Packs may ship optional display annotations for any concept they want to polish; the list itself emerges from the graph at make time.
  3. Compose mode default — union or intersection? Union. Matches the analyst’s first-click instinct (expanding the highlighted set). Intersection is one click away via a sidebar toggle when the boundary question is the right question.
  4. Concept colour vs layer colour. Highlight + dim only at V1. The layer palette keeps its meaning. Cluster identity rides on sidebar swatch + the opacity cut. Coloured ring overlays remain a Phase 2 option if the dim/highlight cut feels too subtle.
  5. Should clusters appear as compound parents? No at V1. The diamond’s geometric meaning is layer × area; re-layouting on cluster selection would distort that. Clusters are an overlay on the existing geometry.
  6. SMEbit anchors → automatic cluster? Yes at V1 — but in a separate sidebar section. Both kinds ship together; CHAINS holds auto-discovered column-flow concepts, SUBJECT MATTER holds SMEbit-anchor clusters. Both share the same visibleConcepts state and compose mode, but the mental distinction (flow vs explanation) gets its own header so it doesn’t get collapsed.
  7. Running visible-count under CHAINS? Yes. A single line under each section header showing the running visible count as the selection composes — answers “did my last toggle change what I’m looking at” without forcing the analyst to count.

Three axes. The diagram reads three ways. The analyst’s three questions all have a home.


Numerical neighbors:Sense 41: The Relevance — Which Rings Are For You · Sense 43: The Cascade — Configuration as Layered Authority

jazzisnow jinflow is a jazzisnow product
v0.64.7 · built 2026-09-20 19:48 UTC