Sense 27: The Funnel — Narrow at the Source

Sense 27 · Steady · Last touched 2026-08-20

last_verified: 2026-07-27

Synced from docs/design/sense_27_the_funnel.md in the engine repo — that’s the source; this page is a build-time mirror.

The cloud should never hold the dataset.

It should hold the picture of the dataset.

Status: proposed Date: 2026-05-04 Author: the owner + Claude (conversation, post-OOM Material Map)

The Problem, Concretely

On 2026-05-04 the proxy-mode JinDesk (jinflow-proxy.fly.dev, 512MB) was OOM-killed mid-render of /numetrix.rmc/map. The sidebar populated, the page rendered, then a single Material Map loader pulled enough rows over the Cloudflare Tunnel to exhaust the cloud machine’s memory. Both Fly machines restarted. The in-memory proxy registry vanished. Every subsequent request 404’d until the local tunnel re-registered.

The bug isn’t proxy-specific. It is the shape of every loader in JinDesk today:

DuckDB  →  cloud Node  →  JSON.parse → array of objects → SSR render → HTML+payload
              ↑
         where the bill is paid

Replace “DuckDB” with the local KLS, R2 KLS, or remote tunnel — the middle hop is the same. The cloud machine is acting as a buffer between data and pixels. Buffers grow with the dataset. Tenants grow. Hospitals grow. The buffer wins.

This will keep happening on this-is.jinflow.io, on numetrix.jinflow.io, on hrcentral.jinflow.io, and on every P2P2P session — not just on the page that happened to OOM today.

It is a structural defect. We should fix it structurally.

The Principle

Data must narrow to display granularity at the source.

A heatmap with 500 visible cells does not need 50,000 rows. A histogram with 30 buckets does not need 50,000 rows. A timeline with one point per day does not need 50,000 rows. The number of rows that leaves the database should equal the number of marks the user will actually see — plus a small constant for hover detail and drill-down handles.

If a panel needs all the rows, it is not a panel. It is a table. Tables paginate.

This is the funnel: the dataset is wide at the bottom, narrows through aggregation as it leaves the database, and arrives at the cloud already at display resolution. The cloud machine becomes a thin pass-through. Memory pressure becomes proportional to the picture, not the dataset.

The Lens — naming the unit of narrowing

The Funnel is a principle. The unit it operates on is a Lens.

A Lens is a parameterized projection onto a dataset that declares:

Picture shape — how many marks, what kind (cells, points, nodes, rows). The visible result.
Narrowing recipe — composed steps drawn from three families: sample (USING SAMPLE N), aggregate (GROUP BY to N bins / ntile / time bucket), select (ORDER BY ... LIMIT N). Real-world Lenses chain them, e.g. sample 5,000 rows → bin to 500 cells → keep top 50 by count.
Drill-in tree — clicking any feature produces either a tabular drill-down (the 600 rows in this cell, paginated) or a sub-Lens (a tighter Map / Lab / Salon over those rows). Both are first-class. The sub-Lens path is what makes drill-in spectacular: Lenses all the way down, each narrower than the last, until a single row is the only thing left.

Map, Salon, and Lab are renderers of Lenses, not separate query worlds. Each consumes the same query handle and presents it in its own grammar:

A Map renders a Lens as space — heatmap cells, scatter points, timeline buckets. Auto-detected from axis types.
A Salon renders a Lens as a curated walk — selected rows, in order, with narrative.
A Lab renders a Lens as a search — start wide, narrow on filter, drill to a row.

Form-follows-Data still picks the renderer from the data shape. The Lens is what they all hold.

Where a Lens lives

A Lens is a first-class AFS artifact, like a signal or a thesis:

afs/lenses/
  lens_material_spend_heatmap.yaml
  lens_signal_severity_strip.yaml
  ...

A page declares which Lenses fill its component slots:

# page.yaml (sketch)
components:
  map:    { lens: lens_material_spend_heatmap }
  salon:  { lens: lens_top_anomalies_walk }
  lab:    { lens: lens_price_explorer }

Sub-Lenses reference each other by id — the drill-in tree becomes a graph of named Lenses. One lens_material_spend_heatmap shows up in a Map, in a Salon walk, and as a Lab default — same SQL, same drill-in tree, three presentation grammars.

Lenses inherit through the standard 4-level cascade (engine → pack → tenant → entity). They do not add a new dimension to the config engine; they are objects with their own folder, like any other AFS artifact.

Model dependencies — what gold must expose

The Lens runtime is generic: it composes SQL against binds_to without knowing about signals, prices, or any domain concept. That cleanliness only holds if the model layer exposes the columns Lenses actually need. When a Lens references a derived or cross-cutting attribute (worst severity across signals, total annual spend across movements, …), the model layer must pre-build it — the Lens runtime never reaches sideways.

Two infrastructure models drop in to make this work for any entity, not just material:

gold_entity_signal_summary — one row per (entity_type, entity_id), materialised after signals build, from UNION ALL of every signal_findings__* table. Carries the columns Lenses (and theses, perspectives, verdicts) reach for: signal_count, max_severity_band, severities_active, worst_signal_id, latest_finding_at. Sparse rows fill with signal_count = 0 and max_severity_band = 'none' so left-joins-then-coalesce disappears from query code. The summary IS observability — chartable, watchable over time (Sense 21), and a natural meeting point of the jinflow: operational findings aggregated, ready for strategic queries.
Per-entity analytical views — e.g. gold_material_view = gold_materials LEFT JOIN gold_material_annual_spend LEFT JOIN gold_entity_signal_summary (where entity_type = material). Master tables stay clean (attributes only). Aggregates live in focused models. Views compose. The entity_registry resolves binds_to: material to the view, not the master — Lenses always see the analytical shape.

Why option 3 (one summary table across all entities, vs. per- entity gold extensions or runtime joins from the Lens):

Symmetry across entity types — same column shape on every entity that has signals.
Build-time JOIN, not request-time JOIN. Lenses recompute on every interaction; the savings compound.
Lens runtime stays signal-blind; signal-awareness lives in the model, not the composer.
Cross-entity reasoning becomes ordinary SQL: “suppliers whose materials have many high-severity findings” is a JOIN.
Signal-set independence: add a signal → it auto-contributes; drop one → it auto-disappears. Salon endpoint already does this dynamically; option 3 makes it static.

Build order: signals → summary → per-entity view → Lens runtime queries the view. Standard dbt refs handle the order; the summary becomes a new layer downstream of signals (still within gold — no need for a “platinum” tier).

Three Levers

These are not alternatives. They are a sequence.

Lever 1 — Define Lenses; render through them

Every Map / Salon / Lab is the rendering of a Lens. The loader compiles the Lens’s narrowing recipe to SQL — composed SAMPLE / GROUP BY / LIMIT — and ships only the resulting picture.

For Material Map specifically: declare lens_material_spend_heatmap that samples (if needed), bins material rows by spend × severity into ~500 cells, and declares two drill-ins — a tabular list of the rows in a cell, and a sub-Lens that re-Maps just those rows over a finer axis. Click a cell, choose the drill: get a table, or zoom into a sub-Map.

This is the move with the highest leverage and the lowest blast radius. It does not change SvelteKit, does not change the browser, does not change the proxy. It changes the SQL the loader compiles, by introducing a new artifact (the Lens YAML) that the loader reads.

Lever 2 — Decouple data fetch from page render

The cloud SSRs the page shell only — sidebar, breadcrumbs, filter scaffolding, panel placeholders. Each panel fetches its own data via /api/panel/<route>/<panel-id> over fetch after mount. The cloud machine never holds a unified payload of “everything this page needs”; it holds N small panel responses serially, garbage-collecting between them.

This is the standard pattern in mature BI tools. We’ve been holding off because “loader returns the world” is simpler when the world is small. Once the world stops being small, the pattern flips.

Lever 3 — Streaming proxy pipe

The local proxy emits NDJSON (one row per line). The cloud forwards as application/x-ndjson straight through to the browser. The cloud is bytes-in/bytes-out — no JSON.parse, no object materialization, no buffer.

Highest payoff for proxy mode but only useful after Lever 2 lands. SvelteKit’s load() cannot consume a stream; the page must be fetching data from the browser, not from the loader, for streaming to mean anything.

Recommended Sequence

Lever 1 across the components that hurt today — define the first Lenses for Material Map, Signal Findings, Consumption trend. Add the lenses/ artifact folder + compiler + loader integration. Migrate Material Map as the reference Lens; the others follow the same pattern.
Bump the cloud machine to 1GB as a temporary breathing room while Lever 1 lands. Roll back after.
Lever 2 when a Lever-1 panel still hurts (i.e. the aggregated query itself returns too much, which would be surprising).
Lever 3 only if Lever 2 isn’t enough — likely never for the panels we have, possibly necessary for future tenants with millions of rows of detail data.

Risks and Downsides

Lens-shaped risks

Composition order matters silently. Sample-then-bin and bin-then-sample give different pictures from the same dataset. Lens YAML must declare an explicit order, the compiler must enforce it, and the order must be visible to the user (e.g. in a Lens metadata bar) when the picture is investigation-grade.
Sub-Lens loops. Lens A drills into Lens B drills into Lens A. With named ids and graph references this is easy to write by accident. Compiler must detect cycles. Probably also enforce a max drill depth.
Lens duplication. “Material Map heatmap” and “Material Map heatmap with stricter filter” become two near-identical YAMLs. Inheritance / extension at the Lens level (one Lens extends: another, overrides params) prevents this — but adds a feature surface the cascade has to honour.
The Lens vocabulary leaks into URL state. If a sub-Lens is reachable by URL, the URL has to encode “we are now in sub-lens X with these drill-in params”. URL design choices there are load-bearing for shareability.

Correctness

Aggregation hides outliers. A 500-cell heatmap collapses rows; a single anomalous material disappears into its bin. We must keep an “investigate” affordance — every cell drill-down must reach the underlying detail, otherwise the funnel becomes a fog. The Map Salon already has this in spirit; it must be load-bearing now.
Filters and aggregations interact badly. If the user filters on a column the aggregation collapsed, the loader has to recompile the SQL — not just re-execute it. The config engine’s filter machinery has to learn the difference between filters that compose with aggregation (WHERE clauses on non-collapsed columns) and filters that invalidate it. Today everything is a WHERE; tomorrow some filters will reach into GROUP BY.
Display granularity drift. A panel author picks “100 cells” today; the data grows; 100 cells become coarse. The panel still renders, but the picture loses information. There is no test for “the picture is wrong.” We need a rule: granularity is adaptive (responds to row count) or declarative-with-bounds, not a fixed magic number.

UX

Loss of point identity. Clicking a single material on the current Map highlights one row. After binning, clicking highlights a cell of N rows. Drill-down is fine — but the one-click identity is gone. For some users this is a feature (the noise was the problem); for others it’s a regression. Worth user-testing before declaring done.
Sort + top-N at the edge. “Top 50 materials by spend” pre-aggregated is a single GROUP BY; sorted at the user’s whim becomes a re-query. Today the table sorts client-side on a fully materialized array. After Lever 2, sort = network round-trip. Users will feel the latency unless we cache.
Page-shell flash. Lever 2 turns a single SSR-rendered page into a shell + N async panel fetches. Without care the user sees an empty layout for ~300ms before panels populate. Solvable with skeletons and parallel fetches, but it’s a visible UX cost compared to “page comes back fully rendered.”

Engineering

Config engine grows another dimension. display.granularity layered onto the four-level cascade is real complexity. Every panel YAML grows fields. The mental model of “Form follows Data” is preserved but the configuration surface widens. We should resist adding settings that a sane default could fill.
query() semantics split. Today every loader calls query(sql, params) and gets rows. After this, we may want aggregateQuery(sql, granularity, params) or similar. The type system stops being one shape. Prefer to keep the surface unified — granularity is a property of the SQL, not of the function.
Existing panels need migration. Every panel that today blindly fetches a table must be inspected, classified (table / chart / map / drill-down), and migrated. This is weeks of work even at a steady pace, and behaviour will subtly change in the process. Test fixtures (Playworks) must extend to cover post-Funnel rendering.
Map Salon’s auto-detection conflicts with declared granularity. Today MapPanel decides between heatmap / scatter / strip / timeline from axis types. After this, it also has to honour declared bin counts. The grammar grows; panels must be honest about what they need.

Operational

Wrong granularity is silent. If a panel declares granularity: 100 cells and the dataset is 80 cells, the picture is correct but the GROUP BY did unnecessary work. If the dataset is 80,000 cells, the picture loses information. Neither case errors. We need a build-time or runtime warn when granularity mismatch is severe.
More expensive query plans. GROUP BY with binning is heavier than SELECT *. DuckDB is fast, but the per-page query budget shifts from “network bytes out of cloud” to “CPU time in DuckDB.” For local KLS this is a wash; for R2-backed KLS the network cost goes down and CPU cost stays the same; for proxy mode the local CPU on the user’s machine takes the load. All three are improvements. Measure to confirm.
Cache invalidation. Lever 2’s per-panel fetches make panel-level caching attractive (compose a page from cached panel slices). But cached aggregations need to invalidate on KLS rebuild, and cache keys must encode granularity + filters + tenant. Wrong cache keys silently serve stale pictures. Easy to mess up.

Architectural

Lever 3 abandons SvelteKit’s load() for these routes. That’s not free. Many of our patterns (URL as source of truth, breadcrumbs, error boundaries) lean on load() returning a value. We’d need to keep load() for the page shell and add a parallel data-fetch surface for panels — which is exactly Lever 2. So Lever 3 only happens if Lever 2 has already happened.
URL as source of truth gets harder. Today the URL changes → loader re-runs → page rebuilds. After Lever 2, URL changes → shell stays → panels re-fetch independently. Each panel must subscribe to the URL state changes that affect it. SvelteKit’s stores can do this, but it’s another layer of contract that pages and panels must honour. Easy to break by accident.
The “everything is reactive” model gets pressure. A Svelte 5 page bound to a single data prop is conceptually clean. A page that composes from N independently-loading panels is still composable, but each panel has its own loading / error / empty state. The visual language has to cope with mixed-readiness pages. Loading skeletons are not UX-free.

Operational, again — what we lose by waiting

Every new tenant is a memory roulette. A bigger hospital, a bigger date range, a less-aggressive default filter — and we OOM again on a panel that worked yesterday. We’ve already found this once on Material Map. We will find it many more times.
P2P2P doesn’t scale to real-sized customers. The pitch — “your data, your machine, their browser” — is undermined if the cloud collapses on the first interesting panel. Either we cap dataset sizes (which kills the value prop) or we narrow at the source.
Cloud bills scale with data, not with users. Without the funnel, a 10× tenant means 10× memory, 10× egress bytes, 10× cold-start time. With the funnel, 10× tenant means 1× memory, slightly more SQL CPU. The cost curve bends sharply.

Out of Scope

This sense does not propose:

A new database. DuckDB is fine.
A new orchestration layer. dbt is fine.
A new framework. SvelteKit is fine.
A rewrite of JinDesk. JinDesk is the consumer of this principle; its scaffolding stays.
Caching as a primary mechanism. Caching is what you do when you can’t shrink the payload. We are shrinking the payload first.

Decision

Adopt the principle: the cloud holds the picture, not the dataset. Sequence the work as Lever 1 → bump-then-roll-back → Lever 2 (when needed) → Lever 3 (probably never).

Start with Material Map as the reference implementation. Migrate one panel cleanly, document the pattern, replicate.

How this manifests in jinflow

The Funnel narrows what the cloud sees of the data: aggregate at SQL (don’t transmit rows), decouple shell from panels (don’t ship state both ways), stream NDJSON (don’t buffer). The three levers ship together — the laptop holds the picture, the cloud holds the questions. Applies primarily to the Cloud R2 operating mode where the SQL host stays remote from the viewer.

Operating mode: most visible in Mode 3 (Cloud R2) per Operating Modes

Status: shipped. The three levers are live across all cloud-mode tenants.

Numerical neighbors: ← Sense 27 — Lever 2 — Decoupling shell from panels · Sense 28: The Direct Line — Data Skips the Middle →

jinflow is a jazzisnow product

v0.64.7 · built 2026-09-20 19:48 UTC 