Sense 27 — Lever 2 — Decoupling shell from panels

Sense 27 · Folded in · Last touched 2026-08-20

Folded into Sense sense-27-the-funnel.

• last_verified: 2026-05-10

Synced from docs/design/sense_27_lever_2.md in the engine repo — that’s the source; this page is a build-time mirror.

A design + rationale note for the second of the three Levers in Sense 27 — The Funnel. Written 2026-05-10 as a “where am I” document, not a sprint plan. Lever 2 is not yet sprinted on; this doc explains what it would take, why we haven’t yet, and what trigger should kick it off.

TL;DR

Today, every JinDesk page loader fetches all its data on the cloud and hands the browser a single payload. Lever 2 flips that: the cloud SSRs scaffolding only, and each panel makes its own /api/panel/<route>/<panel-id> fetch from the browser after mount. Cloud memory becomes proportional to the picture, not the dataset.

The pattern is already half-deployed: every Lens-rendered panel works this way. The heavy non-Lens pages (instruments, compare, dimensions) still use the old “loader returns the world” pattern.

Migration is real engineering — multiple days, framework-grain friction, and dual code paths until done. Recommend deferring until a concrete trigger (a second OOM, or commitment to Sense 28).

Why Lever 2 exists

The first OOM on numetrix.rmc/map (May 2026) revealed the underlying shape of every JinDesk loader:

┌──────────┐   query()   ┌──────────┐  serialize  ┌──────────┐
│  DuckDB  │────────────▶│  Cloud   │────────────▶│ Browser  │
└──────────┘             │ Node mem │             └──────────┘
                         └──────────┘
                              ▲
                          single
                          payload
                       (entire page)

Cloud memory grows with the dataset. Tenants grow. Eventually, a 512 MB Fly machine OOMs on a single /map request. Lever 1 (Lenses) cured the specific case by aggregating at the SQL boundary — but it doesn’t generalise to pages that aren’t Lens-rendered.

Lever 2 attacks the buffer itself: panels fetch their own data from the browser. The cloud forwards N small responses serially, garbage-collecting between them.

┌──────────┐    panel A  ┌──────────┐    panel A  ┌──────────┐
│  DuckDB  │────────────▶│  Cloud   │────────────▶│ Browser  │
└──────────┘             │  (gc)    │             └──────────┘
                         └──────────┘
┌──────────┐    panel B  ┌──────────┐    panel B  ┌──────────┐
│  DuckDB  │────────────▶│  Cloud   │────────────▶│ Browser  │
└──────────┘             │  (gc)    │             └──────────┘
                              ⋮

Cloud memory is now bounded by max(panel_size), not sum(panel_size).

What’s already in place

The Lens runtime accidentally implemented half of Lever 2:

• A Lens page (/[tenant]/lens/[lens_id]) loads minimal SSR scaffolding (route shell + manifest)
• The chart panel mounts and hits /api/lens/[lens_id]/cell on click
• The drill-in table mounts and hits another panel endpoint

This is the target pattern for the rest of JinDesk. We just need to bring the non-Lens pages along.

What’s NOT in place

43 routes in explorer/src/routes/[tenant]/ still use SvelteKit’s +page.server.ts load() pattern, where one server function returns “everything this page needs” as a single payload. The heaviest:

Route	Lines in loader	What it loads
/instruments	254	5 tables (signals, theses, verdicts, smebits, bundles)
/compare	220	Snapshot diffs
/dimensions/[entity]	203	Entity rows + sidebar metadata
/notebook	130	Notes + thread state

Each of these is a single round-trip the cloud serves entirely on the server side. If any one of them has to render 100 k rows, the cloud holds 100 k rows in Node memory until the browser has received the whole payload.

What Lever 2 looks like, concretely

The contract: /api/panel/<route>/<panel-id>

Each panel becomes a small server endpoint. URL convention:

GET /api/panel/instruments/signals?<filters>
GET /api/panel/instruments/theses?<filters>
GET /api/panel/instruments/verdicts?<filters>
GET /api/panel/dimensions/<entity>/rows?<filters>
GET /api/panel/dimensions/<entity>/columns

Response: JSON, scoped to that one panel. The cloud reads from DuckDB, serialises one panel’s rows, and frees the array. Next panel call, fresh allocation.

The page

/instruments/+page.server.ts

load() {
  // Returns metadata only — entity types, panel labels, layout config.
  // No data.
  return { panels: ['signals', 'theses', 'verdicts', 'smebits', 'bundles'] }
}

<!-- /instruments/+page.svelte -->
<DagPanel id="signals" />   {/* mounts → fetches /api/panel/instruments/signals */}
<DagPanel id="theses" />    {/* mounts → fetches /api/panel/instruments/theses */}
<DagPanel id="verdicts" />  {/* etc. */}

The <DagPanel> component owns its own loading state, error state, and re-fetch logic. The page above it is basically static.

Where signed-SQL handles attach (Sense 28)

Sense 28 — Direct Line plans to remove the cloud from the data path entirely: cloud signs the SQL, browser fetches direct from the laptop. Sense 28’s plumbing hooks into Lever 2’s panel-fetch pattern: each /api/panel/... response becomes a signed handle instead of inline JSON, the browser makes the actual data fetch against the tunnel. Lever 2 without Sense 28 is still useful (cloud memory bound). Sense 28 without Lever 2 is incoherent — there’s no panel-fetch boundary to attach to.

Why we haven’t done this yet

Three reasons, none of them shameful:

1. Loader-returns-the-world is simpler when the world is small. For a 50-row dimensions table, the migration cost is real and the payoff is invisible. We’ve been (correctly) waiting for the world to stop being small.

2. Half-and-half is the worst state. Some pages on load(), some on /api/panel/* means two patterns in our heads, two failure modes, two skeleton-loading shapes, two debugging stories. Until we commit to migrating all the heavies, we’re better off not starting.

3. SvelteKit’s grain runs the other way. The framework’s load() contract is the idiomatic place for data. Server-rendered HTML with real data IS what load() was designed for. Lever 2 deliberately sidesteps that — the SSR page becomes a loading skeleton, real data arrives later. We lose first-paint-with-content. We work against the framework, not with it. That’s not free.

The trigger to act

We should sprint Lever 2 when ONE of these becomes true:

• Second OOM: another page hits the same wall as /map did. Each one we patch with Lens migration is a Lever-1 victory, but the next page might not be Lens-shaped (e.g. a dense audit log).
• Commitment to Sense 28: signed-SQL handles need a panel-fetch contract. If we sprint Sense 28, Lever 2 lands as a precondition.
• Decision to push hard on proxy.jinflow.io: the cloud’s memory ceiling is the binding constraint there. If P2P2P traffic grows, every gigabyte saved on the proxy is real.

If none of those triggers, we keep deferring. Lever 1 covers the data-density pages we know about. Loader-returns-the-world is fine for everything else.

What a focused sprint would look like

Order: instruments first (the heaviest, plus its shape — multiple independent panels in one DAG view — is the cleanest map onto /api/panel/<route>/<panel-id>).

Phase 1 — Pattern (1-2 days):

1. Establish /api/panel/<route>/<panel-id>/+server.ts convention
2. Migrate /instruments end-to-end: 5 panels, 5 endpoints, loading states, error boundaries
3. Document the contract (a sub-doc here would replace the speculative sketch in the section above)

Phase 2 — Heavy migrations (2-3 days): 4. Apply the same pattern to /compare, /dimensions/[entity], /notebook 5. Audit the other 39 routes and triage: which are heavy enough to migrate, which stay on load()

Phase 3 — Cleanup (1 day): 6. Standard panel skeleton component (loading, error, empty states) 7. Remove the dual-code-path tax where possible

Total: roughly a week of focused work.

Risks worth budgeting

• Hydration mismatch: SSR returns a skeleton, browser hydrates, panel mounts, panel fetches. SvelteKit handles this, but failure modes are subtle. Plan for at least one round of “why is this panel empty for 200 ms before it fills in”.
• Per-panel auth: each /api/panel/* endpoint must enforce the same capability checks the page-level loader did. Easy to forget; the existing view:* capability model needs to flow through.
• URL state: today, page state lives in URL search params and loaders read them. After migration, panels still need access to those params. Two options: pass via query string (panels read URL themselves) or via component props (page reads URL, hands to panels). Pick before starting.
• Test surface: every panel endpoint is independently testable. Good for unit tests, but the integration story (page renders all panels correctly together) needs a harness.
• Logging: a single page-level loader produces a single log line per request. Five panels = five log lines. Adjust observability before this becomes noisy.

What you don’t have to decide tonight

• Tier 3 (NDJSON streaming) follows naturally from Lever 2 + Sense 28 but is a separate decision.
• Whether to migrate ALL 43 routes or just the heavies. Suggest the pragmatic line: migrate when a route’s loader exceeds, say, 80 lines of actual query code.
• Whether to keep load() for SSR-served metadata (page title, layout, panel labels) — almost certainly yes; the loader becomes trivial but doesn’t disappear.

Decision

Hold. Lever 1 has retired the known OOM trigger. Lens-driven pages already have the panel-fetch pattern. The other 43 loaders are not on fire. We revisit when (a) a non-Lens page OOMs, or (b) Sense 28 work starts.

This document is the resting place for the design. Pick it up when the trigger fires.

---

See also:

• Sense 27 — The Funnel (parent design)
• Sense 28 — The Direct Line (the architectural sibling)
• scripts/lenscompile.py, explorer/src/lib/server/lensRuntime.ts (Lever 1, the precedent)
• explorer/src/routes/[tenant]/instruments/+page.server.ts (the canonical migration target)

---

Numerical neighbors: ← Sense 26: The Workshop — the pack ships its own design · Sense 27: The Funnel — Narrow at the Source →