Sense 50: The Beacon

Sense 50 · In bloom · Last touched 2026-08-20

• last_verified: 2026-07-27

Synced from docs/design/sense_50_the_beacon.md in the engine repo — that’s the source; this page is a build-time mirror.

Status: Phase A shipped · Phase B implemented (2026-07-22) · Phases C–D proposed Date: 2026-07-08

Phase B implemented, 2026-07-22. The KLS now self-declares. jinflow make bakes showcase_publish + showcase_subdomain into _<tenant>.tenant (baking.py); jin cloud sync ships a tiny *_kls.beacon.json sidecar next to each working KLS (r2.py); and the cloud boot loop lists + GETs those sidecars (cheap — no KLS body download) and serves (discovered showcases ∪ KLS_FILTER) (klsStore.ts). Additive and fail-safe: any sidecar listing/GET/parse failure falls back to KLS_FILTER exactly as before, and a private tenant’s sidecar carries showcase_publish: false (nothing sensitive), so its body is never pulled to inspect it. KLS_FILTER is now an override, not the source of truth. Publishing a new showcase collapses to jinflow make <t> --clean + jin cloud sync --do-it — no engine-repo edit. Migration: existing showcases keep their KLS_FILTER entries (redundant-but-safe) until their sidecars land, then the entries can be pruned.

Restart gap closed (2026-07-23). Discovery ran only at boot (doInit), and with KLS_SYNC_INTERVAL=0 a showcase published after the last boot (its sidecar freshly on R2) stayed 404 until a manual restart — aeros.aether hit exactly this. Fixed: POST /api/refresh (scope=all — what jin instance refresh / jin ship’s Refresh stage calls) now runs a cheap undiscoveredShowcases() sidecar-list check and, only when a genuinely new showcase appears, triggers one refresh() to attach it. The common refresh stays the lightweight per-entry swap; the heavy re-init fires solely on a new publish. So jin ship is now truly bake-ship-done — no restart. Origin: the delta.jinflow.io publishing incident, 2026-07-07 → 2026-07-08 (five hours) Companion Sense docs: Sense 46 The Roster · Sense 48 The Stage · Sense 49 The Branch Operational how-to docs: Before You Build a Pack · Before You Publish a Showcase to the Cloud

Reading order

• If you are about to publish a new showcase and need to survive today: read the operational cheat-sheet. It walks the twelve traps in order and gets you to a live URL.
• If you are about to implement the fix and want the endgame: this document is the proposal. Skip to the Proposal and Implementation sketch sections.
• If you are new and want the story: read straight through. Section 1 is the origin.

Claim

A showcase tenant should declare itself — through a single artifact carried inside its KLS — and the cloud infrastructure should discover it, route to it, gate it, and render its welcome page automatically. Publishing a new showcase should be one command, not a coordinated edit across eleven declarations in four repositories.

Tonight it took eleven declarations, five hours, and three separate silent-failure modes to publish one synthetic-data demo tenant to a subdomain.

The story that made this Sense necessary

At 20:30 on 2026-07-07 we finished building the riverflow pack and the delta tenant end-to-end: pack scaffolded, KLS built, entities visible in local JinDesk. All the analytical work was done.

Mig asked: “Can we bring it to the cloud as a showcase?” Reasonable question. The pack was on GitHub, the KLS was ready, kind: showcase was declared in the tenant.yml. Every ingredient existed.

The next five hours were spent making the infrastructure understand that.

Here is what we hit, in order:

Round 1 — The engine-repo scavenger hunt (23:30 → 00:15)

Publishing delta.jinflow.io required editing:

1. packs/riverflow/tenant/tenant.yml — add envs: [prod, dev] alongside the existing kind: showcase. Without envs, the KLS gets baked with no lifecycle declaration and the cloud won’t display it.
2. live/riverflow/delta/afs/tenant/tenant.yml — the same edit, again, in the tenant’s live AFS. jinflow afs update does not sync this file (it’s treated as tenant-owned).
3. jinflow make riverflow.delta --clean — a metadata-only rebuild isn’t enough; needs --clean to actually re-bake _delta.tenant with the new envs value.
4. explorer/hosts.yml (engine repo) — add riverflow.delta to the public_tenants: allow-list. Without this, cloud auth returns 401 regardless of the tenant kind.
5. explorer/hosts.yml — new delta.jinflow.io: host block with pack_id, kls_filter, welcome: true. Without this, the subdomain falls through to the default host handler. (The readonly: / readonly_message: keys this step used to name were retired 2026-08-03 — write policy is per-tenant now; see tenant_access_policy.md.)
6. explorer/hosts.yml — add {pack: riverflow, tenant: delta} to the this-is.jinflow.io gallery for discoverability from the showcase landing.
7. fly.toml (engine repo) — add riverflow.delta to the KLS_FILTER env var. The comment above literally says “Update both files together.”
8. fly.dev.toml (engine repo) — same edit, again, for the dev app.

That’s eight declarations across three files, in the engine repo, coordinated with three declarations in two other repositories. Plus a metadata-only rebuild that isn’t metadata-only in practice.

Round 2 — The Fly secret that shadows the engine repo (01:20 → 01:45)

After all eight declarations landed and CI rebuilt cleanly, delta.jinflow.io still showed the fallback screen. Diagnostics revealed:

$ fly ssh console -a this-is-jinflow -C 'printenv KLS_FILTER'
alptrack,interlogic,lexflow,millesime,numetrix.inspire,numetrix.rmc,hrcentral.vai

The container’s KLS_FILTER env var was different from what fly.toml declared. Neither alptrack nor interlogic appeared anywhere in git; riverflow.delta was correctly in fly.toml but not in the container. Root cause: a fly secrets set had been issued at some prior point — Fly secrets override the [env] block in fly.toml and are invisible to git.

Ninth declaration:

9. fly secrets set KLS_FILTER='alptrack,interlogic,lexflow,millesime,numetrix.inspire,numetrix.rmc,hrcentral.vai,riverflow.delta' -a this-is-jinflow — the actual source of truth, unreachable from the codebase.

Round 3 — The welcome directory that must exist by name (02:00 → 02:30)

After the Fly secret was updated, delta.jinflow.io rendered — but not with a pack welcome page. It rendered the multi-tenant picker showing all seven loaded tenants, with Delta as one card among many. The intended behaviour (based on welcome: true in hosts.yml) was a single-tenant welcome landing, like numetrix.jinflow.io shows for the inspire tenant.

Mig’s diagnosis: “opinions coming from HR central or from numetrix, they are led to an introduction page. which is the main pack. Now this is what we currently have not set up. So maybe [the app] tries to open the domain pack introductory page for riverflow, and then fails and then falls back to the picker.”

He was right. loadWelcomeContent('riverflow') in explorer/src/lib/server/instance.ts calls findWelcomeDir('riverflow') which looks for explorer/welcome/riverflow/ — a directory that didn’t exist. Silent null return, silent picker fallback.

Tenth declaration:

10. explorer/welcome/riverflow/welcome.yml — quadrilingual (EN/DE/FR/IT) brand, claim, description, features[3], footer, logo config. The pack’s about.yaml in the pack repo already carries almost identical content — but JinDesk’s welcome loader ignores it and reads a separate file, in a separate location, in the engine repo.
11. explorer/welcome/riverflow/logo.svg — a pack logo, hand-crafted or copied.
12. explorer/welcome/riverflow/favicon.svg — a pack favicon.

Plus a fresh CI build to bake these files into the container image (they’re copied at Docker build time, not read from a volume at runtime).

Silent-failure modes that made debugging hard

Three distinct silent fallbacks fired tonight. Each one produced no error, no log, no diagnostic hint — the app just did something different from what the operator expected.

1. KLS_FILTER silently drops tenants not in its list

The env var is an allow-list. If a tenant isn’t in it, the boot loader simply doesn’t fetch that KLS from R2. There is no warning, no “attempted to load riverflow.delta but not in KLS_FILTER” log line. Delta.jinflow.io showed the “paste a KLS file path” fallback and we spent 20 minutes suspecting cert issues, DNS resolution, R2 upload failures — anything except the env var that was quietly excluding the tenant we had just spent an evening preparing.

2. prod-promote silently deploys a stale image when CI fails

Per Sense 48 The Stage, just prod-promote takes “the bit-identical image currently running on jinflow-dev” and deploys it to prod. If CI failed to build a fresh dev image (as it did on v0.55.13 and v0.55.14, both due to a SvelteKit route-export bug in unrelated code from another sister), prod-promote still executes successfully — it just promotes whatever image was previously on registry.fly.io/jinflow-dev:latest. The operator sees a green promote, checks git and sees the release commit tagged, and reasonably concludes the deploy worked. Prod continues running last week’s code, silently.

The way we discovered this tonight was fly ssh console -a this-is-jinflow -C 'grep -A1 riverflow /app/hosts.yml' returning empty output. That’s a manual, ad-hoc verification. There is no automated “promote succeeded, running image hash X, expected image hash Y, mismatch” check.

3. loadWelcomeContent(packId) silently returns null

If explorer/welcome/<pack>/ doesn’t exist, the function returns null and the caller falls through to the multi-tenant picker. No log, no “expected welcome dir at explorer/welcome/riverflow, not found” warning. The user sees a picker where they expected a welcome page, and there is no in-app indication that a welcome page was even attempted.

What we learned

Three observations that shape the proposal:

The KLS already carries most of the truth

By the time a KLS is built and shipped to R2, it has already baked:

• _<tenant>.tenant — tenant_id, pack, envs, kind, display_name
• _<tenant>.pack_brand — brand name, product name, primary color
• _<tenant>.metadata_bake_stamp — engine version, build time, snapshot tag
• _<tenant>.afs_archive — a compressed BLOB containing the entire pack AFS (about.yaml included)

Everything the cloud needs to decide whether to show this tenant, on what host, with what welcome content is already inside the KLS. The Fly-side declarations don’t add information; they duplicate it, from a different source, and inevitably drift.

The eleven declarations aren’t independent — they’re facets of one intent

Every one of the eleven edits above encodes the same underlying declaration: “This tenant should be publicly reachable at this URL.” The current infrastructure fragments that single intent across:

• Tenant-scoped files (tenant.yml × 2)
• Pack-scoped files (welcome/riverflow/*)
• Engine-scoped configs (hosts.yml, fly.toml, fly.dev.toml)
• Fly-side secrets (KLS_FILTER)
• Fly-side ops (certs add, ship, refresh)

The DX pain is the fragmentation, not any individual declaration.

Silent fallbacks are the true source of the pain

Each of the ten declarations, if missing, produced a silent behaviour rather than an explicit error. Debugging silent behaviour requires a mental model of what should have happened, and the operator building a fresh pack usually doesn’t have that model yet. Every failure mode we hit tonight would have been resolved in minutes if the runtime had produced a log line like:

[WARN] Tenant riverflow.delta declares kind: showcase but is not
       in KLS_FILTER — will not be loaded. Add to KLS_FILTER or
       set KLS_FILTER to include it.

or

[WARN] Host delta.jinflow.io routes to pack_id: riverflow, but
       explorer/welcome/riverflow/welcome.yml does not exist.
       Falling back to picker.

Silent failure modes are what turn a 20-minute onboarding into a five-hour debugging session.

Proposal

Showcase tenants declare themselves through the KLS, and the cloud discovers them.

1. The KLS is the declaration

At build time, jinflow make bakes into the KLS’s _<tenant> schema the fields already present, plus one addition:

• showcase_publish — boolean derived from kind: showcase AND envs contains current-target-env. If true, this tenant is meant to be publicly visible on the cloud.
• showcase_subdomain — string, optional. Defaults to <tenant_id> (so delta → delta.jinflow.io); optionally overrideable in tenant.yml if the tenant slug isn’t URL-safe or the operator wants a custom subdomain.
• welcome_source — either pack_afs (the loader reads from the KLS’s _<tenant>.afs_archive → about.yaml) or engine_welcome_dir (legacy behaviour for backwards compatibility).

2. The cloud discovers, not declares

The Fly-side container, at boot:

1. Lists KLSes in the configured R2 bucket + prefix (r2://jinflow-prod/jinflow-live/).
2. Reads each KLS’s _<tenant> schema.
3. Filters: showcase_publish = true AND envs contains 'prod' (current env).
4. Builds the equivalent of hosts.yml in memory: routing table by subdomain, kls_filter per host, pack branding.
5. Sets internal KLS load list = union of (discovered showcases) ∪ (explicit KLS_FILTER). Legacy KLS_FILTER env var becomes an optional override, not the source of truth.

No engine-repo edit is required to publish a new showcase. The cloud picks it up on next boot (or after a jinflow instance refresh poke). The tenant declares its intent in its own AFS; the cloud discovers it.

3. Welcome content lives with the pack

Instead of explorer/welcome/<pack>/ in the engine repo, welcome content comes from:

1. Preferred: the pack’s about.yaml (already trilingual, already contains claim/description). A small mapping enriches it with features (currently missing) and per-language footer. The pack repo owns the full welcome shape.
2. Fallback: legacy explorer/welcome/<pack>/ remains supported for migration.

loadWelcomeContent(packId) reads the KLS’s baked AFS archive (or the pack repo at build time) rather than a hardcoded engine-repo directory.

4. Loud fallbacks replace silent ones

Every silent-null path documented above gains a warning log line and, where possible, a visible dev-mode banner:

• KLS_FILTER excludes a KLS with showcase_publish=true → warn on boot
• Host routing matches a subdomain to a pack_id whose welcome resolver returns null → warn on request, banner in dev
• prod-promote deploys an image whose git commit doesn’t match the current tagged release → refuse without --force

Implementation sketch

Four phases, decreasing scope, increasing safety. Each phase lists the exact files an implementer touches — no discovery ceremony required.

Phase A — Loud fallbacks (1–2 days)

Ship the warnings first. Nothing self-declares yet, but the next operator hitting the same walls sees an actionable log line within minutes.

Silent-fallback #1 — KLS_FILTER excludes showcase tenants. Emit a boot-time warning when a tenant with kind: showcase is present in R2 but absent from KLS_FILTER.

• File: explorer/src/lib/server/klsStore.ts — the R2 scan loop that decides which KLSes to load.
• Change: after the filter, for every KLS skipped by KLS_FILTER, read its baked _<tenant>.tenant schema; if kind = 'showcase', log [WARN] Showcase tenant <name> in R2 but excluded by KLS_FILTER — add to KLS_FILTER or set to include it.
• Addresses trap #7, #8, #9 in the operational cheat-sheet.

Silent-fallback #2 — loadWelcomeContent(pack) returns null. Emit a request-time warning when a host with welcome: true + pack_id: X cannot resolve a welcome directory.

• File: explorer/src/lib/server/instance.ts — the loadWelcomeContent function (already reviewed; already has an early null return).
• Change: before returning null, log [WARN] Host <hostname> declares pack_id=<pack> but explorer/welcome/<pack>/welcome.yml is missing — falling back to picker.
• Also: expose a dev-mode banner in explorer/src/routes/+page.svelte when the welcome fell back to picker.
• Addresses trap #10 in the cheat-sheet.

Silent-fallback #3 — prod-promote deploys a stale image. Refuse when the promoted image’s commit hash doesn’t match the current tagged release; require --force for legit overrides (e.g. rollback).

• File: jinflow/cli/commands/promote.py (invoked by just prod-promote).
• Change: after selecting the source image, read its git-commit label; compare to git describe --tags --exact-match main on the invoking clone; refuse if mismatched.
• Add --force flag for the rollback case.
• Addresses the release-tagged but not deployed pattern we hit at v0.55.13 and v0.55.14 tonight.

Total scope: three file edits, no new mechanism. Ship it, ratify Phase A, move on to Phase B.

Phase B — KLS-derived auto-discovery (1–2 weeks)

Cloud reads R2, iterates KLSes, discovers the showcase set. Falls back gracefully to KLS_FILTER if metadata is absent or malformed. This is where the tenant starts self-declaring.

Metadata bake extension.

• File: jinflow/cli/commands/baking.py — the metadata-bake step that already stamps _<tenant>.tenant with tenant_id, pack, envs, kind, display_name.
• Change: add showcase_publish as a derived column: kind = 'showcase' AND array_contains(envs, target_env). Also add showcase_subdomain (from tenant/tenant.yml, defaulting to tenant_id).

Cloud boot discovery loop.

• File: explorer/src/lib/server/klsStore.ts — the R2 scan that populates the tenant list.
• Change: after fetching each KLS’s manifest, read its _<tenant>.tenant row. Build an in-memory discovered_showcases set from KLSes where showcase_publish = true. Merge with any explicit KLS_FILTER entries. Log the union.

Legacy env cleanup.

• Delete the alptrack, interlogic orphaned entries in the Fly-side secret. They should not exist anymore; if they do, log the discrepancy.
• Update fly.toml and fly.dev.toml comments to reflect that KLS_FILTER is now an override, not the source of truth.

Phase C — Pack-carried welcome (2–3 weeks)

Welcome content moves from engine explorer/welcome/<pack>/ into the pack’s own repo. Existing packs migrate in a batch.

Schema extension.

• Extend about.yaml to carry features[] (currently only in welcome.yml), footer[lang], and logo/favicon refs to files inside the pack’s own branding/ dir.
• Version bump the about.yaml schema.

Loader refactor.

• File: explorer/src/lib/server/instance.ts — loadWelcomeContent currently reads explorer/welcome/<pack>/welcome.yml.
• Change: prefer the pack’s about.yaml (accessible via the baked _<tenant>.afs_archive BLOB); fall back to the engine directory for backwards compatibility during migration.

Migrate existing packs.

• One PR per pack (numetrix, hrcentral, alptrack, millesime, lexflow, interlogic) — move welcome content from explorer/welcome/<pack>/ into the pack’s about.yaml.
• Delete explorer/welcome/<pack>/ from the engine repo once every pack is migrated.

Phase D — Reserved-subdomain convention (post-Phase B)

Formalise the <tenant>.jinflow.io convention.

CORRECTED 2026-08-08 — there is no wildcard CERTIFICATE. This section claimed one “already in place” and concluded that per-tenant fly certs add was redundant. It is not, and the belief cost a day: vs.jinflow.io and aura.jinflow.io were shipped, routed and public, and simply would not load.

fly certs list -a this-is-jinflow shows nine individually issued hostnames and no wildcard — including delta.jinflow.io, the very host cited below as proof. The explicit cert add was not redundant; it is why delta works.

The wildcard is DNS only — *.jinflow.io A + AAAA, which is a real and deliberate principle (no per-tenant DNS records, ever). Certificates are a separate axis, and each host needs its own.

The failure mode is why this was hard to see. DNS resolves, so the host looks configured; Fly then refuses the TLS handshake, and a browser reports only that the site cannot be reached. Nothing anywhere says “no certificate”. fly certs list is the one place that answers it.

*.jinflow.io covers DNS. Explicit cert adds remain required, per host.

What is actually true.

• Adding a tenant host needs exactly one command and no DNS edit: fly certs add <host> -a <app>.
• Step 11 of the publishing cheat-sheet stays mandatory for every *.jinflow.io host, not only for external domains.

Subdomain override for edge cases.

• In tenant/tenant.yml, allow showcase_subdomain: <slug> to override the default <tenant_id> → subdomain mapping.
• Useful when tenant_id isn’t URL-safe or when two tenants across packs want disambiguation.

Migration path

• Phase A ships without breaking anything. Existing showcases continue working.
• Phase B introduces auto-discovery in addition to the explicit KLS_FILTER. Existing entries in KLS_FILTER continue to be honoured. When auto-discovery covers all existing showcases, prune the env var.
• Phase C migrates welcome content pack-by-pack. Both paths coexist until every pack has been migrated.
• Phase D formalises what has already worked in practice since the wildcard DNS convention was adopted (memory: jinflow.io wildcard DNS principle).

No forcing function. No big-bang rewrite. Additive.

Ready to implement — recommended first step

Phase A, three file edits, ~1 day of work. No new mechanism. No breaking change. Every future operator hitting the same walls resolves in twenty minutes instead of five hours. Highest-value work per hour of any phase.

The three files to touch:

1. explorer/src/lib/server/klsStore.ts — warn on excluded showcases (trap #7-9)
2. explorer/src/lib/server/instance.ts — warn on missing welcome dir (trap #10)
3. jinflow/cli/commands/promote.py — refuse stale-commit-hash promotes (silent-failure #3)

Every subsequent phase gets easier once Phase A ships — the warnings will show us exactly where the discovery gaps still live.

Sense-cluster relations

• Sense 46 The Roster — Roster is the tenant catalog inside a Fly app (which KLSes are loaded); Beacon extends Roster to how new tenants get into it (auto-discovery vs manual declaration).
• Sense 48 The Stage — Stage is env lifecycle (dev/prod, promote pipeline); Beacon reads from Stage’s env declarations to filter which tenants surface where.
• Sense 49 The Branch — Branch is pack composition (a pack can be assembled from branches); Beacon complements it by removing the engine-repo cost of publishing showcase branches.
• Sense 43 The Cascade — Config engine cascade already handles L1 pack → L2 tenant. Beacon is the same principle applied to deployment config: tenant declares → cloud discovers → engine defaults win only on absence.

Open questions

1. Auth semantics. ANSWERED 2026-08-03 by the tenant access policy: they were redundant, and the host flag was the wrong home. kind: is the authored seed for a per-tenant access policy held in the OPS; readonly: is gone from hosts.yml, which is now pure routing.
2. Two tenants, same slug, different packs. If riverflow.delta and (hypothetical) hrcentral.delta both wanted delta.jinflow.io, the reserved-subdomain convention needs a disambiguation rule. Options: first-published-wins, alphabetical pack ordering, or require explicit showcase_subdomain override.
3. Non-showcase tenants that want a public URL. A private customer might want customer.jinflow.io reachable but auth-gated. Currently that’s rmc.jinflow.io shape (require_auth: true). Post-Phase-B, does auth-gated-public-URL become a separate kind: value, or does it stay in the host config?
4. Welcome content in the KLS vs the pack repo. Reading from _<tenant>.afs_archive means the welcome content is frozen at KLS-build time. Reading from the pack repo at request time means it can update out-of-band. Trade-off between staleness and freshness. Probably: KLS-baked for cloud, pack-live for local dev.
5. How does the wildcard cert handle non-jinflow.io subdomains? For customer-specific domains (say, customer.example.com), Beacon doesn’t help — those need explicit cert adds. Beacon is scoped to *.jinflow.io, not universal.

What this unlocks

If Phase A ships alone, tomorrow’s operator hitting the same walls resolves in twenty minutes instead of five hours. That’s already worth the investment.

If Phase B ships, publishing a new showcase collapses to a single command from any environment, without engine-repo access:

# Add kind: showcase to tenant/tenant.yml (in the tenant AFS, one file, one line).
jinflow make riverflow.delta --clean
jinflow ship riverflow.delta --env prod --do-it

Three lines. No engine-repo edit. No hosts.yml scavenger hunt. No KLS_FILTER shadow-secret. No welcome-dir mystery. The KLS carries the intent; the cloud honours it.

If Phase C ships, welcome content lives with the pack in a way that reflects reality — about.yaml already carries the pack’s identity, and JinDesk’s welcome loader stops being a separate ceremony. New packs onboard by writing their about.yaml and nothing more.

If Phase D ships, the last mechanical Fly-side ceremony (fly certs add) goes away for the 95% case.

The endgame: publishing a showcase is the same shape as building a pack. One artifact, one command, discovered automatically. The engine repo doesn’t become the coordination point; the pack + tenant already are.

Acknowledgements

This Sense exists because tonight, five hours in, the operator (Mig) named the pain out loud: “we need an app deployment in order to raise a new showcase…not cool.” And later: “can we open a new sense that extensively describes the struggle and pain, and a way forward?” That naming is what turns a debugging session into a Sense proposal.

The immediate cost tonight was one late evening. The long-term cost was every future operator, every future pack, every future showcase-shaped launch. Beacon is a bet that we can pay the debt once, in code, and reclaim it forever after.

---

Numerical neighbors: ← Sense 49 — The Branch · Sense 51: The Skill — cross-domain analytical patterns →