Sense 31: The Inlet — Where Bytes Come From Is the Channel's Business

Sense 31 · In bloom · Last touched 2026-08-20

• last_verified: 2026-07-27

Synced from docs/design/sense_31_the_inlet.md in the engine repo — that’s the source; this page is a build-time mirror.

Sense 30 is the outward contract: how a finished KLS speaks to the world. Sense 31 is the inward inlet: how bytes get to the engine in the first place. Deliberate mirrors.

The engine doesn’t know where its data was born.

A supplier delivery and a synthetic generation meet at the DLZ (or at source/, for regen-cadence channels). From there forward, jinflow cannot tell them apart.

Status: proposed Date: 2026-05-12 Authors: the owner + Claude (drafting + review, two sessions) Inherits: extractor_discipline.md — every channel honors its rules. Companion impl doc: source_as_canonical_input.md — what concretely changes in the engine.

---

What pulled us in

jinflow today has four competing keys in pipeline.yml, each answering the same question — how does data get into this tenant’s source/? — but treated by the engine as four different things:

Top-level key	Used by	What it means today
extract_entries:	rmc	Walk a DLZ folder, find dated xlsx members, hash-pin each, extract to gen00_raw
channels:	vai	Run a script, expect it to land files in source/<dir>/
generate:	rmc, inspire	A Python script that produces synthetic data; not part of make, run by hand
hooks.pre_sync:	rmc	Run an extractor right before Phase 0b syncs DLZ → source/

The engine branches on which key is present, applies different rules to each, and so the same conceptual thing — an inlet of information feeding the tenant — fragments into four code paths with four sets of conventions, four governance models, and four ways of relating to the DLZ.

This is the seam Sense 31 closes.

---

The Principle

A channel is an independent inlet of information. It is a noun about WHAT — where in the world (or in a generator) is this stream of bytes coming from. SAP SuccessFactors is a channel. An Excel from a kingdom’s clinical-qualifications spreadsheet is a channel. An SNF grants export is a channel. A curated narrative cast is a channel.

Every channel has four properties. The engine reads all four and dispatches on only two:

Property	Question it answers	Engine reads it?	Engine dispatches on it?
source_id	Which specific inlet instance is this?	yes	no — pure identifier
source_system	What canonical shape do its bytes have?	yes	yes (bronze macros dispatch per source_system)
origin	World or generator?	yes, as metadata	no
cadence	Dated series, or single live output?	yes	yes (governance only — pinning, log shape, invocation rule)

A fifth, mechanical property names what physically materializes the inlet:

Property	Question	Engine’s role
script	What program lands canonical-shape bytes?	invoke it; never inspect what it does

synthetic is never a source_id and never a source_system. It is exclusively a value of origin. A source_id names a real conceptual inlet (vai_curated_cast, vai_publications_seed, ror, wikidata, kingdom_pernet_excel); a source_system names a shape (sap_sf, opale, ror, events). The fact that bytes were generated rather than delivered is a qualifier on origin, never an identifier on its own. This catches a class of mis-modeling where the synthetic property leaks into identity space and makes the engine think “synthetic” is a kind of source.

Creator-agnostic from DLZ onward

The strongest statement of the principle:

World deliveries and synthetic generations are indistinguishable from DLZ onward.

The trust boundary is the DLZ. Before it: where bytes come into existence — the script’s business. After it: what jinflow does with them — the engine’s business. rmc (real OPALE) and inspire (synthetic OPALE) become structurally the same pipeline: same source_system, same bronze, same gold, same signals. The only difference is which script lands the bytes. Bronze does not know, cannot ask, and will not be told.

This is the same shape of stance as Sense 16 (“your data, your machine, their browser”) and Sense 27 (“narrow at the source”) — a statement about what jinflow chooses to be deliberately ignorant of.

Why origin still matters

Origin is metadata, not flow. It matters for governance, audit, sovereignty, and deployment — not for engine code paths:

Property	World	Synthetic
Audit chain	Back to a supplier, a delivery date, a hash	Back to a seed, a profile, a generator version
Mobility	Stays at the customer site (PII, IP)	Travels freely (demos, docs, dev laptops, public cloud)
Reproducibility	What was delivered is what was delivered	Bit-exact rebuildable from seed
Mutability	Frozen at delivery	Re-generatable with new defect rates / new scale / new shape
Reason to exist	This is the actual world	Demo without real data; develop without real data; inject known defects; scale-test; mirror a real tenant’s statistical fingerprint (“inspired by”)

The engine sees the passport stamp but treats every traveler the same.

---

The source/ Layout — locked

Four-level structure, uniformly applied:

source/<channel_id>/<timestamp>/<file>/<artifact>.<ext>
       │            │            │      │
       inlet        delivery     upstream  individual
       (provider    event        file     artifact
       identity)    (timestamped (xlsx,   (sheet of xlsx,
                    snapshot)    csv,     row set of csv,
                                 json…)   image, …)

Every level is mandatory. Every file has an <artifact> (even for single-artifact formats like CSV — the file’s .csv lives under a folder bearing the file’s stem; the duplication is the cost of uniformity, and it leaves room for the same logical entity to ship in multiple formats later, e.g. persons/persons.csv and persons/persons.parquet side by side).

Channel id

• Tenant-scoped. No need to prefix with the tenant name; the filesystem already lives inside the tenant’s source/ tree.
• Descriptive. Names what the inlet is, not where it came from conceptually. Example: rmc’s inlet is the OPALE system → channel id opale. vai’s curated narrative → channel id curated_cast.
• Carries no quality info. No synthetic_, no seed_, no _v2, no _raw. Origin and version are metadata in pipeline.yml and extract_log.jsonl — not identity. A channel’s provider can change over its lifetime (synthetic → world, manual export → API feed) without its identity changing.

Timestamp — YYYYMMDDSSSS

12 chars, no separators. YYYYMMDD is the calendar date, SSSS is a 4-digit sequence number scoped per channel per day starting at 0000:

202605130000   ← first arrival/regen on 2026-05-13
202605130001   ← second arrival/regen on 2026-05-13
…
202605139999   ← 10,000th of the day (more than enough headroom)

String-sortable. Same-day re-delivery (rare for world channels, plausible for regen channels under iteration) is unambiguous. Date is set by the engine to the generator-run clock by default; the --delivery-date YYYY-MM-DD flag overrides it (e.g., to back-date a regen to mirror a real-world delivery moment).

Same-hash skip

If a regen produces a sha256-identical result to the latest existing <timestamp>/ for the channel, the engine does not write a new timestamp directory. It logs a no-op (hash match) entry in extract_log.jsonl. The audit log still records that a regen was attempted; the filesystem stays clean of duplicates.

Where source_system lives

Not in the path. source_system is:

• A contract qualifier declared in pipeline.yml per channel.
• A per-delivery audit fact recorded in extract_log.jsonl (the actual source_system at that delivery’s moment, in case the channel’s source_system evolved between deliveries).
• A bronze dispatch key read from either of the above — never from filesystem position.

Keeping it out of the path means a channel whose source_system evolves over its lifetime (rare, but possible — e.g., an Excel-based inlet that migrates to a REST API) doesn’t have to relocate its historical deliveries on disk. The channel identity is stable; the shape can change.

Files within a delivery

A delivery (one timestamp dir) typically contains multiple files. For rmc, the OPALE channel delivers six xlsx files per delivery (e00_ou_structure, e1_material_master, e3e4_billing, e5_material_movements, e6_suppliers, h1h4_reference). Each xlsx becomes a folder; the sheets become CSVs inside.

For vai’s curated_cast channel, the generator emits three csv files per regen (persons, org_units, contracts). Each csv gets its own folder (with the csv inside); same shape, different fan-out.

# rmc — one channel (opale), multiple files per delivery, multiple sheets per file
source/opale/202602170000/e00_ou_structure/Tabelle1.csv
source/opale/202602170000/e1_material_master/Material_A.csv
source/opale/202602170000/e1_material_master/Material_B.csv
source/opale/202602170000/e3e4_billing/billing_hospitalisation.csv
source/opale/202602170000/h1h4_reference/H1.csv
…

# vai — one channel (curated_cast), multiple files per regen, one artifact per file
source/curated_cast/202605130000/persons/persons.csv
source/curated_cast/202605130000/org_units/org_units.csv
source/curated_cast/202605130000/contracts/contracts.csv

Delivery cadence semantics

• cadence: delivery — channel’s script (or the universal xlsx_to_csv.py) writes to source/<channel>/<timestamp>/… directly. No DLZ-to-source-sync hop. Earlier framing in this Sense called this a “DLZ → source/ mapping”; the locked design collapses that — extraction lands canonical bytes in source/ in one step.
• cadence: regen — same target, same shape. The generator writes to source/<channel>/<timestamp>/… directly. There is no cadence-specific layout.

Bronze does not know which cadence produced its inputs. The filesystem layout is uniform; extract_log.jsonl records the audit detail (cadence, hash, seed/profile/version for regen, upstream source_system, source filename + hash for delivery).

Implementation status (2026-05-14)

The strict-Sense-31 path is enforced in the engine:

• Phase 0b skips the DLZ→source mirror for any source bound to a channel in pipeline.yml. A tenant whose declared sources are all channels skips Phase 0b entirely. There is no silent fallback — the per-source status row reads Sense 31 <cadence> channel (script writes source/ directly). Undeclared sources still get the legacy mirror.
• Phase 1a (validate) and Phase 1b (enrich) discover the latest source/<channel>/<YYYYMMDDSSSS>/ dir across all declared channels and consume it directly. csvschemacheck.py resolves entity files at <csv_root>/<file_id>/<file> when <csv_root>/<file> is absent, so schemas validate nested layouts without per-script changes.
• Phase 1b for enrich-equipped tenants flattens single-file artifacts (where the artifact name matches <file_id>.tsv|csv) up one level and preserves date-series subdirs verbatim. The flatten lives in enrich_csvs.py so each pack owns its own enrichment semantics.

Engine extract scripts (Shape B) receive --source-root and are expected to write directly to source/<channel>/<YYYYMMDDSSSS>/<file_id>/<artifact>. The DLZ writeback is gone — DLZ is raw-arrival only.

CLI verbs

• jin channels [tenant] — print the tenant’s Sense 31 inventory: every channel with id, source_id, cadence, origin, latest <YYYYMMDDSSSS> on disk, file count, script. Highlights any source/ subdirectories that aren’t declared channels.
• jin regen <channel> [--tenant <id>] [--delivery-date YYYY-MM-DD] [--dry-run] — invoke a channel’s script with the right --source-root / --tenant arguments, append one record to afs/state/extract_log.jsonl. --dry-run prints the resolved command and exits so the operator sees exactly what will be touched.

Removed escape hatches

Two intermediate fallback flags were introduced during the migration and then dropped once the migration was complete:

• active: false per-channel flag — staging signal during migration; removed once all channels of the target tenant landed on Sense 31. If a channel isn’t ready, leave it out of pipeline.yml entirely.
• extraction: false master switch — legacy “synthetic tenant” override. Redundant under Sense 31 because Phase 0 already filters regen-cadence channels by their per-channel cadence. Honored for back-compat with a deprecation note; will be removed in a future release. New pipeline.yml files should declare each channel’s cadence and not use the master switch.

---

The Generator

A generator is a kind of extractor. Both materialize canonical-shape bytes; they differ only in what they consume:

• An extractor consumes a world signal — a supplier delivery, an API feed, a database dump.
• A generator consumes a specification — a seed, a profile, a version.

Both honor the same output contract: canonical-shape bytes for the declared source_system. Both run via the channel’s script:. Both produce an extract_log.jsonl entry. The engine invokes them identically.

That gives the generator side a formal contract:

Aspect	Today	Formalized
Output contract	Varies — some write gen00_raw, some write DLZ, some write source/ directly	Always canonical-shape: DLZ (delivery cadence) or source/ (regen cadence)
Inputs	Bespoke flags per generator	Standard: --seed, --profile, --out, --delivery-date (optional, regen only when cadence=delivery)
Provenance	None tracked	Logged as an extract_log entry with origin=synthetic, seed=…, profile=…, generator_version=…, plus sha256 of output for determinism check
Determinism	Implicit (some are, some aren’t)	Required — same seed + profile + version = bit-exact output (same sha256)

A pack ships its generators alongside its extractors. Both are first-class jinflow citizens, not bolt-ons.

---

The Cadence

A channel declares its contract once, not per arrival:

• cadence: delivery — “I emit a series indexed by date; each member is hash-pinned; the append-only log records every member that crosses the boundary.”
• cadence: regen — “I have one current output that gets rewritten when refreshed; no series, no per-member pin, no per-arrival log.”

A delivery only exists as a concept inside a cadence: delivery channel — it’s an arrival event (date, hash, byte count, validator outcome) within that contract. A cadence: regen channel has no deliveries; it has a current state.

Cadence is orthogonal to origin. A synthetic channel can be cadence: delivery if you want to keep historical snapshots of generator runs. Defaults: synthetic → regen, world → delivery. But defaults, not couplings.

Invocation rule:

• cadence: delivery channels are checked against the DLZ on every jin make (incremental via hash — only new dated members are extracted).
• cadence: regen channels fire only on explicit invocation (jin regen <channel> or pack-specific tooling). They do not fire as part of jin make. The engine reads whatever the last regen left in source/.

---

Topology — Logical and Physical

The KLS bakes two views of the channel topology, so JinDesk (and audit tools) can render them side by side.

KLS table	Reads	Granularity
_<tenant>.channels	from pipeline.yml at make-time	one row per declared channel
_<tenant>.deliveries	from extract_log.jsonl	one row per consumed arrival (delivery cadence) or regen event

_<tenant>.channels (logical — what was declared):

Column	Type	Notes
channel_id	VARCHAR	from pipeline.yml id:
source_id	VARCHAR	conceptual inlet identifier
source_system	VARCHAR	canonical shape (drives bronze dispatch)
origin	VARCHAR	world / synthetic
cadence	VARCHAR	delivery / regen
script	VARCHAR	relative path under afs/scripts/
output_directory	VARCHAR	landing folder under DLZ or source/
output_files	VARCHAR[]	declared filenames
declared_at	TIMESTAMP	from the make event that baked this row

_<tenant>.deliveries (physical — what was consumed):

Column	Type	Notes
channel_id	VARCHAR	FK → channels
event_kind	VARCHAR	delivery / regen
event_date	DATE	delivery date (for delivery) or regen date (for regen)
sha256	VARCHAR	of the output bytes
byte_count	BIGINT
file_count	INT
source_path	VARCHAR	DLZ path (delivery) or null (regen-direct)
seed	VARCHAR	synthetic only
profile	VARCHAR	synthetic only
generator_version	VARCHAR	synthetic only
ingested_at	TIMESTAMP	when extract_log recorded it
outcome	VARCHAR	pass / fail / skip-incremental

Together: declared inlets vs. materialized arrivals, queryable. Drift between the two becomes a first-class observable (channel declared but no deliveries yet; deliveries arriving for a channel not declared; expected cadence not honored). JinDesk renders this as a tenant’s “Inlet Topology” panel — a Sense 13 Canvas natural fit.

---

Editing channels — Atelier territory (Sense 29)

A channel is a declared artefact in pipeline.yml. The editing surface is the Atelier (Sense 29), not the engine. Sense 29’s recent extension nails the artefact taxonomy and the validation tiers; channels slot in cleanly without changing either.

Where Channel lives in Sense 29’s taxonomy: the Declarative shape tier — alongside Signal, Perspective, Lens, Entity, and Lineage YAMLs. (Sense 29’s current taxonomy table doesn’t list Channel yet — should be added when Sense 31 lands. The slot is structurally identical: a form with schema-validated fields, no SQL coercion, examples-in-place from existing channels in the same AFS.)

Sense 29’s three-tier validation maps directly to channel YAML:

Tier	What it checks for a channel
A — Schema (live, non-blocking)	cadence ∈ {delivery, regen}, origin ∈ {world, synthetic}, source_system matches an entry in csv_schemas/, output.directory matches the cadence-appropriate root
B — Integrity (on save, blocking)	source_id is unique within the tenant, script path exists under afs/scripts/, output.directory does not collide with another channel, output.files non-empty
C — Compile (post-commit, passive badge)	Dry-run invocation succeeds (delivery channels: walk the DLZ folder; regen channels: import the script and check the function signature). Failure surfaces as ⚠ on the channel card, does not revert the commit.

Sense 29’s vocabulary stability principle hardens Sense 31’s identity rules: the Atelier shows origin: synthetic consistently as the value of a qualifier — never as a source identifier, never as a source system name. The Atelier’s enum-on-source_id field literally won’t accept synthetic as a value, closing the mis-modeling gap by construction rather than by convention. Same for source_system.

The engine still does not read channel YAML at edit time. The Atelier does. The engine reads the baked _<tenant>.channels table from the KLS, populated by jin make. Editing and running stay on different clocks — Sense 29’s “never run jin make at save-time” applies to channels exactly as it does to signals.

Channel edits gate on the Operator role, not Author. Sense 29’s auth matrix introduces Operator specifically for this tier — Author covers all analytical writes (Strategic prose, Captured knowledge, declarative analytics), Operator adds the Infrastructural tier on top. The justification is blast-radius: a broken Subject Matter is a cosmetic blemish; a broken channel stops the pipeline. Same Atelier shell, narrower capability set — and the audit trail (Sense 21) records exactly which Operator changed which channel.

---

What This Changes

Four conflated keys collapse to one. Channels contain files; files contain artifacts. The pipeline.yml expresses this directly:

# Before — six extract_entries flat at the top
extract_entries:
  - id: e00_ou_structure
    source: { path: "opale/xlsx/e00_ou_structure/" }
    sheet: "Tabelle1"
    header_row: 6
  - id: e1_material_master
    source: { path: "opale/xlsx/e1_material_master/" }
    all_sheets: true
    header_row: 3
  - id: e3e4_billing
    source: { path: "opale/xlsx/e3e4_billing/" }
    …
generate: my_synth.py
hooks:
  pre_sync:
    - my_extractor.py

# After — ONE channel (the hospital's operational data feed);
# files declared within. Channel id names the inlet's PURPOSE,
# not its source_system shape (which lives in source_system: below).
channels:
  - id: material_flow
    source_system: opale
    origin: world
    cadence: delivery
    purpose: |
      Hospital's primary operational data feed — material movements,
      billing, procurement, cost-centre and dept structure. Multiple
      xlsx files arrive per delivery.
    files:
      - id: e00_ou_structure
        source: { path: "opale/xlsx/e00_ou_structure/" }
        sheet: "Tabelle1"
        header_row: 6
      - id: e1_material_master
        source: { path: "opale/xlsx/e1_material_master/" }
        all_sheets: true
        header_row: 3
      - id: e3e4_billing
        source: { path: "opale/xlsx/e3e4_billing/" }
        …

# Regen-cadence channel — generator emits multiple files per regen
  - id: curated_cast
    source_system: curated_cast
    origin: synthetic
    cadence: regen
    script: generate_tenant.py
    files:
      - id: persons         # → source/curated_cast/<ts>/persons/persons.csv
      - id: org_units       # → source/curated_cast/<ts>/org_units/org_units.csv
      - id: contracts       # → source/curated_cast/<ts>/contracts/contracts.csv

Channels per tenant (locked design)

Tenant	Channel id	source_system	Files	Notes
rmc	material_flow	opale	e00_ou_structure, e1_material_master, e3e4_billing, e5_material_movements, e6_suppliers, h1h4_reference	One inlet (hospital’s operational data feed); 6 xlsx files per delivery, each with one or more sheets.
inspire	material_flow	opale	(same as rmc)	Synthetic mirror of rmc’s feed. Only diff: origin: synthetic + cadence: regen + the script is a generator instead of xlsx_to_csv. rmc and inspire become structurally identical pipelines downstream of source/.
rmc	material_flow	sap_ish	(script handles its own internal structure)	Same role as rmc’s material_flow, different upstream system. Shape B — tenant-scoped script.
vai	curated_cast	curated_cast	persons, org_units, contracts	The 18-cast narrative anchor — every signal narrative references it.
vai	personnel	sap_sf	per_person, per_personal, per_email, emp_employment, emp_job, fo_department, fo_job_code, fo_position	Employee/job/position master. SAP SF shape today; could become a real SAP SF feed without renaming the channel.
vai	publications	publications	publications	Research output feed; will UNION ALL with ZORA / OpenAlex / Pure later.
vai	grants	grants	grants	Research funding feed; will UNION ALL with SNF P3 / ERC public DB / internal CRIS later.
vai	lifecycle_events	events	events	Hire / promote / depart / sabbatical / parental leave / retirement timeline.

The naming pattern across tenants: channel id describes the inlet’s purpose in the tenant’s domain. rmc, inspire, and rmc all have a material_flow channel because that’s what their primary inlet does — provides the hospital’s operational record. They differ in source_system (opale vs sap_ish) — but that’s a contract qualifier, not part of the inlet’s identity.

Vocabulary discipline: channel ids are short, descriptive, tenant-scoped. No tenant prefix (the tenant is already implicit). No quality info (_synthetic_, _seed_, _v2_ are metadata, not identity). No source_system in the name (that’s source_system: below it). The channel names what the inlet is; the contract qualifiers tell you how it currently realises that purpose.

Scale: the model must stay terse at tens-to-hundreds of channels. Channel definitions are short structured records; prose belongs in registry text or per-channel docs, not bloating pipeline.yml. The KLS topology tables (above) make the scale browsable as queryable metadata, not as a YAML wall.

---

What This Lets Fall Away

• Calibration moves outside the engine. Calibration has been a make phase since day one and has never gated, informed, or shaped a single build. On the Playworks pattern: a tool that watches the engine from outside, consuming KLSes and emitting drift reports — not a step the engine traverses. Three scripts retired, one make phase retired, no downstream consumer affected.

• The synthetic-vs-real distinction stops being engine state. No more generate: key, no more hooks.pre_sync:, no more extract_entries:. One channels list, one contract.

• gen00_raw / gen01_canonical / gen02_enriched collapse. Extractors write directly to source/<source_system>/<entry>/<date>/ in canonical shape. Bronze reads from there. The three-stage build cascade goes away.

• DLZ becomes optional infrastructure, not a permanent dependency. source/ is the canonical local cache; DLZ is the transient delivery surface for delivery-cadence channels. If DLZ is gone, source/ is enough to rebuild. If DLZ is present, delivery-cadence channels can refresh from it.

• build/ becomes truly disposable. Wipe it, run jin make, get the same KLS from source/.

• Tenants stop being special cases. Each is a tuple of channels. Adding a tenant is adding rows to that tuple, not picking a code path.

---

How this manifests in jinflow

Every tenant AFS carries pipeline.yml at the root: the declarative contract for what crosses the source boundary. Each entry pins a file by SHA-256, declares its expected schema, and routes it through the right extractor. jin make Phase 0 runs the extractors with failure-intolerant validation; every run appends to extract_log.jsonl as an audit trail. jin inspect lets you read the contract without touching the extractors.

• Files: afs/scripts/pipeline.yml per tenant (the contract) · afs/state/extract_log.jsonl (the audit trail, append-only)
• CLI: jin make runs Phase 0 + everything after · jin make --extract-only runs Phase 0 alone · jin inspect renders the contract · jin inspect --check verifies SHA-256 + schema without re-extracting · jin inspect --contract shows the full delivery specification

Status: shipped — steady. The six source-types (system_export, expert_curation, reference_data, pipeline_config, api_feed, db_extract) are live; SHA-256 pinning is enforced on every jin make.

---

Stance

A channel is the inlet. The script is the door. Origin is the passport stamp. Cadence is the contract. The KLS bakes both the declared topology and the consumed deliveries so the picture is auditable. The engine reads everything and dispatches on none of the identity.

Where bytes come from is the channel’s business. What jinflow does with them is the engine’s. The two never need to know each other.

The companion implementation doc spells out exactly which lines in which files change, and in what order. This Sense is about why those changes form a coherent picture.

---

Numerical neighbors: ← Sense 30: The Wire — The Outward Contract · Sense 32: The Chores — The Pulse That Doesn’t Beat Itself →