Aller au contenu

Sense 54 — The Typology

Ce contenu n’est pas encore disponible dans votre langue.

Sense 54 · In bloom · Last touched 2026-08-20

  • last_verified: 2026-07-27

Synced from docs/design/sense_54_the_typology.md in the engine repo — that’s the source; this page is a build-time mirror.

Status: proposal · 2026-07-11 Author: claude (Opus 4.7) with Mig Sense number: 54 (verified free on origin/main; Sense 53 The Codex claimed the previous slot)

Refined 2026-07-12. A working session with Mig substantially sharpened this Sense. The current model — semantic type as a data-stream invariant declared at both the ingestion and Gold gates, conversions as typed edges, type-is-not- configuration, and KLS inspector tables replacing the JSON side-file — is in § Evolution — the stream-invariant model (2026-07-12) at the end. Where the sections below conflict with it, that section wins; the most-affected passages are flagged inline.

Every Gold column declares what it is. Not at the signal boundary. Not in the UI. At the site of birth: the entity YAML. Types cascade downward, standards attach where they apply, and every downstream layer inherits — signal aggregates, JinDesk formatting, contract validation, cross-pack composition.

The Codex hosts the standards. The Typology types the columns those standards apply to. Together they answer the metrologist’s question — is this a measurement or is it just a number? — for every value that leaves Gold.

Pack builders and canon creators — anyone who declares an entity in an AFS. The Typology is platform vocabulary of the same rank as the Codex: not engine-internal machinery, but training material a pack author learns once and applies across every entity they ship. A data-creator script for a new sensor type consults the Typology to know what kinds are available and which Codex standards it may bind to. A signal author reads it (indirectly, via the compiler) to inherit column types instead of restating them. An JinDesk render path reads it to know how to format a value in a locale. Everyone in the pack-build path sees the Typology; no one has to reinvent it.

Because signals are the wrong site.

Sense 14.2 Phase 5 typed the aggregates: a signal’s aggregate: {column, type, unit} block validates that a physical quantity carries a UCUM-compatible unit, that a currency carries an ISO 4217 code, that a duration is ISO 8601. Every pack that adopted Phase 5 gained this discipline at the signal boundary.

But the column being read has been read a hundred times before the signal ever touches it — by Bronze macros, by Silver validators, by Gold views, by JinDesk, by every dbt test, by every notebook, by every downstream pack. Along that journey, the type is inferred by each reader from the column name or from context. Mostly they infer correctly. When they don’t, it’s silent.

The signal aggregate is where we notice the type mismatch. The entity YAML is where we should declare it.

Metrology has known this for 150 years. A measurement is not a number — it is a number + type + unit + uncertainty + reference standard. The Bureau International des Poids et Mesures does not accept a value without all five. The Guide to the Expression of Uncertainty in Measurement (JCGM 100:2008 — already in the Codex) says the same. Software conventionally strips this at the CSV boundary and asks users to remember. jinflow’s Four Convictions argue for restoring it: a value that has lost its type has lost its meaning, and a platform that declares itself “Transparent” should not participate in that stripping.

The Typology is the fix: no Gold column may present a value without declaring what kind of value it is.

Mirroring [[sense_53_the_codex]] and [[sense_15_the_observation]]:

  1. “What is this column?” — the analyst asks “the value here is 72.4 — is that milligrams per litre? Something else?” Answer: read the entity YAML; every column has a declared type.
  2. “Does the platform know what this column means?” — the reviewer asks “or is the meaning locked in the author’s head?” Answer: the type is a declaration, not a comment. Compilers read it. Signals inherit it. JinDesk renders it.
  3. “Can I compose across packs?” — the pack builder asks “if riverflow declares mass_g and numetrix declares mass_kg, will jinflow catch it?” Answer: yes, because both declare the same UCUM-typed kind, and the compiler knows they’re comparable up to a unit conversion.

Any framework that answers only one of the three is decoration. A framework that answers all three is a Typology.

The Four Convictions read through metrology

Section titled “The Four Convictions read through metrology”

Read the [[project_four_convictions]] through the lens of measurement:

  • Declarative. A number without a declared type is undeclared. The Typology says: if you claim your Gold contains measurements, you must declare each measurement’s kind. The absence of a declaration is not a permission to guess; it is a defect the compiler flags.
  • Transparent. A reader must be able to trace mass_kg → SI base unit → BIPM. Today the trace stops at the column name. The Typology extends it all the way to the standard.
  • Independent. A KLS travels with its AFS archive (see [[project_the_jinflow]]). If the AFS declares types, the KLS is metrologically self-describing — the receiver needs no ambient knowledge to know what the values mean. This is the difference between shipping data and shipping science.
  • Human. A person reads “72.4 mg/L per USGS PC00040, uncertainty ±0.3” and understands. A person reads “72.4” and asks. The Typology makes the first form possible without hand-authoring it in every panel.

An assayed measurement, not a bare number. Assayed = declared type, declared unit where applicable, declared standard where one applies, declared uncertainty policy where meaningful. The declaration is what makes the number legible as science.

Some columns are not measurements. A status enum is a category. A site_id is an identifier. A notes field is uncontrolled text. The Typology has a place for each of these — not everything is a physical quantity, but everything is something, and the Typology insists you say what.

Why the type belongs to the column, not the signal

Section titled “Why the type belongs to the column, not the signal”

The signal is the arithmetic. It sums, averages, correlates, thresholds. The type of the inputs is not the signal’s business — it’s the substrate’s business. Asking a signal author to know the type of every column they touch is like asking a chef to declare the wattage of the oven every time they bake a loaf. The oven’s wattage is the oven’s declaration, not the recipe’s.

Move the declaration to the site of birth, and every recipe becomes shorter, safer, and more composable.

The load-bearing principle: types are declarations, not runtime data

Section titled “The load-bearing principle: types are declarations, not runtime data”

The Typology follows the discipline that TypeScript and Rust follow: types exist to be verified at compile time; they do not travel with runtime values. Every operation the Typology governs — signal aggregation, dimensional arithmetic, standard binding, FK inheritance — is proven correct or refused before SQL reaches DuckDB. The KLS then carries only data whose semantics have already been asserted.

This is not a stylistic choice. It falls directly out of jinflow’s declarative posture: if the AFS is the source of truth, the compiler must be the only authority that decides whether the AFS is internally consistent, and the KLS must never be asked to re-verify what the AFS already declared. Baking types into the KLS would create two truth sites (the declaration and the baked reflection) that could disagree — the same failure mode that makes runtime type systems in dynamic languages feel unsafe.

Three consequences, all load-bearing:

  • Declarations live in AFS YAML. Entity column_types: blocks. Codex standards/*.yaml. Pack manifests. Pure text, pure git, human-readable, review-able, diff-able. There is no other declaration site.
  • Verification lives in the compilers. typologycheck.py reads the declarations, walks the processing graph (signal aggregates, dbt refs, cross-entity joins), and refuses illegal operations at build time. Every SQL statement that reaches DuckDB has already been proven correct against the declared types.
  • Runtime artifacts (KLS) carry data, not declarations. The KLS never gets a _typology_* table. Coverage rollups, standard bindings, and per-column type manifests live in state/ as build artifacts (JSON, git-tracked) — read by JinDesk, jin doctor, CI, and any downstream tool without touching the database. (Superseded 2026-07-12 — the artifact moved into two inspector-only KLS meta tables, _<tenant>.column_types / _typology_coverage, because the cloud JinDesk has no AFS on hand. The principle holds — no processing reads them; they are debug symbols, not runtime type tags. See § Evolution.)

The bridge between AFS and KLS is the compiler. Both sides speak to it; neither speaks to the other. The KLS stays about data. The AFS stays about declarations. Types never travel across the boundary because they never need to.

Entity YAMLs already carry a columns: block, but today it is a role registry — six slots (primary_key, business_id, label, detail, timestamp, amount) mapping roles to column names. The Typology adds a sibling block, column_types:, that describes each column’s type:

# entities/sensors.yaml — Riverflow
entity_id: sensors
dbt_model: gold_sensors
columns: # role registry — unchanged
primary_key: sensor_key
business_id: sensor_id
label: serial_number
timestamp: last_seen_at
column_types: # ← new: per-column typology
sensor_id:
kind: identifier
subkind: business_id
scope: entity_local # ids that are unique within one entity type
serial_number:
kind: identifier
subkind: external
standard: null # no external standard — vendor-assigned
model_family:
kind: category
values: [RK1, RK2, RK3]
standard: null
current_site_id:
kind: identifier
subkind: foreign_key
references: deployment_sites.site_id
nullable: true
nullable_reason: sensor_off_fleet
status:
kind: category
values: [active, in_stock, decommissioned, lost]
standard: null # pack-defined vocabulary
last_seen_at:
kind: temporal
subkind: instant
standard: iso_8601 # ← Codex reference
timezone: utc
precision: second
first_seen_at:
kind: temporal
subkind: instant
standard: iso_8601
timezone: utc

For measurement columns:

entities/measurements.yaml
column_types:
measured_value:
kind: quantity
quantity_kind: mass_concentration
unit: mg/L # UCUM-validated via Codex
standard: ucum
uncertainty_column: measured_value_uncertainty # column carrying u(x)
uncertainty_policy: jcgm_100_2008
measured_value_uncertainty:
kind: quantity
quantity_kind: mass_concentration
unit: mg/L
standard: ucum
role: uncertainty_of # points back
role_target: measured_value
observed_property_id:
kind: identifier
subkind: foreign_key
references: observed_properties.observed_property_id
standard: usgs_parameter_codes # ← Codex reference
phenomenon_time:
kind: temporal
subkind: instant
standard: iso_8601
timezone: utc
precision: subsecond

The Typology recognises eight top-level kind values. Every column is exactly one:

KindDescriptionExample
quantityA measured value that has units and (usually) uncertaintymass_g, temperature_c, pressure_kpa
identifierA key that identifies an entity — primary, business, external, foreignsensor_id, serial_number, iso_country_code
categoryA value drawn from a bounded, unordered vocabularystatus, event_type, region_name
ordinalA value drawn from an ordered scale — rank/grade/level. Orderable but not summablelevel (hierarchy depth), severity_rank, who_air_quality_band
temporalA time-shaped value — instant, interval, duration, periodphenomenon_time, deployment_window, retention_period
booleanTwo-valued truth (with optional third state for tri-state cases)is_active, has_calibration
narrativeUncontrolled text intended for human reading, not for grouping or filteringnotes, interpretation, technician_comment
geometryStructured spatial data with a coordinate reference systemboundary_geojson, geometry_geojson

Eight is enough to express any Gold column we’ve seen across the seven adopting packs plus the geometric columns Cartography V2 introduces. Sub-kinds refine — identifier.foreign_key, temporal.duration, geometry.polygon — without multiplying the top level.

Note on the seventh kind. The original 2026-07-11 draft of this Sense described six kinds, deferring the geometry question to Cartography Phase 2. Sense 52 V2 (2026-07-11 spec) locked the decision: geometry earns its own top-level kind rather than being crammed into narrative as a serialized-string subkind. Reasoning: a polygon is not narrative. It’s structured data with a coordinate reference system, a serialization format, and spatial semantics that participate in future spatial queries. Modelling it as narrative would lose exactly the semantic content the Typology exists to preserve — and forcing every downstream reader to re-parse it out of an opaque string would violate the “types are declarations” principle by pushing type recovery to runtime. The seventh-kind path costs one more line in the kinds table; the compromise path costs every future geometric column its type-safety.

Note on the eighth kind. ordinal was added 2026-08-18. Earlier drafts modelled ordered scales as a subkind, category.ordinal — a convenient shorthand that hid a real distinction. An ordinal is not a flavour of category; it is a different measurement scale (Stevens’ nominal → ordinal → interval → ratio). Two behaviours turn on the difference. First, order: min/max/median are the ordinal’s native aggregates (they are meaningless on a nominal category), so the ingestion-agg gate accepts ordinal alongside quantity/temporal as orderable. Second, non-additivity: rank 1 + rank 2 is not rank 3, so an ordinal carries no rollup — it neither sums nor averages up a hierarchy. That makes ordinal the honest home for material_classifications.level (a hierarchy depth: ordered, comparable, un-summable), which under the six-kind vocabulary had no true kind — category lost the order, quantity invented an arithmetic the values don’t have. A subkind could not have fixed the agg gate or the rollup question, because both are decided on kind, not subkind. Hence a top-level kind, not category.ordinal.

Three subkinds worth naming explicitly because they name recurring patterns:

  • category.meta_annotation — a categorical column that annotates another column rather than standing alone. Quality flags (quality_flag, qartod_flag), uncertainty-source classifications (uncertainty_source), packet-status codes (packet_status). Declares annotates: <other_column_name> so the compiler and JinDesk know the linkage.
  • quantity.geodetic_latitude / quantity.geodetic_longitude — coordinate pairs are physical quantities (degrees) but they carry a projection standard the Typology must know about. Both subkinds bind to standard: wgs84 from the Codex. Elevation stays as quantity with the appropriate SI unit (m) and an optional geoid-reference note.
  • geometry.polygon / geometry.multipolygon / geometry.linestring — the three primitive subkinds Cartography V2 introduces. Each declares a serialization standard: (initially rfc_7946_geojson, with room for future WKT / TopoJSON / vector-tile bindings) and a crs: (initially wgs84, with room for tenant-specific projections like Swiss LV95). The Codex binding_kinds field enforces that a geometry column may only bind a standard whose binding_kinds list includes geometry — no accidental standard: ucum on a polygon.

The Typology honours the Sense 43 cascade:

Superseded 2026-07-12. Two claims below are retracted: (a) name-based L0 defaults are not a build-time behaviour — no name inference in make, only in schema authoring; and (b) tenant/perspective override does not apply to type (type is not configuration and does not cascade — only display config does). See § Evolution §4–§5.

  • Engine (L0) — defines the eight kinds, their subkinds, and their Codex-standard hooks. Ships default unit/standard mappings for well-known column names (ends_with('_at') → temporal.instant, standard: iso_8601, ends_with('_geojson') → geometry.polygon, standard: rfc_7946_geojson, crs: wgs84) so early packs get sensible defaults.
  • Pack (L1) — declares column_types: per entity in the pack repo. This is the primary site of authoring.
  • Tenant (L2) — may override a pack’s declaration when a tenant genuinely differs (a custom status value, a non-standard unit). Overrides are narrowing or substituting, never widening.
  • Perspective (L3) — reserved for future use. Not expected in Phase 1.

Cascade merge is deep — a pack declares column_types.measured_value.unit: mg/L, a tenant may add column_types.measured_value.reference_range: {low: 0, high: 50} without restating the unit.

Once declared, types flow through every downstream layer:

  • Silver — already preserves the column; now the compiler asserts the Silver column shape matches the declared type (e.g. a quantity column must be a numeric SQL type, a temporal.instant must be a timestamp).
  • Gold — the Gold contract JSON (which already lives as a git-tracked build artifact, not in the KLS) is regenerated to embed the types, so cross-pack readers can consume the contract without reading the entity YAMLs.
  • Signals — the aggregate: {column, type, unit} block from Sense 14.2 becomes derived rather than declared. Signal authors write aggregate: {column: measured_value} and the compiler fills in type: quantity, unit: mg/L, standard: ucum from the Typology. Signal YAMLs get shorter.
  • JinDesk — column formatting, precision, unit-aware display, and Codex badges all read from the Typology instead of per-column display: config. The Sense 43 display: cascade stops carrying redundant type declarations.
  • Bell / Landscape — surface assay coverage as a first-class quality metric: what fraction of a pack’s Gold columns carry declared types? What fraction of those bind to a Codex standard?

The relationship to Sense 53 and Sense 14.2

Section titled “The relationship to Sense 53 and Sense 14.2”

The three form a stack, and Sense 54 is the missing floor:

┌────────────────────────────────────────────────────────────┐
│ Sense 14.2 — Typed signals │
│ Signal aggregates carry type + unit + standard │
│ Inherit from the Typology when column is declared │
├────────────────────────────────────────────────────────────┤
│ Sense 54 — The Typology ← this doc │
│ Every Gold column carries a declared type │
│ References standards from the Codex where they apply │
├────────────────────────────────────────────────────────────┤
│ Sense 53 — The Codex │
│ Platform-hosted registry of external standards │
│ UCUM, ISO 8601, ISO 4217, USGS PC codes, SensorThings, │
│ JCGM 100:2008, ... │
└────────────────────────────────────────────────────────────┘

Read top-down: a signal aggregates typed columns that reference platform standards. Read bottom-up: the platform hosts the standards, packs bind their columns to those standards, signals inherit the bindings and do arithmetic on top.

The three senses were designed in the wrong order — signals first (Sense 14.2, 2026-05→07), standards next (Sense 53, 2026-07-09), columns last (Sense 54, this doc). That’s fine. The order of design rarely matches the order of dependency. What matters is that the three now fit.

The Typology and the Codex share ownership of two mechanisms. Neither belongs entirely to one Sense; both are cleanest as a division of labour:

  • The binding_kinds field on Codex standards. Each engine/standards/*.yaml gains a binding_kinds list declaring which of the six Typology kinds a standard is legitimate to bind to. The list is authored honestly — if a standard has multiple legitimate roles (ISO 4217 values are quantities and ISO 4217 codes are identifiers; USGS PC codes act as FK identifiers and as bounded categories), the list enumerates all of them rather than pretending one is primary. Most standards carry one kind; some carry multiple; some are structural rather than column-typing and carry an empty list, telling the checker to skip them entirely.

    Standardbinding_kinds
    ucum[quantity]
    iso_4217[quantity, identifier] (currency values and currency codes)
    iso_8601[temporal]
    iso_80000[quantity]
    jcgm_100_2008_gum[quantity] (uncertainty policy on measured values)
    jinflow_naming[identifier]
    wgs84[quantity] (geodetic coordinates)
    usgs_parameter_codes[identifier, category] (pcode as FK to observed_properties; pcode as bounded vocab used directly)
    iso_7027[quantity] (turbidity FNU/NTU)
    astm_d3977[quantity] (SSC gravimetric)
    wmo_guide_no_8[quantity] (air temperature, relative humidity)
    ogc_sensorthings_1_1[] (structural entity contract)
    ogc_waterml_2_0[] (structural)
    rfc_7946_geojson[geometry] (moved from [] when Cartography V2 landed the seventh kind)
    odbl[] (licence, not column-typing)

    This turns standard: ucum in a column declaration from a stringly-typed reference into a compile-time-checked pairing — typologycheck.py refuses standard: ucum on an identifier column, because the Codex tells it that’s not a legal binding. Sub-kind refinement (quantity.temperature, quantity.currency) is deferred to Phase 2; Phase 1 checks top-level kind only. Sense 53 owns the field’s schema; Sense 54 owns the checker that consumes it.

  • The coverage rollup — state/typology_coverage.json. The Typology compile writes a JSON artifact (one entry per Gold column: entity_id, column, has_type, has_standard, standard_id, kind) to state/typology_coverage.json and logs the write to the Sense 47 Log with an SHA-256 for auditability. This is a build artifact, not a database view — the KLS never sees it. The Codex reader consumes the JSON for the pack-adoption panel that answers “what fraction of this pack’s columns are assayed, and against which standards?” — “riverflow is 71% assayed, 62% UCUM-bound, 100% ISO 8601-bound.” Real numbers, not marketing. Multiple downstream tools read the same artifact: JinDesk, jin doctor, CI, packaging. No round-trip to the database because there was never a reason for a round-trip. The Codex owns the JSON’s schema (via standard_contract.v1.json’s coverage-block); the Typology compile writes the rows; anything with filesystem access consumes them. This absorbs the “grounding grade” idea that was briefly a candidate Sense 55 (retired 2026-07-11 into this rollup during sister-9’s review): a per-signal grounding grade is mean(column_grades) where each column_grade is the fill-rate of that column’s Typology declaration. Same insight; the rollup lives here, one level above the per-column truth.

For declarations that reference a Codex standard, typologycheck.py must be able to resolve the reference. The engine loads all engine/standards/*.yaml at startup and exposes them as a lookup dictionary keyed by standard_id. The Typology checker consults this dictionary and refuses declarations that reference unknown standards. Cheap, and it upgrades every standard binding from a string to a typed reference. The loader lives in the Codex module — Sense 54 is a consumer, not the owner.

Every pack has hundreds of untyped Gold columns. Making them all typed on day one is a fantasy. Making them typed over time, with visible progress, is achievable.

Four moves:

  1. Assay coverage as a first-class metric. Every jin make writes state/typology_coverage.json with one entry per Gold column: entity_id, column, has_type, has_standard, standard_id, kind. A pack-level rollup lives in the same file: packs/numetrix is 34% assayed, packs/riverflow is 71%. Coverage appears on the Landscape, on the Bell, and in the jin doctor output. The JSON’s schema is owned by the Codex (see the shared-surfaces section above); the Typology compile writes the artifact; every downstream reader (JinDesk, CI, packaging) consumes it directly from filesystem. The KLS is not involved — types are declarations, not runtime data.
  2. The zero-day defaults. The engine ships name-based inference rules: ends_with('_at') → temporal.instant, starts_with('is_') → boolean, ends_with('_id') → identifier. On day one, ~40% of most packs’ columns get typed for free. The residual is the interesting part — measurement quantities, categorical enums, foreign keys — which is exactly where the human declaration adds value.
  3. The strict-mode toggle. Packs opt in per entity to typology: strict — meaning the compiler fails if any column in that entity lacks a declared type. Strict mode spreads entity-by-entity, at the pack’s own pace, until eventually the pack manifest reaches typology: strict at the top level. That’s the moment the pack is fully assayed.
  4. The sidecar hint channel (Phase 3+, optional). Data-creator scripts (synthetic generators, extractor pipelines) may ship a sidecar manifest declaring the shape of the data they produce — kind, unit, expected range, per output column. The Typology compile warns when the entity YAML declaration diverges from the sidecar hint. Not authoritative — the entity YAML always wins — but a diagnostic that catches “generator was updated, entity YAML wasn’t.” Analogous to how OpenAPI generators produce type hints for downstream consumers. Deferrable; not part of Phase 1.

Riverflow’s substrate is the natural pilot: measurement columns already carry unit_of_measure metadata, foreign keys already reference OGC-modelled ObservedProperties, calibration binds uncertainty back to a Calibration event. But full riverflow is ~200 columns across 19 entities — enough to prove the shape and enough to exhaust a session budget before Phase 1 lands.

The pilot narrows to three entities that together hit the interesting patterns:

  • sensors — 7 known-shape columns. Covers identifier, identifier.foreign_key, category, temporal.instant. The simplest testbed for the declaration mechanics.
  • measurements — 14 columns including the quantity-with-uncertainty pattern (result_value + uncertainty + uncertainty_source + quality_flag). Exercises quantity, category.meta_annotation, and the standard binding to UCUM and ISO 7027 / ASTM D3977 (two of the three new Codex standards landed on claude/9th on 2026-07-11).
  • observed_properties — 12 columns with an existing USGS parameter-code binding via pcode. Turns a de-facto binding into a declared one and validates the Codex-reference mechanism against real content.

If these three entities compile cleanly with the column_types: block, land the Typology in JinDesk, and produce a non-empty state/typology_coverage.json, Phase 1 is proven. The remaining 16 riverflow entities become clerical work; the migration to numetrix, hrcentral, alptrack, and millesime becomes the next natural cohort.

The compile-time verification discipline lets typologycheck.py refuse six categories of illegal operation before dbt runs a single query. Each of these is a class of bug that today shows up as silent Bronze-to-Gold data corruption and is caught (if at all) months later when a signal produces a suspicious number. Moving them to compile time turns them into build failures — JinDesk never renders a wrong answer because dbt never ran.

  1. Unit mismatch on aggregate — a signal aggregating a mg/L column and declaring unit: g/L output without a UCUM conversion in the aggregate spec → refused.
  2. Dimensional incompatibility — arithmetic that would subtract a temperature from a mass, or add currency to a duration → refused (the ISO 80000 quantity-kind lattice tells the checker the operation is undefined).
  3. Method mismatch — a threshold rule written against ISO 7027 turbidity applied to a column that doesn’t declare standard: iso_7027 → warned (the Codex binding_kinds field says the standard is legitimately bindable; the column just doesn’t cite it).
  4. Broken FK inheritance — a signal joining on sensor_id where the entity YAML declares the FK actually references a different table → refused (the identifier.foreign_key subkind carries its target; the checker walks it).
  5. Cross-pack unit drift — a mass_g column in one pack and a mass_kg column in another compared or subtracted without conversion → catchable at cross-pack composition (both packs declare units; the checker computes the compatibility).
  6. Uncertainty-drop violation — a signal aggregating measurement columns without carrying the paired uncertainty column through per JCGM 100:2008 → warned (the uncertainty_column: <name> link on the primary quantity tells the checker what to expect).

All six are pure compile-time work. Zero database growth. Zero runtime cost. Every one of them is what a scientific reviewer would ask about a KLS delivered to them — “how do you know the units are consistent? how do you know the joins didn’t drop uncertainty?” — and the answer becomes “the build refused the wrong versions.”

  • Not a rewrite of Sense 14.2. The typed-signals machinery keeps working. The change is that the signal YAML gets shorter because the type is inherited from the column instead of restated.
  • Not a rewrite of Sense 53. The Codex hosts the standards regardless of whether any column binds to them. The Typology is one reader of the Codex; SensorThings API endpoints, Bell disclosures, and JinDesk standards pages are other readers.
  • Not a display concern. The Sense 43 display: cascade keeps its role: precision, format, locale. The Typology says what a column is; display: says how to render it in this locale. Two vocabularies, one cascade shape.
  • Not a validation framework. The Typology declares. It doesn’t check that a value in the Gold table conforms to its declared type — that’s a downstream job for Silver validation, dbt tests, or jin doctor. Declaration and validation are separable, and mixing them is what has bitten every schema system that ever tried.
  • Not a separate coverage-scoring Sense. A briefly-considered Sense 55 “Bedrock” would have introduced a per-signal grounding grade (dimension + unit + method standard + range + uncertainty). Sister-9 retired that proposal during this Sense’s review (2026-07-11) because it was the same insight one level below its natural home: the coverage rollup lives in the Typology, and a signal grounding grade is mean(column_grades). Same result, cleaner home, one fewer Sense.
  • Every adopting pack walks its Gold columns and assigns types. For a 200-column pack, this is 200 declarations. Realistic estimate: 8–16 hours per pack for a first pass, mostly clerical, guided by name-based defaults.
  • Two new compiler steps — typologycheck.py (reads declarations, walks the processing graph, refuses illegal operations) and typologycompile.py (writes state/typology_coverage.json and logs the write to the Sense 47 Log). Both mirror the shape of signalcheck/signalcompile. Neither writes anything to the KLS.
  • One new dbt macro — column_type(entity, column) — so Silver validators can {{ column_type('sensors', 'last_seen_at') }} to read the declared type at build time.
  • JinDesk refactor to consume the typed contract instead of per-column display config. Modest, because the display cascade is already the reader; only the source of the type field moves.
  • Signal YAMLs shrink 15–25% because types are inherited.
  • Cross-pack composition becomes safe — mass_kg in riverflow and mass_kg in numetrix are known to be the same quantity kind and the compiler can prevent an accidental subtraction of temperature from mass.
  • JinDesk’s unit-aware rendering, Codex badge display, and precision formatting all become automatic instead of hand-authored per column.
  • “Does jinflow speak SI?” gets a per-column answer, not a per-pack marketing claim. Every quantity column that binds to UCUM contributes to the platform’s assayed surface.
  • The KLS becomes metrologically self-describing. A KLS shipped to a scientific reviewer carries not just the values but the vocabulary that makes them values.

Phase 1 — Declaration site + coverage artifact. Add the column_types: block to the entity YAML schema. Ship typologycheck.py (validation) and typologycompile.py (writes state/typology_coverage.json, logs to Sense 47). No behavioural changes downstream — this is observation, not enforcement. Ship name-based zero-day defaults so packs open Phase 2 with 40% coverage for free. Nothing lands in the KLS. (Superseded 2026-07-12 — see § Phase 1 as shipped: typologycompile bakes two inspector-only KLS tables (not a JSON side-file); there are no name-based defaults (explicit only); pre-flight is fail-hard, not observe-only.)

Phase 2 — Signal inheritance. Modify signalcompile.py so aggregate: {column: X} (without explicit type or unit) reads the declared type from the Typology. Existing signals with explicit types continue to work; the compiler warns when the explicit type disagrees with the declared type.

The Phase 2 stress test is riverflow’s RegulatoryThreshold entity (landed on claude/9th 2026-07-11): a threshold_value column whose unit varies by row context — g/L for SSC, mg/L for DO, Cel for temperature, 1 for pH. Sense 14.2 Phase 5 could not express this and had to drop the entity-level aggregate: block. A Typology declaration per row context (probably a unit_column: unit_of_measure field on the type declaration, indirecting through the sibling unit_of_measure column) is the natural expression. If Phase 2 does not gracefully handle “type varies by row context”, we haven’t solved the composition problem.

Phase 3 — JinDesk inheritance. JinDesk’s rendering path reads Typology first, display: cascade second. Pack-level display: blocks for column-level precision/format become deprecated in favour of the Typology declaration.

Phase 4 — Strict mode. Per-entity typology: strict toggle. Compiler fails if any column in that entity lacks a declaration. Adoption is pack-by-pack, entity-by-entity, with coverage metrics visible throughout.

Phase 5 — Codex enforcement. Types that name a Codex standard get checked — a quantity column with unit: mg/L, standard: ucum gets validated against the UCUM parser at compile time. Column-level Codex conformance appears in the Bell and on the Landscape.

A person reading the Typology should be able to answer, of any Gold column, three things in under a minute:

  1. What kind of value is this?
  2. What are its units, standards, and uncertainty policy, if any?
  3. Where in the AFS is that declaration written?

If the answers exist and are pointer-verifiable to a YAML file, we have a Typology. If they exist only in the author’s head, we have a schema pretending to be one.

The Codex made jinflow’s external vocabulary legible. The Typology makes jinflow’s internal vocabulary legible. Together they close the loop.

Evolution — the stream-invariant model (2026-07-12)

Section titled “Evolution — the stream-invariant model (2026-07-12)”

This section is the outcome of a working session with Mig on 2026-07-12. It refines the model above and, where it conflicts with earlier sections, it is authoritative. The most-affected earlier passages carry inline “Superseded 2026-07-12” markers pointing here; they are retained for history.

A semantic type is not a label on a Gold column at one site. It is the identity of a data stream — its meaning, dimension, and unit — that is invariant under storage and changes only under an explicit, declared conversion. It is not a fact about one place in the pipeline; it is a property that travels with the stream through every stage of the river.

This reframes the whole Sense. Typing a Gold column is not the goal — it is one of two gates on a stream whose type we track end to end.

2. Two type layers — and the storage type is the floor

Section titled “2. Two type layers — and the storage type is the floor”

There are two distinct type layers, complementary, not competing:

LayerWhereWhat it saysScope
Storage typecsv_schemas/{source_system}.ymlstring / number / date / boolean / integer — how it’s encodedper source system
Semantic typethe Typology (column_types:)quantity+UCUM+dimension, identifier+FK, category+vocab, temporal — what it meanssource-agnostic

The storage layer already exists: csvschemacheck.py validates source CSVs against the declared schema (L0 existence → L1 syntax → L2 headers), and that schema already carries a type: per column — a storage type. The semantic type sits on top of the storage type as its floor: a quantity must rest on a number/integer, a temporal on a date, a boolean on a boolean. A semantic kind on an incompatible storage type is a compile error — make stops. (Storage type is per-source because encoding varies; semantic type is source-agnostic because meaning does not — a concentration is a concentration whether OPALE or SAP delivered it.)

3. The type lives at both gates — and the redundancy is the safety

Section titled “3. The type lives at both gates — and the redundancy is the safety”

The semantic type is declared at two gates, and carried through the stages between:

  • At ingestion (earliest possible) — a prerequisite to entering the pipeline. “This stream delivers °C / counts / meters.”
  • At Gold — a prerequisite to every instrument we apply. “This column is SSC in g/L, averageable.”

Between the gates is the river, where conversions happen (raw counts → NTU → g/L; m → ft; m × m → m²). Declaring the type at both gates looks redundant — and in the best case it is. But the redundancy is a checksum: the source type carried through the declared conversions must equal the independently declared Gold type. When they agree, all is well. When they disagree, the mismatch localizes the conversion bug to the exact stage where the carried type stopped matching — and make stops there and names it. Data that never reaches Gold still needs a type: an untyped intermediate feeds a derivation that does reach Gold, and that is precisely where silent conversion errors hide today. Typing as early as possible is what makes the failure loud and local instead of a suspicious number three layers downstream.

4. Conversions are typed edges — explicit only, no build-time magic

Section titled “4. Conversions are typed edges — explicit only, no build-time magic”

Each conversion is a typed edge: a declared operation taking a stream of type A to type B by a known rule — a commensurable unit conversion (m→ft), a standard-cited calibration that legally changes the dimension (counts→NTU via ISO 7027), or a dimensional derivation (m×m→m²). Because the type is known as early as possible, a conversion is type-driven and type-checked rather than a hand-coded transform that is silently wrong. °C + m is refused.

No name-based inference anywhere in the pipeline. “The column ends in _km2, so it must be km²” is forbidden — it is lying with a heuristic, right until someone names a column depth_km. This retracts the “zero-day name-based defaults” of the migration ramp and Phase 1 below (ends_with('_at') → temporal, “40% for free”), and it retires the pack config.yml unit-guessing regex. Magic is allowed only when authoring a schema — a scaffolding tool may propose “area_km2 looks like a quantity in km²” for a human to review and commit. Once a schema is injected into make, it is read literally; there is zero inference at build time. Suggestion-time magic ≠ build-time magic. This is how trust and safety are maintained: the truthful space is fixed, explicitly, in a reviewed schema, before a single row is built. make is the analytical security boundary, and it stops on any violation (fail-hard, like the extraction contract).

5. Type is not configuration (the Sense 43 boundary)

Section titled “5. Type is not configuration (the Sense 43 boundary)”

The Cascade (Sense 43) is a config engine: values produced and consumed independently, contextual, overridable, cascaded L0→L3. Type is the opposite. Type and data are one — you cannot override a declared type any more than you can relabel °C as °F; changing it means it is different data. Type is upstream of the config engine and never enters it. This supersedes the cascade section’s “tenant may override a declaration” — overrides apply to display config, never to type.

What config may touch are the truthful leftovers — you can only “configure the truth”: a render-unit via a commensurable conversion (show m³/s as L/s), precision, and at most a choice among commensurable legal aggregations (mean vs weighted-mean) — never whether an illegal one is allowed. The type carries the dimension; the dimension bounds exactly which configurations are truthful. Illegal moves are refused, not warned: relabel-without-convert, incommensurable render (“length in seconds”), illegal rollup (a concentration as additive).

The measure roll-up policy is a type property, not config. Whether a value may be summed is a fact of the quantity — extensive (count, load → summable) vs intensive (concentration, temperature → not). It is declared once, truthfully, on the type, immutable, and does not cascade. It is named rollup: (vocabulary additive | averageable | non_additive | distinct_count) — not aggregation:, to avoid collision with Sense 43’s documented config-merge “aggregation policy” (replace/union/append/concat, how config keys merge across cascade levels). Different beast.

Complete type coverage is a data-engineering requirement, not a metric. “There is only one possible unit for a stream that delivers °C.” An untyped Gold column is unfinished data engineering, surfaced loudly; strict mode (Phase 4) refuses to build an entity below full truthful coverage.

6. Where the artifacts live — KLS inspector tables, not a JSON side-file

Section titled “6. Where the artifacts live — KLS inspector tables, not a JSON side-file”

This supersedes the state/typology_coverage.json model throughout the sections above. The load-bearing principle still holds — nothing in processing reads a type — but the artifact moved, for a concrete reason: the cloud JinDesk has no AFS on hand, so a git-tracked side-file cannot reach a browser in Frankfurt. typologycompile bakes two tables into the tenant meta schema:

  • _<tenant>.column_types — one row per declared column (entity_id, dbt_model, column_name, kind, subkind, standard, unit, unit_column, rollup, ref_target).
  • _<tenant>.typology_coverage — one row per Gold entity, the discipline gauge.

These are debug symbols (DWARF), not runtime type tags (RTTI). Nothing in compile / dbt / signal processing may read them — types are inherited at compile time; these tables exist only so JinDesk (an inspector) can show a declared type and derive presentation. The moment a signal reads column_types to decide a type at query time, it becomes the runtime-type-info model we avoid. The guard is loud in the bake code and in a DuckDB COMMENT ON TABLE. They are write-once, rebuilt every make. This is the same pattern as tenant_standards (a497231d): descriptive metadata the cloud viewer needs travels in the KLS meta schema. (Reconciles with §“types are declarations, not runtime data”: processing never reads these; they are inspector-only.)

Coverage is reconciled against the actually built Gold columns (read from information_schema in the KLS the compile is writing into), so it reports the honest gauge — typed / untyped / phantom (declared-but-absent, usually a typo) — not the declaration alone.

  • column_types: on the three testbed entities (34 columns).
  • typologycheck.py — top-level kind validity + Codex binding_kinds conformance + rollup vocabulary. Explicit-only, no name inference, fail-hard.
  • typologycompile.py — bakes the two inspector-only tables; coverage reconciled against built Gold; pre-flight aborts on any invalid type (never bake a lie).
  • ucum widened to [quantity, category] (56586a2b, sister-9) so unit-of-record columns (result_unit, unit_ucum) bind legally.

The typed river — type carried through every stage, conversions as typed edges, make refusing any edge that does not preserve or legally transform the type — is the North Star. It is the same dimensional-propagation machinery already flagged for derived columns (§ derived-unit computation), applied stream-wide. Phase 1 anchors the Gold gate and the reconciliation floor; Phases 2–5 build outward toward the full river.

The V1 type_conversions edge is per-(entity, column) with a single standard — exact for a mono-property column, but result_value is polymorphic (§2 / unit_column), and its conversion method varies by observed property just as its unit does: turbidity’s L1→L2 hop is ISO 7027 (nephelometry), a temperature reading’s is a thermistor calibration — not nephelometry. So a single-standard edge tags every result_value chain with iso_7027: correct for turbidity, wrong for temp/level. (sister-9 flagged this after wiring the Rail to the edge table, 117aa460; the demo host seeds a turbidity reading, so it renders correctly today.)

The gap is the same polymorphism that unit_column bears. The discriminator is the general fix; reshape is a special-case simplification — and the platform supports both (polymorphism is first-class; normalize only where it’s reasonable — see feedback_polymorphism_and_normalization_both). The team converged here (sister-9 + Mig, 2026-07-13):

  • The discriminator (general). A per-observed_property variant→standard map on derived_from, keyed by the same column unit_column leans on (observed_property_id). Kept pure declaration (a declared map, not a gold_observed_properties data-join) so the edge table stays checksum-grade. Correctness first, citations later: the first win needs no new Codex — map op_water_temp → standard_ref: null so temp/level stop wrongly citing ISO 7027. A thermistor calibration may have no governing external standard, and null is the honest answer; positive citations (stage-discharge → USGS rating methods) are seeded as they exist.
  • Reshape (the special case, not a replacement). Splitting result_value into per-property columns dissolves the discriminator — but only when the discriminator has a small, fixed domain, and it walks away from the OGC SensorThings Observation model riverflow deliberately chose. Currency is the counter-proof: an open domain with time-indexed FX (amount + currency) can never be reshaped into amount_usd / amount_eur / … — you keep (amount, currency) and discriminate. So reshape is the exception; the discriminator is the rule.

A layer separation to keep straight. A refinement edge is a cross-dimension conversion (counts→NTU→g/L), method-cited, a type_conversions row. A currency conversion (CHF→EUR) is same-dimension, rate-based, time-indexed — it is a render-time data join (a rate table, the same shape as gold_calibrations), not a type-level standard and not an edge. Money’s FX pressures the data/join layer, never the edge-standard design.

The one genuine hinge is the riverflow reshape timeline — the pipeline owner’s call, not the Typology’s. If a reshape is imminent for other reasons, the discriminator’s positive-citation work is throwaway (the null-correctness fix is not — nulling a lie is honest regardless of shape). Until then, the V1 edge is documented (in the riverflow declarations) as turbidity-scoped.

  • [[sense_14_2_typed_signals]] — the aggregate-boundary types this Sense pushes down to the substrate
  • [[sense_53_the_codex]] — the platform’s standards registry, which the Typology references (and which gains a binding_kinds field to make bindings compile-time-checked)
  • [[sense_43_the_cascade]] — the cascade machinery types will inherit through
  • [[sense_15_the_observation]] — the “signed strategic claim” idea; this Sense signs every column with its type
  • [[sense_47_the_log]] — the write-once execution log that records typologycompile runs and gives the coverage artifact its audit chain
  • [[sense_52_the_cartography]] — the geometry-serialization edge case (RFC 7946 GeoJSON) that this Sense defers to Cartography Phase 2
  • [[project_unit_standards_architecture]] — three-level unit standards (ISO 80000, UCUM, quantity kinds) that the quantity kind builds on
  • [[project_four_convictions]] — the philosophical frame this proposal lives under

Recently landed Codex standards that become Phase 1 binding targets (all on claude/9th, 2026-07-11):

  • ISO 7027 — turbidity nephelometry (quantity.mass_concentration in NTU or FNU)
  • ASTM D3977 — suspended sediment concentration gravimetric (quantity.mass_concentration in mg/L)
  • WMO Guide No. 8 — meteorological instrument conventions (air temperature, relative humidity — deferred to a future riverflow phenomenon)

Numerical neighbors: ← Sense 53 — The Codex · Sense 55: The Broadcast →

jazzisnow jinflow is a jazzisnow product
v0.65.0 · built 2026-11-13 10:20 UTC