Playworks — data-aware browser tests

Playworks is jinflow’s browser-driven test harness for JinDesk. It runs Playwright against a live JinDesk instance (local or cloud) and asserts the rendered app matches the data inside your KLS.

The tests are data-aware: a manifest compiled from your AFS + KLS lists every signal, thesis, route, and oracle count. The tests read that manifest. Adding a new signal in YAML, running jin make, and re-running Playworks gives you one more test scenario for free — no test code changes needed.

The shape of the harness

AFS + KLS  ─(jin make)─▶  *.playworks.json (sibling to KLS)
                                   │
                                   ▼
                       ┌───────────┴───────────────┐
                       │  generated/routes.spec    │  HTTP 200 per route
                       │  generated/cardinal.spec  │  KLS counts match manifest
                       │  generated/ui-cardinal    │  rendered DOM matches oracle
                       │  authored/*.spec          │  hand-written invariants
                       └───────────────────────────┘

The two-track model

Playworks has two kinds of specs, and you’ll grow both as the project matures.

Generated specs — manifest-driven

Three suites, all in explorer/playworks/generated/:

Suite	What it asserts
routes.spec.ts	Every declared route returns HTTP 200. Catches renamed routes, data that crashes a detail page, server 500s on specific entity ids.
cardinal.spec.ts	The KLS row count for each signal-findings table matches what the manifest recorded at build time. Catches manifest/KLS drift when a KLS is rebuilt without regenerating the manifest.
ui-cardinal.spec.ts	The rendered DOM (e.g. the “27 findings” pill on a signal page) matches the oracle. Catches the case where the data is right but the UI mis-reads it.

These suites scale automatically with the manifest:

• Today’s hrcentral.vai build emits 40 signals + 12 theses + 798,146 findings.
• That gives 159 route checks + 59 ui-cardinal checks against hrcentral.vai.
• A smaller pack like hrcentral.vai emits 34 routes + 10 ui-cardinals.

You don’t write these tests — you ship signals and they appear.

Authored specs — hand-written invariants

In explorer/playworks/authored/. Each file scopes to a feature or vocabulary the generator can’t reason about: filter pills, keyboard shortcuts, sidebar order, regression-class bugs. Examples from the current suite:

• findings-numetrix.spec.ts — polarity pill clicks, R/O/N keyboard shortcuts, whole-page slicing.
• instruments-numetrix.spec.ts — a before/after canvas-buffer compare that catches the “thesis-class regression” pattern (silent layer-toggle no-op).
• sidebar-numetrix.spec.ts — section order: Strategy → Model → Expertise → Tools → Backstage.
• board-numetrix.spec.ts — Lens-browser smoke: page mounts, tabs render, canvas accepts clicks without crashing.
• anchors-numetrix.spec.ts — pack-wide vocabulary anchors (no legacy “Probe / Hypothesis / Diagnosis” labels survive).

Authored specs live close to the feature they cover, use locale-tolerant matchers, and prefer URL state and structural CSS over Tailwind class names.

Tagged subsets

Both tracks can be tagged. Two are in use today:

Tag	Cadence	Purpose
@canary	Post-deploy fast gate (≤ 60s)	The smallest set that proves “the deploy didn’t break the boring stuff.” Currently 35 tests.
@visual	On-demand + nightly	Visual baselines (screenshots committed to git). Slow + flaky for fast gates; runs separately.

Run a tagged subset:

npm run playworks:canary       # only @canary
npm run playworks:visual       # only @visual

The invitation canary (opt-in — costs a Clerk MAU)

One @canary spec is special: the invitation-acceptance flow (invitation-flow.spec.ts) authenticates a real user against the PROD Clerk instance and fires a real invitation email. Clerk bills by monthly active user, so every run is a billable +1 PROD MAU (the spec deletes the user afterward, but that is hygiene — it does not un-count the month’s MAU).

It is therefore off by default and never rides along with routine smoke. It runs only when explicitly opted in:

just smoke --invitation          # the deliberate PROD-auth check (+1 MAU)
PLAYWORKS_INVITATION_FLOW=1 …     # local iteration against a dev server

The intended cadence is nightly (a scheduled just smoke --invitation). Every other @canary spec runs anonymously against the demo surface and costs nothing.

Running locally

From explorer/:

# 1. Compile the manifest (auto by `jin make`; manual if you skipped that step)
python3 -m jinflow.playworks --tenant numetrix.inspire

# 2. Install Playwright browsers (one-time setup)
npx playwright install chromium

# 3. Start the dev server with the SAME tenant attached
just dev-solo numetrix.inspire   # avoids the multi-tenant SIGBUS — see "Known issues"

# 4. In another terminal, run Playworks
npm run playworks:local

Or against the deployed JinDesk:

npm run playworks:cloud

Or fully custom:

PLAYWORKS_BASE_URL=https://staging.jinflow.io \
PLAYWORKS_TENANT=hrcentral.vai \
npm run playworks

Per-tenant scenario counts

The number of scenarios depends on the active tenant’s manifest. Today’s totals:

Tenant	Total scenarios	@canary
hrcentral.vai	270	35
numetrix.rmc	206	35
numetrix.inspire	191	35
hrcentral.vai	96	35

Authored specs (51 today) are constant across tenants and skip with a clear message when the active tenant’s pack doesn’t match (anchors-numetrix.* skip on hrcentral.vai, etc.). The variation is in the generated suites.

Environment variables

Variable	Default	Purpose
PLAYWORKS_BASE_URL	http://localhost:4000	Where to point Playwright
PLAYWORKS_TENANT	numetrix.inspire (via npm scripts)	Composite id; drives manifest + tenant-aware specs
PLAYWORKS_MANIFEST	auto from tenant	Explicit path to *.playworks.json
PLAYWORKS_ORACLE	auto from manifest	Explicit KLS path for cardinal tests (read-only open)

The ?playworks=1 flag

JinDesk’s tenant layout cooperates with Playworks. When the URL carries ?playworks=1, the layout:

• Suppresses WelcomePrompt (the modal that asks “what’s your name”).
• Suppresses PackSplash (the brand-color overlay on first load).
• Suppresses the early-boot #jinflow-loading splash.
• Stamps data-app-ready="1" on <body> once hydration finishes.

The playworksPage fixture in playworks/fixtures/playworksPage.ts appends the flag automatically and waits for the marker. Without it, every spec would re-implement the same 10 lines of “dismiss the splash, seed identity, wait for ready.”

Adding a new authored spec

The minimum:

playworks/authored/my-feature.spec.ts

import { test, expect } from '../fixtures/playworksPage.js';

const TENANT = process.env.PLAYWORKS_TENANT ?? 'numetrix.inspire';
const isNumetrix = TENANT.startsWith('numetrix.');

test.describe('My feature', () => {
  test.skip(!isNumetrix, 'tenant is not numetrix; skipping');

  test('does the thing @canary', async ({ playworksPage }) => {
    test.setTimeout(45_000);
    await playworksPage.go(`/${TENANT}/findings`);

    // Behavior, not implementation:
    await playworksPage.page.getByRole('button', { name: /Risk/ }).click();
    await playworksPage.page.waitForFunction(
      () => /[?&]polarity=negative/.test(window.location.search)
    );
  });
});

Five conventions worth keeping:

1. Skip cleanly when the active tenant is wrong (test.skip(!isNumetrix, ...)).
2. Use the playworksPage fixture, not bare page — it handles the splash + identity dance.
3. Prefer URL state and getByRole over CSS class names — survives Tailwind refactors.
4. Locale-tolerant matchers — translated labels are the normal case.
5. @canary only the highest-leverage tests. A 60s post-deploy gate dies if you canary-tag everything.

What Playworks doesn’t do

Worth being explicit:

• No unit-level component tests. That’s vitest territory (see *.test.ts colocated with source).
• No engine / CLI tests. That’s pytest in tests/.
• No SQL / dbt model tests. That’s dbt test (or, increasingly, the cardinal.spec.ts pattern that asserts via DuckDB read).

Each test type has a parallel harness in jinflow’s roadmap — Cliworks for the CLI, Proxyworks for the tunnel, Evolveworks for the AI surface — but the principle stays: data-aware over schema-aware, behavior over implementation.

Known issues

duckdb-async SIGBUS under sustained load

When the dev server holds attachments to multiple tenants in one Node process and Playwright drives ~140+ queries through it, the duckdb-async native addon can SIGBUS. Symptoms: the dev server crashes mid-suite; ERR_CONNECTION_REFUSED on subsequent specs.

Workarounds:

• just dev-solo <tenant> — start the dev server with a single tenant attached.
• For local CI matrices: spawn one dev server per tenant in separate processes.
• Cloud canary doesn’t hit this — each Fly instance serves one tenant via R2.

Manifest staleness

If you jin make produces a new KLS but the manifest sibling didn’t regenerate (e.g. a partial run, or a stale *.playworks.json from an older build), cardinal.spec.ts fails first with a precise mismatch. Re-run jin make to get a fresh paired manifest.

Related

• Make (Build) — what produces the manifest
• Diagnostics — the other half of “is this build healthy?”
• Findings — filter and navigate — exercised heavily by findings-numetrix.spec.ts