diff --git a/.github/scripts/drill-recorded.sh b/.github/scripts/drill-recorded.sh index 514e986..876e547 100755 --- a/.github/scripts/drill-recorded.sh +++ b/.github/scripts/drill-recorded.sh @@ -1,8 +1,8 @@ #!/usr/bin/env bash set -euo pipefail -# drill-recorded.sh [] [] — assert that the version -# this tree claims to ship has a DRILL RECORD in drill/RUNS.md. +# drill-recorded.sh [] [] — assert that the version +# this tree claims to ship has a DRILL RECORD at /.md. # # CONTRIBUTING says a release carries the full real-hardware drill. Nothing # checked that, so no release in this family ever carried one: the step lived @@ -12,44 +12,81 @@ set -euo pipefail # the gate moves into CI, where it is asserted on every release PR rather than # recalled. # +# ONE FILE PER VERSION — WHY THE PARSER IS GONE +# +# The first cut kept every record in one drill/RUNS.md and asked awk which +# section belonged to this version. That bought a heading grammar: em-dash +# field matching, an optional ' — DATE' tail, a whole-version comparison so +# 0.2.0-rc1 could not satisfy 0.2.0, a '(NF == 5 || $6 == dash)' tail +# constraint to stay in step with box's twin, and a non-blank body rule. +# +# All of it existed ONLY because records shared a file — and in review this +# repo shipped two defects out of that complexity: a `sed '/./,$!d'` +# extraction where `.` matches a space, so a heading followed by one tab +# satisfied the gate; and heading-grammar drift from box's stricter form. +# Two defects, on the one check whose entire job is to demand evidence. +# +# One file per version makes nearly all of it UNREPRESENTABLE. `0.2.0.md` and +# `0.2.0-rc1.md` are simply different files — the whole-version rule is the +# filesystem's, not a comparison anyone can get wrong. There is no heading to +# parse, so there is no grammar to drift from box's. What is left is a +# question a shell can ask directly: does the file exist, and does it say +# anything. +# +# The directory is plain `drills/`, NOT `.drills/`. Dot-prefixed directories +# are invisible to globs without `dotglob`, which is the exact blind spot that +# produced #118, #121 here and box#116 — a sweep that looks green because it +# never descended into the directory holding the thing it was meant to check. +# # WHAT IT ASSERTS, AND WHAT IT DELIBERATELY DOES NOT # # It asserts a RECORD EXISTS — not that the drill passed. That is the whole # design. A maintainer may ship on a failed or partial drill; what they may not # do is ship on silence. Requiring a record makes a waiver a deliberate, -# reviewable commit (a section saying who waived it and what is untested) -# instead of the default outcome of forgetting. A guard that demanded a PASS -# would be argued with and eventually bypassed; one that demands EVIDENCE has -# nothing to argue about. +# reviewable commit (a file saying who waived it and what is untested) instead +# of the default outcome of forgetting. A guard that demanded a PASS would be +# argued with and eventually bypassed; one that demands EVIDENCE has nothing to +# argue about. # # PER-REPO, ON PURPOSE # -# This reads cast's OWN drill/RUNS.md. It does not reach into box or rig to ask +# This reads cast's OWN drills/. It does not reach into box or rig to ask # whether the family drilled. A cross-repo lookup has a failure mode this repo # keeps refusing: when the fetch fails — no network, moved file, renamed repo, # a token without read on the other repo — the honest answers are "unknown" and # "blocked", but the shape such code actually takes degrades to "pass". Same # class as the unreadable check rollup that read as "nothing is failing". # -# There is also nothing to look up. The drill is ONE orchestrated run over the -# whole stack (CONTRIBUTING.md, "Releasing"): rig bootstraps the host and -# installs box, box mints a seed, the seed calls rig back to converge, and -# cast's legs run on the result. rig therefore sits BELOW box and ABOVE it — -# the three repos are mutually recursive, not linearly ordered, and their -# releases are NOT published in a fixed sequence. The run pins CANDIDATE refs -# (RIG_REPO/RIG_REF at mint time), so no repo must ship before another can be -# drilled. Each repo records its own legs from that run, citing the shared run -# ID and the other repos' SHAs — which is how three records reassemble into one -# run without any repo reading another's file. +# There is also nothing to look up. The three repos' drills are INDEPENDENT +# (CONTRIBUTING.md, "Releasing") — run in any order, on any schedule, in +# separate sittings. What makes that safe is that every drill pins the SAME +# FIXED SET OF CANDIDATE REFS (RIG_REPO/RIG_REF at mint time), so each one +# exercises the combination that will ship rather than whatever main happens +# to be that afternoon. +# +# That pinning, not sequencing, is what dissolves the box<->rig recursion. box +# and rig ARE mutually recursive — rig builds the host that runs box, box's +# seed calls rig back to converge the guest — but candidate refs are static +# identifiers that exist as soon as the release branches do, long before any +# drill runs. A cycle at runtime becomes independent tests against one fixed +# pair, and no repo must ship before another can be drilled. The three +# releases are NOT published in a fixed sequence. +# +# Each repo also drills a DIFFERENT thing: box asserts the isolation contract, +# rig asserts convergence, cast asserts promotion. Three different exercises +# over a shared substrate — which is exactly why the records are per-repo. +# Each cites the shared run ID naming the pinned set, plus the other repos' +# SHAs, so three records still reassemble into one picture without any repo +# reading another's file. # # A file of its own, not a clause inlined in ci.yml, for the same reason as # release-notes.sh and changelog-monotonic.sh: test/release.test.ts drives the # REAL script against fixtures, so what the tests prove is what CI runs. -runs="${1:-drill/RUNS.md}" +drills="${1:-drills}" version_file="${2:-package.json}" -[ "$#" -le 2 ] || { echo "usage: drill-recorded.sh [] []" >&2; exit 2; } +[ "$#" -le 2 ] || { echo "usage: drill-recorded.sh [] []" >&2; exit 2; } [ -f "$version_file" ] || { echo "drill-recorded: no such file: $version_file" >&2; exit 1; } # cast's version lives in package.json (there is no VERSION file), so this @@ -76,69 +113,34 @@ esac # A bare version is a release ceremony tree: this is the tree whose merge IS # the release, so this is where the evidence has to exist. -[ -f "$runs" ] || { - { - echo "drill-recorded: version $ver is a release, but there is no $runs at all." - echo - echo " The release drill has to be RECORDED, not just performed. Create the file" - echo " with the run's section (see the format below) and commit it." - } >&2 - exit 1 -} +# +# The record is /.md, and it must contain at least one +# NON-WHITESPACE character. That second clause is the one surviving piece of +# the whitespace defect found in review (#138): a file of only spaces, tabs +# and newlines is a file, and `[ -f ]` is happy with it, but it is not a +# record — an evidence-free release for the price of an invisible character. +# `grep -q '[^[:space:]]'` asks the question the old `sed '/./,$!d'` only +# claimed to: `.` matches a space, a POSIX class does not. +record="$drills/$ver.md" -# $5 of a drill heading ('## Release drill — 0.2.0 — 2026-07-21') is the bare -# version, compared WHOLE — so 0.2.0 can never be satisfied by a 0.2.0-rc1 -# section, or the reverse, and no regex-escaping of dots. Exactly the trap -# release-notes.sh solves the same way, with the same awk field split, so the -# two cannot drift apart about what a heading is. The trailing ' — DATE' is -# optional and unread: the gate is about the version, and the date is for the -# humans reading the log. -# -# grab is re-armed by every '## ' line, so the section ends at the next one — -# a record cannot borrow the body of the record below it. -# -# `grab && NF` is the non-blank rule, and it is load-bearing rather than -# tidiness. NF is 0 on a line that is empty OR contains only whitespace, so -# `record` is non-empty exactly when a line with real content exists. The -# first cut of this piped through `sed '/./,$!d'` and the comment here claimed -# "a heading with nothing but whitespace under it extracts to the empty -# string" — which is precisely what that pipeline did NOT guarantee, because -# `.` matches a space. A heading followed by one tab satisfied the gate. The -# comment documented the intended contract and the code did not meet it, which -# on a gate is the whole ballgame: an evidence-free release for the price of an -# invisible character. Found by all three reviewers on #138, independently. -# -# The '(NF == 5 || $6 == dash)' tail constraint keeps this in step with box's -# twin: without it '## Release drill — 0.2.0 stray words' counts as a record -# here and does not there. Two sibling guards disagreeing about what the same -# heading means is the same trap as disagreeing with release-notes.sh. -record="$(awk -v ver="$ver" -v dash="—" ' - /^## / { - grab = ($2 == "Release" && $3 == "drill" && $4 == dash && $5 == ver \ - && (NF == 5 || $6 == dash)) - next - } - grab && NF { print } -' "$runs")" - -if [ -z "$record" ]; then +if [ ! -f "$record" ] || ! grep -q '[^[:space:]]' "$record"; then { - echo "drill-recorded: $runs has no drill record for version '$ver'." + echo "drill-recorded: version $ver is a release, but there is no drill record at $record." echo - echo " A release PR's version must have a NON-EMPTY section headed:" + echo " A release PR's version must have a NON-EMPTY file named for it:" echo - echo " ## Release drill — $ver — YYYY-MM-DD" + echo " $drills/$ver.md" echo - echo " (An empty section under a correct heading counts as no record. The" - echo " heading matches the version WHOLE — a '$ver-rc1' section does not" - echo " satisfy '$ver', and vice versa.)" + echo " (A file that exists but holds only whitespace counts as no record." + echo " One file per version, so '$ver-rc1.md' is a different record and" + echo " does not satisfy '$ver', or the other way round.)" echo echo " To unblock, either:" echo " * run the drill and record it — the legs (team, apply, idempotent" echo " diff, smoke, inventory, emit-draft, fleet, destroy, read-only" echo " guard), the numbers, and what failed; or" - echo " * record an explicit maintainer WAIVER for this version in $runs," - echo " saying who waived it and what is untested." + echo " * record an explicit maintainer WAIVER for this version in that" + echo " file, saying who waived it and what is untested." echo echo " The waiver is allowed on purpose: this gate requires a RECORD, not a" echo " passing result, so shipping without a drill stays possible — and" @@ -147,4 +149,4 @@ if [ -z "$record" ]; then exit 1 fi -echo "drill-recorded: $runs carries a drill record for $ver ($(printf '%s\n' "$record" | grep -c .) line(s))" +echo "drill-recorded: $record carries a drill record for $ver ($(grep -c '' "$record") line(s))" diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 48a9705..79bad59 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -85,8 +85,8 @@ jobs: # drill went missing in the first place. # # It requires a RECORD, not a PASS: a maintainer waiver is legal, and is - # itself a section in drill/RUNS.md. Skipping stays possible and stays - # visible. + # itself the content of drills/.md. Skipping stays possible and + # stays visible. - name: a release version has a drill record run: bash .github/scripts/drill-recorded.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index ed1380a..ff940e3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,7 +9,7 @@ actually cutting it, and this file starts there. ### Added -- CI refuses a release PR with no drill record in `drill/RUNS.md` +- CI refuses a release PR with no drill record at `drills/.md` - An application can declare HTTP basic auth, and `apply` sets it (#76) - `cast github-app create` / `cast github-app register` run the App Manifest flow (#7) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d7eacb7..c6e028a 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -117,10 +117,14 @@ on box#83's shape): 2. **Drill, and record it.** Before the PR can be handed over, run the full real-hardware drill — two live Coolify instances, the whole A→B promotion: team, apply, an idempotent diff, smoke, inventory, emit-draft, fleet, - destroy, and the read-only guard — and record it in - [drill/RUNS.md](drill/RUNS.md) under a heading naming the version: + destroy, and the read-only guard — and record it in a file named for the + version, one record per version: - ## Release drill — X.Y.Z — YYYY-MM-DD + drills/X.Y.Z.md + + The name matches `package.json`'s `version` exactly, and the file must hold + at least one non-whitespace character. See + [drills/README.md](drills/README.md) for what a record contains. [.github/scripts/drill-recorded.sh](.github/scripts/drill-recorded.sh) enforces this on every release PR (a `-dev` tree has no ship claim and @@ -131,46 +135,56 @@ on box#83's shape): So the release flow is: **draft → ready → bot round → drill → `state:needs-human` → maintainer merge (which IS the release).** - **The drill is ONE orchestrated run over the whole stack**, not three - independent exercises. In order: + **The three repos' drills are independent.** Run them in any order, on any + schedule, in separate sittings. They are not three phases of one script. - 1. `rig bootstrap … --host yes` on a bare Debian host — this installs box - and runs box's `setup-host`; - 2. `box new` mints a creds-free seed; - 3. the seed converges on first boot: its cloud-init curls rig's installer - and runs `rig bootstrap -box`; - 4. cast's legs on top of the converged result — two live Coolify - instances and the full A→B promotion. + What makes that safe is that every drill **pins the same fixed set of + candidate refs**, so each one exercises exactly the combination that will + ship rather than whatever `main` happens to be that afternoon. The run + drills **candidate refs, not released artifacts**: `RIG_REPO` and `RIG_REF` + are mint-time environment variables (default `heavy-duty/rig@main`), so a + run pins the exact commits under test. - Note that rig appears **twice**, below box and above it. box and rig are - mutually recursive, not linearly ordered: rig builds the host that runs - box, and box's seed calls rig back to converge the guest. cast sits on top - of whatever that produces. So there is no "previous repo" for a repo to - drill against, and **no fixed order in which the three releases must be - published.** + That pinning — **not sequencing** — is what dissolves the box↔rig + recursion. box and rig *are* mutually recursive: rig builds the host that + runs box, and box's seed calls rig back to converge the guest. But + candidate refs are static identifiers that exist as soon as the release + branches do, long before any drill runs, so a cycle at runtime becomes + independent tests against one fixed pair. No repo has to be released + before another can be drilled, and there is **no fixed order in which the + three releases must be published.** - The run drills **candidate refs, not released artifacts**. `RIG_REPO` and - `RIG_REF` are mint-time environment variables (default - `heavy-duty/rig@main`), so a run pins the exact commits under test. That - dissolves the chicken-and-egg the mutual recursion would otherwise create: - no repo has to be released before another can be drilled. + Each repo also drills a **different thing**: box asserts the isolation + contract (the VM trust boundary), rig asserts convergence (a machine + reaches its role, idempotently), cast asserts promotion (A→B reproduces, + and the diff is idempotent). Three different exercises sharing a + substrate — which is exactly why the records are per-repo. + + cast's legs are the **least coupled** of the three: two Coolify instances + can be stood up by hand, as the July drill did for instance B via a + parameterised compose file. Within a single drill you of course bring the + substrate up before probing it — a host before a guest before Coolify — + but that is how you run *a* drill, not an ordering rule *between repos*. Drilling the candidate **is** drilling the release. A release PR's diff is the version file and `CHANGELOG.md` — nothing executable differs between the tree that was drilled and the tree that ships, so the evidence carries across the ceremony commit. - One run emits one shared **run ID**. Each repo records ITS OWN legs under - its own `## Release drill — X.Y.Z — DATE`, citing that run ID and the - other two repos' commit SHAs — which is what lets three separate records - be reassembled into the single run they came from. The guard still reads - only this repo's file: cast never queries box's or rig's drill log to - decide whether cast may ship, because a cross-repo lookup degrades to - "pass" the moment it fails to resolve — the unreadable-rollup bug wearing - a different hat. + Each repo records ITS OWN legs in its own `drills/X.Y.Z.md`, citing the + shared **run ID** that names the pinned set and the other two repos' commit + SHAs — which is what lets separate records be reassembled into one picture. + The guard still reads only this repo's files: cast never queries box's or + rig's drill records to decide whether cast may ship, because a cross-repo + lookup degrades to "pass" the moment it fails to resolve — the + unreadable-rollup bug wearing a different hat. + + If a defect shows up only in the combination: patch, re-drill, re-record. + The three releases converge on a set that holds together; they are not + required to be right in one pass. A maintainer **waiver** is possible — but it must be RECORDED in - `drill/RUNS.md` for that version, saying who waived it and what is + `drills/X.Y.Z.md` for that version, saying who waived it and what is untested. The guard requires a *record*, not a passing result, so skipping the drill stays possible and stays visible and deliberate. 3. **Merge. That's the ship decision — nothing else to do.** diff --git a/LABELS.md b/LABELS.md index c264c19..b589208 100644 --- a/LABELS.md +++ b/LABELS.md @@ -36,7 +36,7 @@ PR carries as many as apply. | `blocker:conflict` | `#B60205` | GitHub says `CONFLICTING` — the agent owes a **rebase** | it merges cleanly | | `blocker:ci-red` | `#B60205` | a check failed — the agent owes a **fix**, which a rebase will not provide | checks are green | | `blocker:unrequested` | `#E99695` | this head has no verdict from somebody — never reviewed, or staled by a push — and **nobody was asked** for one | reviews are requested | -| `blocker:drill-pending` | `#E99695` | a `release` PR whose version has **no drill record** in [drill/RUNS.md](drill/RUNS.md) — the ceremony is correct but *unevidenced* | the record (or a recorded maintainer waiver) lands for that version | +| `blocker:drill-pending` | `#E99695` | a `release` PR whose version has **no drill record** at [`drills/.md`](drills/README.md) — the ceremony is correct but *unevidenced* | the record (or a recorded maintainer waiver) lands for that version | `blocker:drill-pending` is the only blocker that says nothing is *wrong* with the code. The version bump, the stamp, the re-arm can all be perfect; what is @@ -160,7 +160,7 @@ gh label create "blocker:ci-red" --color B60205 --description "A check is gh label create "blocker:unrequested" --color E99695 --description "Somebody still owes a verdict and nobody was asked for one" --force # needs a MAINTAINER account — the bot 403s on label creation, and until this # runs the reconciler cannot apply it and `blocked` stands in. -gh label create "blocker:drill-pending" --color E99695 --description "Release PR with no drill record in drill/RUNS.md — ceremony correct, unevidenced" --force +gh label create "blocker:drill-pending" --color E99695 --description "Release PR with no drill record at drills/.md — ceremony correct, unevidenced" --force # retired — the reconciler strips it; delete it once no PR carries it # gh label delete "state:needs-rebase" gh label create "state:needs-human" --color 8250DF --description "No blockers, all bots approve — waiting on the human reviewer" --force diff --git a/drill/RUNS.md b/drill/RUNS.md deleted file mode 100644 index 436bf6c..0000000 --- a/drill/RUNS.md +++ /dev/null @@ -1,94 +0,0 @@ -# Drill runs - -The log of cast's **real-hardware drill legs** — one section per run, appended. -A release PR's version must have a section here before CI will let it merge -(`.github/scripts/drill-recorded.sh`, wired into ci.yml). - -**This file is the record, not the instrument.** cast has **no drill harness -script of its own yet**. Its legs are run by hand, against the documented -procedure: two live Coolify instances and the full A→B promotion — - - team → apply → diff (idempotent) → smoke → inventory → emit-draft → - fleet → destroy → read-only guard - -A harness would make the run reproducible; it would not make it recorded. Those -are separate problems, and this file is the second one. When cast grows a -harness, its output gets pasted into a section here in the same format — the -gate does not change. - -## Per-repo, by construction - -cast records **cast's own** legs. It does not read box's or rig's drill log to -decide whether cast may ship: a cross-repo lookup silently degrades to "pass" -the moment it fails to resolve — the unreadable-rollup class of bug, where a -guard that cannot read its input reports the happy answer. - -The legs above are the **top of one orchestrated run over the whole stack** -(CONTRIBUTING.md, *Releasing*): `rig bootstrap --host yes` on bare Debian -installs box and runs `setup-host`; `box new` mints a seed; the seed converges -by calling `rig bootstrap -box`; cast's legs run on the result. rig -appears twice — below box and above it — so the three repos are mutually -recursive, not linearly ordered, and their releases are **not** published in a -fixed order. - -The run drills candidate refs rather than released artifacts (`RIG_REPO` / -`RIG_REF` are mint-time variables, default `heavy-duty/rig@main`), which is -what makes that possible: no repo has to ship before another can be drilled. -A record therefore cites the shared **run ID** and the other two repos' commit -SHAs — three records, one run, reassemblable — while each repo's evidence -still lives in its own tree. - -## Format - -The gate looks for a heading of exactly this shape, and requires the section -under it to be non-empty: - - ## Release drill — X.Y.Z — YYYY-MM-DD - -The version is matched **whole**: a `0.2.0` release is not satisfied by a -`0.2.0-rc1` section, or the other way round. - -A record says three things: **what ran**, **the numbers**, and **what failed**. -"Nothing failed" is a finding and is worth a line; an empty section is not a -record and the gate refuses it. - -### Example of the shape - -Illustrative only — no run has happened yet, and the placeholder version keeps -this block from ever satisfying the gate for a real release. - - ## Release drill — X.Y.Z — YYYY-MM-DD - - Run ID: `drill-YYYYMMDD-NN` (the stack run these legs are the top of). - Stack under test: box ``, rig ``, cast `` — candidate - refs, pinned at mint time via RIG_REPO/RIG_REF. - Instances: A `coolify-a.example` (v4.x), B `coolify-b.example` (v4.x). - Manifest: `examples/two-env.yaml`, 3 applications, 2 environments. - Operator: @maintainer. Elapsed: 41m. - - | Leg | Result | Notes | - |---|---|---| - | team | pass | 2 teams, 4 members reconciled | - | apply (A→B) | pass | 3 apps created, 11 env vars set | - | diff (idempotent) | pass | second apply: 0 changes | - | smoke | pass | 3/3 endpoints 200 | - | inventory | pass | 3 apps, 2 envs, matches manifest | - | emit-draft | pass | draft matches inventory round-trip | - | fleet | pass | both instances listed, versions read | - | destroy | pass | 3 apps removed, absence asserted | - | read-only guard | pass | write refused against B with the guard on | - - Failures: none. One rough edge: `smoke` needed a 20s retry window on B — - filed as #NNN, not a release blocker. - -## Waivers - -A maintainer may ship without a passing drill, but **not without a record**. -The waiver is a section under the same heading saying who waived it, why, and -what is untested — a deliberate, reviewable commit. The gate requires a record, -not a pass, precisely so that skipping is visible. - -## Records - -*None yet.* cast has recorded no drill runs. New runs are appended below, -newest first. diff --git a/drills/README.md b/drills/README.md new file mode 100644 index 0000000..7345e8e --- /dev/null +++ b/drills/README.md @@ -0,0 +1,145 @@ +# Drills + +Per-release evidence: **one file per version**, named `.md`, where +`` matches `package.json`'s `version` exactly — `0.2.0` is recorded in +`0.2.0.md`, `0.2.0-rc1` in `0.2.0-rc1.md`. + +A release PR's version must have its file here, holding at least one +non-whitespace character, before CI will let it merge +(`.github/scripts/drill-recorded.sh`, wired into ci.yml). + +## One file per version, and why the parser went away + +Records used to share a single `drill/RUNS.md`, which meant the gate had to +*parse* it: a heading grammar, an em-dash field match, an optional ` — DATE` +tail, a whole-version comparison, a non-blank-body rule. That machinery existed +only because records shared a file — and it shipped two defects in review: a +`sed '/./,$!d'` extraction where `.` matches a space (so a heading followed by +one tab satisfied the gate), and heading-grammar drift from box's stricter form. + +One file per version makes nearly all of that unrepresentable. `0.2.0.md` and +`0.2.0-rc1.md` are different files, so the whole-version rule is the +filesystem's rather than a comparison anyone can get wrong, and there is no +heading to drift. One rule survives: a file of only whitespace is not a record. + +The directory is plain `drills/`, **not** `.drills/`. Dot-prefixed directories +are invisible to globs without `dotglob` — the blind spot behind #118, #121 and +box#116. + +## What a record should contain + +- **What ran** — which legs, against which manifest. +- **On what host** — the instances, their versions, who operated it. +- **The pinned candidate refs** — box, rig and cast SHAs under test. +- **The numbers** — counts, elapsed time, whatever the legs emit. +- **What failed** — and "nothing failed" is itself a finding worth a line. + +**A failed drill is still a valid record.** The gate wants evidence, not +success. A maintainer may ship on a failed or partial drill — what they may not +do is ship on silence, so a **waiver** is also a legitimate record: say who +waived it, why, and what is untested. Requiring a record makes skipping a +deliberate, reviewable commit instead of the default outcome of forgetting. + +## This directory is the record, not the instrument + +cast has **no drill harness script of its own**. Its legs run by hand against +the documented procedure: two live Coolify instances and the full A→B +promotion — + + team → apply → diff (idempotent) → smoke → inventory → emit-draft → + fleet → destroy → read-only guard + +A harness would make the run reproducible; it would not make it recorded. Those +are separate problems, and this directory is the second one. + +## Per-repo, by construction + +cast records **cast's own** legs. It does not read box's or rig's drill records +to decide whether cast may ship: a cross-repo lookup silently degrades to +"pass" the moment it fails to resolve — the unreadable-rollup class of bug, +where a guard that cannot read its input reports the happy answer. + +**The three repos' drills are independent.** Run them in any order, on any +schedule, in separate sittings. They are not phases of one script. + +What makes that safe is that every drill **pins the same fixed set of candidate +refs** (`RIG_REPO` / `RIG_REF` are mint-time variables, default +`heavy-duty/rig@main`), so each drill exercises exactly the combination that +will ship rather than whatever `main` happens to be that afternoon. The run +drills **candidate refs, not released artifacts**. + +That pinning — **not sequencing** — is what dissolves the box↔rig recursion. +box and rig *are* mutually recursive: rig builds the host that runs box, and +box's seed calls rig back to converge the guest. But candidate refs are static +identifiers that exist as soon as the release branches do, long before any +drill runs, so a cycle at runtime becomes independent tests against one fixed +pair. No repo has to be released before another can be drilled, and there is +**no fixed order in which the three releases must be published**. + +Each repo also drills a **different thing**: box asserts the isolation contract +(the VM trust boundary), rig asserts convergence (a machine reaches its role, +idempotently), cast asserts promotion (A→B reproduces, and the diff is +idempotent). Three different exercises sharing a substrate — which is exactly +why the records are per-repo. + +cast's legs are the **least coupled** of the three. Two Coolify instances can +be stood up by hand; the July drill did exactly that for instance B, via a +parameterised compose file. Nothing about cast's drill requires box or rig to +have been drilled first, or at all, on that day. + +Within a single drill you obviously bring the substrate up before probing it — +a host before a guest before Coolify. That is how you run *a* drill, not an +ordering rule *between repos*. + +If a defect shows up only in the combination: patch, re-drill, re-record. The +three releases converge on a set that holds together; they are not required to +be right in one pass. + +**Drilling the candidate is drilling the release.** A release PR's diff is the +version file and `CHANGELOG.md` — nothing executable differs between the tree +that was drilled and the tree that ships, so the evidence carries across the +ceremony commit. + +Each record cites the shared **run ID** naming the pinned set, plus the other +two repos' SHAs — which is what lets separate records be reassembled into one +picture, while each repo's evidence still lives in its own tree. + +## Worked example + +Illustrative only. The version below is a **placeholder that can never collide +with a real release** — a realistic-looking version here would be a real record +for it, and the gate would wave that release through on documentation. The +mechanism changed with the move to one file per version; the caution did not. + +`drills/9.9.9.md`: + +```markdown +# Release drill — 9.9.9 — YYYY-MM-DD + +Run ID: `drill-YYYYMMDD-NN` (names the pinned candidate set). +Stack under test: box ``, rig ``, cast `` — candidate refs, +pinned at mint time via RIG_REPO/RIG_REF. +Instances: A `coolify-a.example` (v4.x), B `coolify-b.example` (v4.x), +B stood up by hand from the parameterised compose file. +Manifest: `examples/two-env.yaml`, 3 applications, 2 environments. +Operator: @maintainer. Elapsed: 41m. + +| Leg | Result | Notes | +|---|---|---| +| team | pass | 2 teams, 4 members reconciled | +| apply (A→B) | pass | 3 apps created, 11 env vars set | +| diff (idempotent) | pass | second apply: 0 changes | +| smoke | pass | 3/3 endpoints 200 | +| inventory | pass | 3 apps, 2 envs, matches manifest | +| emit-draft | pass | draft matches inventory round-trip | +| fleet | pass | both instances listed, versions read | +| destroy | pass | 3 apps removed, absence asserted | +| read-only guard | pass | write refused against B with the guard on | + +Failures: none. One rough edge: `smoke` needed a 20s retry window on B — +filed as #NNN, not a release blocker. +``` + +## Records + +*None yet.* cast has recorded no drill runs. diff --git a/test/release.test.ts b/test/release.test.ts index 3ceb3d3..c0b9989 100644 --- a/test/release.test.ts +++ b/test/release.test.ts @@ -735,115 +735,134 @@ describe("changelog-monotonic.sh — release headings are append-only (#133)", ( // gate moves the requirement out of a reviewer's memory and into the tree. // // What it asserts is that a RECORD EXISTS, never that the drill passed — a -// maintainer waiver is legal and is itself a section in drill/RUNS.md. That is -// the point: skipping stays possible and stays a deliberate, reviewable -// commit. So the cases below are all about presence, emptiness, and whether -// the version was matched whole; none of them inspect a result. +// maintainer waiver is legal and is itself the content of drills/.md. +// That is the point: skipping stays possible and stays a deliberate, +// reviewable commit. So the cases below are all about presence and emptiness; +// none of them inspect a result. // -// Every fixture carries its OWN package.json and RUNS.md inside the temp dir. -// Pointing the guard at the repo's real package.json would make these cases -// change meaning on every version bump — green or red depending on where the -// ceremony happens to be standing, which is the opposite of a fixture. +// ONE FILE PER VERSION. The earlier design kept every record in one +// drill/RUNS.md, so the guard parsed headings — and the heading grammar was +// where both of this feature's review defects lived (a whitespace bypass, and +// drift from box's stricter form). `0.2.0.md` and `0.2.0-rc1.md` are simply +// different files, so the whole-version rule is now the filesystem's rather +// than a comparison that can be got wrong, and the tests that existed only to +// pin heading syntax (including the stray-tail case) are gone with it. +// +// Every fixture carries its OWN package.json and its OWN drills dir inside a +// temp tree. Pointing the guard at the repo's real package.json would make +// these cases change meaning on every version bump — green or red depending +// on where the ceremony happens to be standing, the opposite of a fixture. describe("drill-recorded.sh — a release version has a drill record", () => { const work = tmp("cast-drill-"); - /** A fixture tree: its own package.json + drill runs file. */ - function treeWith(name: string, version: string, runs?: string): string { + /** + * A fixture tree: its own package.json, plus an optional `drills/` dir. + * + * @param records omit for NO drills dir at all; pass a map of + * `.md` -> contents (possibly empty) to create the dir. + */ + function treeWith( + name: string, + version: string, + records?: Record, + ): string { const dir = join(work, name); mkdirSync(dir, { recursive: true }); writeFileSync( join(dir, "package.json"), `${JSON.stringify({ name: "cast", version }, null, 2)}\n`, ); - if (runs !== undefined) writeFileSync(join(dir, "RUNS.md"), runs); + if (records !== undefined) { + mkdirSync(join(dir, "drills"), { recursive: true }); + for (const [file, body] of Object.entries(records)) { + writeFileSync(join(dir, "drills", file), body); + } + } return dir; } const check = (dir: string) => - run("bash", [DRILL, "RUNS.md", "package.json"], {}, dir); + run("bash", [DRILL, "drills", "package.json"], {}, dir); - const heading = (v: string) => `## Release drill — ${v} — 2026-07-21`; const legs = - "\nInstances: A and B, live. All legs pass: team, apply, diff, smoke,\ninventory, emit-draft, fleet, destroy, read-only guard.\n"; + "# Release drill\n\nInstances: A and B, live. All legs pass: team, apply,\ndiff, smoke, inventory, emit-draft, fleet, destroy, read-only guard.\n"; - it("a -dev tree passes with no drill record at all — nothing ships from it", async () => { + it("a -dev tree passes with no drills dir at all — nothing ships from it", async () => { const r = await check(treeWith("dev", "0.1.2-dev")); expect(r.code).toBe(0); expect(r.output).toContain("development tree"); }); it("a bare version with a matching, non-empty record passes", async () => { - const runs = `# Drill runs\n\n${heading("0.2.0")}\n${legs}`; - const r = await check(treeWith("ok", "0.2.0", runs)); + const r = await check(treeWith("ok", "0.2.0", { "0.2.0.md": legs })); expect(r.code).toBe(0); expect(r.output).toContain("0.2.0"); }); - it("a bare version with NO record fails, naming the version", async () => { - const runs = "# Drill runs\n\n*None yet.*\n"; - const r = await check(treeWith("none", "0.2.0", runs)); + it("a bare version with NO drills dir fails, naming the version", async () => { + const r = await check(treeWith("nodir", "0.2.0")); expect(r.code).toBe(1); - expect(r.output).toContain("no drill record for version '0.2.0'"); + expect(r.output).toContain("version 0.2.0 is a release"); + expect(r.output).toContain("drills/0.2.0.md"); }); - // A heading with nothing under it is ceremony without evidence — the exact - // shape a hurried release produces, and the one a heading-only check would - // wave through. release-notes.sh refuses an empty section for the same - // reason. - it("a present-but-EMPTY record fails — a heading is not a record", async () => { - const runs = `# Drill runs\n\n${heading("0.2.0")}\n\n\n## Notes\n\nUnrelated.\n`; - const r = await check(treeWith("empty", "0.2.0", runs)); + it("a drills dir with no file for THIS version fails", async () => { + const r = await check( + treeWith("othersonly", "0.2.0", { "0.1.0.md": legs, "README.md": legs }), + ); expect(r.code).toBe(1); - expect(r.output).toContain("no drill record for version '0.2.0'"); + expect(r.output).toContain("drills/0.2.0.md"); }); - // ...and whitespace is not evidence either. The first cut extracted with - // `sed '/./,$!d'`, where `.` matches a space, so a heading followed by one - // tab passed the gate — while the comment above the extractor claimed the - // opposite. An evidence-free release for the price of an invisible - // character, on the one check whose whole job is to demand evidence. - // Found independently by all three reviewers on #138. - it("a record body of only spaces and tabs fails (#138)", async () => { - const runs = `# Drill runs\n\n${heading("0.2.0")}\n \n\t\n\n## Notes\n\nUnrelated.\n`; - const r = await check(treeWith("blank", "0.2.0", runs)); + // A file that exists but says nothing is ceremony without evidence — the + // exact shape a hurried release produces, and the one an `[ -f ]` check + // would wave through. release-notes.sh refuses an empty section likewise. + it("a present-but-EMPTY record fails — a filename is not a record", async () => { + const r = await check(treeWith("empty", "0.2.0", { "0.2.0.md": "" })); expect(r.code).toBe(1); - expect(r.output).toContain("no drill record for version '0.2.0'"); + expect(r.output).toContain("drills/0.2.0.md"); }); - // The sibling guards must agree about what a heading IS, not just about the - // version in it. box#149 requires the version to be the last field or be - // followed by the em dash; without the same constraint here, this heading is - // a record in cast and not in box. - it("a stray tail after the version is not a heading — in step with box#149", async () => { - const runs = `# Drill runs\n\n## Release drill — 0.2.0 stray words\n${legs}`; - const r = await check(treeWith("tail", "0.2.0", runs)); + // ...and whitespace is not evidence either. This is the one surviving piece + // of the heading-parser era: the first cut extracted with `sed '/./,$!d'`, + // where `.` matches a space, so a heading followed by one tab passed the + // gate — while the comment above the extractor claimed the opposite. An + // evidence-free release for the price of an invisible character, on the one + // check whose whole job is to demand evidence. Found independently by all + // three reviewers on #138. The file layout changed; this rule did not. + it("a record of only spaces, tabs and newlines fails (#138)", async () => { + const r = await check( + treeWith("blank", "0.2.0", { "0.2.0.md": " \n\t\n \n" }), + ); expect(r.code).toBe(1); - expect(r.output).toContain("no drill record for version '0.2.0'"); + expect(r.output).toContain("drills/0.2.0.md"); }); - // The version is matched WHOLE, both directions — release-notes.sh's trap, - // solved the same way (awk field equality, no regex, no dot-escaping). A - // release candidate's drill is not the release's drill: different tree, - // different build, and a prefix match would silently accept it. + // The version is matched WHOLE, both directions — but now by the FILESYSTEM + // rather than by a comparison. A release candidate's drill is not the + // release's drill: different tree, different build, and under the old + // heading parser a prefix match would have silently accepted it. it("0.2.0-rc1's record does NOT satisfy 0.2.0", async () => { - const runs = `# Drill runs\n\n${heading("0.2.0-rc1")}\n${legs}`; - const r = await check(treeWith("rc-for-bare", "0.2.0", runs)); + const r = await check( + treeWith("rc-for-bare", "0.2.0", { "0.2.0-rc1.md": legs }), + ); expect(r.code).toBe(1); - expect(r.output).toContain("no drill record for version '0.2.0'"); + expect(r.output).toContain("drills/0.2.0.md"); }); it("...and 0.2.0's record does NOT satisfy 0.2.0-rc1", async () => { - const runs = `# Drill runs\n\n${heading("0.2.0")}\n${legs}`; - const r = await check(treeWith("bare-for-rc", "0.2.0-rc1", runs)); + const r = await check( + treeWith("bare-for-rc", "0.2.0-rc1", { "0.2.0.md": legs }), + ); expect(r.code).toBe(1); - expect(r.output).toContain("no drill record for version '0.2.0-rc1'"); + expect(r.output).toContain("drills/0.2.0-rc1.md"); }); it("...while each still matches its own record", async () => { - const runs = `# Drill runs\n\n${heading("0.2.0")}\n${legs}\n${heading("0.2.0-rc1")}\n${legs}`; - expect((await check(treeWith("both-a", "0.2.0", runs))).code).toBe(0); - expect((await check(treeWith("both-b", "0.2.0-rc1", runs))).code).toBe(0); + const both = { "0.2.0.md": legs, "0.2.0-rc1.md": legs }; + expect((await check(treeWith("both-a", "0.2.0", both))).code).toBe(0); + expect((await check(treeWith("both-b", "0.2.0-rc1", both))).code).toBe(0); }); // The message is the whole user interface of a blocking guard. A failure @@ -851,22 +870,16 @@ describe("drill-recorded.sh — a release version has a drill record", () => { // than satisfied — including the waiver, which must be visibly ALLOWED or // somebody will route around the gate instead of recording one. it("the failure names the unblock: run the drill, or record a waiver", async () => { - const r = await check(treeWith("unblock", "0.2.0", "# Drill runs\n")); + const r = await check(treeWith("unblock", "0.2.0", {})); expect(r.code).toBe(1); - expect(r.output).toContain("## Release drill — 0.2.0"); + expect(r.output).toContain("drills/0.2.0.md"); expect(r.output).toContain("run the drill and record it"); expect(r.output).toContain("WAIVER"); expect(r.output).toContain("RECORD, not a"); }); - it("a missing runs file on a release tree refuses — never a silent pass", async () => { - const r = await check(treeWith("norunsfile", "0.2.0")); - expect(r.code).toBe(1); - expect(r.output).toContain("no RUNS.md at all"); - }); - it("a missing version file refuses by path", async () => { - const r = await run("bash", [DRILL, "RUNS.md", "nope.json"], {}, work); + const r = await run("bash", [DRILL, "drills", "nope.json"], {}, work); expect(r.code).toBe(1); expect(r.output).toContain("no such file"); }); @@ -885,12 +898,20 @@ describe("drill-recorded.sh — a release version has a drill record", () => { expect(r.code).toBe(0); }); - it("drill/RUNS.md documents the heading format the gate parses", () => { - const runs = readFileSync(join(ROOT, "drill/RUNS.md"), "utf8"); - expect(runs).toContain("## Release drill — X.Y.Z — YYYY-MM-DD"); + it("drills/README.md documents the naming rule and the waiver", () => { + const doc = readFileSync(join(ROOT, "drills/README.md"), "utf8"); + expect(doc).toContain(".md"); + // The placeholder example version can never collide with a real release: + // under any scheme, a realistic version in the docs is a real record. + expect(doc).toContain("drills/9.9.9.md"); // Per-repo by construction: cast records cast's legs and never reads - // another repo's log to decide whether cast may ship. - expect(runs).toMatch(/waiver/i); + // another repo's record to decide whether cast may ship. + expect(doc).toMatch(/waiver/i); + expect(doc).toMatch(/failed drill is still a valid record/i); + }); + + it("the old single-file log is gone — records are per version", () => { + expect(existsSync(join(ROOT, "drill/RUNS.md"))).toBe(false); }); it("ci.yml runs it, ungated by event or label", () => { @@ -914,24 +935,31 @@ describe("drill-recorded.sh — a release version has a drill record", () => { expect(block).not.toMatch(/^ {8}if:/m); }); - it("CONTRIBUTING documents the gate and the ONE-RUN stack drill", () => { + // The three drills are INDEPENDENT — any order, any schedule, separate + // sittings. What makes that safe is that each pins the same fixed candidate + // refs, so every drill exercises the combination that will ship. That + // pinning, not sequencing, is what dissolves the box<->rig recursion: the + // refs are static identifiers that exist as soon as the release branches + // do. The docs must not re-acquire an ordering rule between repos. + it("CONTRIBUTING documents the gate and the INDEPENDENT, ref-pinned drills", () => { const doc = readFileSync(join(ROOT, "CONTRIBUTING.md"), "utf8"); - expect(doc).toContain("drill/RUNS.md"); + expect(doc).toContain("drills/X.Y.Z.md"); expect(doc).toContain("drill-recorded.sh"); - // One orchestrated run over the whole stack, in order: rig builds the - // host (installing box), box mints a seed, the seed calls rig back to - // converge, cast's legs run on the result. - expect(doc).toContain("--host yes"); - expect(doc).toContain("box new"); - expect(doc).toContain("rig bootstrap -box"); - // rig sits BELOW box and ABOVE it — mutually recursive, not linear. So - // the docs must not promise a fixed release order, and must say the run - // pins CANDIDATE refs, which is what dissolves the chicken-and-egg. + expect(doc).toMatch(/drills are independent/i); + expect(doc).toMatch(/any order/i); + expect(doc).toMatch(/pins the same fixed set of\s+candidate\s+refs/i); + expect(doc).toMatch(/not sequencing/i); + // box and rig stay mutually recursive; the pinning is what makes that a + // non-problem, so both halves have to survive together. expect(doc).toMatch(/mutually recursive/); expect(doc).toContain("RIG_REF"); expect(doc).toMatch(/candidate refs, not released artifacts/); expect(doc).toMatch(/no fixed order|not.*published in a fixed order/i); - // Three records, one run: each repo cites the shared run ID. + // Each repo drills a different thing — which is WHY records are per-repo. + expect(doc).toMatch(/isolation\s+contract/i); + expect(doc).toMatch(/convergence/i); + expect(doc).toMatch(/promotion/i); + // Separate records, one pinned set: each cites the shared run ID. expect(doc).toMatch(/run ID/i); }); });