heavy-duty/cast
7
0
Fork 0
Code Issues 4 Pull requests Projects Releases 3 Packages Wiki Activity Actions

Merge pull request #139 from dan-claude-bot/refactor/drills-per-version

Browse source 
refactor: one drill record per version, in drills/
This commit is contained in:
Daniel Marin 2026-07-21 18:41:48 +01:00 committed by GitHub
commit 2e6fe1b1b2
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
8 changed files with 412 additions and 295 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

148
.github/scripts/drill-recorded.sh vendored
View file

@ -1,8 +1,8 @@
#!/usr/bin/env bash
set -euo pipefail
# drill-recorded.sh [<runs-file>] [<version-file>] — assert that the version
# this tree claims to ship has a DRILL RECORD in drill/RUNS.md.
# drill-recorded.sh [<drills-dir>] [<version-file>] — assert that the version
# this tree claims to ship has a DRILL RECORD at <drills-dir>/<version>.md.
#
# CONTRIBUTING says a release carries the full real-hardware drill. Nothing
# checked that, so no release in this family ever carried one: the step lived
@ -12,44 +12,81 @@ set -euo pipefail
# the gate moves into CI, where it is asserted on every release PR rather than
# recalled.
#
# ONE FILE PER VERSION — WHY THE PARSER IS GONE
#
# The first cut kept every record in one drill/RUNS.md and asked awk which
# section belonged to this version. That bought a heading grammar: em-dash
# field matching, an optional ' — DATE' tail, a whole-version comparison so
# 0.2.0-rc1 could not satisfy 0.2.0, a '(NF == 5 || $6 == dash)' tail
# constraint to stay in step with box's twin, and a non-blank body rule.
#
# All of it existed ONLY because records shared a file — and in review this
# repo shipped two defects out of that complexity: a `sed '/./,$!d'`
# extraction where `.` matches a space, so a heading followed by one tab
# satisfied the gate; and heading-grammar drift from box's stricter form.
# Two defects, on the one check whose entire job is to demand evidence.
#
# One file per version makes nearly all of it UNREPRESENTABLE. `0.2.0.md` and
# `0.2.0-rc1.md` are simply different files — the whole-version rule is the
# filesystem's, not a comparison anyone can get wrong. There is no heading to
# parse, so there is no grammar to drift from box's. What is left is a
# question a shell can ask directly: does the file exist, and does it say
# anything.
#
# The directory is plain `drills/`, NOT `.drills/`. Dot-prefixed directories
# are invisible to globs without `dotglob`, which is the exact blind spot that
# produced #118, #121 here and box#116 — a sweep that looks green because it
# never descended into the directory holding the thing it was meant to check.
#
# WHAT IT ASSERTS, AND WHAT IT DELIBERATELY DOES NOT
#
# It asserts a RECORD EXISTS — not that the drill passed. That is the whole
# design. A maintainer may ship on a failed or partial drill; what they may not
# do is ship on silence. Requiring a record makes a waiver a deliberate,
# reviewable commit (a section saying who waived it and what is untested)
# instead of the default outcome of forgetting. A guard that demanded a PASS
# would be argued with and eventually bypassed; one that demands EVIDENCE has
# nothing to argue about.
# reviewable commit (a file saying who waived it and what is untested) instead
# of the default outcome of forgetting. A guard that demanded a PASS would be
# argued with and eventually bypassed; one that demands EVIDENCE has nothing to
# argue about.
#
# PER-REPO, ON PURPOSE
#
# This reads cast's OWN drill/RUNS.md. It does not reach into box or rig to ask
# This reads cast's OWN drills/. It does not reach into box or rig to ask
# whether the family drilled. A cross-repo lookup has a failure mode this repo
# keeps refusing: when the fetch fails — no network, moved file, renamed repo,
# a token without read on the other repo — the honest answers are "unknown" and
# "blocked", but the shape such code actually takes degrades to "pass". Same
# class as the unreadable check rollup that read as "nothing is failing".
#
# There is also nothing to look up. The drill is ONE orchestrated run over the
# whole stack (CONTRIBUTING.md, "Releasing"): rig bootstraps the host and
# installs box, box mints a seed, the seed calls rig back to converge, and
# cast's legs run on the result. rig therefore sits BELOW box and ABOVE it —
# the three repos are mutually recursive, not linearly ordered, and their
# releases are NOT published in a fixed sequence. The run pins CANDIDATE refs
# (RIG_REPO/RIG_REF at mint time), so no repo must ship before another can be
# drilled. Each repo records its own legs from that run, citing the shared run
# ID and the other repos' SHAs — which is how three records reassemble into one
# run without any repo reading another's file.
# There is also nothing to look up. The three repos' drills are INDEPENDENT
# (CONTRIBUTING.md, "Releasing") — run in any order, on any schedule, in
# separate sittings. What makes that safe is that every drill pins the SAME
# FIXED SET OF CANDIDATE REFS (RIG_REPO/RIG_REF at mint time), so each one
# exercises the combination that will ship rather than whatever main happens
# to be that afternoon.
#
# That pinning, not sequencing, is what dissolves the box<->rig recursion. box
# and rig ARE mutually recursive — rig builds the host that runs box, box's
# seed calls rig back to converge the guest — but candidate refs are static
# identifiers that exist as soon as the release branches do, long before any
# drill runs. A cycle at runtime becomes independent tests against one fixed
# pair, and no repo must ship before another can be drilled. The three
# releases are NOT published in a fixed sequence.
#
# Each repo also drills a DIFFERENT thing: box asserts the isolation contract,
# rig asserts convergence, cast asserts promotion. Three different exercises
# over a shared substrate — which is exactly why the records are per-repo.
# Each cites the shared run ID naming the pinned set, plus the other repos'
# SHAs, so three records still reassemble into one picture without any repo
# reading another's file.
#
# A file of its own, not a clause inlined in ci.yml, for the same reason as
# release-notes.sh and changelog-monotonic.sh: test/release.test.ts drives the
# REAL script against fixtures, so what the tests prove is what CI runs.
runs="${1:-drill/RUNS.md}"
drills="${1:-drills}"
version_file="${2:-package.json}"
[ "$#" -le 2 ] || { echo "usage: drill-recorded.sh [<runs-file>] [<version-file>]" >&2; exit 2; }
[ "$#" -le 2 ] || { echo "usage: drill-recorded.sh [<drills-dir>] [<version-file>]" >&2; exit 2; }
[ -f "$version_file" ] || { echo "drill-recorded: no such file: $version_file" >&2; exit 1; }
# cast's version lives in package.json (there is no VERSION file), so this
@ -76,69 +113,34 @@ esac
# A bare version is a release ceremony tree: this is the tree whose merge IS
# the release, so this is where the evidence has to exist.
[ -f "$runs" ] || {
{
echo "drill-recorded: version $ver is a release, but there is no $runs at all."
echo
echo " The release drill has to be RECORDED, not just performed. Create the file"
echo " with the run's section (see the format below) and commit it."
} >&2
exit 1
}
#
# The record is <drills-dir>/<version>.md, and it must contain at least one
# NON-WHITESPACE character. That second clause is the one surviving piece of
# the whitespace defect found in review (#138): a file of only spaces, tabs
# and newlines is a file, and `[ -f ]` is happy with it, but it is not a
# record — an evidence-free release for the price of an invisible character.
# `grep -q '[^[:space:]]'` asks the question the old `sed '/./,$!d'` only
# claimed to: `.` matches a space, a POSIX class does not.
record="$drills/$ver.md"
# $5 of a drill heading ('## Release drill — 0.2.0 — 2026-07-21') is the bare
# version, compared WHOLE — so 0.2.0 can never be satisfied by a 0.2.0-rc1
# section, or the reverse, and no regex-escaping of dots. Exactly the trap
# release-notes.sh solves the same way, with the same awk field split, so the
# two cannot drift apart about what a heading is. The trailing ' — DATE' is
# optional and unread: the gate is about the version, and the date is for the
# humans reading the log.
#
# grab is re-armed by every '## ' line, so the section ends at the next one —
# a record cannot borrow the body of the record below it.
#
# `grab && NF` is the non-blank rule, and it is load-bearing rather than
# tidiness. NF is 0 on a line that is empty OR contains only whitespace, so
# `record` is non-empty exactly when a line with real content exists. The
# first cut of this piped through `sed '/./,$!d'` and the comment here claimed
# "a heading with nothing but whitespace under it extracts to the empty
# string" — which is precisely what that pipeline did NOT guarantee, because
# `.` matches a space. A heading followed by one tab satisfied the gate. The
# comment documented the intended contract and the code did not meet it, which
# on a gate is the whole ballgame: an evidence-free release for the price of an
# invisible character. Found by all three reviewers on #138, independently.
#
# The '(NF == 5 || $6 == dash)' tail constraint keeps this in step with box's
# twin: without it '## Release drill — 0.2.0 stray words' counts as a record
# here and does not there. Two sibling guards disagreeing about what the same
# heading means is the same trap as disagreeing with release-notes.sh.
record="$(awk -v ver="$ver" -v dash="—" '
/^## / {
grab = ($2 == "Release" && $3 == "drill" && $4 == dash && $5 == ver \
&& (NF == 5 || $6 == dash))
next
}
grab && NF { print }
' "$runs")"
if [ -z "$record" ]; then
if [ ! -f "$record" ] || ! grep -q '[^[:space:]]' "$record"; then
{
echo "drill-recorded: $runs has no drill record for version '$ver'."
echo "drill-recorded: version $ver is a release, but there is no drill record at $record."
echo
echo " A release PR's version must have a NON-EMPTY section headed:"
echo " A release PR's version must have a NON-EMPTY file named for it:"
echo
echo " ## Release drill — $ver — YYYY-MM-DD"
echo " $drills/$ver.md"
echo
echo " (An empty section under a correct heading counts as no record. The"
echo " heading matches the version WHOLE — a '$ver-rc1' section does not"
echo " satisfy '$ver', and vice versa.)"
echo " (A file that exists but holds only whitespace counts as no record."
echo " One file per version, so '$ver-rc1.md' is a different record and"
echo " does not satisfy '$ver', or the other way round.)"
echo
echo " To unblock, either:"
echo " * run the drill and record it — the legs (team, apply, idempotent"
echo " diff, smoke, inventory, emit-draft, fleet, destroy, read-only"
echo " guard), the numbers, and what failed; or"
echo " * record an explicit maintainer WAIVER for this version in $runs,"
echo " saying who waived it and what is untested."
echo " * record an explicit maintainer WAIVER for this version in that"
echo " file, saying who waived it and what is untested."
echo
echo " The waiver is allowed on purpose: this gate requires a RECORD, not a"
echo " passing result, so shipping without a drill stays possible — and"
@ -147,4 +149,4 @@ if [ -z "$record" ]; then
exit 1
fi
echo "drill-recorded: $runs carries a drill record for $ver ($(printf '%s\n' "$record" | grep -c .) line(s))"
echo "drill-recorded: $record carries a drill record for $ver ($(grep -c '' "$record") line(s))"

4
.github/workflows/ci.yml vendored
View file

@ -85,8 +85,8 @@ jobs:
# drill went missing in the first place.
#
# It requires a RECORD, not a PASS: a maintainer waiver is legal, and is
# itself a section in drill/RUNS.md. Skipping stays possible and stays
# visible.
# itself the content of drills/<version>.md. Skipping stays possible and
# stays visible.
- name: a release version has a drill record
run: bash .github/scripts/drill-recorded.sh

2
CHANGELOG.md
View file

@ -9,7 +9,7 @@ actually cutting it, and this file starts there.
### Added
- CI refuses a release PR with no drill record in `drill/RUNS.md`
- CI refuses a release PR with no drill record at `drills/<version>.md`
- An application can declare HTTP basic auth, and `apply` sets it (#76)
- `cast github-app create` / `cast github-app register` run the App Manifest
flow (#7)

78
CONTRIBUTING.md
View file

@ -117,10 +117,14 @@ on box#83's shape):
2. **Drill, and record it.** Before the PR can be handed over, run the full
real-hardware drill — two live Coolify instances, the whole A→B promotion:
team, apply, an idempotent diff, smoke, inventory, emit-draft, fleet,
destroy, and the read-only guard — and record it in
[drill/RUNS.md](drill/RUNS.md) under a heading naming the version:
destroy, and the read-only guard — and record it in a file named for the
version, one record per version:
## Release drill — X.Y.Z — YYYY-MM-DD
drills/X.Y.Z.md
The name matches `package.json`'s `version` exactly, and the file must hold
at least one non-whitespace character. See
[drills/README.md](drills/README.md) for what a record contains.
[.github/scripts/drill-recorded.sh](.github/scripts/drill-recorded.sh)
enforces this on every release PR (a `-dev` tree has no ship claim and
@ -131,46 +135,56 @@ on box#83's shape):
So the release flow is: **draft → ready → bot round → drill →
`state:needs-human` → maintainer merge (which IS the release).**
**The drill is ONE orchestrated run over the whole stack**, not three
independent exercises. In order:
**The three repos' drills are independent.** Run them in any order, on any
schedule, in separate sittings. They are not three phases of one script.
1. `rig bootstrap … --host yes` on a bare Debian host — this installs box
and runs box's `setup-host`;
2. `box new` mints a creds-free seed;
3. the seed converges on first boot: its cloud-init curls rig's installer
and runs `rig bootstrap <tenant>-box`;
4. cast's legs on top of the converged result — two live Coolify
instances and the full A→B promotion.
What makes that safe is that every drill **pins the same fixed set of
candidate refs**, so each one exercises exactly the combination that will
ship rather than whatever `main` happens to be that afternoon. The run
drills **candidate refs, not released artifacts**: `RIG_REPO` and `RIG_REF`
are mint-time environment variables (default `heavy-duty/rig@main`), so a
run pins the exact commits under test.
Note that rig appears **twice**, below box and above it. box and rig are
mutually recursive, not linearly ordered: rig builds the host that runs
box, and box's seed calls rig back to converge the guest. cast sits on top
of whatever that produces. So there is no "previous repo" for a repo to
drill against, and **no fixed order in which the three releases must be
published.**
That pinning — **not sequencing** — is what dissolves the box↔rig
recursion. box and rig *are* mutually recursive: rig builds the host that
runs box, and box's seed calls rig back to converge the guest. But
candidate refs are static identifiers that exist as soon as the release
branches do, long before any drill runs, so a cycle at runtime becomes
independent tests against one fixed pair. No repo has to be released
before another can be drilled, and there is **no fixed order in which the
three releases must be published.**
The run drills **candidate refs, not released artifacts**. `RIG_REPO` and
`RIG_REF` are mint-time environment variables (default
`heavy-duty/rig@main`), so a run pins the exact commits under test. That
dissolves the chicken-and-egg the mutual recursion would otherwise create:
no repo has to be released before another can be drilled.
Each repo also drills a **different thing**: box asserts the isolation
contract (the VM trust boundary), rig asserts convergence (a machine
reaches its role, idempotently), cast asserts promotion (A→B reproduces,
and the diff is idempotent). Three different exercises sharing a
substrate — which is exactly why the records are per-repo.
cast's legs are the **least coupled** of the three: two Coolify instances
can be stood up by hand, as the July drill did for instance B via a
parameterised compose file. Within a single drill you of course bring the
substrate up before probing it — a host before a guest before Coolify —
but that is how you run *a* drill, not an ordering rule *between repos*.
Drilling the candidate **is** drilling the release. A release PR's diff is
the version file and `CHANGELOG.md` — nothing executable differs between
the tree that was drilled and the tree that ships, so the evidence carries
across the ceremony commit.
One run emits one shared **run ID**. Each repo records ITS OWN legs under
its own `## Release drill — X.Y.Z — DATE`, citing that run ID and the
other two repos' commit SHAs — which is what lets three separate records
be reassembled into the single run they came from. The guard still reads
only this repo's file: cast never queries box's or rig's drill log to
decide whether cast may ship, because a cross-repo lookup degrades to
"pass" the moment it fails to resolve — the unreadable-rollup bug wearing
a different hat.
Each repo records ITS OWN legs in its own `drills/X.Y.Z.md`, citing the
shared **run ID** that names the pinned set and the other two repos' commit
SHAs — which is what lets separate records be reassembled into one picture.
The guard still reads only this repo's files: cast never queries box's or
rig's drill records to decide whether cast may ship, because a cross-repo
lookup degrades to "pass" the moment it fails to resolve — the
unreadable-rollup bug wearing a different hat.
If a defect shows up only in the combination: patch, re-drill, re-record.
The three releases converge on a set that holds together; they are not
required to be right in one pass.
A maintainer **waiver** is possible — but it must be RECORDED in
`drill/RUNS.md` for that version, saying who waived it and what is
`drills/X.Y.Z.md` for that version, saying who waived it and what is
untested. The guard requires a *record*, not a passing result, so skipping
the drill stays possible and stays visible and deliberate.
3. **Merge. That's the ship decision — nothing else to do.**

4
LABELS.md
View file

@ -36,7 +36,7 @@ PR carries as many as apply.
| `blocker:conflict` | `#B60205` | GitHub says `CONFLICTING` — the agent owes a **rebase** | it merges cleanly |
| `blocker:ci-red` | `#B60205` | a check failed — the agent owes a **fix**, which a rebase will not provide | checks are green |
| `blocker:unrequested` | `#E99695` | this head has no verdict from somebody — never reviewed, or staled by a push — and **nobody was asked** for one | reviews are requested |
| `blocker:drill-pending` | `#E99695` | a `release` PR whose version has **no drill record** in [drill/RUNS.md](drill/RUNS.md) — the ceremony is correct but *unevidenced* | the record (or a recorded maintainer waiver) lands for that version |
| `blocker:drill-pending` | `#E99695` | a `release` PR whose version has **no drill record** at [`drills/<version>.md`](drills/README.md) — the ceremony is correct but *unevidenced* | the record (or a recorded maintainer waiver) lands for that version |
`blocker:drill-pending` is the only blocker that says nothing is *wrong* with
the code. The version bump, the stamp, the re-arm can all be perfect; what is
@ -160,7 +160,7 @@ gh label create "blocker:ci-red" --color B60205 --description "A check is
gh label create "blocker:unrequested" --color E99695 --description "Somebody still owes a verdict and nobody was asked for one" --force
# needs a MAINTAINER account — the bot 403s on label creation, and until this
# runs the reconciler cannot apply it and `blocked` stands in.
gh label create "blocker:drill-pending" --color E99695 --description "Release PR with no drill record in drill/RUNS.md — ceremony correct, unevidenced" --force
gh label create "blocker:drill-pending" --color E99695 --description "Release PR with no drill record at drills/<version>.md — ceremony correct, unevidenced" --force
# retired — the reconciler strips it; delete it once no PR carries it
# gh label delete "state:needs-rebase"
gh label create "state:needs-human" --color 8250DF --description "No blockers, all bots approve — waiting on the human reviewer" --force

94
drill/RUNS.md
View file

@ -1,94 +0,0 @@
# Drill runs
The log of cast's **real-hardware drill legs** — one section per run, appended.
A release PR's version must have a section here before CI will let it merge
(`.github/scripts/drill-recorded.sh`, wired into ci.yml).
**This file is the record, not the instrument.** cast has **no drill harness
script of its own yet**. Its legs are run by hand, against the documented
procedure: two live Coolify instances and the full A→B promotion —
team → apply → diff (idempotent) → smoke → inventory → emit-draft →
fleet → destroy → read-only guard
A harness would make the run reproducible; it would not make it recorded. Those
are separate problems, and this file is the second one. When cast grows a
harness, its output gets pasted into a section here in the same format — the
gate does not change.
## Per-repo, by construction
cast records **cast's own** legs. It does not read box's or rig's drill log to
decide whether cast may ship: a cross-repo lookup silently degrades to "pass"
the moment it fails to resolve — the unreadable-rollup class of bug, where a
guard that cannot read its input reports the happy answer.
The legs above are the **top of one orchestrated run over the whole stack**
(CONTRIBUTING.md, *Releasing*): `rig bootstrap --host yes` on bare Debian
installs box and runs `setup-host`; `box new` mints a seed; the seed converges
by calling `rig bootstrap <tenant>-box`; cast's legs run on the result. rig
appears twice — below box and above it — so the three repos are mutually
recursive, not linearly ordered, and their releases are **not** published in a
fixed order.
The run drills candidate refs rather than released artifacts (`RIG_REPO` /
`RIG_REF` are mint-time variables, default `heavy-duty/rig@main`), which is
what makes that possible: no repo has to ship before another can be drilled.
A record therefore cites the shared **run ID** and the other two repos' commit
SHAs — three records, one run, reassemblable — while each repo's evidence
still lives in its own tree.
## Format
The gate looks for a heading of exactly this shape, and requires the section
under it to be non-empty:
## Release drill — X.Y.Z — YYYY-MM-DD
The version is matched **whole**: a `0.2.0` release is not satisfied by a
`0.2.0-rc1` section, or the other way round.
A record says three things: **what ran**, **the numbers**, and **what failed**.
"Nothing failed" is a finding and is worth a line; an empty section is not a
record and the gate refuses it.
### Example of the shape
Illustrative only — no run has happened yet, and the placeholder version keeps
this block from ever satisfying the gate for a real release.
## Release drill — X.Y.Z — YYYY-MM-DD
Run ID: `drill-YYYYMMDD-NN` (the stack run these legs are the top of).
Stack under test: box `<sha>`, rig `<sha>`, cast `<sha>` — candidate
refs, pinned at mint time via RIG_REPO/RIG_REF.
Instances: A `coolify-a.example` (v4.x), B `coolify-b.example` (v4.x).
Manifest: `examples/two-env.yaml`, 3 applications, 2 environments.
Operator: @maintainer. Elapsed: 41m.
| Leg | Result | Notes |
|---|---|---|
| team | pass | 2 teams, 4 members reconciled |
| apply (A→B) | pass | 3 apps created, 11 env vars set |
| diff (idempotent) | pass | second apply: 0 changes |
| smoke | pass | 3/3 endpoints 200 |
| inventory | pass | 3 apps, 2 envs, matches manifest |
| emit-draft | pass | draft matches inventory round-trip |
| fleet | pass | both instances listed, versions read |
| destroy | pass | 3 apps removed, absence asserted |
| read-only guard | pass | write refused against B with the guard on |
Failures: none. One rough edge: `smoke` needed a 20s retry window on B —
filed as #NNN, not a release blocker.
## Waivers
A maintainer may ship without a passing drill, but **not without a record**.
The waiver is a section under the same heading saying who waived it, why, and
what is untested — a deliberate, reviewable commit. The gate requires a record,
not a pass, precisely so that skipping is visible.
## Records
*None yet.* cast has recorded no drill runs. New runs are appended below,
newest first.

145
drills/README.md Normal file
View file

@ -0,0 +1,145 @@
# Drills
Per-release evidence: **one file per version**, named `<version>.md`, where
`<version>` matches `package.json`'s `version` exactly — `0.2.0` is recorded in
`0.2.0.md`, `0.2.0-rc1` in `0.2.0-rc1.md`.
A release PR's version must have its file here, holding at least one
non-whitespace character, before CI will let it merge
(`.github/scripts/drill-recorded.sh`, wired into ci.yml).
## One file per version, and why the parser went away
Records used to share a single `drill/RUNS.md`, which meant the gate had to
*parse* it: a heading grammar, an em-dash field match, an optional ` — DATE`
tail, a whole-version comparison, a non-blank-body rule. That machinery existed
only because records shared a file — and it shipped two defects in review: a
`sed '/./,$!d'` extraction where `.` matches a space (so a heading followed by
one tab satisfied the gate), and heading-grammar drift from box's stricter form.
One file per version makes nearly all of that unrepresentable. `0.2.0.md` and
`0.2.0-rc1.md` are different files, so the whole-version rule is the
filesystem's rather than a comparison anyone can get wrong, and there is no
heading to drift. One rule survives: a file of only whitespace is not a record.
The directory is plain `drills/`, **not** `.drills/`. Dot-prefixed directories
are invisible to globs without `dotglob` — the blind spot behind #118, #121 and
box#116.
## What a record should contain
- **What ran** — which legs, against which manifest.
- **On what host** — the instances, their versions, who operated it.
- **The pinned candidate refs** — box, rig and cast SHAs under test.
- **The numbers** — counts, elapsed time, whatever the legs emit.
- **What failed** — and "nothing failed" is itself a finding worth a line.
**A failed drill is still a valid record.** The gate wants evidence, not
success. A maintainer may ship on a failed or partial drill — what they may not
do is ship on silence, so a **waiver** is also a legitimate record: say who
waived it, why, and what is untested. Requiring a record makes skipping a
deliberate, reviewable commit instead of the default outcome of forgetting.
## This directory is the record, not the instrument
cast has **no drill harness script of its own**. Its legs run by hand against
the documented procedure: two live Coolify instances and the full A→B
promotion —
team → apply → diff (idempotent) → smoke → inventory → emit-draft →
fleet → destroy → read-only guard
A harness would make the run reproducible; it would not make it recorded. Those
are separate problems, and this directory is the second one.
## Per-repo, by construction
cast records **cast's own** legs. It does not read box's or rig's drill records
to decide whether cast may ship: a cross-repo lookup silently degrades to
"pass" the moment it fails to resolve — the unreadable-rollup class of bug,
where a guard that cannot read its input reports the happy answer.
**The three repos' drills are independent.** Run them in any order, on any
schedule, in separate sittings. They are not phases of one script.
What makes that safe is that every drill **pins the same fixed set of candidate
refs** (`RIG_REPO` / `RIG_REF` are mint-time variables, default
`heavy-duty/rig@main`), so each drill exercises exactly the combination that
will ship rather than whatever `main` happens to be that afternoon. The run
drills **candidate refs, not released artifacts**.
That pinning — **not sequencing** — is what dissolves the box↔rig recursion.
box and rig *are* mutually recursive: rig builds the host that runs box, and
box's seed calls rig back to converge the guest. But candidate refs are static
identifiers that exist as soon as the release branches do, long before any
drill runs, so a cycle at runtime becomes independent tests against one fixed
pair. No repo has to be released before another can be drilled, and there is
**no fixed order in which the three releases must be published**.
Each repo also drills a **different thing**: box asserts the isolation contract
(the VM trust boundary), rig asserts convergence (a machine reaches its role,
idempotently), cast asserts promotion (A→B reproduces, and the diff is
idempotent). Three different exercises sharing a substrate — which is exactly
why the records are per-repo.
cast's legs are the **least coupled** of the three. Two Coolify instances can
be stood up by hand; the July drill did exactly that for instance B, via a
parameterised compose file. Nothing about cast's drill requires box or rig to
have been drilled first, or at all, on that day.
Within a single drill you obviously bring the substrate up before probing it —
a host before a guest before Coolify. That is how you run *a* drill, not an
ordering rule *between repos*.
If a defect shows up only in the combination: patch, re-drill, re-record. The
three releases converge on a set that holds together; they are not required to
be right in one pass.
**Drilling the candidate is drilling the release.** A release PR's diff is the
version file and `CHANGELOG.md` — nothing executable differs between the tree
that was drilled and the tree that ships, so the evidence carries across the
ceremony commit.
Each record cites the shared **run ID** naming the pinned set, plus the other
two repos' SHAs — which is what lets separate records be reassembled into one
picture, while each repo's evidence still lives in its own tree.
## Worked example
Illustrative only. The version below is a **placeholder that can never collide
with a real release** — a realistic-looking version here would be a real record
for it, and the gate would wave that release through on documentation. The
mechanism changed with the move to one file per version; the caution did not.
`drills/9.9.9.md`:
```markdown
# Release drill — 9.9.9 — YYYY-MM-DD
Run ID: `drill-YYYYMMDD-NN` (names the pinned candidate set).
Stack under test: box `<sha>`, rig `<sha>`, cast `<sha>` — candidate refs,
pinned at mint time via RIG_REPO/RIG_REF.
Instances: A `coolify-a.example` (v4.x), B `coolify-b.example` (v4.x),
B stood up by hand from the parameterised compose file.
Manifest: `examples/two-env.yaml`, 3 applications, 2 environments.
Operator: @maintainer. Elapsed: 41m.
| Leg | Result | Notes |
|---|---|---|
| team | pass | 2 teams, 4 members reconciled |
| apply (A→B) | pass | 3 apps created, 11 env vars set |
| diff (idempotent) | pass | second apply: 0 changes |
| smoke | pass | 3/3 endpoints 200 |
| inventory | pass | 3 apps, 2 envs, matches manifest |
| emit-draft | pass | draft matches inventory round-trip |
| fleet | pass | both instances listed, versions read |
| destroy | pass | 3 apps removed, absence asserted |
| read-only guard | pass | write refused against B with the guard on |
Failures: none. One rough edge: `smoke` needed a 20s retry window on B —
filed as #NNN, not a release blocker.
```
## Records
*None yet.* cast has recorded no drill runs.

230
test/release.test.ts
View file

@ -735,115 +735,134 @@ describe("changelog-monotonic.sh — release headings are append-only (#133)", (
// gate moves the requirement out of a reviewer's memory and into the tree.
//
// What it asserts is that a RECORD EXISTS, never that the drill passed — a
// maintainer waiver is legal and is itself a section in drill/RUNS.md. That is
// the point: skipping stays possible and stays a deliberate, reviewable
// commit. So the cases below are all about presence, emptiness, and whether
// the version was matched whole; none of them inspect a result.
// maintainer waiver is legal and is itself the content of drills/<ver>.md.
// That is the point: skipping stays possible and stays a deliberate,
// reviewable commit. So the cases below are all about presence and emptiness;
// none of them inspect a result.
//
// Every fixture carries its OWN package.json and RUNS.md inside the temp dir.
// Pointing the guard at the repo's real package.json would make these cases
// change meaning on every version bump — green or red depending on where the
// ceremony happens to be standing, which is the opposite of a fixture.
// ONE FILE PER VERSION. The earlier design kept every record in one
// drill/RUNS.md, so the guard parsed headings — and the heading grammar was
// where both of this feature's review defects lived (a whitespace bypass, and
// drift from box's stricter form). `0.2.0.md` and `0.2.0-rc1.md` are simply
// different files, so the whole-version rule is now the filesystem's rather
// than a comparison that can be got wrong, and the tests that existed only to
// pin heading syntax (including the stray-tail case) are gone with it.
//
// Every fixture carries its OWN package.json and its OWN drills dir inside a
// temp tree. Pointing the guard at the repo's real package.json would make
// these cases change meaning on every version bump — green or red depending
// on where the ceremony happens to be standing, the opposite of a fixture.
describe("drill-recorded.sh — a release version has a drill record", () => {
const work = tmp("cast-drill-");
/** A fixture tree: its own package.json + drill runs file. */
function treeWith(name: string, version: string, runs?: string): string {
/**
* A fixture tree: its own package.json, plus an optional `drills/` dir.
*
* @param records omit for NO drills dir at all; pass a map of
* `<version>.md` -> contents (possibly empty) to create the dir.
*/
function treeWith(
name: string,
version: string,
records?: Record<string, string>,
): string {
const dir = join(work, name);
mkdirSync(dir, { recursive: true });
writeFileSync(
join(dir, "package.json"),
`${JSON.stringify({ name: "cast", version }, null, 2)}\n`,
);
if (runs !== undefined) writeFileSync(join(dir, "RUNS.md"), runs);
if (records !== undefined) {
mkdirSync(join(dir, "drills"), { recursive: true });
for (const [file, body] of Object.entries(records)) {
writeFileSync(join(dir, "drills", file), body);
}
}
return dir;
}
const check = (dir: string) =>
run("bash", [DRILL, "RUNS.md", "package.json"], {}, dir);
run("bash", [DRILL, "drills", "package.json"], {}, dir);
const heading = (v: string) => `## Release drill — ${v} — 2026-07-21`;
const legs =
"\nInstances: A and B, live. All legs pass: team, apply, diff, smoke,\ninventory, emit-draft, fleet, destroy, read-only guard.\n";
"# Release drill\n\nInstances: A and B, live. All legs pass: team, apply,\ndiff, smoke, inventory, emit-draft, fleet, destroy, read-only guard.\n";
it("a -dev tree passes with no drill record at all — nothing ships from it", async () => {
it("a -dev tree passes with no drills dir at all — nothing ships from it", async () => {
const r = await check(treeWith("dev", "0.1.2-dev"));
expect(r.code).toBe(0);
expect(r.output).toContain("development tree");
});
it("a bare version with a matching, non-empty record passes", async () => {
const runs = `# Drill runs\n\n${heading("0.2.0")}\n${legs}`;
const r = await check(treeWith("ok", "0.2.0", runs));
const r = await check(treeWith("ok", "0.2.0", { "0.2.0.md": legs }));
expect(r.code).toBe(0);
expect(r.output).toContain("0.2.0");
});
it("a bare version with NO record fails, naming the version", async () => {
const runs = "# Drill runs\n\n*None yet.*\n";
const r = await check(treeWith("none", "0.2.0", runs));
it("a bare version with NO drills dir fails, naming the version", async () => {
const r = await check(treeWith("nodir", "0.2.0"));
expect(r.code).toBe(1);
expect(r.output).toContain("no drill record for version '0.2.0'");
expect(r.output).toContain("version 0.2.0 is a release");
expect(r.output).toContain("drills/0.2.0.md");
});
// A heading with nothing under it is ceremony without evidence — the exact
// shape a hurried release produces, and the one a heading-only check would
// wave through. release-notes.sh refuses an empty section for the same
// reason.
it("a present-but-EMPTY record fails — a heading is not a record", async () => {
const runs = `# Drill runs\n\n${heading("0.2.0")}\n\n\n## Notes\n\nUnrelated.\n`;
const r = await check(treeWith("empty", "0.2.0", runs));
it("a drills dir with no file for THIS version fails", async () => {
const r = await check(
treeWith("othersonly", "0.2.0", { "0.1.0.md": legs, "README.md": legs }),
);
expect(r.code).toBe(1);
expect(r.output).toContain("no drill record for version '0.2.0'");
expect(r.output).toContain("drills/0.2.0.md");
});
// ...and whitespace is not evidence either. The first cut extracted with
// `sed '/./,$!d'`, where `.` matches a space, so a heading followed by one
// tab passed the gate — while the comment above the extractor claimed the
// opposite. An evidence-free release for the price of an invisible
// character, on the one check whose whole job is to demand evidence.
// Found independently by all three reviewers on #138.
it("a record body of only spaces and tabs fails (#138)", async () => {
const runs = `# Drill runs\n\n${heading("0.2.0")}\n \n\t\n\n## Notes\n\nUnrelated.\n`;
const r = await check(treeWith("blank", "0.2.0", runs));
// A file that exists but says nothing is ceremony without evidence — the
// exact shape a hurried release produces, and the one an `[ -f ]` check
// would wave through. release-notes.sh refuses an empty section likewise.
it("a present-but-EMPTY record fails — a filename is not a record", async () => {
const r = await check(treeWith("empty", "0.2.0", { "0.2.0.md": "" }));
expect(r.code).toBe(1);
expect(r.output).toContain("no drill record for version '0.2.0'");
expect(r.output).toContain("drills/0.2.0.md");
});
// The sibling guards must agree about what a heading IS, not just about the
// version in it. box#149 requires the version to be the last field or be
// followed by the em dash; without the same constraint here, this heading is
// a record in cast and not in box.
it("a stray tail after the version is not a heading — in step with box#149", async () => {
const runs = `# Drill runs\n\n## Release drill — 0.2.0 stray words\n${legs}`;
const r = await check(treeWith("tail", "0.2.0", runs));
// ...and whitespace is not evidence either. This is the one surviving piece
// of the heading-parser era: the first cut extracted with `sed '/./,$!d'`,
// where `.` matches a space, so a heading followed by one tab passed the
// gate — while the comment above the extractor claimed the opposite. An
// evidence-free release for the price of an invisible character, on the one
// check whose whole job is to demand evidence. Found independently by all
// three reviewers on #138. The file layout changed; this rule did not.
it("a record of only spaces, tabs and newlines fails (#138)", async () => {
const r = await check(
treeWith("blank", "0.2.0", { "0.2.0.md": " \n\t\n \n" }),
);
expect(r.code).toBe(1);
expect(r.output).toContain("no drill record for version '0.2.0'");
expect(r.output).toContain("drills/0.2.0.md");
});
// The version is matched WHOLE, both directions — release-notes.sh's trap,
// solved the same way (awk field equality, no regex, no dot-escaping). A
// release candidate's drill is not the release's drill: different tree,
// different build, and a prefix match would silently accept it.
// The version is matched WHOLE, both directions — but now by the FILESYSTEM
// rather than by a comparison. A release candidate's drill is not the
// release's drill: different tree, different build, and under the old
// heading parser a prefix match would have silently accepted it.
it("0.2.0-rc1's record does NOT satisfy 0.2.0", async () => {
const runs = `# Drill runs\n\n${heading("0.2.0-rc1")}\n${legs}`;
const r = await check(treeWith("rc-for-bare", "0.2.0", runs));
const r = await check(
treeWith("rc-for-bare", "0.2.0", { "0.2.0-rc1.md": legs }),
);
expect(r.code).toBe(1);
expect(r.output).toContain("no drill record for version '0.2.0'");
expect(r.output).toContain("drills/0.2.0.md");
});
it("...and 0.2.0's record does NOT satisfy 0.2.0-rc1", async () => {
const runs = `# Drill runs\n\n${heading("0.2.0")}\n${legs}`;
const r = await check(treeWith("bare-for-rc", "0.2.0-rc1", runs));
const r = await check(
treeWith("bare-for-rc", "0.2.0-rc1", { "0.2.0.md": legs }),
);
expect(r.code).toBe(1);
expect(r.output).toContain("no drill record for version '0.2.0-rc1'");
expect(r.output).toContain("drills/0.2.0-rc1.md");
});
it("...while each still matches its own record", async () => {
const runs = `# Drill runs\n\n${heading("0.2.0")}\n${legs}\n${heading("0.2.0-rc1")}\n${legs}`;
expect((await check(treeWith("both-a", "0.2.0", runs))).code).toBe(0);
expect((await check(treeWith("both-b", "0.2.0-rc1", runs))).code).toBe(0);
const both = { "0.2.0.md": legs, "0.2.0-rc1.md": legs };
expect((await check(treeWith("both-a", "0.2.0", both))).code).toBe(0);
expect((await check(treeWith("both-b", "0.2.0-rc1", both))).code).toBe(0);
});
// The message is the whole user interface of a blocking guard. A failure
@ -851,22 +870,16 @@ describe("drill-recorded.sh — a release version has a drill record", () => {
// than satisfied — including the waiver, which must be visibly ALLOWED or
// somebody will route around the gate instead of recording one.
it("the failure names the unblock: run the drill, or record a waiver", async () => {
const r = await check(treeWith("unblock", "0.2.0", "# Drill runs\n"));
const r = await check(treeWith("unblock", "0.2.0", {}));
expect(r.code).toBe(1);
expect(r.output).toContain("## Release drill — 0.2.0");
expect(r.output).toContain("drills/0.2.0.md");
expect(r.output).toContain("run the drill and record it");
expect(r.output).toContain("WAIVER");
expect(r.output).toContain("RECORD, not a");
});
it("a missing runs file on a release tree refuses — never a silent pass", async () => {
const r = await check(treeWith("norunsfile", "0.2.0"));
expect(r.code).toBe(1);
expect(r.output).toContain("no RUNS.md at all");
});
it("a missing version file refuses by path", async () => {
const r = await run("bash", [DRILL, "RUNS.md", "nope.json"], {}, work);
const r = await run("bash", [DRILL, "drills", "nope.json"], {}, work);
expect(r.code).toBe(1);
expect(r.output).toContain("no such file");
});
@ -877,20 +890,50 @@ describe("drill-recorded.sh — a release version has a drill record", () => {
expect(r.output).toContain("usage:");
});
// The REAL tree, run with the REAL defaults — the same discipline as the
// arming rule's "the REAL tree is armed". Whatever state the ceremony is in,
// this repo must satisfy its own gate.
it("the real tree satisfies its own gate, with default arguments", async () => {
// The REAL tree, run with the REAL defaults. The property is that the
// guard's VERDICT IS CORRECT FOR THIS TREE — not that it always passes.
//
// The previous wording here claimed "whatever state the ceremony is in, this
// repo must satisfy its own gate", and that is exactly wrong: a ceremony tree
// CANNOT satisfy the gate until a human has run the drill and written the
// record, which is the entire point of the gate. Demanding exit 0 made this
// suite un-greenable on every release branch before its drill, and surfaced
// as a `build` failure rather than as the gate doing its job — the same
// misattribution shape as box#146. Caught when box#148 went red for the
// wrong-looking reason.
it("the guard's verdict on the real tree matches the tree's own state", async () => {
const version = realVersion();
const r = await run("bash", [DRILL], {}, ROOT);
if (version.endsWith("-dev")) {
// Vacuous: nothing ships from a development tree.
expect(r.code).toBe(0);
expect(r.output).toContain("development tree");
return;
}
// A ceremony tree: green only once its record exists.
const recorded =
existsSync(join(ROOT, "drills", `${version}.md`)) &&
readFileSync(join(ROOT, "drills", `${version}.md`), "utf8").trim() !== "";
expect(r.code).toBe(recorded ? 0 : 1);
if (!recorded) expect(r.output).toContain("no drill record");
});
it("drill/RUNS.md documents the heading format the gate parses", () => {
const runs = readFileSync(join(ROOT, "drill/RUNS.md"), "utf8");
expect(runs).toContain("## Release drill — X.Y.Z — YYYY-MM-DD");
it("drills/README.md documents the naming rule and the waiver", () => {
const doc = readFileSync(join(ROOT, "drills/README.md"), "utf8");
expect(doc).toContain("<version>.md");
// The placeholder example version can never collide with a real release:
// under any scheme, a realistic version in the docs is a real record.
expect(doc).toContain("drills/9.9.9.md");
// Per-repo by construction: cast records cast's legs and never reads
// another repo's log to decide whether cast may ship.
expect(runs).toMatch(/waiver/i);
// another repo's record to decide whether cast may ship.
expect(doc).toMatch(/waiver/i);
expect(doc).toMatch(/failed drill is still a valid record/i);
});
it("the old single-file log is gone — records are per version", () => {
expect(existsSync(join(ROOT, "drill/RUNS.md"))).toBe(false);
});
it("ci.yml runs it, ungated by event or label", () => {
@ -914,24 +957,31 @@ describe("drill-recorded.sh — a release version has a drill record", () => {
expect(block).not.toMatch(/^ {8}if:/m);
});
it("CONTRIBUTING documents the gate and the ONE-RUN stack drill", () => {
// The three drills are INDEPENDENT — any order, any schedule, separate
// sittings. What makes that safe is that each pins the same fixed candidate
// refs, so every drill exercises the combination that will ship. That
// pinning, not sequencing, is what dissolves the box<->rig recursion: the
// refs are static identifiers that exist as soon as the release branches
// do. The docs must not re-acquire an ordering rule between repos.
it("CONTRIBUTING documents the gate and the INDEPENDENT, ref-pinned drills", () => {
const doc = readFileSync(join(ROOT, "CONTRIBUTING.md"), "utf8");
expect(doc).toContain("drill/RUNS.md");
expect(doc).toContain("drills/X.Y.Z.md");
expect(doc).toContain("drill-recorded.sh");
// One orchestrated run over the whole stack, in order: rig builds the
// host (installing box), box mints a seed, the seed calls rig back to
// converge, cast's legs run on the result.
expect(doc).toContain("--host yes");
expect(doc).toContain("box new");
expect(doc).toContain("rig bootstrap <tenant>-box");
// rig sits BELOW box and ABOVE it — mutually recursive, not linear. So
// the docs must not promise a fixed release order, and must say the run
// pins CANDIDATE refs, which is what dissolves the chicken-and-egg.
expect(doc).toMatch(/drills are independent/i);
expect(doc).toMatch(/any order/i);
expect(doc).toMatch(/pins the same fixed set of\s+candidate\s+refs/i);
expect(doc).toMatch(/not sequencing/i);
// box and rig stay mutually recursive; the pinning is what makes that a
// non-problem, so both halves have to survive together.
expect(doc).toMatch(/mutually recursive/);
expect(doc).toContain("RIG_REF");
expect(doc).toMatch(/candidate refs, not released artifacts/);
expect(doc).toMatch(/no fixed order|not.*published in a fixed order/i);
// Three records, one run: each repo cites the shared run ID.
// Each repo drills a different thing — which is WHY records are per-repo.
expect(doc).toMatch(/isolation\s+contract/i);
expect(doc).toMatch(/convergence/i);
expect(doc).toMatch(/promotion/i);
// Separate records, one pinned set: each cites the shared run ID.
expect(doc).toMatch(/run ID/i);
});
});