Merge pull request #149 from dan-claude-bot/feat/drill-gate

feat: CI refuses a release PR with no drill record
This commit is contained in:
Daniel Marin 2026-07-21 17:17:39 +01:00 committed by GitHub
commit 115b9c026e
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
6 changed files with 373 additions and 4 deletions

128
.github/scripts/drill-recorded.sh vendored Executable file
View file

@ -0,0 +1,128 @@
#!/usr/bin/env bash
set -euo pipefail
# drill-recorded.sh [<runs-file>] [<version-file>] — assert that a RELEASE
# tree carries a drill record: that the full real-hardware drill this repo
# says a release rests on was actually run for this version, and written
# down in drill/RUNS.md.
#
# CONTRIBUTING.md has said since #96 that "this PR is where the release
# ritual hangs: the full drill on real hardware, recorded in drill/RUNS.md".
# No release has ever done it. #95, #114 and #148 all shipped as a VERSION
# bump plus a CHANGELOG.md stamp and nothing else, and the file they were
# supposed to append to has no '## Release drill' section anywhere in it.
# That is three releases through the same gap, because the gate was a
# sentence in a document and the only thing standing on it was a reviewer
# remembering to ask. A reviewer finally did — which is the point: the ONE
# time it was caught is the time somebody happened to look, and that is not a
# gate, it is luck with good manners.
#
# So the rule moves into CI, where it fires on every release PR whether or
# not anyone is paying attention. The rule, keyed on VERSION for the same
# reason changelog-armed.sh is — the two states are genuinely different:
#
# VERSION ends in -dev -> PASS. A development tree ships nothing, so
# there is nothing for it to have proven. Almost
# every PR in this repo is this case, and a guard
# that nagged all of them would be turned off.
# VERSION is bare -> the ceremony tree, the one about to ship.
# drill/RUNS.md MUST carry a section headed
# '## Release drill — <version>' (optionally with
# ' — <date>' after it), and that section must
# have prose in it.
#
# What this guard asserts is a RECORD, deliberately — not a passing drill.
# CI cannot run the drill: it wants real hardware, a real Incus, and the
# better part of an hour (see ci.yml, which says exactly this about the
# rehearsal job it runs instead). What CI can do is refuse to let a release
# claim a ritual it left no evidence of. That also leaves the maintainer
# waiver intact and honest: a release that must ship without a full drill
# records WHY under its own heading, which is a deliberate, reviewable commit
# in the diff — rather than the silent skip that got us here.
#
# A file of its own (not inlined in ci.yml) so test/release.sh can drive it
# against constructed trees for both states — the same discipline as
# changelog-armed.sh and release-notes.sh.
runs="${1:-drill/RUNS.md}"
version_file="${2:-VERSION}"
[ -f "$version_file" ] || { echo "drill-recorded: no such file: $version_file" >&2; exit 1; }
ver="$(tr -d '[:space:]' < "$version_file")"
[ -n "$ver" ] || { echo "drill-recorded: $version_file is empty" >&2; exit 1; }
case "$ver" in
*-dev)
# Nothing to assert, and saying so is the point: the operator reading a
# green log should be able to tell "the guard passed" from "the guard
# decided this tree was not its business".
echo "drill-recorded: VERSION '$ver' is a development tree — nothing to assert; only ceremony trees ship"
exit 0
;;
esac
[ -f "$runs" ] || { echo "drill-recorded: no such file: $runs" >&2; exit 1; }
# The section for this version: everything between its own heading and the
# next '## '. The version is compared WHOLE — as a field, never as a
# substring or a regex — so '0.9.0' can never be satisfied by a
# '0.9.0-rc1' section (or vice versa), and there are no dots to escape.
# release-notes.sh solves the identical trap the identical way; the two
# scripts must not disagree about what "the section for X" means.
#
# The heading shape is '## Release drill — <ver>' with an OPTIONAL
# ' — <date>' tail, i.e. fields: '##' 'Release' 'drill' '—' '<ver>' ['—' ...].
# Pinning the leading fields as well as the version keeps some other '## '
# heading that merely mentions the number from counting as a record.
#
# The em dash is passed IN as a variable rather than written into the awk
# program, because '\x' escapes in an awk string are a gawk extension and CI
# runs on ubuntu-latest, where awk is mawk.
#
# `grab && NF` is the non-blank rule, and it is load-bearing rather than
# tidiness. NF is 0 on a line that is empty OR contains only whitespace, so
# `record` ends up non-empty exactly when a line with actual content exists.
# The first cut of this piped through `sed '/./,$!d'` instead, which drops
# leading blank lines but keeps a line of spaces — `.` matches a space. A
# heading followed by nothing but a tab therefore satisfied the guard and
# shipped an evidence-free release, while the failure text below promised "at
# least one non-blank line". The implementation was looser than its own
# contract, which on a gate is the whole ballgame: the bypass costs one
# invisible character. Caught by all three reviewers on #149, independently.
record="$(awk -v ver="$ver" -v dash="—" '
/^## / {
grab = ($2 == "Release" && $3 == "drill" && $4 == dash && $5 == ver \
&& (NF == 5 || $6 == dash))
next
}
grab && NF { print }
' "$runs")"
if [ -z "$record" ]; then
cat >&2 <<EOF
drill-recorded: VERSION is '$ver' — a release — and $runs has no drill record
for it. The heading this looks for is:
## Release drill — $ver — DATE
...with at least one non-blank line under it. Either the section is absent
entirely, or it is present and empty; both mean the same thing, which is
that this release is asserting a ritual it has left no evidence of.
The unblock is to RUN THE DRILL (drill/drill.sh, on real hardware) and
record it in $runs under that heading — what it measured, what it found,
what it cost. CI cannot run the drill for you; it can only refuse a release
that never ran one.
If this release must ship without a full drill, that is a maintainer's call
to make and it is still recorded: write the section under the same heading
and say plainly that the drill was WAIVED and why. The guard requires a
record, not a passing result — so a skip is a visible, reviewable line in
the diff rather than the silent gap that let #95, #114 and #148 all ship
unproven. See CONTRIBUTING.md, "Releases".
EOF
exit 1
fi
echo "drill-recorded: VERSION '$ver' has a drill record in $runs"

View file

@ -88,6 +88,23 @@ jobs:
# the fallback it resolves to the pushed branch, whose merge base with
# HEAD is HEAD or its parent: containment passes vacuously, exactly as the
# old `if` intended, while uniqueness now runs on every push.
# ...and a RELEASE tree has a drill record. Its own step for the same
# reason as the two above — when it goes red the log names the invariant
# that broke — and a third invariant again: armed is about the shape of
# the changelog, monotonic about the diff, this one about whether the
# ritual CONTRIBUTING.md hangs on the release PR actually happened.
#
# NOT pull-request-only, and for the same reasoning the monotonic guard
# was un-gated in #143: the release does not finish at the PR. The merge
# that publishes it is a PUSH to main carrying the same bare VERSION,
# and it is release.yml's trigger — so a check that only ever ran on the
# PR would leave the tree that actually ships unasserted, and any bare
# VERSION reaching main by another route (a direct push, a backfill,
# the manual tag path) unasserted forever. On a -dev tree, which is
# every ordinary push and PR, the script no-ops green by design, so
# running it everywhere costs nothing and closes the route.
- name: a release tree has a drill record
run: bash .github/scripts/drill-recorded.sh
- name: no shipped changelog heading was deleted or duplicated
env:
CHANGELOG_MONOTONIC_STRICT: '1'

View file

@ -17,6 +17,7 @@ which records not just what changed but what each drill run proved.
- Every fresh mint marks a `pristine` snapshot, before rig converges anything
(#104, heavy-duty/rig#62)
- A mint that converges a tenant role marks a `bootstrapped` snapshot (#130)
- CI refuses a release PR with no drill record in `drill/RUNS.md`
### Changed

View file

@ -156,10 +156,78 @@ A release is a PR, and merging it ships it
stamp passes it by construction — rewriting `## Unreleased` into
`## X.Y.Z — DATE` adds a heading and removes none.
This PR is where the release ritual hangs:
the full drill on real hardware, recorded in
[drill/RUNS.md](drill/RUNS.md) — CI proves the tier's semantics on every
PR, a release still proves the boundary.
**The drill gates the release, and CI enforces it.** The release PR's
flow is: draft → ready → bot round → **drill**`state:needs-human`
maintainer merge (which *is* the release). CI proves the tier's semantics
on every PR; the drill is what proves the VM trust boundary, and it wants
real hardware and the better part of an hour, so it runs on the release
PR's branch and nowhere else.
Record it in [drill/RUNS.md](drill/RUNS.md) under its own heading:
```markdown
## Release drill — X.Y.Z — DATE
```
...with the run under it — what it measured, what it found, what it cost.
[.github/scripts/drill-recorded.sh](.github/scripts/drill-recorded.sh)
asserts exactly that on every tree with a bare `VERSION`, which is every
`release` PR and the merge that publishes it; a `-dev` tree passes with
nothing to assert. It is **no longer a thing a reviewer has to remember**.
It became CI's job because remembering did not work. The sentence this
paragraph replaces described a step **no release had ever performed**
`drill/RUNS.md` carried no `## Release drill` section at all — and #95,
#114 and #148 all shipped through the gap as a `VERSION` bump plus a
`CHANGELOG.md` stamp. The one time it was caught was the one time somebody
happened to look, which is not a gate.
**The drill is ONE orchestrated run over the whole stack**, not three
drills in a queue. box and rig are **mutually recursive**, so there is no
linear order to put them in: rig sits *below* box as the host-builder
(`rig bootstrap --host yes` installs box and runs `setup-host`) and
*above* it as the guest-converger (a `box new` seed's cloud-init curls
rig's installer and runs `rig bootstrap <tenant>-box`). box's own source
says as much — the seeds "invert the rig→box install edge (rig#28: rig
installs box on hosts; now box guests install rig)". The run goes:
1. `rig bootstrap --host yes` on a bare Debian host — installs box, runs
`setup-host`
2. `box new` mints a creds-free seed
3. the seed converges itself via `rig bootstrap <tenant>-box`
4. cast on top
It drills **candidate refs, not released artifacts.** `RIG_REPO` and
`RIG_REF` are mint-time environment variables (defaults
`heavy-duty/rig` and `main`, `bin/box`), so a run pins the exact commits
under test. That dissolves the chicken-and-egg: **no repo has to be
released before another can be drilled.** And drilling the candidate *is*
drilling the release — a release PR's diff is `VERSION` plus
`CHANGELOG.md` and nothing else, so no executable byte differs between
the tree that was drilled and the tree that ships.
One run emits **one shared run ID**. Each repo records its own legs under
its own `## Release drill — X.Y.Z — DATE`, citing that run ID and the
other two repos' commit SHAs, so the three records reconcile into a single
run afterwards. The guard reads only this repo's file — it asserts box's
record exists, never the other two.
**A known gap, and box is where it belongs.** A *released* box still
defaults `RIG_REF` to `main`, so what a user mints a week after a drill is
not the combination that was drilled — the guest converges against
whatever rig's main has become since. Pinning `RIG_REF` to a released rig
tag in the templates is the outstanding step from
[#81](https://github.com/heavy-duty/box/issues/81) (rig#32 step 5), and it
is what would make a drilled combination reproducible for users. This PR
does not fix it; box's source already says the two directions "track main
unpinned today, said honestly ... until a release flow exists".
A maintainer **waiver** is possible, and it is still written down. The
guard requires a *record*, not a passing result, so a release that must
ship without a full drill puts the section under the same heading and says
plainly that the drill was waived and why. Skipping then costs a
deliberate, reviewable line in the diff — which is precisely what the
three silent skips above did not.
2. **The maintainer's merge IS the release.**
[release.yml](.github/workflows/release.yml) fires on the merged,
`release`-labeled PR and asserts before creating anything: `VERSION` at

View file

@ -36,6 +36,22 @@ PR carries as many as apply.
| `blocker:conflict` | `#B60205` | GitHub says `CONFLICTING` — the agent owes a **rebase** | it merges cleanly |
| `blocker:ci-red` | `#B60205` | a check failed — the agent owes a **fix**, which a rebase will not provide | checks are green |
| `blocker:unrequested` | `#E99695` | this head has no verdict from somebody — never reviewed, or staled by a push — and **nobody was asked** for one | reviews are requested |
| `blocker:drill-pending` | `#B60205` | a `release` PR whose version has no drill record in [drill/RUNS.md](drill/RUNS.md) — the ceremony is **correct but unevidenced** | the drill is run and recorded (or a waiver is recorded) under `## Release drill — X.Y.Z` |
`blocker:drill-pending` is the one blocker that says the diff is *fine*. The
version is stamped, the changelog is right, CI's other guards are green — what
is missing is the evidence that the release was proven on real hardware, which
[.github/scripts/drill-recorded.sh](.github/scripts/drill-recorded.sh) refuses
to let a release ship without (CONTRIBUTING.md, "Releases"). Naming it
separately from `blocker:ci-red` matters because the two owe different work: a
red check is a fix in the branch, a pending drill is an hour on a real host.
It must be **created by a maintainer account** — the bot account gets a `403`
creating labels, so the labels workflow's bootstrap dispatch cannot mint it.
Until it exists in the repo, `blocked` stands in: it is the closest true thing
(the PR is waiting on something outside itself) and, usefully, the staleness
sweep already skips it, so a release parked overnight on a drill does not
collect a `stale`.
One rule joins the axes: **`state:needs-human` requires zero blockers.** Any
blocker means the work is the agent's, whatever the review round says.

View file

@ -278,6 +278,145 @@ check "CONTRIBUTING: the ceremony re-arms '## Unreleased' after stamping" 0 "" \
check "CONTRIBUTING: ...and names the guard that enforces it" 0 "" \
grep -qF 'changelog-armed.sh' "$ROOT/CONTRIBUTING.md"
# ---------------------------------------------------------------------------
# drill-recorded.sh — a RELEASE tree has a drill record in drill/RUNS.md.
#
# The gap this closes is not a drift or a race; it is a rule that was only
# ever a sentence. CONTRIBUTING.md has said since #96 that the release PR is
# where the full real-hardware drill hangs — and #95, #114 and #148 all
# shipped without one, because nothing but a reviewer's memory stood on it.
#
# Both states are constructed as real trees, same discipline as the armed
# block above, and for a sharper reason here: the -dev pass is not a
# convenience, it is what keeps the guard installable. A version of this rule
# that fired on every tree would be red on every ordinary PR in the repo and
# would be switched off inside a day.
#
# Every fixture carries its OWN VERSION and its OWN RUNS.md. Reaching for
# $ROOT/VERSION instead is exactly the coupling #146 had to fix: the suite
# then passes or fails on whichever state the repo happens to be in, so it
# goes red on the ceremony tree — the one tree where the release suite most
# needs to be trustworthy.
# ---------------------------------------------------------------------------
DRILLED="$ROOT/.github/scripts/drill-recorded.sh"
check "drill-recorded: runnable bash" 0 "" bash -n "$DRILLED"
check "drill-recorded: executable" 0 "" test -x "$DRILLED"
# dtree <dir> <version> <runs-body...> — a two-file tree, self-contained
dtree() {
local d="$WORK/$1" v="$2"; shift 2
mkdir -p "$d"
printf '%s\n' "$v" > "$d/VERSION"
{ echo "# Drill run log"; echo; printf '%s\n' "$@"; } > "$d/RUNS.md"
echo "$d"
}
drilled() { bash "$DRILLED" "$1/RUNS.md" "$1/VERSION"; }
# --- the -dev steady state: nothing to assert ------------------------------
# No record at all, and that must PASS. This is the case that makes the guard
# survivable on an ordinary PR.
T="$(dtree drill-dev 0.8.1-dev 'No drill sections in here at all.')"
check "drill-recorded: a -dev tree with NO record at all passes" 0 "nothing to assert" drilled "$T"
check "drill-recorded: ...and says why it declined to judge" 0 "development tree" drilled "$T"
# --- the ceremony tree: a record is required and must carry prose ----------
T="$(dtree drill-ok 0.9.0 \
'## Release drill — 0.9.0 — 2026-07-21' '' '- 47/47 on real hardware' '' \
'## Run history' '' '- older prose')"
check "drill-recorded: a bare VERSION with a matching non-empty record passes" \
0 "has a drill record" drilled "$T"
# The date tail is optional; the heading without it is still a record.
T="$(dtree drill-nodate 0.9.0 '## Release drill — 0.9.0' '' '- ran it, 47/47')"
check "drill-recorded: ...and the ' — DATE' tail is optional" 0 "has a drill record" drilled "$T"
T="$(dtree drill-none 0.9.0 '## Run history' '' '- prose about earlier runs')"
check "drill-recorded: a bare VERSION with NO record fails" 1 "no drill record" drilled "$T"
check "drill-recorded: ...and the failure names the version" 1 "VERSION is '0.9.0'" drilled "$T"
check "drill-recorded: ...and names the unblock — run the drill and record it" \
1 "RUN THE DRILL" drilled "$T"
check "drill-recorded: ...and offers the recorded maintainer waiver as the other way out" \
1 "WAIVED" drilled "$T"
# A heading with nothing under it is the same failure as no heading: the
# ceremony wrote the shape and never wrote the evidence.
T="$(dtree drill-empty 0.9.0 '## Release drill — 0.9.0 — 2026-07-21' '' '' '## Run history' '' '- older prose')"
check "drill-recorded: a bare VERSION whose record is present but EMPTY fails" \
1 "no drill record" drilled "$T"
# ...and WHITESPACE is not evidence either. The first cut extracted with
# `sed '/./,$!d'`, where `.` matches a space — so a heading followed by one
# tab passed the guard while the failure text promised "at least one non-blank
# line". An evidence-free release for the price of an invisible character, on
# the one check whose entire job is to demand evidence. All three reviewers on
# #149 found it independently, which is the level of scrutiny it deserved and
# not a level it should ever need again.
T="$(dtree drill-blank 0.9.0 '## Release drill — 0.9.0 — 2026-07-21' ' ' ' ' '' '## Run history' '' '- older prose')"
check "drill-recorded: a record body of only spaces and tabs fails (#149)" \
1 "no drill record" drilled "$T"
# --- the version is matched WHOLE, both directions -------------------------
# release-notes.sh's trap, in a new file: a substring match would let an rc
# drill stand in for the release it was a candidate for, which is the one
# thing a release-candidate drill by definition did not measure.
T="$(dtree drill-rc-only 0.9.0 '## Release drill — 0.9.0-rc1 — 2026-07-20' '' '- the rc drill')"
check "drill-recorded: a 0.9.0-rc1 record does NOT satisfy 0.9.0" 1 "no drill record" drilled "$T"
T="$(dtree drill-rel-only 0.9.0-rc1 '## Release drill — 0.9.0 — 2026-07-21' '' '- the release drill')"
check "drill-recorded: ...and a 0.9.0 record does NOT satisfy 0.9.0-rc1" 1 "no drill record" drilled "$T"
# ...and with both present, each resolves to its own section and neither
# borrows the other's prose.
T="$(dtree drill-both 0.9.0 \
'## Release drill — 0.9.0-rc1 — 2026-07-20' '' '- the rc drill' '' \
'## Release drill — 0.9.0 — 2026-07-21' '' '- the release drill')"
check "drill-recorded: with both present, the release resolves to its own" \
0 "has a drill record" drilled "$T"
# The sharp version of the same claim: the rc section is full, the release's
# own section is EMPTY. If emptiness were judged over anything but this
# version's own lines, this tree would pass on the rc's prose.
T="$(dtree drill-borrow 0.9.0 \
'## Release drill — 0.9.0-rc1 — 2026-07-20' '' '- the rc drill' '' \
'## Release drill — 0.9.0 — 2026-07-21' '')"
check "drill-recorded: ...and an empty release section cannot borrow the rc's prose" \
1 "no drill record" drilled "$T"
# --- degenerate trees refuse rather than pass by accident ------------------
check "drill-recorded: a missing VERSION refuses by path" 1 "no such file" \
bash "$DRILLED" "$WORK/drill-ok/RUNS.md" "$WORK/nope-version"
mkdir -p "$WORK/drill-empty-ver"; : > "$WORK/drill-empty-ver/VERSION"
check "drill-recorded: an empty VERSION refuses" 1 "is empty" \
bash "$DRILLED" "$WORK/drill-ok/RUNS.md" "$WORK/drill-empty-ver/VERSION"
# The runs file is only needed once the tree is a release — a -dev tree must
# not be refused for a file it has no business reading.
check "drill-recorded: a release tree with a missing runs file refuses by path" \
1 "no such file" bash "$DRILLED" "$WORK/nope-runs.md" "$WORK/drill-ok/VERSION"
check "drill-recorded: ...but a -dev tree does not even look for it" 0 "nothing to assert" \
bash "$DRILLED" "$WORK/nope-runs.md" "$WORK/drill-dev/VERSION"
# --- and the tree under test ----------------------------------------------
check "drill-recorded: THIS tree passes its own guard" 0 "" \
bash "$DRILLED" "$ROOT/drill/RUNS.md" "$ROOT/VERSION"
# The guard is only a guard if CI runs it, and only a rule if the document
# that used to carry the rule now points at it.
check "ci.yml: runs the drill-recorded guard" 0 "" \
grep -qF 'drill-recorded.sh' "$ROOT/.github/workflows/ci.yml"
check "CONTRIBUTING: names the guard that enforces the drill" 0 "" \
grep -qF 'drill-recorded.sh' "$ROOT/CONTRIBUTING.md"
check "CONTRIBUTING: documents the drill record's heading format" 0 "" \
grep -qF '## Release drill — X.Y.Z — DATE' "$ROOT/CONTRIBUTING.md"
check "CONTRIBUTING: the drill is ONE run over the whole stack, not a queue" 0 "" \
grep -qF 'ONE orchestrated run' "$ROOT/CONTRIBUTING.md"
# The load-bearing correction: box and rig are mutually recursive, so a
# "release A before you can drill B" rule is not merely bureaucratic, it is
# unsatisfiable in principle. Pinning the mechanism that dissolves it —
# mint-time candidate refs — because that is the claim a future editor is
# most likely to "simplify" back into a linear order.
check "CONTRIBUTING: ...drilling candidate refs, not released artifacts" 0 "" \
grep -qF 'candidate refs, not released artifacts' "$ROOT/CONTRIBUTING.md"
check "CONTRIBUTING: ...and the RIG_REF gap is stated as outstanding (#81)" 0 "" \
grep -qF 'RIG_REF' "$ROOT/CONTRIBUTING.md"
check "LABELS: documents blocker:drill-pending" 0 "" \
grep -qF 'blocker:drill-pending' "$ROOT/LABELS.md"
# ---------------------------------------------------------------------------
# changelog-monotonic.sh (#122) — no SHIPPED release heading may be DELETED.
#