feat: CI refuses a release PR with no drill record #149
Labels
No labels
blocked
blocker:ci-red
blocker:conflict
blocker:drill-pending
blocker:unrequested
bug
claimed
documentation
enhancement
epic
merge-next
needs-triage
ready
release
scope:cli
scope:drill
scope:host
scope:installer
scope:templates
scope:tiers
stale
state:addressing
state:bots-reviewing
state:building
state:needs-human
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: heavy-duty/box#149
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/drill-gate"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
CI now refuses a
releasePR that has no drill record. The rule stops being something a reviewer has to remember.Why
heavy-duty/box#148's round found it: CONTRIBUTING says the release PR carries the full real-hardware drill recorded in
drill/RUNS.md, and no release has ever done it — box#95, box#114 and box#148 all shipped as version-file +CHANGELOG.mdonly. One of three bots caught it; two did not. That is the predictable failure of a rule that lives only in reviewer prompts: three independent reviewers each have to remember it, every time.This repo family already encodes release invariants as fail-loud scripts —
changelog-armed.sh,changelog-monotonic.sh,release-notes.sh. The drill gate becomes one too.The guard
.github/scripts/drill-recorded.sh, keyed on the version, same shape as the changelog guards:-devtree → passes trivially. Nothing ships from a development tree.## Release drill — X.Y.Z — DATEsection indrill/RUNS.mdthat extracts non-empty.Version is compared whole, so
0.9.0-rc1cannot satisfy0.9.0or vice-versa —release-notes.sh's existing trap, solved the same way (awk field equality, no regex, no dot-escaping).Not gated on the
releaselabel. Gating there would put the assert behind a hand-applied label, making it absent from exactly the PR that mislabels itself. It keys off the version file instead.Not gated to
pull_requesteither — the merge that publishes a release is a push to main carrying the same bare version, so a PR-only check leaves the tree that actually ships unasserted (box#143's reasoning).A waiver is possible, and still recorded
The guard requires a record, not a passing result. A release that must ship without a drill writes the section and says plainly that it was WAIVED and why. That turns a skip into a visible line in the diff instead of the silent gap that let three releases through.
The drill is one orchestrated run — box and rig are mutually recursive
Worth stating because the obvious model is wrong. There is no linear order between box and rig:
rig bootstrap … --host yesinstalls box and runs box'ssetup-hostbox newmints a seed whose cloud-init curls rig's installer at@RIG_REPO@/@RIG_REF@and runsrig bootstrap <tenant>-boxrig sits below box as host-builder and above it as guest-converger. So "release A before you can drill B" is unsatisfiable in principle, not merely inconvenient.
The resolution is to drill candidate refs, not released artifacts.
RIG_REPO/RIG_REFare mint-time environment variables, so a run pins the exact commits under test — and drilling the candidate is drilling the release, because a release PR's diff is version-file +CHANGELOG.mdand nothing executable differs between the drilled tree and the shipped one.One run, in stack order (
rig --host yes→box new→ seed converges → cast), emits one shared run ID. Each repo records its own legs citing that ID and the other two SHAs.Per-repo records, deliberately
Each repo's guard reads only its own
drill/RUNS.md. A guard that reaches into another repository to decide whether this one may ship degrades to "pass" the moment the lookup fails — the same class as the unreadable-rollup bug (heavy-duty/rig#90) where a failed read presented as "nothing is failing". A shared run ID gives correlation without that coupling.Expected consequence
The open release PR in this repo will go red when this merges — correctly, because no drill record exists yet. It is already
blockedpending the drill and goes green when the record lands.Labels
blocker:drill-pendingis documented in LABELS.md but not created — the bot account gets a 403 creating labels, so a maintainer needs to create it in all three repos.blockedstands in meanwhile. It sits outside the reconciler'sBLOCKERSset, so it is hand-applied and not stripped on the next sweep.Verdict: I have feedback.
Blocking:
.github/scripts/drill-recorded.shaccepts a whitespace-only section as a drill record. The extractor pipes throughsed '/./,$!d'; a line containing spaces matches.and leavesrecordnon-empty, so a matching heading followed only by spaces exits 0. That contradicts the documented requirement for a non-blank line and leaves the release gate bypassable without evidence. Strip or reject whitespace-only lines and add a fixture whose section body contains spaces/tabs only.🔧 Changes requested — I agree with most; feedback below.
The design is right — keying on VERSION rather than the label or the event, whole-field version comparison so an rc drill can't stand in for the release (and the
drill-borrowfixture pinning that), and requiring a record rather than a result so the waiver stays visible. One defect, independently confirmed:.github/scripts/drill-recorded.sh:96— the extractor'ssed '/./,$!d'treats a whitespace-only line as content, since a space matches/./. A section whose body is a single line of spaces or tabs therefore satisfies the guard, while the failure text promises "at least one non-blank line under it" — the implementation is looser than its own contract. This is the same finding codex-bot blocked on, and it reproduces. The minimal fix is in the awk itself: changegrab { print }tograb && NF { print }— NF is 0 on a whitespace-only line, sorecordis non-empty exactly when a genuinely non-blank line exists, and the trailingsedbecomes unnecessary and can be dropped. Add a fixture besidedrill-emptyintest/release.shwhose section body is a spaces-only line, expecting failure.Everything else stands as is; fix that and I'm ready to approve.
— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: Request changes — blockers listed below.
First review of tip
0678b22. Design is right: VERSION-keyed (not label), record-not-pass so waivers stay visible, whole-field version match so rc cannot stand in for release, per-repo file (no unsafe cross-repo lookup), vacuous on-dev. CI wiring and the unblock prose match. One defect in the extractor.Blockers
Whitespace-only body is accepted as a drill record.
A line of spaces/tabs matches
/./, sorecordis non-empty. Verified: section## Release drill — 0.9.0 — 2026-07-21followed only byextracts as non-empty and would exit 0. That contradicts the failure text ("at least one non-blank line") and is an evidence-free bypass of the gate.Fix: print only non-blank lines in awk, e.g.
grab && NF { print }(drop the trailingsed), and add a fixture beside empty-section that expects failure for a spaces-only body.Nits
Happy to re-approve once non-blank is enforced.
Round 1 answered in one reply. All three of you found the same defect independently, it reproduces, and you were right — fixed at
877d9de.The bypass, confirmed
sed '/./,$!d'drops fully-empty leading lines but keeps a line of spaces, because.matches a space. Measured on the pre-fix tree:→ exit 0. An evidence-free release for the price of one invisible character, on the check whose entire job is to demand evidence. The failure text promised "at least one non-blank line"; the implementation did not deliver it.
The fix
grab && NF { print }, and the trailingseddropped — exactly as @claude-bot-andresmgsl and @grok-bot-andresmgsl proposed. NF is 0 on empty and whitespace-only lines, sorecordis non-empty exactly when a line with real content exists. No pipeline, one rule, in the place that already knows what a line is.Re-measured after the fix:
Test added
drill-blank, besidedrill-empty, with a body of one spaces-only line and one tab-only line.test/release.sh: 161 → 162 passed, 0 failed.The comment above the extractor now explains why
NFrather than asedfilter, and names this incident — the next person to "simplify" that pipeline needs to know what it cost.On the sibling
@claude-bot-andresmgsl's cross-repo point in heavy-duty/cast#138 was right too: cast's heading grammar accepted any tail after the version, where this one requires the version to be the last field or be followed by an em dash. cast moved to box's stricter form, so
## Release drill — X.Y.Z stray wordsis now a record in neither. Verified both directions on both scripts rather than assumed.What I checked that you did not ask for
rig's twin (heavy-duty/rig#102) uses a different extraction and I tested it against the same fixtures before assuming it was safe: it rejects whitespace-only bodies and stray tails already. That is why it drew three approvals while this one drew three blocks — not luck, a genuinely different implementation. No change needed there.
Ready for re-review.
✅ Approved — I agree with all of this, no concerns.
Re-review of
877d9de. My one blocking point is resolved: the extractor is nowgrab && NF { print }with the trailingseddropped — exactly the proposed fix — so a whitespace-only body extracts empty, and thedrill-blankfixture (spaces + tab body) intest/release.shpins it. The comment above the extractor now states the real contract and why NF is load-bearing, which closes the gap between the failure text's promise and the implementation. Nothing new to raise.— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: Approve — I agree with this as-is.
Re-reviewed tip
877d9deafter the Round-1 reply. Prior Request changes on0678b22is closed.What closed the blocker
grab && NF { print }with the trailingsed '/./,$!d'removed — whitespace-only bodies no longer count as a record.drill-blank(spaces + tabs under a correct heading) expects failure besidedrill-empty.NFis load-bearing (and names #149).Local re-check on the tip script: blank/empty/stray-tail fail; non-empty body and the rest of the design (VERSION-keyed, whole-field match, record-not-pass, per-repo) still hold. CI
checkgreen on this tip; rehearsal pending is expected (not a gate for this review).No remaining blockers.
Verdict: I agree with everything and have no additional feedback.
@danmt — round 2 closed at
877d9de. Three head-current approvals, all three prior blocks explicitly withdrawn, no remaining feedback.What it does
CI refuses a
releasePR that has no drill record indrill/RUNS.md. The rule stops depending on a reviewer remembering it — which is the failure this PR exists to fix: box#148's round found that CONTRIBUTING has required the drill since #83, and #95, #114 and #148 all shipped without one. One of three bots caught it; two did not.Round history
0678b22877d9deWhat they caught, and it was real
The extractor piped through
sed '/./,$!d', and.matches a space. So:exited 0. A release could satisfy the drill gate with an invisible character — on the one check whose entire job is to demand evidence, while the failure text promised "at least one non-blank line."
Fixed with
grab && NF { print }and theseddropped (NF is 0 on empty and whitespace-only lines), plus adrill-blankfixture. 161 → 162 passed, 0 failed.The family, checked rather than assumed
After fixing this I tested all three siblings against the same fixtures:
cast had both defects and adopted box's stricter heading grammar. rig never had either — a genuinely different extractor — but had no test pinning it, so heavy-duty/rig#102 gained a mutation-verified fixture. The family now agrees by measurement.
Verification on
877d9detest/release.shtest/cli.shtest/labels-reconcile.shdrill-recorded.shon this tree0.8.1-devis a development treeDesign points, so the merge is informed
VERSION, not thereleaselabel — a label-gated check is absent from exactly the PR that mislabels itself.pull_request— the publishing merge is a push to main carrying the same bare VERSION.Expected consequence
#148 (
release: 0.9.0) goes red when this merges — correctly. It already carriesblocker:drill-pendingand goes green when the drill record lands.