forked from heavy-duty/box
f63fa952d4
Drill records move from sections inside drill/RUNS.md to one file per version: drills/<version>.md. drill/RUNS.md is untouched — it stays the harness's own run log, traps table and lore. drills/ is release evidence only, and the docs now say which is which. The old guard parsed headings: em-dash field matching, an optional ' — DATE' tail, whole-version comparison so 0.9.0-rc1 could not satisfy 0.9.0, avoiding \x escapes because CI runs mawk not gawk, and a non-blank body rule. Every one of those existed only because records shared one file, and two separate defects were found in review because of that complexity — the sed '/./,$!d' whitespace bypass, and heading-grammar drift from the sibling repos. One file per version makes almost all of it unrepresentable: 0.9.0.md and 0.9.0-rc1.md are simply different files, so whole-version matching is free rather than a trap. The guard keeps only the rule that was never about headings — a record of pure whitespace is not a record — and keeps the failure message that names the version, the path, the unblock, and the recorded-waiver escape hatch. Plain drills/, not .drills/ — a dot-directory is invisible to globs without dotglob, which is what caused #116 and #118. Also corrects the release-drill framing: the three repos' drills are INDEPENDENT and run in any order. What dissolves the box<->rig recursion is that every drill pins the same fixed candidate refs, not sequencing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
82 lines
3.4 KiB
Markdown
82 lines
3.4 KiB
Markdown
# drills/ — release evidence, one file per version
|
|
|
|
This directory holds the **evidence that a release was proven on real
|
|
hardware**. One file per shipped version, named exactly for the version:
|
|
|
|
```
|
|
drills/0.9.0.md
|
|
drills/0.9.0-rc1.md
|
|
```
|
|
|
|
The name must match the contents of `VERSION` exactly.
|
|
[.github/scripts/drill-recorded.sh](../.github/scripts/drill-recorded.sh)
|
|
refuses any tree with a bare `VERSION` that has no such file, or whose file is
|
|
blank. A `-dev` tree passes with nothing to assert.
|
|
|
|
Because each version owns a file, `0.9.0` and `0.9.0-rc1` can never be
|
|
confused for one another — they are simply different paths. That used to take
|
|
careful whole-version field matching inside one shared file; now it is free.
|
|
|
|
## This is not `drill/RUNS.md`
|
|
|
|
Two different artifacts, and the distinction is load-bearing:
|
|
|
|
| | what it is |
|
|
|---|---|
|
|
| [`drill/RUNS.md`](../drill/RUNS.md) | the **harness's own history** — every run of `drill/drill.sh`, the traps table, the lore about what broke and why. It is not release-scoped and it is not going anywhere. |
|
|
| `drills/<version>.md` | **release evidence** — the record that *this version* was drilled before it shipped. Release-scoped, one file, gated by CI. |
|
|
|
|
Appending to `drill/RUNS.md` does not satisfy the release gate, and is not
|
|
meant to. Keep using it for what it has always been for.
|
|
|
|
## What a record should contain
|
|
|
|
- **What ran** — which drill, how many probes, `drill/drill.sh` invocation.
|
|
- **On what host** — the machine, the OS, the Incus version. "Real hardware"
|
|
is the claim; name the hardware.
|
|
- **The pinned candidate refs** — the exact `BOX_REF` / `RIG_REF` /
|
|
`CAST_REF` under test, and the other repos' commit SHAs. A drill that does
|
|
not say what it drilled proves nothing later.
|
|
- **The shared run ID**, so this record reconciles with the sibling repos'.
|
|
- **The numbers** — passed, failed, how long it took.
|
|
- **What failed**, plainly.
|
|
|
|
**A failed drill is still a valid record.** The gate wants *evidence*, not
|
|
success. A record saying "83/85, criterion (m) regressed, here is the issue"
|
|
is a good record. So is a maintainer's written waiver explaining why this
|
|
release shipped without a full drill. What the gate refuses is silence — #95,
|
|
#114 and #148 all shipped unproven because a skip left no trace.
|
|
|
|
## Worked example
|
|
|
|
The version below is a **placeholder that can never be a real release**.
|
|
Copy the shape, not the number.
|
|
|
|
```markdown
|
|
# Release drill — 9.9.9
|
|
|
|
- **Run ID:** `drill-9.9.9-20260721-01` (shared with rig, cast)
|
|
- **Host:** bare Debian 13, Ryzen 7 5800X / 64 GB, Incus 6.0.2
|
|
- **Date:** 2026-07-21
|
|
- **Candidate refs:**
|
|
- box `release/9.9.9` @ `abc1234`
|
|
- rig `release/4.4.4` @ `def5678` (minted with `RIG_REF=release/4.4.4`)
|
|
- cast `release/2.2.2` @ `9abcdef`
|
|
|
|
## What ran
|
|
|
|
`bash drill/drill.sh --ref release/9.9.9` — the full end-to-end: install the
|
|
stack, mint every template cold, snapshot and restore, uninstall to zero
|
|
residue. Then `drill/multiuser.sh` for the two-user grant matrix.
|
|
|
|
## Result
|
|
|
|
**84/85 passed, 1 failed.** 41 minutes wall clock.
|
|
|
|
- Failed: `multiuser.sh` criterion (m) — the raw instance kept a stale route
|
|
after teardown. Filed as #999. Judged not release-blocking: it affects
|
|
teardown residue on a host that is about to be wiped, not the trust
|
|
boundary itself.
|
|
- The VM boundary probes (the 85-probe isolation contract) passed clean,
|
|
which is the assertion this repo's drill exists to make.
|
|
```
|