History

codex-bot-andresmgsl 3e3d8bbd5a Some checks failed labels / labels (pull_request) Failing after 17s Details ci / check (pull_request) Has been cancelled Details ci / rehearsal (pull_request) Has been cancelled Details fix: preserve evidence across drill failures		2026-08-18 07:04:33 +00:00
..
0.9.0.md	docs: record the 0.9.0 drill as waived, with what that leaves unproven	2026-07-21 17:42:38 +00:00
README.md	fix: preserve evidence across drill failures	2026-08-18 07:04:33 +00:00

README.md

drills/ — release evidence, one file per version

This directory holds the evidence that a release was proven on real hardware. One file per shipped version, named exactly for the version:

drills/0.9.0.md
drills/0.9.0-rc1.md

The name must match the contents of VERSION exactly. the pinned ceremony drill-recorded action refuses any tree with a bare VERSION that has no such file, or whose file is blank. A -dev tree passes with nothing to assert.

Because each version owns a file, 0.9.0 and 0.9.0-rc1 can never be confused for one another — they are simply different paths. That used to take careful whole-version field matching inside one shared file; now it is free.

This is not `drill/RUNS.md`

Two different artifacts, and the distinction is load-bearing:

	what it is
`drill/RUNS.md`	the harness's own history — every run of `drill/drill.sh`, the traps table, the lore about what broke and why. It is not release-scoped and it is not going anywhere.
`drills/<version>.md`	release evidence — the record that this version was drilled before it shipped. Release-scoped, one file, gated by CI.

Appending to drill/RUNS.md does not satisfy the release gate, and is not meant to. Keep using it for what it has always been for.

What a record should contain

What ran — which drill, how many probes, drill/drill.sh invocation.
On what host — the machine, the OS, the Incus version. "Real hardware" is the claim; name the hardware.
The pinned candidate refs — the exact BOX_REF / RIG_REF / CAST_REF under test, and the other repos' commit SHAs. A drill that does not say what it drilled proves nothing later.
The shared run ID, so this record reconciles with the sibling repos'.
The numbers — passed, failed, how long it took.
What failed, plainly.

A failed drill is still a valid record. The gate wants evidence, not success. A record saying "83/85, criterion (m) regressed, here is the issue" is a good record. So is a maintainer's written waiver explaining why this release shipped without a full drill. What the gate refuses is silence — #95, #114 and #148 all shipped unproven because a skip left no trace.

Generate the factual draft

The harness can write the facts it already knows directly into a new record:

bash drill/drill.sh --ref release/0.9.1 \
  --run-id drill-0.9.1-20260818-01 \
  --emit-record drills/0.9.1.md

--run-id wins over DRILL_RUN_ID; if neither is set, the harness generates drill-<version>-<UTC date>-01. Supply the same ID explicitly to each family drill so their records reconcile. The emitter refuses to overwrite an existing file, and its Markdown contains no terminal colour escapes.

When record emission is requested, the harness resolves the requested box and rig refs before their installers run and feeds those installers the resulting full commit SHAs. The draft keeps both the human-facing refs and the immutable SHAs. If either ref cannot be pinned, the drill refuses before mutating the host and still writes the failed preflight record.

The generated file is a draft, not finished release evidence. It records the host, invocation, elapsed time, findings, and the exact box and rig refs and SHAs the run selected. Before committing it, verify those facts and add the operator judgement that a script cannot make: hardware detail, the meaning of any failures or skips, and whether they block release.

Worked example

The version below is a placeholder that can never be a real release. Copy the shape, not the number.