cluade-reviewer-andresmgsl
1179d3142f
#109 shipped `rig forgejo-runner` and the drill never touched it, so the release gate could pass with the whole family broken. Its only coverage was 15 argument-handling checks in test/cli.sh. The Forgejo half cannot be a copy of the GitHub one. Measured against forgejo.heavyduty.builders (8.0.3+gitea-1.22.0), not read from the docs: GET actions/tasks carries NO `conclusion` field — `status` holds the terminal outcome — `id` is a global task id while the URL ends in run_number, and the payload lists ASSIGNED tasks only, so a queued run is invisible (measured: 200s of total_count 0 while the web UI showed the run). That makes "no new id" the only signal the runner never took the job, which is the verdict this leg exists to produce. forgejo_run_verdict() is the testable core of that, grep-and-sed rather than jq for the same reason json_field() is. Tests were written first and cover the cases that must NOT pass: a stale run with the pre-dispatch id, an assigned-but-still-running task, and a cancelled run. Refs #129 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
124 lines
6.3 KiB
Markdown
124 lines
6.3 KiB
Markdown
# Drill records
|
|
|
|
Per-release evidence for rig's real-hardware drill. **One file per version**,
|
|
named for the version exactly as `VERSION` carries it:
|
|
|
|
drills/<version>.md
|
|
|
|
So `0.3.0` is recorded in `drills/0.3.0.md`, and `0.3.0-rc1` in
|
|
`drills/0.3.0-rc1.md`. They are different files, which is the whole point: the
|
|
filesystem does the whole-version comparison that an earlier single-file
|
|
version of this had to do with an awk extractor and a heading grammar. A
|
|
record for the candidate cannot be mistaken for evidence for the final.
|
|
|
|
The directory is `drills/`, not `.drills/` — a dot-directory is invisible to
|
|
any glob without `dotglob`, which is how #70 here and box#116 / box#118 all
|
|
happened.
|
|
|
|
**This directory is the record, not the instrument.** The instrument is
|
|
[`drill/drill.sh`](../drill/README.md) (#105): it runs the legs, asserts the
|
|
pinned refs actually landed, decides idempotence by a mechanical state diff,
|
|
and emits the record file this directory holds. rig does not reach into
|
|
another repo's harness to decide whether rig may ship: a cross-repo lookup
|
|
that fails silently degrades to "pass", which is the UNREADABLE-vs-NONE shape
|
|
#90 fixed. The gate reads a file in this repo, and nothing else.
|
|
|
|
## What the gate requires
|
|
|
|
The `drill-recorded` guard (heavy-duty/ceremony's action, pinned in
|
|
`ci.yml`) runs on every PR. On a `-dev` tree it
|
|
asserts nothing — a development tree has no release to evidence. On a bare
|
|
`VERSION` — a release ceremony tree — it requires `drills/<version>.md` to
|
|
exist and to hold at least one non-whitespace character. An empty file, or one
|
|
of only spaces and tabs, is not a record (box#149 and cast#138 both shipped an
|
|
extractor where a single tab satisfied the gate).
|
|
|
|
The guard requires a **record**, not a passing result. A maintainer waiver is
|
|
a legitimate outcome of a release — but it is written in that version's file,
|
|
so that skipping the drill is a deliberate, reviewable commit rather than a
|
|
silence. **A failed drill is still a valid record**: the gate wants evidence,
|
|
not success.
|
|
|
|
## The drill
|
|
|
|
rig's legs (#105; `drill/drill.sh` runs them):
|
|
|
|
- `rig bootstrap <role>` converges the machine to its role — then runs
|
|
**again**, and the captured state must diff **empty** (idempotence,
|
|
decided mechanically). On a host=yes role this is also what installs the
|
|
pinned box and asserts its host stack stands.
|
|
- `bash test/db-integration.sh` against a real Postgres on the machine
|
|
- the GitHub runner lifecycle — register, take a job, deregister — against a
|
|
fork
|
|
- a coolify install, pinned, `AUTOUPDATE=false`
|
|
|
|
box and rig are **mutually recursive**: `rig bootstrap --host yes` installs box
|
|
and runs box's `setup-host`, while box's guests converge back through rig's
|
|
installer. Within a single drill you naturally bring the substrate up before
|
|
probing it — a host before a guest — but that is how you run a drill, not an
|
|
ordering rule between repos.
|
|
|
|
**The three repos' drills are independent.** Run them in any order, on any
|
|
schedule, in separate sittings. What makes that safe is that every drill
|
|
**pins the same fixed set of candidate refs**: rig's drill runs `--host yes`
|
|
with `BOX_REF=release/<box-version>`, so it exercises the box that will
|
|
actually ship; box's drill mints with `RIG_REF=release/<rig-version>`, so it
|
|
exercises the rig that will actually ship. Both measure the same pair. The
|
|
record also cites the **rig-templates SHA** the converge read (#110) — the
|
|
candidate tree's `RIG_TEMPLATES_PIN` unless the drill was pointed elsewhere
|
|
via `RIG_TEMPLATES_REF` — so the mechanism+registry pair a release freezes
|
|
is the pair the drill proved.
|
|
|
|
That — not sequencing — is what dissolves the box↔rig recursion. The refs are
|
|
static identifiers that exist as soon as the release branches do, long before
|
|
any drill runs, so a cycle at runtime becomes two independent tests against
|
|
one fixed pair. It also means **candidate refs, not released artifacts**:
|
|
`RIG_REPO`/`RIG_REF` are mint-time environment variables, so no repo has to be
|
|
released before another can be drilled. Drilling the candidate *is* drilling
|
|
the release, because a release PR's diff is `VERSION` + `CHANGELOG.md` and
|
|
nothing executable differs.
|
|
|
|
Each repo drills in a **different way** and asserts a different thing: rig
|
|
asserts **convergence** (a machine reaches its role, idempotently), box asserts
|
|
the **isolation contract** (the VM trust boundary), cast asserts **promotion**
|
|
(A→B reproduces, the diff is idempotent). Three different exercises sharing a
|
|
substrate, not three phases of one script — which is exactly why the records
|
|
are per-repo.
|
|
|
|
Drills that share a substrate share **one run ID**; each repo records its own
|
|
legs in its own file, citing that run ID and the other repos' commit SHAs, so
|
|
the records can be joined after the fact. If a defect shows up only in the
|
|
combination: patch, re-drill, re-record. The three releases converge on a set
|
|
that holds together; they are not required to be right in one pass. Releases
|
|
do **not** have to be published in a fixed order.
|
|
|
|
## What a record should contain
|
|
|
|
What ran, on what host, the pinned candidate refs, the numbers, and what
|
|
failed. Below is the *shape*, in a file named `drills/9.9.9.md` — a version
|
|
that can never collide with a real release. **No drill has been recorded here
|
|
yet**; this log starts empty rather than reconstructing runs from memory, since
|
|
an invented number is worse than no number.
|
|
|
|
```markdown
|
|
# Release drill — 9.9.9 — 2026-01-01
|
|
|
|
Run ID: drill-2026-01-01-a. Host: bare Debian 13 cloud image, 4 vCPU / 8 GB.
|
|
Candidate refs: box@1a2b3c4 (BOX_REF=release/0.4.0), rig@5d6e7f8, cast@9a0b1c2.
|
|
|
|
| Leg | Result |
|
|
| --- | --- |
|
|
| convergence — bootstrap staging-server reaches its role | PASS (312s) |
|
|
| re-converge (idempotence) | clean, no changes |
|
|
| --host yes: pinned box installed, host stack up | PASS — box doctor clean |
|
|
| `test/db-integration.sh` | PASS — 14 passed, 0 failed |
|
|
| runner lifecycle against a fork | PASS — registered, took a job, deregistered clean |
|
|
| forgejo runner lifecycle (you/drill-probe) | PASS — registered, took a job, removed (stale row needs deleting by hand) |
|
|
| coolify install (4.1.2) | PASS (6 min) |
|
|
|
|
Failed: `rig users apply` left one revoked key in `authorized_keys`
|
|
(filed #NNN). Everything else clean.
|
|
```
|
|
|
|
State what failed. A record with no failures listed reads as "nothing broke",
|
|
so if a leg was not run, say that instead of omitting it.
|