forked from heavy-duty/rig
cluade-reviewer-andresmgsl
1179d3142f
#109 shipped `rig forgejo-runner` and the drill never touched it, so the release gate could pass with the whole family broken. Its only coverage was 15 argument-handling checks in test/cli.sh. The Forgejo half cannot be a copy of the GitHub one. Measured against forgejo.heavyduty.builders (8.0.3+gitea-1.22.0), not read from the docs: GET actions/tasks carries NO `conclusion` field — `status` holds the terminal outcome — `id` is a global task id while the URL ends in run_number, and the payload lists ASSIGNED tasks only, so a queued run is invisible (measured: 200s of total_count 0 while the web UI showed the run). That makes "no new id" the only signal the runner never took the job, which is the verdict this leg exists to produce. forgejo_run_verdict() is the testable core of that, grep-and-sed rather than jq for the same reason json_field() is. Tests were written first and cover the cases that must NOT pass: a stale run with the pre-dispatch id, an assigned-but-still-running task, and a cancelled run. Refs #129 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
114 lines
6.2 KiB
Markdown
114 lines
6.2 KiB
Markdown
# The drill — running it
|
|
|
|
`drill/drill.sh` is the instrument; `drills/` is the record it feeds
|
|
(see [drills/README.md](../drills/README.md) for what a record means and
|
|
how the three repos' drills relate). rig's drill asserts **convergence**:
|
|
a machine reaches its role, idempotently. This file is the procedure —
|
|
written down so a run is repeatable, not reconstructed from memory each
|
|
release (#105, and #107's debt).
|
|
|
|
## What you need
|
|
|
|
- **A throwaway Debian 13 machine** you can format, reached as root. The
|
|
drill hardens its sshd, renames it, joins it to a tailnet, and installs
|
|
box/Incus, Coolify and Actions runners on it. It is not coming back.
|
|
The machine is its own reset — there is no teardown script and no need
|
|
for one.
|
|
- **The pinned candidate refs, both of them.** `--rig-ref` and
|
|
`--box-ref` are required; the harness refuses to run without them and
|
|
refuses to continue if what installed disagrees with what was asked
|
|
(`INSTALLED_FROM`, both trees). Since heavy-duty/rig#103 landed, both
|
|
installers have sane defaults when unpinned — box installs the
|
|
`BOX_RELEASE` pin (currently `0.9.0`), rig's `install.sh` resolves the
|
|
latest release — and a sane default is exactly why the drill will not
|
|
let a ref go unstated: an unpinned run silently drills a shipping pair
|
|
that is not the candidate, and the record it leaves looks clean.
|
|
- **A single-use, tagged tailscale pre-auth key** in `TS_AUTHKEY`
|
|
(`tag:local` for the default `staging-server` role — bootstrap refuses
|
|
`tag:server` outside the control-plane shapes).
|
|
- **A users file** (`--users`) naming at least one operator — leg 1
|
|
asserts the accounts and keys actually converged.
|
|
- **For leg 3** (runner lifecycle): a fork to register against
|
|
(`--runner-repo you/rig`) carrying a `workflow_dispatch` workflow —
|
|
default name `drill.yml` — whose job has `runs-on: [self-hosted, drill]`
|
|
and does something trivial (`echo drilled`). Tokens come from an
|
|
authenticated `gh`, or from `RUNNER_TOKEN` / `RUNNER_REMOVE_TOKEN`.
|
|
Without a fork the leg **skips, loudly, into the record**.
|
|
- **For leg 3's Forgejo half** (#129): `--forgejo-instance <url>` and
|
|
`--forgejo-runner-repo <owner>/<repo>`, where that repo carries the same
|
|
`workflow_dispatch` workflow — but with `runs-on: drill`, because a
|
|
Forgejo runner matches the bare label it registered with. Tokens come from
|
|
`FORGEJO_RUNNER_TOKEN` (a registration token) or `FORGEJO_API_TOKEN`, which
|
|
mints one and is also what dispatches the job. `--forgejo-ref` names the
|
|
branch to dispatch (default `main`): Forgejo's dispatch endpoint requires a
|
|
ref in the body, where GitHub's defaults to the repo's default branch.
|
|
Without an instance and a repo this half **skips, loudly and separately**.
|
|
Note there is no removal token — Forgejo has no deregistration endpoint, so
|
|
the leg removes locally and the record tells you to delete the stale runner
|
|
row by hand.
|
|
- **For leg 4** (coolify): a version pin, `--coolify-version 4.1.2`.
|
|
No pin, no leg — rig's own `coolify install` refuses to default a
|
|
version and so does its drill. The skip is recorded.
|
|
- **A run ID** (`--run-id`) when this drill shares a substrate with
|
|
box's or cast's — the shared ID is what lets the per-repo records be
|
|
joined afterwards. Defaults to `drill-<date>`.
|
|
|
|
## Running it
|
|
|
|
From a checkout of this repo on the throwaway machine (the record lands
|
|
in the checkout's `drills/`):
|
|
|
|
```sh
|
|
TS_AUTHKEY=tskey-... bash drill/drill.sh \
|
|
--rig-ref release/0.4.0 --box-ref 0.9.0 \
|
|
--users ./drill-users --run-id drill-2026-07-24-a \
|
|
--coolify-version 4.1.2 --runner-repo you/rig --yes
|
|
```
|
|
|
|
`--box-ref` is a tag on purpose: since #103 the box that ships is the
|
|
`BOX_RELEASE` tag, so a `release/…` branch is the wrong thing to pin for
|
|
box — while a release branch stays exactly right for rig's own candidate.
|
|
|
|
It runs unattended from there. Legs execute as 1, 4, 2, 3 — Coolify's
|
|
installer is what puts Docker on the box and the db leg needs a daemon —
|
|
and the record lists them as they ran. A failing check never aborts the
|
|
run (`set -u`, no `-e`: a failing check is data), and the summary counts
|
|
passes, failures and skips separately.
|
|
|
|
## What it asserts
|
|
|
|
1. **Convergence, and idempotence.** `rig bootstrap <role> --users …`
|
|
reaches the declared role, asserted on *effective* state — the marker,
|
|
`sshd -T`, the granted tailnet tag, the operators' accounts and keys.
|
|
Then bootstrap runs **again**, and the state captured before and after
|
|
the re-run must diff **empty**. The diff is mechanical; "watched it
|
|
not obviously break" is exactly what this leg exists to replace.
|
|
Riding along, the `--host yes` assertions: the **pinned** box
|
|
installed (`INSTALLED_FROM` matches `--box-ref`, fatal if not),
|
|
`box doctor` passes. It stops there and says so in the output — the
|
|
isolation boundary is **box's** drill's assertion, never rig's.
|
|
2. **db** — `test/db-integration.sh` from the *installed* tree: a real
|
|
dump/restore round-trip. Its clean-skip contract (no Docker → loud
|
|
skip, exit 0) survives into the record as a SKIP, never a pass.
|
|
3. **Runner lifecycle** — register against the fork, dispatch the drill
|
|
workflow and watch the runner take it, deregister, and assert the
|
|
box's registration is actually gone. Runs **once per forge**: `rig runner`
|
|
against GitHub, then `rig forgejo-runner` against a Forgejo instance
|
|
(#129). Both families ship, so a release that evidences only one
|
|
evidences half of what it ships; each half skips separately, so a record
|
|
can honestly show one forge drilled and the other not.
|
|
4. **Coolify** — installed at the pin, `AUTOUPDATE=false` landed in the
|
|
effective `.env`, container running.
|
|
|
|
## The record
|
|
|
|
The run always ends by writing `drills/<version>.md` (the version is the
|
|
installed tree's own `VERSION`) — on failures too: **a failed drill is a
|
|
valid record**; the gate wants evidence, not success. Skipped legs are
|
|
named as not-run so the record can never read as a clean sweep. Commit
|
|
the file on the release branch; the `drill-recorded` guard reads that
|
|
file and nothing else.
|
|
|
|
The instrument's own honesty — the refusals, the skip accounting, the
|
|
capture-and-diff, the emitter — is `test/drill.sh`'s job, and CI runs it
|
|
on every PR. The live four-leg run is a release's job, once per cycle.
|