From 1179d3142fe6acfdcf1eaf381a6255fc1a82a72f Mon Sep 17 00:00:00 2001 From: cluade-reviewer-andresmgsl Date: Thu, 30 Jul 2026 23:27:36 +0000 Subject: [PATCH 1/4] feat(drill): a forgejo-runner lifecycle leg beside the GitHub one MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #109 shipped `rig forgejo-runner` and the drill never touched it, so the release gate could pass with the whole family broken. Its only coverage was 15 argument-handling checks in test/cli.sh. The Forgejo half cannot be a copy of the GitHub one. Measured against forgejo.heavyduty.builders (8.0.3+gitea-1.22.0), not read from the docs: GET actions/tasks carries NO `conclusion` field — `status` holds the terminal outcome — `id` is a global task id while the URL ends in run_number, and the payload lists ASSIGNED tasks only, so a queued run is invisible (measured: 200s of total_count 0 while the web UI showed the run). That makes "no new id" the only signal the runner never took the job, which is the verdict this leg exists to produce. forgejo_run_verdict() is the testable core of that, grep-and-sed rather than jq for the same reason json_field() is. Tests were written first and cover the cases that must NOT pass: a stale run with the pre-dispatch id, an assigned-but-still-running task, and a cancelled run. Refs #129 Co-Authored-By: Claude Opus 5 (1M context) --- changelog.d/129.md | 3 + drill/README.md | 20 ++++- drill/drill.sh | 182 ++++++++++++++++++++++++++++++++++++++++++++- drills/README.md | 1 + test/drill.sh | 60 ++++++++++++++- 5 files changed, 259 insertions(+), 7 deletions(-) create mode 100644 changelog.d/129.md diff --git a/changelog.d/129.md b/changelog.d/129.md new file mode 100644 index 0000000..c6ff707 --- /dev/null +++ b/changelog.d/129.md @@ -0,0 +1,3 @@ +### Added + +- The release drill exercises `rig forgejo-runner` beside `rig runner`, so both shipped runner families carry evidence (#129) diff --git a/drill/README.md b/drill/README.md index efbbc62..fd54cf7 100644 --- a/drill/README.md +++ b/drill/README.md @@ -11,7 +11,7 @@ release (#105, and #107's debt). - **A throwaway Debian 13 machine** you can format, reached as root. The drill hardens its sshd, renames it, joins it to a tailnet, and installs - box/Incus, Coolify and a GitHub runner on it. It is not coming back. + box/Incus, Coolify and Actions runners on it. It is not coming back. The machine is its own reset — there is no teardown script and no need for one. - **The pinned candidate refs, both of them.** `--rig-ref` and @@ -34,6 +34,18 @@ release (#105, and #107's debt). and does something trivial (`echo drilled`). Tokens come from an authenticated `gh`, or from `RUNNER_TOKEN` / `RUNNER_REMOVE_TOKEN`. Without a fork the leg **skips, loudly, into the record**. +- **For leg 3's Forgejo half** (#129): `--forgejo-instance ` and + `--forgejo-runner-repo /`, where that repo carries the same + `workflow_dispatch` workflow — but with `runs-on: drill`, because a + Forgejo runner matches the bare label it registered with. Tokens come from + `FORGEJO_RUNNER_TOKEN` (a registration token) or `FORGEJO_API_TOKEN`, which + mints one and is also what dispatches the job. `--forgejo-ref` names the + branch to dispatch (default `main`): Forgejo's dispatch endpoint requires a + ref in the body, where GitHub's defaults to the repo's default branch. + Without an instance and a repo this half **skips, loudly and separately**. + Note there is no removal token — Forgejo has no deregistration endpoint, so + the leg removes locally and the record tells you to delete the stale runner + row by hand. - **For leg 4** (coolify): a version pin, `--coolify-version 4.1.2`. No pin, no leg — rig's own `coolify install` refuses to default a version and so does its drill. The skip is recorded. @@ -80,7 +92,11 @@ passes, failures and skips separately. skip, exit 0) survives into the record as a SKIP, never a pass. 3. **Runner lifecycle** — register against the fork, dispatch the drill workflow and watch the runner take it, deregister, and assert the - box's registration is actually gone. + box's registration is actually gone. Runs **once per forge**: `rig runner` + against GitHub, then `rig forgejo-runner` against a Forgejo instance + (#129). Both families ship, so a release that evidences only one + evidences half of what it ships; each half skips separately, so a record + can honestly show one forge drilled and the other not. 4. **Coolify** — installed at the pin, `AUTOUPDATE=false` landed in the effective `.env`, container running. diff --git a/drill/drill.sh b/drill/drill.sh index a6f161d..2e7412b 100644 --- a/drill/drill.sh +++ b/drill/drill.sh @@ -4,13 +4,16 @@ # ⚠ DESTRUCTIVE, AND MEANT TO BE. Run it on a THROWAWAY Debian machine you # can format. It wipes any installed rig and reinstalls from the pinned # ref, hardens sshd, sets the hostname, joins the tailnet, installs box -# and its Incus stack, installs Coolify and a GitHub Actions runner. +# and its Incus stack, installs Coolify and Actions runners (GitHub, and +# Forgejo when --forgejo-instance is given). # Never run it on a machine you care about. # # TS_AUTHKEY=tskey-... bash drill/drill.sh \ # --rig-ref release/0.4.0 --box-ref 0.9.0 \ # --users ./drill-users --run-id drill-2026-07-24-a \ -# --coolify-version 4.1.2 --runner-repo you/rig --yes +# --coolify-version 4.1.2 --runner-repo you/rig \ +# --forgejo-instance https://forgejo.example.com \ +# --forgejo-runner-repo you/drill-probe --yes # (--box-ref is a tag: since #103 the box that ships is the BOX_RELEASE tag.) # rig's drill asserts CONVERGENCE — a machine reaches its role, idempotently. # The legs (drills/README.md, issue #105): @@ -22,6 +25,10 @@ # the isolation boundary is box's drill's assertion, not this one's). # 2. db — the real dump/restore round-trip, test/db-integration.sh. # 3. runner lifecycle — register, take a job, deregister, against a fork. +# Runs once per forge: `rig runner` against GitHub (--runner-repo), and +# `rig forgejo-runner` against a Forgejo instance (--forgejo-instance + +# --forgejo-runner-repo). Both forges ship, so both need evidence; each +# skips loudly and separately when its inputs are absent (#129). # 4. coolify install — at a pinned version, AUTOUPDATE=false. # # Execution order is 1, 4, 2, 3 — coolify's installer is what puts Docker on @@ -67,6 +74,11 @@ RECORD="${DRILL_RECORD:-}" COOLIFY_VERSION="${DRILL_COOLIFY_VERSION:-}" RUNNER_REPO="${DRILL_RUNNER_REPO:-}" RUNNER_WORKFLOW="${DRILL_RUNNER_WORKFLOW:-drill.yml}" +FJ_INSTANCE="${DRILL_FORGEJO_INSTANCE:-}" +FJ_RUNNER_REPO="${DRILL_FORGEJO_RUNNER_REPO:-}" +# The branch the dispatch names. Forgejo's dispatch endpoint requires a ref in +# the body — unlike GitHub's, which defaults to the repo's default branch. +FJ_REF="${DRILL_FORGEJO_REF:-main}" YES=0 while [ $# -gt 0 ]; do @@ -83,7 +95,10 @@ while [ $# -gt 0 ]; do --coolify-version) COOLIFY_VERSION="$2"; shift 2 ;; --runner-repo) RUNNER_REPO="$2"; shift 2 ;; --runner-workflow) RUNNER_WORKFLOW="$2"; shift 2 ;; - -h|--help) sed -n '2,33p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + --forgejo-instance) FJ_INSTANCE="$2"; shift 2 ;; + --forgejo-runner-repo) FJ_RUNNER_REPO="$2"; shift 2 ;; + --forgejo-ref) FJ_REF="$2"; shift 2 ;; + -h|--help) sed -n '2,40p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "drill: unknown option: $1 (see --help)" >&2; exit 2 ;; esac done @@ -108,6 +123,50 @@ phase(){ printf '\n\033[1m══ %s\033[0m\n' "$*"; } LEG_NAMES=(); LEG_RESULTS=() leg() { LEG_NAMES+=("$1"); LEG_RESULTS+=("$2"); } +# forgejo_run_verdict — the verdict for OUR dispatch: +# success | failed | pending. Reads GET /repos/{o}/{r}/actions/tasks, whose +# shape is NOT GitHub's and was measured against forgejo.heavyduty.builders +# (8.0.3+gitea-1.22.0) on 2026-07-30 rather than read from the docs: +# +# * There is no `conclusion` field. `status` carries the terminal outcome +# directly ("success"), where GitHub splits status:completed + +# conclusion:success. Reading `conclusion` here gets an empty string on +# every run, which would grade a green job as failed. +# * `id` is a GLOBAL task id; the run's own URL ends in `run_number`. The +# pre-dispatch guard therefore compares `id`, exactly as the GitHub leg +# compares databaseId — an old run must never be read as this one. +# * The payload lists ASSIGNED tasks only. A run sitting queued is simply +# absent (measured: 200s of total_count 0 while the web UI showed the run +# as "job is not started"). So "no new id" is the ONLY signal that the +# runner never took the job — which is the verdict this leg exists for. +# * A task appears when ASSIGNED, so it can be seen mid-flight. A +# non-terminal status is pending, not failed; grading a running job as a +# failure would make the leg flaky inside its own watch window. +# +# grep-and-sed, not jq: a throwaway drill machine has neither jq nor an +# authenticated forge CLI, the same constraint json_field() carries in +# commands/lib/runner-config.sh. `id` is a bare number, which json_field's +# quoted-value shape cannot read, so this reads both forms itself. +forgejo_run_verdict() { + # `run_status`, not `status`: the latter is read-only in zsh, and while this + # file is bash, the function is awk-extracted and sourced by other shells in + # test harnesses — where the assignment fails silently and grades every run + # as pending. + local pre="$1" file="$2" id run_status + id="$(grep -o '"id"[[:space:]]*:[[:space:]]*[0-9][0-9]*' "$file" 2>/dev/null \ + | head -n1 | sed 's/.*:[[:space:]]*//')" + [ -n "$id" ] || { echo pending; return 0; } + [ "$id" != "$pre" ] || { echo pending; return 0; } + run_status="$(grep -o '"status"[[:space:]]*:[[:space:]]*"[^"]*"' "$file" 2>/dev/null \ + | head -n1 | sed 's/.*:[[:space:]]*"//; s/"$//')" + case "$run_status" in + success) echo success ;; + failure | cancelled | skipped | timedout) echo failed ;; + *) echo pending ;; + esac +} + + # run_logged — run a long command with its narration in a file # and a dot every 5s on the terminal: a silent multi-minute apt/install run is # indistinguishable from a wedge, and that ambiguity has cost box whole @@ -690,6 +749,123 @@ else fi fi +# ============================================================================= +phase "Leg 3 — forgejo runner lifecycle against an instance" +# ============================================================================= +# The same leg as above, for the other forge. Both forges ship a runner family +# (#109 added `rig forgejo-runner` beside `rig runner`), so a release that +# evidences only GitHub evidences half of what it ships (#129). +# +# Three things differ from the GitHub leg, all measured against +# forgejo.heavyduty.builders (8.0.3+gitea-1.22.0) on 2026-07-30, not read: +# +# * Scope is the TOKEN's, never a flag — `rig forgejo-runner install` refuses +# --repo on purpose (commands/forgejo-runner-install.sh:160). The repo here +# is only where the registration token is minted from, and where the +# dispatched workflow lives. +# * The mint path is /repos///actions/runners/registration-token. The +# instance's own swagger documents /repos///runners/registration-token +# — WITHOUT /actions/ — and that path 404s. Do not "fix" this to match the +# published API reference. +# * There is no deregistration endpoint, so there is no removal token and no +# remote deregistration: `rig forgejo-runner remove` is local-only by +# design (commands/forgejo-runner-remove.sh:7-11) and the runner row +# survives in the UI until a human deletes it. The record says so rather +# than implying a clean remote teardown the way the GitHub leg can. +# +# Tokens: FORGEJO_RUNNER_TOKEN (a registration token) is used directly; else +# FORGEJO_API_TOKEN mints one from FJ_RUNNER_REPO. Without an instance, a repo, +# or a token source the leg SKIPS loudly and the record says it did not run. +if [ -z "$FJ_INSTANCE" ] || [ -z "$FJ_RUNNER_REPO" ]; then + skip "forgejo runner lifecycle: no --forgejo-instance/--forgejo-runner-repo given — the leg did not run" + leg "forgejo runner lifecycle" "SKIPPED — no instance/repo provided" +else + fj_reg="${FORGEJO_RUNNER_TOKEN:-}" + if [ -z "$fj_reg" ] && [ -n "${FORGEJO_API_TOKEN:-}" ]; then + fj_reg="$(curl -fsSL -H "Authorization: token ${FORGEJO_API_TOKEN}" \ + "${FJ_INSTANCE%/}/api/v1/repos/${FJ_RUNNER_REPO}/actions/runners/registration-token" 2>/dev/null \ + | grep -o '"token"[[:space:]]*:[[:space:]]*"[^"]*"' | head -n1 \ + | sed 's/.*:[[:space:]]*"//; s/"$//')" + fi + if [ -z "$fj_reg" ]; then + skip "forgejo runner lifecycle: no FORGEJO_RUNNER_TOKEN and no FORGEJO_API_TOKEN to mint one — the leg did not run" + leg "forgejo runner lifecycle ($FJ_RUNNER_REPO)" "SKIPPED — no registration token source" + else + FJ_NAME="drill-$(hostname)-$$" + # The label MUST carry a docker:// image: forgejo-runner runs jobs in + # containers, and a bare label leaves runs-on matched but unrunnable. + if FORGEJO_RUNNER_TOKEN="$fj_reg" run_logged /tmp/drill-forgejo-runner-install.log \ + rig forgejo-runner install --instance "$FJ_INSTANCE" --name "$FJ_NAME" \ + --labels 'drill:docker://node:22-bookworm'; then + ok "rig forgejo-runner install --instance $FJ_INSTANCE exited 0 (registered as $FJ_NAME)" + else + no "forgejo-runner install FAILED — tail: $(tail -3 /tmp/drill-forgejo-runner-install.log | tr '\n' ' ')" + fi + rig forgejo-runner status 2>/dev/null | grep -qF "${FJ_INSTANCE%/}" \ + && ok "forgejo-runner status names the instance: $FJ_INSTANCE" \ + || no "forgejo-runner status does not name ${FJ_INSTANCE}" + + fj_took=none + if [ -n "${FORGEJO_API_TOKEN:-}" ]; then + fj_api="${FJ_INSTANCE%/}/api/v1/repos/${FJ_RUNNER_REPO}" + # Read the newest ASSIGNED task id BEFORE dispatching, same guard as the + # GitHub leg: an already-completed run must never be read as ours. + fj_pre_body="$(curl -fsSL -H "Authorization: token ${FORGEJO_API_TOKEN}" \ + "$fj_api/actions/tasks" 2>/dev/null || echo '{}')" + printf '%s' "$fj_pre_body" > /tmp/drill-forgejo-pre.json + fj_pre="$(grep -o '"id"[[:space:]]*:[[:space:]]*[0-9][0-9]*' /tmp/drill-forgejo-pre.json \ + 2>/dev/null | head -n1 | sed 's/.*:[[:space:]]*//')" + if curl -fsSL -o /dev/null -X POST -H "Authorization: token ${FORGEJO_API_TOKEN}" \ + -H "Content-Type: application/json" -d "{\"ref\":\"${FJ_REF}\"}" \ + "$fj_api/actions/workflows/${RUNNER_WORKFLOW}/dispatches" 2>/dev/null; then + inf "dispatched $RUNNER_WORKFLOW on $FJ_RUNNER_REPO — waiting for the runner to take it (≤5 min)…" + fj_took=timeout + for _i in $(seq 1 30); do + sleep 10 + curl -fsSL -H "Authorization: token ${FORGEJO_API_TOKEN}" \ + "$fj_api/actions/tasks" -o /tmp/drill-forgejo-tasks.json 2>/dev/null || continue + case "$(forgejo_run_verdict "${fj_pre:-}" /tmp/drill-forgejo-tasks.json)" in + success) fj_took=success; break ;; + failed) fj_took=failed; break ;; + *) : ;; # pending — queued, or assigned and still running + esac + done + else + fj_took=nodispatch + fi + case "$fj_took" in + success) ok "the forgejo runner took a job and it succeeded ($RUNNER_WORKFLOW)" ;; + failed) no "the dispatched job completed UNSUCCESSFULLY — the runner ran it, the workflow failed; read the run on $FJ_RUNNER_REPO" ;; + # A queued task is INVISIBLE in this API until a runner claims it, so a + # timeout means "nothing ever took it". Two causes, and the second one + # is not rig's: the runs-on label may not match, or the daemon's poller + # can go quiet — a restarted daemon claims a minutes-old backlog in + # about a second. Check 'systemctl restart forgejo-runner' before + # reading this as a rig defect. + timeout) no "the dispatched job was never taken within 5 min — check the workflow's runs-on is 'drill', then restart forgejo-runner and re-read (a quiet poller looks exactly like this)" ;; + nodispatch) no "could not dispatch $RUNNER_WORKFLOW on $FJ_RUNNER_REPO — does it carry that workflow, with workflow_dispatch, on its default branch?" ;; + esac + else + skip "took a job: not attempted — no FORGEJO_API_TOKEN to dispatch $RUNNER_WORKFLOW with" + fi + + # No removal token exists on this forge — remove is local by design. + rig forgejo-runner remove >/dev/null 2>&1 \ + && note "forgejo-runner removed locally — Forgejo has no deregistration endpoint, so DELETE the stale '$FJ_NAME' row under $FJ_RUNNER_REPO > Settings > Actions > Runners by hand" \ + || no "forgejo-runner remove FAILED" + rig forgejo-runner status >/dev/null 2>&1 \ + && no "forgejo-runner status still answers after remove — the removal did not take" \ + || ok "forgejo-runner status confirms: nothing registered" + + leg "forgejo runner lifecycle ($FJ_RUNNER_REPO)" \ + "$(case "$fj_took" in + success) echo "PASS — registered, took a job, removed (stale row needs deleting by hand)" ;; + none) echo "PARTIAL — registered and removed; took a job: not attempted (no FORGEJO_API_TOKEN)" ;; + *) echo "FAIL — see Failed below" ;; + esac)" + fi +fi + # ============================================================================= phase "Summary" # ============================================================================= diff --git a/drills/README.md b/drills/README.md index 5b2927a..581e0db 100644 --- a/drills/README.md +++ b/drills/README.md @@ -113,6 +113,7 @@ Candidate refs: box@1a2b3c4 (BOX_REF=release/0.4.0), rig@5d6e7f8, cast@9a0b1c2. | --host yes: pinned box installed, host stack up | PASS — box doctor clean | | `test/db-integration.sh` | PASS — 14 passed, 0 failed | | runner lifecycle against a fork | PASS — registered, took a job, deregistered clean | +| forgejo runner lifecycle (you/drill-probe) | PASS — registered, took a job, removed (stale row needs deleting by hand) | | coolify install (4.1.2) | PASS (6 min) | Failed: `rig users apply` left one revoked key in `authorized_keys` diff --git a/test/drill.sh b/test/drill.sh index 8c01a08..04b8f38 100644 --- a/test/drill.sh +++ b/test/drill.sh @@ -50,10 +50,10 @@ trap 'rm -rf "$WORK"' EXIT # --- the functions under test, extracted ------------------------------------- FNS="$WORK/drill-fns.sh" -for fn in tree_of assert_installed_from classify_leg capture_state emit_record; do +for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict; do awk "/^${fn}\(\) \{/,/^\}/" "$ROOT/drill/drill.sh" >> "$FNS" done -for fn in tree_of assert_installed_from classify_leg capture_state emit_record; do +for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict; do check "extraction guards the awk: ${fn}() landed" 0 "${fn}() {" grep -F "${fn}() {" "$FNS" done # shellcheck source=/dev/null @@ -201,6 +201,54 @@ check "an all-green record says every leg ran and passed" 0 "Every leg ran and e # ============================================================================= # the shipped script itself # ============================================================================= +# ============================================================================= +# forgejo_run_verdict — did OUR dispatched run land, and how (#129) +# ============================================================================= +# The Forgejo half of the runner leg cannot reuse the GitHub reader. Measured +# against forgejo.heavyduty.builders (8.0.3+gitea-1.22.0) on 2026-07-30, a +# completed run in GET /repos/{o}/{r}/actions/tasks carries NO `conclusion` +# field at all — `status` holds the terminal outcome directly, where GitHub +# splits status:completed + conclusion:success. And `id` is a global task id +# (25) while the run's own URL ends in run_number (1), so the pre-dispatch +# guard has to compare `id`. +# +# The payload is also an ASSIGNED-task view: it reads total_count 0 for as +# long as a run sits queued (measured: 200s), so "no new id" is the ONLY +# signal that the runner never took the job. That is the verdict this leg +# exists to produce, which is why it gets its own function and its own tests. +FJ="$WORK/fj"; mkdir -p "$FJ" +printf '%s' '{"workflow_runs":[],"total_count":0}' > "$FJ/empty.json" +printf '%s' '{"workflow_runs":[{"id":25,"status":"success","run_number":1,"url":"https://f/o/r/actions/runs/1"}],"total_count":1}' > "$FJ/new-success.json" +printf '%s' '{"workflow_runs":[{"id":25,"status":"failure","run_number":1,"url":"https://f/o/r/actions/runs/1"}],"total_count":1}' > "$FJ/new-failure.json" +printf '%s' '{"workflow_runs":[{"id":25,"status":"cancelled","run_number":1,"url":"https://f/o/r/actions/runs/1"}],"total_count":1}' > "$FJ/new-cancelled.json" +printf '%s' '{"workflow_runs":[{"id":24,"status":"success","run_number":1,"url":"https://f/o/r/actions/runs/1"}],"total_count":1}' > "$FJ/stale-only.json" + +check "verdict: an empty task list is PENDING, never a pass" 0 "pending" \ + forgejo_run_verdict "" "$FJ/empty.json" +check "verdict: a queued run the runner never took stays PENDING" 0 "pending" \ + forgejo_run_verdict "24" "$FJ/stale-only.json" +check "verdict: OUR new run, status success, is SUCCESS" 0 "success" \ + forgejo_run_verdict "24" "$FJ/new-success.json" +check "verdict: status carries the outcome — failure is FAILED, not success" 0 "failed" \ + forgejo_run_verdict "24" "$FJ/new-failure.json" +check "verdict: a cancelled run is FAILED, not silently passed" 0 "failed" \ + forgejo_run_verdict "24" "$FJ/new-cancelled.json" +check "verdict: the first run ever (no pre-id) still resolves" 0 "success" \ + forgejo_run_verdict "" "$FJ/new-success.json" +# A task appears in this payload the moment it is ASSIGNED, which can be before +# it finishes — so a non-terminal status must read as pending, not as a failure. +# Calling a still-running job "failed" would make the leg flaky in exactly the +# window the leg is watching. +printf '%s' '{"workflow_runs":[{"id":25,"status":"running","run_number":1}],"total_count":1}' > "$FJ/new-running.json" +check "verdict: an assigned-but-running task is PENDING, not FAILED" 0 "pending" \ + forgejo_run_verdict "24" "$FJ/new-running.json" + +# The anti-false-positive guard, stated as its own case: an OLD completed run +# with the SAME id as pre_id must never be read as this dispatch's result. +check "verdict: a pre-existing success with the pre-id is NOT our run" 0 "pending" \ + forgejo_run_verdict "25" "$FJ/new-success.json" + + # Arg refusals fire before the root check (repo doctrine, bootstrap.sh:114), # which is what makes them provable here without a throwaway machine. check "drill.sh refuses to run without BOTH refs pinned (#103)" 2 "--box-ref" \ @@ -217,6 +265,14 @@ check "an unknown flag dies loudly, exit 2" 2 "unknown option" \ bash "$ROOT/drill/drill.sh" --frobnicate check "--help prints the header and exits 0" 0 "THROWAWAY" \ bash "$ROOT/drill/drill.sh" --help +check "--forgejo-instance is a known flag (the leg's opt-in)" 2 "--users is required" \ + bash "$ROOT/drill/drill.sh" --rig-ref r --box-ref b --forgejo-instance https://f.example.com --yes +check "--forgejo-runner-repo is a known flag" 2 "--users is required" \ + bash "$ROOT/drill/drill.sh" --rig-ref r --box-ref b --forgejo-runner-repo o/r --yes +check "--help names the forgejo runner leg's flags" 0 "--forgejo-instance" \ + bash "$ROOT/drill/drill.sh" --help +check "--forgejo-ref is a known flag (Forgejo's dispatch needs a ref)" 2 "--users is required" \ + bash "$ROOT/drill/drill.sh" --rig-ref r --box-ref b --forgejo-ref dev --yes echo "---" echo "$PASS passed, $FAIL failed" -- 2.45.2 From 9c0e508b76957882f9d83b5d6b0d00cf165670c3 Mon Sep 17 00:00:00 2001 From: cluade-reviewer-andresmgsl Date: Fri, 31 Jul 2026 00:13:59 +0000 Subject: [PATCH 2/4] fix(drill): read every task, and fail a configured leg that cannot mint MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Addresses grok's and kimi's REQUEST_CHANGES on !130. 1. forgejo_run_verdict read only the FIRST entry of actions/tasks. That payload accumulates, so the moment a repo is drilled twice our run shares it with older ones — and nothing documents the sort order. A green job then reports as a timeout: a false FAILURE on the gate this leg exists to provide. It now inspects every entry and lets the newest id above pre_id decide. Newlines are stripped first, so a pretty-printed payload parses like a compact one. 2. A mint that yielded nothing degraded to SKIPPED "no registration token source" — violating #129's own acceptance ("token source present but the instance is unreachable -> the leg FAILS; it must not skip and must not pass") and sending the operator to check an env var they had already set. forgejo_token_verdict separates absent inputs from a configured leg that could not mint; only the former skips. 3. The pre---yes confirm block still announced a GitHub runner alone. Refs #129 Co-Authored-By: Claude Opus 5 (1M context) --- drill/drill.sh | 71 ++++++++++++++++++++++++++++++++++++++------------ test/drill.sh | 46 ++++++++++++++++++++++++++++++-- 2 files changed, 98 insertions(+), 19 deletions(-) diff --git a/drill/drill.sh b/drill/drill.sh index 2e7412b..06d1b7f 100644 --- a/drill/drill.sh +++ b/drill/drill.sh @@ -143,29 +143,61 @@ leg() { LEG_NAMES+=("$1"); LEG_RESULTS+=("$2"); } # non-terminal status is pending, not failed; grading a running job as a # failure would make the leg flaky inside its own watch window. # +# EVERY entry is inspected, and the NEWEST id above pre_id decides — never +# entry[0]. actions/tasks accumulates, so the moment a repo is drilled twice +# our run shares the payload with older ones, and nothing documents the sort +# order. Reading the first entry made a green job report as a timeout, a false +# FAILURE on the gate this leg exists to provide (grok/kimi on !130). +# # grep-and-sed, not jq: a throwaway drill machine has neither jq nor an # authenticated forge CLI, the same constraint json_field() carries in # commands/lib/runner-config.sh. `id` is a bare number, which json_field's -# quoted-value shape cannot read, so this reads both forms itself. +# quoted-value shape cannot read, so this reads both forms itself. Newlines are +# stripped first so a pretty-printed payload parses identically to a compact +# one — the instance documents neither. forgejo_run_verdict() { - # `run_status`, not `status`: the latter is read-only in zsh, and while this - # file is bash, the function is awk-extracted and sourced by other shells in - # test harnesses — where the assignment fails silently and grades every run - # as pending. - local pre="$1" file="$2" id run_status - id="$(grep -o '"id"[[:space:]]*:[[:space:]]*[0-9][0-9]*' "$file" 2>/dev/null \ - | head -n1 | sed 's/.*:[[:space:]]*//')" - [ -n "$id" ] || { echo pending; return 0; } - [ "$id" != "$pre" ] || { echo pending; return 0; } - run_status="$(grep -o '"status"[[:space:]]*:[[:space:]]*"[^"]*"' "$file" 2>/dev/null \ - | head -n1 | sed 's/.*:[[:space:]]*"//; s/"$//')" - case "$run_status" in - success) echo success ;; + local pre="$1" file="$2" obj id best_id="" best_st="" + [ -r "$file" ] || { echo pending; return 0; } + while IFS= read -r obj; do + [ -n "$obj" ] || continue + id="$(printf '%s' "$obj" | grep -o '"id"[[:space:]]*:[[:space:]]*[0-9][0-9]*' \ + | head -n1 | sed 's/.*:[[:space:]]*//')" + [ -n "$id" ] || continue + # Strictly newer than the pre-dispatch id. Equal is the run that was + # already there; lower is older still. + if [ -n "$pre" ]; then + [ "$id" -gt "$pre" ] 2>/dev/null || continue + fi + if [ -z "$best_id" ] || [ "$id" -gt "$best_id" ] 2>/dev/null; then + best_id="$id" + best_st="$(printf '%s' "$obj" | grep -o '"status"[[:space:]]*:[[:space:]]*"[^"]*"' \ + | head -n1 | sed 's/.*:[[:space:]]*"//; s/"$//')" + fi + done </dev/null | grep -o '{[^{}]*}') +EOF + [ -n "$best_id" ] || { echo pending; return 0; } + case "$best_st" in + success) echo success ;; failure | cancelled | skipped | timedout) echo failed ;; - *) echo pending ;; + *) echo pending ;; esac } +# forgejo_token_verdict — ok | mint-failed | no-source. +# #129's acceptance: "Token source present but the instance is unreachable -> +# the leg FAILS; it must not skip and must not pass". A mint that yields +# nothing — unreachable instance, under-scoped token, wrong repo — is a +# CONFIGURED leg failing, and reporting it as "no token source" both writes +# SKIPPED where the record owes a FAIL and sends the operator to check an env +# var they already set. Absent inputs are the only honest skip. +forgejo_token_verdict() { + if [ -n "$1" ]; then echo ok + elif [ -n "$2" ]; then echo mint-failed + else echo no-source + fi +} + # run_logged — run a long command with its narration in a file # and a dot every 5s on the terminal: a silent multi-minute apt/install run is @@ -411,7 +443,7 @@ This will, ON THIS HOST ($(hostname)): · run 'rig bootstrap $ROLE --users $USERS_FILE' — sshd hardening, hostname change, tailnet join, box ($BOXREPO@$BOXREF) + its Incus stack — TWICE (the second run is the idempotence assertion) - · install Coolify${COOLIFY_VERSION:+ $COOLIFY_VERSION} and a GitHub runner${RUNNER_REPO:+ against $RUNNER_REPO} + · install Coolify${COOLIFY_VERSION:+ $COOLIFY_VERSION}, a GitHub runner${RUNNER_REPO:+ against $RUNNER_REPO} and a Forgejo runner${FJ_INSTANCE:+ against $FJ_INSTANCE}${FJ_RUNNER_REPO:+ ($FJ_RUNNER_REPO)} Only do this on a THROWAWAY machine you can format. EOF [ -t 0 ] || { echo "drill: no TTY to confirm on — pass --yes if you mean it." >&2; exit 2; } @@ -787,7 +819,12 @@ else | grep -o '"token"[[:space:]]*:[[:space:]]*"[^"]*"' | head -n1 \ | sed 's/.*:[[:space:]]*"//; s/"$//')" fi - if [ -z "$fj_reg" ]; then + fj_tok_verdict="$(forgejo_token_verdict "$fj_reg" "${FORGEJO_API_TOKEN:-}")" + if [ "$fj_tok_verdict" = mint-failed ]; then + # Configured, and it did not work. Never a skip: see forgejo_token_verdict. + no "registration-token mint FAILED against ${FJ_INSTANCE} — is it reachable, and does FORGEJO_API_TOKEN own ${FJ_RUNNER_REPO}? (the token is never printed)" + leg "forgejo runner lifecycle ($FJ_RUNNER_REPO)" "FAIL — registration-token mint failed" + elif [ "$fj_tok_verdict" = no-source ]; then skip "forgejo runner lifecycle: no FORGEJO_RUNNER_TOKEN and no FORGEJO_API_TOKEN to mint one — the leg did not run" leg "forgejo runner lifecycle ($FJ_RUNNER_REPO)" "SKIPPED — no registration token source" else diff --git a/test/drill.sh b/test/drill.sh index 04b8f38..5dd965b 100644 --- a/test/drill.sh +++ b/test/drill.sh @@ -50,10 +50,10 @@ trap 'rm -rf "$WORK"' EXIT # --- the functions under test, extracted ------------------------------------- FNS="$WORK/drill-fns.sh" -for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict; do +for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict forgejo_token_verdict; do awk "/^${fn}\(\) \{/,/^\}/" "$ROOT/drill/drill.sh" >> "$FNS" done -for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict; do +for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict forgejo_token_verdict; do check "extraction guards the awk: ${fn}() landed" 0 "${fn}() {" grep -F "${fn}() {" "$FNS" done # shellcheck source=/dev/null @@ -243,6 +243,48 @@ printf '%s' '{"workflow_runs":[{"id":25,"status":"running","run_number":1}],"tot check "verdict: an assigned-but-running task is PENDING, not FAILED" 0 "pending" \ forgejo_run_verdict "24" "$FJ/new-running.json" +# grok/kimi on !130: the reader must not stop at the FIRST run. actions/tasks +# accumulates — the moment a repo is drilled twice, our run shares the payload +# with older ones, and nothing documents the sort order. Reading entry[0] makes +# a green job read as a timeout, which is a FALSE FAILURE on the very gate this +# leg exists to provide. +printf '%s' '{"workflow_runs":[{"id":24,"status":"success"},{"id":25,"status":"success"}],"total_count":2}' > "$FJ/oldest-first.json" +printf '%s' '{"workflow_runs":[{"id":25,"status":"success"},{"id":24,"status":"success"}],"total_count":2}' > "$FJ/newest-first.json" +printf '%s' '{"workflow_runs":[{"id":25,"status":"running"},{"id":26,"status":"success"}],"total_count":2}' > "$FJ/ours-not-first.json" +printf '%s' '{"workflow_runs":[{"id":23,"status":"success"},{"id":24,"status":"failure"}],"total_count":2}' > "$FJ/all-stale.json" +# Pretty-printed: the instance may or may not compact its JSON, and a parser +# that silently depends on one-line objects is a latent failure (kimi, !130). +cp /tmp/fjfix/pretty.json "$FJ/pretty.json" +check "verdict: ours is LAST in the payload — order must not decide" 0 "success" \ + forgejo_run_verdict "24" "$FJ/oldest-first.json" +check "verdict: ours is FIRST in the payload — same answer" 0 "success" \ + forgejo_run_verdict "24" "$FJ/newest-first.json" +check "verdict: a stale RUNNING entry ahead of ours does not mask it" 0 "success" \ + forgejo_run_verdict "24" "$FJ/ours-not-first.json" +check "verdict: every entry at or below pre is stale — PENDING" 0 "pending" \ + forgejo_run_verdict "24" "$FJ/all-stale.json" +check "verdict: a pretty-printed payload parses too" 0 "success" \ + forgejo_run_verdict "24" "$FJ/pretty.json" + +# ============================================================================= +# forgejo_token_verdict — a configured leg that cannot mint must FAIL, not SKIP +# ============================================================================= +# #129's own acceptance: "Token source present but the instance is unreachable +# -> the leg FAILS; it must not skip and must not pass". A mint that returns +# nothing because the instance is unreachable, the token is under-scoped or the +# repo name is wrong is a CONFIGURED leg failing — reporting "no token source" +# sends the operator to check an env var they already set, and writes SKIPPED +# where the record owes a FAIL. That is the UNREADABLE-vs-NONE shape +# drills/README.md names. +check "token: a resolved registration token is ok" 0 "ok" \ + forgejo_token_verdict "reg-tok" "" +check "token: an explicit token wins even with no API token" 0 "ok" \ + forgejo_token_verdict "reg-tok" "" +check "token: no token at all and no API token is a genuine SKIP" 0 "no-source" \ + forgejo_token_verdict "" "" +check "token: API token offered but mint produced nothing is a FAILURE" 0 "mint-failed" \ + forgejo_token_verdict "" "api-tok" + # The anti-false-positive guard, stated as its own case: an OLD completed run # with the SAME id as pre_id must never be read as this dispatch's result. check "verdict: a pre-existing success with the pre-id is NOT our run" 0 "pending" \ -- 2.45.2 From e42ba31a093e9323189f50ff89070e692db478f5 Mon Sep 17 00:00:00 2001 From: cluade-reviewer-andresmgsl Date: Fri, 31 Jul 2026 00:29:06 +0000 Subject: [PATCH 3/4] fix(drill): the pre-dispatch baseline folds max too, and the suite is hermetic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round 2 of !130. Both blockers were mine, and one was the same mistake I had just fixed one layer up. 1. fj_pre still took `head -n1`. The verdict folds max over every id, but the PRE-DISPATCH snapshot did not, so an oldest-first payload named an old run as the baseline — and a later poll finding the same body then read the PREVIOUS drill's run as this dispatch's result. That is a false PASS on the take-a-job assertion, strictly worse than the false FAIL entry[0] caused inside the verdict. Both sides now share forgejo_max_task_id, and a test composes them the way the leg does so the pair cannot drift apart again. 2. test/drill.sh copied its pretty-printed fixture from /tmp/fjfix — a scratch path that existed only on the box the fix was built on. Everywhere else the cp failed, the guard returned pending, and the suite was 65/66. The claimed 66/66 was true on one machine. The fixture is written inline like every other one; verified by deleting the scratch dir and running the suite from a clean tree under env -i. Refs #129 Co-Authored-By: Claude Opus 5 (1M context) --- drill/drill.sh | 25 +++++++++++++++++++++++-- test/drill.sh | 38 +++++++++++++++++++++++++++++++++++--- 2 files changed, 58 insertions(+), 5 deletions(-) diff --git a/drill/drill.sh b/drill/drill.sh index 06d1b7f..106341d 100644 --- a/drill/drill.sh +++ b/drill/drill.sh @@ -184,6 +184,28 @@ EOF esac } +# forgejo_max_task_id — the highest numeric task id in the payload, +# empty when there is none. This is the PRE-DISPATCH baseline, and it must fold +# max exactly as forgejo_run_verdict does: taking the first id instead names an +# OLD run as the baseline whenever the payload is not newest-first (the order is +# undocumented). A later poll that finds the same body then reads the PREVIOUS +# drill's run as this dispatch's result — a false PASS on the take-a-job +# assertion, which is worse than the false FAIL the same mistake caused inside +# the verdict (grok/kimi, !130). The GitHub leg is safe from this only because +# `gh run list --limit 1` contracts newest-first; this API contracts nothing. +forgejo_max_task_id() { + local file="$1" id best="" + [ -r "$file" ] || return 0 + while IFS= read -r id; do + [ -n "$id" ] || continue + if [ -z "$best" ] || [ "$id" -gt "$best" ] 2>/dev/null; then best="$id"; fi + done </dev/null \ + | grep -o '"id"[[:space:]]*:[[:space:]]*[0-9][0-9]*' | sed 's/.*:[[:space:]]*//') +EOF + printf '%s\n' "$best" +} + # forgejo_token_verdict — ok | mint-failed | no-source. # #129's acceptance: "Token source present but the instance is unreachable -> # the leg FAILS; it must not skip and must not pass". A mint that yields @@ -850,8 +872,7 @@ else fj_pre_body="$(curl -fsSL -H "Authorization: token ${FORGEJO_API_TOKEN}" \ "$fj_api/actions/tasks" 2>/dev/null || echo '{}')" printf '%s' "$fj_pre_body" > /tmp/drill-forgejo-pre.json - fj_pre="$(grep -o '"id"[[:space:]]*:[[:space:]]*[0-9][0-9]*' /tmp/drill-forgejo-pre.json \ - 2>/dev/null | head -n1 | sed 's/.*:[[:space:]]*//')" + fj_pre="$(forgejo_max_task_id /tmp/drill-forgejo-pre.json)" if curl -fsSL -o /dev/null -X POST -H "Authorization: token ${FORGEJO_API_TOKEN}" \ -H "Content-Type: application/json" -d "{\"ref\":\"${FJ_REF}\"}" \ "$fj_api/actions/workflows/${RUNNER_WORKFLOW}/dispatches" 2>/dev/null; then diff --git a/test/drill.sh b/test/drill.sh index 5dd965b..17855a5 100644 --- a/test/drill.sh +++ b/test/drill.sh @@ -50,10 +50,10 @@ trap 'rm -rf "$WORK"' EXIT # --- the functions under test, extracted ------------------------------------- FNS="$WORK/drill-fns.sh" -for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict forgejo_token_verdict; do +for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict forgejo_token_verdict forgejo_max_task_id; do awk "/^${fn}\(\) \{/,/^\}/" "$ROOT/drill/drill.sh" >> "$FNS" done -for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict forgejo_token_verdict; do +for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict forgejo_token_verdict forgejo_max_task_id; do check "extraction guards the awk: ${fn}() landed" 0 "${fn}() {" grep -F "${fn}() {" "$FNS" done # shellcheck source=/dev/null @@ -254,7 +254,15 @@ printf '%s' '{"workflow_runs":[{"id":25,"status":"running"},{"id":26,"status":"s printf '%s' '{"workflow_runs":[{"id":23,"status":"success"},{"id":24,"status":"failure"}],"total_count":2}' > "$FJ/all-stale.json" # Pretty-printed: the instance may or may not compact its JSON, and a parser # that silently depends on one-line objects is a latent failure (kimi, !130). -cp /tmp/fjfix/pretty.json "$FJ/pretty.json" +# Written HERE, like every other fixture: a suite that copies from a scratch +# path passes only on the box that built it (grok/kimi, !130 round 2). +printf '%s\n' '{ + "workflow_runs": [ + {"id": 24, "status": "success"}, + {"id": 25, "name": "drill", "status": "success"} + ], + "total_count": 2 +}' > "$FJ/pretty.json" check "verdict: ours is LAST in the payload — order must not decide" 0 "success" \ forgejo_run_verdict "24" "$FJ/oldest-first.json" check "verdict: ours is FIRST in the payload — same answer" 0 "success" \ @@ -266,6 +274,30 @@ check "verdict: every entry at or below pre is stale — PENDING" 0 "pending" \ check "verdict: a pretty-printed payload parses too" 0 "success" \ forgejo_run_verdict "24" "$FJ/pretty.json" +# The PRE-DISPATCH snapshot has the same multi-entry hazard as the verdict, and +# getting it wrong is worse: a `head -n1` pre-id on an oldest-first payload +# names an OLD run as the baseline, so a later poll that finds the same body +# reports the PREVIOUS drill's run as ours — a false PASS on the take-a-job +# assertion, where the entry[0] bug only produced a false failure (grok, !130). +# Both sides must fold max over every id, which is why they share one function. +check "max id: oldest-first payload yields the NEWEST id, not the first" 0 "25" \ + forgejo_max_task_id "$FJ/oldest-first.json" +check "max id: newest-first payload yields the same answer" 0 "25" \ + forgejo_max_task_id "$FJ/newest-first.json" +check "max id: an empty payload has no id at all" 0 "" \ + forgejo_max_task_id "$FJ/empty.json" +check "max id: a pretty-printed payload folds too" 0 "25" \ + forgejo_max_task_id "$FJ/pretty.json" +# The false PASS, pinned end to end: snapshot the oldest-first body, dispatch, +# the runner never takes it so the body is unchanged — the verdict must stay +# pending. With head -n1 this returned success. +# The two halves composed exactly as the leg composes them. +verdict_after_no_new_run() { forgejo_run_verdict "$(forgejo_max_task_id "$1")" "$1"; } +check "no new run after dispatch: max-id baseline keeps it PENDING (false-PASS guard)" 0 "pending" \ + verdict_after_no_new_run "$FJ/oldest-first.json" +check "…and the same composition on a pretty payload" 0 "pending" \ + verdict_after_no_new_run "$FJ/pretty.json" + # ============================================================================= # forgejo_token_verdict — a configured leg that cannot mint must FAIL, not SKIP # ============================================================================= -- 2.45.2 From 9afcdf858f24e501e8b3ce856d457c58194e867e Mon Sep 17 00:00:00 2001 From: cluade-reviewer-andresmgsl Date: Fri, 31 Jul 2026 20:04:16 +0000 Subject: [PATCH 4/4] fix(drill): the leg row is the whole lifecycle, not just the job MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit codex, grok and kimi all landed on the same hole at e42ba31, and they are right. The record row keyed only on fj_took, so install/status/remove failures called `no` without gating it. A dispatched job only needs SOMETHING answering runs-on: drill — and this leg removes locally, telling the operator to delete the stale runner by hand, so a leftover drill-labeled runner from the previous drill is the designed-for aftermath, not a contrived case. It answers the fixture exactly, and the row then read "PASS — registered, took a job, removed" for a drill whose install never happened. The exit code was never wrong: every one of those failures also called `no`. What was wrong is the row, and the row is what outlives the run in drills/.md. A row claiming a lifecycle that did not happen is exactly what the gate exists to refuse — kimi's framing, and it is the right one. forgejo_leg_row folds all five assertions, and the leg no longer dispatches after install or status has failed: a foreign runner's success could only ever be evidence about someone else's runner. The regression composes the real functions over a payload carrying a newer successful run, the way the leg composes them. The GitHub leg has the identical shape and is byte-frozen by #129's contract; left alone deliberately, and worth a sibling issue for triage. Refs #129 Co-Authored-By: Claude Opus 5 (1M context) --- drill/drill.sh | 71 +++++++++++++++++++++++++++++++++++++++----------- test/drill.sh | 53 +++++++++++++++++++++++++++++++++++-- 2 files changed, 107 insertions(+), 17 deletions(-) diff --git a/drill/drill.sh b/drill/drill.sh index 106341d..a5ba7b8 100644 --- a/drill/drill.sh +++ b/drill/drill.sh @@ -184,6 +184,35 @@ EOF esac } +# forgejo_leg_row +# The record row for the Forgejo runner leg. PASS requires the WHOLE lifecycle, +# not just the take-a-job outcome. +# +# Keying the row on alone let it read "PASS — registered, took a job, +# removed" when install had failed, because the dispatched job only needs +# SOMETHING answering runs-on: drill — and this leg removes locally, telling the +# operator to delete the stale runner by hand, so a leftover drill-labeled +# runner from the previous drill is the designed-for aftermath rather than a +# contrived case (codex/grok/kimi on !130). drills/.md is the release's +# durable evidence; a row claiming a lifecycle that did not happen is exactly +# what the gate exists to refuse. +# +# The drill's exit code was never wrong here — every one of those failures also +# called `no`. What was wrong is the row, and the row is what outlives the run. +forgejo_leg_row() { + local install_ok="$1" status_ok="$2" took="$3" remove_ok="$4" absent_ok="$5" + if [ "$install_ok" != 1 ] || [ "$status_ok" != 1 ] \ + || [ "$remove_ok" != 1 ] || [ "$absent_ok" != 1 ]; then + echo "FAIL — see Failed below" + return 0 + fi + case "$took" in + success) echo "PASS — registered, took a job, removed (stale row needs deleting by hand)" ;; + none) echo "PARTIAL — registered and removed; took a job: not attempted (no FORGEJO_API_TOKEN)" ;; + *) echo "FAIL — see Failed below" ;; + esac +} + # forgejo_max_task_id — the highest numeric task id in the payload, # empty when there is none. This is the PRE-DISPATCH baseline, and it must fold # max exactly as forgejo_run_verdict does: taking the first id instead names an @@ -851,21 +880,31 @@ else leg "forgejo runner lifecycle ($FJ_RUNNER_REPO)" "SKIPPED — no registration token source" else FJ_NAME="drill-$(hostname)-$$" + fj_install_ok=0 fj_status_ok=0 fj_remove_ok=0 fj_absent_ok=0 # The label MUST carry a docker:// image: forgejo-runner runs jobs in # containers, and a bare label leaves runs-on matched but unrunnable. if FORGEJO_RUNNER_TOKEN="$fj_reg" run_logged /tmp/drill-forgejo-runner-install.log \ rig forgejo-runner install --instance "$FJ_INSTANCE" --name "$FJ_NAME" \ --labels 'drill:docker://node:22-bookworm'; then ok "rig forgejo-runner install --instance $FJ_INSTANCE exited 0 (registered as $FJ_NAME)" + fj_install_ok=1 else no "forgejo-runner install FAILED — tail: $(tail -3 /tmp/drill-forgejo-runner-install.log | tr '\n' ' ')" fi - rig forgejo-runner status 2>/dev/null | grep -qF "${FJ_INSTANCE%/}" \ - && ok "forgejo-runner status names the instance: $FJ_INSTANCE" \ - || no "forgejo-runner status does not name ${FJ_INSTANCE}" + if rig forgejo-runner status 2>/dev/null | grep -qF "${FJ_INSTANCE%/}"; then + ok "forgejo-runner status names the instance: $FJ_INSTANCE"; fj_status_ok=1 + else + no "forgejo-runner status does not name ${FJ_INSTANCE}" + fi fj_took=none - if [ -n "${FORGEJO_API_TOKEN:-}" ]; then + # Do NOT dispatch once install or status has failed. The job would be taken + # by whatever else answers runs-on: drill — a stale runner this leg's own + # hand-delete caveat leaves behind — and its success would be evidence about + # someone else's runner (codex/grok/kimi, !130). + if [ "$fj_install_ok" != 1 ] || [ "$fj_status_ok" != 1 ]; then + skip "took a job: not attempted — install or status failed, and a foreign runner answering 'drill' could only manufacture a false pass" + elif [ -n "${FORGEJO_API_TOKEN:-}" ]; then fj_api="${FJ_INSTANCE%/}/api/v1/repos/${FJ_RUNNER_REPO}" # Read the newest ASSIGNED task id BEFORE dispatching, same guard as the # GitHub leg: an already-completed run must never be read as ours. @@ -908,19 +947,21 @@ else fi # No removal token exists on this forge — remove is local by design. - rig forgejo-runner remove >/dev/null 2>&1 \ - && note "forgejo-runner removed locally — Forgejo has no deregistration endpoint, so DELETE the stale '$FJ_NAME' row under $FJ_RUNNER_REPO > Settings > Actions > Runners by hand" \ - || no "forgejo-runner remove FAILED" - rig forgejo-runner status >/dev/null 2>&1 \ - && no "forgejo-runner status still answers after remove — the removal did not take" \ - || ok "forgejo-runner status confirms: nothing registered" + if rig forgejo-runner remove >/dev/null 2>&1; then + note "forgejo-runner removed locally — Forgejo has no deregistration endpoint, so DELETE the stale '$FJ_NAME' row under $FJ_RUNNER_REPO > Settings > Actions > Runners by hand" + fj_remove_ok=1 + else + no "forgejo-runner remove FAILED" + fi + if rig forgejo-runner status >/dev/null 2>&1; then + no "forgejo-runner status still answers after remove — the removal did not take" + else + ok "forgejo-runner status confirms: nothing registered"; fj_absent_ok=1 + fi leg "forgejo runner lifecycle ($FJ_RUNNER_REPO)" \ - "$(case "$fj_took" in - success) echo "PASS — registered, took a job, removed (stale row needs deleting by hand)" ;; - none) echo "PARTIAL — registered and removed; took a job: not attempted (no FORGEJO_API_TOKEN)" ;; - *) echo "FAIL — see Failed below" ;; - esac)" + "$(forgejo_leg_row "$fj_install_ok" "$fj_status_ok" "$fj_took" \ + "$fj_remove_ok" "$fj_absent_ok")" fi fi diff --git a/test/drill.sh b/test/drill.sh index 17855a5..7b94eb7 100644 --- a/test/drill.sh +++ b/test/drill.sh @@ -50,10 +50,10 @@ trap 'rm -rf "$WORK"' EXIT # --- the functions under test, extracted ------------------------------------- FNS="$WORK/drill-fns.sh" -for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict forgejo_token_verdict forgejo_max_task_id; do +for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict forgejo_token_verdict forgejo_max_task_id forgejo_leg_row; do awk "/^${fn}\(\) \{/,/^\}/" "$ROOT/drill/drill.sh" >> "$FNS" done -for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict forgejo_token_verdict forgejo_max_task_id; do +for fn in tree_of assert_installed_from classify_leg capture_state emit_record forgejo_run_verdict forgejo_token_verdict forgejo_max_task_id forgejo_leg_row; do check "extraction guards the awk: ${fn}() landed" 0 "${fn}() {" grep -F "${fn}() {" "$FNS" done # shellcheck source=/dev/null @@ -298,6 +298,55 @@ check "no new run after dispatch: max-id baseline keeps it PENDING (false-PASS g check "…and the same composition on a pretty payload" 0 "pending" \ verdict_after_no_new_run "$FJ/pretty.json" +# ============================================================================= +# forgejo_leg_row — the row is the WHOLE lifecycle, not just the job +# ============================================================================= +# codex/grok/kimi on !130: keying the record row on the take-a-job outcome alone +# lets it read "PASS — registered, took a job, removed" when install failed, so +# long as SOMETHING answered runs-on: drill. That is not contrived — this leg +# removes locally and tells the operator to delete the stale runner by hand, so +# a leftover drill-labeled runner from the previous drill is the DESIGNED-FOR +# aftermath, and it answers the fixture exactly. +# +# drills/.md is the release's durable evidence. A row claiming a lifecycle +# that did not happen is precisely what the gate exists to refuse, so PASS +# requires every assertion, not just the interesting one. +check "leg row: everything succeeded is the only PASS" 0 "PASS" \ + forgejo_leg_row 1 1 success 1 1 +check "leg row: install failed cannot PASS, even when a foreign runner took the job" 0 "FAIL" \ + forgejo_leg_row 0 1 success 1 1 +check "leg row: status failed cannot PASS either" 0 "FAIL" \ + forgejo_leg_row 1 0 success 1 1 +check "leg row: remove failed cannot PASS" 0 "FAIL" \ + forgejo_leg_row 1 1 success 0 1 +check "leg row: a runner still registered after remove cannot PASS" 0 "FAIL" \ + forgejo_leg_row 1 1 success 1 0 +check "leg row: no dispatch attempted, everything else clean, is PARTIAL" 0 "PARTIAL" \ + forgejo_leg_row 1 1 none 1 1 +check "leg row: a job that was never taken is a FAIL" 0 "FAIL" \ + forgejo_leg_row 1 1 timeout 1 1 +check "leg row: PARTIAL requires a clean lifecycle too" 0 "FAIL" \ + forgejo_leg_row 0 1 none 1 1 + +# The end-to-end shape codex/grok/kimi asked for, composed the way the leg +# composes it: a tasks payload carrying a NEWER successful run (as a foreign +# drill-labeled runner would produce) must still not yield a PASS row when the +# drill's own install failed. This is the exact false-evidence case. +row_after_failed_install() { + forgejo_leg_row 0 1 "$(forgejo_run_verdict "$(forgejo_max_task_id "$1")" "$1")" 1 1 +} +printf '%s' '{"workflow_runs":[{"id":24,"status":"success"},{"id":99,"status":"success"}],"total_count":2}' \ + > "$FJ/foreign-runner-took-it.json" +check "install failed + a newer successful run in the payload is still FAIL, never PASS" 0 "FAIL" \ + row_after_failed_install "$FJ/foreign-runner-took-it.json" +# …and the same payload with a clean lifecycle is the PASS, so the check above +# is discriminating rather than always-FAIL. +row_after_clean_install() { + forgejo_leg_row 1 1 "$(forgejo_run_verdict "" "$1")" 1 1 +} +check "…while the same payload with a clean lifecycle does PASS" 0 "PASS" \ + row_after_clean_install "$FJ/foreign-runner-took-it.json" + # ============================================================================= # forgejo_token_verdict — a configured leg that cannot mint must FAIL, not SKIP # ============================================================================= -- 2.45.2