diff --git a/changelog.d/202.md b/changelog.d/202.md new file mode 100644 index 0000000..9f27adf --- /dev/null +++ b/changelog.d/202.md @@ -0,0 +1,64 @@ +### Added + +- `docs/RUNNER-PROBES.md` documents the standing runner-probe venue, + `heavy-duty/ceremony-runner-probe` — the place runner-only facts are measured + on demand, ruled as option A by the operator (#202). + +- `drills/README.md` cross-links it beside the disposal rule, so the exception + is visible where the dangerous habit lives (#202). + +- The runbook states that the drill disposal rule does **not** apply to it. + Archiving it defeats its purpose, and that is exactly how the three existing + drill repos each became unavailable (#202). + +- It records that a probe must run as an Actions job under the workflow token: + the same call answers 500 there and 204 under a PAT, so a probe run any other + way produces a confident wrong answer (#202). + +- Creating the repo is recorded as the operator's step, measured rather than + assumed: a fleet identity gets 403 on org repo creation and 201 in its own + namespace (#202). + +- It carries an executable two-layer arming procedure: an immutable candidate + code SHA and an armed workflow commit on top of it. A single layer is + self-referential — rewriting a workflow makes a new commit, and a commit + cannot contain its own object ID (#202). + +- Callers are pinned by layer: composite actions to the candidate code SHA, + reusable workflows to the armed SHA, which is the only revision whose inner + checkout points at the fork (#202). + +- The arming gate asserts what each carrier IS, not only that the old literal + is gone: every `repository:` equals the fork, every `CEREMONY_SELF_REF` value + equal the candidate code SHA, and callers match the layer they belong to + (#202). + +- It enumerates the carriers from the tree rather than encoding a count, and + distinguishes ceremony's internal self-checkouts from the consumer checkouts + that must stay `${{ github.repository }}` (#202). + +- Both published snippets are ShellCheck-clean when extracted and linted + directly, not merely as part of the repository sweep (#202). + +- The checker validates the MANIFEST against the target it was given, so a + manifest that describes a wrong arming consistently — wrong fork, or the + armed SHA where the candidate belongs — refuses instead of matching a tree + rewritten to the same wrong value (#202). + +- The manifest is generated from the PRE-arming tree, which is the only order + that enumerates the carriers that must change (#202). + +- Both published snippets were driven against a constructed candidate/probe + pair: deletion, both role swaps, wrong owner, wrong + SHA, wrong path, a deleted caller class and an extra carrier all refuse, and + the armed control passes (#202). + +- The manifest records complete caller coordinates, so a path swapped under the + right owner and SHA is caught (#202). + +- Generator and checker share one domain — ceremony callers — so a third-party + `actions/checkout` is neither manifested nor reported as unrecognised (#202). + +- Probe results are written to an issue in the probe repo and carried to the + ceremony issue by a human, so the probe holds no path that can write to the + live board (#202). diff --git a/docs/RUNNER-PROBES.md b/docs/RUNNER-PROBES.md new file mode 100644 index 0000000..1e98bc2 --- /dev/null +++ b/docs/RUNNER-PROBES.md @@ -0,0 +1,345 @@ +# Runner probes + +**Not a drill.** A drill rehearses the release doors on a disposable repo and +ends. This is the opposite shape: one **standing** repo that exists so that +runner-only facts can be measured on demand, and it is **never archived**. + +`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes. +Ruled by the operator as option (A) of ceremony#202 (#5631). + +## Why a standing repo, when drills are disposable + +Some facts are only true inside Actions, under the token Actions injects, and +no local harness or PAT can reproduce them. The worked example is ceremony#192: + +``` +DELETE /issues/{n}/labels/{id} -> 500 under ${{ github.token }} in a workflow +DELETE /issues/{n}/labels/{id} -> 204 under a maintainer PAT, same call +``` + +A probe that runs anywhere else passes and proves nothing. Before this venue +existed the answer was "un-archive a drill repo", which was requested three +times in two days across two issues and never became anything — the three +drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived, +and each was minted for one probe and then wanted again. + +## The disposal rule above does NOT apply here + +The rehearsal section says the builder archives the scratch repo and the +operator deletes it. **That rule is for drills.** Archiving this repo defeats +its entire purpose, and it is the failure mode the three archived drill repos +demonstrate — each was archived correctly, by the rule, and each then had to be +un-archived or replaced. + +So: never archive it, never delete it, and if you find it archived, un-archive +it rather than minting a fourth one. + +## Standing it up is the operator's step + +Bot identities cannot create repositories in `heavy-duty`. Measured +2026-08-05 with a fleet identity holding the `repo` scope: + +``` +POST /api/v1/orgs/heavy-duty/repos -> 403 "not allowed to create repository in organization" +POST /api/v1/user/repos -> 201 (personal namespace only) +``` + +This is the same shape as the drill delete: a deliberate permission boundary, +not a misconfiguration. Do not retry it, and do not work around it by putting +the venue in a personal namespace — **not because a personal namespace is +proven unable to reach the org's runner** (that was not measured; the probe +repository above was deleted immediately, so nothing about runner or secret +reach was established), but because @andres ruled an **org-owned standing +venue** (#5631). A personally-owned repo is a different thing from the one that +was decided on, and cannot satisfy #202's named acceptance target. + +If runner or secret reach turns out to matter, measure it once the venue +exists rather than assuming it here. + +## Running a probe + +1. Reset the repo to a clean state — the probe's own fixtures only, no + leftovers from the last one. A probe that inherits state is a probe whose + result you cannot attribute. +2. **Arm it against the candidate** (below) — two layers, candidate code and + armed workflow — if the probe is about ceremony's own machinery rather than + about a bare API call. +3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole + point of the venue and the one step that cannot be shortcut. A `curl` from a + laptop with a PAT answers a different question — see the 204/500 split + above — and a probe run that way is worse than no probe, because it produces + a confident wrong answer. +4. **The job writes its raw results into an issue in the PROBE repo** — + `heavy-duty/ceremony-runner-probe` — not into ceremony. Logs age out; + ceremony#192's run 701 survived only because the job wrote its findings + into an issue it created. +5. **A human then records the probe issue's URL and the Actions run number on + the ceremony issue the probe serves.** That hop is deliberate and is the + whole of the boundary: the probe workflow holds no credential and no code + path that can write to `heavy-duty/ceremony`, so "the probe reports its + findings" and "the probe cannot touch the live board" stay compatible + rather than contradicting each other (@codex-reviewer-andresmgsl, #202 + review). + +## Arming a candidate ref + +A probe that exercises ceremony's own machinery needs the candidate tree +reachable from a `uses:` line. This is the fork-ref shape `drills/README.md` +step 2 points at, written out — and it has **two layers**, which is the part +that is easy to get wrong and impossible to fix afterwards. + +**Why two.** The candidate's own workflows contain +`repository: heavy-duty/ceremony` beside `ref: ${{ env.CEREMONY_SELF_REF }}`, +so they must be rewritten to point at the fork and at the candidate. But +rewriting them **creates a new commit**, and a commit cannot contain its own +object ID. A single-layer arming is therefore self-referential: pin the callers +to the pre-rewrite SHA and they load the *unarmed* workflows; pin them to the +post-rewrite one and you are asking a commit to embed itself +(@codex-reviewer-andresmgsl, #202 review). + +So: + +| layer | what it is | what it carries | +|---|---|---| +| **candidate code SHA** | the immutable tree under test | `actions/`, `lib/` — untouched | +| **armed workflow SHA** | a small child commit on top of it | workflows rewritten to the fork + `CEREMONY_SELF_REF` = the candidate code SHA | + +### The procedure + +1. **Push the candidate tree** to a fork under the identity that will run the + probe — one branch, `/ceremony@probe-` — and record its + SHA. Steps 1 and 2 advance the tip of that **same** branch; there are two + commits, not two branches. That is + the **candidate code SHA**. Never create a branch on + `heavy-duty/ceremony` named like a tag: it shadows that tag for every + consumer until somebody remembers to delete it. +2. **Write the manifest FIRST, from the pre-arming tree, then commit the + arming.** The manifest enumerates the carriers *that must change*, so it is + generated before they do — running it afterwards would enumerate + already-rewritten rows and lose the canonical internal-checkout ones + entirely (@codex-reviewer-andresmgsl, #202 review). In that same + fork branch rewrite, for **every** carrier the manifest below enumerates: + ceremony's own internal `repository:` checkouts → `/ceremony`, and + **every** `CEREMONY_SELF_REF` value → the **candidate code SHA** from step 1. + There were three self-ref carriers on `main` at the time of writing and the + count is not a constant — derive it, do not remember it + (@glm-reviewer-andresmgsl, @codex-reviewer-andresmgsl, #202 review). The + **consumer** checkouts (`${{ github.repository }}`) are left alone. Record + the resulting SHA: that is the **armed workflow SHA**. +3. **Pin the probe repo's callers by layer**, because they are not the same + thing: + - composite-action callers → + `/ceremony/actions/@`; + - reusable-workflow callers → + `/ceremony/.github/workflows/@`, since + that is the only revision whose inner checkout is rewritten. +4. **Gate the arming against a MANIFEST, byte for byte.** Every weaker shape + has a hole, and each of these was found in a published draft of this file + (@codex-reviewer-andresmgsl, #202 review): + + | weaker check | what slips through | + |---|---| + | "the old literal is absent" | a carrier rewritten to the wrong fork, or to the *armed* SHA | + | "every extracted value equals X" | a carrier that **vanished** — nothing to compare | + | "each value is one of {fork, dynamic}" | a **role swap**: an internal checkout made dynamic, a consumer checkout pointed at the fork | + | "the SHA suffix matches" | `wrong-owner/ceremony/actions/foo@` | + | "known callers match" | an **unrecognised** caller, or none at all | + | "the owner and the sha are right for the kind" | a **layer swap**: `…/actions/x@` labelled a workflow caller satisfies both | + + So the arming step **writes a manifest** — one line per carrier, `path`, + `kind`, `full expected value` — and the gate compares the tree's actual + carriers against it as a set. A deletion, a role swap, a wrong fork, a wrong + SHA, an extra carrier and a missing caller are then all the same kind of + failure: the sets differ. + + **Generate it while arming**, from the tree you are arming, so the manifest + cannot drift from the repository: + + ```sh + #!/usr/bin/env bash + # write-manifest + # + # Run against the PRE-ARMING tree and the UNPINNED probe: this records what + # each carrier must BECOME, so it has to see them before they change. + # + # `|| true` on every extraction, for the same reason the checker needs it: + # git grep exits 1 on no-match and `set -e` would abort BEFORE the manifest + # is written — silently, which is how the first version of this generator + # produced no file and no diagnostic when a probe exercised only one layer + # (@codex-reviewer-andresmgsl). A probe need not use both. + set -euo pipefail + candidate="$1"; probe="$2"; fork="$3"; code_sha="$4"; armed_sha="$5" + # shellcheck disable=SC2016 # `${{ github.repository }}` is literal YAML, not a shell expansion + { + git -C "$candidate" grep -n 'CEREMONY_SELF_REF:' -- .github/workflows \ + | cut -d: -f1,2 | sed "s|$|\tself_ref\t$code_sha|" || true + git -C "$candidate" grep -n 'repository: heavy-duty/ceremony' -- .github/workflows \ + | cut -d: -f1,2 | sed "s|$|\tinternal_repo\t$fork|" || true + git -C "$candidate" grep -n 'repository: ${{ github.repository }}' -- .github/workflows \ + | cut -d: -f1,2 | sed 's|$|\tconsumer_repo\t${{ github.repository }}|' || true + # Callers record the COMPLETE expected coordinate, not just the sha: the + # path is as rewritable as the owner, and a manifest that stores only the + # suffix cannot notice `…/actions/wrong-one@`. + git -C "$probe" grep -nE 'uses:[[:space:]]*[^[:space:]]*/ceremony/\.github/workflows/' -- .github \ + | sed -E "s|^([^:]+):([0-9]+):.*/ceremony/(\.github/workflows/[^@[:space:]]+)@.*|\\1:\\2\\tworkflow_caller\\t$fork/\\3@$armed_sha|" || true + git -C "$probe" grep -nE 'uses:[[:space:]]*[^[:space:]]*/ceremony/actions/' -- .github \ + | sed -E "s|^([^:]+):([0-9]+):.*/ceremony/(actions/[^@[:space:]]+)@.*|\\1:\\2\\taction_caller\\t$fork/\\3@$code_sha|" || true + } | sort >manifest.tsv + + # Zero ceremony callers is a refusal by name; one layer only is fine. + callers="$(grep -cE '(workflow|action)_caller' manifest.tsv || true)" + [ "$callers" -gt 0 ] || { echo "manifest: no ceremony callers found in $probe" >&2; exit 1; } + ``` + + Then arm — rewrite and commit — and check the result against it: + + ```sh + #!/usr/bin/env bash + # check-arming + set -euo pipefail + armed="$1"; probe="$2"; fork="$3"; code_sha="$4"; armed_sha="$5"; manifest="$6" + fail() { echo "arming incomplete: $*" >&2; exit 1; } + + # `|| true` on every extraction: git grep exits 1 when nothing matches, and + # under `set -e` that would kill this script BEFORE the comparison — so a + # carrier class that vanished ENTIRELY produced silence instead of a + # refusal. Silence is the worst of the three outcomes; the comparison below + # is what must report it. + [ "$(grep -cE '(workflow|action)_caller' "$manifest" || true)" -gt 0 ] \ + || fail "manifest names no ceremony callers — it cannot prove an arming" + + # THE MANIFEST ITSELF IS CHECKED AGAINST THE TARGET, not trusted. Comparing + # only tree-vs-manifest proves consistency, and a manifest generated with the + # armed SHA where the candidate SHA belonged — or with the wrong fork — + # describes a WRONG arming perfectly. The tree would then match it and the + # gate would pass (@codex-reviewer-andresmgsl, #202 review). + # shellcheck disable=SC2016 # `${{ github.repository }}` below is literal YAML + while IFS=$'\t' read -r loc kind want; do + case "$kind" in + self_ref) [ "$want" = "$code_sha" ] || fail "manifest $loc: self_ref should be the CANDIDATE sha" ;; + internal_repo) [ "$want" = "$fork" ] || fail "manifest $loc: internal repo should be $fork" ;; + consumer_repo) [ "$want" = '${{ github.repository }}' ] \ + || fail "manifest $loc: consumer checkout must stay dynamic" ;; + # THE KIND MUST BIND TO THE PATH CLASS, not only to the owner and the + # sha. The path class is what SAYS which layer a caller is, so checking + # the sha against the kind while letting the kind float free accepts a + # consistent layer swap — `…/actions/x@` declared workflow_caller + # passes every owner and sha test (@codex-reviewer-andresmgsl, #202 + # review). Decompose once, then let the kind fix BOTH coordinates. + workflow_caller|action_caller) + owner="${want%%/ceremony/*}"; rest="${want#*/ceremony/}" + path="${rest%@*}"; sha="${want##*@}" + [ "$owner/ceremony" = "$fork" ] \ + || fail "manifest $loc: caller owner should be $fork" + case "$kind" in + workflow_caller) + case "$path" in .github/workflows/?*) : ;; + *) fail "manifest $loc: workflow_caller must resolve at .github/workflows/, not '$path'" ;; + esac + [ "$sha" = "$armed_sha" ] || fail "manifest $loc: workflow caller should be the ARMED sha" ;; + action_caller) + case "$path" in actions/?*) : ;; + *) fail "manifest $loc: action_caller must resolve at actions/, not '$path'" ;; + esac + [ "$sha" = "$code_sha" ] || fail "manifest $loc: action caller should be the CANDIDATE sha" ;; + esac ;; + *) fail "manifest $loc: unknown kind '$kind'" ;; + esac + done <"$manifest" + + actual="$(mktemp)"; trap 'rm -f "$actual"' EXIT + { + git -C "$armed" grep -nP '(?<=CEREMONY_SELF_REF: ")[^"]+' -- .github/workflows \ + | sed -E 's/^([^:]+):([0-9]+):.*CEREMONY_SELF_REF: "([^"]*)".*/\1:\2\tself_ref\t\3/' || true + git -C "$armed" grep -nE 'repository: .+' -- .github/workflows \ + | sed -E 's|^([^:]+):([0-9]+):[[:space:]]*repository:[[:space:]]*(.*)$|\1:\2\t__repo__\t\3|' || true + # Only CEREMONY callers, matching the generator's domain exactly — a + # third-party `actions/checkout` is not this gate's business, and + # extracting it here while the generator ignores it made every probe fail + # as an "unrecognised carrier" (@codex-reviewer-andresmgsl). A wrong OWNER + # is still caught: `wrong-owner/ceremony/...` matches this pattern. + git -C "$probe" grep -nE 'uses:[[:space:]]*[^[:space:]]*/ceremony/' -- .github \ + | sed -E 's|^([^:]+):([0-9]+):[[:space:]]*-?[[:space:]]*uses:[[:space:]]*(.*)$|\1:\2\t__uses__\t\3|' || true + } | sort >"$actual" + + # every manifest line must be present with its EXACT expected value, and the + # kinds must match — a role swap changes the kind, not just the value. + while IFS=$'\t' read -r loc kind want; do + case "$kind" in + self_ref) have="$(awk -F'\t' -v l="$loc" '$1==l && $2=="self_ref"{print $3}' "$actual")" ;; + internal_repo|consumer_repo) + have="$(awk -F'\t' -v l="$loc" '$1==l && $2=="__repo__"{print $3}' "$actual")" ;; + workflow_caller|action_caller) + have="$(awk -F'\t' -v l="$loc" '$1==l && $2=="__uses__"{print $3}' "$actual")" ;; + esac + [ -n "$have" ] || fail "carrier vanished: $loc ($kind)" + # ONE comparison for every kind: the manifest already carries the complete + # expected value, so owner, path AND sha are checked at once. Checking the + # owner and the sha separately let `…/actions/wrong-one@` + # through (@codex-reviewer-andresmgsl). + [ "$have" = "$want" ] || fail "$loc ($kind): expected '$want', found '$have'" + done <"$manifest" + + # and nothing UNRECOGNISED: every uses:/repository: in the trees must appear + # in the manifest, so an added carrier is a failure rather than a silence. + while IFS=$'\t' read -r loc _ _; do + grep -qF "$loc"$'\t' "$manifest" || fail "carrier not in manifest: $loc" + done <"$actual" + ``` + + **Why a manifest rather than a longer list of assertions.** The carrier set + is a property of the tree at the moment of arming; any list written into + this document is stale the next time a workflow is added. The manifest is + generated from the tree, recorded in the result issue (step 6), and is the + thing a later reader compares against — so "what was armed" is evidence + rather than recollection. + +5. **Invoke the probe by the event it is about**, and record which: a + `workflow_dispatch`, or the real board event under test. A probe that fires + a different event than the one under test proves something else. +6. **The result issue records all of it**: the fork repository, the candidate + code SHA, the armed workflow SHA, every rewritten carrier, the workflow + invoked and the run number. Those are what make the result reproducible; + without the two SHAs distinguished, a later reader cannot tell which tree + answered. +7. **Reset removes the candidate-specific EXECUTABLE state**: the caller stubs, + the probe workflow, and the fork's probe branch — whose tip carries both the + candidate commit and the armed commit on top of it — so the next probe + cannot inherit a pin it did not choose. **Result issues are never deleted.** + They may be closed or relabelled; deleting them would recreate the + expiring-log problem this venue exists to avoid. + +## Who may reset it + +**Operator-owned until ruled otherwise.** #202's task 4 asks who may reset the +venue, and creating the repo is the operator's step, so the access policy is +his to set at the same time (@codex-reviewer-andresmgsl, #202 review). + +Two levels, deliberately separated: + +- **content reset** — removing probe branches, workflows and fixtures; the + ordinary between-probes operation. It does **not** include deleting result + issues, which are the evidence and are immutable once written + (@codex-reviewer-andresmgsl, #202 review); +- **archive / delete / admin** — which is where the drill rule's damage came + from, and which no bot identity should hold here. + +If fleet identities are given push access for content reset, this section +records that; until then, ask. + +## What must never happen here + +No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no +runs attributable to a probe. The venue exists so that the live board does not +have to be the test fixture. + +## The probes this venue owes + +- **ceremony#192** — that the repaired sweep actually lifts a label under the + workflow token, which is the half its acceptance criteria cannot get from + the hermetic contract tests. +- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches` + works on this instance with a valid ref and inputs. Measured so far: + `GET /actions/workflows` 404s and the dispatch route answers 500 rather than + a 4xx, which is not enough to port against. +- A 0.6.0 consumer exercise once ceremony#198 has merged. diff --git a/drills/README.md b/drills/README.md index e25f97f..b75847f 100644 --- a/drills/README.md +++ b/drills/README.md @@ -110,3 +110,11 @@ non-empty even though neither release door reads that file (#217, #237). missing or blank. A waived drill is still a record: the file says WAIVED and why — a maintainer's call, visible and reviewable in the release PR's diff, never a silent skip. + +--- + +**Standing runner probes are not drills.** The disposal rule above — builder +archives, operator deletes — is for the disposable scratch repo a drill runs +in. `heavy-duty/ceremony-runner-probe` is the opposite shape: it stands, and +archiving it is the failure mode that made all three previous drill repos +unavailable. See [docs/RUNNER-PROBES.md](../docs/RUNNER-PROBES.md) (#202).