Merge pull request 'drills/README.md — the standing runner-probe venue, and why the drill disposal rule does not apply to it (#202)' (#207) from build/202-runner-probe-venue into main

Reviewed-on: #207
Reviewed-by: codex-reviewer-andresmgsl <andres+2@heavyduty.builders>
Reviewed-by: glm-reviewer-andresmgsl <andres+5@heavyduty.builders>
Reviewed-by: kimi-reviewer-andresmgsl <andres+4@heavyduty.builders>

changelog.d/202.md

@@ -0,0 +1,64 @@
+### Added
+
+- `docs/RUNNER-PROBES.md` documents the standing runner-probe venue,
+  `heavy-duty/ceremony-runner-probe` — the place runner-only facts are measured
+  on demand, ruled as option A by the operator (#202).
+
+- `drills/README.md` cross-links it beside the disposal rule, so the exception
+  is visible where the dangerous habit lives (#202).
+
+- The runbook states that the drill disposal rule does **not** apply to it.
+  Archiving it defeats its purpose, and that is exactly how the three existing
+  drill repos each became unavailable (#202).
+
+- It records that a probe must run as an Actions job under the workflow token:
+  the same call answers 500 there and 204 under a PAT, so a probe run any other
+  way produces a confident wrong answer (#202).
+
+- Creating the repo is recorded as the operator's step, measured rather than
+  assumed: a fleet identity gets 403 on org repo creation and 201 in its own
+  namespace (#202).
+
+- It carries an executable two-layer arming procedure: an immutable candidate
+  code SHA and an armed workflow commit on top of it. A single layer is
+  self-referential — rewriting a workflow makes a new commit, and a commit
+  cannot contain its own object ID (#202).
+
+- Callers are pinned by layer: composite actions to the candidate code SHA,
+  reusable workflows to the armed SHA, which is the only revision whose inner
+  checkout points at the fork (#202).
+
+- The arming gate asserts what each carrier IS, not only that the old literal
+  is gone: every `repository:` equals the fork, every `CEREMONY_SELF_REF` value
+  equal the candidate code SHA, and callers match the layer they belong to
+  (#202).
+
+- It enumerates the carriers from the tree rather than encoding a count, and
+  distinguishes ceremony's internal self-checkouts from the consumer checkouts
+  that must stay `${{ github.repository }}` (#202).
+
+- Both published snippets are ShellCheck-clean when extracted and linted
+  directly, not merely as part of the repository sweep (#202).
+
+- The checker validates the MANIFEST against the target it was given, so a
+  manifest that describes a wrong arming consistently — wrong fork, or the
+  armed SHA where the candidate belongs — refuses instead of matching a tree
+  rewritten to the same wrong value (#202).
+
+- The manifest is generated from the PRE-arming tree, which is the only order
+  that enumerates the carriers that must change (#202).
+
+- Both published snippets were driven against a constructed candidate/probe
+  pair: deletion, both role swaps, wrong owner, wrong
+  SHA, wrong path, a deleted caller class and an extra carrier all refuse, and
+  the armed control passes (#202).
+
+- The manifest records complete caller coordinates, so a path swapped under the
+  right owner and SHA is caught (#202).
+
+- Generator and checker share one domain — ceremony callers — so a third-party
+  `actions/checkout` is neither manifested nor reported as unrecognised (#202).
+
+- Probe results are written to an issue in the probe repo and carried to the
+  ceremony issue by a human, so the probe holds no path that can write to the
+  live board (#202).

docs/RUNNER-PROBES.md

@@ -0,0 +1,345 @@
+# Runner probes
+
+**Not a drill.** A drill rehearses the release doors on a disposable repo and
+ends. This is the opposite shape: one **standing** repo that exists so that
+runner-only facts can be measured on demand, and it is **never archived**.
+
+`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes.
+Ruled by the operator as option (A) of ceremony#202 (#5631).
+
+## Why a standing repo, when drills are disposable
+
+Some facts are only true inside Actions, under the token Actions injects, and
+no local harness or PAT can reproduce them. The worked example is ceremony#192:
+
+```
+DELETE /issues/{n}/labels/{id}   ->  500   under ${{ github.token }} in a workflow
+DELETE /issues/{n}/labels/{id}   ->  204   under a maintainer PAT, same call
+```
+
+A probe that runs anywhere else passes and proves nothing. Before this venue
+existed the answer was "un-archive a drill repo", which was requested three
+times in two days across two issues and never became anything — the three
+drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived,
+and each was minted for one probe and then wanted again.
+
+## The disposal rule above does NOT apply here
+
+The rehearsal section says the builder archives the scratch repo and the
+operator deletes it. **That rule is for drills.** Archiving this repo defeats
+its entire purpose, and it is the failure mode the three archived drill repos
+demonstrate — each was archived correctly, by the rule, and each then had to be
+un-archived or replaced.
+
+So: never archive it, never delete it, and if you find it archived, un-archive
+it rather than minting a fourth one.
+
+## Standing it up is the operator's step
+
+Bot identities cannot create repositories in `heavy-duty`. Measured
+2026-08-05 with a fleet identity holding the `repo` scope:
+
+```
+POST /api/v1/orgs/heavy-duty/repos   ->  403  "not allowed to create repository in organization"
+POST /api/v1/user/repos              ->  201  (personal namespace only)
+```
+
+This is the same shape as the drill delete: a deliberate permission boundary,
+not a misconfiguration. Do not retry it, and do not work around it by putting
+the venue in a personal namespace — **not because a personal namespace is
+proven unable to reach the org's runner** (that was not measured; the probe
+repository above was deleted immediately, so nothing about runner or secret
+reach was established), but because @andres ruled an **org-owned standing
+venue** (#5631). A personally-owned repo is a different thing from the one that
+was decided on, and cannot satisfy #202's named acceptance target.
+
+If runner or secret reach turns out to matter, measure it once the venue
+exists rather than assuming it here.
+
+## Running a probe
+
+1. Reset the repo to a clean state — the probe's own fixtures only, no
+   leftovers from the last one. A probe that inherits state is a probe whose
+   result you cannot attribute.
+2. **Arm it against the candidate** (below) — two layers, candidate code and
+   armed workflow — if the probe is about ceremony's own machinery rather than
+   about a bare API call.
+3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole
+   point of the venue and the one step that cannot be shortcut. A `curl` from a
+   laptop with a PAT answers a different question — see the 204/500 split
+   above — and a probe run that way is worse than no probe, because it produces
+   a confident wrong answer.
+4. **The job writes its raw results into an issue in the PROBE repo** —
+   `heavy-duty/ceremony-runner-probe` — not into ceremony. Logs age out;
+   ceremony#192's run 701 survived only because the job wrote its findings
+   into an issue it created.
+5. **A human then records the probe issue's URL and the Actions run number on
+   the ceremony issue the probe serves.** That hop is deliberate and is the
+   whole of the boundary: the probe workflow holds no credential and no code
+   path that can write to `heavy-duty/ceremony`, so "the probe reports its
+   findings" and "the probe cannot touch the live board" stay compatible
+   rather than contradicting each other (@codex-reviewer-andresmgsl, #202
+   review).
+
+## Arming a candidate ref
+
+A probe that exercises ceremony's own machinery needs the candidate tree
+reachable from a `uses:` line. This is the fork-ref shape `drills/README.md`
+step 2 points at, written out — and it has **two layers**, which is the part
+that is easy to get wrong and impossible to fix afterwards.
+
+**Why two.** The candidate's own workflows contain
+`repository: heavy-duty/ceremony` beside `ref: ${{ env.CEREMONY_SELF_REF }}`,
+so they must be rewritten to point at the fork and at the candidate. But
+rewriting them **creates a new commit**, and a commit cannot contain its own
+object ID. A single-layer arming is therefore self-referential: pin the callers
+to the pre-rewrite SHA and they load the *unarmed* workflows; pin them to the
+post-rewrite one and you are asking a commit to embed itself
+(@codex-reviewer-andresmgsl, #202 review).
+
+So:
+
+| layer | what it is | what it carries |
+|---|---|---|
+| **candidate code SHA** | the immutable tree under test | `actions/`, `lib/` — untouched |
+| **armed workflow SHA** | a small child commit on top of it | workflows rewritten to the fork + `CEREMONY_SELF_REF` = the candidate code SHA |
+
+### The procedure
+
+1. **Push the candidate tree** to a fork under the identity that will run the
+   probe — one branch, `<identity>/ceremony@probe-<issue>` — and record its
+   SHA. Steps 1 and 2 advance the tip of that **same** branch; there are two
+   commits, not two branches. That is
+   the **candidate code SHA**. Never create a branch on
+   `heavy-duty/ceremony` named like a tag: it shadows that tag for every
+   consumer until somebody remembers to delete it.
+2. **Write the manifest FIRST, from the pre-arming tree, then commit the
+   arming.** The manifest enumerates the carriers *that must change*, so it is
+   generated before they do — running it afterwards would enumerate
+   already-rewritten rows and lose the canonical internal-checkout ones
+   entirely (@codex-reviewer-andresmgsl, #202 review). In that same
+   fork branch rewrite, for **every** carrier the manifest below enumerates:
+   ceremony's own internal `repository:` checkouts → `<identity>/ceremony`, and
+   **every** `CEREMONY_SELF_REF` value → the **candidate code SHA** from step 1.
+   There were three self-ref carriers on `main` at the time of writing and the
+   count is not a constant — derive it, do not remember it
+   (@glm-reviewer-andresmgsl, @codex-reviewer-andresmgsl, #202 review). The
+   **consumer** checkouts (`${{ github.repository }}`) are left alone. Record
+   the resulting SHA: that is the **armed workflow SHA**.
+3. **Pin the probe repo's callers by layer**, because they are not the same
+   thing:
+   - composite-action callers →
+     `<identity>/ceremony/actions/<name>@<candidate-code-sha>`;
+   - reusable-workflow callers →
+     `<identity>/ceremony/.github/workflows/<file>@<armed-workflow-sha>`, since
+     that is the only revision whose inner checkout is rewritten.
+4. **Gate the arming against a MANIFEST, byte for byte.** Every weaker shape
+   has a hole, and each of these was found in a published draft of this file
+   (@codex-reviewer-andresmgsl, #202 review):
+
+   | weaker check | what slips through |
+   |---|---|
+   | "the old literal is absent" | a carrier rewritten to the wrong fork, or to the *armed* SHA |
+   | "every extracted value equals X" | a carrier that **vanished** — nothing to compare |
+   | "each value is one of {fork, dynamic}" | a **role swap**: an internal checkout made dynamic, a consumer checkout pointed at the fork |
+   | "the SHA suffix matches" | `wrong-owner/ceremony/actions/foo@<right-sha>` |
+   | "known callers match" | an **unrecognised** caller, or none at all |
+   | "the owner and the sha are right for the kind" | a **layer swap**: `…/actions/x@<armed>` labelled a workflow caller satisfies both |
+
+   So the arming step **writes a manifest** — one line per carrier, `path`,
+   `kind`, `full expected value` — and the gate compares the tree's actual
+   carriers against it as a set. A deletion, a role swap, a wrong fork, a wrong
+   SHA, an extra carrier and a missing caller are then all the same kind of
+   failure: the sets differ.
+
+   **Generate it while arming**, from the tree you are arming, so the manifest
+   cannot drift from the repository:
+
+   ```sh
+   #!/usr/bin/env bash
+   # write-manifest <candidate-checkout> <probe-checkout> <fork> <code-sha> <armed-sha>
+   #
+   # Run against the PRE-ARMING tree and the UNPINNED probe: this records what
+   # each carrier must BECOME, so it has to see them before they change.
+   #
+   # `|| true` on every extraction, for the same reason the checker needs it:
+   # git grep exits 1 on no-match and `set -e` would abort BEFORE the manifest
+   # is written — silently, which is how the first version of this generator
+   # produced no file and no diagnostic when a probe exercised only one layer
+   # (@codex-reviewer-andresmgsl). A probe need not use both.
+   set -euo pipefail
+   candidate="$1"; probe="$2"; fork="$3"; code_sha="$4"; armed_sha="$5"
+   # shellcheck disable=SC2016 # `${{ github.repository }}` is literal YAML, not a shell expansion
+   {
+     git -C "$candidate" grep -n 'CEREMONY_SELF_REF:' -- .github/workflows \
+       | cut -d: -f1,2 | sed "s|$|\tself_ref\t$code_sha|" || true
+     git -C "$candidate" grep -n 'repository: heavy-duty/ceremony' -- .github/workflows \
+       | cut -d: -f1,2 | sed "s|$|\tinternal_repo\t$fork|" || true
+     git -C "$candidate" grep -n 'repository: ${{ github.repository }}' -- .github/workflows \
+       | cut -d: -f1,2 | sed 's|$|\tconsumer_repo\t${{ github.repository }}|' || true
+     # Callers record the COMPLETE expected coordinate, not just the sha: the
+     # path is as rewritable as the owner, and a manifest that stores only the
+     # suffix cannot notice `…/actions/wrong-one@<right-sha>`.
+     git -C "$probe" grep -nE 'uses:[[:space:]]*[^[:space:]]*/ceremony/\.github/workflows/' -- .github \
+       | sed -E "s|^([^:]+):([0-9]+):.*/ceremony/(\.github/workflows/[^@[:space:]]+)@.*|\\1:\\2\\tworkflow_caller\\t$fork/\\3@$armed_sha|" || true
+     git -C "$probe" grep -nE 'uses:[[:space:]]*[^[:space:]]*/ceremony/actions/' -- .github \
+       | sed -E "s|^([^:]+):([0-9]+):.*/ceremony/(actions/[^@[:space:]]+)@.*|\\1:\\2\\taction_caller\\t$fork/\\3@$code_sha|" || true
+   } | sort >manifest.tsv
+
+   # Zero ceremony callers is a refusal by name; one layer only is fine.
+   callers="$(grep -cE '(workflow|action)_caller' manifest.tsv || true)"
+   [ "$callers" -gt 0 ] || { echo "manifest: no ceremony callers found in $probe" >&2; exit 1; }
+   ```
+
+   Then arm — rewrite and commit — and check the result against it:
+
+   ```sh
+   #!/usr/bin/env bash
+   # check-arming <armed-checkout> <probe-checkout> <fork> <code-sha> <armed-sha> <manifest.tsv>
+   set -euo pipefail
+   armed="$1"; probe="$2"; fork="$3"; code_sha="$4"; armed_sha="$5"; manifest="$6"
+   fail() { echo "arming incomplete: $*" >&2; exit 1; }
+
+   # `|| true` on every extraction: git grep exits 1 when nothing matches, and
+   # under `set -e` that would kill this script BEFORE the comparison — so a
+   # carrier class that vanished ENTIRELY produced silence instead of a
+   # refusal. Silence is the worst of the three outcomes; the comparison below
+   # is what must report it.
+   [ "$(grep -cE '(workflow|action)_caller' "$manifest" || true)" -gt 0 ] \
+     || fail "manifest names no ceremony callers — it cannot prove an arming"
+
+   # THE MANIFEST ITSELF IS CHECKED AGAINST THE TARGET, not trusted. Comparing
+   # only tree-vs-manifest proves consistency, and a manifest generated with the
+   # armed SHA where the candidate SHA belonged — or with the wrong fork —
+   # describes a WRONG arming perfectly. The tree would then match it and the
+   # gate would pass (@codex-reviewer-andresmgsl, #202 review).
+   # shellcheck disable=SC2016 # `${{ github.repository }}` below is literal YAML
+   while IFS=$'\t' read -r loc kind want; do
+     case "$kind" in
+       self_ref)        [ "$want" = "$code_sha" ] || fail "manifest $loc: self_ref should be the CANDIDATE sha" ;;
+       internal_repo)   [ "$want" = "$fork" ]     || fail "manifest $loc: internal repo should be $fork" ;;
+       consumer_repo)   [ "$want" = '${{ github.repository }}' ] \
+                          || fail "manifest $loc: consumer checkout must stay dynamic" ;;
+       # THE KIND MUST BIND TO THE PATH CLASS, not only to the owner and the
+       # sha. The path class is what SAYS which layer a caller is, so checking
+       # the sha against the kind while letting the kind float free accepts a
+       # consistent layer swap — `…/actions/x@<armed>` declared workflow_caller
+       # passes every owner and sha test (@codex-reviewer-andresmgsl, #202
+       # review). Decompose once, then let the kind fix BOTH coordinates.
+       workflow_caller|action_caller)
+                        owner="${want%%/ceremony/*}"; rest="${want#*/ceremony/}"
+                        path="${rest%@*}"; sha="${want##*@}"
+                        [ "$owner/ceremony" = "$fork" ] \
+                          || fail "manifest $loc: caller owner should be $fork"
+                        case "$kind" in
+                          workflow_caller)
+                            case "$path" in .github/workflows/?*) : ;;
+                              *) fail "manifest $loc: workflow_caller must resolve at .github/workflows/<file>, not '$path'" ;;
+                            esac
+                            [ "$sha" = "$armed_sha" ] || fail "manifest $loc: workflow caller should be the ARMED sha" ;;
+                          action_caller)
+                            case "$path" in actions/?*) : ;;
+                              *) fail "manifest $loc: action_caller must resolve at actions/<name>, not '$path'" ;;
+                            esac
+                            [ "$sha" = "$code_sha" ] || fail "manifest $loc: action caller should be the CANDIDATE sha" ;;
+                        esac ;;
+       *)               fail "manifest $loc: unknown kind '$kind'" ;;
+     esac
+   done <"$manifest"
+
+   actual="$(mktemp)"; trap 'rm -f "$actual"' EXIT
+   {
+     git -C "$armed" grep -nP '(?<=CEREMONY_SELF_REF: ")[^"]+' -- .github/workflows \
+       | sed -E 's/^([^:]+):([0-9]+):.*CEREMONY_SELF_REF: "([^"]*)".*/\1:\2\tself_ref\t\3/' || true
+     git -C "$armed" grep -nE 'repository: .+' -- .github/workflows \
+       | sed -E 's|^([^:]+):([0-9]+):[[:space:]]*repository:[[:space:]]*(.*)$|\1:\2\t__repo__\t\3|' || true
+     # Only CEREMONY callers, matching the generator's domain exactly — a
+     # third-party `actions/checkout` is not this gate's business, and
+     # extracting it here while the generator ignores it made every probe fail
+     # as an "unrecognised carrier" (@codex-reviewer-andresmgsl). A wrong OWNER
+     # is still caught: `wrong-owner/ceremony/...` matches this pattern.
+     git -C "$probe" grep -nE 'uses:[[:space:]]*[^[:space:]]*/ceremony/' -- .github \
+       | sed -E 's|^([^:]+):([0-9]+):[[:space:]]*-?[[:space:]]*uses:[[:space:]]*(.*)$|\1:\2\t__uses__\t\3|' || true
+   } | sort >"$actual"
+
+   # every manifest line must be present with its EXACT expected value, and the
+   # kinds must match — a role swap changes the kind, not just the value.
+   while IFS=$'\t' read -r loc kind want; do
+     case "$kind" in
+       self_ref)        have="$(awk -F'\t' -v l="$loc" '$1==l && $2=="self_ref"{print $3}' "$actual")" ;;
+       internal_repo|consumer_repo)
+                        have="$(awk -F'\t' -v l="$loc" '$1==l && $2=="__repo__"{print $3}' "$actual")" ;;
+       workflow_caller|action_caller)
+                        have="$(awk -F'\t' -v l="$loc" '$1==l && $2=="__uses__"{print $3}' "$actual")" ;;
+     esac
+     [ -n "$have" ] || fail "carrier vanished: $loc ($kind)"
+     # ONE comparison for every kind: the manifest already carries the complete
+     # expected value, so owner, path AND sha are checked at once. Checking the
+     # owner and the sha separately let `…/actions/wrong-one@<right-sha>`
+     # through (@codex-reviewer-andresmgsl).
+     [ "$have" = "$want" ] || fail "$loc ($kind): expected '$want', found '$have'"
+   done <"$manifest"
+
+   # and nothing UNRECOGNISED: every uses:/repository: in the trees must appear
+   # in the manifest, so an added carrier is a failure rather than a silence.
+   while IFS=$'\t' read -r loc _ _; do
+     grep -qF "$loc"$'\t' "$manifest" || fail "carrier not in manifest: $loc"
+   done <"$actual"
+   ```
+
+   **Why a manifest rather than a longer list of assertions.** The carrier set
+   is a property of the tree at the moment of arming; any list written into
+   this document is stale the next time a workflow is added. The manifest is
+   generated from the tree, recorded in the result issue (step 6), and is the
+   thing a later reader compares against — so "what was armed" is evidence
+   rather than recollection.
+
+5. **Invoke the probe by the event it is about**, and record which: a
+   `workflow_dispatch`, or the real board event under test. A probe that fires
+   a different event than the one under test proves something else.
+6. **The result issue records all of it**: the fork repository, the candidate
+   code SHA, the armed workflow SHA, every rewritten carrier, the workflow
+   invoked and the run number. Those are what make the result reproducible;
+   without the two SHAs distinguished, a later reader cannot tell which tree
+   answered.
+7. **Reset removes the candidate-specific EXECUTABLE state**: the caller stubs,
+   the probe workflow, and the fork's probe branch — whose tip carries both the
+   candidate commit and the armed commit on top of it — so the next probe
+   cannot inherit a pin it did not choose. **Result issues are never deleted.**
+   They may be closed or relabelled; deleting them would recreate the
+   expiring-log problem this venue exists to avoid.
+
+## Who may reset it
+
+**Operator-owned until ruled otherwise.** #202's task 4 asks who may reset the
+venue, and creating the repo is the operator's step, so the access policy is
+his to set at the same time (@codex-reviewer-andresmgsl, #202 review).
+
+Two levels, deliberately separated:
+
+- **content reset** — removing probe branches, workflows and fixtures; the
+  ordinary between-probes operation. It does **not** include deleting result
+  issues, which are the evidence and are immutable once written
+  (@codex-reviewer-andresmgsl, #202 review);
+- **archive / delete / admin** — which is where the drill rule's damage came
+  from, and which no bot identity should hold here.
+
+If fleet identities are given push access for content reset, this section
+records that; until then, ask.
+
+## What must never happen here
+
+No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no
+runs attributable to a probe. The venue exists so that the live board does not
+have to be the test fixture.
+
+## The probes this venue owes
+
+- **ceremony#192** — that the repaired sweep actually lifts a label under the
+  workflow token, which is the half its acceptance criteria cannot get from
+  the hermetic contract tests.
+- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches`
+  works on this instance with a valid ref and inputs. Measured so far:
+  `GET /actions/workflows` 404s and the dispatch route answers 500 rather than
+  a 4xx, which is not enough to port against.
+- A 0.6.0 consumer exercise once ceremony#198 has merged.

drills/README.md

@@ -110,3 +110,11 @@ non-empty even though neither release door reads that file (#217, #237).
 missing or blank. A waived drill is still a record: the file says WAIVED and
 why — a maintainer's call, visible and reviewable in the release PR's diff,
 never a silent skip.
+
+---
+
+**Standing runner probes are not drills.** The disposal rule above — builder
+archives, operator deletes — is for the disposable scratch repo a drill runs
+in. `heavy-duty/ceremony-runner-probe` is the opposite shape: it stands, and
+archiving it is the failure mode that made all three previous drill repos
+unavailable. See [docs/RUNNER-PROBES.md](../docs/RUNNER-PROBES.md) (#202).