ceremony/docs/RUNNER-PROBES.md

# Runner probes

**Not a drill.** A drill rehearses the release doors on a disposable repo and
ends. This is the opposite shape: one **standing** repo that exists so that
runner-only facts can be measured on demand, and it is **never archived**.

`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes.
Ruled by the operator as option (A) of ceremony#202 (#5631).

## Why a standing repo, when drills are disposable

Some facts are only true inside Actions, under the token Actions injects, and
no local harness or PAT can reproduce them. The worked example is ceremony#192:

```
DELETE /issues/{n}/labels/{id}   ->  500   under ${{ github.token }} in a workflow
DELETE /issues/{n}/labels/{id}   ->  204   under a maintainer PAT, same call
```

A probe that runs anywhere else passes and proves nothing. Before this venue
existed the answer was "un-archive a drill repo", which was requested three
times in two days across two issues and never became anything — the three
drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived,
and each was minted for one probe and then wanted again.

## The disposal rule above does NOT apply here

The rehearsal section says the builder archives the scratch repo and the
operator deletes it. **That rule is for drills.** Archiving this repo defeats
its entire purpose, and it is the failure mode the three archived drill repos
demonstrate — each was archived correctly, by the rule, and each then had to be
un-archived or replaced.

So: never archive it, never delete it, and if you find it archived, un-archive
it rather than minting a fourth one.

## Standing it up is the operator's step

Bot identities cannot create repositories in `heavy-duty`. Measured
2026-08-05 with a fleet identity holding the `repo` scope:

```
POST /api/v1/orgs/heavy-duty/repos   ->  403  "not allowed to create repository in organization"
POST /api/v1/user/repos              ->  201  (personal namespace only)
```

This is the same shape as the drill delete: a deliberate permission boundary,
not a misconfiguration. Do not retry it, and do not work around it by putting
the venue in a personal namespace — **not because a personal namespace is
proven unable to reach the org's runner** (that was not measured; the probe
repository above was deleted immediately, so nothing about runner or secret
reach was established), but because @andres ruled an **org-owned standing
venue** (#5631). A personally-owned repo is a different thing from the one that
was decided on, and cannot satisfy #202's named acceptance target.

If runner or secret reach turns out to matter, measure it once the venue
exists rather than assuming it here.

## Running a probe

1. Reset the repo to a clean state — the probe's own fixtures only, no
   leftovers from the last one. A probe that inherits state is a probe whose
   result you cannot attribute.
2. **Arm it against the candidate** (below) — two layers, candidate code and
   armed workflow — if the probe is about ceremony's own machinery rather than
   about a bare API call.
3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole
   point of the venue and the one step that cannot be shortcut. A `curl` from a
   laptop with a PAT answers a different question — see the 204/500 split
   above — and a probe run that way is worse than no probe, because it produces
   a confident wrong answer.
4. **The job writes its raw results into an issue in the PROBE repo** —
   `heavy-duty/ceremony-runner-probe` — not into ceremony. Logs age out;
   ceremony#192's run 701 survived only because the job wrote its findings
   into an issue it created.
5. **A human then records the probe issue's URL and the Actions run number on
   the ceremony issue the probe serves.** That hop is deliberate and is the
   whole of the boundary: the probe workflow holds no credential and no code
   path that can write to `heavy-duty/ceremony`, so "the probe reports its
   findings" and "the probe cannot touch the live board" stay compatible
   rather than contradicting each other (@codex-reviewer-andresmgsl, #202
   review).

## Arming a candidate ref

A probe that exercises ceremony's own machinery needs the candidate tree
reachable from a `uses:` line. This is the fork-ref shape `drills/README.md`
step 2 points at, written out — and it has **two layers**, which is the part
that is easy to get wrong and impossible to fix afterwards.

**Why two.** The candidate's own workflows contain
`repository: heavy-duty/ceremony` beside `ref: ${{ env.CEREMONY_SELF_REF }}`,
so they must be rewritten to point at the fork and at the candidate. But
rewriting them **creates a new commit**, and a commit cannot contain its own
object ID. A single-layer arming is therefore self-referential: pin the callers
to the pre-rewrite SHA and they load the *unarmed* workflows; pin them to the
post-rewrite one and you are asking a commit to embed itself
(@codex-reviewer-andresmgsl, #202 review).

So:

| layer | what it is | what it carries |
|---|---|---|
| **candidate code SHA** | the immutable tree under test | `actions/`, `lib/` — untouched |
| **armed workflow SHA** | a small child commit on top of it | workflows rewritten to the fork + `CEREMONY_SELF_REF` = the candidate code SHA |

### The procedure

1. **Push the candidate tree** to a fork under the identity that will run the
   probe — one branch, `<identity>/ceremony@probe-<issue>` — and record its
   SHA. Steps 1 and 2 advance the tip of that **same** branch; there are two
   commits, not two branches. That is
   the **candidate code SHA**. Never create a branch on
   `heavy-duty/ceremony` named like a tag: it shadows that tag for every
   consumer until somebody remembers to delete it.
2. **Commit the arming on top of it.** In that same fork branch, rewrite every
   workflow carrier — `repository:` → `<identity>/ceremony`, and both
   `CEREMONY_SELF_REF` values → the **candidate code SHA** from step 1. Record
   the resulting SHA. That is the **armed workflow SHA**.
3. **Pin the probe repo's callers by layer**, because they are not the same
   thing:
   - composite-action callers →
     `<identity>/ceremony/actions/<name>@<candidate-code-sha>`;
   - reusable-workflow callers →
     `<identity>/ceremony/.github/workflows/<file>@<armed-workflow-sha>`, since
     that is the only revision whose inner checkout is rewritten.
4. **Gate the arming POSITIVELY, and against the tree each check is about.**
   Absence of the old literal proves nothing: a negative grep stays green if
   `CEREMONY_SELF_REF` names a tag or the *armed* SHA, if a carrier went to the
   wrong fork, or if a carrier vanished instead of being rewritten. So assert
   what each carrier **is** — and enumerate them, because the set is a property
   of the current tree rather than a number to remember
   (@codex-reviewer-andresmgsl, #202 review).

   **The carriers on `main` today.** Re-derive this list each time rather than
   trusting it; workflows come and go:

   ```sh
   git grep -n 'CEREMONY_SELF_REF:' -- .github/workflows
   git grep -n 'repository:'        -- .github/workflows
   ```

   | carrier | on current `main` | armed value |
   |---|---|---|
   | `CEREMONY_SELF_REF` | `labels-sweep.yml`, `labels.yml`, `release.yml` | the **candidate code** SHA |
   | internal self-checkout | `labels-sweep.yml`, `labels.yml`, `release.yml` ×2 | `<identity>/ceremony` |
   | **consumer** checkout | `labels-sweep.yml`, `labels.yml`, `release-exercise.yml` | **unchanged** — `${{ github.repository }}` |

   That third row is the one an over-broad rewrite destroys. Those checkouts
   fetch the **caller's** repository; pointing them at the fork changes what
   the probe exercises into something else entirely.

   ```sh
   #!/usr/bin/env bash
   # check-arming <armed-checkout> <probe-checkout> <fork> <code-sha> <armed-sha>
   set -euo pipefail
   armed="$1"; probe="$2"; fork="$3"; code_sha="$4"; armed_sha="$5"
   fail() { echo "arming incomplete: $*" >&2; exit 1; }

   # (a) every self-ref carrier names the CANDIDATE CODE sha — enumerated, so a
   #     carrier that vanished is a failure rather than one fewer loop pass.
   mapfile -t refs < <(git -C "$armed" grep -hoP '(?<=CEREMONY_SELF_REF: ")[^"]+' \
     -- .github/workflows)
   [ "${#refs[@]}" -gt 0 ] || fail "no CEREMONY_SELF_REF carriers found"
   for v in "${refs[@]}"; do [ "$v" = "$code_sha" ] || fail "CEREMONY_SELF_REF=$v"; done

   # (b) internal checkouts point at the fork; consumer checkouts stay dynamic.
   mapfile -t repos < <(git -C "$armed" grep -hoP '(?<=repository: ).*' \
     -- .github/workflows)
   for v in "${repos[@]}"; do
     case "$v" in
       '${{ github.repository }}') ;;          # the caller's repo — must NOT change
       "$fork") ;;                             # an internal self-checkout, armed
       *) fail "repository: $v" ;;
     esac
   done

   # (c) in the PROBE repo: workflow callers pin the armed sha, action callers
   #     the code sha.
   mapfile -t uses < <(git -C "$probe" grep -hoP '(?<=uses: ).*' -- .github)
   for u in "${uses[@]}"; do
     case "$u" in
       */.github/workflows/*@*) [ "${u##*@}" = "$armed_sha" ] || fail "workflow caller $u" ;;
       */actions/*@*)           [ "${u##*@}" = "$code_sha" ]  || fail "action caller $u" ;;
     esac
   done

   # (d) the negative check, as a cheap extra rather than as the proof.
   ! git -C "$armed" grep -qE '(uses:|repository:)[[:space:]]*heavy-duty/ceremony' \
     -- .github || fail "canonical coordinate still present"
   ```

   **`mapfile`, not a pipeline.** `git grep | while … fail` runs the loop in a
   subshell, so `fail` exits *that* subshell and the gate continues. Collect
   first, validate after — and run under `set -euo pipefail`, which the snippet
   declares rather than assumes.

5. **Invoke the probe by the event it is about**, and record which: a
   `workflow_dispatch`, or the real board event under test. A probe that fires
   a different event than the one under test proves something else.
6. **The result issue records all of it**: the fork repository, the candidate
   code SHA, the armed workflow SHA, every rewritten carrier, the workflow
   invoked and the run number. Those are what make the result reproducible;
   without the two SHAs distinguished, a later reader cannot tell which tree
   answered.
7. **Reset removes the candidate-specific EXECUTABLE state**: the caller stubs,
   the probe workflow, and the fork's probe branch — whose tip carries both the
   candidate commit and the armed commit on top of it — so the next probe
   cannot inherit a pin it did not choose. **Result issues are never deleted.**
   They may be closed or relabelled; deleting them would recreate the
   expiring-log problem this venue exists to avoid.

## Who may reset it

**Operator-owned until ruled otherwise.** #202's task 4 asks who may reset the
venue, and creating the repo is the operator's step, so the access policy is
his to set at the same time (@codex-reviewer-andresmgsl, #202 review).

Two levels, deliberately separated:

- **content reset** — removing probe branches, workflows and fixtures; the
  ordinary between-probes operation. It does **not** include deleting result
  issues, which are the evidence and are immutable once written
  (@codex-reviewer-andresmgsl, #202 review);
- **archive / delete / admin** — which is where the drill rule's damage came
  from, and which no bot identity should hold here.

If fleet identities are given push access for content reset, this section
records that; until then, ask.

## What must never happen here

No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no
runs attributable to a probe. The venue exists so that the live board does not
have to be the test fixture.

## The probes this venue owes

- **ceremony#192** — that the repaired sweep actually lifts a label under the
  workflow token, which is the half its acceptance criteria cannot get from
  the hermetic contract tests.
- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches`
  works on this instance with a valid ref and inputs. Measured so far:
  `GET /actions/workflows` 404s and the dispatch route answers 500 rather than
  a 4xx, which is not enough to port against.
- A 0.6.0 consumer exercise once ceremony#198 has merged.