ceremony/docs/RUNNER-PROBES.md

# Runner probes

**Not a drill.** A drill rehearses the release doors on a disposable repo and
ends. This is the opposite shape: one **standing** repo that exists so that
runner-only facts can be measured on demand, and it is **never archived**.

`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes.
Ruled by the operator as option (A) of ceremony#202 (#5631).

## Why a standing repo, when drills are disposable

Some facts are only true inside Actions, under the token Actions injects, and
no local harness or PAT can reproduce them. The worked example is ceremony#192:

```
DELETE /issues/{n}/labels/{id}   ->  500   under ${{ github.token }} in a workflow
DELETE /issues/{n}/labels/{id}   ->  204   under a maintainer PAT, same call
```

A probe that runs anywhere else passes and proves nothing. Before this venue
existed the answer was "un-archive a drill repo", which was requested three
times in two days across two issues and never became anything — the three
drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived,
and each was minted for one probe and then wanted again.

## The disposal rule above does NOT apply here

The rehearsal section says the builder archives the scratch repo and the
operator deletes it. **That rule is for drills.** Archiving this repo defeats
its entire purpose, and it is the failure mode the three archived drill repos
demonstrate — each was archived correctly, by the rule, and each then had to be
un-archived or replaced.

So: never archive it, never delete it, and if you find it archived, un-archive
it rather than minting a fourth one.

## Standing it up is the operator's step

Bot identities cannot create repositories in `heavy-duty`. Measured
2026-08-05 with a fleet identity holding the `repo` scope:

```
POST /api/v1/orgs/heavy-duty/repos   ->  403  "not allowed to create repository in organization"
POST /api/v1/user/repos              ->  201  (personal namespace only)
```

This is the same shape as the drill delete: a deliberate permission boundary,
not a misconfiguration, and the same rule applies — do not retry it, and do not
work around it by putting the venue in a personal namespace, where the org's
runner and secrets do not reach. Ask the operator.

## Running a probe

1. Reset the repo to a clean state — the probe's own fixtures only, no
   leftovers from the last one. A probe that inherits state is a probe whose
   result you cannot attribute.
2. **Arm it against the candidate ref** (below), if the probe is about
   ceremony's own code rather than about a bare API call.
3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole
   point of the venue and the one step that cannot be shortcut. A `curl` from a
   laptop with a PAT answers a different question — see the 204/500 split
   above — and a probe run that way is worse than no probe, because it produces
   a confident wrong answer.
4. **The job writes its raw results into an issue in the PROBE repo** —
   `heavy-duty/ceremony-runner-probe` — not into ceremony. Logs age out;
   ceremony#192's run 701 survived only because the job wrote its findings
   into an issue it created.
5. **A human then records the probe issue's URL and the Actions run number on
   the ceremony issue the probe serves.** That hop is deliberate and is the
   whole of the boundary: the probe workflow holds no credential and no code
   path that can write to `heavy-duty/ceremony`, so "the probe reports its
   findings" and "the probe cannot touch the live board" stay compatible
   rather than contradicting each other (@codex-reviewer-andresmgsl, #202
   review).

## Arming a candidate ref

A probe that exercises ceremony's own machinery needs the candidate tree
reachable from a `uses:` line. The shape is the drill rehearsal's, reused
rather than reinvented (`drills/README.md` step 2):

1. **The candidate is a commit SHA, on a fork ref.** Push the candidate tree to
   a fork under the identity running the probe —
   `<identity>/ceremony@probe-<issue>` — and take its canonical SHA. Never
   create a branch on `heavy-duty/ceremony` named like a tag: it shadows that
   tag for every consumer until somebody remembers to delete it.
2. **Install `docs/CONSUMERS.md`'s caller stubs** in the probe repo, pinned to
   that SHA — every `uses:` reference, not some of them.
3. **Rewrite `CEREMONY_SELF_REF` in the candidate itself** to the same
   canonical SHA. A candidate carries the tag it has not created yet, so the
   self-consumption checkout cannot resolve until it is rewritten. Both
   carriers — `labels.yml` and `release.yml` — take the same value.
4. **Invoke the probe by the event it is about**, and record which: a
   `workflow_dispatch` of the caller, or the real board event the probe is
   testing. A probe that fires a different event than the one under test
   proves something else.
5. **Record the fork ref, the rewritten pin, the workflow invoked and the run
   number** in the probe repo's result issue. Those four are what make the
   result reproducible; without the pin especially, a later reader cannot tell
   which tree answered.
6. **Reset removes the candidate-specific fixtures**: the caller stubs, the
   probe workflow and the result branch, so the next probe does not inherit a
   pin it did not choose. The result *issue* stays — it is the evidence.

## Who may reset it

**Operator-owned until ruled otherwise.** #202's task 4 asks who may reset the
venue, and creating the repo is the operator's step, so the access policy is
his to set at the same time (@codex-reviewer-andresmgsl, #202 review).

Two levels, deliberately separated:

- **content reset** — deleting probe branches, workflows, fixtures and stale
  result issues; the ordinary between-probes operation;
- **archive / delete / admin** — which is where the drill rule's damage came
  from, and which no bot identity should hold here.

If fleet identities are given push access for content reset, this section
records that; until then, ask.

## What must never happen here

No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no
runs attributable to a probe. The venue exists so that the live board does not
have to be the test fixture.

## The probes this venue owes

- **ceremony#192** — that the repaired sweep actually lifts a label under the
  workflow token, which is the half its acceptance criteria cannot get from
  the hermetic contract tests.
- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches`
  works on this instance with a valid ref and inputs. Measured so far:
  `GET /actions/workflows` 404s and the dispatch route answers 500 rather than
  a 4xx, which is not enough to port against.
- A 0.6.0 consumer exercise once ceremony#198 has merged.