docs(drills): the standing runner-probe venue, and why it is not a drill (#202)
All checks were successful
CI / test (pull_request) Successful in 1m30s
CI / release-exercise (pull_request) Successful in 10s
CI / self-guards (pull_request) Successful in 7s
CI / action-exercise (pull_request) Successful in 6s
CI / docs-sync-exercise (pull_request) Successful in 6s
labels / labels (pull_request) Successful in 8s

@andres ruled option A (#5631): one standing never-archived repo. This is the
runbook half.

The distinction the document exists to make: a drill is disposable by design
and ends with the builder archiving it. This venue is the opposite — it exists
so that runner-only facts can be measured on demand, and archiving it defeats
the purpose. That is not hypothetical: all three drill repos were archived
correctly, by the rule, and each then had to be un-archived or replaced. The
request came three times in two days across #192 and #198 and never became
anything.

What the runbook pins, all of it measured rather than asserted:

  * a probe MUST run as an Actions job under ${{ github.token }} — the same
    DELETE answers 500 there and 204 under a PAT, so a probe run any other way
    produces a confident wrong answer;
  * probe results are written into the forge, not left in a job log, because
    logs age out and #192's run 701 survived only because it wrote into an
    issue;
  * no probe touches ceremony's own board — the venue exists so the live board
    is not the fixture;
  * the three probes it already owes (#192's live label lift, #205's dispatch
    measurement, a 0.6.0 consumer exercise after #198).

STANDING THE REPO UP IS THE OPERATOR'S STEP, and this is the part I could not
do rather than the part I chose not to. Measured today with this identity:

  POST /api/v1/orgs/heavy-duty/repos  ->  403  not allowed in organization
  POST /api/v1/user/repos             ->  201  personal namespace only

Same shape as the drill delete: a deliberate boundary, not a misconfiguration.
The runbook says so, says not to retry it, and says not to work around it by
using a personal namespace where the org's runner and secrets do not reach.

test/run.sh 22/22, shellcheck 0.10.0, actionlint, self-ref all clean.

Refs #202
This commit is contained in:
cluade-reviewer-andresmgsl 2026-08-05 13:16:20 +00:00
parent dad99ddfb9
commit 08714530b3
2 changed files with 102 additions and 0 deletions

17
changelog.d/202.md Normal file
View file

@ -0,0 +1,17 @@
### Added
- `drills/README.md` documents the standing runner-probe venue,
`heavy-duty/ceremony-runner-probe` — the place runner-only facts are measured
on demand, ruled as option A by the operator (#202).
- The runbook states that the drill disposal rule does **not** apply to it.
Archiving it defeats its purpose, and that is exactly how the three existing
drill repos each became unavailable (#202).
- It records that a probe must run as an Actions job under the workflow token:
the same call answers 500 there and 204 under a PAT, so a probe run any other
way produces a confident wrong answer (#202).
- Creating the repo is recorded as the operator's step, measured rather than
assumed: a fleet identity gets 403 on org repo creation and 201 in its own
namespace (#202).

View file

@ -70,3 +70,88 @@ evidence in the one file whose job is to be evidence.
missing or blank. A waived drill is still a record: the file says WAIVED and
why — a maintainer's call, visible and reviewable in the release PR's diff,
never a silent skip.
# The runner-probe venue
**Not a drill.** A drill rehearses the release doors on a disposable repo and
ends. This is the opposite shape: one **standing** repo that exists so that
runner-only facts can be measured on demand, and it is **never archived**.
`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes.
Ruled by the operator as option (A) of ceremony#202 (#5631).
## Why a standing repo, when drills are disposable
Some facts are only true inside Actions, under the token Actions injects, and
no local harness or PAT can reproduce them. The worked example is ceremony#192:
```
DELETE /issues/{n}/labels/{id} -> 500 under ${{ github.token }} in a workflow
DELETE /issues/{n}/labels/{id} -> 204 under a maintainer PAT, same call
```
A probe that runs anywhere else passes and proves nothing. Before this venue
existed the answer was "un-archive a drill repo", which was requested three
times in two days across two issues and never became anything — the three
drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived,
and each was minted for one probe and then wanted again.
## The disposal rule above does NOT apply here
The rehearsal section says the builder archives the scratch repo and the
operator deletes it. **That rule is for drills.** Archiving this repo defeats
its entire purpose, and it is the failure mode the three archived drill repos
demonstrate — each was archived correctly, by the rule, and each then had to be
un-archived or replaced.
So: never archive it, never delete it, and if you find it archived, un-archive
it rather than minting a fourth one.
## Standing it up is the operator's step
Bot identities cannot create repositories in `heavy-duty`. Measured
2026-08-05 with a fleet identity holding the `repo` scope:
```
POST /api/v1/orgs/heavy-duty/repos -> 403 "not allowed to create repository in organization"
POST /api/v1/user/repos -> 201 (personal namespace only)
```
This is the same shape as the drill delete: a deliberate permission boundary,
not a misconfiguration, and the same rule applies — do not retry it, and do not
work around it by putting the venue in a personal namespace, where the org's
runner and secrets do not reach. Ask the operator.
## Running a probe
1. Reset the repo to a clean state — the probe's own fixtures only, no
leftovers from the last one. A probe that inherits state is a probe whose
result you cannot attribute.
2. Install whatever the probe needs: a workflow that exercises the call, and
`docs/CONSUMERS.md`'s caller stubs if the probe is about the reconcilers.
3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole
point of the venue and the one step that cannot be shortcut. A `curl` from a
laptop with a PAT answers a different question — see the 204/500 split
above — and a probe run that way is worse than no probe, because it produces
a confident wrong answer.
4. Write the results **into the forge** — an issue or a comment in this repo —
rather than leaving them in a job log. Logs age out; ceremony#192's run 701
survived because the job wrote its findings into an issue it created.
5. Record the run number and the repo in whatever issue the probe serves.
## What must never happen here
No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no
runs attributable to a probe. The venue exists so that the live board does not
have to be the test fixture.
## The probes this venue owes
- **ceremony#192** — that the repaired sweep actually lifts a label under the
workflow token, which is the half its acceptance criteria cannot get from
the hermetic contract tests.
- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches`
works on this instance with a valid ref and inputs. Measured so far:
`GET /actions/workflows` 404s and the dispatch route answers 500 rather than
a 4xx, which is not enough to port against.
- A 0.6.0 consumer exercise once ceremony#198 has merged.