heavy-duty/ceremony
7
0
Fork 2
Code Issues 1 Pull requests Projects Releases 8 Packages Wiki Activity Actions

drills/README.md — the standing runner-probe venue, and why the drill disposal rule does not apply to it (#202) #207

Merged
andres merged 14 commits from build/202-runner-probe-venue into main 2026-08-05 17:18:59 +00:00
Conversation 11 Commits 14 Files changed 3 +417
3 changed files with 417 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

64
changelog.d/202.md Normal file
View file

@ -0,0 +1,64 @@
### Added
- `docs/RUNNER-PROBES.md` documents the standing runner-probe venue,
`heavy-duty/ceremony-runner-probe` — the place runner-only facts are measured
on demand, ruled as option A by the operator (#202).
- `drills/README.md` cross-links it beside the disposal rule, so the exception
is visible where the dangerous habit lives (#202).
- The runbook states that the drill disposal rule does **not** apply to it.
Archiving it defeats its purpose, and that is exactly how the three existing
drill repos each became unavailable (#202).
- It records that a probe must run as an Actions job under the workflow token:
the same call answers 500 there and 204 under a PAT, so a probe run any other
way produces a confident wrong answer (#202).
- Creating the repo is recorded as the operator's step, measured rather than
assumed: a fleet identity gets 403 on org repo creation and 201 in its own
namespace (#202).
- It carries an executable two-layer arming procedure: an immutable candidate
code SHA and an armed workflow commit on top of it. A single layer is
self-referential — rewriting a workflow makes a new commit, and a commit
cannot contain its own object ID (#202).
- Callers are pinned by layer: composite actions to the candidate code SHA,
reusable workflows to the armed SHA, which is the only revision whose inner
checkout points at the fork (#202).
- The arming gate asserts what each carrier IS, not only that the old literal
is gone: every `repository:` equals the fork, every `CEREMONY_SELF_REF` value
equal the candidate code SHA, and callers match the layer they belong to
(#202).
- It enumerates the carriers from the tree rather than encoding a count, and
distinguishes ceremony's internal self-checkouts from the consumer checkouts
that must stay `${{ github.repository }}` (#202).
- Both published snippets are ShellCheck-clean when extracted and linted
directly, not merely as part of the repository sweep (#202).
- The checker validates the MANIFEST against the target it was given, so a
manifest that describes a wrong arming consistently — wrong fork, or the
armed SHA where the candidate belongs — refuses instead of matching a tree
rewritten to the same wrong value (#202).
- The manifest is generated from the PRE-arming tree, which is the only order
that enumerates the carriers that must change (#202).
- Both published snippets were driven against a constructed candidate/probe
pair: deletion, both role swaps, wrong owner, wrong
SHA, wrong path, a deleted caller class and an extra carrier all refuse, and
the armed control passes (#202).
- The manifest records complete caller coordinates, so a path swapped under the
right owner and SHA is caught (#202).
- Generator and checker share one domain — ceremony callers — so a third-party
`actions/checkout` is neither manifested nor reported as unrecognised (#202).
- Probe results are written to an issue in the probe repo and carried to the
ceremony issue by a human, so the probe holds no path that can write to the
live board (#202).

345
docs/RUNNER-PROBES.md Normal file
View file

@ -0,0 +1,345 @@
# Runner probes
**Not a drill.** A drill rehearses the release doors on a disposable repo and
ends. This is the opposite shape: one **standing** repo that exists so that
runner-only facts can be measured on demand, and it is **never archived**.
`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes.
Ruled by the operator as option (A) of ceremony#202 (#5631).
## Why a standing repo, when drills are disposable
Some facts are only true inside Actions, under the token Actions injects, and
no local harness or PAT can reproduce them. The worked example is ceremony#192:
```
DELETE /issues/{n}/labels/{id} -> 500 under ${{ github.token }} in a workflow
DELETE /issues/{n}/labels/{id} -> 204 under a maintainer PAT, same call
```
A probe that runs anywhere else passes and proves nothing. Before this venue
existed the answer was "un-archive a drill repo", which was requested three
times in two days across two issues and never became anything — the three
drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived,
and each was minted for one probe and then wanted again.
## The disposal rule above does NOT apply here
The rehearsal section says the builder archives the scratch repo and the
operator deletes it. **That rule is for drills.** Archiving this repo defeats
its entire purpose, and it is the failure mode the three archived drill repos
demonstrate — each was archived correctly, by the rule, and each then had to be
un-archived or replaced.
So: never archive it, never delete it, and if you find it archived, un-archive
it rather than minting a fourth one.
## Standing it up is the operator's step
Bot identities cannot create repositories in `heavy-duty`. Measured
2026-08-05 with a fleet identity holding the `repo` scope:
```
POST /api/v1/orgs/heavy-duty/repos -> 403 "not allowed to create repository in organization"
POST /api/v1/user/repos -> 201 (personal namespace only)
```
This is the same shape as the drill delete: a deliberate permission boundary,
not a misconfiguration. Do not retry it, and do not work around it by putting
the venue in a personal namespace — **not because a personal namespace is
proven unable to reach the org's runner** (that was not measured; the probe
repository above was deleted immediately, so nothing about runner or secret
reach was established), but because @andres ruled an **org-owned standing
venue** (#5631). A personally-owned repo is a different thing from the one that
was decided on, and cannot satisfy #202's named acceptance target.
If runner or secret reach turns out to matter, measure it once the venue
exists rather than assuming it here.
## Running a probe
1. Reset the repo to a clean state — the probe's own fixtures only, no
leftovers from the last one. A probe that inherits state is a probe whose
result you cannot attribute.
2. **Arm it against the candidate** (below) — two layers, candidate code and
armed workflow — if the probe is about ceremony's own machinery rather than
about a bare API call.
3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole
point of the venue and the one step that cannot be shortcut. A `curl` from a
laptop with a PAT answers a different question — see the 204/500 split
above — and a probe run that way is worse than no probe, because it produces
a confident wrong answer.
4. **The job writes its raw results into an issue in the PROBE repo** —
`heavy-duty/ceremony-runner-probe` — not into ceremony. Logs age out;
ceremony#192's run 701 survived only because the job wrote its findings
into an issue it created.
5. **A human then records the probe issue's URL and the Actions run number on
the ceremony issue the probe serves.** That hop is deliberate and is the
whole of the boundary: the probe workflow holds no credential and no code
path that can write to `heavy-duty/ceremony`, so "the probe reports its
findings" and "the probe cannot touch the live board" stay compatible
rather than contradicting each other (@codex-reviewer-andresmgsl, #202
review).
## Arming a candidate ref
A probe that exercises ceremony's own machinery needs the candidate tree
reachable from a `uses:` line. This is the fork-ref shape `drills/README.md`
step 2 points at, written out — and it has **two layers**, which is the part
that is easy to get wrong and impossible to fix afterwards.
**Why two.** The candidate's own workflows contain
`repository: heavy-duty/ceremony` beside `ref: ${{ env.CEREMONY_SELF_REF }}`,
so they must be rewritten to point at the fork and at the candidate. But
rewriting them **creates a new commit**, and a commit cannot contain its own
object ID. A single-layer arming is therefore self-referential: pin the callers
to the pre-rewrite SHA and they load the *unarmed* workflows; pin them to the
post-rewrite one and you are asking a commit to embed itself
(@codex-reviewer-andresmgsl, #202 review).
So:
| layer | what it is | what it carries |
|---|---|---|
| **candidate code SHA** | the immutable tree under test | `actions/`, `lib/` — untouched |
| **armed workflow SHA** | a small child commit on top of it | workflows rewritten to the fork + `CEREMONY_SELF_REF` = the candidate code SHA |
### The procedure
1. **Push the candidate tree** to a fork under the identity that will run the
probe — one branch, `<identity>/ceremony@probe-<issue>` — and record its
SHA. Steps 1 and 2 advance the tip of that **same** branch; there are two
commits, not two branches. That is
the **candidate code SHA**. Never create a branch on
`heavy-duty/ceremony` named like a tag: it shadows that tag for every
consumer until somebody remembers to delete it.
2. **Write the manifest FIRST, from the pre-arming tree, then commit the
arming.** The manifest enumerates the carriers *that must change*, so it is
generated before they do — running it afterwards would enumerate
already-rewritten rows and lose the canonical internal-checkout ones
entirely (@codex-reviewer-andresmgsl, #202 review). In that same
fork branch rewrite, for **every** carrier the manifest below enumerates:
ceremony's own internal `repository:` checkouts → `<identity>/ceremony`, and
**every** `CEREMONY_SELF_REF` value → the **candidate code SHA** from step 1.
There were three self-ref carriers on `main` at the time of writing and the
count is not a constant — derive it, do not remember it
(@glm-reviewer-andresmgsl, @codex-reviewer-andresmgsl, #202 review). The
**consumer** checkouts (`${{ github.repository }}`) are left alone. Record
the resulting SHA: that is the **armed workflow SHA**.
3. **Pin the probe repo's callers by layer**, because they are not the same
thing:
- composite-action callers →
`<identity>/ceremony/actions/<name>@<candidate-code-sha>`;
- reusable-workflow callers →
`<identity>/ceremony/.github/workflows/<file>@<armed-workflow-sha>`, since
that is the only revision whose inner checkout is rewritten.
4. **Gate the arming against a MANIFEST, byte for byte.** Every weaker shape
has a hole, and each of these was found in a published draft of this file
(@codex-reviewer-andresmgsl, #202 review):
| weaker check | what slips through |
|---|---|
| "the old literal is absent" | a carrier rewritten to the wrong fork, or to the *armed* SHA |
| "every extracted value equals X" | a carrier that **vanished** — nothing to compare |
| "each value is one of {fork, dynamic}" | a **role swap**: an internal checkout made dynamic, a consumer checkout pointed at the fork |
| "the SHA suffix matches" | `wrong-owner/ceremony/actions/foo@<right-sha>` |
| "known callers match" | an **unrecognised** caller, or none at all |
| "the owner and the sha are right for the kind" | a **layer swap**: `…/actions/x@<armed>` labelled a workflow caller satisfies both |
So the arming step **writes a manifest** — one line per carrier, `path`,
`kind`, `full expected value` — and the gate compares the tree's actual
carriers against it as a set. A deletion, a role swap, a wrong fork, a wrong
SHA, an extra carrier and a missing caller are then all the same kind of
failure: the sets differ.
**Generate it while arming**, from the tree you are arming, so the manifest
cannot drift from the repository:
```sh
#!/usr/bin/env bash
# write-manifest <candidate-checkout> <probe-checkout> <fork> <code-sha> <armed-sha>
#
# Run against the PRE-ARMING tree and the UNPINNED probe: this records what
# each carrier must BECOME, so it has to see them before they change.
#
# `|| true` on every extraction, for the same reason the checker needs it:
# git grep exits 1 on no-match and `set -e` would abort BEFORE the manifest
# is written — silently, which is how the first version of this generator
# produced no file and no diagnostic when a probe exercised only one layer
# (@codex-reviewer-andresmgsl). A probe need not use both.
set -euo pipefail
candidate="$1"; probe="$2"; fork="$3"; code_sha="$4"; armed_sha="$5"
# shellcheck disable=SC2016 # `${{ github.repository }}` is literal YAML, not a shell expansion
{
git -C "$candidate" grep -n 'CEREMONY_SELF_REF:' -- .github/workflows \
| cut -d: -f1,2 | sed "s|$|\tself_ref\t$code_sha|" || true
git -C "$candidate" grep -n 'repository: heavy-duty/ceremony' -- .github/workflows \
| cut -d: -f1,2 | sed "s|$|\tinternal_repo\t$fork|" || true
git -C "$candidate" grep -n 'repository: ${{ github.repository }}' -- .github/workflows \
| cut -d: -f1,2 | sed 's|$|\tconsumer_repo\t${{ github.repository }}|' || true
# Callers record the COMPLETE expected coordinate, not just the sha: the
# path is as rewritable as the owner, and a manifest that stores only the
# suffix cannot notice `…/actions/wrong-one@<right-sha>`.
git -C "$probe" grep -nE 'uses:[[:space:]]*[^[:space:]]*/ceremony/\.github/workflows/' -- .github \
| sed -E "s|^([^:]+):([0-9]+):.*/ceremony/(\.github/workflows/[^@[:space:]]+)@.*|\\1:\\2\\tworkflow_caller\\t$fork/\\3@$armed_sha|" || true
git -C "$probe" grep -nE 'uses:[[:space:]]*[^[:space:]]*/ceremony/actions/' -- .github \
| sed -E "s|^([^:]+):([0-9]+):.*/ceremony/(actions/[^@[:space:]]+)@.*|\\1:\\2\\taction_caller\\t$fork/\\3@$code_sha|" || true
} | sort >manifest.tsv
# Zero ceremony callers is a refusal by name; one layer only is fine.
callers="$(grep -cE '(workflow|action)_caller' manifest.tsv || true)"
[ "$callers" -gt 0 ] || { echo "manifest: no ceremony callers found in $probe" >&2; exit 1; }
```
Then arm — rewrite and commit — and check the result against it:
```sh
#!/usr/bin/env bash
# check-arming <armed-checkout> <probe-checkout> <fork> <code-sha> <armed-sha> <manifest.tsv>
set -euo pipefail
armed="$1"; probe="$2"; fork="$3"; code_sha="$4"; armed_sha="$5"; manifest="$6"
fail() { echo "arming incomplete: $*" >&2; exit 1; }
# `|| true` on every extraction: git grep exits 1 when nothing matches, and
# under `set -e` that would kill this script BEFORE the comparison — so a
# carrier class that vanished ENTIRELY produced silence instead of a
# refusal. Silence is the worst of the three outcomes; the comparison below
# is what must report it.
[ "$(grep -cE '(workflow|action)_caller' "$manifest" || true)" -gt 0 ] \
|| fail "manifest names no ceremony callers — it cannot prove an arming"
# THE MANIFEST ITSELF IS CHECKED AGAINST THE TARGET, not trusted. Comparing
# only tree-vs-manifest proves consistency, and a manifest generated with the
# armed SHA where the candidate SHA belonged — or with the wrong fork —
# describes a WRONG arming perfectly. The tree would then match it and the
# gate would pass (@codex-reviewer-andresmgsl, #202 review).
# shellcheck disable=SC2016 # `${{ github.repository }}` below is literal YAML
while IFS=$'\t' read -r loc kind want; do
case "$kind" in
self_ref) [ "$want" = "$code_sha" ] || fail "manifest $loc: self_ref should be the CANDIDATE sha" ;;
internal_repo) [ "$want" = "$fork" ] || fail "manifest $loc: internal repo should be $fork" ;;
consumer_repo) [ "$want" = '${{ github.repository }}' ] \
|| fail "manifest $loc: consumer checkout must stay dynamic" ;;
# THE KIND MUST BIND TO THE PATH CLASS, not only to the owner and the
# sha. The path class is what SAYS which layer a caller is, so checking
# the sha against the kind while letting the kind float free accepts a
# consistent layer swap — `…/actions/x@<armed>` declared workflow_caller
# passes every owner and sha test (@codex-reviewer-andresmgsl, #202
# review). Decompose once, then let the kind fix BOTH coordinates.
workflow_caller|action_caller)
owner="${want%%/ceremony/*}"; rest="${want#*/ceremony/}"
path="${rest%@*}"; sha="${want##*@}"
[ "$owner/ceremony" = "$fork" ] \
|| fail "manifest $loc: caller owner should be $fork"
case "$kind" in
workflow_caller)
case "$path" in .github/workflows/?*) : ;;
*) fail "manifest $loc: workflow_caller must resolve at .github/workflows/<file>, not '$path'" ;;
esac
[ "$sha" = "$armed_sha" ] || fail "manifest $loc: workflow caller should be the ARMED sha" ;;
action_caller)
case "$path" in actions/?*) : ;;
*) fail "manifest $loc: action_caller must resolve at actions/<name>, not '$path'" ;;
esac
[ "$sha" = "$code_sha" ] || fail "manifest $loc: action caller should be the CANDIDATE sha" ;;
esac ;;
*) fail "manifest $loc: unknown kind '$kind'" ;;
esac
done <"$manifest"
actual="$(mktemp)"; trap 'rm -f "$actual"' EXIT
{
git -C "$armed" grep -nP '(?<=CEREMONY_SELF_REF: ")[^"]+' -- .github/workflows \
| sed -E 's/^([^:]+):([0-9]+):.*CEREMONY_SELF_REF: "([^"]*)".*/\1:\2\tself_ref\t\3/' || true
git -C "$armed" grep -nE 'repository: .+' -- .github/workflows \
| sed -E 's|^([^:]+):([0-9]+):[[:space:]]*repository:[[:space:]]*(.*)$|\1:\2\t__repo__\t\3|' || true
# Only CEREMONY callers, matching the generator's domain exactly — a
# third-party `actions/checkout` is not this gate's business, and
# extracting it here while the generator ignores it made every probe fail
# as an "unrecognised carrier" (@codex-reviewer-andresmgsl). A wrong OWNER
# is still caught: `wrong-owner/ceremony/...` matches this pattern.
git -C "$probe" grep -nE 'uses:[[:space:]]*[^[:space:]]*/ceremony/' -- .github \
| sed -E 's|^([^:]+):([0-9]+):[[:space:]]*-?[[:space:]]*uses:[[:space:]]*(.*)$|\1:\2\t__uses__\t\3|' || true
} | sort >"$actual"
# every manifest line must be present with its EXACT expected value, and the
# kinds must match — a role swap changes the kind, not just the value.
while IFS=$'\t' read -r loc kind want; do
case "$kind" in
self_ref) have="$(awk -F'\t' -v l="$loc" '$1==l && $2=="self_ref"{print $3}' "$actual")" ;;
internal_repo|consumer_repo)
have="$(awk -F'\t' -v l="$loc" '$1==l && $2=="__repo__"{print $3}' "$actual")" ;;
workflow_caller|action_caller)
have="$(awk -F'\t' -v l="$loc" '$1==l && $2=="__uses__"{print $3}' "$actual")" ;;
esac
[ -n "$have" ] || fail "carrier vanished: $loc ($kind)"
# ONE comparison for every kind: the manifest already carries the complete
# expected value, so owner, path AND sha are checked at once. Checking the
# owner and the sha separately let `…/actions/wrong-one@<right-sha>`
# through (@codex-reviewer-andresmgsl).
[ "$have" = "$want" ] || fail "$loc ($kind): expected '$want', found '$have'"
done <"$manifest"
# and nothing UNRECOGNISED: every uses:/repository: in the trees must appear
# in the manifest, so an added carrier is a failure rather than a silence.
while IFS=$'\t' read -r loc _ _; do
grep -qF "$loc"$'\t' "$manifest" || fail "carrier not in manifest: $loc"
done <"$actual"
```
**Why a manifest rather than a longer list of assertions.** The carrier set
is a property of the tree at the moment of arming; any list written into
this document is stale the next time a workflow is added. The manifest is
generated from the tree, recorded in the result issue (step 6), and is the
thing a later reader compares against — so "what was armed" is evidence
rather than recollection.
5. **Invoke the probe by the event it is about**, and record which: a
`workflow_dispatch`, or the real board event under test. A probe that fires
a different event than the one under test proves something else.
6. **The result issue records all of it**: the fork repository, the candidate
code SHA, the armed workflow SHA, every rewritten carrier, the workflow
invoked and the run number. Those are what make the result reproducible;
without the two SHAs distinguished, a later reader cannot tell which tree
answered.
7. **Reset removes the candidate-specific EXECUTABLE state**: the caller stubs,
the probe workflow, and the fork's probe branch — whose tip carries both the
candidate commit and the armed commit on top of it — so the next probe
cannot inherit a pin it did not choose. **Result issues are never deleted.**
They may be closed or relabelled; deleting them would recreate the
expiring-log problem this venue exists to avoid.
## Who may reset it
**Operator-owned until ruled otherwise.** #202's task 4 asks who may reset the
venue, and creating the repo is the operator's step, so the access policy is
his to set at the same time (@codex-reviewer-andresmgsl, #202 review).
Two levels, deliberately separated:
- **content reset** — removing probe branches, workflows and fixtures; the
ordinary between-probes operation. It does **not** include deleting result
issues, which are the evidence and are immutable once written
(@codex-reviewer-andresmgsl, #202 review);
- **archive / delete / admin** — which is where the drill rule's damage came
from, and which no bot identity should hold here.
If fleet identities are given push access for content reset, this section
records that; until then, ask.
## What must never happen here
No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no
runs attributable to a probe. The venue exists so that the live board does not
have to be the test fixture.
## The probes this venue owes
- **ceremony#192** — that the repaired sweep actually lifts a label under the
workflow token, which is the half its acceptance criteria cannot get from
the hermetic contract tests.
- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches`
works on this instance with a valid ref and inputs. Measured so far:
`GET /actions/workflows` 404s and the dispatch route answers 500 rather than
a 4xx, which is not enough to port against.
- A 0.6.0 consumer exercise once ceremony#198 has merged.

8
drills/README.md
View file

@ -110,3 +110,11 @@ non-empty even though neither release door reads that file (#217, #237).
missing or blank. A waived drill is still a record: the file says WAIVED and
why — a maintainer's call, visible and reviewable in the release PR's diff,
never a silent skip.
---
**Standing runner probes are not drills.** The disposal rule above — builder
archives, operator deletes — is for the disposable scratch repo a drill runs
in. `heavy-duty/ceremony-runner-probe` is the opposite shape: it stands, and
archiving it is the failure mode that made all three previous drill repos
unavailable. See [docs/RUNNER-PROBES.md](../docs/RUNNER-PROBES.md) (#202).