forked from heavy-duty/ceremony
cluade-reviewer-andresmgsl
e27acd8ab9
@codex-reviewer-andresmgsl found that the procedure was not executable as
written, and the reason is structural rather than a wording slip.
The candidate's workflows carry `repository: heavy-duty/ceremony` beside
`ref: ${{ env.CEREMONY_SELF_REF }}`, so arming must rewrite them. But
rewriting CREATES A NEW COMMIT, and a commit cannot embed its own object ID. So
a single-layer arming is self-referential: pin the callers to the pre-rewrite
SHA and they load the UNARMED workflows; pin them to the post-rewrite SHA and
you are asking that commit to contain itself. My step 3 asked for exactly that.
Two layers, stated as a table because the distinction is the whole thing:
candidate code SHA the immutable tree under test — actions/, lib/
armed workflow SHA a child commit whose workflows point at the fork and
whose CEREMONY_SELF_REF is the candidate code SHA
And callers pin by layer, because they are not the same thing: composite
actions to the candidate code SHA, reusable workflows to the armed SHA, which
is the only revision whose inner checkout is rewritten.
The completeness check becomes a mechanical non-zero gate — `git grep` for
executable `uses:`/`repository:` carriers over the ARMED tree, exiting non-zero
on any hit — rather than "every remaining hit must be prose". A partial rewrite
does not announce itself: it silently tests canonical main.
The result issue records both SHAs, not one, or a later reader cannot tell
which tree answered.
test/run.sh 28/28; shellcheck 0.10.0 and changelog-armed clean.
Refs #202
184 lines
9.1 KiB
Markdown
184 lines
9.1 KiB
Markdown
# Runner probes
|
|
|
|
**Not a drill.** A drill rehearses the release doors on a disposable repo and
|
|
ends. This is the opposite shape: one **standing** repo that exists so that
|
|
runner-only facts can be measured on demand, and it is **never archived**.
|
|
|
|
`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes.
|
|
Ruled by the operator as option (A) of ceremony#202 (#5631).
|
|
|
|
## Why a standing repo, when drills are disposable
|
|
|
|
Some facts are only true inside Actions, under the token Actions injects, and
|
|
no local harness or PAT can reproduce them. The worked example is ceremony#192:
|
|
|
|
```
|
|
DELETE /issues/{n}/labels/{id} -> 500 under ${{ github.token }} in a workflow
|
|
DELETE /issues/{n}/labels/{id} -> 204 under a maintainer PAT, same call
|
|
```
|
|
|
|
A probe that runs anywhere else passes and proves nothing. Before this venue
|
|
existed the answer was "un-archive a drill repo", which was requested three
|
|
times in two days across two issues and never became anything — the three
|
|
drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived,
|
|
and each was minted for one probe and then wanted again.
|
|
|
|
## The disposal rule above does NOT apply here
|
|
|
|
The rehearsal section says the builder archives the scratch repo and the
|
|
operator deletes it. **That rule is for drills.** Archiving this repo defeats
|
|
its entire purpose, and it is the failure mode the three archived drill repos
|
|
demonstrate — each was archived correctly, by the rule, and each then had to be
|
|
un-archived or replaced.
|
|
|
|
So: never archive it, never delete it, and if you find it archived, un-archive
|
|
it rather than minting a fourth one.
|
|
|
|
## Standing it up is the operator's step
|
|
|
|
Bot identities cannot create repositories in `heavy-duty`. Measured
|
|
2026-08-05 with a fleet identity holding the `repo` scope:
|
|
|
|
```
|
|
POST /api/v1/orgs/heavy-duty/repos -> 403 "not allowed to create repository in organization"
|
|
POST /api/v1/user/repos -> 201 (personal namespace only)
|
|
```
|
|
|
|
This is the same shape as the drill delete: a deliberate permission boundary,
|
|
not a misconfiguration. Do not retry it, and do not work around it by putting
|
|
the venue in a personal namespace — **not because a personal namespace is
|
|
proven unable to reach the org's runner** (that was not measured; the probe
|
|
repository above was deleted immediately, so nothing about runner or secret
|
|
reach was established), but because @andres ruled an **org-owned standing
|
|
venue** (#5631). A personally-owned repo is a different thing from the one that
|
|
was decided on, and cannot satisfy #202's named acceptance target.
|
|
|
|
If runner or secret reach turns out to matter, measure it once the venue
|
|
exists rather than assuming it here.
|
|
|
|
## Running a probe
|
|
|
|
1. Reset the repo to a clean state — the probe's own fixtures only, no
|
|
leftovers from the last one. A probe that inherits state is a probe whose
|
|
result you cannot attribute.
|
|
2. **Arm it against the candidate** (below) — two layers, candidate code and
|
|
armed workflow — if the probe is about ceremony's own machinery rather than
|
|
about a bare API call.
|
|
3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole
|
|
point of the venue and the one step that cannot be shortcut. A `curl` from a
|
|
laptop with a PAT answers a different question — see the 204/500 split
|
|
above — and a probe run that way is worse than no probe, because it produces
|
|
a confident wrong answer.
|
|
4. **The job writes its raw results into an issue in the PROBE repo** —
|
|
`heavy-duty/ceremony-runner-probe` — not into ceremony. Logs age out;
|
|
ceremony#192's run 701 survived only because the job wrote its findings
|
|
into an issue it created.
|
|
5. **A human then records the probe issue's URL and the Actions run number on
|
|
the ceremony issue the probe serves.** That hop is deliberate and is the
|
|
whole of the boundary: the probe workflow holds no credential and no code
|
|
path that can write to `heavy-duty/ceremony`, so "the probe reports its
|
|
findings" and "the probe cannot touch the live board" stay compatible
|
|
rather than contradicting each other (@codex-reviewer-andresmgsl, #202
|
|
review).
|
|
|
|
## Arming a candidate ref
|
|
|
|
A probe that exercises ceremony's own machinery needs the candidate tree
|
|
reachable from a `uses:` line. This is the fork-ref shape `drills/README.md`
|
|
step 2 points at, written out — and it has **two layers**, which is the part
|
|
that is easy to get wrong and impossible to fix afterwards.
|
|
|
|
**Why two.** The candidate's own workflows contain
|
|
`repository: heavy-duty/ceremony` beside `ref: ${{ env.CEREMONY_SELF_REF }}`,
|
|
so they must be rewritten to point at the fork and at the candidate. But
|
|
rewriting them **creates a new commit**, and a commit cannot contain its own
|
|
object ID. A single-layer arming is therefore self-referential: pin the callers
|
|
to the pre-rewrite SHA and they load the *unarmed* workflows; pin them to the
|
|
post-rewrite one and you are asking a commit to embed itself
|
|
(@codex-reviewer-andresmgsl, #202 review).
|
|
|
|
So:
|
|
|
|
| layer | what it is | what it carries |
|
|
|---|---|---|
|
|
| **candidate code SHA** | the immutable tree under test | `actions/`, `lib/` — untouched |
|
|
| **armed workflow SHA** | a small child commit on top of it | workflows rewritten to the fork + `CEREMONY_SELF_REF` = the candidate code SHA |
|
|
|
|
### The procedure
|
|
|
|
1. **Push the candidate tree** to a fork under the identity that will run the
|
|
probe — `<identity>/ceremony@probe-<issue>` — and record its SHA. That is
|
|
the **candidate code SHA**. Never create a branch on
|
|
`heavy-duty/ceremony` named like a tag: it shadows that tag for every
|
|
consumer until somebody remembers to delete it.
|
|
2. **Commit the arming on top of it.** In that same fork branch, rewrite every
|
|
workflow carrier — `repository:` → `<identity>/ceremony`, and both
|
|
`CEREMONY_SELF_REF` values → the **candidate code SHA** from step 1. Record
|
|
the resulting SHA. That is the **armed workflow SHA**.
|
|
3. **Pin the probe repo's callers by layer**, because they are not the same
|
|
thing:
|
|
- composite-action callers →
|
|
`<identity>/ceremony/actions/<name>@<candidate-code-sha>`;
|
|
- reusable-workflow callers →
|
|
`<identity>/ceremony/.github/workflows/<file>@<armed-workflow-sha>`, since
|
|
that is the only revision whose inner checkout is rewritten.
|
|
4. **Gate the rewrite mechanically, against the armed tree** — not against
|
|
whichever checkout happens to be current, and not by eyeballing prose:
|
|
|
|
```sh
|
|
git -C <armed-checkout> grep -nE \
|
|
'(uses:|repository:)[[:space:]]*heavy-duty/ceremony' -- .github \
|
|
&& { echo "arming incomplete" >&2; exit 1; }
|
|
```
|
|
|
|
Exit non-zero on any hit. A partial rewrite does not fail loudly on its
|
|
own — it silently tests canonical `main`, and the probe's answer is then
|
|
about the wrong tree.
|
|
5. **Invoke the probe by the event it is about**, and record which: a
|
|
`workflow_dispatch`, or the real board event under test. A probe that fires
|
|
a different event than the one under test proves something else.
|
|
6. **The result issue records all of it**: the fork repository, the candidate
|
|
code SHA, the armed workflow SHA, every rewritten carrier, the workflow
|
|
invoked and the run number. Those are what make the result reproducible;
|
|
without the two SHAs distinguished, a later reader cannot tell which tree
|
|
answered.
|
|
7. **Reset removes the candidate-specific EXECUTABLE state**: the caller stubs,
|
|
the probe workflow, the candidate and armed branches — so the next probe
|
|
cannot inherit a pin it did not choose. **Result issues are never deleted.**
|
|
They may be closed or relabelled; deleting them would recreate the
|
|
expiring-log problem this venue exists to avoid.
|
|
|
|
## Who may reset it
|
|
|
|
**Operator-owned until ruled otherwise.** #202's task 4 asks who may reset the
|
|
venue, and creating the repo is the operator's step, so the access policy is
|
|
his to set at the same time (@codex-reviewer-andresmgsl, #202 review).
|
|
|
|
Two levels, deliberately separated:
|
|
|
|
- **content reset** — removing probe branches, workflows and fixtures; the
|
|
ordinary between-probes operation. It does **not** include deleting result
|
|
issues, which are the evidence and are immutable once written
|
|
(@codex-reviewer-andresmgsl, #202 review);
|
|
- **archive / delete / admin** — which is where the drill rule's damage came
|
|
from, and which no bot identity should hold here.
|
|
|
|
If fleet identities are given push access for content reset, this section
|
|
records that; until then, ask.
|
|
|
|
## What must never happen here
|
|
|
|
No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no
|
|
runs attributable to a probe. The venue exists so that the live board does not
|
|
have to be the test fixture.
|
|
|
|
## The probes this venue owes
|
|
|
|
- **ceremony#192** — that the repaired sweep actually lifts a label under the
|
|
workflow token, which is the half its acceptance criteria cannot get from
|
|
the hermetic contract tests.
|
|
- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches`
|
|
works on this instance with a valid ref and inputs. Measured so far:
|
|
`GET /actions/workflows` 404s and the dispatch route answers 500 rather than
|
|
a 4xx, which is not enough to port against.
|
|
- A 0.6.0 consumer exercise once ceremony#198 has merged.
|