forked from heavy-duty/ceremony
docs(runner-probes): its own document, an arming procedure, and the evidence boundary made consistent (#202)
@codex-reviewer-andresmgsl's four gaps.
1. BRANCH UPDATED TO CURRENT MAIN. The commit's parent was pre-#204 dad99dd, so
its green 22-file run did not contain the six test files and rules that
landed with the sync. Merged main in — no rewrite — and re-verified against
the 28-file suite the operator would actually receive.
2. WHO MAY RESET IT is now a section, and it says operator-owned until ruled
otherwise, with content reset separated from archive/delete/admin. The
access policy is set when the repo is created, which is the operator's step,
so the two belong together. Flagged for @andres rather than assumed.
3. AN EXECUTABLE ARMING PROCEDURE replaces "install whatever the probe needs":
fork ref and canonical SHA, caller stubs pinned to it, BOTH
CEREMONY_SELF_REF carriers rewritten, the event invoked recorded by name,
and what reset removes afterwards. It reuses the drill rehearsal's fork-ref
pattern rather than inventing a floating pin, including its rule against
ever creating a tag-shaped branch on heavy-duty/ceremony.
4. THE EVIDENCE CONTRADICTION IS RESOLVED. "Write results into an issue in this
repo" and "no probe touches ceremony's board" could not both be followed in
a file where "this repo" reads as ceremony. The job now writes raw results
into the PROBE repo, and a human carries the issue URL and run number to the
ceremony issue. The probe workflow holds no credential and no code path that
can write to ceremony, which is what makes the two rules compatible.
Placement: the operational contract moves to docs/RUNNER-PROBES.md, with a
short cross-link in drills/README.md beside the disposal rule it excepts — the
exception stays visible where the dangerous habit lives, and neither document
grows a second top-level heading.
test/run.sh 28/28 on the updated branch; shellcheck 0.10.0, actionlint,
self-ref, marker, vendored and changelog-armed clean.
Refs #202
This commit is contained in:
parent
2037275a9c
commit
e61bb91476
3 changed files with 154 additions and 84 deletions
|
|
@ -1,9 +1,12 @@
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
- `drills/README.md` documents the standing runner-probe venue,
|
- `docs/RUNNER-PROBES.md` documents the standing runner-probe venue,
|
||||||
`heavy-duty/ceremony-runner-probe` — the place runner-only facts are measured
|
`heavy-duty/ceremony-runner-probe` — the place runner-only facts are measured
|
||||||
on demand, ruled as option A by the operator (#202).
|
on demand, ruled as option A by the operator (#202).
|
||||||
|
|
||||||
|
- `drills/README.md` cross-links it beside the disposal rule, so the exception
|
||||||
|
is visible where the dangerous habit lives (#202).
|
||||||
|
|
||||||
- The runbook states that the drill disposal rule does **not** apply to it.
|
- The runbook states that the drill disposal rule does **not** apply to it.
|
||||||
Archiving it defeats its purpose, and that is exactly how the three existing
|
Archiving it defeats its purpose, and that is exactly how the three existing
|
||||||
drill repos each became unavailable (#202).
|
drill repos each became unavailable (#202).
|
||||||
|
|
@ -15,3 +18,11 @@
|
||||||
- Creating the repo is recorded as the operator's step, measured rather than
|
- Creating the repo is recorded as the operator's step, measured rather than
|
||||||
assumed: a fleet identity gets 403 on org repo creation and 201 in its own
|
assumed: a fleet identity gets 403 on org repo creation and 201 in its own
|
||||||
namespace (#202).
|
namespace (#202).
|
||||||
|
|
||||||
|
- It carries an executable arming procedure — fork ref, canonical SHA, both
|
||||||
|
`CEREMONY_SELF_REF` carriers rewritten, and what the result must record
|
||||||
|
(#202).
|
||||||
|
|
||||||
|
- Probe results are written to an issue in the probe repo and carried to the
|
||||||
|
ceremony issue by a human, so the probe holds no path that can write to the
|
||||||
|
live board (#202).
|
||||||
|
|
|
||||||
136
docs/RUNNER-PROBES.md
Normal file
136
docs/RUNNER-PROBES.md
Normal file
|
|
@ -0,0 +1,136 @@
|
||||||
|
# Runner probes
|
||||||
|
|
||||||
|
**Not a drill.** A drill rehearses the release doors on a disposable repo and
|
||||||
|
ends. This is the opposite shape: one **standing** repo that exists so that
|
||||||
|
runner-only facts can be measured on demand, and it is **never archived**.
|
||||||
|
|
||||||
|
`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes.
|
||||||
|
Ruled by the operator as option (A) of ceremony#202 (#5631).
|
||||||
|
|
||||||
|
## Why a standing repo, when drills are disposable
|
||||||
|
|
||||||
|
Some facts are only true inside Actions, under the token Actions injects, and
|
||||||
|
no local harness or PAT can reproduce them. The worked example is ceremony#192:
|
||||||
|
|
||||||
|
```
|
||||||
|
DELETE /issues/{n}/labels/{id} -> 500 under ${{ github.token }} in a workflow
|
||||||
|
DELETE /issues/{n}/labels/{id} -> 204 under a maintainer PAT, same call
|
||||||
|
```
|
||||||
|
|
||||||
|
A probe that runs anywhere else passes and proves nothing. Before this venue
|
||||||
|
existed the answer was "un-archive a drill repo", which was requested three
|
||||||
|
times in two days across two issues and never became anything — the three
|
||||||
|
drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived,
|
||||||
|
and each was minted for one probe and then wanted again.
|
||||||
|
|
||||||
|
## The disposal rule above does NOT apply here
|
||||||
|
|
||||||
|
The rehearsal section says the builder archives the scratch repo and the
|
||||||
|
operator deletes it. **That rule is for drills.** Archiving this repo defeats
|
||||||
|
its entire purpose, and it is the failure mode the three archived drill repos
|
||||||
|
demonstrate — each was archived correctly, by the rule, and each then had to be
|
||||||
|
un-archived or replaced.
|
||||||
|
|
||||||
|
So: never archive it, never delete it, and if you find it archived, un-archive
|
||||||
|
it rather than minting a fourth one.
|
||||||
|
|
||||||
|
## Standing it up is the operator's step
|
||||||
|
|
||||||
|
Bot identities cannot create repositories in `heavy-duty`. Measured
|
||||||
|
2026-08-05 with a fleet identity holding the `repo` scope:
|
||||||
|
|
||||||
|
```
|
||||||
|
POST /api/v1/orgs/heavy-duty/repos -> 403 "not allowed to create repository in organization"
|
||||||
|
POST /api/v1/user/repos -> 201 (personal namespace only)
|
||||||
|
```
|
||||||
|
|
||||||
|
This is the same shape as the drill delete: a deliberate permission boundary,
|
||||||
|
not a misconfiguration, and the same rule applies — do not retry it, and do not
|
||||||
|
work around it by putting the venue in a personal namespace, where the org's
|
||||||
|
runner and secrets do not reach. Ask the operator.
|
||||||
|
|
||||||
|
## Running a probe
|
||||||
|
|
||||||
|
1. Reset the repo to a clean state — the probe's own fixtures only, no
|
||||||
|
leftovers from the last one. A probe that inherits state is a probe whose
|
||||||
|
result you cannot attribute.
|
||||||
|
2. **Arm it against the candidate ref** (below), if the probe is about
|
||||||
|
ceremony's own code rather than about a bare API call.
|
||||||
|
3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole
|
||||||
|
point of the venue and the one step that cannot be shortcut. A `curl` from a
|
||||||
|
laptop with a PAT answers a different question — see the 204/500 split
|
||||||
|
above — and a probe run that way is worse than no probe, because it produces
|
||||||
|
a confident wrong answer.
|
||||||
|
4. **The job writes its raw results into an issue in the PROBE repo** —
|
||||||
|
`heavy-duty/ceremony-runner-probe` — not into ceremony. Logs age out;
|
||||||
|
ceremony#192's run 701 survived only because the job wrote its findings
|
||||||
|
into an issue it created.
|
||||||
|
5. **A human then records the probe issue's URL and the Actions run number on
|
||||||
|
the ceremony issue the probe serves.** That hop is deliberate and is the
|
||||||
|
whole of the boundary: the probe workflow holds no credential and no code
|
||||||
|
path that can write to `heavy-duty/ceremony`, so "the probe reports its
|
||||||
|
findings" and "the probe cannot touch the live board" stay compatible
|
||||||
|
rather than contradicting each other (@codex-reviewer-andresmgsl, #202
|
||||||
|
review).
|
||||||
|
|
||||||
|
## Arming a candidate ref
|
||||||
|
|
||||||
|
A probe that exercises ceremony's own machinery needs the candidate tree
|
||||||
|
reachable from a `uses:` line. The shape is the drill rehearsal's, reused
|
||||||
|
rather than reinvented (`drills/README.md` step 2):
|
||||||
|
|
||||||
|
1. **The candidate is a commit SHA, on a fork ref.** Push the candidate tree to
|
||||||
|
a fork under the identity running the probe —
|
||||||
|
`<identity>/ceremony@probe-<issue>` — and take its canonical SHA. Never
|
||||||
|
create a branch on `heavy-duty/ceremony` named like a tag: it shadows that
|
||||||
|
tag for every consumer until somebody remembers to delete it.
|
||||||
|
2. **Install `docs/CONSUMERS.md`'s caller stubs** in the probe repo, pinned to
|
||||||
|
that SHA — every `uses:` reference, not some of them.
|
||||||
|
3. **Rewrite `CEREMONY_SELF_REF` in the candidate itself** to the same
|
||||||
|
canonical SHA. A candidate carries the tag it has not created yet, so the
|
||||||
|
self-consumption checkout cannot resolve until it is rewritten. Both
|
||||||
|
carriers — `labels.yml` and `release.yml` — take the same value.
|
||||||
|
4. **Invoke the probe by the event it is about**, and record which: a
|
||||||
|
`workflow_dispatch` of the caller, or the real board event the probe is
|
||||||
|
testing. A probe that fires a different event than the one under test
|
||||||
|
proves something else.
|
||||||
|
5. **Record the fork ref, the rewritten pin, the workflow invoked and the run
|
||||||
|
number** in the probe repo's result issue. Those four are what make the
|
||||||
|
result reproducible; without the pin especially, a later reader cannot tell
|
||||||
|
which tree answered.
|
||||||
|
6. **Reset removes the candidate-specific fixtures**: the caller stubs, the
|
||||||
|
probe workflow and the result branch, so the next probe does not inherit a
|
||||||
|
pin it did not choose. The result *issue* stays — it is the evidence.
|
||||||
|
|
||||||
|
## Who may reset it
|
||||||
|
|
||||||
|
**Operator-owned until ruled otherwise.** #202's task 4 asks who may reset the
|
||||||
|
venue, and creating the repo is the operator's step, so the access policy is
|
||||||
|
his to set at the same time (@codex-reviewer-andresmgsl, #202 review).
|
||||||
|
|
||||||
|
Two levels, deliberately separated:
|
||||||
|
|
||||||
|
- **content reset** — deleting probe branches, workflows, fixtures and stale
|
||||||
|
result issues; the ordinary between-probes operation;
|
||||||
|
- **archive / delete / admin** — which is where the drill rule's damage came
|
||||||
|
from, and which no bot identity should hold here.
|
||||||
|
|
||||||
|
If fleet identities are given push access for content reset, this section
|
||||||
|
records that; until then, ask.
|
||||||
|
|
||||||
|
## What must never happen here
|
||||||
|
|
||||||
|
No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no
|
||||||
|
runs attributable to a probe. The venue exists so that the live board does not
|
||||||
|
have to be the test fixture.
|
||||||
|
|
||||||
|
## The probes this venue owes
|
||||||
|
|
||||||
|
- **ceremony#192** — that the repaired sweep actually lifts a label under the
|
||||||
|
workflow token, which is the half its acceptance criteria cannot get from
|
||||||
|
the hermetic contract tests.
|
||||||
|
- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches`
|
||||||
|
works on this instance with a valid ref and inputs. Measured so far:
|
||||||
|
`GET /actions/workflows` 404s and the dispatch route answers 500 rather than
|
||||||
|
a 4xx, which is not enough to port against.
|
||||||
|
- A 0.6.0 consumer exercise once ceremony#198 has merged.
|
||||||
|
|
@ -111,87 +111,10 @@ missing or blank. A waived drill is still a record: the file says WAIVED and
|
||||||
why — a maintainer's call, visible and reviewable in the release PR's diff,
|
why — a maintainer's call, visible and reviewable in the release PR's diff,
|
||||||
never a silent skip.
|
never a silent skip.
|
||||||
|
|
||||||
# The runner-probe venue
|
---
|
||||||
|
|
||||||
**Not a drill.** A drill rehearses the release doors on a disposable repo and
|
**Standing runner probes are not drills.** The disposal rule above — builder
|
||||||
ends. This is the opposite shape: one **standing** repo that exists so that
|
archives, operator deletes — is for the disposable scratch repo a drill runs
|
||||||
runner-only facts can be measured on demand, and it is **never archived**.
|
in. `heavy-duty/ceremony-runner-probe` is the opposite shape: it stands, and
|
||||||
|
archiving it is the failure mode that made all three previous drill repos
|
||||||
`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes.
|
unavailable. See [docs/RUNNER-PROBES.md](../docs/RUNNER-PROBES.md) (#202).
|
||||||
Ruled by the operator as option (A) of ceremony#202 (#5631).
|
|
||||||
|
|
||||||
## Why a standing repo, when drills are disposable
|
|
||||||
|
|
||||||
Some facts are only true inside Actions, under the token Actions injects, and
|
|
||||||
no local harness or PAT can reproduce them. The worked example is ceremony#192:
|
|
||||||
|
|
||||||
```
|
|
||||||
DELETE /issues/{n}/labels/{id} -> 500 under ${{ github.token }} in a workflow
|
|
||||||
DELETE /issues/{n}/labels/{id} -> 204 under a maintainer PAT, same call
|
|
||||||
```
|
|
||||||
|
|
||||||
A probe that runs anywhere else passes and proves nothing. Before this venue
|
|
||||||
existed the answer was "un-archive a drill repo", which was requested three
|
|
||||||
times in two days across two issues and never became anything — the three
|
|
||||||
drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived,
|
|
||||||
and each was minted for one probe and then wanted again.
|
|
||||||
|
|
||||||
## The disposal rule above does NOT apply here
|
|
||||||
|
|
||||||
The rehearsal section says the builder archives the scratch repo and the
|
|
||||||
operator deletes it. **That rule is for drills.** Archiving this repo defeats
|
|
||||||
its entire purpose, and it is the failure mode the three archived drill repos
|
|
||||||
demonstrate — each was archived correctly, by the rule, and each then had to be
|
|
||||||
un-archived or replaced.
|
|
||||||
|
|
||||||
So: never archive it, never delete it, and if you find it archived, un-archive
|
|
||||||
it rather than minting a fourth one.
|
|
||||||
|
|
||||||
## Standing it up is the operator's step
|
|
||||||
|
|
||||||
Bot identities cannot create repositories in `heavy-duty`. Measured
|
|
||||||
2026-08-05 with a fleet identity holding the `repo` scope:
|
|
||||||
|
|
||||||
```
|
|
||||||
POST /api/v1/orgs/heavy-duty/repos -> 403 "not allowed to create repository in organization"
|
|
||||||
POST /api/v1/user/repos -> 201 (personal namespace only)
|
|
||||||
```
|
|
||||||
|
|
||||||
This is the same shape as the drill delete: a deliberate permission boundary,
|
|
||||||
not a misconfiguration, and the same rule applies — do not retry it, and do not
|
|
||||||
work around it by putting the venue in a personal namespace, where the org's
|
|
||||||
runner and secrets do not reach. Ask the operator.
|
|
||||||
|
|
||||||
## Running a probe
|
|
||||||
|
|
||||||
1. Reset the repo to a clean state — the probe's own fixtures only, no
|
|
||||||
leftovers from the last one. A probe that inherits state is a probe whose
|
|
||||||
result you cannot attribute.
|
|
||||||
2. Install whatever the probe needs: a workflow that exercises the call, and
|
|
||||||
`docs/CONSUMERS.md`'s caller stubs if the probe is about the reconcilers.
|
|
||||||
3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole
|
|
||||||
point of the venue and the one step that cannot be shortcut. A `curl` from a
|
|
||||||
laptop with a PAT answers a different question — see the 204/500 split
|
|
||||||
above — and a probe run that way is worse than no probe, because it produces
|
|
||||||
a confident wrong answer.
|
|
||||||
4. Write the results **into the forge** — an issue or a comment in this repo —
|
|
||||||
rather than leaving them in a job log. Logs age out; ceremony#192's run 701
|
|
||||||
survived because the job wrote its findings into an issue it created.
|
|
||||||
5. Record the run number and the repo in whatever issue the probe serves.
|
|
||||||
|
|
||||||
## What must never happen here
|
|
||||||
|
|
||||||
No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no
|
|
||||||
runs attributable to a probe. The venue exists so that the live board does not
|
|
||||||
have to be the test fixture.
|
|
||||||
|
|
||||||
## The probes this venue owes
|
|
||||||
|
|
||||||
- **ceremony#192** — that the repaired sweep actually lifts a label under the
|
|
||||||
workflow token, which is the half its acceptance criteria cannot get from
|
|
||||||
the hermetic contract tests.
|
|
||||||
- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches`
|
|
||||||
works on this instance with a valid ref and inputs. Measured so far:
|
|
||||||
`GET /actions/workflows` 404s and the dispatch route answers 500 rather than
|
|
||||||
a 4xx, which is not enough to port against.
|
|
||||||
- A 0.6.0 consumer exercise once ceremony#198 has merged.
|
|
||||||
|
|
|
||||||
Loading…
Reference in a new issue