drills/README.md — the standing runner-probe venue, and why the drill disposal rule does not apply to it (#202) #207
3 changed files with 154 additions and 84 deletions
|
|
@ -1,9 +1,12 @@
|
|||
### Added
|
||||
|
||||
- `drills/README.md` documents the standing runner-probe venue,
|
||||
- `docs/RUNNER-PROBES.md` documents the standing runner-probe venue,
|
||||
`heavy-duty/ceremony-runner-probe` — the place runner-only facts are measured
|
||||
on demand, ruled as option A by the operator (#202).
|
||||
|
||||
- `drills/README.md` cross-links it beside the disposal rule, so the exception
|
||||
is visible where the dangerous habit lives (#202).
|
||||
|
||||
- The runbook states that the drill disposal rule does **not** apply to it.
|
||||
Archiving it defeats its purpose, and that is exactly how the three existing
|
||||
drill repos each became unavailable (#202).
|
||||
|
|
@ -15,3 +18,11 @@
|
|||
- Creating the repo is recorded as the operator's step, measured rather than
|
||||
assumed: a fleet identity gets 403 on org repo creation and 201 in its own
|
||||
namespace (#202).
|
||||
|
||||
- It carries an executable arming procedure — fork ref, canonical SHA, both
|
||||
`CEREMONY_SELF_REF` carriers rewritten, and what the result must record
|
||||
(#202).
|
||||
|
||||
- Probe results are written to an issue in the probe repo and carried to the
|
||||
ceremony issue by a human, so the probe holds no path that can write to the
|
||||
live board (#202).
|
||||
|
|
|
|||
136
docs/RUNNER-PROBES.md
Normal file
136
docs/RUNNER-PROBES.md
Normal file
|
|
@ -0,0 +1,136 @@
|
|||
# Runner probes
|
||||
|
||||
**Not a drill.** A drill rehearses the release doors on a disposable repo and
|
||||
ends. This is the opposite shape: one **standing** repo that exists so that
|
||||
runner-only facts can be measured on demand, and it is **never archived**.
|
||||
|
||||
`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes.
|
||||
Ruled by the operator as option (A) of ceremony#202 (#5631).
|
||||
|
||||
## Why a standing repo, when drills are disposable
|
||||
|
||||
Some facts are only true inside Actions, under the token Actions injects, and
|
||||
no local harness or PAT can reproduce them. The worked example is ceremony#192:
|
||||
|
||||
```
|
||||
DELETE /issues/{n}/labels/{id} -> 500 under ${{ github.token }} in a workflow
|
||||
DELETE /issues/{n}/labels/{id} -> 204 under a maintainer PAT, same call
|
||||
```
|
||||
|
||||
A probe that runs anywhere else passes and proves nothing. Before this venue
|
||||
existed the answer was "un-archive a drill repo", which was requested three
|
||||
times in two days across two issues and never became anything — the three
|
||||
drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived,
|
||||
and each was minted for one probe and then wanted again.
|
||||
|
||||
## The disposal rule above does NOT apply here
|
||||
|
||||
The rehearsal section says the builder archives the scratch repo and the
|
||||
operator deletes it. **That rule is for drills.** Archiving this repo defeats
|
||||
its entire purpose, and it is the failure mode the three archived drill repos
|
||||
demonstrate — each was archived correctly, by the rule, and each then had to be
|
||||
un-archived or replaced.
|
||||
|
||||
So: never archive it, never delete it, and if you find it archived, un-archive
|
||||
it rather than minting a fourth one.
|
||||
|
||||
## Standing it up is the operator's step
|
||||
|
||||
Bot identities cannot create repositories in `heavy-duty`. Measured
|
||||
2026-08-05 with a fleet identity holding the `repo` scope:
|
||||
|
||||
```
|
||||
POST /api/v1/orgs/heavy-duty/repos -> 403 "not allowed to create repository in organization"
|
||||
POST /api/v1/user/repos -> 201 (personal namespace only)
|
||||
```
|
||||
|
||||
This is the same shape as the drill delete: a deliberate permission boundary,
|
||||
not a misconfiguration, and the same rule applies — do not retry it, and do not
|
||||
work around it by putting the venue in a personal namespace, where the org's
|
||||
runner and secrets do not reach. Ask the operator.
|
||||
|
||||
## Running a probe
|
||||
|
||||
1. Reset the repo to a clean state — the probe's own fixtures only, no
|
||||
leftovers from the last one. A probe that inherits state is a probe whose
|
||||
result you cannot attribute.
|
||||
2. **Arm it against the candidate ref** (below), if the probe is about
|
||||
ceremony's own code rather than about a bare API call.
|
||||
3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole
|
||||
point of the venue and the one step that cannot be shortcut. A `curl` from a
|
||||
laptop with a PAT answers a different question — see the 204/500 split
|
||||
above — and a probe run that way is worse than no probe, because it produces
|
||||
a confident wrong answer.
|
||||
4. **The job writes its raw results into an issue in the PROBE repo** —
|
||||
`heavy-duty/ceremony-runner-probe` — not into ceremony. Logs age out;
|
||||
ceremony#192's run 701 survived only because the job wrote its findings
|
||||
into an issue it created.
|
||||
5. **A human then records the probe issue's URL and the Actions run number on
|
||||
the ceremony issue the probe serves.** That hop is deliberate and is the
|
||||
whole of the boundary: the probe workflow holds no credential and no code
|
||||
path that can write to `heavy-duty/ceremony`, so "the probe reports its
|
||||
findings" and "the probe cannot touch the live board" stay compatible
|
||||
rather than contradicting each other (@codex-reviewer-andresmgsl, #202
|
||||
review).
|
||||
|
||||
## Arming a candidate ref
|
||||
|
||||
A probe that exercises ceremony's own machinery needs the candidate tree
|
||||
reachable from a `uses:` line. The shape is the drill rehearsal's, reused
|
||||
rather than reinvented (`drills/README.md` step 2):
|
||||
|
||||
1. **The candidate is a commit SHA, on a fork ref.** Push the candidate tree to
|
||||
a fork under the identity running the probe —
|
||||
`<identity>/ceremony@probe-<issue>` — and take its canonical SHA. Never
|
||||
create a branch on `heavy-duty/ceremony` named like a tag: it shadows that
|
||||
tag for every consumer until somebody remembers to delete it.
|
||||
2. **Install `docs/CONSUMERS.md`'s caller stubs** in the probe repo, pinned to
|
||||
that SHA — every `uses:` reference, not some of them.
|
||||
3. **Rewrite `CEREMONY_SELF_REF` in the candidate itself** to the same
|
||||
canonical SHA. A candidate carries the tag it has not created yet, so the
|
||||
self-consumption checkout cannot resolve until it is rewritten. Both
|
||||
carriers — `labels.yml` and `release.yml` — take the same value.
|
||||
4. **Invoke the probe by the event it is about**, and record which: a
|
||||
`workflow_dispatch` of the caller, or the real board event the probe is
|
||||
testing. A probe that fires a different event than the one under test
|
||||
proves something else.
|
||||
5. **Record the fork ref, the rewritten pin, the workflow invoked and the run
|
||||
number** in the probe repo's result issue. Those four are what make the
|
||||
result reproducible; without the pin especially, a later reader cannot tell
|
||||
which tree answered.
|
||||
6. **Reset removes the candidate-specific fixtures**: the caller stubs, the
|
||||
probe workflow and the result branch, so the next probe does not inherit a
|
||||
pin it did not choose. The result *issue* stays — it is the evidence.
|
||||
|
||||
## Who may reset it
|
||||
|
||||
**Operator-owned until ruled otherwise.** #202's task 4 asks who may reset the
|
||||
venue, and creating the repo is the operator's step, so the access policy is
|
||||
his to set at the same time (@codex-reviewer-andresmgsl, #202 review).
|
||||
|
||||
Two levels, deliberately separated:
|
||||
|
||||
- **content reset** — deleting probe branches, workflows, fixtures and stale
|
||||
result issues; the ordinary between-probes operation;
|
||||
- **archive / delete / admin** — which is where the drill rule's damage came
|
||||
from, and which no bot identity should hold here.
|
||||
|
||||
If fleet identities are given push access for content reset, this section
|
||||
records that; until then, ask.
|
||||
|
||||
## What must never happen here
|
||||
|
||||
No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no
|
||||
runs attributable to a probe. The venue exists so that the live board does not
|
||||
have to be the test fixture.
|
||||
|
||||
## The probes this venue owes
|
||||
|
||||
- **ceremony#192** — that the repaired sweep actually lifts a label under the
|
||||
workflow token, which is the half its acceptance criteria cannot get from
|
||||
the hermetic contract tests.
|
||||
- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches`
|
||||
works on this instance with a valid ref and inputs. Measured so far:
|
||||
`GET /actions/workflows` 404s and the dispatch route answers 500 rather than
|
||||
a 4xx, which is not enough to port against.
|
||||
- A 0.6.0 consumer exercise once ceremony#198 has merged.
|
||||
|
|
@ -111,87 +111,10 @@ missing or blank. A waived drill is still a record: the file says WAIVED and
|
|||
why — a maintainer's call, visible and reviewable in the release PR's diff,
|
||||
never a silent skip.
|
||||
|
||||
# The runner-probe venue
|
||||
---
|
||||
|
||||
**Not a drill.** A drill rehearses the release doors on a disposable repo and
|
||||
ends. This is the opposite shape: one **standing** repo that exists so that
|
||||
runner-only facts can be measured on demand, and it is **never archived**.
|
||||
|
||||
`heavy-duty/ceremony-runner-probe` — private, standing, reset between probes.
|
||||
Ruled by the operator as option (A) of ceremony#202 (#5631).
|
||||
|
||||
## Why a standing repo, when drills are disposable
|
||||
|
||||
Some facts are only true inside Actions, under the token Actions injects, and
|
||||
no local harness or PAT can reproduce them. The worked example is ceremony#192:
|
||||
|
||||
```
|
||||
DELETE /issues/{n}/labels/{id} -> 500 under ${{ github.token }} in a workflow
|
||||
DELETE /issues/{n}/labels/{id} -> 204 under a maintainer PAT, same call
|
||||
```
|
||||
|
||||
A probe that runs anywhere else passes and proves nothing. Before this venue
|
||||
existed the answer was "un-archive a drill repo", which was requested three
|
||||
times in two days across two issues and never became anything — the three
|
||||
drill repos (`ceremony-drill-0.4.1`, `-0.4.1-final`, `-191`) are all archived,
|
||||
and each was minted for one probe and then wanted again.
|
||||
|
||||
## The disposal rule above does NOT apply here
|
||||
|
||||
The rehearsal section says the builder archives the scratch repo and the
|
||||
operator deletes it. **That rule is for drills.** Archiving this repo defeats
|
||||
its entire purpose, and it is the failure mode the three archived drill repos
|
||||
demonstrate — each was archived correctly, by the rule, and each then had to be
|
||||
un-archived or replaced.
|
||||
|
||||
So: never archive it, never delete it, and if you find it archived, un-archive
|
||||
it rather than minting a fourth one.
|
||||
|
||||
## Standing it up is the operator's step
|
||||
|
||||
Bot identities cannot create repositories in `heavy-duty`. Measured
|
||||
2026-08-05 with a fleet identity holding the `repo` scope:
|
||||
|
||||
```
|
||||
POST /api/v1/orgs/heavy-duty/repos -> 403 "not allowed to create repository in organization"
|
||||
POST /api/v1/user/repos -> 201 (personal namespace only)
|
||||
```
|
||||
|
||||
This is the same shape as the drill delete: a deliberate permission boundary,
|
||||
not a misconfiguration, and the same rule applies — do not retry it, and do not
|
||||
work around it by putting the venue in a personal namespace, where the org's
|
||||
runner and secrets do not reach. Ask the operator.
|
||||
|
||||
## Running a probe
|
||||
|
||||
1. Reset the repo to a clean state — the probe's own fixtures only, no
|
||||
leftovers from the last one. A probe that inherits state is a probe whose
|
||||
result you cannot attribute.
|
||||
2. Install whatever the probe needs: a workflow that exercises the call, and
|
||||
`docs/CONSUMERS.md`'s caller stubs if the probe is about the reconcilers.
|
||||
3. **Run it as an Actions job under `${{ github.token }}`.** This is the whole
|
||||
point of the venue and the one step that cannot be shortcut. A `curl` from a
|
||||
laptop with a PAT answers a different question — see the 204/500 split
|
||||
above — and a probe run that way is worse than no probe, because it produces
|
||||
a confident wrong answer.
|
||||
4. Write the results **into the forge** — an issue or a comment in this repo —
|
||||
rather than leaving them in a job log. Logs age out; ceremony#192's run 701
|
||||
survived because the job wrote its findings into an issue it created.
|
||||
5. Record the run number and the repo in whatever issue the probe serves.
|
||||
|
||||
## What must never happen here
|
||||
|
||||
No probe touches `heavy-duty/ceremony`'s board. No labels, no comments, no
|
||||
runs attributable to a probe. The venue exists so that the live board does not
|
||||
have to be the test fixture.
|
||||
|
||||
## The probes this venue owes
|
||||
|
||||
- **ceremony#192** — that the repaired sweep actually lifts a label under the
|
||||
workflow token, which is the half its acceptance criteria cannot get from
|
||||
the hermetic contract tests.
|
||||
- **ceremony#205** — whether `POST /actions/workflows/{file}/dispatches`
|
||||
works on this instance with a valid ref and inputs. Measured so far:
|
||||
`GET /actions/workflows` 404s and the dispatch route answers 500 rather than
|
||||
a 4xx, which is not enough to port against.
|
||||
- A 0.6.0 consumer exercise once ceremony#198 has merged.
|
||||
**Standing runner probes are not drills.** The disposal rule above — builder
|
||||
archives, operator deletes — is for the disposable scratch repo a drill runs
|
||||
in. `heavy-duty/ceremony-runner-probe` is the opposite shape: it stands, and
|
||||
archiving it is the failure mode that made all three previous drill repos
|
||||
unavailable. See [docs/RUNNER-PROBES.md](../docs/RUNNER-PROBES.md) (#202).
|
||||
|
|
|
|||
Loading…
Reference in a new issue