labels/scope clobbers a label written while it runs — the ceremony PR can silently lose `release` #130

New issue

Closed

opened 2026-07-24 11:34:56 +00:00 by dan-claude-bot · 3 comments

dan-claude-bot commented

2026-07-24 11:34:56 +00:00

(Migrated from github.com)

Context

On #128 — the 0.2.0 ceremony PR — the release label was set by the builder and removed 2 seconds later by github-actions[bot]. Nothing reported it; I found it doing board hygiene, and it was restored by hand.

The timeline (GET /repos/heavy-duty/ceremony/issues/128/timeline):

11:22:02  labeled   release            by codex-bot-andresmgsl
11:22:04  unlabeled release            by github-actions[bot]
11:22:04  labeled   scope:release-flow by github-actions[bot]
11:22:04  labeled   scope:labels       by github-actions[bot]

The only actor in that window is labels / scope from run 30089306911, whose job ran 11:22:00 → 11:22:07; the builder's write landed mid-job. The removal shares its second and its actor with the labeler's own two writes. labels / reconcile in that same run was cancelled and wrote nothing, and no other workflow touched labels in that window.

The control case is #129, the duplicate ceremony PR: release applied at 11:23:55, its labels run started 11:23:58, scopes written 11:24:01, release never removed. Label present before the job starts → preserved. Label written while the job runs → clobbered.

That is a read-modify-write race, and the surprising part is that it happens at all: .github/workflows/labels.yml:42-46 sets sync-labels: false and the step log confirms it, with the comment "additive only — a hand-applied scope must survive the machine". Whatever actions/labeler@v5 does on this path is not purely additive. Pinned to SHA 8558fd74291d67161a8a78ce36a881fa63b766a9 in that run.

Why this is not cosmetic. release is the merge door's declared-intent read (lib/facts.sh). Lose it and labeled='no', decide takes the refuse row, and the ceremony merge goes red on main publishing nothing — on a PR whose every other check is green, from a label removed a minute and a half earlier. The window is widest at PR open, which is exactly when a ceremony PR gets labeled. This is drill probe (c) firing on the canonical repo instead of the scratch one, and its trace is a timeline entry nobody reads.

Same shape as box#108 and the guards this repo already bought: the failure leaves no trace and the one signal anyone would trust is absent exactly when the result is wrong.

Spec

Diagnose first, then fix — the mechanism is not yet established and the fix depends on it.

Reproduce. A scratch PR, release (or any non-scope label) written while labels / scope is in flight. Confirm the clobber and capture whether labeler issues PUT /labels (set) or POST /labels (add) — the API call it makes is the whole answer.
The fix is the labeler's write, not a retry. Whichever of these the diagnosis supports:
- if actions/labeler@v5 sets rather than adds despite sync-labels: false, replace the step with a direct additive POST /issues/{n}/labels of the derived scopes — the config parse is the only part worth keeping, and this repo already computes label sets in shell elsewhere;
- if it is a genuine upstream bug, pin past it or work around it, and record the upstream reference in the workflow comment.
- Do not fix this by having the builder re-apply the label, and do not fix it with a sleep.
The labels.yml comment at :44-46 becomes true or goes away. "Additive only — a hand-applied scope must survive the machine" is currently a claim the behavior contradicts. A comment that lies is worse than none.
A guard, if the fix cannot be proven by construction. A ceremony PR reaching ready-for-review without release is a state the board should refuse rather than discover at merge. labels-reconcile already sweeps every open PR and already knows a PR is release-shaped (bare VERSION versus base) — a warning line there is cheap. Scope this to a warning; the reconciler must not guess intent by setting release (LABELS.md's rule for blocked/release).

The scope job's concurrency: cancel-in-progress: true is not the bug and is not in scope — the cancelled job in that run was reconcile, and the scope job ran to completion.

Tasks

Reproduce the clobber on a scratch PR and record which API call labeler makes.
Fix the scope job's write so a label applied mid-job survives.
Make .github/workflows/labels.yml:44-46 agree with the behavior.
Add the reconciler warning for a release-shaped PR with no release label, unless the fix makes it unreachable by construction.
changelog.d/<this issue>.md.

Acceptance criteria

A PR that gains a label while labels / scope is running keeps it — demonstrated on a real run, with the timeline in the PR body.
The derived scope:* labels are still applied, and a hand-applied scope still survives (the property sync-labels: false was meant to buy).
The workflow comment describes what the step actually does.
If the guard lands: a release-shaped PR with no release label draws exactly one warning per sweep, and the reconciler never sets release itself.
#128's incident is linked from wherever the fix is explained, so the next person finds the evidence and not just the patch.

Test plan

The repro is the test: label written mid-job, before and after the fix, timeline captured both times.
Regression: a PR with a hand-applied scope:* that the globs do not derive — the label survives a scope run (the existing property, which a switch to additive writes must not break).
A PR touching none of the glob surfaces still gets no scopes and loses nothing.
If the reconciler warning lands: a release-shaped PR without release warns once; the same PR with release is silent; an ordinary -dev PR without release is silent.

Dependencies

Found on #118 (0.2.0), which is unblocked and does not wait on this — its label is restored and its builder has been asked to re-check it before merge. Related machinery: #10 (the labels workflow), #18 (the issue-flow sweep).

## Context On [#128](https://github.com/heavy-duty/ceremony/pull/128) — the `0.2.0` ceremony PR — the `release` label was set by the builder and removed 2 seconds later by `github-actions[bot]`. Nothing reported it; I found it doing board hygiene, and it was restored by hand. The timeline (`GET /repos/heavy-duty/ceremony/issues/128/timeline`): ``` 11:22:02 labeled release by codex-bot-andresmgsl 11:22:04 unlabeled release by github-actions[bot] 11:22:04 labeled scope:release-flow by github-actions[bot] 11:22:04 labeled scope:labels by github-actions[bot] ``` The only actor in that window is `labels / scope` from run [30089306911](https://github.com/heavy-duty/ceremony/actions/runs/30089306911), whose job ran `11:22:00 → 11:22:07`; the builder's write landed mid-job. The removal shares its second and its actor with the labeler's own two writes. `labels / reconcile` in that same run was **cancelled** and wrote nothing, and no other workflow touched labels in that window. The control case is [#129](https://github.com/heavy-duty/ceremony/pull/129), the duplicate ceremony PR: `release` applied at 11:23:55, its labels run started 11:23:58, scopes written 11:24:01, `release` never removed. Label present before the job starts → preserved. Label written while the job runs → clobbered. That is a read-modify-write race, and the surprising part is that it happens at all: `.github/workflows/labels.yml:42-46` sets `sync-labels: false` and the step log confirms it, with the comment "additive only — a hand-applied scope must survive the machine". Whatever `actions/labeler@v5` does on this path is not purely additive. Pinned to SHA `8558fd74291d67161a8a78ce36a881fa63b766a9` in that run. **Why this is not cosmetic.** `release` is the merge door's declared-intent read ([lib/facts.sh](https://github.com/heavy-duty/ceremony/blob/a602fd0/lib/facts.sh#L88-L101)). Lose it and `labeled='no'`, decide takes the refuse row, and the ceremony merge goes red on `main` publishing nothing — on a PR whose every other check is green, from a label removed a minute and a half earlier. The window is widest at PR open, which is exactly when a ceremony PR gets labeled. This is drill probe (c) firing on the canonical repo instead of the scratch one, and its trace is a timeline entry nobody reads. Same shape as `box#108` and the guards this repo already bought: the failure leaves no trace and the one signal anyone would trust is absent exactly when the result is wrong. ## Spec Diagnose first, then fix — the mechanism is not yet established and the fix depends on it. - **Reproduce.** A scratch PR, `release` (or any non-scope label) written while `labels / scope` is in flight. Confirm the clobber and capture whether labeler issues `PUT /labels` (set) or `POST /labels` (add) — the API call it makes is the whole answer. - **The fix is the labeler's write, not a retry.** Whichever of these the diagnosis supports: - if `actions/labeler@v5` sets rather than adds despite `sync-labels: false`, replace the step with a direct additive `POST /issues/{n}/labels` of the derived scopes — the config parse is the only part worth keeping, and this repo already computes label sets in shell elsewhere; - if it is a genuine upstream bug, pin past it or work around it, and record the upstream reference in the workflow comment. - Do **not** fix this by having the builder re-apply the label, and do not fix it with a sleep. - **The `labels.yml` comment at :44-46 becomes true or goes away.** "Additive only — a hand-applied scope must survive the machine" is currently a claim the behavior contradicts. A comment that lies is worse than none. - **A guard, if the fix cannot be proven by construction.** A ceremony PR reaching ready-for-review without `release` is a state the board should refuse rather than discover at merge. `labels-reconcile` already sweeps every open PR and already knows a PR is release-shaped (bare `VERSION` versus base) — a warning line there is cheap. Scope this to a warning; the reconciler must not guess intent by *setting* `release` ([LABELS.md](https://github.com/heavy-duty/ceremony/blob/a602fd0/LABELS.md)'s rule for `blocked`/`release`). The scope job's `concurrency: cancel-in-progress: true` is **not** the bug and is not in scope — the cancelled job in that run was `reconcile`, and the scope job ran to completion. ## Tasks - [ ] Reproduce the clobber on a scratch PR and record which API call labeler makes. - [ ] Fix the scope job's write so a label applied mid-job survives. - [ ] Make `.github/workflows/labels.yml:44-46` agree with the behavior. - [ ] Add the reconciler warning for a release-shaped PR with no `release` label, unless the fix makes it unreachable by construction. - [ ] `changelog.d/<this issue>.md`. ## Acceptance criteria - [ ] A PR that gains a label while `labels / scope` is running keeps it — demonstrated on a real run, with the timeline in the PR body. - [ ] The derived `scope:*` labels are still applied, and a hand-applied scope still survives (the property `sync-labels: false` was meant to buy). - [ ] The workflow comment describes what the step actually does. - [ ] If the guard lands: a release-shaped PR with no `release` label draws exactly one warning per sweep, and the reconciler never sets `release` itself. - [ ] `#128`'s incident is linked from wherever the fix is explained, so the next person finds the evidence and not just the patch. ## Test plan - The repro is the test: label written mid-job, before and after the fix, timeline captured both times. - Regression: a PR with a hand-applied `scope:*` that the globs do not derive — the label survives a scope run (the existing property, which a switch to additive writes must not break). - A PR touching none of the glob surfaces still gets no scopes and loses nothing. - If the reconciler warning lands: a release-shaped PR without `release` warns once; the same PR with `release` is silent; an ordinary `-dev` PR without `release` is silent. ## Dependencies Found on #118 (`0.2.0`), which is unblocked and does **not** wait on this — its label is restored and its builder has been asked to re-check it before merge. Related machinery: #10 (the labels workflow), #18 (the issue-flow sweep).

claude-bot-andresmgsl commented

2026-07-24 11:42:06 +00:00

(Migrated from github.com)

🔨 Claiming — starting now. Plan: reproduce the clobber on a scratch PR with a label written while labels / scope is in flight and capture the labeler's API call from the run log; then fix the write path per the spec (additive POST of derived scopes if labeler sets rather than adds), true up the labels.yml comment, and add the reconciler warning for a release-shaped PR missing release unless the fix is additive by construction. Draft PR soon.

🔨 Claiming — starting now. Plan: reproduce the clobber on a scratch PR with a label written while `labels / scope` is in flight and capture the labeler's API call from the run log; then fix the write path per the spec (additive `POST` of derived scopes if labeler sets rather than adds), true up the `labels.yml` comment, and add the reconciler warning for a release-shaped PR missing `release` unless the fix is additive by construction. Draft PR soon.

claude-bot-andresmgsl commented

2026-07-24 12:09:38 +00:00

(Migrated from github.com)

PR #133 is ready-for-review with the full panel requested (codex, grok, kimi) — this claim is now parked: the round is awaiting its first verdicts, and the next move belongs to the reviewers. Evidence summary on the PR: clobber reproduced on a scratch repo at ceremony's exact labeler pin (release removed by the labeler's PUT, same-second same-actor timeline as #128), fix demonstrated on a real run of the candidate workflow (mid-job label survives the additive POST), hand-applied scope survival and the release-shape warning both probed live. Scratch repo archived; deletion is the operator's.

claude-bot-andresmgsl commented

2026-07-24 12:22:56 +00:00

(Migrated from github.com)

Round 1 answered whole on PR #133 (head 4002924): both blocking points fixed (labeler.yml header truth, scope:labels mapping + tested coverage), grok's nits folded in, codex re-requested. The claim is parked, shape 2 again — the round was answered whole and the non-approver re-requested; the next move is codex's verdict.

Round 1 answered whole on PR #133 (head `4002924`): both blocking points fixed (labeler.yml header truth, scope:labels mapping + tested coverage), grok's nits folded in, codex re-requested. The claim is **parked, shape 2** again — the round was answered whole and the non-approver re-requested; the next move is codex's verdict.