Found by the first post-merge sweep after the 0.6.0 merge — #198's own
acceptance probe — not by review. Three PRs in one run:
labels: #208: could not read the head commit's date:
forge_api: HTTP 404 from 'GET repos/heavy-duty/ceremony/commits/f3a1336…'
— blocker:unrequested not judged this pass
Measured against this instance:
forgejo repos/{o}/{r}/commits/{sha} -> 404
forgejo repos/{o}/{r}/git/commits/{sha} -> 200, date under `.created`
github repos/{o}/{r}/commits/{sha} -> 200, date nested
A fourth asymmetry, alongside the three lib/forge-forgejo.sh's header already
records. #198 ported this call site onto the shim with GitHub's path unchanged
— correct against GitHub, and the block it lives in (#236 D2) arrived WITH the
merge, so nothing here had ever executed it.
So it becomes a verb rather than a path at the call site: the caller wants one
timestamp and should not have to know either shape.
Cost while it stood was bounded and loud rather than silent — guarded_read
refused and the sweep said so — but blocker:unrequested could never be judged
on this forge.
The tests pin each backend's PATH and FIELD, because a stubbed forge_api cannot
catch a wrong path; that is exactly how this shipped and why it took a live
sweep to find. Swapping the paths reds the forgejo pair; swapping the fields
reds the github one.
test/run.sh 28/28 under jq 1.7 and jq 1.6; forge-backends 124/124; shellcheck
0.10.0 and actionlint clean.
Refs #209
@codex-reviewer-andresmgsl's four gaps, all real, all taken.
1. PRESERVED IDS COME FROM THE ISSUE. The removal path read only .labels[].name
and then re-resolved every preserved label through the repository-wide list
— so preservation depended on a paginated read with nothing to do with this
issue, and an incomplete one would drop a bystander. It now keeps
name<TAB>id from the issue payload, subtracts removals by name, and resolves
ONLY added names. Fixture: a bystander on the issue with id 14 that is
absent from the repo-list fixture entirely must still survive the PUT.
2. AN ABSENT REMOVAL WRITES NOTHING. I had it PUT the unchanged set, arguing
the write proved the sweep reached the forge. The GET already proves that,
and replacing a set with itself opens ceremony#128's window for no state
change — most calls here are exactly this case, since the reconcilers call
--remove-label unconditionally. Short-circuits when the wanted set equals
the current one. This was the policy-shaped choice flagged for @andres; the
reviewer's reasoning is better than mine was.
3. EVERY LABEL MUTATION REACHES THE TALLY. The marker was only on the primary
state edit, so clearing `merge-next` and both `stale` edits could fail into
the generic per-PR branch and still finish `reconciled.` and exit 0. All
four sites go through one `label_write` helper, so a future call site cannot
reopen it by forgetting to mark itself. Probe: a failed NON-primary write
(unstale on a blocked PR) must fail the sweep.
4. NO SUCCESS TOKEN IN A FAILURE TAIL. "NOT reconciled." still contains
"reconciled.", which a log-tail consumer greps for. The line is now
"sweep incomplete", and the test asserts the whole output is free of the
token rather than only of the success prefix.
Also added the two fault boundaries the acceptance plan named and the fixtures
never proved: a failed current-label GET and a failed replacement PUT, each
non-zero with the backend's verb/path/status diagnostic.
Mutation-tested, each gap separately: bypassing the tally reds 3, re-resolving
preserved ids reds 7, writing the unchanged set reds 1.
forge-backends 115/115 (was 110), labels-reconcile 175/175 (was 172),
test/run.sh 22/22 under jq 1.7 and jq 1.6, shellcheck 0.10.0 and actionlint
clean.
Refs #192
Two defects, one cause, and the second is why the first survived a week.
THE WRITE. Removal was a per-label `DELETE .../labels/{id}` loop. On this
instance that call returns HTTP 500 for every removal under the token the
sweep actually holds — measured inside Actions, probe run 701, where the same
`PUT .../labels` with the desired full set returns 200 including the empty set
for a full clear. A PAT gets 204 on the same DELETE, which is exactly why it
went unseen: it fails only for `${{ github.token }}`.
Net effect before this: on Forgejo the state machine could only ever ADD
labels. Every `state:*` transition needing the previous state cleared and every
`blocker:*` that should lift was inert. Both PRs open right now carry stale
`blocker:*` labels that are false and that nothing can remove.
So the removal path is read-current, compute-wanted, one PUT — the same shape
the assignee branch beside it already used. An ADD-ONLY call keeps its additive
POST: ceremony#128 lost a `release` label to a read-modify-write that clobbered
a concurrent set, and forge_labels_add stays pinned against ever doing that.
The window is accepted here and only here, where the caller asked to REMOVE
and no additive verb can say that. An unresolvable --add-label refuses before
any write, so a replacement PUT can never drop a label nobody asked to remove.
THE REPORTING. `labels-reconcile` logged `WARNING: label edit failed`, fell
through, and `main` printed `reconciled.` and exited 0 — while
`issueflow-reconcile` treated the identical 500 as fatal. One cause, two
contradictory policies, and the wrong one hid the write defect.
A failed write is fatal now, and the tally reaches main's exit code. That
second half is load-bearing: making reconcile_pr fatal alone is not enough,
because the loop swallows a per-PR non-zero into a log line and finishes. The
per-PR tolerance is right and stays — one bad PR must not blind the board — but
it now applies to READS. A sweep that could not write exits non-zero and never
prints `reconciled.`
The diagnostic says what was attempted and that it did not happen. The old text
blamed a missing label and told the operator to bootstrap, when the label was
present and the call returned 500 — #101's rule is report, do not diagnose.
Mutation-tested, all three ways: restoring the warn-and-continue reds 5 cases,
removing the tally reds 2, restoring the DELETE loop reds 7.
test/run.sh 22 files 0 failed under jq 1.7 and jq 1.6; shellcheck 0.10.0 and
actionlint clean.
Refs #192
`git merge` of upstream `8c3a4d1` onto `dad99dd`, common ancestor `84bb1a4`.
18 hunks in 10 files; `lib/forge.sh`, `lib/forge-github.sh` and
`lib/forge-forgejo.sh` conflict in none and come out byte-identical.
The resolutions the issue decided: VERSION and both CEREMONY_SELF_REF
carriers take upstream's numbers; `.github/labels.conf` and `drills/0.4.1.md`
keep this forge's; CHANGELOG keeps both sides and names the upstream commit
this tree carries.
The part the hunks did not contain. Upstream's 0.5.0/0.6.0 work added whole
functions to files this tree already owned, so `git merge` took its side
without raising a conflict — and with them, EIGHT runtime `gh` call sites
that #188 had removed. Seven are ported onto the shim: two reads and four
comment writes in issueflow-reconcile, and labels-reconcile's HEAD_COMMIT_AT
read. The eighth is `gh workflow run` in labels.yml, which a workflow cannot
declare a client for and whose Forgejo equivalent this instance answers with
500 rather than a 4xx — named with its reason rather than ported on a guess.
test/no-runtime-gh.test.sh makes the rule mechanical, because reviewing the
diff could not: four reviewers reading it each found a different subset, and
the contract suite stubs `gh`, so a reintroduced call site passes it.
Three seams the resolution decides are silent when resolved wrongly, and each
now has a case that fails on the wrong one: the merged record's `merged_at`
third column (without it every sort key ties and the highest PR number comes
back), the open gather's one-BODY-row-per-line feed (a whole decoded body as
one record loses every declaration including the first), and the whole-board
read whose COLLISION_FLAGS/WINDOW_FLAGS consumers auto-merged.
The open gather carries CLOSING rows as well as BODY rows. `Refs` alone would
drop every `Closes #N` link on the open side and reclaim a claim the PR was
holding — the existing base64 round-trip case is red without it.
actions/refs-not-closing declares CEREMONY_FORGE_CLIENT=gh: its only gather
is GraphQL, which Forgejo does not serve at all. #199 ports it.
test/run.sh: 28 test files, 0 failed. shellcheck and actionlint clean.
Refs #198
`gh api` prints a 5xx response body to stdout AND exits non-zero, and
GitHub's 5xx body is a JSON object. Inside the per-issue subshell that
payload passed `has("pull_request") | not`, emptied `.labels[]`, and
`queue_decision` — correct on the input it was handed — wrote
`needs-triage` onto a healthy epic. The run then logged `reconciled.`
and exited 0 (crew#329, #247).
errexit could not have caught it: a command whose status is tested by
`||` runs with errexit suppressed, and the suppression extends through
the whole subshell body, so the `|| log` handler is what disables the
errexit that would have aborted at the failed read. Removing the handler
revives errexit and loses #91's resilience, and an inline `set -e` does
not re-arm it. Explicit per-read checks are the mechanism.
Every read inside that subshell is now checked — the issue read on its
status AND on its payload shape (an HTTP 200 whose body is `null` exits
0 and empties the label set just the same), both reads in
`last_issue_activity`, and the comments read in
`issue_comment_has_marker`. On failure the issue is left exactly as it
is, the reason rides its own `#$n:` line, and the subshell exits with a
distinguished status the sweep counts, so a deliberate skip is not
reported as a crash and a genuine crash is still named byte-identically.
`read_failure_reason` moves to lib/read.sh beside a new `guarded_read`,
sourced by both reconcilers: labels-reconcile's copy was the only one,
and the issue surface needs the identical rule.
Refs #247
The predicate's fixtures cannot see the read that sets HEAD_COMMIT_AT, so a
sweep probe drives it both ways: read, and the blocker is written off a dated
head; denied, and the denial is named on its own line while the state still
converges — this read narrows one blocker, it does not skip the PR the way an
unreadable rollup does. Renaming the assignment reds the probe.
The read also moves after the mergeability/checks skip: a PR the sweep walks
away from must not pay for a call whose only consumer is a blocker that pass
will never decide.
Refs #236
`blocker:unrequested` is the one blocker that names an act the author must
perform, and it never knew whether performing it was permitted. BUILDER.md's
review round requires a green check at the head before requesting, so a
builder waiting out a pending run is complying — and the blocker fired on
compliance (crew#318 ~12:44Z, ceremony#235 12:30Z, both 2026-08-03).
Gate the branch on CHECKS ∈ SUCCESS | NONE (D1): PENDING is CI's move, which
state:addressing already says, and FAILURE belongs to blocker:ci-red rather
than to a second label on the same stall. Then require the supporting facts —
the head's own date and the round's newest submitted review — to have stood
for RECONCILE_UNREQUESTED_GRACE (default 300s, D2), measured off those
timestamps because this sweep is stateless per pass. A timestamp that cannot
be read refuses the blocker.
Refs #236
Codex 1566 held APPROVE: only Forgejo stubs covered forge_timeline and
forge_pr_activity. Pin the github twins as 1:1 extractions (timeline
paginate; issue comments + flat pulls comments + commits).
Cluade #4879: drop 2>/dev/null on the labels-reconcile activity call site
so a failed read still degrades last_activity but names the failure in the
job log (keep || true).
Panel-unanimous batch that was staged unpushed on 57abe15 (#4853):
- forge_timeline: project Forgejo label events into the GitHub shape
so the ruling ladder fires on this forge (measured mapping #4849)
- forge_pr_activity: stop calling /pulls/{n}/comments (404 here); use
reviews with comments_count > 0 for inline comments (#4844)
- ci.yml: install shellcheck before lint, mirroring actionlint — the
act-22.04 runner image does not ship it
Status captured before jq so an unreadable timeline cannot report empty.
Term 4. GitHub clears requested_reviewers when a verdict lands, so the field
answers "who still owes a verdict" by itself. Forgejo never clears it —
measured: rig!140 listed all three panelists with all three verdicts in, and
rig!146 still lists three while MERGED, so the field is stale even on a
closed PR.
Read raw on Forgejo that is not a cosmetic over-count. `requested` drives
three decisions, and a permanently-true field pins a PR at
state:bots-reviewing for life and stops blocker:unrequested from ever being
true: the sweep believes a round is live forever and no staleness can
correct it.
So the requested set is intersected with who has NOT submitted a verdict for
the current head, derived from /pulls/{n}/reviews — the read that is true on
both forges. On GitHub the filter removes nothing, because the field is
already accurate; term 5 holds by construction rather than by care.
A STALE approval — an approval of an older head — still owes a verdict. That
is the case that matters: treating it as answered would let a stale round
read as complete, which is the shape #136 exists to prevent.
Mutation-verified both ways: reading the field raw again reds three cases,
and treating STALE as answered reds two.
Also documents @grok-reviewer-andresmgsl's ask (#4763): every panel= account
must be able to read the repo, or the forge refuses the review request —
422 naming the account on Forgejo. A real failure mode for private
consumers, and it fails loudly rather than sweeping blind.
Refs #188
Term 1 completed. All 52 runtime gh call sites in the three reconcilers and
lib/ruling.sh now go through forge_* verbs; the three remaining matches in
labels-reconcile are prose in comments. lib/facts.sh is deliberately
untouched — it is the release door, and the ruling keeps release.yml out of
this issue.
The CEREMONY_FORGE_CLIENT:-gh wrappers die here, in the same commit as the
sites they described, so the tree is never in a state where the declaration
lies. main() now runs forge_preflight then forge_select "".
Two sites needed judgment rather than substitution:
- labels-scope's write is forge_labels_add, a genuine additive POST on
both backends, NOT forge_issue_edit --add-label. ceremony#128 turns on
that write not being a read-modify-PUT: the labeler action computed
(labels-at-job-start union derived) and PUT the whole set, silently
dropping a label applied while the job ran. Routing it through a generic
edit verb would have quietly reopened that.
- the human-review request is forge_request_reviewer. Contrary to my
earlier reading, POST /pulls/{n}/requested_reviewers DOES exist on
Forgejo — 422 naming the reviewer's access without it, 201 with it. The
earlier 404 was a GET, which the endpoint does not serve, plus a
username that did not exist.
Test churn, all of it the term-5 boundary move:
- the suites select the github backend, so their existing gh() stubs stay
the boundary and keep intercepting;
- stubs strip the paging the shim injects, so fixtures stay keyed on the
logical endpoint (inlined in the PATH stub, which is a standalone
executable and cannot see a shell function);
- fixtures renamed off the per_page suffix for the same reason;
- recorded-mutation assertions now match the verb, not the raw gh line;
- gh() stubs carry SC2317: they are reached through the backend now, so
shellcheck can no longer see the call path.
Refs #188
The preflight half of #188, landed first so it stands alone: the forge is
decided once, before any sweep, and a client that cannot speak it exits
non-zero with a named reason.
Measured against forgejo.heavyduty.builders at 84bb1a4 — two of the three
actions reported SUCCESS having read nothing:
labels-scope exit 0 "no .github/labeler.yml" (the file is HTTP 200)
labels-reconcile exit 0 "reconciled." (zero PRs enumerated)
issueflow-reconcile exit 1 "unexpected end of JSON input"
labels-reconcile's blind-sweep warning (#96) could not fire: it counts
unreadable PRs against a list `gh pr list` never produced, and a process
substitution's failure does not trip set -e, so total stayed 0. Installing
gh makes it worse, silencing the one loud failure.
Detection is measured, not inferred from docs: a real forgejo-runner v6.3.1
job (probe task 278) shows Forgejo populating the whole GITHUB_* namespace,
so GITHUB_ACTIONS proves nothing. GITHUB_API_URL's shape, GITEA_ACTIONS and
GITHUB_SERVER_URL do. The same probe shows the runner image carries neither
gh nor stoke, which is what makes the forgejo backend REST.
Tests declare CEREMONY_FORGE at the forge boundary rather than stubbing gh
and staying silent about the forge — the boundary move term 5 asks for.
Refs #188
panel[z]]=b parsed at the round-1 head: the case pattern only proves
some ]= occurs, so the stray ] stayed inside the login and the real
author silently fell back to the base panel — the misroute D4 exists to
refuse. The login charset is now enforced with the bracket-specific
diagnostic; codex's probe and an invalid-character row are the new
must-fail fixtures.
Refs #224
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
round_outranks_draft consults the round before draft short-circuits: a
re-drafted PR carrying CHANGES_REQUESTED, an owed round-reply, or
push-staled approvals reads state:addressing; a live panel request on a
draft surfaces as state:bots-reviewing rather than being absorbed
(the must-not-paper-over combination, decided as: visible). Approvals do
not outrank draft, so a draft never reads needs-human, and a virgin
draft is byte-identical to before. LABELS.md's state:building row makes
draft evidence, not the definition.
Refs #205
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One resolution point (panel_for_author) feeds set_required_bots; the
author's row when the conf defines one, the base panel= otherwise, minus
the author in either case. Bracket prefixes are matched quoted so the
case patterns cannot glob (D7, panela= tripwire). configured_label_rows
skips the rows so a dispatch bootstrap cannot mint a label named after
one. BUILDER.md/REVIEWER.md carry the one D9 wording; CONSUMERS.md
publishes the row as unreleased with the parse-failure warning.
Refs #224
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The shared reconcile concurrency group displaces queued sweeps as
CANCELLED, and the displaced run's successor attaches to a different PR —
so on the victim the newest self entry stayed CANCELLED, scored FAILURE,
and the sweep set blocker:ci-red off its own corpse every cadence
(crew#227). Drop rollup entries whose workflowName matches SELF_WORKFLOW
(defaulting to the ambient GITHUB_WORKFLOW — the caller's name, so no
workflow edit and no hardcoded consumer name) before the newest-per-context
collapse; an empty name filters nothing. A self-only rollup now honestly
scores NONE, and a genuine foreign failure still blocks beside a cancelled
self entry — the must-fail guard against re-opening #136.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
During the ceremony#111 freeze, rig#126/#128 carried blocked beside
state:needs-human — the round had finished, but the hold said the merge
must not happen, and rig#126 was merged seven minutes after the
reconciler wrote the green label. decide_state() only joined the two
axes through blockers(), which emits branch facts; the hand-set blocked
label was never consulted.
blocked becomes the second exclusion on state:needs-human, exactly
parallel to needs-ruling: round says needs-human + has_label blocked ->
state:addressing. Deliberately not a blockers() emission — BLOCKERS is
machine-owned and the converge loop would strip the live hold on the
next tick, the same trap #51 names for needs-ruling.
Ruling record: discussion 122, armed default A fired 2026-07-25T09:00Z.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
checks_state discards a CANCELLED entry only when its context group holds
at least one non-cancelled sibling — before the sort, so the duplicate the
repo-global reconcile queue evicted (after it had already attached a check
to the head) cannot outvote the success that did its work (#136a17e497,
#1334002924). An all-cancelled context never reported at all and still
classifies FAILURE; {FAILURE older, CANCELLED newest} keeps its red.
The fixture that pinned the opposite rule imagined a cancelled run
replacing a success; it never saw one that replaced nothing. Rewritten
with its reason, plus the recorded a17e497 shape, the all-cancelled
groups, and the cancelled-over-FAILURE case.
Closes#139
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Bare X.Y.Z at the head where the base says otherwise, no release label,
not a draft: the sweep emits one :⚠️: per pass naming both
versions. A warning only — release is declared intent and the reconciler
never guesses intent (LABELS.md). Version read via the API, both
backends, jq not node; unreadable reads nag nobody. Plus the yq test
contract in CI and fixture tests for the guard matrix.
Part of #130.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reconciler's mergeability/checks read kept its correct degrade but
threw the reason away: 2>/dev/null dropped gh's stderr, leaving a
permanent denial and a network hiccup byte-identical in the log (#95 had
to infer a cause from a control case, and the inference did not survive
incubator#48/#49). Capture stderr into a variable via a temp file (D2),
emit it as its own '#N: read failed: …' line beside the byte-identical
counted line (D1), collapsed and bounded by a pure helper (D3/D4), and
lead blind_sweep_warning with the sampled observed reason, demoting the
permissions hint from stated cause to named candidate (D5).
Part of #101 groundwork; tests and changelog follow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LABELS.md publishes them as deleted at bootstrap; nothing deleted them —
incubator's first dispatch left `good first issue` standing. One registry
(retired_label_names) beside core_label_rows, dispatch-only, through run,
tolerant of absence and refusal (#93).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Issue side: the claim-reclaim clock stops under a pending ruling (the
decision still sees an unassigned claim), an already-applied stale heals
off, and reconcile_ruling runs for any flagged issue on any queue state.
PR side: reconcile_ruling rides the (#51) stale section's real-activity
computation. test/ruling.test.sh pins the window boundaries, newest-event
anchoring, per-event marker scoping, the markerless nudge reset, the
unreadable-timeline rule, and that no scenario writes a label.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
