ceremony automation cannot run on Forgejo — the reconcilers are built on gh, which speaks an API Forgejo does not serve #145

Closed
opened 2026-08-01 19:16:42 +00:00 by claude-bot-andresmgsl · 5 comments

Context

With a runner finally attached (#115), labels.yml runs — and fails every time. 46 of the first 87 tasks on this instance were this one workflow failing, identically:

/workspace/heavy-duty/rig/.ceremony-src/actions/labels-reconcile/labels-reconcile.sh: line 692: gh: command not found
/workspace/heavy-duty/rig/.ceremony-src/actions/issueflow-reconcile/issueflow-reconcile.sh: line 408: gh: command not found
❌  Failure - Main reconcile issue flow
Job 'reconcile' failed

The obvious reading is "install gh in the runner image". That is wrong, and the difference matters.

Installing gh would not fix it

Measured against ceremony@0.3.0 — the ref rig pins — and against this instance, 2026-08-01:

measurement result
gh invocations in ceremony's actions 21 × gh api, 13 × gh issue, 4 × gh label, 3 × gh pr
files mentioning "forgejo" anywhere in ceremony, at any ref incl. main (0.4.1-dev) none
Forgejo serving GitHub's API shape — /api/v3/, what gh targets 404
Forgejo serving its own shape — /api/v1/version 200

gh speaks GitHub's REST API. Forgejo implements Gitea's, at a different path, with a different schema. There is no GH_HOST setting that bridges them: GitHub Enterprise support in gh still expects /api/v3/.

So the reconciler is not missing a binary. It is pointed at a forge whose API it cannot speak, and every one of those 41 call sites would have to be re-expressed.

Why this is bigger than a red workflow

It is the same class as #122 / !128, one layer down. That round fixed panel= so the config names accounts that exist here. This says the machinery that would read that config cannot run here at all — so the labels reconciler, the work-queue sweep and the state:* machine are all inert on this forge regardless of configuration.

Concretely, it means every claim of the form "the sweep will catch it" is currently false on this instance. Several decisions this month leaned on that, including leaving stale state:* labels for the reconciler to correct, and #129's queue-state conflict sitting unresolved because "nothing recomputes it for us".

It also retires a premise from #116, which was closed on the strength of aligning actors: aligning them was necessary and is not sufficient.

This is a ruling, not a repair

The fix does not live in rig, and the options have very different costs:

  1. Make ceremony forge-agnostic — abstract the ~41 gh calls behind a shim with a Forgejo backend. The real fix, upstream, and by far the largest.
  2. Accept that label automation stays on GitHub. rig's board lives here, its automation lives there, and the state:* labels on this instance stay hand-managed. Cheapest, and honest, but it means the doctrine's machinery never runs on the forge the work is on.
  3. Stop running labels.yml here — disable the caller on this instance so 46 red runs per sweep stop being noise, whichever of 1 or 2 is chosen. This is worth doing immediately either way.

I recommend 3 now, then a decision between 1 and 2, because the current state is the worst of all: a workflow that fails on every trigger, teaching everyone to ignore red.

Tasks

  • @andres rules between 1 and 2
  • Regardless: stop labels.yml from running against this instance until that lands (option 3)
  • If 1: open the issue on heavy-duty/ceremony; rig's part is then only a pin bump
  • If 2: say so in .ceremony/ and in rig's own docs, so the next reader does not expect a sweep that will not come
  • Re-open or supersede the parts of #116 this contradicts

Acceptance criteria

  • labels.yml no longer produces a failing run on every trigger on this instance
  • The chosen answer is written down where someone reading rig's board will find it
  • Any doctrine text claiming the sweep runs here is corrected or scoped to GitHub

Test plan

Measurement rather than tests, since the deliverable is a decision:

  • Re-run the numbers above against whatever ref ceremony is pinned to at the time.
  • If 1 lands, labels.yml must reach a success on this instance, not merely stop failing.

Dependencies

Surfaced by #115. Contradicts a premise of the closed #116. Independent of #144 — that one is a missing tool and genuinely fixable in rig; this one is an API mismatch and is not.


@andres — filed, not fixed, per your instruction. Flagging needs-ruling because options 1 and 2 differ in cost by an order of magnitude and the choice reaches outside this repo.

## Context With a runner finally attached (#115), `labels.yml` runs — and fails every time. **46 of the first 87 tasks on this instance were this one workflow failing**, identically: ``` /workspace/heavy-duty/rig/.ceremony-src/actions/labels-reconcile/labels-reconcile.sh: line 692: gh: command not found /workspace/heavy-duty/rig/.ceremony-src/actions/issueflow-reconcile/issueflow-reconcile.sh: line 408: gh: command not found ❌ Failure - Main reconcile issue flow Job 'reconcile' failed ``` The obvious reading is "install `gh` in the runner image". That is wrong, and the difference matters. ## Installing `gh` would not fix it Measured against `ceremony@0.3.0` — the ref rig pins — and against this instance, 2026-08-01: | measurement | result | |---|---| | `gh` invocations in ceremony's actions | **21 × `gh api`, 13 × `gh issue`, 4 × `gh label`, 3 × `gh pr`** | | files mentioning "forgejo" anywhere in ceremony, at **any** ref incl. `main` (`0.4.1-dev`) | **none** | | Forgejo serving GitHub's API shape — `/api/v3/`, what `gh` targets | **404** | | Forgejo serving its own shape — `/api/v1/version` | **200** | `gh` speaks GitHub's REST API. Forgejo implements Gitea's, at a different path, with a different schema. There is no `GH_HOST` setting that bridges them: GitHub Enterprise support in `gh` still expects `/api/v3/`. So the reconciler is not missing a binary. **It is pointed at a forge whose API it cannot speak**, and every one of those 41 call sites would have to be re-expressed. ## Why this is bigger than a red workflow It is the same class as #122 / !128, one layer down. That round fixed `panel=` so the config names accounts that exist here. This says the machinery that would *read* that config cannot run here at all — so the labels reconciler, the work-queue sweep and the `state:*` machine are all inert on this forge regardless of configuration. Concretely, it means every claim of the form "the sweep will catch it" is currently false on this instance. Several decisions this month leaned on that, including leaving stale `state:*` labels for the reconciler to correct, and #129's queue-state conflict sitting unresolved because "nothing recomputes it for us". It also retires a premise from **#116**, which was closed on the strength of aligning actors: aligning them was necessary and is not sufficient. ## This is a ruling, not a repair The fix does not live in rig, and the options have very different costs: 1. **Make ceremony forge-agnostic** — abstract the ~41 `gh` calls behind a shim with a Forgejo backend. The real fix, upstream, and by far the largest. 2. **Accept that label automation stays on GitHub.** rig's board lives here, its automation lives there, and the `state:*` labels on this instance stay hand-managed. Cheapest, and honest, but it means the doctrine's machinery never runs on the forge the work is on. 3. **Stop running `labels.yml` here** — disable the caller on this instance so 46 red runs per sweep stop being noise, whichever of 1 or 2 is chosen. This is worth doing *immediately* either way. I recommend **3 now, then a decision between 1 and 2**, because the current state is the worst of all: a workflow that fails on every trigger, teaching everyone to ignore red. ## Tasks - [ ] @andres rules between 1 and 2 - [ ] Regardless: stop `labels.yml` from running against this instance until that lands (option 3) - [ ] If 1: open the issue on `heavy-duty/ceremony`; rig's part is then only a pin bump - [ ] If 2: say so in `.ceremony/` and in rig's own docs, so the next reader does not expect a sweep that will not come - [ ] Re-open or supersede the parts of #116 this contradicts ## Acceptance criteria - [ ] `labels.yml` no longer produces a failing run on every trigger on this instance - [ ] The chosen answer is written down where someone reading rig's board will find it - [ ] Any doctrine text claiming the sweep runs here is corrected or scoped to GitHub ## Test plan Measurement rather than tests, since the deliverable is a decision: - Re-run the numbers above against whatever ref ceremony is pinned to at the time. - If 1 lands, `labels.yml` must reach a **success** on this instance, not merely stop failing. ## Dependencies Surfaced by #115. Contradicts a premise of the closed #116. Independent of #144 — that one is a missing tool and genuinely fixable in rig; this one is an API mismatch and is not. --- @andres — filed, not fixed, per your instruction. Flagging `needs-ruling` because options 1 and 2 differ in cost by an order of magnitude and the choice reaches outside this repo.
claude-bot-andresmgsl added the
bug
scope:labels
needs-ruling
labels 2026-08-01 19:16:43 +00:00
Owner

Same way we did with the forgejo-runner, we are gonna use this forgejo instance (https://forgejo.heavyduty.builders) and the cli stoke (https://forgejo.heavyduty.builders/heavy-duty/stoke) to replace all the gh commands and github locked stuff to use forgejo...

Same way we did with the forgejo-runner, we are gonna use this forgejo instance (https://forgejo.heavyduty.builders) and the cli stoke (https://forgejo.heavyduty.builders/heavy-duty/stoke) to replace all the gh commands and github locked stuff to use forgejo...
Author
Member

The gh dependency, measured — and it is not in this repo

@andres asked on !146 why labels / labels is red. @grok answered correctly; this adds the measurement behind the answer, and one fact that changes this issue's shape.

Why it fails

GET /api/v3/repos/heavy-duty/rig  -> 404     # the shape gh speaks
GET /api/v1/repos/heavy-duty/rig  -> 200     # the shape this instance serves
GH_HOST=forgejo.heavyduty.builders gh api repos/heavy-duty/rig
  -> Not found.  gh: HTTP 404

Driven against this instance just now, not read from docs. Installing gh on the runner changes nothing — the client is pointed at an API surface that does not exist here. Confirmed as stated.

The fact that matters: the gh calls are upstream, not in rig

rig/.github/workflows/labels.yml is a three-line caller:

jobs:
  labels:
    uses: heavy-duty/ceremony/.github/workflows/labels.yml@0.3.0

Everything else lives in heavy-duty/ceremony. Counted at tag 0.3.0:

ceremony action gh invocations
labels-reconcile 20
issueflow-reconcile 18
labels-scope 3
total 41 — 21 gh api, 13 gh issue, 4 gh label, 3 gh pr

So no change inside rig can turn this job green. rig's caller is already correct; the port @andres described — "use the stoke CLI to replace all the gh commands" — is 41 call sites in ceremony's three composite actions, and rig only consumes the result. That is worth stating plainly on this issue, because its title reads as rig work and the fix is not rig's to make.

Two consequences for whoever takes the ruling:

  1. This issue is offsite in ceremony's sense — its deliverable is a PR in another repository. It cannot close on a rig commit.
  2. The 41 sites are not uniform: gh api (21) is a URL-shape port, while gh issue / gh pr / gh label (20) are subcommands stoke would need equivalents for. Worth checking stoke's surface covers all four verbs before committing to the port — stoke issue currently offers list and create only, with no comment or label subcommand, which is why every agent on this board writes issue comments through raw curl against /api/v1/ rather than through stoke.

That last point is the one I would want ruled on before anyone starts: port ceremony to stoke, or teach ceremony to speak /api/v1/ directly? The second needs no new CLI surface and no stoke release in the loop.

@andres — evidence only; the ruling on this issue stays yours, and #144 is unaffected either way.

## The `gh` dependency, measured — and it is not in this repo @andres asked on !146 why `labels / labels` is red. @grok answered correctly; this adds the measurement behind the answer, and one fact that changes this issue's shape. ### Why it fails ``` GET /api/v3/repos/heavy-duty/rig -> 404 # the shape gh speaks GET /api/v1/repos/heavy-duty/rig -> 200 # the shape this instance serves GH_HOST=forgejo.heavyduty.builders gh api repos/heavy-duty/rig -> Not found. gh: HTTP 404 ``` Driven against this instance just now, not read from docs. Installing `gh` on the runner changes nothing — the client is pointed at an API surface that does not exist here. Confirmed as stated. ### The fact that matters: the `gh` calls are upstream, not in rig `rig/.github/workflows/labels.yml` is a **three-line caller**: ```yaml jobs: labels: uses: heavy-duty/ceremony/.github/workflows/labels.yml@0.3.0 ``` Everything else lives in **heavy-duty/ceremony**. Counted at tag `0.3.0`: | ceremony action | `gh` invocations | |---|---| | `labels-reconcile` | 20 | | `issueflow-reconcile` | 18 | | `labels-scope` | 3 | | **total** | **41** — 21 `gh api`, 13 `gh issue`, 4 `gh label`, 3 `gh pr` | So **no change inside rig can turn this job green.** rig's caller is already correct; the port @andres described — *"use the stoke CLI to replace all the gh commands"* — is 41 call sites in ceremony's three composite actions, and rig only consumes the result. That is worth stating plainly on this issue, because its title reads as rig work and the fix is not rig's to make. Two consequences for whoever takes the ruling: 1. This issue is `offsite` in ceremony's sense — its deliverable is a PR in another repository. It cannot close on a rig commit. 2. The 41 sites are not uniform: `gh api` (21) is a URL-shape port, while `gh issue` / `gh pr` / `gh label` (20) are subcommands stoke would need equivalents for. Worth checking stoke's surface covers all four verbs before committing to the port — `stoke issue` currently offers `list` and `create` only, with no comment or label subcommand, which is why every agent on this board writes issue comments through raw `curl` against `/api/v1/` rather than through stoke. That last point is the one I would want ruled on before anyone starts: **port ceremony to stoke, or teach ceremony to speak `/api/v1/` directly?** The second needs no new CLI surface and no stoke release in the loop. @andres — evidence only; the ruling on this issue stays yours, and #144 is unaffected either way.
Author
Member

Upstream issue filed — heavy-duty/ceremony#219

@andres — created as you asked, for discussion before anyone builds.

It is on GitHub, not this instance, and that was a decision worth stating. ceremony develops on github.com/heavy-duty/ceremony: issues run to #218 with activity today and releases through 0.4.1. The heavy-duty/ceremony mirror here stops at #187 and is stale, so an issue filed on it would be invisible to the bench that owns the code.

No duplicate — zero ceremony issues mention Forgejo; the topic was unfiled there.

Re-measured against ceremony main (80da0a8), not the 0.3.0 tag I counted on this issue earlier. The number moved: 65 runtime gh invocations, not 41 — 26 gh api, 20 gh issue, 6 gh pr, 5 gh release, 4 gh label, 3 gh workflow, 1 gh run, across issueflow-reconcile (21), labels-reconcile (20), lib/ruling.sh (7), release.yml (7), labels-scope (3), lib/facts.sh (2). There is no forge abstraction to build on: no GH_HOST indirection, no shim in lib/.

Two findings that bear on your "use stoke" direction, both measured rather than assumed:

  1. gh api graphql is in there, and Forgejo has no GraphQL API at all — those sites need a different answer, not a translated one.
  2. stoke cannot do this job today. Its whole surface is auth · repo · issue (list, create) · pr (list, create, merge) · branch · collaborator · org · user — no label command, no issue-comment command, no review/checks/statuses/timeline reads. The reconcilers need all of those. So porting to stoke means building most of stoke first. That is why every agent on this board writes comments with raw curl against /api/v1/.

So the upstream issue frames three options — port to stoke, speak the REST API directly, or a two-backend shim in lib/ — and deliberately leaves the Spec undecided with needs-triage, because ceremony'''s own contract says an issue with an open question is not ready. That open question is the discussion you wanted.

It also carries the requested_reviewers trap I hit on !140: Forgejo does not clear that field when a verdict lands, so anything reading it as "who still owes a verdict" over-counts and would park PRs that are ready.

This issue stays as it is — needs-ruling, consumer-side. Nothing here can fix it; the code is upstream.

## Upstream issue filed — [heavy-duty/ceremony#219](https://github.com/heavy-duty/ceremony/issues/219) @andres — created as you asked, for discussion before anyone builds. **It is on GitHub, not this instance, and that was a decision worth stating.** ceremony develops on `github.com/heavy-duty/ceremony`: issues run to #218 with activity today and releases through **0.4.1**. The `heavy-duty/ceremony` mirror here stops at #187 and is stale, so an issue filed on it would be invisible to the bench that owns the code. **No duplicate** — zero ceremony issues mention Forgejo; the topic was unfiled there. **Re-measured against ceremony `main` (`80da0a8`), not the 0.3.0 tag I counted on this issue earlier.** The number moved: **65** runtime `gh` invocations, not 41 — 26 `gh api`, 20 `gh issue`, 6 `gh pr`, 5 `gh release`, 4 `gh label`, 3 `gh workflow`, 1 `gh run`, across `issueflow-reconcile` (21), `labels-reconcile` (20), `lib/ruling.sh` (7), `release.yml` (7), `labels-scope` (3), `lib/facts.sh` (2). There is no forge abstraction to build on: no `GH_HOST` indirection, no shim in `lib/`. Two findings that bear on your "use stoke" direction, both measured rather than assumed: 1. **`gh api graphql` is in there, and Forgejo has no GraphQL API at all** — those sites need a different answer, not a translated one. 2. **stoke cannot do this job today.** Its whole surface is `auth · repo · issue (list, create) · pr (list, create, merge) · branch · collaborator · org · user` — no label command, no issue-comment command, no review/checks/statuses/timeline reads. The reconcilers need all of those. So porting to stoke means building most of stoke first. That is why every agent on this board writes comments with raw `curl` against `/api/v1/`. So the upstream issue frames three options — port to stoke, speak the REST API directly, or a two-backend shim in `lib/` — and deliberately leaves the Spec **undecided** with `needs-triage`, because ceremony'''s own contract says an issue with an open question is not `ready`. That open question is the discussion you wanted. It also carries the `requested_reviewers` trap I hit on !140: Forgejo does not clear that field when a verdict lands, so anything reading it as "who still owes a verdict" over-counts and would park PRs that are ready. This issue stays as it is — `needs-ruling`, consumer-side. Nothing here can fix it; the code is upstream.
Author
Member

Correction — the upstream issue is heavy-duty/ceremony#188, on this instance

@andres — I filed it on GitHub first; that was wrong, and it is corrected. ceremony#188 is the live one, created with stoke issue create.

The GitHub copy is closed as not-planned with a pointer here. I could not delete it: DeleteIssue needs repo-admin, which this identity does not have — anyone with admin should delete it rather than leave the duplicate.

Two things changed in the move, both because the issue should be measured against the tree it lives beside:

  • The Forgejo heavy-duty/ceremony is not a mirror — it is a real repo at 84bb1a4 (0.4.1-dev), one cycle behind the GitHub tree I had measured (80da0a8, 0.4.2-dev).
  • So the count is 61 runtime gh invocations on this tree, not the 65 I quoted from the other one: 26 gh api, 20 gh issue, 6 gh pr, 5 gh release, 4 gh label, across issueflow-reconcile (21), labels-reconcile (20), lib/ruling.sh (7), release.yml (7), labels-scope (3), lib/facts.sh (2), self-release.yml (1). Two of them are gh api graphql, which Forgejo has no equivalent for at all.

Everything else stands: no forge abstraction exists, stoke today has no label, comment, review, checks or timeline surface, and the Spec is deliberately left undecided under needs-triage so the bench settles A/B/C first.

This issue stays needs-ruling and consumer-side.

## Correction — the upstream issue is [heavy-duty/ceremony#188](https://forgejo.heavyduty.builders/heavy-duty/ceremony/issues/188), on this instance @andres — I filed it on GitHub first; that was wrong, and it is corrected. **ceremony#188** is the live one, created with `stoke issue create`. The GitHub copy is **closed as not-planned** with a pointer here. I could not delete it: `DeleteIssue` needs repo-admin, which this identity does not have — anyone with admin should delete it rather than leave the duplicate. Two things changed in the move, both because the issue should be measured against the tree it lives beside: - The Forgejo `heavy-duty/ceremony` is **not a mirror** — it is a real repo at `84bb1a4` (`0.4.1-dev`), one cycle behind the GitHub tree I had measured (`80da0a8`, `0.4.2-dev`). - So the count is **61** runtime `gh` invocations on *this* tree, not the 65 I quoted from the other one: 26 `gh api`, 20 `gh issue`, 6 `gh pr`, 5 `gh release`, 4 `gh label`, across `issueflow-reconcile` (21), `labels-reconcile` (20), `lib/ruling.sh` (7), `release.yml` (7), `labels-scope` (3), `lib/facts.sh` (2), `self-release.yml` (1). Two of them are `gh api graphql`, which Forgejo has no equivalent for at all. Everything else stands: no forge abstraction exists, `stoke` today has no label, comment, review, checks or timeline surface, and the Spec is deliberately left undecided under `needs-triage` so the bench settles A/B/C first. This issue stays `needs-ruling` and consumer-side.
Author
Member

Ruling resolved — decision recorded; remaining rig work is #149

Label events re-read before this write: needs-ruling set 2026-08-01T19:16:43Z, never lifted since — this comment closes that episode.

The decision, recorded: @andres ruled option 1 the same evening the flag went up (comment above): port ceremony's machinery to this Forgejo rather than keeping automation on GitHub or hand-managing labels forever. That ruling was executed upstream — ceremony#188 (filed from this issue, now closed) delivered the forge abstraction, and ceremony 0.6.1 on this instance ships lib/forge.sh + lib/forge-forgejo.sh, citing rig's run 979 as the motivating failure. Agreement was reached and then built; nothing here waits on a human any longer, so the label comes off in this same move.

Where this issue's remaining substance went: rig's part was always "then only a pin bump" (this issue's own task list), and that is now minted to contract as #149 — bump 0.3.0 → 0.6.1 across labels.yml, release.yml, the six ci.yml guard refs, the new labels-sweep.yml/refs-guard.yml callers, and the .ceremony/ re-vendor. Its post-merge criterion — a green labels.yml run on this instance — carries the acceptance criterion this issue could not check.

Point-by-point against this issue's own tasks:

task outcome
@andres rules between 1 and 2 option 1, 2026-08-01
stop labels.yml failing here (option 3) not done separately; #149 is now the shortest path — the noise is still live (runs 2461–2468 today, all red)
if 1: open the ceremony issue ceremony#188, closed delivered
if 2: document n/a
re-open/supersede parts of #116 #116 stays closed; its premise gap is answered by the port itself

Closing as superseded by #149: the ruling this issue existed to obtain is recorded above, and every remaining checkbox lives in #149's contract. The board keeps exactly one open item for this failure, and it is buildable today.

## Ruling resolved — decision recorded; remaining rig work is #149 Label events re-read before this write: `needs-ruling` set 2026-08-01T19:16:43Z, never lifted since — this comment closes that episode. **The decision, recorded:** @andres ruled **option 1** the same evening the flag went up ([comment above](https://forgejo.heavyduty.builders/heavy-duty/rig/issues/145#issuecomment-4350)): port ceremony's machinery to this Forgejo rather than keeping automation on GitHub or hand-managing labels forever. That ruling was executed upstream — [ceremony#188](https://forgejo.heavyduty.builders/heavy-duty/ceremony/issues/188) (filed from this issue, now closed) delivered the forge abstraction, and ceremony `0.6.1` on this instance ships `lib/forge.sh` + `lib/forge-forgejo.sh`, citing rig's run 979 as the motivating failure. Agreement was reached and then built; nothing here waits on a human any longer, so the label comes off in this same move. **Where this issue's remaining substance went:** rig's part was always "then only a pin bump" (this issue's own task list), and that is now minted to contract as **#149** — bump `0.3.0 → 0.6.1` across `labels.yml`, `release.yml`, the six `ci.yml` guard refs, the new `labels-sweep.yml`/`refs-guard.yml` callers, and the `.ceremony/` re-vendor. Its post-merge criterion — a **green** `labels.yml` run on this instance — carries the acceptance criterion this issue could not check. Point-by-point against this issue's own tasks: | task | outcome | |---|---| | @andres rules between 1 and 2 | **option 1**, 2026-08-01 | | stop `labels.yml` failing here (option 3) | not done separately; #149 is now the shortest path — the noise is still live (runs 2461–2468 today, all red) | | if 1: open the ceremony issue | ceremony#188, closed delivered | | if 2: document | n/a | | re-open/supersede parts of #116 | #116 stays closed; its premise gap is answered by the port itself | Closing as **superseded by #149**: the ruling this issue existed to obtain is recorded above, and every remaining checkbox lives in #149's contract. The board keeps exactly one open item for this failure, and it is buildable today.
claude-bot-andresmgsl removed the
needs-ruling
label 2026-08-17 23:20:49 +00:00
Sign in to join this conversation.
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/rig#145
No description provided.