codex, grok and kimi all landed on the same hole at e42ba31, and they are
right. The record row keyed only on fj_took, so install/status/remove failures
called `no` without gating it. A dispatched job only needs SOMETHING answering
runs-on: drill — and this leg removes locally, telling the operator to delete
the stale runner by hand, so a leftover drill-labeled runner from the previous
drill is the designed-for aftermath, not a contrived case. It answers the
fixture exactly, and the row then read "PASS — registered, took a job,
removed" for a drill whose install never happened.
The exit code was never wrong: every one of those failures also called `no`.
What was wrong is the row, and the row is what outlives the run in
drills/<v>.md. A row claiming a lifecycle that did not happen is exactly what
the gate exists to refuse — kimi's framing, and it is the right one.
forgejo_leg_row folds all five assertions, and the leg no longer dispatches
after install or status has failed: a foreign runner's success could only ever
be evidence about someone else's runner. The regression composes the real
functions over a payload carrying a newer successful run, the way the leg
composes them.
The GitHub leg has the identical shape and is byte-frozen by #129's contract;
left alone deliberately, and worth a sibling issue for triage.
Refs #129
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 2 of !130. Both blockers were mine, and one was the same mistake I had
just fixed one layer up.
1. fj_pre still took `head -n1`. The verdict folds max over every id, but the
PRE-DISPATCH snapshot did not, so an oldest-first payload named an old run
as the baseline — and a later poll finding the same body then read the
PREVIOUS drill's run as this dispatch's result. That is a false PASS on the
take-a-job assertion, strictly worse than the false FAIL entry[0] caused
inside the verdict. Both sides now share forgejo_max_task_id, and a test
composes them the way the leg does so the pair cannot drift apart again.
2. test/drill.sh copied its pretty-printed fixture from /tmp/fjfix — a scratch
path that existed only on the box the fix was built on. Everywhere else the
cp failed, the guard returned pending, and the suite was 65/66. The claimed
66/66 was true on one machine. The fixture is written inline like every
other one; verified by deleting the scratch dir and running the suite from
a clean tree under env -i.
Refs #129
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Addresses grok's and kimi's REQUEST_CHANGES on !130.
1. forgejo_run_verdict read only the FIRST entry of actions/tasks. That
payload accumulates, so the moment a repo is drilled twice our run shares
it with older ones — and nothing documents the sort order. A green job
then reports as a timeout: a false FAILURE on the gate this leg exists to
provide. It now inspects every entry and lets the newest id above pre_id
decide. Newlines are stripped first, so a pretty-printed payload parses
like a compact one.
2. A mint that yielded nothing degraded to SKIPPED "no registration token
source" — violating #129's own acceptance ("token source present but the
instance is unreachable -> the leg FAILS; it must not skip and must not
pass") and sending the operator to check an env var they had already set.
forgejo_token_verdict separates absent inputs from a configured leg that
could not mint; only the former skips.
3. The pre---yes confirm block still announced a GitHub runner alone.
Refs #129
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
#109 shipped `rig forgejo-runner` and the drill never touched it, so the
release gate could pass with the whole family broken. Its only coverage was
15 argument-handling checks in test/cli.sh.
The Forgejo half cannot be a copy of the GitHub one. Measured against
forgejo.heavyduty.builders (8.0.3+gitea-1.22.0), not read from the docs:
GET actions/tasks carries NO `conclusion` field — `status` holds the terminal
outcome — `id` is a global task id while the URL ends in run_number, and the
payload lists ASSIGNED tasks only, so a queued run is invisible (measured:
200s of total_count 0 while the web UI showed the run). That makes "no new
id" the only signal the runner never took the job, which is the verdict this
leg exists to produce.
forgejo_run_verdict() is the testable core of that, grep-and-sed rather than
jq for the same reason json_field() is. Tests were written first and cover
the cases that must NOT pass: a stale run with the pre-dispatch id, an
assigned-but-still-running task, and a cancelled run.
Refs #129
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
claude REQUEST_CHANGES on 1c9a245: the all-candidates-fail path still
interpolated multi-line BOX_MANUAL into one warn, orphaning the or: line
and leaving try: non-pasteable on the default host.
- box_manual_emit log|warn prints one recovery line at a time
- single-candidate hosts (GitHub) get a bare pasteable command (no try:)
- multi-candidate keeps try:/or: prefixes
- never ${BOX_MANUAL} inside a log/warn string
- cli.sh asserts the emission invariant
codex REQUEST_CHANGES on !114 / #125: multi-candidate recovery was
rendered as `curl A | bash; if that 404s: curl B | bash`, which
`bash -n` rejects (exit 2). Operators on RIG_SKIP_BOX_INSTALL / no-curl
/ all-candidates-failed paths were handed non-executable recovery.
- box_manual_cmd / box_manual_text: one pasteable command per candidate
- display as separate `try:` / `or:` lines (newlines, no prose join)
- post-probe rewrite still collapses to the URL that worked
- execution tests: both Forgejo URLs visible; each command passes bash -n
Claude re-review on c7d56d3:
- Group the installer-fn extracts in test/release.sh (SC2129).
- BOX_MANUAL on multi-candidate hosts lists every URL so BOX_REF=main
never hands the operator only a 404ing /raw/tag/ form.
- Execute box_install_urls under the harness (tag-before-branch order).
- Cover Forgejo's no-release 404 path in resolve_latest_tag.
Claude REQUEST_CHANGES on !114: the latest-release path must never fall
through to refs/heads/<tag> (would install a branch while INSTALLED_FROM
still names the release). release_tag_url is the single refs/tags URL on
every forge; regression test drives the heads-only stub.
Also restore coolify Documentation= to github.com — forge flip needs an
@andres ruling, not a shipped default in this PR.
Address codex REQUEST_CHANGES + claude's live Forgejo measurements:
- ref_candidate_urls is host-only: same refs/tags then refs/heads on every
forge (Forgejo serves both paths; bare /archive/<ref> no longer special).
- bootstrap box raw fetch tries /raw/tag/ then /raw/branch/ and never
guesses kind from spelling; download is separate from execute.
- BOX_HOST defaults through RIG_HOST; comments narrow the zero-GitHub claim
(box@0.9.0 installer still hardcodes GitHub for its archive).
- SC2016 suppressions on the new grep assertions; tests updated.
install.sh resolves releases and archive URLs through RIG_HOST (default
GitHub), using Forgejo's single /archive/<ref>.tar.gz form and the same
/releases/latest redirect grammar. Bootstrap's box fetch gets BOX_HOST
with forge-aware raw-file URLs. Refusal hints and coolify Documentation=
stop pointing only at GitHub.
Closes#111
CONTRIBUTING's changelog rule is explicit — the fragment answers one question,
what is different in this version; why it was wrong and what it implies belong
in the PR body and the commit message, which is where anyone chasing the
reasoning already goes. The entry carried both.
Review: codex-reviewer-andresmgsl on !113.
Refs #112
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The six ceremony actions in ci.yml are step `uses:` references, and a bare
step reference resolves through FORGEJO__actions__DEFAULT_ACTIONS_URL — which
this forge points at code.forgejo.org, where heavy-duty/ceremony does not
exist. Measured with a registered runner on the instance: `git clone
https://code.forgejo.org/heavy-duty/ceremony` -> 404, so every guard fails the
moment rig's own CI runs on the forge. Those six become absolute.
The two reusable-workflow callers do NOT move, and this is the correction to
the original "make all eight absolute" ruling. A reusable `uses:` never
consults DEFAULT_ACTIONS_URL; it resolves against the runner's own instance,
so both callers already fetch ceremony from this forge and expand. Absolutising
release.yml's would actively break CI: ceremony's docs-sync reads rig's pin out
of that one line with a grep anchored to the bare spelling, so the prefix makes
the pin invisible and docs-sync exits 1 with "no pin line ... found none".
actions/checkout@v4 stays bare in all three places — the mirror serves it, and
that is exactly what DEFAULT_ACTIONS_URL is for. The pin stays @0.3.0
throughout; no .ceremony/ edit, no pin bump.
test/cli.sh pins all three shapes, including the docs-sync grep verbatim, so
ceremony's parser and rig's caller cannot drift apart silently.
Refs #112
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
codex's non-blocking nit, folded in by the maintainer's call now that the round
is open anyway.
The resolve-latest path validated what rig worked out for itself
(""|*[!0-9.]*), and an explicit --version went straight into the download URL
unchecked — so the value rig is HANDED, which is the one that reaches a URL
path, was the only one nobody looked at. `--version ../../etc/passwd` was
interpolated rather than refused.
Not a trust boundary: this command is already root and the operator typed the
flag. It is a diagnosis defect. A pin that cannot name a release should fail by
name, next to the flag, rather than as an opaque 404 forty lines later.
install.sh's valid_version is the same instinct.
Four rows: two refusals, and two that reach "must run as root" — which is how a
non-root suite proves a good pin got THROUGH validation rather than merely
failing later. Mutation-checked: the refusals exit 1 instead of 2 without the
guard.
756 passed (was 752), release 31, drill 43, shellcheck clean.
Round finding from @codex-reviewer-andresmgsl, elevated to blocking by
@grok-reviewer-andresmgsl and @kimi-reviewer-andresmgsl. Unanimous, and right.
creds.md called the registration token "short-lived" and said it was "consumed
at registration". Both are GitHub's facts, copied across the forge boundary
with the rest of the sibling's shape. Forgejo's primary source, read rather
than inferred:
models/actions/runner_token.go — ActionRunnerToken has NO expiry field. Only
IsActive, Created, Updated. NewRunnerToken flips IsActive false on prior
tokens at the same scope and only there, so a token dies when somebody mints
its replacement, never on a clock.
routers/api/actions/runner/runner.go — Register reads the token, refuses it
when !IsActive ("please use the latest one"), and returns WITHOUT setting
IsActive = false. Registration does not spend it. One token registers as many
runners as it is shown to.
So it is long-lived and reusable — the precise opposite of the adjective, and
GitHub's really does expire in about an hour, which is why runner-install.sh is
correct to use it.
This is not a wording nit because of where the wording lives. creds.md is
spliced into the ci-box's own CONTEXT.md: it is the paragraph an agent INSIDE
the box reads about its own credentials. Telling that reader the token
self-expires is telling it a leaked one stops mattering on its own, while it is
still registering runners.
Pinned, not merely fixed, per codex's ask — the phrase arrived by copying from
the GitHub sibling, so the same copy can bring it back. Four rows: absence from
both files, and presence of the true claim, so the pin cannot be satisfied by
deleting the sentence instead of correcting it. The first draft of the CIBOX
pin was a phrase match and passed against the exact text it was written to
catch — the old wording wrapped across two comment lines. It is a plain absence
check now, and the file explains the ban without spelling the word.
Mutation-checked: all four go red against the old wording, green after.
Maintainer ruling on the issue-number collision raised in the PR body.
CONTRIBUTING's rule is "cite the issue or PR — (#96)", and fragments are
assembled into CHANGELOG.md verbatim, so a fragment is the one place the
convention is actually load-bearing. `forgejo#109` links on neither forge and
would seed a second citation spelling in every future fragment.
The prose distinction stays where it earns its keep: PR bodies and review
comments still write forgejo#N, because that is where a reader has both
numbering schemes in front of them at once.
Self-review of the whole head after the panel converged. One real defect,
reproduced verbatim before it was touched.
runner_version_of is a pipeline, and every forgejo-runner-* command runs under
set -euo pipefail. When the binary exits non-zero the pipeline fails, and
`PRESENT_VER="$(runner_version_of "$BIN")"` is an ASSIGNMENT — so install died
at that line, exit 1, printing nothing at all. The lib documented the opposite
("empty when it cannot answer"), and the refusal written for exactly this case
twenty lines later — "the download landed but cannot run" — was unreachable
code that could never have fired.
The shape is not hypothetical. `[ -x ]` yes, runnable no is what a truncated or
wrong-arch download leaves behind, and a ci-box's template preinstalls the
binary at mint — so the silent death sat on the one path this command family
exists to serve.
json_field, two files away, already carries `|| true` with this reasoning
written above it: "callers run under set -e with pipefail, where a grep that
matches nothing would otherwise kill the script with no message". The same
guard, for the same reason, at both sites: the lib reader and install's own
read of --version.
The test could not have caught it. vparse drove the lib through `bash -c` with
NO shell options, and its "bad" stub exits 0 — it proved garbage-in-empty-out
in a permissive shell, which is not where the function is used. vparse now runs
under set -euo pipefail, and a stub that exits non-zero is a third row.
Mutation-checked: that row exits 1 against the unpatched lib and 0 against the
fixed one. Install's site is grep-pinned, since reaching it for real needs root
and a downloaded binary.
748 passed (was 746), release 31, drill 43, shellcheck clean.
The operator turned off REQUIRE_SIGNIN_VIEW on forgejo.heavyduty.builders, so
the blocker this PR documented no longer applies to that instance. Re-measured
anonymously: heavy-duty/rig, ceremony, box, cast and stoke all answer 200 on
the API, the web page and git ls-remote, where rig 404'd on every one of those
this morning.
More to the point, the mechanism is now proven rather than argued: a
credential-less templates_resolve with
RIG_TEMPLATES_HOST=https://forgejo.heavyduty.builders fetches and extracts a
real archive end to end. That was the one part of piece 1 that could not be
demonstrated while the gate was up.
The requirement itself is unchanged and the refusal text still names it — it
follows from the mint's creds-free contract, not from one server's config, and
any other instance hosting a registry needs the same setting. What changed is
that this instance now satisfies it, so the docs say "verified satisfied"
instead of "blocked".
The plan doc keeps the original analysis under a heading that marks it
historical: it is why the knob exists, and deleting it would lose the
reasoning.
forgejo#109
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The maintainer ruled on the conflict kimi found reviewing !110: keep
DEFAULT_ACTIONS_URL=https://code.forgejo.org — actions/* are exactly what that
setting is designed to resolve, and the mirror serves them — and make the eight
first-party ceremony references absolute instead.
The README stated the conflict but deliberately left the direction open. It now
names the ruling and shows the absolute form, and points at forgejo#112 where
the workflow edits are tracked. Those edits are not made here: they touch
release ceremony, and folding them into a PR three reviewers have already read
would invalidate those reviews for no benefit.
forgejo#109
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Net-new review findings from grok and kimi on !110. Their items 1-3 were
codex's, already fixed in 1933b07; these are the ones only they raised.
grok #4 — the two downloaders would drift. docs/templates/ci-box/install.sh
and the download block in forgejo-runner-install.sh were near-copies, and grok
named the exact consequence with the exact evidence: fail-open survived in BOTH
while a grep for "checksum mismatch" passed against both, because the string it
looked for sat right beside the branch it could not see. The whole policy —
fetch, unreadable, mismatch — is now fetch_and_verify_sha256, byte-identical in
both files and diffed by test/cli.sh. They cannot share a lib: the command
sources commands/lib/, and the template is a registry definition that runs
standalone inside a mint with rig's tree nowhere in reach, which is the same
situation valid_version faces between bin/rig and install.sh. Mutation-checked
by drifting one copy's message and confirming the diff goes red.
kimi #2 — the labeler could not see this family. scope:runner matched
commands/runner-*.sh only, so forgejo-runner-*.sh and the staged ci-box
definition scored no scope at all. Globs extended and the label's description
now says either forge rather than GitHub.
kimi #4 — remove stranded a unit whose user was gone. The missing-user check
exited 0 before the unit was ever looked at, so a deleted account with a
leftover forgejo-runner.service reported "nothing to remove" while the
absence-assert that never ran implied the opposite. The unit is now checked
independently. Auditing that fix surfaced a hazard kimi did not mention: with
the user gone RUNNER_DIR is "", and the later unguarded "$RUNNER_DIR/.rig-labels"
would have expanded to "/.rig-labels" — an rm at the filesystem root, as root.
Every RUNNER_DIR path is now gated, and a test pins that none is unguarded.
kimi #1 — the README handed out a config that breaks rig's own gates.
DEFAULT_ACTIONS_URL is a single fallback and rig's workflows need two origins;
measured: code.forgejo.org serves actions/checkout (200) but not
heavy-duty/ceremony (404), which lives on the Forgejo instance. With the value
the README recommended, all eight ceremony references fail to resolve. The
section now states the conflict with the counts, says which references would
break, and explicitly does NOT pick a side — that is an infra decision, and
rig's CI running on Forgejo is not something rig forgejo-runner depends on.
Asked the maintainer for direction.
746/31/43 pass, shellcheck clean, labeler.yml parses.
forgejo#109
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three defects from review !110, all the same family — a stated contract the
code did not keep.
--version was swallowed on the path this command exists for. The download
block skipped on mere presence, copying runner-install.sh's shape without its
justification: actions/runner SELF-UPDATES, so freezing it would only make
GitHub refuse its jobs. forgejo-runner does not self-update, so nothing else
ever moves the version — and a ci-box's template preinstalls the binary at
mint, which meant the documented deterministic-pin lever could never fire on a
ci-box. It now converges toward the pin, downward included, because a pin is
an instruction and not a floor; absent a pin an existing binary is left alone,
since chasing latest would make a re-run an unrequested upgrade. The decision
moved to runner_download_decision in the lib as a pure function: the first
attempt at a test here grepped for a log string and survived the logic being
disabled, which is exactly the weak test the review warned about. The binary
is now renamed into place rather than written over — the converge path runs
while the daemon is live, and in-place is ETXTBSY.
.rig-labels outlived the registration it described. The write had escaped the
registration branch, where runner-install.sh correctly keeps its copy, so a
plain re-run stamped this invocation's labels over a registration made with
different ones and status then reported confidently wrong labels while Forgejo
still held the originals. Scoped again, and an EXPLICIT --labels on a re-run
now warns that Forgejo owns labels from registration time rather than letting
the request evaporate silently.
The checksum gate failed open. A missing .sha256 warned and installed anyway,
contradicting both the README and the template's own comment about unverified
root downloads. The original reasoning — do not let an upstream layout change
break installs — reasons about the wrong failure: a layout change breaks the
BINARY url too, so "binary yes, checksum no" is the shape of an interfered
fetch, which is precisely what the checksum exists to catch. Both paths refuse
now, with no bypass flag; if upstream really moves its assets that is a rig PR
editing the URL, not an operator improvising past a security gate.
Tests: the checksum paths are now DRIVEN against a stub curl through the real
template install.sh — matching, missing, mismatched and empty — instead of
grepped, and all three fixes were mutation-checked by reverting each and
confirming the suite goes red.
739/31/43 pass, shellcheck clean.
forgejo#109
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rig's CI story was GitHub-shaped end to end. This makes it work against a
self-hosted Forgejo, in three pieces.
The registry fetch becomes forge-aware. templates_resolve hardcoded three
github.com archive URLs; RIG_TEMPLATES_HOST now selects the grammar, because
the forges genuinely differ — GitHub serves refs/tags, refs/heads and bare
paths, Forgejo serves exactly one, and emitting the other two there would mean
two guaranteed 404s per fetch and a failure message listing URLs that never
could have worked. Measured against forgejo.heavyduty.builders, not inferred.
The default stays GitHub, so every existing caller is unchanged. install.sh's
snapshot reads the same variable through a byte-identical copy of the builder,
diffed by the tests: a snapshot cached from a forge converge would never fetch
from is worse than no snapshot, and the pin-in-the-name staleness guard cannot
catch a wrong-ORIGIN snapshot, only an old one.
ci-box is a tenant, not a machine role. The topology is a fleet machine
hosting boxes, one of which runs CI — a '-box' guest by rig's own family rule.
That also deletes the docker-in-docker layer the usual setup needs:
bootstrap-tenant.sh already installs Docker and adds the tenant user to the
group, and the isolation a privileged dind sidecar buys is already paid for by
a box that is network-isolated, inbound-less and disposable. rig runner
install refuses Docker for good reason — it converges a MACHINE, where the
blast radius is the machine. Here it is a guest that gets thrown away.
rig forgejo-runner is a new family beside rig runner, which is untouched.
Forgejo registers against an INSTANCE and the token carries the scope, so
there is no --repo to converge toward and nothing to compare; folding that
into one command would make every guard bimodal to share a flag name while the
contract underneath differs. assert_runner_instance asks the same
trust-boundary question about the axis Forgejo actually has. There is no
repoint and no --local, and both absences are explained where an operator
arriving from the GitHub sibling will hit them.
Forgejo's .runner holds the runner's own long-lived token, unlike GitHub's, so
it is installed 0600 and the mode is re-asserted on every converge — a mode
that drifted leaks the secret silently, since nothing fails and the runner
keeps working. status reports it and never prints the token.
Both downloads verify the published .sha256 before installing: this binary
lands as root and is executed by a systemd unit.
bootstrap --undo learns the guard for the same hazard on the other forge, and
it matters more here — Forgejo has no deregistration endpoint, so the ghost it
would strand has to be deleted by hand.
Known prerequisite, documented rather than assumed: the fetch is
unauthenticated by contract, and a Forgejo with REQUIRE_SIGNIN_VIEW=true
answers 404 for repos it reports as public. Hosting a registry there needs
FORGEJO__service__REQUIRE_SIGNIN_VIEW=false. The refusal names that case,
because it is indistinguishable from a wrong ref.
forgejo#109
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The service converge+assert block becomes converge_cron(), behavior-
identical, so test/cli.sh can lift it verbatim (the drop_incus precedent)
and drive it: the already-enabled/active re-run is a no-op, a disabled
stopped unit converges, and an unrecoverably masked or start-dead unit
dies nonzero naming cron and #162 — proven by execution, not only by
structural pins. The converge logs now state the probe fact instead of
claiming an unverified success (panel nit).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agent tenants exist to run the cron-driven duty engine, and the engine's
installer runs unprivileged — it can detect a missing cron but never
apt-get it. Install cron with the shared toolbelt, then assert the
effective state, not the package: crontab on PATH AND cron.service
enabled and active, converging (unmask/enable/start) best-effort first.
A masked daemon with the binary present is exactly the silent-inert box
the issue reports, so the service asserts are the authority and either
failing dies naming cron.
staging-box stays exempt with the rest of the agent-assert block: no
agent, no duty engine. Machine roles (bootstrap.sh) are deliberately
not widened — the issue names tenants as the demonstrated gap.
Closes#162
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
json_string_array took the first "Tags" array anywhere in the document,
justified by Self-before-Peer field order. An untagged Self omits the key
entirely (Go omitempty), so the match fell through into Peer and returned
a peer's tag — false-refusing every login join and false-verifying
untagged authkey joins on any tailnet with a tagged node. Extract Self by
brace counting (Location nests an object, so slicing to the next key
would truncate) and read the array inside it.
Refs #160
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
