forgejo-runner install reports 'installed and running' for a crash-looping unit — Type=simple is active the instant it forks #141

New issue

Open

opened 2026-08-01 19:04:39 +00:00 by claude-bot-andresmgsl · 0 comments

claude-bot-andresmgsl commented

2026-08-01 19:04:39 +00:00

Member

Context

rig forgejo-runner install reported success on a box where the runner could never start. From @andres's ci-box, 2026-08-01:

rig-forgejo-runner: runner ci-forgejo-box (v12.13.2) installed and running

Meanwhile:

● forgejo-runner.service - Forgejo Actions runner
     Active: activating (auto-restart) (Result: exit-code)
    Process: 1925 ExecStart=/usr/local/bin/forgejo-runner daemon (code=exited, status=1/FAILURE)
   forgejo-runner.service: Scheduled restart job, restart counter is at 76.

The runner showed Offline / Never in Forgejo for ~13 minutes while rig had already said "installed and running".

The defect

commands/forgejo-runner-install.sh:498-503:

active=""
for _ in 1 2 3 4 5 6; do
  if systemctl is-active forgejo-runner >/dev/null 2>&1; then active=1; break; fi
  sleep 2
done
[ -n "$active" ] || die "forgejo-runner.service is not active after 12s …"

The unit is Type=simple. systemd marks such a unit active the instant it forks, before the process has done anything at all — and systemctl enable --now does not return until that has happened. So rig's first poll always sees active, breaks immediately, and declares success. The daemon then exited 1 after ~25 ms and entered a 10-second restart loop.

This is not a flaky race that occasionally misfires. It is near-deterministic: the check cannot fail for any crash-looping Type=simple unit, because there is always an active moment between fork and death.

Why it is rig's own named bug class

commands/bootstrap.sh, on the box install, puts it exactly:

Don't trust the exit code — prove the effective state (issue #12). An installer can exit 0 having done less than it claims […] asserting what was REQUESTED […] instead of what actually TOOK.

is-active immediately after enable --now asserts what was requested. The claim "installed and running" is the strongest claim this command makes, and it is the one least supported.

Spec

Assert the service is still up after it has had a chance to fail, and distinguish activating from active.

Poll for a settled state: the unit must be active and remain so across the window, rather than breaking on the first sighting.
Treat activating as not-yet-healthy. systemctl is-active exits non-zero for it, but the loop's early break means the distinction is never reached.
On failure, name the restart counter and point at the journal — NRestarts climbing is the single clearest signal of this shape.
Do not lengthen the wait as the fix. A longer sleep with the same early break has the same hole.

Tasks

Replace the break-on-first-sighting loop with a settled-state assert
Refuse when the unit is activating at the end of the window, naming the restart count
test/cli.sh: stub systemctl so a unit that reports active once then activating fails the assert
changelog.d/<n>.md

Acceptance criteria

A crash-looping unit makes install exit non-zero — driven with a stubbed systemctl, not reasoned
A genuinely healthy unit still passes, unchanged
The refusal names the journal and the restart counter
bash test/cli.sh passes; CI's shellcheck -x step is clean

Test plan

Stub systemctl to report active on the first call and activating thereafter — today's code passes, the fix must fail.
Must fail: revert to the early break and the new test goes red.
A stub reporting active throughout must still succeed.

Dependencies

Surfaced by #141 (no Docker preflight), which is what made the daemon exit — but independent of it: this assert would have lied for any startup failure, not only a missing Docker.

@andres — filed from your ci-box report, not to be fixed yet, per your instruction.

## Context `rig forgejo-runner install` reported success on a box where the runner could never start. From @andres's ci-box, 2026-08-01: ``` rig-forgejo-runner: runner ci-forgejo-box (v12.13.2) installed and running ``` Meanwhile: ``` ● forgejo-runner.service - Forgejo Actions runner Active: activating (auto-restart) (Result: exit-code) Process: 1925 ExecStart=/usr/local/bin/forgejo-runner daemon (code=exited, status=1/FAILURE) forgejo-runner.service: Scheduled restart job, restart counter is at 76. ``` The runner showed **Offline / Never** in Forgejo for ~13 minutes while rig had already said "installed and running". ## The defect `commands/forgejo-runner-install.sh:498-503`: ```bash active="" for _ in 1 2 3 4 5 6; do if systemctl is-active forgejo-runner >/dev/null 2>&1; then active=1; break; fi sleep 2 done [ -n "$active" ] || die "forgejo-runner.service is not active after 12s …" ``` The unit is `Type=simple`. systemd marks such a unit **active the instant it forks**, before the process has done anything at all — and `systemctl enable --now` does not return until that has happened. So rig's *first* poll always sees `active`, breaks immediately, and declares success. The daemon then exited 1 after ~25 ms and entered a 10-second restart loop. This is not a flaky race that occasionally misfires. It is near-deterministic: the check cannot fail for any crash-looping `Type=simple` unit, because there is always an `active` moment between fork and death. ## Why it is rig's own named bug class `commands/bootstrap.sh`, on the box install, puts it exactly: > Don't trust the exit code — prove the effective state (issue #12). An installer can exit 0 having done less than it claims […] asserting what was REQUESTED […] instead of what actually TOOK. `is-active` immediately after `enable --now` asserts what was requested. The claim "installed and running" is the strongest claim this command makes, and it is the one least supported. ## Spec Assert the service is *still* up after it has had a chance to fail, and distinguish `activating` from `active`. - Poll for a **settled** state: the unit must be `active` **and** remain so across the window, rather than breaking on the first sighting. - Treat `activating` as not-yet-healthy. `systemctl is-active` exits non-zero for it, but the loop's early `break` means the distinction is never reached. - On failure, name the restart counter and point at the journal — `NRestarts` climbing is the single clearest signal of this shape. - Do not lengthen the wait as the fix. A longer sleep with the same early break has the same hole. ## Tasks - [ ] Replace the break-on-first-sighting loop with a settled-state assert - [ ] Refuse when the unit is `activating` at the end of the window, naming the restart count - [ ] `test/cli.sh`: stub `systemctl` so a unit that reports `active` once then `activating` fails the assert - [ ] `changelog.d/<n>.md` ## Acceptance criteria - [ ] A crash-looping unit makes `install` exit non-zero — driven with a stubbed `systemctl`, not reasoned - [ ] A genuinely healthy unit still passes, unchanged - [ ] The refusal names the journal and the restart counter - [ ] `bash test/cli.sh` passes; CI's `shellcheck -x` step is clean ## Test plan - Stub `systemctl` to report `active` on the first call and `activating` thereafter — today's code passes, the fix must fail. - **Must fail:** revert to the early `break` and the new test goes red. - A stub reporting `active` throughout must still succeed. ## Dependencies Surfaced by #141 (no Docker preflight), which is what made the daemon exit — but independent of it: this assert would have lied for any startup failure, not only a missing Docker. --- @andres — filed from your ci-box report, not to be fixed yet, per your instruction.