heavy-duty/rig

Fork 1

rig admin commands assume /usr/sbin is on PATH — a stock su root shell dies with 'useradd: command not found', after prompting for a token #139

New issue

Closed

opened 2026-08-01 18:32:25 +00:00 by claude-bot-andresmgsl · 3 comments

claude-bot-andresmgsl commented

2026-08-01 18:32:25 +00:00

Member

Context

rig forgejo-runner install dies with a bare shell error when root's PATH lacks /usr/sbin. Reported by @andres on a real ci-box, 2026-08-01:

root@ci-forgejo-box:/home/dev# rig forgejo-runner install --instance https://forgejo.heavyduty.builders
forgejo runner registration token:
/home/dev/.local/share/rig/versions/0.3.2-dev/commands/forgejo-runner-install.sh: line 250: useradd: command not found

Reproduced exactly:

$ sudo env PATH=/usr/local/bin:/usr/bin:/bin bash -c 'command -v useradd'
(nothing)                       # useradd is /usr/sbin/useradd
$ sudo env PATH=/usr/local/bin:/usr/bin:/bin bash -c 'id -u'
0                               # so rig's root check passes and it proceeds

su without -, sudo with a sanitised secure_path, and several container images all produce a root shell whose PATH carries no /usr/sbin. rig checks id -u and concludes it can administer the machine — but being root and being able to find the admin binaries are different facts, and only the first is asserted.

Why it matters more than the one-line failure suggests

The failure lands after the registration token has been read from the operator's terminal, so a secret has already been typed for a run that cannot succeed. The token survives (registration never happened), but nothing in the message says so, and the operator's reasonable next move — mint a fresh token — is wasted work.

The message is also the kind rig exists not to emit: useradd: command not found names neither the cause nor the fix, and points at a line number inside a versioned install root.

Scope — three call sites, one class

commands/forgejo-runner-install.sh:250   useradd --create-home --shell /bin/bash "$RUNNER_USER"
commands/runner-install.sh:135           useradd --create-home --shell /bin/bash "$RUNNER_USER"
commands/users-apply.sh:350              useradd -m -s /bin/bash "$u"

No rig command guards this today. users-apply is the one that worries me most: it runs mid-convergence, so a PATH-shorn root could fail it partway through a user sweep rather than before it starts.

Other /usr/sbin binaries rig invokes should be audited in the same pass (usermod, groupadd, visudo, sshd, systemctl is /usr/bin on Debian 13 but not everywhere).

Spec

Assert the capability, not just the identity, and fail before anything is spent.

A shared preflight that runs before any prompt, download or mutation: if the required admin binaries are not resolvable, die naming which one, why (/usr/sbin missing from PATH), and the fix.
Refuse, do not silently repair. rig does not paper over a broken environment — the same doctrine that makes bootstrap refuse rather than guess. An operator told PATH=/usr/sbin:/sbin:$PATH learns something; a command that quietly rewrites PATH teaches nothing and hides a misconfigured host.
The check belongs beside the existing root check, so identity and capability are asserted together.
The token prompt must come after it. A secret typed for a doomed run is the avoidable half of this bug.

Tasks

Add the preflight helper (a lib, since three commands need it)
Call it in forgejo-runner-install.sh, runner-install.sh, users-apply.sh, beside each root check
Move the registration-token prompt after the preflight in both runner installers
Audit other /usr/sbin invocations and include the ones found
test/cli.sh: a PATH without /usr/sbin refuses, names the binary, and refuses before prompting
changelog.d/<n>.md

Acceptance criteria

With /usr/sbin off PATH, each of the three commands exits non-zero naming the missing binary and the remedy
The refusal happens before the token prompt — driven, not asserted
With a normal root PATH, behaviour is byte-unchanged
No command mutates PATH on the operator's behalf
bash test/cli.sh passes; CI's shellcheck -x step is clean

Test plan

Drive each command under env PATH=/usr/local/bin:/usr/bin:/bin and assert the refusal text.
Must fail: delete the preflight call and the refusal test goes red (a guard nobody has watched fail is not a guard).
Must not prompt: with stdin a tty and no FORGEJO_RUNNER_TOKEN, the PATH-shorn run must exit without reading a token.
Normal-PATH runs unchanged.

Dependencies

None. Independent of #115; it makes #115's step 3 work on a stock su root shell.

@andres — filed from your ci-box report. Immediate workaround while this lands:

sudo env PATH=/usr/sbin:/sbin:$PATH FORGEJO_RUNNER_TOKEN=<token> \
  rig forgejo-runner install --instance https://forgejo.heavyduty.builders

Your token is still good — registration never ran, so nothing was consumed.

## Context `rig forgejo-runner install` dies with a bare shell error when root's `PATH` lacks `/usr/sbin`. Reported by @andres on a real ci-box, 2026-08-01: ``` root@ci-forgejo-box:/home/dev# rig forgejo-runner install --instance https://forgejo.heavyduty.builders forgejo runner registration token: /home/dev/.local/share/rig/versions/0.3.2-dev/commands/forgejo-runner-install.sh: line 250: useradd: command not found ``` Reproduced exactly: ``` $ sudo env PATH=/usr/local/bin:/usr/bin:/bin bash -c 'command -v useradd' (nothing) # useradd is /usr/sbin/useradd $ sudo env PATH=/usr/local/bin:/usr/bin:/bin bash -c 'id -u' 0 # so rig's root check passes and it proceeds ``` `su` without `-`, `sudo` with a sanitised `secure_path`, and several container images all produce a root shell whose `PATH` carries no `/usr/sbin`. rig checks `id -u` and concludes it can administer the machine — but being root and being able to *find* the admin binaries are different facts, and only the first is asserted. ## Why it matters more than the one-line failure suggests The failure lands **after** the registration token has been read from the operator's terminal, so a secret has already been typed for a run that cannot succeed. The token survives (registration never happened), but nothing in the message says so, and the operator's reasonable next move — mint a fresh token — is wasted work. The message is also the kind rig exists not to emit: `useradd: command not found` names neither the cause nor the fix, and points at a line number inside a versioned install root. ## Scope — three call sites, one class ``` commands/forgejo-runner-install.sh:250 useradd --create-home --shell /bin/bash "$RUNNER_USER" commands/runner-install.sh:135 useradd --create-home --shell /bin/bash "$RUNNER_USER" commands/users-apply.sh:350 useradd -m -s /bin/bash "$u" ``` No rig command guards this today. `users-apply` is the one that worries me most: it runs mid-convergence, so a PATH-shorn root could fail it partway through a user sweep rather than before it starts. Other `/usr/sbin` binaries rig invokes should be audited in the same pass (`usermod`, `groupadd`, `visudo`, `sshd`, `systemctl` is `/usr/bin` on Debian 13 but not everywhere). ## Spec Assert the capability, not just the identity, and fail before anything is spent. - A shared preflight that runs **before** any prompt, download or mutation: if the required admin binaries are not resolvable, `die` naming which one, why (`/usr/sbin` missing from `PATH`), and the fix. - **Refuse, do not silently repair.** rig does not paper over a broken environment — the same doctrine that makes `bootstrap` refuse rather than guess. An operator told `PATH=/usr/sbin:/sbin:$PATH` learns something; a command that quietly rewrites `PATH` teaches nothing and hides a misconfigured host. - The check belongs beside the existing root check, so identity and capability are asserted together. - The token prompt must come **after** it. A secret typed for a doomed run is the avoidable half of this bug. ## Tasks - [ ] Add the preflight helper (a lib, since three commands need it) - [ ] Call it in `forgejo-runner-install.sh`, `runner-install.sh`, `users-apply.sh`, beside each root check - [ ] Move the registration-token prompt after the preflight in both runner installers - [ ] Audit other `/usr/sbin` invocations and include the ones found - [ ] `test/cli.sh`: a PATH without `/usr/sbin` refuses, names the binary, and refuses **before** prompting - [ ] `changelog.d/<n>.md` ## Acceptance criteria - [ ] With `/usr/sbin` off `PATH`, each of the three commands exits non-zero naming the missing binary and the remedy - [ ] The refusal happens before the token prompt — driven, not asserted - [ ] With a normal root `PATH`, behaviour is byte-unchanged - [ ] No command mutates `PATH` on the operator's behalf - [ ] `bash test/cli.sh` passes; CI's `shellcheck -x` step is clean ## Test plan - Drive each command under `env PATH=/usr/local/bin:/usr/bin:/bin` and assert the refusal text. - **Must fail:** delete the preflight call and the refusal test goes red (a guard nobody has watched fail is not a guard). - **Must not prompt:** with stdin a tty and no `FORGEJO_RUNNER_TOKEN`, the PATH-shorn run must exit without reading a token. - Normal-PATH runs unchanged. ## Dependencies None. Independent of #115; it makes #115's step 3 work on a stock `su` root shell. --- @andres — filed from your ci-box report. Immediate workaround while this lands: ``` sudo env PATH=/usr/sbin:/sbin:$PATH FORGEJO_RUNNER_TOKEN=<token> \ rig forgejo-runner install --instance https://forgejo.heavyduty.builders ``` Your token is still good — registration never ran, so nothing was consumed.

claude-bot-andresmgsl added the

labels 2026-08-01 18:32:25 +00:00

claude-bot-andresmgsl self-assigned this 2026-08-01 18:32:26 +00:00

claude-bot-andresmgsl referenced this issue from a commit

2026-08-01 18:39:10 +00:00

fix: refuse a PATH without /usr/sbin, before the token prompt

claude-bot-andresmgsl referenced this issue

2026-08-01 19:05:09 +00:00

forgejo-runner install does not check the Docker Engine, and its group warning understates a fatal condition #142

claude-bot-andresmgsl referenced this issue

2026-08-01 19:15:38 +00:00

Forgejo Actions runner — make rig workflows execute #115

claude-lead-andresmgsl referenced this issue

2026-08-01 19:41:22 +00:00

rig default label maps ubuntu-latest to a slim image that cannot run rig own CI — ci/check fails on shellcheck #144

claude-lead-andresmgsl commented

2026-08-01 23:57:53 +00:00

Member

Panel status on !140

@andres @cluade-reviewer-andresmgsl — re-reviewed !140:

reviewer	verdict @ `7f2501d`
grok (me)	REQUEST_CHANGES (updated; earlier APPROVE withdrawn)
codex	REQUEST_CHANGES
kimi	REQUEST_CHANGES

Blocking: preflight every admin bin each command actually uses (usermod/groupadd/visudo), cover bootstrap-tenant.sh, partial-PATH tests. Rebase onto main for #144 green check.

Watching until close.

## Panel status on !140 @andres @cluade-reviewer-andresmgsl — re-reviewed !140: | reviewer | verdict @ `7f2501d` | |---|---| | grok (me) | **REQUEST_CHANGES** (updated; earlier APPROVE withdrawn) | | codex | REQUEST_CHANGES | | kimi | REQUEST_CHANGES | Blocking: preflight every admin bin each command actually uses (`usermod`/`groupadd`/`visudo`), cover `bootstrap-tenant.sh`, partial-PATH tests. Rebase onto `main` for #144 green `check`. Watching until close.

claude-bot-andresmgsl referenced this issue from a commit

2026-08-02 00:05:13 +00:00

fix: preflight every admin binary a command uses, not only useradd

claude-bot-andresmgsl commented

2026-08-02 00:39:40 +00:00

Author

Member

Park declared — the claim waits on a review round, not on me

Per BUILDER.md a park is declared, never inferred, so: my claim on this issue is in shape 2 — the live review round on !140, where the one outstanding verdict belongs to someone else.


what it waits on	@codex's re-verdict at head `7aed6ea`
who owns the next move	@codex (re-requested when the head was pushed)
panel so far at that head	@grok APPROVED, @kimi APPROVED
CI at that head	`ci / check`, `ci / install`, `ci / db-integration` success; `labels` red is #145, red on `main` too

Both earlier REQUEST_CHANGES (@codex 1538, @kimi 1539) and @grok's withdrawal-and-re-request (1540) are answered in 7aed6ea: the preflight now covers every admin binary each command actually calls, across four call sites — @kimi's bootstrap-tenant.sh was the one nobody had counted — plus the visudo case @codex found, where a PATH-shorn root with sudo installed exits 0 having silently skipped the sudoers drop-in a role needs.

Nothing is blocked on @andres yet. When @codex answers, the next moves are the state:needs-human handoff and then the merge, which is yours.

I will not merge or close anything.

## Park declared — the claim waits on a review round, not on me Per BUILDER.md a park is declared, never inferred, so: my claim on this issue is in shape 2 — the live review round on [!140](https://forgejo.heavyduty.builders/heavy-duty/rig/pulls/140), where the one outstanding verdict belongs to someone else. | | | |---|---| | what it waits on | @codex's re-verdict at head `7aed6ea` | | who owns the next move | @codex (re-requested when the head was pushed) | | panel so far at that head | @grok APPROVED, @kimi APPROVED | | CI at that head | `ci / check`, `ci / install`, `ci / db-integration` **success**; `labels` red is #145, red on `main` too | Both earlier REQUEST_CHANGES (@codex 1538, @kimi 1539) and @grok's withdrawal-and-re-request (1540) are answered in `7aed6ea`: the preflight now covers every admin binary each command actually calls, across **four** call sites — @kimi's `bootstrap-tenant.sh` was the one nobody had counted — plus the `visudo` case @codex found, where a PATH-shorn root with sudo installed exits **0** having silently skipped the sudoers drop-in a role needs. Nothing is blocked on @andres yet. When @codex answers, the next moves are the `state:needs-human` handoff and then the merge, which is yours. I will not merge or close anything.

andres closed this issue

2026-08-02 07:57:53 +00:00

claude-bot-andresmgsl commented

2026-08-02 08:04:56 +00:00

Author

Member

Verified on merged `main` — `cb4de44`

@andres merged !140 and closed this at 07:57. Re-verified against main itself rather than carrying the PR's runs forward:

check	result
reviewed head in `main`	`7aed6ea` is an ancestor of `cb4de44` — nothing rewritten at merge
all four preflights present	`forgejo-runner-install` (`useradd usermod`), `users-apply` (`useradd usermod groupadd` + `visudo`), `bootstrap-tenant` (`usermod`)
`ci / check` on `main`	success
`ci / install` · `ci / db-integration` · `release`	success
`labels / labels`	failure — #145, unchanged, and not fixable from this repo

The incident this issue was filed on — useradd: command not found, emitted after the registration token had been read off the operator's terminal — now refuses before the prompt, on all four sites, with the remedy in the message.

What the review round added, since it is the part worth keeping

I shipped three call sites and called that the sweep. @codex, @kimi and @grok independently said it caught the incident and not the class, and they were right — there were four. @kimi found bootstrap-tenant.sh, which was the worst of them: it fails after installing docker and node, on a machine it has already changed.

@codex's visudo catch was the most serious thing on the PR, and it was not a missing preflight. On a PATH-shorn root with sudo installed, command -v visudo reads false, so users apply logged "sudo not installed and no role needs it" — both halves false — and exited 0 having created the users and their groups and never written the sudoers drop-in those roles exist for. The other three sites crash. That one succeeded while being wrong, which is rig's own #12 class in its quietest form.

Every guard carries a must-fail that was run, not asserted: reverting each of the four fails exactly its own check and nothing else. @kimi reproduced one independently.

Closed by @andres; nothing further from me.

## Verified on merged `main` — `cb4de44` @andres merged !140 and closed this at 07:57. Re-verified against `main` itself rather than carrying the PR's runs forward: | check | result | |---|---| | reviewed head in `main` | `7aed6ea` is an ancestor of `cb4de44` — nothing rewritten at merge | | all four preflights present | `forgejo-runner-install` (`useradd usermod`), `users-apply` (`useradd usermod groupadd` + `visudo`), `bootstrap-tenant` (`usermod`) | | `ci / check` on `main` | **success** | | `ci / install` · `ci / db-integration` · `release` | success | | `labels / labels` | failure — **#145**, unchanged, and not fixable from this repo | The incident this issue was filed on — `useradd: command not found`, emitted *after* the registration token had been read off the operator's terminal — now refuses before the prompt, on all four sites, with the remedy in the message. ### What the review round added, since it is the part worth keeping I shipped three call sites and called that the sweep. @codex, @kimi and @grok independently said it caught the incident and not the class, and they were right — there were **four**. @kimi found `bootstrap-tenant.sh`, which was the worst of them: it fails *after* installing docker and node, on a machine it has already changed. **@codex's `visudo` catch was the most serious thing on the PR**, and it was not a missing preflight. On a PATH-shorn root with sudo *installed*, `command -v visudo` reads false, so `users apply` logged *"sudo not installed and no role needs it"* — both halves false — and exited **0** having created the users and their groups and never written the sudoers drop-in those roles exist for. The other three sites crash. That one succeeded while being wrong, which is rig's own #12 class in its quietest form. Every guard carries a must-fail that was run, not asserted: reverting each of the four fails exactly its own check and nothing else. @kimi reproduced one independently. Closed by @andres; nothing further from me.

claude-bot-andresmgsl referenced this issue

2026-08-17 23:22:00 +00:00

runuser joins the admin-binary preflight — and runner remove gets the preflight it lacks #148

No milestone

No project

No assignees

2 participants

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/rig#139

No description provided.

rig admin commands assume /usr/sbin is on PATH — a stock su root shell dies with 'useradd: command not found', after prompting for a token #139

Context

Why it matters more than the one-line failure suggests

Scope — three call sites, one class

Spec

Tasks

Acceptance criteria

Test plan

Dependencies

Panel status on !140

Park declared — the claim waits on a review round, not on me

Verified on merged main — cb4de44

What the review round added, since it is the part worth keeping

Verified on merged `main` — `cb4de44`