heavy-duty/rig

Fork 1

bootstrap: create an admin user, and make the root door role-aware — a uniform 'lock root' would cut Coolify off from the fleet #17

New issue

Closed

opened 2026-07-17 14:18:32 +00:00 by dan-claude-bot · 0 comments

dan-claude-bot commented

2026-07-17 14:18:32 +00:00

(Migrated from github.com)

rig hardens the SSH door but never creates a human to walk through it. Every box rig builds is administered as root, and the only reason that is survivable is the prohibit-password drop-in. "Create an admin user and lock root" is the obvious next step — and half of it is a trap that would cut the control plane off from the fleet.

The trap, measured rather than assumed

"Lock root" is not one action. It is four, and they do not behave alike. Verified empirically against a live sshd (OpenSSH 10.0p2 / Debian 13), not inferred from hardening guides:

Measure	Key-based root SSH after	Verdict
`passwd -l root` (shadow → `!*`)	✅ works	harmless
`PermitRootLogin prohibit-password`	✅ works	safe — what rig does today
`usermod --expiredate 1 root`	❌ `Access denied ... by PAM account configuration`	breaks
root shell → `/usr/sbin/nologin`	❌ `This account is currently not available.`	breaks
`PermitRootLogin no`	❌ `Permission denied (publickey)`	breaks

passwd(1) is explicit that locking a password is not disabling an account: "this does not disable the account. The user may still be able to login using another authentication token (e.g. an SSH key). To disable the account, administrators should use usermod --expiredate 1." shadow(5) agrees: a ! or * password field means "the user will not be able to use a unix password to log in (but the user may log in the system by other means)."

So the danger is precisely the guides' advice. Every hardening checklist pairs passwd -l root with usermod --expiredate 1 or a nologin shell. The first half is a no-op on a box whose root has no password anyway (Debian cloud images ship *); the second half is what takes the fleet down. If rig grows a --lock-root, it must mean exactly one of these, and say which.

Cheap footgun to know about: once root's shell is nologin, chsh itself fails (pam_shells). Recover with usermod -s /bin/bash root.

Why `PermitRootLogin no` is not available on every role

Coolify needs root SSH, and more specifically than the README says:

The control plane needs root SSH to itself. Coolify reaches its own host over SSH via host.docker.internal — server 0 is not special-cased in the auth path — and scripts/install.sh hard-fails off root (if [ $EUID != 0 ]; then ... exit; fi, carrying a literal # TODO: Ask for a user). Non-root localhost is unsupported (#4245, open).
Managed servers default to root. Non-root exists (since v4.0.0-beta.259, 2024-04) but is still labelled experimental in v4.1.2, and requires your-user ALL=(ALL) NOPASSWD: ALL — structurally, because Coolify SSHes with -o RequestTTY=no and never pipes a sudo password, so a prompt could not be answered. Docs; installation docs still say "Log in as the root user (non-root users are not fully supported yet)."

workload CAN drop root — via Coolify's non-root mode — but it is not a privilege reduction, and probably not worth it yet. This corrects an earlier claim in this issue that workload's root door "must stay". It doesn't have to; the question is whether closing it buys anything.

Verified in Coolify source (v4.1.2), not inferred:

remote_process() — the central command executor — applies parseCommandsByLineForSudo() whenever Server::isNonRoot() (a plain user !== 'root' string compare, app/Models/Server.php:1416). InstallDocker runs through it, so Coolify installs Docker itself on a non-root box via sudo curl … | sh. The box does not need Docker pre-installed.
The cost is your-user ALL=(ALL) NOPASSWD: ALL — required structurally, since Coolify SSHes -o RequestTTY=no and never pipes a sudo password. The docs call this "not the most secure way… we will improve this in the future, by adding more granular permissions on binaries."

So it works. But three things make it a poor trade today:

NOPASSWD: ALL is root-equivalent. The coolify user can sudo -i at will — strictly more privilege than the docker-group membership rig already refuses in runner install as "a gratuitous path to root". Attribution becomes cooperative, not enforced: it records what Coolify did in normal operation, not what an attacker who took the account did.
The sudo layer is fragile. It is line-by-line string munging against a hand-maintained bash-keyword denylist, plus special-cases like rewriting docker exec → sudo docker exec. Experimental, and labelled so in the UI two years after it landed (v4.0.0-beta.259, 2024-04).
There is no documented root→non-root migration. The docs only cover adding new non-root servers. An already-registered root workload box must be re-onboarded (change the user field, re-select the key — through the #4245 UX bug), not flipped in place.

What closing root actually buys — and a cheaper way to get it. The goal is attribution: "was that Coolify or a person?" That is answerable today, for free — sshd logs the key fingerprint on every login (Accepted publickey for root … SHA256:<fp>), and Coolify's key is distinct from the operator's. auditd adds per-command attribution without touching Coolify's code path. The only thing non-root uniquely unlocks over that is PermitRootLogin no — but prohibit-password already means key-only root with no password surface, so the marginal gain is small against an experimental dependency and a NOPASSWD: ALL that is root by another name.

Recommendation: leave workload on prohibit-password; get attribution from sshd fingerprints + auditd. Revisit if Coolify ships the promised granular-sudo permissions — that is when non-root stops being root-with-extra-steps. The row below reflects this as conditional, not forbidden, so the decision is the operator's and stays revisitable.

So the root door closes per role, and the constraint is real only where something depends on it:

Role	Coolify-managed	Root SSH	Why
`control-plane`	yes — to itself	must stay (`prohibit-password`)	Coolify SSHes to its own host; non-root localhost unimplemented (#4245)
`workload`	yes — remote	can close, conditionally — not recommended yet	requires adopting experimental non-root + `NOPASSWD: ALL`; attribution is cheaper via logging
`runner`	no	can close once an admin user is proven	no Coolify involved
`dev` (#12)	no — never Coolify-registered	can close once an admin user is proven	no Coolify involved

A uniform --lock-root would still be a fleet outage — it would take out control-plane. A role-aware one is safe, and on workload it should be an opt-in the operator reaches for deliberately, not a default rig applies.

Proposed

1. An admin user, on every role. A non-root account with sudo, an SSH key, and no supplementary groups beyond sudo — never docker, same reasoning as runner install.

runner install already sets the precedent for user creation on a rig box (useradd --create-home, runuser, refuses root), so this is a known shape, not a new one.

2. Role-aware root policy. runner and dev may close root's SSH door freely. control-plane must not, and rig should refuse rather than warn if asked — the same way runner refuses tag:server; a flag that silently bricks the control plane's access to itself is worse than no flag. workload is the middle case: closing root there is possible (Coolify non-root mode, see above) but not the default and not recommended yet — if it's ever offered, it must be an explicit opt-in that also provisions the coolify user + NOPASSWD sudoers, never something --lock-root does implicitly.

3. If "lock root" ships at all, it means PermitRootLogin no on the permitted roles — not usermod --expiredate, not a nologin shell, both of which break rig's own bootstrap re-run path too (rig is run as root over SSH; convergence dies with the door). passwd -l root can ship everywhere as a convergent no-op-in-practice, but be honest in the README that it buys approximately nothing on a box whose root has no password to begin with.

The lockout problem — and a way out

Closing root's door on a box whose admin key does not work means rescue mode. rig can verify a lot locally: the account exists and is unlocked, authorized_keys is non-empty with sane ownership/perms, sudo -n true succeeds under runuser, the shell is valid, and sshd -T -C user=<admin> resolves to something that actually permits the login (an AllowUsers/Match block elsewhere can quietly exclude them). What rig cannot verify is that the operator holds the private key. Only an inbound connection proves that.

Seeding the admin's authorized_keys from root's own turns that unprovable claim into a proven one. The operator is right now connected as root using one of those keys — that is live evidence the private key exists and is in their hands, which is strictly better than any check rig could invent. It also needs no new argument, and a public key is not a secret, so nothing about "no credential, ever" bends.

Two wrinkles that need a decision, not a default:

Coolify writes its own key into root's authorized_keys when it registers a server. Blind-copying hands the admin account to Coolify's key. Not a breach — Coolify already has root there — but it is sloppy, and it means the admin's keys drift as Coolify's do.
Convergence vs. intent. Re-seeding every run means a key the operator deliberately removed from the admin account comes back on the next re-run. Seeding once at creation avoids that but is, strictly, not convergent — an honest exception worth naming in the code rather than papering over.
Cloud images can carry command=/from= restrictions or forced commands in root's authorized_keys. Copied verbatim, those silently follow to the admin user.

Open question — where does the admin key come from?

(a) Seed from root's authorized_keys at creation, never re-seed. Zero new arguments, live proof the key works, one honest non-convergent step. My lean.
(b) --admin-key <pubkey> explicit. Convergent, immune to Coolify's key drift and to inherited command= restrictions — but the operator can hand over a key they do not hold, which is exactly the lockout (a) rules out.

(a) is safer against lockout; (b) is cleaner against drift. They can also compose: seed from root, let --admin-key override.

Why this blocks #12

box's host/setup-host.sh is written for a sudo-capable non-root user: it calls sudo apt-get, and if $USER is not in incus-admin it adds the group, prints "re-login and re-run", and exit 0. Run as root out of rig bootstrap dev, that returns success having done half the job — precisely the "claimed success without checking effective state" failure the sshd first-wins bug already taught this repo once.

So the dev role needs the admin user to exist and to be the identity setup-host.sh runs under. This issue is a prerequisite for #12, not a neighbour of it.

Constraints

Nothing org-specific: no real usernames or keys in the repo. The admin user's name is an argument with a generic default.
The admin user gets sudo, never docker.
Argument validation before the root check; exit 2 usage / 1 refusal; convergent second run — the existing global constraints apply unchanged.
Never close root's door in the same breath as opening the admin's, without the checks above passing. If a check fails, the door stays open and rig says why.

Rehearsal

The Incus container rehearsal cannot see this (no cloud-init drop-in, no Coolify). The real rehearsal is a throwaway cloud box: bootstrap, confirm inbound SSH as the admin user from a second terminal — the one check that matters and the one rig cannot self-assert — then close root's door, then re-run rig and confirm it is still convergent through the admin path. Assert effective state (sshd -T -C user=…, sudo -n true), never the files rig wrote.

rig hardens the SSH door but never creates a human to walk through it. Every box rig builds is administered as `root`, and the only reason that is survivable is the `prohibit-password` drop-in. "Create an admin user and lock root" is the obvious next step — and half of it is a trap that would cut the control plane off from the fleet. ## The trap, measured rather than assumed "Lock root" is not one action. It is four, and they do not behave alike. Verified empirically against a live sshd (OpenSSH 10.0p2 / Debian 13), not inferred from hardening guides: | Measure | Key-based root SSH after | Verdict | |---|---|---| | `passwd -l root` (shadow → `!*`) | ✅ works | **harmless** | | `PermitRootLogin prohibit-password` | ✅ works | **safe — what rig does today** | | `usermod --expiredate 1 root` | ❌ `Access denied ... by PAM account configuration` | **breaks** | | root shell → `/usr/sbin/nologin` | ❌ `This account is currently not available.` | **breaks** | | `PermitRootLogin no` | ❌ `Permission denied (publickey)` | **breaks** | `passwd(1)` is explicit that locking a password is not disabling an account: *"this does not disable the account. The user may still be able to login using another authentication token (e.g. an SSH key). To disable the account, administrators should use `usermod --expiredate 1`."* `shadow(5)` agrees: a `!` or `*` password field means *"the user will not be able to use a unix password to log in (but the user may log in the system by other means)."* **So the danger is precisely the guides' advice.** Every hardening checklist pairs `passwd -l root` with `usermod --expiredate 1` or a nologin shell. The first half is a no-op on a box whose root has no password anyway (Debian cloud images ship `*`); the second half is what takes the fleet down. If rig grows a `--lock-root`, it must mean exactly one of these, and say which. > Cheap footgun to know about: once root's shell is `nologin`, `chsh` itself fails (`pam_shells`). Recover with `usermod -s /bin/bash root`. ## Why `PermitRootLogin no` is not available on every role Coolify needs root SSH, and more specifically than the README says: - **The control plane needs root SSH to *itself*.** Coolify reaches its own host over SSH via `host.docker.internal` — server 0 is not special-cased in the auth path — and `scripts/install.sh` hard-fails off root (`if [ $EUID != 0 ]; then ... exit; fi`, carrying a literal `# TODO: Ask for a user`). Non-root localhost is unsupported ([#4245](https://github.com/coollabsio/coolify/issues/4245), open). - **Managed servers default to root.** Non-root exists (since `v4.0.0-beta.259`, 2024-04) but is still labelled **experimental** in `v4.1.2`, and requires `your-user ALL=(ALL) NOPASSWD: ALL` — structurally, because Coolify SSHes with `-o RequestTTY=no` and never pipes a sudo password, so a prompt could not be answered. [Docs](https://coolify.io/docs/knowledge-base/server/non-root-user); installation docs still say *"Log in as the root user (non-root users are not fully supported yet)."* **`workload` CAN drop root — via Coolify's non-root mode — but it is not a privilege reduction, and probably not worth it yet.** This corrects an earlier claim in this issue that workload's root door "must stay". It doesn't have to; the question is whether closing it buys anything. Verified in Coolify source (`v4.1.2`), not inferred: - `remote_process()` — the central command executor — applies `parseCommandsByLineForSudo()` whenever `Server::isNonRoot()` (a plain `user !== 'root'` string compare, `app/Models/Server.php:1416`). `InstallDocker` runs through it, so **Coolify installs Docker itself on a non-root box** via `sudo curl … | sh`. The box does not need Docker pre-installed. - The cost is `your-user ALL=(ALL) NOPASSWD: ALL` — required structurally, since Coolify SSHes `-o RequestTTY=no` and never pipes a sudo password. The docs call this *"not the most secure way… we will improve this in the future, by adding more granular permissions on binaries."* So it works. But three things make it a poor trade **today**: 1. **`NOPASSWD: ALL` is root-equivalent.** The `coolify` user can `sudo -i` at will — strictly *more* privilege than the docker-group membership rig already refuses in `runner install` as "a gratuitous path to root". Attribution becomes **cooperative, not enforced**: it records what Coolify did in normal operation, not what an attacker who took the account did. 2. **The sudo layer is fragile.** It is line-by-line string munging against a hand-maintained bash-keyword denylist, plus special-cases like rewriting `docker exec` → `sudo docker exec`. Experimental, and labelled so in the UI two years after it landed (`v4.0.0-beta.259`, 2024-04). 3. **There is no documented root→non-root migration.** The [docs](https://coolify.io/docs/knowledge-base/server/non-root-user) only cover *adding new* non-root servers. An already-registered root workload box must be re-onboarded (change the user field, re-select the key — through the [#4245](https://github.com/coollabsio/coolify/issues/4245) UX bug), not flipped in place. **What closing root actually buys — and a cheaper way to get it.** The goal is attribution: "was that Coolify or a person?" That is answerable **today, for free** — sshd logs the key fingerprint on every login (`Accepted publickey for root … SHA256:<fp>`), and Coolify's key is distinct from the operator's. `auditd` adds per-command attribution without touching Coolify's code path. The *only* thing non-root uniquely unlocks over that is `PermitRootLogin no` — but `prohibit-password` already means key-only root with no password surface, so the marginal gain is small against an experimental dependency and a `NOPASSWD: ALL` that is root by another name. **Recommendation: leave `workload` on `prohibit-password`; get attribution from sshd fingerprints + `auditd`. Revisit if Coolify ships the promised granular-sudo permissions** — that is when non-root stops being root-with-extra-steps. The row below reflects this as *conditional*, not *forbidden*, so the decision is the operator's and stays revisitable. So the root door closes **per role**, and the constraint is real only where something depends on it: | Role | Coolify-managed | Root SSH | Why | |---|---|---|---| | `control-plane` | yes — *to itself* | **must stay** (`prohibit-password`) | Coolify SSHes to its own host; non-root localhost unimplemented ([#4245](https://github.com/coollabsio/coolify/issues/4245)) | | `workload` | yes — remote | **can close, conditionally** — not recommended yet | requires adopting experimental non-root + `NOPASSWD: ALL`; attribution is cheaper via logging | | `runner` | no | **can close** once an admin user is proven | no Coolify involved | | `dev` (#12) | no — never Coolify-registered | **can close** once an admin user is proven | no Coolify involved | A uniform `--lock-root` would still be a fleet outage — it would take out `control-plane`. A role-aware one is safe, and on `workload` it should be an opt-in the operator reaches for deliberately, not a default rig applies. ## Proposed **1. An admin user, on every role.** A non-root account with `sudo`, an SSH key, and no supplementary groups beyond `sudo` — **never `docker`**, same reasoning as `runner install`. `runner install` already sets the precedent for user creation on a rig box (`useradd --create-home`, `runuser`, refuses `root`), so this is a known shape, not a new one. **2. Role-aware root policy.** `runner` and `dev` may close root's SSH door freely. `control-plane` **must not**, and rig should **refuse** rather than warn if asked — the same way `runner` refuses `tag:server`; a flag that silently bricks the control plane's access to itself is worse than no flag. `workload` is the middle case: closing root there is *possible* (Coolify non-root mode, see above) but **not the default and not recommended yet** — if it's ever offered, it must be an explicit opt-in that also provisions the `coolify` user + `NOPASSWD` sudoers, never something `--lock-root` does implicitly. **3. If "lock root" ships at all, it means `PermitRootLogin no`** on the permitted roles — not `usermod --expiredate`, not a nologin shell, both of which break rig's own `bootstrap` re-run path too (rig is run as root over SSH; convergence dies with the door). `passwd -l root` can ship everywhere as a convergent no-op-in-practice, but be honest in the README that it buys approximately nothing on a box whose root has no password to begin with. ## The lockout problem — and a way out Closing root's door on a box whose admin key does not work means rescue mode. rig can verify a lot locally: the account exists and is unlocked, `authorized_keys` is non-empty with sane ownership/perms, `sudo -n true` succeeds under `runuser`, the shell is valid, and `sshd -T -C user=<admin>` resolves to something that actually permits the login (an `AllowUsers`/`Match` block elsewhere can quietly exclude them). What rig **cannot** verify is that the operator holds the private key. Only an inbound connection proves that. **Seeding the admin's `authorized_keys` from root's own turns that unprovable claim into a proven one.** The operator is *right now* connected as root using one of those keys — that is live evidence the private key exists and is in their hands, which is strictly better than any check rig could invent. It also needs no new argument, and a public key is not a secret, so nothing about "no credential, ever" bends. Two wrinkles that need a decision, not a default: - **Coolify writes its own key into root's `authorized_keys`** when it registers a server. Blind-copying hands the admin account to Coolify's key. Not a breach — Coolify already has root there — but it is sloppy, and it means the admin's keys drift as Coolify's do. - **Convergence vs. intent.** Re-seeding every run means a key the operator *deliberately removed* from the admin account comes back on the next re-run. Seeding once at creation avoids that but is, strictly, not convergent — an honest exception worth naming in the code rather than papering over. - Cloud images can carry `command=`/`from=` restrictions or forced commands in root's `authorized_keys`. Copied verbatim, those silently follow to the admin user. ### Open question — where does the admin key come from? - **(a) Seed from root's `authorized_keys` at creation**, never re-seed. Zero new arguments, live proof the key works, one honest non-convergent step. My lean. - **(b) `--admin-key <pubkey>` explicit.** Convergent, immune to Coolify's key drift and to inherited `command=` restrictions — but the operator can hand over a key they do not hold, which is exactly the lockout (a) rules out. (a) is safer against lockout; (b) is cleaner against drift. They can also compose: seed from root, let `--admin-key` override. ## Why this blocks #12 `box`'s `host/setup-host.sh` is written for a **sudo-capable non-root user**: it calls `sudo apt-get`, and if `$USER` is not in `incus-admin` it adds the group, prints "re-login and re-run", and **`exit 0`**. Run as root out of `rig bootstrap dev`, that returns *success* having done half the job — precisely the "claimed success without checking effective state" failure the sshd first-wins bug already taught this repo once. So the `dev` role needs the admin user to exist *and* to be the identity `setup-host.sh` runs under. This issue is a prerequisite for #12, not a neighbour of it. ## Constraints - Nothing org-specific: no real usernames or keys in the repo. The admin user's name is an argument with a generic default. - The admin user gets `sudo`, never `docker`. - Argument validation before the root check; exit 2 usage / 1 refusal; convergent second run — the existing global constraints apply unchanged. - **Never close root's door in the same breath as opening the admin's, without the checks above passing.** If a check fails, the door stays open and rig says why. ## Rehearsal The Incus container rehearsal cannot see this (no cloud-init drop-in, no Coolify). The real rehearsal is a throwaway cloud box: bootstrap, confirm inbound SSH **as the admin user from a second terminal** — the one check that matters and the one rig cannot self-assert — then close root's door, then re-run rig and confirm it is still convergent *through the admin path*. Assert effective state (`sshd -T -C user=…`, `sudo -n true`), never the files rig wrote.

claude-lead-andresmgsl referenced this issue

2026-07-29 15:23:22 +00:00

feat: install channel is forge-agnostic (RIG_HOST / BOX_HOST) #114

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/rig#17

No description provided.