fix(bootstrap): sshd hardening never applied on cloud images #6

Merged

dan-claude-bot merged 2 commits from fix/sshd-hardening-first-wins into main

2026-07-12 15:31:35 +00:00

dan-claude-bot commented

2026-07-12 15:30:39 +00:00

(Migrated from github.com)

The bug

rig bootstrap wrote its hardening drop-in as 99-rig.conf. Cloud images ship
/etc/ssh/sshd_config.d/50-cloud-init.conf carrying PasswordAuthentication yes,
and sshd_config is first-wins — "for each keyword, the first obtained value
will be used" (sshd_config(5)) — with Include expanding its glob in lexical
order. So 50- was read before 99-, and every keyword rig set was silently
discarded.

Every Hetzner box rig has bootstrapped was still serving passwordauthentication yes. Confirmed on coolify-box (CX23) and ci-runner (CX43) via sshd -T, and
from off-tailnet via ssh -o PreferredAuthentications=none:

$ sshd -T | grep -Ei 'passwordauthentication|permitrootlogin'
permitrootlogin without-password
passwordauthentication yes          # <-- rig set `no`; cloud-init won
$ ls /etc/ssh/sshd_config.d/
50-cloud-init.conf  99-rig.conf

Root logins were never exposed (PermitRootLogin resolved to prohibit-password
via Debian's stock config, not via rig), but any password-bearing non-root account
was reachable on a public port 22.

Why it shipped green

bootstrap asserted that the file existed, not what sshd actually resolved —
and the Incus rehearsal runs in a pristine Debian container that has no cloud-init
drop-in to lose to. The one gate that could have caught this was structurally
blind to it. That is the real defect; the filename is just its symptom.

Fixes

Name the drop-in 00-rig.conf so it is read first and actually wins. Sweeps a
stale 99-rig.conf on re-run, so existing boxes converge by re-running bootstrap.
Assert the effective config (sshd -T), and refuse to claim a hardened box
otherwise. Verifying the artifact instead of the effect is what let this through.
Validate the merged config before bouncing the daemon. The first cut restarted
ssh and checked afterwards — on a box whose only door is SSH, restarting against a
config sshd won't parse leaves no listener and no way back. sshd -t now parses
everything sshd would (ours, cloud-init's, any third-party file); on failure the
drop-in is rolled back and the daemon is left untouched. Verified: a bad neighbour
drop-in exits 255 and never reaches systemctl restart.
Set the system hostname, not just the tailnet one. A box reached as coolify-box
greeted the operator as root@internal-tooling. The shell prompt is the only "am I
on the right box" signal before a destructive command, and it was lying on every box
rig built — it very nearly sent a diagnostic to the wrong host while this bug was
being chased.
Defer the pre-auth key prompt to the join path. rig is convergent by contract, but
re-running it to pick up this fix demanded a Tailscale key it would never spend —
friction on precisely the repair path that matters.

Operator follow-up

Re-run rig bootstrap on coolify-box and ci-runner once this lands; both converge
(no pre-auth key needed — they are already joined). Bootstrap the new prod CX33 with the
fixed rig from the start.

bash -n + shellcheck clean.

🤖 Generated with Claude Code

## The bug `rig bootstrap` wrote its hardening drop-in as `99-rig.conf`. Cloud images ship `/etc/ssh/sshd_config.d/50-cloud-init.conf` carrying `PasswordAuthentication yes`, and **`sshd_config` is first-wins** — *"for each keyword, the first obtained value will be used"* (`sshd_config(5)`) — with `Include` expanding its glob in lexical order. So `50-` was read before `99-`, and every keyword rig set was silently discarded. **Every Hetzner box rig has bootstrapped was still serving `passwordauthentication yes`.** Confirmed on `coolify-box` (CX23) and `ci-runner` (CX43) via `sshd -T`, and from off-tailnet via `ssh -o PreferredAuthentications=none`: ``` $ sshd -T | grep -Ei 'passwordauthentication|permitrootlogin' permitrootlogin without-password passwordauthentication yes # <-- rig set `no`; cloud-init won $ ls /etc/ssh/sshd_config.d/ 50-cloud-init.conf 99-rig.conf ``` Root logins were never exposed (`PermitRootLogin` resolved to `prohibit-password` via Debian's stock config, not via rig), but any password-bearing non-root account was reachable on a public port 22. ## Why it shipped green `bootstrap` asserted that *the file existed*, not what `sshd` actually resolved — and the Incus rehearsal runs in a pristine Debian container that has **no cloud-init drop-in to lose to**. The one gate that could have caught this was structurally blind to it. That is the real defect; the filename is just its symptom. ## Fixes 1. **Name the drop-in `00-rig.conf`** so it is read first and actually wins. Sweeps a stale `99-rig.conf` on re-run, so existing boxes converge by re-running `bootstrap`. 2. **Assert the effective config** (`sshd -T`), and refuse to claim a hardened box otherwise. Verifying the artifact instead of the effect is what let this through. 3. **Validate the merged config before bouncing the daemon.** The first cut restarted `ssh` and checked afterwards — on a box whose only door is SSH, restarting against a config `sshd` won't parse leaves no listener and no way back. `sshd -t` now parses everything `sshd` would (ours, cloud-init's, any third-party file); on failure the drop-in is rolled back and the daemon is left untouched. Verified: a bad neighbour drop-in exits 255 and never reaches `systemctl restart`. 4. **Set the system hostname**, not just the tailnet one. A box reached as `coolify-box` greeted the operator as `root@internal-tooling`. The shell prompt is the only "am I on the right box" signal before a destructive command, and it was lying on every box rig built — it very nearly sent a diagnostic to the wrong host while this bug was being chased. 5. **Defer the pre-auth key prompt to the join path.** rig is convergent by contract, but re-running it to *pick up this fix* demanded a Tailscale key it would never spend — friction on precisely the repair path that matters. ## Operator follow-up Re-run `rig bootstrap` on `coolify-box` and `ci-runner` once this lands; both converge (no pre-auth key needed — they are already joined). Bootstrap the new prod CX33 with the fixed rig from the start. `bash -n` + `shellcheck` clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/rig#6

No description provided.