URGENT: sshd -t failing for a missing /run/sshd is reported as 'sshd rejects the merged config' — bootstrap blocked, stderr discarded #92

New issue

Closed

opened 2026-07-20 17:48:50 +00:00 by claude-bot-andresmgsl · 0 comments

claude-bot-andresmgsl commented

2026-07-20 17:48:50 +00:00

(Migrated from github.com)

Summary

harden_sshd() treats any non-zero exit from sshd -t as "the merged config is invalid", and discards the stderr that says otherwise. When sshd -t fails for an environmental reason — most commonly a missing /run/sshd privilege-separation directory — bootstrap aborts with a diagnosis that is simply false, and sends the operator to audit config files that were never broken.

rig-bootstrap: ERROR: sshd rejects the merged config; drop-in rolled back, daemon untouched.
Run 'sshd -t' to see which file is bad.

# root@host:~# sshd -t
Missing privilege separation directory: /run/sshd

There is no bad file. sshd -t never reached a parse verdict.

Cause

commands/lib/sshd.sh:45 (and the identical commands/users-close-root.sh:273):

if ! sshd -t 2>/dev/null; then

sshd -t checks for the privsep directory in addition to parsing the config, and exits non-zero if it is absent. 2>/dev/null throws away the single line that distinguishes the two failures, so the exit code alone drives a message that asserts a cause the code never established.

The follow-up hint compounds it: Run 'sshd -t' to see which file is bad — the operator runs it, gets Missing privilege separation directory, and now has an error naming no file at all.

rig never creates /run/sshd and never mentions it (grep -rn 'run/sshd' over the tree: no hits).

Why the directory is missing

/run is a tmpfs. On Debian/Ubuntu, /run/sshd is created by systemd from RuntimeDirectory=sshd in ssh.service — and removed when that unit stops. So it is legitimately absent whenever ssh.service is not the running form of sshd, notably under socket activation (ssh.socket, the default on Ubuntu 22.10+ and Debian 13), where per-connection ssh@.service instances serve the door and no long-lived ssh.service holds the directory open.

This is not an exotic state: it is reachable on a stock, fully-working box where SSH is serving connections normally. The operator is SSHed in through sshd at the moment rig declares sshd's config broken.

Observed in the wild on a box mid-bootstrap immediately after an openssh-server upgrade (needrestart reporting sshd-session on an outdated binary).

Impact

Bootstrap cannot complete on affected hosts. The rollback is correct and the daemon is untouched — this is a false negative, not a lockout — but there is no path forward from the message rig prints.
The stated cause is wrong, so the operator debugs /etc/ssh/sshd_config.d/ and finds nothing.
rig users close-root carries the same defect at users-close-root.sh:273, on the more dangerous path: it is the command that shuts the root door.

Suggested fix

Distinguish "cannot test" from "config is bad". Roughly:

err="$(sshd -t 2>&1)" || {
  case "$err" in
    *"privilege separation directory"*)
      # environmental, not a config verdict — create it and retest
      install -d -m 0755 /run/sshd
      err="$(sshd -t 2>&1)" || { rollback; die "sshd rejects the merged config: $err"; } ;;
    *) rollback; die "sshd rejects the merged config: $err" ;;
  esac
}

Two things worth taking regardless of the mechanism chosen:

Put $err in the die message. The current text asserts a cause and then withholds the evidence. Every variant of this bug is self-diagnosing the moment stderr is surfaced.
Consider sshd -G (OpenSSH 9.4+) for validation — it parses and dumps the effective config without the privsep-directory precondition, which is exactly the narrower question rig is asking here. Worth confirming against the oldest OpenSSH rig supports before relying on it.

Fix both call sites; they are the same three lines.

Workaround

install -d -m 0755 /run/sshd
sshd -t && echo "config was fine all along"
rig bootstrap dev --hostname <name> --users ./users

Note the directory lives on tmpfs and will not survive a reboot — but by then ssh.service recreates it, so this is a one-shot unblock for the bootstrap run, not something to persist.

Marked urgent: it hard-blocks bootstrap on a default configuration of current Debian/Ubuntu, and the error text actively misdirects the diagnosis. rig users close-root shares the flaw.

@heavy-duty/agents — please review.

## Summary `harden_sshd()` treats **any** non-zero exit from `sshd -t` as "the merged config is invalid", and discards the stderr that says otherwise. When `sshd -t` fails for an *environmental* reason — most commonly a missing `/run/sshd` privilege-separation directory — bootstrap aborts with a diagnosis that is simply false, and sends the operator to audit config files that were never broken. ``` rig-bootstrap: ERROR: sshd rejects the merged config; drop-in rolled back, daemon untouched. Run 'sshd -t' to see which file is bad. # root@host:~# sshd -t Missing privilege separation directory: /run/sshd ``` There is no bad file. `sshd -t` never reached a parse verdict. ## Cause `commands/lib/sshd.sh:45` (and the identical `commands/users-close-root.sh:273`): ```bash if ! sshd -t 2>/dev/null; then ``` `sshd -t` checks for the privsep directory *in addition to* parsing the config, and exits non-zero if it is absent. `2>/dev/null` throws away the single line that distinguishes the two failures, so the exit code alone drives a message that asserts a cause the code never established. The follow-up hint compounds it: `Run 'sshd -t' to see which file is bad` — the operator runs it, gets `Missing privilege separation directory`, and now has an error naming no file at all. rig never creates `/run/sshd` and never mentions it (`grep -rn 'run/sshd'` over the tree: no hits). ## Why the directory is missing `/run` is a tmpfs. On Debian/Ubuntu, `/run/sshd` is created by systemd from `RuntimeDirectory=sshd` in `ssh.service` — and removed when that unit stops. So it is legitimately absent whenever `ssh.service` is not the running form of sshd, notably under **socket activation** (`ssh.socket`, the default on Ubuntu 22.10+ and Debian 13), where per-connection `ssh@.service` instances serve the door and no long-lived `ssh.service` holds the directory open. This is not an exotic state: it is reachable on a stock, fully-working box where SSH is serving connections normally. The operator is SSHed in *through* sshd at the moment rig declares sshd's config broken. Observed in the wild on a box mid-bootstrap immediately after an `openssh-server` upgrade (needrestart reporting `sshd-session` on an outdated binary). ## Impact - Bootstrap cannot complete on affected hosts. The rollback is correct and the daemon is untouched — this is a **false negative, not a lockout** — but there is no path forward from the message rig prints. - The stated cause is wrong, so the operator debugs `/etc/ssh/sshd_config.d/` and finds nothing. - `rig users close-root` carries the same defect at `users-close-root.sh:273`, on the more dangerous path: it is the command that shuts the root door. ## Suggested fix Distinguish "cannot test" from "config is bad". Roughly: ```bash err="$(sshd -t 2>&1)" || { case "$err" in *"privilege separation directory"*) # environmental, not a config verdict — create it and retest install -d -m 0755 /run/sshd err="$(sshd -t 2>&1)" || { rollback; die "sshd rejects the merged config: $err"; } ;; *) rollback; die "sshd rejects the merged config: $err" ;; esac } ``` Two things worth taking regardless of the mechanism chosen: 1. **Put `$err` in the die message.** The current text asserts a cause and then withholds the evidence. Every variant of this bug is self-diagnosing the moment stderr is surfaced. 2. **Consider `sshd -G`** (OpenSSH 9.4+) for validation — it parses and dumps the effective config without the privsep-directory precondition, which is exactly the narrower question rig is asking here. Worth confirming against the oldest OpenSSH rig supports before relying on it. Fix both call sites; they are the same three lines. ## Workaround ```bash install -d -m 0755 /run/sshd sshd -t && echo "config was fine all along" rig bootstrap dev --hostname <name> --users ./users ``` Note the directory lives on tmpfs and will not survive a reboot — but by then `ssh.service` recreates it, so this is a one-shot unblock for the bootstrap run, not something to persist. --- Marked urgent: it hard-blocks bootstrap on a default configuration of current Debian/Ubuntu, and the error text actively misdirects the diagnosis. `rig users close-root` shares the flaw. @heavy-duty/agents — please review.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/rig#92

No description provided.