All three reviewers, same substance: 'command -v box' proves the CLI
landed, not that setup-host took effect — and the success line claimed
both. The claim is now split to match its proofs: PATH proves the
install; 'box doctor' (box's own effective-state verdict — the daemon
stays box's domain) gates "host set up". A failed doctor WARNS with the
verdict verb and the manual path, and claims nothing it cannot prove.
Also: the coolify marker guard matched 'role=control-plane ' by its
trailing space, coupling it to the marker's field formatting — a bare
'role=control-plane' line now reads the same (claude-bot's nit), with a
fixture proving it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Issue #12's review comment named the failure shape exactly: box's
setup-host is written for a sudo-capable user and one of its paths exits
0 after only adding a group, asking for a re-login — so an installer's
exit code can claim a success that never took effect. That is the sshd
first-wins bug's shape, and rig's doctrine is to assert effective state.
The check stays deliberately light: command -v box proves the one
artifact rig asked the installer for. Anything deeper — daemon, pool,
network — is box's domain; rig never interrogates Incus, so the success
log hands the operator 'box doctor' (box's own effective-state verdict)
instead of reimplementing it. A hollow success WARNS with the manual
pointer, never dies: box is the host extra, and the OS+tailnet core is
already done and asserted by the time this block runs.
Tests grep the shipped script (the check needs root + network to
exercise): the call, the warn wording, the delegation to box doctor,
and a fail-closed line-number assert that the check follows the
installer run. Rides along: the README rename greps (#12) — the stale
heavy-duty/claudebox slug is negative-grepped out for good.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A host=yes box exists to run guest boxes, so bootstrap finishes the job
instead of printing "next: install the box CLI and run 'box setup-host'".
After the role marker is written, on host=yes it installs the box CLI
globally and lets box's OWN setup-host build the Incus stack.
rig DELEGATES to box; it never touches Incus itself — the same design law
`rig users apply` enforces ("rig NEVER installs Incus: box's setup-host
owns the daemon and its group"). rig does not apt-install incus, does not
configure the daemon, does not create the incus group. It runs box's global
installer as root with BOX_YES=1 (non-interactive AND keeps setup-host);
box installs Incus. Two tools converging one daemon is drift by construction.
- Convergent: box's installer is a no-op once box is installed, so re-running
bootstrap changes nothing.
- Opt-out: RIG_SKIP_BOX_INSTALL=1 skips; also skips gracefully (with a manual
pointer) when curl or the network is missing — box is the host EXTRA, so a
failed box install never aborts a bootstrap that otherwise succeeded.
- Pinnable: BOX_REPO / BOX_REF (default heavy-duty/box@main).
- Runs only AFTER the role marker write, so a box that failed to become what
it claims never installs box on a half-built host.
The world-readable global install path (box under /opt/box, readable by every
non-root user) depends on box PR #71; until it merges box's root install lands
in /root. Noted in a comment and the plan doc.
Completes rig#12 (the dev role — the Incus claudebox host) and rig#25
(machine classes: host-class installs box + rig users).
Tests: 8 new bootstrap checks (guard on host=yes, BOX_YES install, pin
defaults, RIG_SKIP_BOX_INSTALL opt-out, negative-grep that rig never
apt-installs incus, box-after-marker ordering, manual-pointer on skip).
154 passed, 0 failed; shellcheck -x clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three refusals, one doctrine: detect, refuse, name the repair — and never
back out state rig did not create.
- verify_effective_tag grows the same <back-out|keep> mode discipline as
verify_user_owned. First join keeps the logout-and-die on an untagged key;
the already-joined path now refuses WITHOUT logout — the untagged node may
be a login-joined workstation (untagged by design) that a join=authkey
re-run must not tear off the tailnet. The die names both ways out.
- verify_user_owned fails CLOSED on a stalled backend: empty tags is its
success signal, so a 30s poll that never saw Running waved a tagged node
on a slow tailscaled through as user-owned. state!=Running now dies in
both modes, logging nothing out — nothing was verified, so the repair is
to re-run and verify, not to undo a join that may be fine.
- The permitrootlogin acceptance is class-gated. class=human keeps
no|prohibit-password|without-password (`no` is the close-root state).
class=server accepts only prohibit-password|without-password: root SSH is
the control plane's automation door, and `no` there means a leftover
00-rig-users.conf from a former class=human life has fleet management
silently dead. Refused loudly, drop-in named, never auto-removed —
silently reopening a root door is worse than a loud stop.
Harness greps pin all three die messages so a deleted guard cannot ship
green (repo precedent: the tag-refusal greps).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
class decides root SSH's fate, and this is human's: install
/etc/ssh/sshd_config.d/00-rig-users.conf (PermitRootLogin no), where the NAME
is the mechanism — sshd_config is first-wins, the Include glob expands
lexically, and '-' sorts before '.', so it is read before bootstrap's
00-rig.conf and wins. Gated three ways, no --force: a marker must exist
(never shut the root door blind), it must say class=human (on a server root
is the control plane's automation identity — closing it severs fleet
management), and some rig-admin member must already hold a non-empty
authorized_keys (never close the only door). The gate's policy lives in the
lib as assert_marker_human so the harness proves every refusal against
fixture markers as non-root; RIG_ROLE_MARKER keeps the command pointable at
the same fixtures. Apply is bootstrap's validate-then-apply shape verbatim —
cmp-guard, sshd -t on the merged config before the restart with rollback,
then the sshd -T effective assertion. Bootstrap's own permitrootlogin
assertion widens to accept 'no': the closed door is strictly harder, never
broken, and by first-wins bootstrap cannot reopen it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Roles become presets over three orthogonal traits declared in one map:
class (who lives here), host (runs VMs), join (authkey or interactive
login). Every per-role behavior now keys off the traits — the /dev/kvm
advisory rides host=yes, the next-steps log rides class and host — and
tag:server is derived policy, not a trait: only control-plane and
workload are shapes the control plane manages, so every other role
refuses the effective tag. join=login inverts the tag assertion (a
user-owned node must come up untagged; a tag is refused and backed out
on first join, refused without back-out on a box already joined) and
refuses a set TS_AUTHKEY before the root check. The verified shape is
recorded convergently in /etc/rig/role as ground truth for rig users.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rig used to pass --ts-tag to `tailscale up --advertise-tags`, stating the
tailnet tag a second time with no way to know whether its request and the
key's own tags agreed. It asserted the tag it REQUESTED, never the tag control
GRANTED — the sshd first-wins bug in a different hat, and the same scar (both
M900s joined tag:server, retagged by hand, unnoticed).
Collapse the two sources of truth onto one: the key.
- `tailscale up` drops --advertise-tags; the key's tags apply.
- After join, poll `tailscale status --json` for `.Self.Tags` (netmap ground
truth, not `debug prefs`) until tags appear or BackendState=Running, on BOTH
the fresh-join and already-joined paths.
- UNTAGGED -> hard refusal: `tailscale logout` to back the user-owned node out,
then die naming the fix (mint a tagged key).
- Role policy moves onto the effective tag: a runner must not have tag:server
among the tags the key actually granted. Strictly stronger than before.
- --ts-tag is removed, and dies exit 2 with a message pointing at the key
(consuming its value), not an "unknown flag".
- New array-aware reader json_string_array in lib/runner-config.sh (jq-free,
never fails under set -e), with its own unit tests; bootstrap sources the lib.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`bootstrap` skips `tailscale up` when the box has already joined — which is what
lets a re-run work without spending a pre-auth key. But skipping it also skipped
`--hostname`, so the TAILNET name never converged: a box that joined under the
wrong name (e.g. `--hostname` omitted, defaulting to the ROLE) stayed misnamed
permanently, and re-running rig — the documented repair path — could not fix it.
rig is convergent by contract; this was the one field that wasn't.
The already-joined path now compares the current tailnet hostname against
`--hostname` and reconciles it with `tailscale set --hostname` (no re-auth, no
key). A failure warns rather than dying: a cosmetic name is not worth failing a
bootstrap over.
Safe by construction: Tailscale ACLs cannot bind a rule's dst to a hostname (it
must be a tag, an IP, or a `hosts` alias — which is precisely why acl.hujson pins
coolify-box to an IP), so a rename cannot silently void a grant. It also will not
clobber a deliberate rename — a machine renamed in the admin console keeps that
name and is no longer overridden by the device hostname.
Found in the same session as the sshd first-wins bug (#6): an operator ran
`bootstrap control-plane` on the prod box without `--hostname`, and no re-run
could undo the resulting name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit restarted ssh and only checked `sshd -T` afterwards. On a
box whose only door is SSH, restarting against a config sshd refuses to parse
leaves no listener and no way back in — the same shape as the firewall-before-
bootstrap lockout this session already found in the migration runbook: commit
to the irreversible act, then verify.
Now `sshd -t` parses the MERGED config (our drop-in, cloud-init's, and any
third-party file) before the restart; on failure the drop-in is rolled back and
the daemon is left untouched. Verified: a bad neighbour drop-in exits 255 and
never reaches `systemctl restart`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rig wrote its hardening drop-in as 99-rig.conf. Cloud images ship
50-cloud-init.conf carrying `PasswordAuthentication yes`, and sshd_config is
FIRST-wins ("for each keyword, the first obtained value will be used" —
sshd_config(5)) with Include expanding its glob in lexical order. So 50-
was read before 99- and every keyword rig set was silently discarded.
Every Hetzner box rig has bootstrapped was still serving
`passwordauthentication yes` — confirmed today on coolify-box (CX23) and
ci-runner (CX43) by `sshd -T`, and from off-tailnet by
`ssh -o PreferredAuthentications=none`. Root logins were never exposed
(PermitRootLogin resolved to prohibit-password via Debian's stock config),
but any password-bearing non-root account was reachable on a public port 22.
Three fixes:
1. Name the drop-in 00-rig.conf so it is read first and actually wins; sweep
a stale 99-rig.conf on re-run so existing boxes converge.
2. Assert the EFFECTIVE config with `sshd -T` and refuse to claim a hardened
box otherwise. Asserting that the file existed is what let this ship green
— the Incus rehearsal has no cloud-init drop-in to lose to, so the bug was
invisible to the one gate that could have caught it.
3. Set the system hostname, not just the tailnet one. A box reached as
`coolify-box` greeted the operator as `root@internal-tooling`; the shell
prompt is the only "am I on the right box" signal before a destructive
command, and it was lying on every box rig built.
Also defer the pre-auth key prompt to the join path. rig is convergent by
contract, but re-running it to pick up this fix demanded a Tailscale key it
would never spend — friction on precisely the repair path that matters.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On Debian, /etc/os-release defines VERSION="13 (trixie)". runner-install
sourced it into the main shell for the Debian-family guard, overwriting the
script's empty $VERSION: the latest-release resolution was skipped and the
download URL became .../v13 (trixie)/... -> curl (3) malformed URL. A
--version pin was clobbered the same way (guards run after arg parsing).
Read ID/ID_LIKE via a subshell in both runner-install and bootstrap (same
pattern, no collision there yet), and add a harness guard that fails on any
future main-shell sourcing of os-release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the Incus rehearsal (prod-migration Task 4): cloud images ship
openssh-server, container/VM images do not; the hardening drop-in and
'systemctl restart ssh' both presume it. A rig box is SSH-managed by
definition, so the dependency is explicit now. No-op on cloud images.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
