rig made every box except one: the Incus host that runs the claudeboxes was
hand-built, so "every box is rig-made, reproducibly" had a hole exactly where
an agent runs. Add `rig bootstrap dev`, same shape as the other roles —
idempotent, convergent, a second run is a no-op.
dev reuses all the shared machinery (00-rig.conf sshd drop-in + sshd -T
effective assert, hostname convergence, tailscale join) and adds:
- Tag policy: defaults TS_TAG to tag:local and REFUSES tag:server (exit 2,
mirroring the runner refusal). tag:server's ACL grants :22, so a mis-tagged
dev host would hand the control plane free SSH — the exact bug that made both
M900s retag-by-hand jobs. Correct-tag-only is enforced, not documented.
- Incus install + init, dev-only: apt-get install incus, then `incus admin
init --auto` ONCE. --auto is not idempotent, so a prior init is detected by
its artefacts (a storage pool AND a root disk on the default profile) and
re-init skipped, keeping a second run a true no-op. Effective state is
asserted after (default profile root disk, incusbr0) rather than trusting
init's exit code.
- A comment recording that the guest claudeboxes deliberately do NOT join the
tailnet — the host joins, guests are reached via ProxyJump through it; an
"enrol the guests" convenience would be the bug.
Unit tests cover arg parsing and the tag:server refusal; incus init and the
effective tag:local assertion need a real host and belong in the rehearsal.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`bootstrap` skips `tailscale up` when the box has already joined — which is what
lets a re-run work without spending a pre-auth key. But skipping it also skipped
`--hostname`, so the TAILNET name never converged: a box that joined under the
wrong name (e.g. `--hostname` omitted, defaulting to the ROLE) stayed misnamed
permanently, and re-running rig — the documented repair path — could not fix it.
rig is convergent by contract; this was the one field that wasn't.
The already-joined path now compares the current tailnet hostname against
`--hostname` and reconciles it with `tailscale set --hostname` (no re-auth, no
key). A failure warns rather than dying: a cosmetic name is not worth failing a
bootstrap over.
Safe by construction: Tailscale ACLs cannot bind a rule's dst to a hostname (it
must be a tag, an IP, or a `hosts` alias — which is precisely why acl.hujson pins
coolify-box to an IP), so a rename cannot silently void a grant. It also will not
clobber a deliberate rename — a machine renamed in the admin console keeps that
name and is no longer overridden by the device hostname.
Found in the same session as the sshd first-wins bug (#6): an operator ran
`bootstrap control-plane` on the prod box without `--hostname`, and no re-run
could undo the resulting name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit restarted ssh and only checked `sshd -T` afterwards. On a
box whose only door is SSH, restarting against a config sshd refuses to parse
leaves no listener and no way back in — the same shape as the firewall-before-
bootstrap lockout this session already found in the migration runbook: commit
to the irreversible act, then verify.
Now `sshd -t` parses the MERGED config (our drop-in, cloud-init's, and any
third-party file) before the restart; on failure the drop-in is rolled back and
the daemon is left untouched. Verified: a bad neighbour drop-in exits 255 and
never reaches `systemctl restart`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rig wrote its hardening drop-in as 99-rig.conf. Cloud images ship
50-cloud-init.conf carrying `PasswordAuthentication yes`, and sshd_config is
FIRST-wins ("for each keyword, the first obtained value will be used" —
sshd_config(5)) with Include expanding its glob in lexical order. So 50-
was read before 99- and every keyword rig set was silently discarded.
Every Hetzner box rig has bootstrapped was still serving
`passwordauthentication yes` — confirmed today on coolify-box (CX23) and
ci-runner (CX43) by `sshd -T`, and from off-tailnet by
`ssh -o PreferredAuthentications=none`. Root logins were never exposed
(PermitRootLogin resolved to prohibit-password via Debian's stock config),
but any password-bearing non-root account was reachable on a public port 22.
Three fixes:
1. Name the drop-in 00-rig.conf so it is read first and actually wins; sweep
a stale 99-rig.conf on re-run so existing boxes converge.
2. Assert the EFFECTIVE config with `sshd -T` and refuse to claim a hardened
box otherwise. Asserting that the file existed is what let this ship green
— the Incus rehearsal has no cloud-init drop-in to lose to, so the bug was
invisible to the one gate that could have caught it.
3. Set the system hostname, not just the tailnet one. A box reached as
`coolify-box` greeted the operator as `root@internal-tooling`; the shell
prompt is the only "am I on the right box" signal before a destructive
command, and it was lying on every box rig built.
Also defer the pre-auth key prompt to the join path. rig is convergent by
contract, but re-running it to pick up this fix demanded a Tailscale key it
would never spend — friction on precisely the repair path that matters.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On Debian, /etc/os-release defines VERSION="13 (trixie)". runner-install
sourced it into the main shell for the Debian-family guard, overwriting the
script's empty $VERSION: the latest-release resolution was skipped and the
download URL became .../v13 (trixie)/... -> curl (3) malformed URL. A
--version pin was clobbered the same way (guards run after arg parsing).
Read ID/ID_LIKE via a subshell in both runner-install and bootstrap (same
pattern, no collision there yet), and add a harness guard that fails on any
future main-shell sourcing of os-release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the Incus rehearsal (prod-migration Task 4): cloud images ship
openssh-server, container/VM images do not; the hardening drop-in and
'systemctl restart ssh' both presume it. A rig box is SSH-managed by
definition, so the dependency is explicit now. No-op on cloud images.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
