The Coolify control-plane database holds the GitHub App private key, every
registered server's SSH key, and every environment value for every environment
it manages. Backing it up was a manual runbook step, and the dump script lived
in cast — the off-box tool, whose src never references it. It runs on the box,
as root, under a scheduler: that is rig's job description.
It matters beyond tidiness. The dump is forensics, not a restore path — a lost
control plane is rebuilt fresh and reconciled from the manifest. So there will
be a next control-plane box, and as a runbook step it was born un-backed-up,
depending on someone remembering mid-incident. Now it is backed up from birth.
rig installs the machinery and templates /etc/coolify-dump.env empty at 0600,
never reading it back — no credential passes through rig. The script's own
guards make an unfilled file fail the unit loudly rather than ship plaintext.
systemd timer over cron: EnvironmentFile is the right idiom for 0600 secrets,
failures surface in systemctl status instead of being mailed into the void, and
Persistent=true catches a run missed while the box was down.
Two hazards the cast script missed, carried into the unit:
- aws-cli >= 2.23 enables default upload checksums that S3-compatible backends
reject; Debian 13 ships 2.23.6, so the unit defaults both checksum knobs to
when_required.
- A failed pg_dump piped into age still yields a valid, tiny, encrypted file
that uploads cleanly every night and looks exactly like a working backup. The
script now refuses to upload an empty artifact.
Closes#8
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`bootstrap` skips `tailscale up` when the box has already joined — which is what
lets a re-run work without spending a pre-auth key. But skipping it also skipped
`--hostname`, so the TAILNET name never converged: a box that joined under the
wrong name (e.g. `--hostname` omitted, defaulting to the ROLE) stayed misnamed
permanently, and re-running rig — the documented repair path — could not fix it.
rig is convergent by contract; this was the one field that wasn't.
The already-joined path now compares the current tailnet hostname against
`--hostname` and reconciles it with `tailscale set --hostname` (no re-auth, no
key). A failure warns rather than dying: a cosmetic name is not worth failing a
bootstrap over.
Safe by construction: Tailscale ACLs cannot bind a rule's dst to a hostname (it
must be a tag, an IP, or a `hosts` alias — which is precisely why acl.hujson pins
coolify-box to an IP), so a rename cannot silently void a grant. It also will not
clobber a deliberate rename — a machine renamed in the admin console keeps that
name and is no longer overridden by the device hostname.
Found in the same session as the sshd first-wins bug (#6): an operator ran
`bootstrap control-plane` on the prod box without `--hostname`, and no re-run
could undo the resulting name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit restarted ssh and only checked `sshd -T` afterwards. On a
box whose only door is SSH, restarting against a config sshd refuses to parse
leaves no listener and no way back in — the same shape as the firewall-before-
bootstrap lockout this session already found in the migration runbook: commit
to the irreversible act, then verify.
Now `sshd -t` parses the MERGED config (our drop-in, cloud-init's, and any
third-party file) before the restart; on failure the drop-in is rolled back and
the daemon is left untouched. Verified: a bad neighbour drop-in exits 255 and
never reaches `systemctl restart`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rig wrote its hardening drop-in as 99-rig.conf. Cloud images ship
50-cloud-init.conf carrying `PasswordAuthentication yes`, and sshd_config is
FIRST-wins ("for each keyword, the first obtained value will be used" —
sshd_config(5)) with Include expanding its glob in lexical order. So 50-
was read before 99- and every keyword rig set was silently discarded.
Every Hetzner box rig has bootstrapped was still serving
`passwordauthentication yes` — confirmed today on coolify-box (CX23) and
ci-runner (CX43) by `sshd -T`, and from off-tailnet by
`ssh -o PreferredAuthentications=none`. Root logins were never exposed
(PermitRootLogin resolved to prohibit-password via Debian's stock config),
but any password-bearing non-root account was reachable on a public port 22.
Three fixes:
1. Name the drop-in 00-rig.conf so it is read first and actually wins; sweep
a stale 99-rig.conf on re-run so existing boxes converge.
2. Assert the EFFECTIVE config with `sshd -T` and refuse to claim a hardened
box otherwise. Asserting that the file existed is what let this ship green
— the Incus rehearsal has no cloud-init drop-in to lose to, so the bug was
invisible to the one gate that could have caught it.
3. Set the system hostname, not just the tailnet one. A box reached as
`coolify-box` greeted the operator as `root@internal-tooling`; the shell
prompt is the only "am I on the right box" signal before a destructive
command, and it was lying on every box rig built.
Also defer the pre-auth key prompt to the join path. rig is convergent by
contract, but re-running it to pick up this fix demanded a Tailscale key it
would never spend — friction on precisely the repair path that matters.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On Debian, /etc/os-release defines VERSION="13 (trixie)". runner-install
sourced it into the main shell for the Debian-family guard, overwriting the
script's empty $VERSION: the latest-release resolution was skipped and the
download URL became .../v13 (trixie)/... -> curl (3) malformed URL. A
--version pin was clobbered the same way (guards run after arg parsing).
Read ID/ID_LIKE via a subshell in both runner-install and bootstrap (same
pattern, no collision there yet), and add a harness guard that fails on any
future main-shell sourcing of os-release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the Incus rehearsal (prod-migration Task 4): cloud images ship
openssh-server, container/VM images do not; the hardening drop-in and
'systemctl restart ssh' both presume it. A rig box is SSH-managed by
definition, so the dependency is explicit now. No-op on cloud images.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
