Seven review findings on the users family, each with the harness check that
would have caught it:
- Invoker gate (apply + close-root): %rig's sudoers rule is binary-scoped but
not argument-scoped, so `sudo rig users apply --file <me-as-admin>` made
role rig silently root-equivalent through the very command that granted it.
Identity management now refuses any sudo invoker outside rig-admin; direct
root (bring-up, a root shell) proceeds.
- Offboarding revokes SSH, not just the password: a '!'-locked password is
not a closed door under UsePAM — Debian sshd still honors the pubkey. A
dropped user's account is now expired (usermod -L -e 1, the switch PAM
actually enforces) and authorized_keys is renamed to
authorized_keys.revoked-by-rig — access revoked, data kept, convergence
never destroys. Present users get their expiry cleared idempotently, so a
re-added user comes back to life.
- The ledger remembers: two-field lines ('name active' / 'name revoked',
legacy bare names read as active), so dropped users no longer vanish from
rig's memory on the next rewrite. status now reports the ledger state
corroborated by the account's real expiry — passwd -S read L for everyone
(apply locks all passwords always), so its locked/active was meaningless —
and flags a mismatch loudly as drift.
- Perms are part of the converged state: ~/.ssh and authorized_keys ownership
and mode converge on every run, not only when content changes — StrictModes
treats them as load-bearing, so drifted perms were a broken login that
"already converged" lied about. Only the content write stays cmp-guarded.
- close-root's admin-door gate checks the StrictModes shape per candidate —
ownership, group/world-writability of home/.ssh/authorized_keys, a real
login shell, an unexpired account — and names which check failed. It proves
the door SHOULD open, not that it does; the separate-session advisory stays
load-bearing.
- Usernames are validated in the parser's one-pass refusal matrix
(^[a-z_][a-z0-9_-]{0,31}$): 'fo|o' corrupted the parser's own '|'-delimited
stream, and a leading '-' read as a useradd flag mid-convergence.
- The box role is trait-aware: on a host=no box an absent incus group skips
the role with a warning and converges everything else — one box-role user
in a fleet-wide file must not abort apply everywhere VMs don't live.
host=yes still dies pointing at box setup-host; a classless marker warns
toward a bootstrap re-run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three refusals, one doctrine: detect, refuse, name the repair — and never
back out state rig did not create.
- verify_effective_tag grows the same <back-out|keep> mode discipline as
verify_user_owned. First join keeps the logout-and-die on an untagged key;
the already-joined path now refuses WITHOUT logout — the untagged node may
be a login-joined workstation (untagged by design) that a join=authkey
re-run must not tear off the tailnet. The die names both ways out.
- verify_user_owned fails CLOSED on a stalled backend: empty tags is its
success signal, so a 30s poll that never saw Running waved a tagged node
on a slow tailscaled through as user-owned. state!=Running now dies in
both modes, logging nothing out — nothing was verified, so the repair is
to re-run and verify, not to undo a join that may be fine.
- The permitrootlogin acceptance is class-gated. class=human keeps
no|prohibit-password|without-password (`no` is the close-root state).
class=server accepts only prohibit-password|without-password: root SSH is
the control plane's automation door, and `no` there means a leftover
00-rig-users.conf from a former class=human life has fleet management
silently dead. Refused loudly, drop-in named, never auto-removed —
silently reopening a root door is worse than a loud stop.
Harness greps pin all three die messages so a deleted guard cannot ship
green (repo precedent: the tag-refusal greps).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
class decides root SSH's fate, and this is human's: install
/etc/ssh/sshd_config.d/00-rig-users.conf (PermitRootLogin no), where the NAME
is the mechanism — sshd_config is first-wins, the Include glob expands
lexically, and '-' sorts before '.', so it is read before bootstrap's
00-rig.conf and wins. Gated three ways, no --force: a marker must exist
(never shut the root door blind), it must say class=human (on a server root
is the control plane's automation identity — closing it severs fleet
management), and some rig-admin member must already hold a non-empty
authorized_keys (never close the only door). The gate's policy lives in the
lib as assert_marker_human so the harness proves every refusal against
fixture markers as non-root; RIG_ROLE_MARKER keeps the command pointable at
the same fixtures. Apply is bootstrap's validate-then-apply shape verbatim —
cmp-guard, sshd -t on the merged config before the restart with rollback,
then the sshd -T effective assertion. Bootstrap's own permitrootlogin
assertion widens to accept 'no': the closed door is strictly harder, never
broken, and by first-wins bootstrap cannot reopen it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Operators become a declared fact, not an accumulation of adduser runs: a
line-based, bash-parseable users file (no YAML, no jq — a rig box has
neither) names each user, their roles, and their keys, and apply converges
the box to exactly that. Roles map to groups (admin→rig-admin with full
NOPASSWD sudo, rig→rig sudo for the rig binary only, box→incus with no
sudo — box's setup-host owns Incus, rig only asserts the group). Every
password stays locked always; the SSH key at the door is the
authentication. A user dropped from the file is found via the /etc/rig/users
ledger and locked, never deleted — deleting frees the uid and rots
attribution. The sudoers drop-in lands only after visudo -c passes, because
a bad file under sudoers.d takes down all of sudo. Class never gates apply
(#26: a shared root login is unattributable, so operators belong on every
class); the marker only colors what root SSH does next. The whole file is
validated in one pass before the root check, every error named with its
line, so refusals are provable in the non-root harness through the sourced
parser.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Roles become presets over three orthogonal traits declared in one map:
class (who lives here), host (runs VMs), join (authkey or interactive
login). Every per-role behavior now keys off the traits — the /dev/kvm
advisory rides host=yes, the next-steps log rides class and host — and
tag:server is derived policy, not a trait: only control-plane and
workload are shapes the control plane manages, so every other role
refuses the effective tag. join=login inverts the tag assertion (a
user-owned node must come up untagged; a tag is refused and backed out
on first join, refused without back-out on a box already joined) and
refuses a set TS_AUTHKEY before the root check. The verified shape is
recorded convergently in /etc/rig/role as ground truth for rig users.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
rig used to pass --ts-tag to `tailscale up --advertise-tags`, stating the
tailnet tag a second time with no way to know whether its request and the
key's own tags agreed. It asserted the tag it REQUESTED, never the tag control
GRANTED — the sshd first-wins bug in a different hat, and the same scar (both
M900s joined tag:server, retagged by hand, unnoticed).
Collapse the two sources of truth onto one: the key.
- `tailscale up` drops --advertise-tags; the key's tags apply.
- After join, poll `tailscale status --json` for `.Self.Tags` (netmap ground
truth, not `debug prefs`) until tags appear or BackendState=Running, on BOTH
the fresh-join and already-joined paths.
- UNTAGGED -> hard refusal: `tailscale logout` to back the user-owned node out,
then die naming the fix (mint a tagged key).
- Role policy moves onto the effective tag: a runner must not have tag:server
among the tags the key actually granted. Strictly stronger than before.
- --ts-tag is removed, and dies exit 2 with a message pointing at the key
(consuming its value), not an "unknown flag".
- New array-aware reader json_string_array in lib/runner-config.sh (jq-free,
never fails under set -e), with its own unit tests; bootstrap sources the lib.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`rig runner install --repo <B>` on a box already registered to repo A
treated the mere existence of .runner as "already registered", skipped
configure, restarted the service still pointed at A, and reported success.
--repo was accepted, validated, and then ignored — leaving B with zero
runners and its `runs-on` jobs queued against one that will never come.
This is the natural next command after a partial `repoint`, and the failure
is worse than a no-op: moving a runner between repos is a trust-boundary
act, so quietly putting it back on the old one defeats the point of the move.
Gate install on the repo .runner actually names. Convergence — the property
worth keeping — is untouched: re-running against the repo the box is already
on still skips registration, never prompts for a token, and exits 0.
Skipping when the repo *differs* was never convergence, only a silently
ignored argument, so it now fails and names both repos, pointing at
`runner repoint` (move) or `runner remove` (start over). An unreadable
.runner is refused too — it is no licence to assume a match.
The .runner reader that `status` and `repoint` each carried is lifted into
commands/lib/runner-config.sh, which now also holds the guard. Its json_field
no longer dies bare under `set -o pipefail` when a key is missing, which is
what `status`'s own ${REPO_URL:-unknown} fallback always assumed.
Tests: the guard is exercised against a fixture .runner (refuses another repo
naming both, points at repoint, no-ops on the same repo, passes an
unregistered box, refuses an unreadable one) plus an ordering assertion that
it precedes svc.sh start — reaching it through the CLI would need root and a
really-registered runner, which the dependency-free harness cannot fabricate.
All three mutants (guard deleted, guard comparing nothing, guard moved below
the service start) go red.
Closes#13
runner install is convergent by skipping: it sees a registered runner and
leaves it alone. So rig could create a runner and never move or destroy one,
and re-pointing a box at a different repo meant hand-rolled config.sh/svc.sh
incantations against an install layout only rig knew about.
- status: repo, name, labels, dir, unit — read-only, no token, no network.
- remove: service down, then deregister. --local wipes the box without
contacting GitHub, leaving a stale entry to delete by hand.
- repoint: remove + re-register in one act, keeping the runner's name and
reusing the binary already on the box.
The service always comes down before deregistration in both paths: GitHub's
removal throws "Uninstall service first" while the service is configured, and
--local bypasses that check entirely, which would strand a running service
pointed at deleted config.
repoint collects both tokens up front — a token you turn out not to have must
fail while the runner is still registered, not halfway through the move.
Labels are the sharp edge: GitHub holds them, the runner does not persist
them, and they are what runs-on matches. install now records what it
registered with so repoint and status can read it back; a runner installed
before that has nothing to read, so repoint falls back to the ci-runner
default and warns before it touches anything.
Both found by the first run on a real control-plane box — neither was
reachable by the argument-parsing tests.
$EDITOR is unset on a freshly-bootstrapped server, which is precisely rig's
target environment. The printed next-step `$EDITOR /etc/coolify-dump.env`
expanded to nothing, so bash tried to EXECUTE the 0600 bindings file and said
"Permission denied" — an error that reads like a filesystem problem and is not
one. Print `nano`.
A bare bucket name in S3_BUCKET reads to `aws` as a LOCAL path, so the upload
died with "Invalid argument type" and a usage dump — after pg_dump had run and
age had encrypted 14MB, with nothing in the error pointing at the actual
mistake. The script now validates the bindings up front: S3_BUCKET must be an
s3:// URI, S3_ENDPOINT must carry a scheme. Both fail with the value quoted and
the reason stated, before a database is read.
Note what still cannot be validated, and now says so in the script: age's X25519
header does not reveal its recipient, so a valid-but-WRONG key (staging's
instead of prod's) yields a flawless backup nobody can open. Only decrypting an
artifact proves the recipient. The printed next-steps now walk through that
read-back explicitly, from a machine holding the private key — never the box.
The dump script ships as an embedded heredoc, so a typo in it would first
surface at 04:00 on a live control plane. test/cli.sh now extracts it and
asserts it is valid bash and that both new guards fire.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Coolify control-plane database holds the GitHub App private key, every
registered server's SSH key, and every environment value for every environment
it manages. Backing it up was a manual runbook step, and the dump script lived
in cast — the off-box tool, whose src never references it. It runs on the box,
as root, under a scheduler: that is rig's job description.
It matters beyond tidiness. The dump is forensics, not a restore path — a lost
control plane is rebuilt fresh and reconciled from the manifest. So there will
be a next control-plane box, and as a runbook step it was born un-backed-up,
depending on someone remembering mid-incident. Now it is backed up from birth.
rig installs the machinery and templates /etc/coolify-dump.env empty at 0600,
never reading it back — no credential passes through rig. The script's own
guards make an unfilled file fail the unit loudly rather than ship plaintext.
systemd timer over cron: EnvironmentFile is the right idiom for 0600 secrets,
failures surface in systemctl status instead of being mailed into the void, and
Persistent=true catches a run missed while the box was down.
Two hazards the cast script missed, carried into the unit:
- aws-cli >= 2.23 enables default upload checksums that S3-compatible backends
reject; Debian 13 ships 2.23.6, so the unit defaults both checksum knobs to
when_required.
- A failed pg_dump piped into age still yields a valid, tiny, encrypted file
that uploads cleanly every night and looks exactly like a working backup. The
script now refuses to upload an empty artifact.
Closes#8
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`bootstrap` skips `tailscale up` when the box has already joined — which is what
lets a re-run work without spending a pre-auth key. But skipping it also skipped
`--hostname`, so the TAILNET name never converged: a box that joined under the
wrong name (e.g. `--hostname` omitted, defaulting to the ROLE) stayed misnamed
permanently, and re-running rig — the documented repair path — could not fix it.
rig is convergent by contract; this was the one field that wasn't.
The already-joined path now compares the current tailnet hostname against
`--hostname` and reconciles it with `tailscale set --hostname` (no re-auth, no
key). A failure warns rather than dying: a cosmetic name is not worth failing a
bootstrap over.
Safe by construction: Tailscale ACLs cannot bind a rule's dst to a hostname (it
must be a tag, an IP, or a `hosts` alias — which is precisely why acl.hujson pins
coolify-box to an IP), so a rename cannot silently void a grant. It also will not
clobber a deliberate rename — a machine renamed in the admin console keeps that
name and is no longer overridden by the device hostname.
Found in the same session as the sshd first-wins bug (#6): an operator ran
`bootstrap control-plane` on the prod box without `--hostname`, and no re-run
could undo the resulting name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit restarted ssh and only checked `sshd -T` afterwards. On a
box whose only door is SSH, restarting against a config sshd refuses to parse
leaves no listener and no way back in — the same shape as the firewall-before-
bootstrap lockout this session already found in the migration runbook: commit
to the irreversible act, then verify.
Now `sshd -t` parses the MERGED config (our drop-in, cloud-init's, and any
third-party file) before the restart; on failure the drop-in is rolled back and
the daemon is left untouched. Verified: a bad neighbour drop-in exits 255 and
never reaches `systemctl restart`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rig wrote its hardening drop-in as 99-rig.conf. Cloud images ship
50-cloud-init.conf carrying `PasswordAuthentication yes`, and sshd_config is
FIRST-wins ("for each keyword, the first obtained value will be used" —
sshd_config(5)) with Include expanding its glob in lexical order. So 50-
was read before 99- and every keyword rig set was silently discarded.
Every Hetzner box rig has bootstrapped was still serving
`passwordauthentication yes` — confirmed today on coolify-box (CX23) and
ci-runner (CX43) by `sshd -T`, and from off-tailnet by
`ssh -o PreferredAuthentications=none`. Root logins were never exposed
(PermitRootLogin resolved to prohibit-password via Debian's stock config),
but any password-bearing non-root account was reachable on a public port 22.
Three fixes:
1. Name the drop-in 00-rig.conf so it is read first and actually wins; sweep
a stale 99-rig.conf on re-run so existing boxes converge.
2. Assert the EFFECTIVE config with `sshd -T` and refuse to claim a hardened
box otherwise. Asserting that the file existed is what let this ship green
— the Incus rehearsal has no cloud-init drop-in to lose to, so the bug was
invisible to the one gate that could have caught it.
3. Set the system hostname, not just the tailnet one. A box reached as
`coolify-box` greeted the operator as `root@internal-tooling`; the shell
prompt is the only "am I on the right box" signal before a destructive
command, and it was lying on every box rig built.
Also defer the pre-auth key prompt to the join path. rig is convergent by
contract, but re-running it to pick up this fix demanded a Tailscale key it
would never spend — friction on precisely the repair path that matters.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On Debian, /etc/os-release defines VERSION="13 (trixie)". runner-install
sourced it into the main shell for the Debian-family guard, overwriting the
script's empty $VERSION: the latest-release resolution was skipped and the
download URL became .../v13 (trixie)/... -> curl (3) malformed URL. A
--version pin was clobbered the same way (guards run after arg parsing).
Read ID/ID_LIKE via a subshell in both runner-install and bootstrap (same
pattern, no collision there yet), and add a harness guard that fails on any
future main-shell sourcing of os-release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the Incus rehearsal (prod-migration Task 4): cloud images ship
openssh-server, container/VM images do not; the hardening drop-in and
'systemctl restart ssh' both presume it. A rig box is SSH-managed by
definition, so the dependency is explicit now. No-op on cloud images.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
