Round-2 convergence (codex + claude-bot): sshd enforces the group
directives against the candidate's ACTUAL membership, and the gate read
only the *Users pair — an admin outside 'AllowGroups sudo' still
reached ADMIN_OK=1, and root closed on a false proof. The gate now
resolves id -Gn and judges both group directives with the *Users
discipline: DenyGroups flags on a held-group literal or ANY
pattern/host-qualified token; AllowGroups, when set, passes only on a
literal token naming a held group (a pattern that would admit proves
nothing — over-refusing stays the safe error). id failing yields no
groups, which makes a set AllowGroups flag: fail closed there too.
Both requested regressions ride the sourced lib (unmet AllowGroups,
DenyGroups naming a held group) plus the pattern/pass cases, and grep
guards pin the shipped gate to the verdicts and to real membership.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three reviewers, same substance, and they were right that it was
lockout-adjacent: the literal grep passed a candidate whom a DenyUsers
PATTERN really denies ('DenyUsers dan*' vs admin 'dan'), and the door
closed on a false proof. The judgment now lives in the lib as a pure
deny_verdict: a literal hit flags, and so does ANY pattern or
host-qualified token — a token the check cannot prove irrelevant counts
as a hit, never as a pass. The asymmetry with AllowUsers is now the
same direction on both sides: every error closes toward repair, never
toward a welded-shut door.
Also (claude-bot): the -C probe resolves Match blocks against a
synthetic addr=127.0.0.1, so Match Address is out of the local proof's
scope — named in --help, the README, and the gate's comment, so the
separate-session advisory reads as load-bearing, not ceremony.
Regressions ride the sourced lib: wildcard (the review's dan* case),
'?', USER@HOST, literal hit, irrelevant-literals pass, plus a grep
guard that the shipped gate consults deny_verdict.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
#17's table said runner 'can close root once an admin user is proven'; the
class model (#26) superseded the per-role call, and close-root refuses on
class=server — runner's class. The gate does not change: the refusal message
now explains itself (server-class machines are automation identities whose
management plane IS root SSH; a CI box meant to be administered like a human
machine is --class human at bootstrap, not an exception), and the README's
identity-model section records the divergence in one paragraph. README also
documents the @root seed token and close-root's reachability proofs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three content-only edits. The philosophy line pointed at
heavy-duty/claudebox, a slug that only works through a GitHub redirect;
the repo is heavy-duty/box now (#12). The identity model section gets
issue #25's at-a-glance class comparison, translated onto the traits
that replaced the class binary, plus the who-installs-what /
who-runs-as-what paragraph. And the host-class box install now says out
loud that it tracks a moving heavy-duty/box@main — box cuts no tags and
its installer resolves refs/heads only, so there is nothing to pin to;
BOX_REPO/BOX_REF are the pin points the day that changes, and
RIG_SKIP_BOX_INSTALL=1 the opt-out. Issue #12's decision was that
silently tracking main on the box that runs the agents is the option
not to pick.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A host=yes box exists to run guest boxes, so bootstrap finishes the job
instead of printing "next: install the box CLI and run 'box setup-host'".
After the role marker is written, on host=yes it installs the box CLI
globally and lets box's OWN setup-host build the Incus stack.
rig DELEGATES to box; it never touches Incus itself — the same design law
`rig users apply` enforces ("rig NEVER installs Incus: box's setup-host
owns the daemon and its group"). rig does not apt-install incus, does not
configure the daemon, does not create the incus group. It runs box's global
installer as root with BOX_YES=1 (non-interactive AND keeps setup-host);
box installs Incus. Two tools converging one daemon is drift by construction.
- Convergent: box's installer is a no-op once box is installed, so re-running
bootstrap changes nothing.
- Opt-out: RIG_SKIP_BOX_INSTALL=1 skips; also skips gracefully (with a manual
pointer) when curl or the network is missing — box is the host EXTRA, so a
failed box install never aborts a bootstrap that otherwise succeeded.
- Pinnable: BOX_REPO / BOX_REF (default heavy-duty/box@main).
- Runs only AFTER the role marker write, so a box that failed to become what
it claims never installs box on a half-built host.
The world-readable global install path (box under /opt/box, readable by every
non-root user) depends on box PR #71; until it merges box's root install lands
in /root. Noted in a comment and the plan doc.
Completes rig#12 (the dev role — the Incus claudebox host) and rig#25
(machine classes: host-class installs box + rig users).
Tests: 8 new bootstrap checks (guard on host=yes, BOX_YES install, pin
defaults, RIG_SKIP_BOX_INSTALL opt-out, negative-grep that rig never
apt-installs incus, box-after-marker ordering, manual-pointer on skip).
154 passed, 0 failed; shellcheck -x clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seven review findings on the users family, each with the harness check that
would have caught it:
- Invoker gate (apply + close-root): %rig's sudoers rule is binary-scoped but
not argument-scoped, so `sudo rig users apply --file <me-as-admin>` made
role rig silently root-equivalent through the very command that granted it.
Identity management now refuses any sudo invoker outside rig-admin; direct
root (bring-up, a root shell) proceeds.
- Offboarding revokes SSH, not just the password: a '!'-locked password is
not a closed door under UsePAM — Debian sshd still honors the pubkey. A
dropped user's account is now expired (usermod -L -e 1, the switch PAM
actually enforces) and authorized_keys is renamed to
authorized_keys.revoked-by-rig — access revoked, data kept, convergence
never destroys. Present users get their expiry cleared idempotently, so a
re-added user comes back to life.
- The ledger remembers: two-field lines ('name active' / 'name revoked',
legacy bare names read as active), so dropped users no longer vanish from
rig's memory on the next rewrite. status now reports the ledger state
corroborated by the account's real expiry — passwd -S read L for everyone
(apply locks all passwords always), so its locked/active was meaningless —
and flags a mismatch loudly as drift.
- Perms are part of the converged state: ~/.ssh and authorized_keys ownership
and mode converge on every run, not only when content changes — StrictModes
treats them as load-bearing, so drifted perms were a broken login that
"already converged" lied about. Only the content write stays cmp-guarded.
- close-root's admin-door gate checks the StrictModes shape per candidate —
ownership, group/world-writability of home/.ssh/authorized_keys, a real
login shell, an unexpired account — and names which check failed. It proves
the door SHOULD open, not that it does; the separate-session advisory stays
load-bearing.
- Usernames are validated in the parser's one-pass refusal matrix
(^[a-z_][a-z0-9_-]{0,31}$): 'fo|o' corrupted the parser's own '|'-delimited
stream, and a leading '-' read as a useradd flag mid-convergence.
- The box role is trait-aware: on a host=no box an absent incus group skips
the role with a warning and converges everything else — one box-role user
in a fleet-wide file must not abort apply everywhere VMs don't live.
host=yes still dies pointing at box setup-host; a classless marker warns
toward a bootstrap re-run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The bootstrap section now documents what shipped: roles as presets over the
three orthogonal traits (class/host/join, every one overridable, custom
states all of them), the derived tag:server policy, the /etc/rig/role marker
that records effective traits so an overridden role never lies, and the
join=login path where the tag assertion inverts — untagged is asserted, a
tag is the refusal. A new identity-model section carries the hybrid access
model: named operators on every class, humans never entering as root, class
deciding root SSH's fate after `rig users apply` — closed on human, kept as
the control plane's automation door on server — with the detection benefit
and the honest attribution-not-privilege caveat stated plainly. Per-command
sections cover apply/status/close-root, including the first-wins drop-in
mechanism and the README-only from= guidance for Coolify's key on servers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The db PR only unit-tested arg parsing; this adds executable proof that
dump/restore actually works end to end.
- test/db-integration.sh: stands up two throwaway Postgres containers whose
superusers DIFFER by construction (src_super vs dst_super), seeds a known
checksummable fixture, runs the real `rig db dump`/`rig db restore`, and
reads the rows back out — proving both invariants db.sh cares about: the
code reads the container's OWN $POSTGRES_USER/$POSTGRES_DB (a hardcoded
`postgres` would break on the non-default source superuser), and
--no-owner --no-acl makes the dump portable across differing superusers (a
plain dump would abort under ON_ERROR_STOP=1 on the missing role). Also
asserts default-outfile naming, restore idempotency (--clean --if-exists),
and the named-[db] scratch-database path. Skips cleanly (exit 0) when Docker
is absent/unreachable or root is unobtainable; always cleans up via trap.
- ci.yml: separate `db-integration` job on ubuntu-latest (Docker preinstalled),
kept apart from the fast shellcheck+cli.sh `check` job so an image pull can't
slow lint feedback.
- README: "Verifying a dump/restore actually works" — the safe manual
round-trip against a real Coolify container via a fresh scratch db, echoing
"a backup you have never read back is not yet a backup."
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rig used to pass --ts-tag to `tailscale up --advertise-tags`, stating the
tailnet tag a second time with no way to know whether its request and the
key's own tags agreed. It asserted the tag it REQUESTED, never the tag control
GRANTED — the sshd first-wins bug in a different hat, and the same scar (both
M900s joined tag:server, retagged by hand, unnoticed).
Collapse the two sources of truth onto one: the key.
- `tailscale up` drops --advertise-tags; the key's tags apply.
- After join, poll `tailscale status --json` for `.Self.Tags` (netmap ground
truth, not `debug prefs`) until tags appear or BackendState=Running, on BOTH
the fresh-join and already-joined paths.
- UNTAGGED -> hard refusal: `tailscale logout` to back the user-owned node out,
then die naming the fix (mint a tagged key).
- Role policy moves onto the effective tag: a runner must not have tag:server
among the tags the key actually granted. Strictly stronger than before.
- --ts-tag is removed, and dies exit 2 with a message pointing at the key
(consuming its value), not an "unknown flag".
- New array-aware reader json_string_array in lib/runner-config.sh (jq-free,
never fails under set -e), with its own unit tests; bootstrap sources the lib.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `rig db dump <container> [outfile]` and
`rig db restore <artifact> <container> [db] [--yes]` — imperative on-box
PostgreSQL tooling, the interactive counterpart to the scheduled,
declarative `coolify backup install`.
Key decisions:
- Dumps carry `--clean --if-exists --no-owner --no-acl`. `--no-owner
--no-acl` is mandatory for cross-instance restores: the target's
superuser differs (Coolify randomizes it), so a plain dump aborts under
ON_ERROR_STOP=1 on the first GRANT/ALTER OWNER for a missing role.
- $POSTGRES_USER/$POSTGRES_DB are read INSIDE the container (single-quoted
`sh -c`), never hardcoded to `postgres` on the host.
- restore connects as the container's own superuser and runs with
ON_ERROR_STOP=1; the optional [db] arg targets a NAMED database in a
shared container, passed in via a container env var rather than string
splicing.
- restore overwrites the target, so it prompts y/N; --yes/--force is the
automation bypass. Artifact existence/non-emptiness is checked before
the confirm gate and before anything touches the DB.
- dump uses pipefail + a sibling temp promoted only on success, and
refuses to keep an empty artifact — a failed pg_dump must never leave a
plausible-looking .gz behind.
Args are validated before the root check (testable without root); guards
are root, Debian-family warn, docker, and gzip/gunzip. Adds CLI tests and
a `### rig db` README section.
Closes#15
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`rig runner install --repo <B>` on a box already registered to repo A
treated the mere existence of .runner as "already registered", skipped
configure, restarted the service still pointed at A, and reported success.
--repo was accepted, validated, and then ignored — leaving B with zero
runners and its `runs-on` jobs queued against one that will never come.
This is the natural next command after a partial `repoint`, and the failure
is worse than a no-op: moving a runner between repos is a trust-boundary
act, so quietly putting it back on the old one defeats the point of the move.
Gate install on the repo .runner actually names. Convergence — the property
worth keeping — is untouched: re-running against the repo the box is already
on still skips registration, never prompts for a token, and exits 0.
Skipping when the repo *differs* was never convergence, only a silently
ignored argument, so it now fails and names both repos, pointing at
`runner repoint` (move) or `runner remove` (start over). An unreadable
.runner is refused too — it is no licence to assume a match.
The .runner reader that `status` and `repoint` each carried is lifted into
commands/lib/runner-config.sh, which now also holds the guard. Its json_field
no longer dies bare under `set -o pipefail` when a key is missing, which is
what `status`'s own ${REPO_URL:-unknown} fallback always assumed.
Tests: the guard is exercised against a fixture .runner (refuses another repo
naming both, points at repoint, no-ops on the same repo, passes an
unregistered box, refuses an unreadable one) plus an ordering assertion that
it precedes svc.sh start — reaching it through the CLI would need root and a
really-registered runner, which the dependency-free harness cannot fabricate.
All three mutants (guard deleted, guard comparing nothing, guard moved below
the service start) go red.
Closes#13
The Coolify control-plane database holds the GitHub App private key, every
registered server's SSH key, and every environment value for every environment
it manages. Backing it up was a manual runbook step, and the dump script lived
in cast — the off-box tool, whose src never references it. It runs on the box,
as root, under a scheduler: that is rig's job description.
It matters beyond tidiness. The dump is forensics, not a restore path — a lost
control plane is rebuilt fresh and reconciled from the manifest. So there will
be a next control-plane box, and as a runbook step it was born un-backed-up,
depending on someone remembering mid-incident. Now it is backed up from birth.
rig installs the machinery and templates /etc/coolify-dump.env empty at 0600,
never reading it back — no credential passes through rig. The script's own
guards make an unfilled file fail the unit loudly rather than ship plaintext.
systemd timer over cron: EnvironmentFile is the right idiom for 0600 secrets,
failures surface in systemctl status instead of being mailed into the void, and
Persistent=true catches a run missed while the box was down.
Two hazards the cast script missed, carried into the unit:
- aws-cli >= 2.23 enables default upload checksums that S3-compatible backends
reject; Debian 13 ships 2.23.6, so the unit defaults both checksum knobs to
when_required.
- A failed pg_dump piped into age still yields a valid, tiny, encrypted file
that uploads cleanly every night and looks exactly like a working backup. The
script now refuses to upload an empty artifact.
Closes#8
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`bootstrap` skips `tailscale up` when the box has already joined — which is what
lets a re-run work without spending a pre-auth key. But skipping it also skipped
`--hostname`, so the TAILNET name never converged: a box that joined under the
wrong name (e.g. `--hostname` omitted, defaulting to the ROLE) stayed misnamed
permanently, and re-running rig — the documented repair path — could not fix it.
rig is convergent by contract; this was the one field that wasn't.
The already-joined path now compares the current tailnet hostname against
`--hostname` and reconciles it with `tailscale set --hostname` (no re-auth, no
key). A failure warns rather than dying: a cosmetic name is not worth failing a
bootstrap over.
Safe by construction: Tailscale ACLs cannot bind a rule's dst to a hostname (it
must be a tag, an IP, or a `hosts` alias — which is precisely why acl.hujson pins
coolify-box to an IP), so a rename cannot silently void a grant. It also will not
clobber a deliberate rename — a machine renamed in the admin console keeps that
name and is no longer overridden by the device hostname.
Found in the same session as the sshd first-wins bug (#6): an operator ran
`bootstrap control-plane` on the prod box without `--hostname`, and no re-run
could undo the resulting name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rig wrote its hardening drop-in as 99-rig.conf. Cloud images ship
50-cloud-init.conf carrying `PasswordAuthentication yes`, and sshd_config is
FIRST-wins ("for each keyword, the first obtained value will be used" —
sshd_config(5)) with Include expanding its glob in lexical order. So 50-
was read before 99- and every keyword rig set was silently discarded.
Every Hetzner box rig has bootstrapped was still serving
`passwordauthentication yes` — confirmed today on coolify-box (CX23) and
ci-runner (CX43) by `sshd -T`, and from off-tailnet by
`ssh -o PreferredAuthentications=none`. Root logins were never exposed
(PermitRootLogin resolved to prohibit-password via Debian's stock config),
but any password-bearing non-root account was reachable on a public port 22.
Three fixes:
1. Name the drop-in 00-rig.conf so it is read first and actually wins; sweep
a stale 99-rig.conf on re-run so existing boxes converge.
2. Assert the EFFECTIVE config with `sshd -T` and refuse to claim a hardened
box otherwise. Asserting that the file existed is what let this ship green
— the Incus rehearsal has no cloud-init drop-in to lose to, so the bug was
invisible to the one gate that could have caught it.
3. Set the system hostname, not just the tailnet one. A box reached as
`coolify-box` greeted the operator as `root@internal-tooling`; the shell
prompt is the only "am I on the right box" signal before a destructive
command, and it was lying on every box rig built.
Also defer the pre-auth key prompt to the join path. rig is convergent by
contract, but re-running it to pick up this fix demanded a Tailscale key it
would never spend — friction on precisely the repair path that matters.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
