rig used to pass --ts-tag to `tailscale up --advertise-tags`, stating the
tailnet tag a second time with no way to know whether its request and the
key's own tags agreed. It asserted the tag it REQUESTED, never the tag control
GRANTED — the sshd first-wins bug in a different hat, and the same scar (both
M900s joined tag:server, retagged by hand, unnoticed).
Collapse the two sources of truth onto one: the key.
- `tailscale up` drops --advertise-tags; the key's tags apply.
- After join, poll `tailscale status --json` for `.Self.Tags` (netmap ground
truth, not `debug prefs`) until tags appear or BackendState=Running, on BOTH
the fresh-join and already-joined paths.
- UNTAGGED -> hard refusal: `tailscale logout` to back the user-owned node out,
then die naming the fix (mint a tagged key).
- Role policy moves onto the effective tag: a runner must not have tag:server
among the tags the key actually granted. Strictly stronger than before.
- --ts-tag is removed, and dies exit 2 with a message pointing at the key
(consuming its value), not an "unknown flag".
- New array-aware reader json_string_array in lib/runner-config.sh (jq-free,
never fails under set -e), with its own unit tests; bootstrap sources the lib.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `rig db dump <container> [outfile]` and
`rig db restore <artifact> <container> [db] [--yes]` — imperative on-box
PostgreSQL tooling, the interactive counterpart to the scheduled,
declarative `coolify backup install`.
Key decisions:
- Dumps carry `--clean --if-exists --no-owner --no-acl`. `--no-owner
--no-acl` is mandatory for cross-instance restores: the target's
superuser differs (Coolify randomizes it), so a plain dump aborts under
ON_ERROR_STOP=1 on the first GRANT/ALTER OWNER for a missing role.
- $POSTGRES_USER/$POSTGRES_DB are read INSIDE the container (single-quoted
`sh -c`), never hardcoded to `postgres` on the host.
- restore connects as the container's own superuser and runs with
ON_ERROR_STOP=1; the optional [db] arg targets a NAMED database in a
shared container, passed in via a container env var rather than string
splicing.
- restore overwrites the target, so it prompts y/N; --yes/--force is the
automation bypass. Artifact existence/non-emptiness is checked before
the confirm gate and before anything touches the DB.
- dump uses pipefail + a sibling temp promoted only on success, and
refuses to keep an empty artifact — a failed pg_dump must never leave a
plausible-looking .gz behind.
Args are validated before the root check (testable without root); guards
are root, Debian-family warn, docker, and gzip/gunzip. Adds CLI tests and
a `### rig db` README section.
Closes#15
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The hand-listed inputs under-covered silently: commands/*.sh does not glob
into lib/, and the previous fix patched that by adding one more path — which
leaves the same hole open for the next subdirectory. Verified: with a broken
script at commands/deep/nested/bad.sh, the hand-listed invocation still exits 0.
`shopt -s globstar` + bin/* **/*.sh covers every script in the repo, including
the extensionless bin/ entrypoints. The list is printed before it runs, so
coverage is visible in the CI log rather than assumed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The runner-* commands source commands/lib/runner-config.sh. Without -x,
shellcheck will not follow that source even though the files carry a
`source=SCRIPTDIR/...` directive, so it reported SC1091 on all three —
and shellcheck exits non-zero on any finding, info-level included.
commands/*.sh does not glob into lib/, so runner-config.sh was being
linted by nothing at all. List it explicitly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`rig runner install --repo <B>` on a box already registered to repo A
treated the mere existence of .runner as "already registered", skipped
configure, restarted the service still pointed at A, and reported success.
--repo was accepted, validated, and then ignored — leaving B with zero
runners and its `runs-on` jobs queued against one that will never come.
This is the natural next command after a partial `repoint`, and the failure
is worse than a no-op: moving a runner between repos is a trust-boundary
act, so quietly putting it back on the old one defeats the point of the move.
Gate install on the repo .runner actually names. Convergence — the property
worth keeping — is untouched: re-running against the repo the box is already
on still skips registration, never prompts for a token, and exits 0.
Skipping when the repo *differs* was never convergence, only a silently
ignored argument, so it now fails and names both repos, pointing at
`runner repoint` (move) or `runner remove` (start over). An unreadable
.runner is refused too — it is no licence to assume a match.
The .runner reader that `status` and `repoint` each carried is lifted into
commands/lib/runner-config.sh, which now also holds the guard. Its json_field
no longer dies bare under `set -o pipefail` when a key is missing, which is
what `status`'s own ${REPO_URL:-unknown} fallback always assumed.
Tests: the guard is exercised against a fixture .runner (refuses another repo
naming both, points at repoint, no-ops on the same repo, passes an
unregistered box, refuses an unreadable one) plus an ordering assertion that
it precedes svc.sh start — reaching it through the CLI would need root and a
really-registered runner, which the dependency-free harness cannot fabricate.
All three mutants (guard deleted, guard comparing nothing, guard moved below
the service start) go red.
Closes#13
runner install is convergent by skipping: it sees a registered runner and
leaves it alone. So rig could create a runner and never move or destroy one,
and re-pointing a box at a different repo meant hand-rolled config.sh/svc.sh
incantations against an install layout only rig knew about.
- status: repo, name, labels, dir, unit — read-only, no token, no network.
- remove: service down, then deregister. --local wipes the box without
contacting GitHub, leaving a stale entry to delete by hand.
- repoint: remove + re-register in one act, keeping the runner's name and
reusing the binary already on the box.
The service always comes down before deregistration in both paths: GitHub's
removal throws "Uninstall service first" while the service is configured, and
--local bypasses that check entirely, which would strand a running service
pointed at deleted config.
repoint collects both tokens up front — a token you turn out not to have must
fail while the runner is still registered, not halfway through the move.
Labels are the sharp edge: GitHub holds them, the runner does not persist
them, and they are what runs-on matches. install now records what it
registered with so repoint and status can read it back; a runner installed
before that has nothing to read, so repoint falls back to the ci-runner
default and warns before it touches anything.
Both found by the first run on a real control-plane box — neither was
reachable by the argument-parsing tests.
$EDITOR is unset on a freshly-bootstrapped server, which is precisely rig's
target environment. The printed next-step `$EDITOR /etc/coolify-dump.env`
expanded to nothing, so bash tried to EXECUTE the 0600 bindings file and said
"Permission denied" — an error that reads like a filesystem problem and is not
one. Print `nano`.
A bare bucket name in S3_BUCKET reads to `aws` as a LOCAL path, so the upload
died with "Invalid argument type" and a usage dump — after pg_dump had run and
age had encrypted 14MB, with nothing in the error pointing at the actual
mistake. The script now validates the bindings up front: S3_BUCKET must be an
s3:// URI, S3_ENDPOINT must carry a scheme. Both fail with the value quoted and
the reason stated, before a database is read.
Note what still cannot be validated, and now says so in the script: age's X25519
header does not reveal its recipient, so a valid-but-WRONG key (staging's
instead of prod's) yields a flawless backup nobody can open. Only decrypting an
artifact proves the recipient. The printed next-steps now walk through that
read-back explicitly, from a machine holding the private key — never the box.
The dump script ships as an embedded heredoc, so a typo in it would first
surface at 04:00 on a live control plane. test/cli.sh now extracts it and
asserts it is valid bash and that both new guards fire.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Coolify control-plane database holds the GitHub App private key, every
registered server's SSH key, and every environment value for every environment
it manages. Backing it up was a manual runbook step, and the dump script lived
in cast — the off-box tool, whose src never references it. It runs on the box,
as root, under a scheduler: that is rig's job description.
It matters beyond tidiness. The dump is forensics, not a restore path — a lost
control plane is rebuilt fresh and reconciled from the manifest. So there will
be a next control-plane box, and as a runbook step it was born un-backed-up,
depending on someone remembering mid-incident. Now it is backed up from birth.
rig installs the machinery and templates /etc/coolify-dump.env empty at 0600,
never reading it back — no credential passes through rig. The script's own
guards make an unfilled file fail the unit loudly rather than ship plaintext.
systemd timer over cron: EnvironmentFile is the right idiom for 0600 secrets,
failures surface in systemctl status instead of being mailed into the void, and
Persistent=true catches a run missed while the box was down.
Two hazards the cast script missed, carried into the unit:
- aws-cli >= 2.23 enables default upload checksums that S3-compatible backends
reject; Debian 13 ships 2.23.6, so the unit defaults both checksum knobs to
when_required.
- A failed pg_dump piped into age still yields a valid, tiny, encrypted file
that uploads cleanly every night and looks exactly like a working backup. The
script now refuses to upload an empty artifact.
Closes#8
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`bootstrap` skips `tailscale up` when the box has already joined — which is what
lets a re-run work without spending a pre-auth key. But skipping it also skipped
`--hostname`, so the TAILNET name never converged: a box that joined under the
wrong name (e.g. `--hostname` omitted, defaulting to the ROLE) stayed misnamed
permanently, and re-running rig — the documented repair path — could not fix it.
rig is convergent by contract; this was the one field that wasn't.
The already-joined path now compares the current tailnet hostname against
`--hostname` and reconciles it with `tailscale set --hostname` (no re-auth, no
key). A failure warns rather than dying: a cosmetic name is not worth failing a
bootstrap over.
Safe by construction: Tailscale ACLs cannot bind a rule's dst to a hostname (it
must be a tag, an IP, or a `hosts` alias — which is precisely why acl.hujson pins
coolify-box to an IP), so a rename cannot silently void a grant. It also will not
clobber a deliberate rename — a machine renamed in the admin console keeps that
name and is no longer overridden by the device hostname.
Found in the same session as the sshd first-wins bug (#6): an operator ran
`bootstrap control-plane` on the prod box without `--hostname`, and no re-run
could undo the resulting name.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit restarted ssh and only checked `sshd -T` afterwards. On a
box whose only door is SSH, restarting against a config sshd refuses to parse
leaves no listener and no way back in — the same shape as the firewall-before-
bootstrap lockout this session already found in the migration runbook: commit
to the irreversible act, then verify.
Now `sshd -t` parses the MERGED config (our drop-in, cloud-init's, and any
third-party file) before the restart; on failure the drop-in is rolled back and
the daemon is left untouched. Verified: a bad neighbour drop-in exits 255 and
never reaches `systemctl restart`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
rig wrote its hardening drop-in as 99-rig.conf. Cloud images ship
50-cloud-init.conf carrying `PasswordAuthentication yes`, and sshd_config is
FIRST-wins ("for each keyword, the first obtained value will be used" —
sshd_config(5)) with Include expanding its glob in lexical order. So 50-
was read before 99- and every keyword rig set was silently discarded.
Every Hetzner box rig has bootstrapped was still serving
`passwordauthentication yes` — confirmed today on coolify-box (CX23) and
ci-runner (CX43) by `sshd -T`, and from off-tailnet by
`ssh -o PreferredAuthentications=none`. Root logins were never exposed
(PermitRootLogin resolved to prohibit-password via Debian's stock config),
but any password-bearing non-root account was reachable on a public port 22.
Three fixes:
1. Name the drop-in 00-rig.conf so it is read first and actually wins; sweep
a stale 99-rig.conf on re-run so existing boxes converge.
2. Assert the EFFECTIVE config with `sshd -T` and refuse to claim a hardened
box otherwise. Asserting that the file existed is what let this ship green
— the Incus rehearsal has no cloud-init drop-in to lose to, so the bug was
invisible to the one gate that could have caught it.
3. Set the system hostname, not just the tailnet one. A box reached as
`coolify-box` greeted the operator as `root@internal-tooling`; the shell
prompt is the only "am I on the right box" signal before a destructive
command, and it was lying on every box rig built.
Also defer the pre-auth key prompt to the join path. rig is convergent by
contract, but re-running it to pick up this fix demanded a Tailscale key it
would never spend — friction on precisely the repair path that matters.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On Debian, /etc/os-release defines VERSION="13 (trixie)". runner-install
sourced it into the main shell for the Debian-family guard, overwriting the
script's empty $VERSION: the latest-release resolution was skipped and the
download URL became .../v13 (trixie)/... -> curl (3) malformed URL. A
--version pin was clobbered the same way (guards run after arg parsing).
Read ID/ID_LIKE via a subshell in both runner-install and bootstrap (same
pattern, no collision there yet), and add a harness guard that fails on any
future main-shell sourcing of os-release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the Incus rehearsal (prod-migration Task 4): cloud images ship
openssh-server, container/VM images do not; the hardening drop-in and
'systemctl restart ssh' both presume it. A rig box is SSH-managed by
definition, so the dependency is explicit now. No-op on cloud images.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
