Control-plane backup plumbing belongs in rig, not the runbook (nightly Coolify dump) #8

New issue

Closed

opened 2026-07-12 19:12:05 +00:00 by dan-claude-bot · 0 comments

dan-claude-bot commented

2026-07-12 19:12:05 +00:00

(Migrated from github.com)

rig coolify install ends by printing:

next: your bootstrap runbook (admin user, API token, GitHub App, S3 destination)

Some of what it hands off genuinely is runbook work — creating an admin user and minting
an API token are console acts against a running Coolify. But one item on that list is not:
the nightly control-plane dump is on-box root plumbing, and it belongs in rig.

What's missing

The Coolify control plane's own Postgres holds the GitHub App private key, every registered
server's SSH key, and every environment variable value for every environment it manages.
Losing it un-backed-up loses the fleet's secrets. Backing it up requires, on the box:

age and an S3 client installed
a dump script at a known path
a scheduled unit that runs it nightly
somewhere for the bindings (age recipient, S3 bucket/endpoint/credentials) to live

Today all four are a manual runbook. That is the gap.

Why it's rig's job, not the runbook's

It is exactly rig's job description. On-box, as root, on a pristine Debian server —
installing packages and a systemd unit. rig already does this one door down: rig runner install creates a user, installs a versioned binary, and registers a systemd service.
A timer for the dump is the same move.

The DR story makes it a correctness bug, not tidiness. The control-plane dump is
explicitly forensics, not a restore path: a lost control plane is rebuilt fresh and
reconciled from the manifest. So there will be a next control-plane box — that is the design.
If the backup plumbing is a manual runbook step, every rebuilt control plane is born
un-backed-up, and the person rebuilding it (stressed, mid-incident) has to remember.
In rig, a control-plane box is backed up from birth.

The script is currently in the wrong repo. It ships in
cast as scripts/dump-coolify-db.sh — but cast is
the off-box tool, and cast's src/ never references it. It is payload, and its first line
is docker exec coolify-db pg_dump …, which only runs where the Coolify containers are. It's
a stowaway from before the rig/cast/infra split re-homed the code. It should live here.

The seam — rig installs machinery, never credentials

rig's philosophy (README) is "plumbing logic only — no hostnames, no bindings, no secrets,
nothing about your infrastructure... stores no credential, ever." That holds cleanly here:

rig installs	the operator supplies
`age` + `awscli`	`AGE_RECIPIENT` (a public key)
the dump script	`S3_BUCKET`, `S3_ENDPOINT`
the systemd service + timer	`AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / region
an empty, `0600` env-file template	— by filling that template, and nothing else

rig writes the template with blank values and never reads it back. The dump script's own
guards (: "${AGE_RECIPIENT:?…}") mean an unfilled file fails the unit loudly rather than
shipping plaintext or a zero-byte artifact — the right failure mode for a backup.

Proposal

New convergent subcommand, so the plumbing can be (re)installed on an existing control plane
without touching Coolify itself:

rig coolify backup install [--schedule <OnCalendar>] [--pg-container <name>]
                           [--pg-user <name>] [--pg-db <name>]

systemd timer, not cron. EnvironmentFile= is the correct idiom for 0600 secrets (no
set -a sourcing wrapper), failures are visible in systemctl status / list-timers rather
than mailed into the void, and Persistent=true catches a run missed while the box was down.
Cron gives none of that and needs a wrapper script to load the env at all.

Two things to carry over that the current cast script gets wrong or omits:

aws-cli ≥ 2.23 breaks against S3-compatible backends. Debian 13 (trixie) ships
awscli 2.23.6, and 2.23 enabled new default upload checksums that Hetzner/MinIO/Ceph
reject. The unit must default AWS_REQUEST_CHECKSUM_CALCULATION=when_required and
AWS_RESPONSE_CHECKSUM_VALIDATION=when_required.
Refuse to upload an empty artifact. A failed pg_dump piped into age still produces a
valid, tiny, encrypted file. Without a size check that uploads cleanly every night and looks
exactly like a working backup.

Out of scope

rig cannot verify the upload actually works — that needs real credentials. The command prints
the manual verification (fill the env file → systemctl start coolify-dump.service → confirm
the object is in the bucket and really is age ciphertext). Reading a backup back is the
operator's gate, and should stay one.

`rig coolify install` ends by printing: ``` next: your bootstrap runbook (admin user, API token, GitHub App, S3 destination) ``` Some of what it hands off genuinely *is* runbook work — creating an admin user and minting an API token are console acts against a running Coolify. But one item on that list is not: **the nightly control-plane dump is on-box root plumbing, and it belongs in rig.** ## What's missing The Coolify control plane's own Postgres holds the GitHub App private key, every registered server's SSH key, and every environment variable value for every environment it manages. Losing it un-backed-up loses the fleet's secrets. Backing it up requires, on the box: - `age` and an S3 client installed - a dump script at a known path - a scheduled unit that runs it nightly - somewhere for the bindings (age recipient, S3 bucket/endpoint/credentials) to live Today all four are a manual runbook. That is the gap. ## Why it's rig's job, not the runbook's **It is exactly rig's job description.** On-box, as root, on a pristine Debian server — installing packages and a systemd unit. rig already does this one door down: `rig runner install` creates a user, installs a versioned binary, and registers a systemd service. A timer for the dump is the same move. **The DR story makes it a correctness bug, not tidiness.** The control-plane dump is explicitly *forensics, not a restore path*: a lost control plane is **rebuilt fresh** and reconciled from the manifest. So there will be a next control-plane box — that is the design. If the backup plumbing is a manual runbook step, every rebuilt control plane is born **un-backed-up**, and the person rebuilding it (stressed, mid-incident) has to remember. In rig, a control-plane box is backed up from birth. **The script is currently in the wrong repo.** It ships in [`cast`](https://github.com/heavy-duty/cast) as `scripts/dump-coolify-db.sh` — but `cast` is the *off-box* tool, and `cast`'s `src/` never references it. It is payload, and its first line is `docker exec coolify-db pg_dump …`, which only runs where the Coolify containers are. It's a stowaway from before the rig/cast/infra split re-homed the code. It should live here. ## The seam — rig installs machinery, never credentials rig's philosophy (README) is *"plumbing logic only — no hostnames, no bindings, no secrets, nothing about your infrastructure... stores no credential, ever."* That holds cleanly here: | rig installs | the operator supplies | | --- | --- | | `age` + `awscli` | `AGE_RECIPIENT` (a **public** key) | | the dump script | `S3_BUCKET`, `S3_ENDPOINT` | | the systemd service + timer | `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / region | | **an empty, `0600` env-file template** | — by filling that template, and nothing else | rig writes the template with **blank values** and never reads it back. The dump script's own guards (`: "${AGE_RECIPIENT:?…}"`) mean an unfilled file fails the unit **loudly** rather than shipping plaintext or a zero-byte artifact — the right failure mode for a backup. ## Proposal New convergent subcommand, so the plumbing can be (re)installed on an existing control plane without touching Coolify itself: ```sh rig coolify backup install [--schedule <OnCalendar>] [--pg-container <name>] [--pg-user <name>] [--pg-db <name>] ``` **systemd timer, not cron.** `EnvironmentFile=` is the correct idiom for `0600` secrets (no `set -a` sourcing wrapper), failures are visible in `systemctl status` / `list-timers` rather than mailed into the void, and `Persistent=true` catches a run missed while the box was down. Cron gives none of that and needs a wrapper script to load the env at all. Two things to carry over that the current `cast` script gets wrong or omits: - **`aws-cli ≥ 2.23` breaks against S3-compatible backends.** Debian 13 (trixie) ships `awscli 2.23.6`, and 2.23 enabled new default upload checksums that Hetzner/MinIO/Ceph reject. The unit must default `AWS_REQUEST_CHECKSUM_CALCULATION=when_required` and `AWS_RESPONSE_CHECKSUM_VALIDATION=when_required`. - **Refuse to upload an empty artifact.** A failed `pg_dump` piped into `age` still produces a valid, tiny, encrypted file. Without a size check that uploads cleanly every night and looks exactly like a working backup. ## Out of scope rig cannot verify the upload actually works — that needs real credentials. The command prints the manual verification (fill the env file → `systemctl start coolify-dump.service` → confirm the object is in the bucket and really is `age` ciphertext). Reading a backup back is the operator's gate, and should stay one.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/rig#8

No description provided.