diff --git a/README.md b/README.md index 4fb46ae..c9b7987 100644 --- a/README.md +++ b/README.md @@ -68,6 +68,7 @@ cast apply --env --all # no repo: EVERY registered proj cast diff / --env [--full] cast diff --env --all [--full] # no repo: EVERY registered project cast capture / --env [--generated ] [--override ] +cast capture / --env --generated-only [--from =] cast inventory / --env cast inventory --env [--emit-draft [--recipient age1…] [--no-secrets]] cast server add --ip --key --env [--user root] [--port 22] @@ -104,7 +105,10 @@ cast team [--env ] point-in-time blueprint of a box. See *Drafting a box that was never declared*. - **`capture`** — the adoption path: reads a hand-built instance's live env and writes the environment's age store from it. See *Adopting a hand-built - instance* below. + instance* below. With **`--generated-only`** it is instead **pass 2 of a + bootstrap**: run *after* `apply`, it fills the store's provider-generated names + (a Coolify-made `DATABASE_URL`) with the values Coolify generated. See *The + bootstrap is two-pass* below. - **`server add`** — uploads a server's private key and registers it with Coolify. - **`smoke`** — contract test against the project's `smoke_target`: proves Coolify's bulk env endpoint still *upserts* rather than replacing. Run it after @@ -327,6 +331,69 @@ The store is encrypted to the environment's `age_recipient` (add it to `age` on stdin: it is never a temp file, never on stdout, never in your shell history. An existing store is not overwritten without `--force`. +## The bootstrap is two-pass: `capture --generated-only` + +Those placeheld names are the reason bootstrapping an environment **cannot be one +pass**. The value does not exist until Coolify makes it: + +```sh +cast capture heavy-duty/incubator --env prod # 1. the store learns every + # name; generated ones are + # placeheld — nothing has + # created them yet +cast apply heavy-duty/incubator --env prod # 2. Coolify creates the + # database, and generates + # the real URL +cast capture heavy-duty/incubator --env prod \ # 3. the store learns THAT + --generated-only --from DATABASE_URL=incubator-db # value +``` + +Until step 3 runs, the store says `pending-coolify-generated` while the live value +is real — the exact state in which the next routine `apply` overwrites a working +secret. It is also on the DR path: *rebuild the control plane from state* means +apply-from-nothing, so every generated secret in every store is a placeholder +again. This used to be a hand `age` re-encrypt against production, with the prod +key in a process substitution. + +`--generated-only` **inverts** capture's rule and changes nothing else: it fills +the `generated_secrets` names and leaves every other name in the store **exactly as +it is, byte for byte** — never re-read from the box, so a secret you rotated by +hand last month survives it. Same typed confirmation, same names-never-values plan. +The store must already exist: pass 2 *fills* names, it does not create them. + +It reads each value from the **database that owns it** (`internal_db_url`) — +never from the consuming app's env, where a generated URL never appears (the app's +env holds what the template resolved to, which at this point is *the placeholder +itself*). And it resolves that database **inside your project and environment +only**, never from the instance-wide `GET /databases` list, which holds every +database on the box — other projects', and umami's own bundled Postgres. + +**It will not guess which database a name comes from.** Nothing carries that edge: +`generated_secrets:` is a flat list of names, and the template knows only +`DATABASE_URL=${DATABASE_URL}`. So cast infers it only when it *cannot* be wrong +(one name, one database) and otherwise refuses, printing the flag you need. A pick +made from the *name* (`REDIS_URL` → the redis one) is wrong **silently**, and what +it writes is a well-formed URL to somebody else's database. + +Four refusals, each a thing that used to be a step in a runbook: + +| | | +| --- | --- | +| **UNMAPPED** | more than one database could be meant → say which, with `--from` | +| **OCCUPIED** | the name already holds a *real* value → filling it silently rotates a live credential. `--force` to mean it | +| **ABSENT** | the name is not in the store at all → pass 1 has not run, or you are pointed at the wrong store | +| **PENDING** | a placeholder in a name nothing here fills → the store would still be a lie | + +Afterwards it **asserts the postcondition**, against the ciphertext now on disk: zero +`pending-coolify-generated` remain, and the name count is unchanged. A store that +lost a name re-encrypts perfectly and reads back perfectly — you would find out at +the next `apply`, in an environment whose plaintext nobody has any more. + +`apply` deliberately does not do this for you after a create. It would close the +window entirely, but it would make the verb that mutates Coolify also mutate the +encrypted store — and hence the git repo — which is a much bigger blast radius for +a verb people run on a schedule. + **[docs/semantics.md](docs/semantics.md)** is the contract behind those commands: what `apply` guarantees (never deletes, never recreates a database, fails loudly rather than recreating on un-updatable drift), the `dockercompose` diff --git a/docs/semantics.md b/docs/semantics.md index 58cb803..390d184 100644 --- a/docs/semantics.md +++ b/docs/semantics.md @@ -471,6 +471,79 @@ the plan. There is no `--yes`: a store written without someone reading the provenance column is the outcome the verb exists to prevent. A closed stdin aborts rather than hanging. +### Pass 2 (`capture --generated-only`) + +An environment that declares `generated_secrets:` **bootstraps in two passes, by +construction** — the value does not exist until Coolify makes it: + + capture → apply → capture --generated-only + (placeholds) (Coolify generates) (the store learns the real value) + +Without pass 2 the store's value for `DATABASE_URL` stays a placeholder while the +live value is real — which is exactly the state in which the next routine `apply` +overwrites a working secret (#47). **The absence of pass 2 is what leaves that gun +loaded**, and it is on the DR path: *rebuild the control plane from state* means +apply-from-nothing, which means every generated secret in every store is a +placeholder again. This used to be a hand `age` re-encrypt against production — +decrypt a fourteen-name store, edit two lines, re-encrypt to the environment's +recipient, holding the prod key, with a `jq` filter that must not pick the wrong row. + +`--generated-only` **inverts** capture's disposition rule and changes nothing else: +the names in `generated_secrets` are the ones it *fills*, and every other name is +left **exactly as the store has it, byte for byte** — never re-read from the box, +which is what makes it safe to run against an environment whose other secrets have +since been rotated by hand. Same verb, same store-writing code path, same typed +confirmation. + +| | | +| --- | --- | +| **fill** | a generated name holding the placeholder → the value from the database that owns it | +| **keep** | every other name → carried over from the store, untouched | +| **UNMAPPED** | cast cannot attribute the name to exactly one database → **refuses** | +| **OCCUPIED** | a generated name already holding a real value → **refuses** (without `--force`) | +| **ABSENT** | a generated name the store does not carry at all → **refuses** | +| **PENDING** | a placeholder in a name nothing here fills → **refuses** | + +**The value is read from the resource that OWNS it.** A generated URL never appears +on the consuming application's env — the app's env holds whatever the template +resolved to, which at this point in the bootstrap is *the placeholder itself*. +Reading the app back would faithfully capture the lie pass 2 exists to correct. It +lives on the **database**, as `internal_db_url`. + +**Resolved inside the project + environment, never instance-wide.** cast reads +`GET /projects/{uuid}/{env}`, whose `postgresqls` / `redis` relations are *this* +environment's and nothing else's. It never calls `GET /databases`, which lists +every database on the box — other projects', and umami's own bundled Postgres — +where picking ours out means matching by name across a list in which a collision +is both possible and silent (#29 in another hat, and the reason the hand-run `jq` +carried a comment about not taking the third row). The scoping is **structural** +rather than a filter cast has to remember to get right. + +**cast will not guess which database a name comes from.** Nothing in the system +carries that edge: `generated_secrets:` is a flat list of *names*, the env template +knows only `DATABASE_URL=${DATABASE_URL}`, and the box does not say. So the +inference is made **only when it cannot be wrong** — one generated name, one +database, no other candidate — and otherwise the run refuses and hands back the +flag: `--from DATABASE_URL=incubator-db`. Reading the type out of the *name* +(`REDIS_URL` → the redis one) is precisely the bug this must not have: a +name-directed pick is wrong **silently**, and what it writes is a perfectly +well-formed URL to somebody else's database. + +**And it asserts the postcondition it exists for** — against the ciphertext now on +disk, decrypted back, not trusted from memory. Zero `pending-coolify-generated` +remain, and the name count is unchanged. Both claims, because they fail in opposite +directions: a store that still holds a placeholder is still a lie, and a store that +*lost* a name on the way through re-encrypts perfectly, reads back perfectly, and +surfaces at the next `apply` as a missing secret in an environment whose plaintext +nobody has any more. That assertion used to be a line in a human runbook — which is +to say, a step that could be skipped. + +`apply` deliberately does **not** do this automatically after a create. It would +close the window entirely, but it would make the verb that mutates Coolify also +mutate the encrypted store and hence the git repo — a much bigger blast radius for +a verb people run on a schedule. A separate, explicit, operator-run verb is the +right first step. + ## Drafts (`inventory --emit-draft`) `inventory` with no repo sweeps an instance. `--emit-draft ` writes that diff --git a/src/capture.ts b/src/capture.ts index 5359fca..2997b3c 100644 --- a/src/capture.ts +++ b/src/capture.ts @@ -239,3 +239,349 @@ export function renderCapturePlan( } return lines.join("\n"); } + +// --------------------------------------------------------------------------- +// Pass 2 — `capture --generated-only` +// +// Bootstrapping an environment that declares generated_secrets is two-pass BY +// CONSTRUCTION, and the two passes are not variations of one verb: +// +// pass 1 `capture` the store learns every name; the generated +// ones are placeheld, because their values do +// not exist yet — nothing has created them. +// (apply) Coolify creates the database and generates +// the real URL. +// pass 2 `capture --generated-only` the store learns THAT value. +// +// Between the two, the store's value for a generated name is a placeholder while +// the live value is real. Everything below exists to close that window without +// the operator hand-editing decrypted plaintext (which is how it was closed +// before: decrypt, edit two lines, re-encrypt, against prod, holding the key). +// +// The flag INVERTS classify()'s disposition rule and nothing else: the names in +// generated_secrets are the ones it fills, and every other name is left exactly +// as the store has it — byte for byte, never re-read from the box. A generated +// name is the only kind of name whose true value cast can find AFTER the fact, +// because it is the only kind the provider owns. +// --------------------------------------------------------------------------- + +// A live database inside the project + environment being filled — the resource +// that OWNS a generated URL. +// +// `url` is `internal_db_url`, and it is read from the DATABASE, never from a +// consuming application's env. It never appears there: the app's env holds +// whatever the template resolved to, which at this point in the bootstrap is +// the placeholder itself. Reading the app back would faithfully capture the +// lie we are here to correct. +export type GeneratedSource = { + resource: string; + type: string; + url: string; +}; + +// One generated name, filled from the resource that owns it. +export type Fill = { + ref: string; + from: { resource: string; type: string }; + // Never rendered. Kept here so the caller can encrypt it, and nowhere else — + // same contract as Disposition.value. + value: string; +}; + +export type GeneratedPlan = { + fills: Fill[]; + // Names the store keeps, byte for byte. Printed as names so the operator can + // see the store is not being rewritten around them. + kept: string[]; + // A generated name cast cannot attribute to exactly one database. Refuses: + // picking would be silent, and picking WRONG writes another box's credentials + // into this environment's store. + unmapped: Array<{ ref: string; why: string }>; + // A generated name whose store value is NOT the placeholder. Refuses without + // --force: it is already filled (someone ran pass 2, or set it by hand), and + // overwriting it is a silent rotation of a live credential. + occupied: string[]; + // A generated name the store does not carry AT ALL. Refuses: pass 2 fills + // names, it does not invent them. A store missing one did not come from pass + // 1, and the name-count postcondition below could not hold anyway. + absent: string[]; + // A store value that is still the placeholder and is NOT in the fill set — + // so this run would leave the store still lying. Refuses. This is the + // postcondition, checked BEFORE the write so it reads as a plan and not as an + // assertion failure over ciphertext already on disk. + stillPending: string[]; +}; + +// Which database does a generated name come from? +// +// The manifest cannot say. `generated_secrets:` is a flat list of NAMES, and +// there is no edge anywhere in the data model from DATABASE_URL to the database +// that owns it — not in the manifest, not in the env template (which knows only +// `DATABASE_URL=${DATABASE_URL}`), and not on the box. +// +// So cast does not guess. It infers ONLY when the inference cannot be wrong — +// one generated name, one database, no other candidate — and otherwise refuses +// and makes the operator state the edge with --from. The temptation is to read +// the type out of the NAME (REDIS_URL → the redis one), and that is precisely +// the bug this verb must not have: a name-directed pick across a list of +// databases is #29 wearing a different hat, and it is wrong SILENTLY. The +// value it would write is a perfectly well-formed URL to somebody else's +// database. +export function resolveGeneratedSources( + generated: string[], + databases: GeneratedSource[], + from: Record, +): { + mapping: Record; + unmapped: Array<{ ref: string; why: string }>; +} { + const mapping: Record = {}; + const unmapped: Array<{ ref: string; why: string }> = []; + const byName = new Map(databases.map((d) => [d.resource, d])); + for (const ref of generated) { + const named = from[ref]; + if (named !== undefined) { + const hit = byName.get(named); + if (!hit) { + unmapped.push({ + ref, + why: `--from ${ref}=${named}, but no database named "${named}" exists in this project+environment`, + }); + continue; + } + mapping[ref] = hit; + continue; + } + if (databases.length === 0) { + unmapped.push({ + ref, + why: "no database exists in this project+environment to fill it from", + }); + continue; + } + // The only inference that cannot be wrong: nothing else it could be. + if (generated.length === 1 && databases.length === 1) { + mapping[ref] = databases[0]; + continue; + } + unmapped.push({ + ref, + why: `${databases.length} databases here (${databases.map((d) => `${d.resource}:${d.type}`).join(", ")}) — cast will not pick by name`, + }); + } + return { mapping, unmapped }; +} + +// Invert the disposition rule. `store` is the DECRYPTED current store; every +// name in it that is not being filled comes out the other side untouched. +export function planGenerated( + generated: string[], + store: Record, + sources: Record, + unmapped: Array<{ ref: string; why: string }>, + opts: { force: boolean } = { force: false }, +): GeneratedPlan { + const generatedSet = new Set(generated); + const fills: Fill[] = []; + const occupied: string[] = []; + const absent: string[] = []; + for (const ref of generated) { + if (!(ref in store)) { + absent.push(ref); + continue; + } + const source = sources[ref]; + // Already reported by resolveGeneratedSources; not also an "occupied". + if (!source) continue; + if (store[ref] !== GENERATED_PLACEHOLDER && !opts.force) { + occupied.push(ref); + continue; + } + fills.push({ + ref, + from: { resource: source.resource, type: source.type }, + value: source.url, + }); + } + const filling = new Set(fills.map((f) => f.ref)); + // A placeholder left standing in a name nobody is filling. The run would + // "succeed" and the store would still be a lie — so it refuses instead, and + // names the flag that fixes it. Generated names that failed to map or were + // refused as occupied are NOT reported here: they already have their own row, + // and one problem should be named once. + const stillPending = Object.keys(store) + .filter( + (k) => + store[k] === GENERATED_PLACEHOLDER && + !filling.has(k) && + !generatedSet.has(k), + ) + .sort(); + // Everything the store keeps byte for byte. A name carrying a refusal above + // is not "kept" — it is the reason the run stops, and printing it in both + // rows would read as though cast had a plan for it. + const flagged = new Set([ + ...unmapped.map((u) => u.ref), + ...occupied, + ...stillPending, + ]); + const kept = Object.keys(store) + .filter((k) => !filling.has(k) && !flagged.has(k)) + .sort(); + return { fills, kept, unmapped, occupied, absent, stillPending }; +} + +export function generatedPlanRefuses(p: GeneratedPlan): boolean { + return ( + p.unmapped.length > 0 || + p.occupied.length > 0 || + p.absent.length > 0 || + p.stillPending.length > 0 + ); +} + +// Names, provenance and the resource a value came FROM. Never values. +// +// capture.ts's rule holds unchanged: the only value-shaped thing printed is +// GENERATED_PLACEHOLDER, a literal constant in this file — and here it is +// printed as the thing being REPLACED, which is the one fact about the store's +// current contents an operator needs in order to believe the plan. +export function renderGeneratedPlan( + p: GeneratedPlan, + ctx: { + orgRepo: string; + env: string; + instance: string; + store: string; + recipient: string; + project: string; + environment: string; + }, +): string { + const lines = [ + `capture --generated-only — ${ctx.orgRepo} ${ctx.env}`, + "", + ` source: instance ${ctx.instance}, project "${ctx.project}", environment "${ctx.environment}"`, + ` store: ${ctx.store}`, + ` recipient: ${ctx.recipient}`, + "", + ]; + const width = Math.max( + 0, + ...[ + ...p.fills.map((f) => f.ref), + ...p.kept, + ...p.occupied, + ...p.absent, + ...p.unmapped.map((u) => u.ref), + ].map((r) => r.length), + ); + for (const f of p.fills) { + lines.push( + ` ${f.ref.padEnd(width)} fill ${GENERATED_PLACEHOLDER} → ${f.from.resource} (${f.from.type}) internal_db_url`, + ); + } + for (const k of p.kept) { + lines.push(` ${k.padEnd(width)} keep unchanged in the store`); + } + for (const u of p.unmapped) { + lines.push(` ${u.ref.padEnd(width)} UNMAPPED ${u.why}`); + } + for (const o of p.occupied) { + lines.push( + ` ${o.padEnd(width)} OCCUPIED already holds a value that is not ${GENERATED_PLACEHOLDER}`, + ); + } + for (const a of p.absent) { + lines.push(` ${a.padEnd(width)} ABSENT not in the store at all`); + } + for (const s of p.stillPending) { + lines.push( + ` ${s.padEnd(width)} PENDING still ${GENERATED_PLACEHOLDER}, and nothing here fills it`, + ); + } + lines.push( + "", + `${p.fills.length} name(s) to fill, ${p.kept.length} left exactly as the store has them`, + ); + if (p.unmapped.length > 0) { + lines.push( + "", + `refusing to write the store: ${p.unmapped.length} generated name(s) cannot be attributed`, + "to exactly one database in this project+environment. Nothing in the manifest, the env", + "template or the box says which database a given name comes from, and cast will not", + "pick by name — a wrong pick writes another database's credentials into this store,", + "and it does it silently. State the edge:", + "", + ...p.unmapped.map((u) => ` --from ${u.ref}=`), + ); + } + if (p.occupied.length > 0) { + lines.push( + "", + `refusing to write the store: ${p.occupied.length} generated name(s) already hold a real`, + "value. Filling them would rotate a live credential — silently, and against whatever is", + "already running on that value. If Coolify's resource really was recreated and the store", + "is stale, say so with --force.", + ); + } + if (p.absent.length > 0) { + lines.push( + "", + `refusing to write the store: ${p.absent.length} generated name(s) are not in the store at`, + "all. --generated-only FILLS names, it does not add them: the store it fills is the one", + "pass 1 wrote, and the name set must come out of this run exactly as it went in. Run", + "`cast capture` first, or check you are pointed at the right store.", + ); + } + if (p.stillPending.length > 0) { + lines.push( + "", + `refusing to write the store: ${p.stillPending.length} name(s) would still hold the`, + `${GENERATED_PLACEHOLDER} literal after this run — the store would still be a lie, and`, + "the next apply would push that literal at a live app. If these are generated too,", + "declare them (manifest `generated_secrets:`, or --generated ).", + ); + } + return lines.join("\n"); +} + +// The postcondition this verb EXISTS for, asserted against the store that was +// actually written — decrypted back off disk, not against the map cast held in +// memory a moment ago. In the hand-run procedure this was a line in a runbook +// ("assert 14 names / zero placeholders"), which is to say it was a step that +// could be, and eventually would be, skipped. +// +// Two claims, and they fail in opposite directions: +// - zero placeholders remain → the store no longer lies about any name +// - the name count is unchanged → and it did not lose one on the way +// +// A store that lost a name re-encrypts perfectly and reads back perfectly; the +// failure surfaces at the next apply, as a missing secret, in an environment +// whose plaintext nobody has any more. +export function assertGeneratedComplete( + before: Record, + after: Record, +): string[] { + const violations: string[] = []; + const pending = Object.keys(after) + .filter((k) => after[k] === GENERATED_PLACEHOLDER) + .sort(); + if (pending.length > 0) { + violations.push( + `${pending.length} name(s) still hold the ${GENERATED_PLACEHOLDER} literal: ${pending.join(", ")}`, + ); + } + const beforeNames = Object.keys(before).sort(); + const afterNames = Object.keys(after).sort(); + if (beforeNames.length !== afterNames.length) { + violations.push( + `the store went in with ${beforeNames.length} name(s) and came out with ${afterNames.length}`, + ); + } + const lost = beforeNames.filter((n) => !(n in after)); + const gained = afterNames.filter((n) => !(n in before)); + if (lost.length > 0) violations.push(`names LOST: ${lost.join(", ")}`); + if (gained.length > 0) violations.push(`names ADDED: ${gained.join(", ")}`); + return violations; +} diff --git a/src/cli.ts b/src/cli.ts index 04505d7..6948093 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -14,11 +14,17 @@ import { smokeTargetFor, } from "./bindings.js"; import { + type GeneratedSource, type LiveEnvs, absentResources, + assertGeneratedComplete, classify, + generatedPlanRefuses, + planGenerated, renderAbsentResources, renderCapturePlan, + renderGeneratedPlan, + resolveGeneratedSources, } from "./capture.js"; import { type CoolifyInstance, @@ -88,6 +94,7 @@ const USAGE = `usage: cast apply / --env [--path ] [-- cast diff / --env [--full] [--project ] [--environment ] cast diff --env --all [--full] # no repo: EVERY registered project cast capture / --env [--path ] [--project ] [--environment ] [--generated ] [--override ] [--force] + cast capture / --env --generated-only [--from =] [--force] # pass 2, AFTER apply cast inventory / --env [--path ] [--project ] [--environment ] [--resource =] cast inventory --env [--instance ] # no repo: SWEEP the whole instance cast inventory --env --emit-draft [--recipient age1…] [--no-secrets] @@ -152,6 +159,24 @@ capture (adopt a hand-built instance into the age secret store): command line — argv is visible in \`ps\`. Repeatable. --force overwrite an existing store (refused by default). +capture --generated-only (PASS 2 — run it AFTER \`apply\` has created the resources): + a manifest with \`generated_secrets:\` bootstraps in two passes, because the value + does not exist until Coolify makes it: pass 1 \`capture\` placeholds those names, + \`apply\` creates the database, and this fills the store with the URL Coolify then + generated. It INVERTS capture's rule — it fills the generated names and leaves + every other name in the store exactly as it is. The store must already exist. + The value is read from the DATABASE that owns it (\`internal_db_url\`), resolved + inside this project+environment only — never from a consuming app's env, where a + generated URL never appears, and never from the instance-wide database list. + --from = which live database NAME is filled from. Required whenever + more than one database could be meant: nothing in the + manifest, the templates or the box says that DATABASE_URL + comes from the postgres one, and cast refuses to guess by + name rather than write another database's credentials into + your store. Repeatable. + --force fill a generated name that already holds a REAL value + (refused by default — it is a silent credential rotation). + inventory --emit-draft (write down what a box has, as a PROPOSAL): --emit-draft emit what the sweep saw as a draft of cast's own inputs — a manifest per project, env templates, an environments.yaml @@ -574,6 +599,48 @@ export function parseResourceAliases( return alias; } +// `--from =` — the edge nothing else in the system carries. +// +// The right side is the database as COOLIFY names it, in this project and +// environment (which is what `capture --generated-only`'s own refusal prints +// for you). Not the manifest's name: --resource exists to reconcile those two +// vocabularies for the diff, and pass 2 reads its value straight off the live +// resource, so the live name is the one that can be checked. +export function parseFromPairs( + pairs: string[], + generated: string[], +): Record { + const out: Record = {}; + for (const pair of pairs) { + const eq = pair.indexOf("="); + if (eq <= 0 || eq === pair.length - 1) { + throw new Error(`--from expects =, got "${pair}"`); + } + const ref = pair.slice(0, eq).trim(); + const db = pair.slice(eq + 1).trim(); + // A --from naming something that is not a generated secret is a no-op that + // LOOKS like it did something: pass 2 fills generated names and nothing + // else, so the flag would be silently ignored and the operator would walk + // away believing they had set a value. + if (!generated.includes(ref)) { + throw new Error( + [ + `--from ${ref}=${db}: ${ref} is not a generated secret in this environment`, + "", + ` generated: ${generated.join(", ") || "(none declared)"}`, + "", + "--generated-only fills the generated names only. A name that is not one of", + "them is carried over from the store untouched, and --from cannot change that.", + "Declare it in the manifest's `generated_secrets:` (or pass --generated )", + "if it really is provider-generated.", + ].join("\n"), + ); + } + out[ref] = db; + } + return out; +} + // Rename live resources to the manifest's vocabulary, once, at the boundary. // Everything downstream — computeDiff, classify, reconcile — then matches by // name as it always has, and none of them needs to know a box was involved. @@ -611,6 +678,62 @@ async function fetchEnv( return Object.fromEntries(envs.map((e) => [e.key, e.real_value ?? e.value])); } +// The databases inside ONE project+environment, each carrying the value it +// OWNS. This is `capture --generated-only`'s only read of the box. +// +// Deliberately NOT `GET /databases`. That route lists every database on the +// INSTANCE — other projects', and umami's own bundled Postgres — so finding +// ours in it means matching by name across a list where a collision is both +// possible and silent (#29 in another hat; the hand-run jq this verb replaces +// had a comment warning not to pick the third row). `GET /projects/{uuid}/{env}` +// cannot express that bug: it eager-loads `postgresqls` and `redis` for THIS +// environment and nothing else (ProjectController@environment_details, +// coollabsio/coolify v4.1.2), so the scoping is structural rather than a filter +// cast has to remember to get right. +// +// `internal_db_url` is an appended model attribute — `protected $appends = +// ['internal_db_url', 'external_db_url', 'database_type', 'server_status']` on +// BOTH app/Models/StandalonePostgresql.php and app/Models/StandaloneRedis.php +// @ v4.1.2. Same key on both; only the URL it builds differs +// (`postgres://user:pw@{uuid}:5432/{db}` vs `redis://user:pw@{uuid}:6379/0`). +// environment_details serializes the models whole — serializeApiResponse +// (bootstrap/helpers/api.php) only sorts keys, and unlike DatabasesController +// it calls no removeSensitiveData() — so the field is present here WITHOUT the +// sensitive-read token permission that `GET /databases` gates it behind +// (`can_read_sensitive` → makeHidden(['internal_db_url', …])). The vendored +// OpenAPI documents neither route's body ("Content is very complex. Will be +// implemented later."); the spec's silence is not evidence of absence (#46). +async function fetchGeneratedSources( + client: CoolifyClient, + projectName: string, + envName: string, +): Promise<{ sources: GeneratedSource[]; urlless: string[] }> { + const uuid = await client.projectUuid(projectName); + const raw = (await client.get( + `/projects/${uuid}/${encodeURIComponent(envName)}`, + )) as { + postgresqls?: Array>; + redis?: Array>; + } | null; + const sources: GeneratedSource[] = []; + const urlless: string[] = []; + const take = (type: string, items: Array> = []) => { + for (const i of items) { + const url = i.internal_db_url; + // A database that is THERE but will not tell us its URL. Never a fill of + // "" — that re-encrypts cleanly and boots the app pointed at nothing. + if (typeof url !== "string" || url === "") { + urlless.push(String(i.name)); + continue; + } + sources.push({ resource: String(i.name), type, url }); + } + }; + take("postgresql", raw?.postgresqls); + take("redis", raw?.redis); + return { sources, urlless }; +} + // The value for an --override, read from the ENVIRONMENT rather than argv. // // A secret passed as a command-line argument is visible in `ps` to every @@ -1030,6 +1153,8 @@ async function main(): Promise { generated: { type: "string", multiple: true }, override: { type: "string", multiple: true }, force: { type: "boolean", default: false }, + "generated-only": { type: "boolean", default: false }, + from: { type: "string", multiple: true }, }, }); const orgRepo = positionals[0]; @@ -1038,6 +1163,9 @@ async function main(): Promise { console.error(USAGE); return 2; } + // Pass 2 of a two-pass bootstrap. Not a different verb: same ceremony, same + // store-writing code path, one inverted disposition rule. See capture.ts. + const generatedOnly = values["generated-only"]; const stateDir = stateDirFrom(values.state); const repoShort = orgRepo.split("/")[1]; const projectName = values.project ?? repoShort; @@ -1047,11 +1175,63 @@ async function main(): Promise { // vocabulary. const coolifyEnv = values.environment ?? envName; const store = secretsFileFor(stateDir, repoShort, envName); + // Flag pairings that can never be honored, refused up front — before a state + // file, a store, an age key or a Coolify is opened (same disposition as + // PATH_IN_PROD_REFUSAL). --override supplies a value for a name cast would + // otherwise CAPTURE, and --generated-only captures nothing; --from names the + // database a GENERATED name comes from, and only pass 2 fills those. + if (generatedOnly && (values.override ?? []).length > 0) { + console.error( + "refuses --override with --generated-only: pass 2 fills the generated names and leaves every other name exactly as the store has it — there is nothing for an override to override. Set the value in pass 1 (`cast capture --override`), or edit it there.", + ); + return 2; + } + if (!generatedOnly && (values.from ?? []).length > 0) { + console.error( + "refuses --from without --generated-only: --from names the database a generated secret is filled FROM, and plain `capture` never fills one — it placeholds them (that is the point of pass 1).", + ); + return 2; + } + // --resource reconciles the MANIFEST's vocabulary with the box's for the + // env-reading pass, and pass 2 reads no env: it takes its value straight off + // the live database, which --from names in the box's own vocabulary. Left + // accepted, the flag would be silently ignored — the exact "the flag missed + // and nothing said so" failure parseResourceAliases refuses for. + if (generatedOnly && (values.resource ?? []).length > 0) { + console.error( + "refuses --resource with --generated-only: pass 2 reads no application env, so there is no manifest-to-box name mapping for it to use. --from names the live database directly, in the box's own vocabulary.", + ); + return 2; + } + // The two passes take OPPOSITE positions on the store, and both are the same + // rule: never destroy values that exist nowhere else. + // + // Pass 1 writes the store from nothing, so an existing one is something it + // must not clobber. Pass 2 fills names INTO the store pass 1 wrote, so an + // absent one is not a blank slate — it means this run is pointed somewhere + // unexpected, and writing would produce a store holding two names out of + // fourteen. + if (generatedOnly && !existsSync(store)) { + console.error( + [ + `refusing to capture --generated-only: ${store} does not exist`, + "", + "Pass 2 FILLS the generated names in a store that pass 1 already wrote — it does", + "not create one. A store written from here would hold only the generated names,", + "and every other name the manifest requires would be silently absent from it.", + "", + "Run `cast capture` first (pass 1), then `apply`, then this.", + ].join("\n"), + ); + return 2; + } // Never overwrite a store by accident. `apply` never deletes; the verb // that WRITES the store gets the same disposition, because the thing it // would destroy is the only copy of values that may not exist anywhere - // else any more. - if (existsSync(store) && !values.force) { + // else any more. (Pass 2 is exempt: it REQUIRES the store to exist, and + // reuses --force for the finer refusal — overwriting a generated name that + // already holds a real value. See planGenerated.) + if (!generatedOnly && existsSync(store) && !values.force) { console.error( [ `refusing to capture: ${store} already exists`, @@ -1113,6 +1293,101 @@ async function main(): Promise { ); return 2; } + if (generatedOnly) { + // Pass 2 needs the age IDENTITY, not just the recipient: it fills names + // into a store it must first read. Everything it does not fill is carried + // over from here byte for byte — never re-read from the box, which is what + // makes this safe to run against a live environment whose other secrets + // have since been rotated by hand. + const keyFile = keyFileFor(envName); + const before = decryptSecrets(store, keyFile); + const generatedNames = [ + ...new Set([...generated, ...(values.generated ?? [])]), + ]; + const { sources, urlless } = await fetchGeneratedSources( + client, + projectName, + coolifyEnv, + ); + // A database that exists but will not report its URL. The only way this + // happens on this route is a Coolify whose shape we do not know — so it + // stops, rather than filling a name with something that is not a URL. + if (urlless.length > 0) { + console.error( + [ + `refusing to capture --generated-only: ${urlless.length} database(s) report no internal_db_url`, + "", + ` ${urlless.join(", ")}`, + "", + "`internal_db_url` is an appended attribute on Coolify's StandalonePostgresql /", + "StandaloneRedis models (v4.1.2) and this route serializes them whole, so its", + "absence means this Coolify is not the shape cast knows. Filling a secret with an", + "empty value would re-encrypt cleanly and boot the app pointed at nothing.", + ].join("\n"), + ); + return 2; + } + const { mapping, unmapped } = resolveGeneratedSources( + generatedNames, + sources, + parseFromPairs(values.from ?? [], generatedNames), + ); + const plan = planGenerated(generatedNames, before, mapping, unmapped, { + force: values.force, + }); + console.log( + renderGeneratedPlan(plan, { + orgRepo, + env: envName, + instance: values.instance ?? binding.instance ?? "default", + store, + recipient, + project: projectName, + environment: coolifyEnv, + }), + ); + if (generatedPlanRefuses(plan)) return 2; + if (plan.fills.length === 0) { + console.log( + "\nnothing to fill — this environment declares no generated secrets, and no name in the store is still pending.", + ); + return 0; + } + if (!(await confirmCapture(envName))) { + console.error("aborted — nothing written"); + return 2; + } + encryptSecrets(recipient, store, { + ...before, + ...Object.fromEntries(plan.fills.map((f) => [f.ref, f.value])), + }); + // The postcondition this verb exists for, asserted against the ciphertext + // that is now on disk — decrypted back, not trusted from memory. In the + // hand-run procedure this was a line in a runbook, which is to say a step + // that could be skipped, and was only ever as good as the operator's + // attention at the end of a long careful thing. + const after = decryptSecrets(store, keyFile); + const violations = assertGeneratedComplete(before, after); + if (violations.length > 0) { + console.error( + [ + "", + `POSTCONDITION FAILED — ${store} was written, and it is not what it should be:`, + "", + ...violations.map((v) => ` - ${v}`), + "", + "This store is suspect. Do not apply from it. Restore the previous ciphertext", + "from the state repo (it is committed) and report this — cast wrote a store whose", + "shape it does not itself accept, which is a bug in cast, not in your invocation.", + ].join("\n"), + ); + return 2; + } + console.log( + `\nwrote ${store} — ${plan.fills.length} name(s) filled, ${Object.keys(after).length} name(s) total (unchanged), zero pending-coolify-generated remaining, encrypted to ${recipient}`, + ); + return 0; + } const aliases = parseResourceAliases( values.resource ?? [], manifestResources(checkout, envName).map((r) => r.name), diff --git a/test/capture-cli.test.ts b/test/capture-cli.test.ts index a45c42e..787d227 100644 --- a/test/capture-cli.test.ts +++ b/test/capture-cli.test.ts @@ -11,7 +11,7 @@ import type { AddressInfo } from "node:net"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { afterEach, beforeAll, describe, expect, it } from "vitest"; -import { decryptSecrets } from "../src/secrets.js"; +import { decryptSecrets, encryptSecrets } from "../src/secrets.js"; // End-to-end: the real CLI, a real age identity, a stub Coolify holding real // live values. The point is the store that comes out the other side — it is @@ -106,10 +106,16 @@ OPENROUTER_API_KEY=\${OPENROUTER_API_KEY} ADMIN_EMAIL=\${ADMIN_EMAIL} `; -function fixture(url: string, opts: { template?: string } = {}) { +function fixture( + url: string, + opts: { template?: string; manifest?: string } = {}, +) { const checkout = mkdtempSync(join(tmpdir(), "cast-co-")); mkdirSync(join(checkout, ".infra", "env"), { recursive: true }); - writeFileSync(join(checkout, ".infra", "manifest.yaml"), MANIFEST); + writeFileSync( + join(checkout, ".infra", "manifest.yaml"), + opts.manifest ?? MANIFEST, + ); writeFileSync( join(checkout, ".infra", "env", "core.staging.env.template"), opts.template ?? TEMPLATE, @@ -332,3 +338,370 @@ describe("cast capture (end to end)", () => { expect(r.output).not.toMatch(/MISSING/); }); }); + +// --------------------------------------------------------------------------- +// Pass 2 — `cast capture --generated-only` (end to end) +// +// The bootstrap this closes: pass 1 wrote a store in which the generated names +// are placeholders, `apply` then created the database, and Coolify generated the +// real URL. Until this runs, the store's value is a lie while the live value is +// real — which is the state that makes the next routine apply overwrite a +// working secret (#47). Everything here runs against a real age identity and a +// real `dist/cli.js`; the store is decrypted afterwards and asserted on. +// --------------------------------------------------------------------------- + +// The values Coolify generated when it created the resources. NOT reachable from +// any application's env — the app's env holds what the template resolved to, +// which at this point in the bootstrap is the placeholder itself. +const PG_URL = "postgres://postgres:GENERATED-PG-PASSWORD@abc123:5432/app"; +const REDIS_URL = "redis://default:GENERATED-REDIS-PASSWORD@def456:6379/0"; +// Another project's database entirely. `GET /databases` would list it right next +// to ours — it is umami's bundled Postgres, the row the hand-run `jq` had to be +// careful not to pick. A name-directed lookup across the instance eventually +// takes it, and what it writes is a well-formed URL to the wrong database. +const UMAMI_URL = "postgres://umami:UMAMI-PASSWORD@zzz999:5432/umami"; + +const GEN_MANIFEST = `project: incubator +environments: + staging: + generated_secrets: [DATABASE_URL, REDIS_URL] + applications: + core: + source: { repo: heavy-duty/incubator, branch: main } + build: { pack: nixpacks, base_directory: / } + domains: ["http://core.example.com"] + env_template: core.staging.env.template + databases: + incubator-db: { type: postgresql } + incubator-redis: { type: redis } +`; + +const GEN_TEMPLATE = `NODE_ENV=production +DATABASE_URL=\${DATABASE_URL} +REDIS_URL=\${REDIS_URL} +MAILGUN_API_KEY=\${MAILGUN_API_KEY} +`; + +// What pass 1 left behind: the two generated names placeheld, the real secret +// captured. Pass 2 must fill the first two and not disturb the third. +const PASS1_STORE = { + DATABASE_URL: "pending-coolify-generated", + REDIS_URL: "pending-coolify-generated", + MAILGUN_API_KEY: "key-REAL-MAILGUN-SECRET", +}; + +// Records every path the CLI asks for, so a test can assert on what cast did +// NOT call — `GET /databases` is instance-wide, and never reaching for it is the +// point, not an implementation detail. +async function stubCoolifyWithDatabases(): Promise { + const hits: string[] = []; + const server = createServer((req, res) => { + const path = (req.url ?? "").replace("/api/v1", ""); + hits.push(path); + const json = (body: unknown) => { + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify(body)); + }; + if (path === "/teams/current") return json({ id: 0, name: "Root Team" }); + if (path === "/projects") + return json([ + { uuid: "p1", name: "incubator" }, + { uuid: "p2", name: "analytics" }, + ]); + // The project+environment document. Coolify's environment_details eager-loads + // these relations and serializes the models whole, so `internal_db_url` (an + // appended attribute on StandalonePostgresql / StandaloneRedis) rides along. + if (path === "/projects/p1/staging") + return json({ + applications: [{ name: "core", uuid: "a1" }], + postgresqls: [ + { name: "incubator-db", uuid: "abc123", internal_db_url: PG_URL }, + ], + redis: [ + { + name: "incubator-redis", + uuid: "def456", + internal_db_url: REDIS_URL, + }, + ], + }); + // Instance-wide: OURS and somebody else's, indistinguishable by name alone. + // Nothing in cast may read this. + if (path === "/databases") + return json([ + { name: "incubator-db", internal_db_url: PG_URL }, + { name: "incubator-redis", internal_db_url: REDIS_URL }, + { name: "incubator-db", internal_db_url: UMAMI_URL }, + ]); + // The app's env — it holds the PLACEHOLDER, which is exactly why the value + // must be read off the database instead. + if (path === "/applications/a1/envs") + return json([ + { + key: "DATABASE_URL", + real_value: "pending-coolify-generated", + value: "REDACTED", + }, + { + key: "MAILGUN_API_KEY", + real_value: "key-REAL-MAILGUN-SECRET", + value: "REDACTED", + }, + ]); + res.writeHead(404); + res.end("{}"); + }); + await new Promise((r) => { + server.listen(0, "127.0.0.1", r); + }); + const stub = { + url: `http://127.0.0.1:${(server.address() as AddressInfo).port}`, + close: () => + new Promise((r) => { + server.close(() => r()); + }), + hits, + }; + stubs.push(stub); + return stub; +} + +// A fixture whose store is already what pass 1 would have written. +function genFixture(url: string, store: Record = PASS1_STORE) { + const f = fixture(url, { manifest: GEN_MANIFEST, template: GEN_TEMPLATE }); + encryptSecrets(recipient, f.store, store); + return f; +} + +const genBase = (f: ReturnType) => [ + ...base(f), + "--generated-only", +]; + +// Pass 2 reads the store before it fills it, so it needs the age IDENTITY — +// not just the recipient pass 1 needed. +const withKey = () => ({ CAST_AGE_KEY_FILE_STAGING: keyFile }); + +const FROM = [ + "--from", + "DATABASE_URL=incubator-db", + "--from", + "REDIS_URL=incubator-redis", +]; + +describe("cast capture --generated-only (end to end)", () => { + it("fills the generated names from the databases that own them, and leaves the rest alone", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url); + const r = await runCapture([...genBase(f), ...FROM], { + stdin: "staging\n", + env: withKey(), + }); + expect(r.code).toBe(0); + + const store = decryptSecrets(f.store, keyFile); + // The postcondition, asserted on the actual ciphertext: zero placeholders, + // and the name count unchanged. + expect(Object.keys(store).sort()).toEqual([ + "DATABASE_URL", + "MAILGUN_API_KEY", + "REDIS_URL", + ]); + expect(Object.values(store)).not.toContain("pending-coolify-generated"); + // Each from the resource that OWNS it — and the redis one is a redis URL, + // not the Postgres URL under a redis name. + expect(store.DATABASE_URL).toBe(PG_URL); + expect(store.REDIS_URL).toBe(REDIS_URL); + // Carried over byte for byte. Pass 2 never re-read it from the box. + expect(store.MAILGUN_API_KEY).toBe("key-REAL-MAILGUN-SECRET"); + // Never the other project's database. + expect(store.DATABASE_URL).not.toContain("UMAMI"); + expect(r.output).toMatch(/zero pending-coolify-generated remaining/); + }); + + // The #29 hazard, structurally: the value is resolved inside the project and + // environment, so the instance-wide list is never even consulted. + it("never reads the instance-wide database list", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url); + const r = await runCapture([...genBase(f), ...FROM], { + stdin: "staging\n", + env: withKey(), + }); + expect(r.code).toBe(0); + expect(stub.hits).toContain("/projects/p1/staging"); + expect(stub.hits).not.toContain("/databases"); + }); + + // capture.ts:166's rule holds in pass 2: the only value-shaped thing printed + // is the placeholder literal being replaced. + it("never prints a secret value to the console", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url); + const r = await runCapture([...genBase(f), ...FROM], { + stdin: "staging\n", + env: withKey(), + }); + expect(r.code).toBe(0); + for (const value of [PG_URL, REDIS_URL, "key-REAL-MAILGUN-SECRET"]) { + expect(r.output).not.toContain(value); + } + expect(r.output).not.toContain("GENERATED-PG-PASSWORD"); + // The NAMES, and where each value came from, are the plan. + expect(r.output).toMatch(/DATABASE_URL\s+fill/); + expect(r.output).toContain("incubator-db (postgresql) internal_db_url"); + expect(r.output).toMatch(/MAILGUN_API_KEY\s+keep/); + }); + + // Nothing in the manifest, the templates or the box says DATABASE_URL comes + // from the postgres one. cast will not guess — it hands back the flag. + it("refuses to guess which database a name comes from, and says how to say it", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url); + const r = await runCapture(genBase(f), { + stdin: "staging\n", + env: withKey(), + }); + expect(r.code).not.toBe(0); + expect(r.output).toMatch(/UNMAPPED/); + expect(r.output).toContain("--from DATABASE_URL="); + // Refused BEFORE the store was touched. + expect(decryptSecrets(f.store, keyFile)).toEqual(PASS1_STORE); + }); + + // The refusal that keeps this from being a silent credential rotation. + it("refuses to overwrite a generated name that already holds a real value", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url, { + ...PASS1_STORE, + DATABASE_URL: "postgres://set:BY-HAND@live:5432/app", + }); + const r = await runCapture([...genBase(f), ...FROM], { + stdin: "staging\n", + env: withKey(), + }); + expect(r.code).not.toBe(0); + expect(r.output).toMatch(/OCCUPIED/); + expect(r.output).toMatch(/rotate a live credential/); + expect(r.output).toMatch(/--force/); + // Untouched — including the name it COULD have filled. A refusal is a stop, + // not a partial write. + const store = decryptSecrets(f.store, keyFile); + expect(store.DATABASE_URL).toBe("postgres://set:BY-HAND@live:5432/app"); + expect(store.REDIS_URL).toBe("pending-coolify-generated"); + }); + + it("--force rotates it deliberately", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url, { + ...PASS1_STORE, + DATABASE_URL: "postgres://set:BY-HAND@live:5432/app", + }); + const r = await runCapture([...genBase(f), ...FROM, "--force"], { + stdin: "staging\n", + env: withKey(), + }); + expect(r.code).toBe(0); + expect(decryptSecrets(f.store, keyFile).DATABASE_URL).toBe(PG_URL); + }); + + // Pass 2 fills a store; it does not create one. A store written from here + // would hold the generated names and nothing else. + it("refuses when the store does not exist — pass 1 has not run", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = fixture(stub.url, { + manifest: GEN_MANIFEST, + template: GEN_TEMPLATE, + }); + const r = await runCapture([...genBase(f), ...FROM], { + stdin: "staging\n", + env: withKey(), + }); + expect(r.code).not.toBe(0); + expect(r.output).toMatch(/does not exist/); + expect(r.output).toMatch(/Pass 2 FILLS the generated names/); + expect(existsSync(f.store)).toBe(false); + }); + + // A placeholder nobody fills would leave the store still lying — and the next + // apply would push that literal at a live app. + it("refuses to leave a placeholder standing in a name it does not fill", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url, { + ...PASS1_STORE, + MAILGUN_API_KEY: "pending-coolify-generated", + }); + const r = await runCapture([...genBase(f), ...FROM], { + stdin: "staging\n", + env: withKey(), + }); + expect(r.code).not.toBe(0); + expect(r.output).toMatch(/MAILGUN_API_KEY\s+PENDING/); + expect(r.output).toMatch(/would still hold the/); + }); + + // The confirmation is the same last gate as pass 1, and it is not "y". + it("aborts, writing nothing, when the confirmation does not name the env", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url); + const r = await runCapture([...genBase(f), ...FROM], { + stdin: "y\n", + env: withKey(), + }); + expect(r.code).not.toBe(0); + expect(r.output).toMatch(/aborted/); + expect(decryptSecrets(f.store, keyFile)).toEqual(PASS1_STORE); + }); + + // A flag pairing that can never be honored: pass 2 captures nothing, so there + // is nothing for an override to override. + it("refuses --override with --generated-only", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url); + const r = await runCapture( + [...genBase(f), ...FROM, "--override", "MAILGUN_API_KEY"], + { stdin: "staging\n", env: withKey() }, + ); + expect(r.code).not.toBe(0); + expect(r.output).toMatch(/refuses --override with --generated-only/); + }); + + it("refuses --from without --generated-only", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = fixture(stub.url, { + manifest: GEN_MANIFEST, + template: GEN_TEMPLATE, + }); + const r = await runCapture([...base(f), ...FROM], { stdin: "staging\n" }); + expect(r.code).not.toBe(0); + expect(r.output).toMatch(/refuses --from without --generated-only/); + }); + + // A --from for a name that is not generated would be silently ignored, and + // the operator would walk away believing they had set a value. + it("refuses a --from naming something that is not a generated secret", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url); + const r = await runCapture( + [...genBase(f), "--from", "MAILGUN_API_KEY=incubator-db"], + { stdin: "staging\n", env: withKey() }, + ); + expect(r.code).not.toBe(0); + expect(r.output).toMatch(/is not a generated secret/); + }); +}); + +// --resource maps manifest names to box names for the env-reading pass, and pass +// 2 reads no env. Accepted, it would be silently ignored. +describe("cast capture --generated-only (flag hygiene)", () => { + it("refuses --resource with --generated-only", async () => { + const stub = await stubCoolifyWithDatabases(); + const f = genFixture(stub.url); + const r = await runCapture( + [...genBase(f), ...FROM, "--resource", "core=Core"], + { stdin: "staging\n", env: withKey() }, + ); + expect(r.code).not.toBe(0); + expect(r.output).toMatch(/refuses --resource with --generated-only/); + }); +}); diff --git a/test/capture.test.ts b/test/capture.test.ts index 8118428..6824fe9 100644 --- a/test/capture.test.ts +++ b/test/capture.test.ts @@ -4,8 +4,14 @@ import { join } from "node:path"; import { describe, expect, it } from "vitest"; import { GENERATED_PLACEHOLDER, + type GeneratedSource, + assertGeneratedComplete, classify, + generatedPlanRefuses, + planGenerated, renderCapturePlan, + renderGeneratedPlan, + resolveGeneratedSources, } from "../src/capture.js"; import { requiredSecrets } from "../src/resolve.js"; @@ -307,3 +313,284 @@ environments: ); }); }); + +// --------------------------------------------------------------------------- +// Pass 2 — capture --generated-only +// --------------------------------------------------------------------------- + +// The live shape this verb exists for: the incubator, mid-bootstrap. Two +// databases in the project+environment, and a third Postgres that belongs to +// umami — the row the hand-run `jq | head` had to be careful not to pick, and +// the one a name-directed lookup across `GET /databases` would eventually take. +const PG: GeneratedSource = { + resource: "incubator-db", + type: "postgresql", + url: "postgres://u:REAL-PG-PASSWORD@abc123:5432/app", +}; +const REDIS: GeneratedSource = { + resource: "incubator-redis", + type: "redis", + url: "redis://default:REAL-REDIS-PASSWORD@def456:6379/0", +}; + +// Pass 1 left these two placeheld; every other name is real and must survive. +const STORE = { + DATABASE_URL: GENERATED_PLACEHOLDER, + REDIS_URL: GENERATED_PLACEHOLDER, + MAILGUN_API_KEY: "key-REAL", + ADMIN_EMAIL: "operator@example.com", +}; +const GENERATED = ["DATABASE_URL", "REDIS_URL"]; + +const GCTX = { + orgRepo: "heavy-duty/incubator", + env: "prod", + instance: "default", + store: "/s/secrets/incubator.prod.env.age", + recipient: "age1abc", + project: "incubator", + environment: "production", +}; + +// The mapping is stated, then the plan is built from it. +const planWith = ( + from: Record, + store = STORE, + databases = [PG, REDIS], + opts?: { force: boolean }, +) => { + const { mapping, unmapped } = resolveGeneratedSources( + GENERATED, + databases, + from, + ); + return planGenerated(GENERATED, store, mapping, unmapped, opts); +}; + +describe("resolveGeneratedSources", () => { + // The whole point: the value comes off the DATABASE, not off an app. + it("maps each generated name to the database the operator named", () => { + const { mapping, unmapped } = resolveGeneratedSources( + GENERATED, + [PG, REDIS], + { DATABASE_URL: "incubator-db", REDIS_URL: "incubator-redis" }, + ); + expect(unmapped).toEqual([]); + expect(mapping.DATABASE_URL).toEqual(PG); + expect(mapping.REDIS_URL).toEqual(REDIS); + }); + + // #29 wearing a different hat. Nothing anywhere says DATABASE_URL comes from + // the postgres one — inferring it from the NAME is exactly the silent wrong + // pick this refuses, because what it writes is a well-formed URL to somebody + // else's database. + it("refuses to guess when more than one database could be meant", () => { + const { mapping, unmapped } = resolveGeneratedSources( + GENERATED, + [PG, REDIS], + {}, + ); + expect(mapping).toEqual({}); + expect(unmapped.map((u) => u.ref)).toEqual(["DATABASE_URL", "REDIS_URL"]); + expect(unmapped[0].why).toMatch(/will not pick by name/); + // It names the candidates rather than picking one. + expect(unmapped[0].why).toContain("incubator-db:postgresql"); + expect(unmapped[0].why).toContain("incubator-redis:redis"); + }); + + // The one inference that cannot be wrong: nothing else it could be. + it("infers the only database when there is exactly one, for one name", () => { + const { mapping, unmapped } = resolveGeneratedSources( + ["DATABASE_URL"], + [PG], + {}, + ); + expect(unmapped).toEqual([]); + expect(mapping.DATABASE_URL).toEqual(PG); + }); + + // ...and still refuses two names against that one database: filling REDIS_URL + // from the Postgres would be a perfectly well-formed lie. + it("does not infer when one database must serve two generated names", () => { + const { unmapped } = resolveGeneratedSources(GENERATED, [PG], {}); + expect(unmapped.map((u) => u.ref)).toEqual(["DATABASE_URL", "REDIS_URL"]); + }); + + it("refuses a --from naming a database that is not in this project+env", () => { + const { unmapped } = resolveGeneratedSources(GENERATED, [PG, REDIS], { + DATABASE_URL: "umami-db", + REDIS_URL: "incubator-redis", + }); + expect(unmapped).toHaveLength(1); + expect(unmapped[0].why).toMatch(/no database named "umami-db" exists/); + }); + + it("refuses when the environment holds no database at all", () => { + const { unmapped } = resolveGeneratedSources(["DATABASE_URL"], [], {}); + expect(unmapped[0].why).toMatch(/no database exists/); + }); +}); + +describe("planGenerated", () => { + it("fills the generated names and keeps every other one byte for byte", () => { + const p = planWith({ + DATABASE_URL: "incubator-db", + REDIS_URL: "incubator-redis", + }); + expect(p.fills).toEqual([ + { + ref: "DATABASE_URL", + from: { resource: "incubator-db", type: "postgresql" }, + value: PG.url, + }, + { + ref: "REDIS_URL", + from: { resource: "incubator-redis", type: "redis" }, + value: REDIS.url, + }, + ]); + // Untouched — and NOT re-read from the box, which is what makes pass 2 safe + // to run against an environment whose other secrets were rotated by hand. + expect(p.kept).toEqual(["ADMIN_EMAIL", "MAILGUN_API_KEY"]); + expect(p.occupied).toEqual([]); + expect(p.absent).toEqual([]); + expect(p.stillPending).toEqual([]); + }); + + // The refusal that stops a silent credential rotation: someone already filled + // it (a previous pass 2, or by hand) and the value is live. + it("refuses to overwrite a generated name that already holds a real value", () => { + const p = planWith( + { DATABASE_URL: "incubator-db", REDIS_URL: "incubator-redis" }, + { ...STORE, DATABASE_URL: "postgres://already:filled@live/db" }, + ); + expect(p.occupied).toEqual(["DATABASE_URL"]); + expect(p.fills.map((f) => f.ref)).toEqual(["REDIS_URL"]); + expect(generatedPlanRefuses(p)).toBe(true); + }); + + it("--force fills it anyway, deliberately", () => { + const p = planWith( + { DATABASE_URL: "incubator-db", REDIS_URL: "incubator-redis" }, + { ...STORE, DATABASE_URL: "postgres://already:filled@live/db" }, + [PG, REDIS], + { force: true }, + ); + expect(p.occupied).toEqual([]); + expect(p.fills.map((f) => f.ref)).toEqual(["DATABASE_URL", "REDIS_URL"]); + expect(generatedPlanRefuses(p)).toBe(false); + }); + + // Pass 2 FILLS names; it does not add them. A store missing one did not come + // from pass 1, and the name-count postcondition could not hold anyway. + it("refuses a generated name the store does not carry at all", () => { + const { REDIS_URL, ...withoutRedis } = STORE; + const p = planWith( + { DATABASE_URL: "incubator-db", REDIS_URL: "incubator-redis" }, + withoutRedis, + ); + expect(p.absent).toEqual(["REDIS_URL"]); + expect(generatedPlanRefuses(p)).toBe(true); + }); + + // A placeholder standing in a name nothing here fills: the run would + // "succeed" and the store would still be a lie. + it("refuses when a name nobody fills would be left still pending", () => { + const p = planWith( + { DATABASE_URL: "incubator-db", REDIS_URL: "incubator-redis" }, + { ...STORE, SESSION_SECRET: GENERATED_PLACEHOLDER }, + ); + expect(p.stillPending).toEqual(["SESSION_SECRET"]); + expect(generatedPlanRefuses(p)).toBe(true); + }); + + it("a refused name is not also reported as kept", () => { + const p = planWith({}); + expect(p.kept).toEqual(["ADMIN_EMAIL", "MAILGUN_API_KEY"]); + expect(p.unmapped).toHaveLength(2); + }); +}); + +describe("renderGeneratedPlan", () => { + // capture.ts:166's rule, unchanged: names, provenance, and the resource a + // value came FROM. The only value-shaped thing printed is the placeholder + // literal being replaced. + it("prints names and never a value", () => { + const out = renderGeneratedPlan( + planWith({ DATABASE_URL: "incubator-db", REDIS_URL: "incubator-redis" }), + GCTX, + ); + expect(out).not.toContain("REAL-PG-PASSWORD"); + expect(out).not.toContain("REAL-REDIS-PASSWORD"); + expect(out).not.toContain("key-REAL"); + expect(out).toContain("DATABASE_URL"); + expect(out).toContain("incubator-db (postgresql) internal_db_url"); + expect(out).toContain(GENERATED_PLACEHOLDER); + // The store is not being rewritten around the names it keeps, and says so. + expect(out).toMatch(/MAILGUN_API_KEY\s+keep/); + }); + + it("hands back a ready-to-paste --from when it will not guess", () => { + const out = renderGeneratedPlan(planWith({}), GCTX); + expect(out).toMatch(/refusing to write the store/); + expect(out).toContain("--from DATABASE_URL="); + expect(out).toContain("--from REDIS_URL="); + }); + + it("says a fill would rotate a live credential", () => { + const out = renderGeneratedPlan( + planWith( + { DATABASE_URL: "incubator-db", REDIS_URL: "incubator-redis" }, + { ...STORE, DATABASE_URL: "postgres://live" }, + ), + GCTX, + ); + expect(out).toMatch(/OCCUPIED/); + expect(out).toMatch(/rotate a live credential/); + expect(out).not.toContain("postgres://live"); + }); +}); + +// The assertion that was a line in a human runbook ("assert 14 names / zero +// placeholders"), which is to say a step that could be skipped. +describe("assertGeneratedComplete", () => { + it("passes when every placeholder is gone and the name set is unchanged", () => { + const after = { ...STORE, DATABASE_URL: PG.url, REDIS_URL: REDIS.url }; + expect(assertGeneratedComplete(STORE, after)).toEqual([]); + }); + + it("catches a placeholder left standing", () => { + const after = { ...STORE, DATABASE_URL: PG.url }; + const v = assertGeneratedComplete(STORE, after); + expect(v).toHaveLength(1); + expect(v[0]).toMatch( + /still hold the pending-coolify-generated literal: REDIS_URL/, + ); + }); + + // A store that LOST a name re-encrypts perfectly and reads back perfectly. + // The failure surfaces at the next apply, in an environment whose plaintext + // nobody has any more. + it("catches a name dropped on the way through", () => { + const { MAILGUN_API_KEY, ...lost } = { + ...STORE, + DATABASE_URL: PG.url, + REDIS_URL: REDIS.url, + }; + const v = assertGeneratedComplete(STORE, lost); + expect(v.join(" ")).toMatch(/went in with 4 name\(s\) and came out with 3/); + expect(v.join(" ")).toMatch(/names LOST: MAILGUN_API_KEY/); + }); + + it("catches a name that was never supposed to be added", () => { + const after = { + ...STORE, + DATABASE_URL: PG.url, + REDIS_URL: REDIS.url, + SURPRISE: "x", + }; + expect(assertGeneratedComplete(STORE, after).join(" ")).toMatch( + /names ADDED: SURPRISE/, + ); + }); +});