diff --git a/README.md b/README.md index e553dee..06ff3a4 100644 --- a/README.md +++ b/README.md @@ -331,6 +331,16 @@ manifest is what knows `DATABASE_URL` comes from a database it declares. (A standing over nothing is worse than no guard, because it reads like one. `--generated ` covers a manifest that hasn't declared them yet.) +> **A database's own URL is better *derived* than stored.** If the value is the +> URL of a database this manifest declares, write `DATABASE_URL=${resource:postgres.url}` +> in the template instead of storing it. cast reads it back from the database it +> created — never into the store, never through a terminal — so there is no +> placeholder, no two-pass dance, and no stored copy to drift or overwrite. A +> rotated password is simply *followed* on the next apply. `generated_secrets:` +> and the pass below stay for the residual class — a provider-generated value +> that genuinely is not derivable. See [semantics.md](docs/semantics.md) → +> *Derived resource URLs*. + An **`--override`**'s value is read from `$CAST_CAPTURE_`, never from the command line: argv is visible in `ps` to every process on the box. It exists for values that must not survive the copy — staging and prod sharing a Mailgun diff --git a/docs/semantics.md b/docs/semantics.md index 10ef9be..b4cc609 100644 --- a/docs/semantics.md +++ b/docs/semantics.md @@ -852,6 +852,75 @@ mutate the encrypted store and hence the git repo — a much bigger blast radius a verb people run on a schedule. A separate, explicit, operator-run verb is the right first step. +## Derived resource URLs (`${resource:.url}`) + +The two-pass bootstrap above **automates** a hand-dance. Derivation **deletes** +it. A `DATABASE_URL` is not a secret anybody authored — it is a fact about a +resource cast itself created, readable from the API that created it, at any time, +for free. So a template can say so directly: + + DATABASE_URL=${resource:postgres.url} + REDIS_URL=${resource:redis.url} + +`${resource:.url}` resolves to the **internal** URL of a database the same +manifest environment declares (`connect_to_docker_network` puts apps on the +Docker network, so it is the internal URL, not the external one). The value is +read back from the live resource's `internal_db_url` — the same field, on the +same `GET /projects/{uuid}/{env}` route, that `capture --generated-only` reads — +**never** stored in the age store, **never** decrypted, **never** printed. + +**It is not a secret ref.** `parseTemplate` classifies `${resource:…}` as a +distinct *derived* var, so `capture` never goes looking for a store name called +`resource:postgres.url`, and `templateRefs` (the required-secret set) never lists +it. The age store shrinks to the things a human actually authored. + +**Resolved in two places, one function.** `fillDerivedEnv` is the only code that +turns a ref into a value, and it runs twice against two different URL maps: + +- **at diff/apply time**, against the databases that **already exist** on the + box. On a re-apply this is the whole story: the app's live `DATABASE_URL` + already equals its database's URL, so the derived var shows **no drift** — + which is what deletes the `secret DATABASE_URL differs` line the store-copy + approach printed on every plan. And if the two have diverged (a password + rotated in Coolify), the diff shows it and `apply` **follows** the live + database, rather than reverting it to a stale stored copy. +- **at apply time in the executor**, against a database this same run just + created. On a from-nothing apply nothing existed to resolve against at plan + time, so the ref rides through the diff unresolved (rendered `DATABASE_URL: + derived from database postgres — apply will set it`) and `syncEnv` fills it + after the create. `apply` acts databases-before-applications (see *Apply acts + in dependency order*), so the database exists by the time the app's env is + written. + +**The unresolved sentinel is never written.** Until it resolves, a derived var +carries a sentinel that is not a legal value; the executor **refuses** to write +one that never resolved, rather than writing a blank — an empty `DATABASE_URL` +boots every consumer pointed at nothing. Coolify mints a database's credentials +at create time and `internal_db_url` is a model accessor built from them (not +from a running container), so the URL is expected the moment the create returns; +if a given Coolify only publishes it once the container is up, the refusal names +the app and the database and says to re-run once it is up — and the second run +resolves it as an ordinary update, because by then the database is live and the +diff fills it. It is graceful either way, and single-pass in the expected one. + +**Validation is at plan time, in the same voice as the dead-`generated_secrets` +check.** A `${resource:X.url}` naming a database the manifest does not declare, or +an attribute other than `.url`, is a hard error before any write — refused by +every verb that opens a template (`apply`, `diff`, `capture`), because a ref that +resolves against nothing is broken for all of them, not just the one about to +write. + +**Scope.** This covers only databases cast itself declares and creates. It does +not touch a service that builds its own URL internally from magic vars against +its *own* bundled database (Coolify's umami is the example): there is no edge for +the manifest to declare there, and a value of cast's would never be read. +`generated_secrets:` and the two-pass bootstrap above **remain** for the residual +class — a provider-generated value that is genuinely not derivable (a service's +own generated credential). What leaves is `DATABASE_URL` / `REDIS_URL`: they stop +being store names at all, so the placeholder they held, and the `apply` refusal +that guarded it, simply cease to exist for them — there is nothing in the store +to overwrite, and nothing to re-encrypt on a rebuild-from-nothing. + ## Drafts (`inventory --emit-draft`) `inventory` with no repo sweeps an instance. `--emit-draft ` writes that diff --git a/src/cli.ts b/src/cli.ts index e358d75..b2098cb 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -66,7 +66,12 @@ import { renderNoRecipient, renderRepoWithDraft, } from "./draft.js"; -import { assertEnvVarPolicy } from "./envtemplate.js"; +import { + type ResolvedEnv, + assertEnvVarPolicy, + fillDerivedEnv, + unresolvedDerived, +} from "./envtemplate.js"; import { type ProjectOutcome, fleetConflict, @@ -89,6 +94,7 @@ import { assertNoReservedEnvNames, reservedHits } from "./reserved.js"; import { PATH_IN_PROD_REFUSAL, desiredFromManifest, + fillDesiredDerived, manifestResources, refusesPathInProd, requiredSecrets, @@ -517,6 +523,16 @@ export async function fetchLive( // the `destination` relation and no endpoint exposes it. See Placement. destinationId: typeof i.destination_id === "number" ? i.destination_id : undefined, + // The URL an application's ${resource:.url} derives from (#60). Only + // databases carry `internal_db_url` (an appended model attribute on + // StandalonePostgresql/StandaloneRedis — see fetchGeneratedSources for the + // provenance); it rides on Live rather than in `fields` because it is never + // a database field cast writes or diffs. aliasLive preserves it while + // renaming to the manifest's vocabulary, so the URL map built downstream is + // keyed by the name a ref actually uses. + ...(kind === "database" && typeof i.internal_db_url === "string" + ? { internalDbUrl: i.internal_db_url } + : {}), })); const live = [ ...map("application", env.applications), @@ -1035,6 +1051,21 @@ async function runProject( l.env = await fetchEnv(ctx.client, l); } } + // Resolve ${resource:.url} against the databases that ALREADY exist on + // the box (#60). On a re-apply this is the whole story — the app's live + // DATABASE_URL already equals its database's URL, so the derived var shows no + // drift, which is what stops the "secret DATABASE_URL differs" noise that ran + // on every plan. On a from-nothing apply the databases are not here yet, so + // their refs stay unresolved through the diff (rendered "apply will set it") + // and the executor fills them after it creates the databases. Keyed by manifest + // name: aliasLive has already renamed live resources, and internalDbUrl rode + // along (see fetchLive). + const resourceUrls = Object.fromEntries( + live + .filter((l) => l.kind === "database" && l.internalDbUrl) + .map((l) => [l.name, l.internalDbUrl as string]), + ); + desired = fillDesiredDerived(desired, resourceUrls); const report = computeDiff(desired, live, ctx.mode, { declaredDestination: projectBinding?.destination_uuid, }); @@ -2992,6 +3023,48 @@ export function buildExecutor( } await client.post(`/databases/${dbUuid}/backups`, body); }; + // Resolve any ${resource:.url} still unresolved when apply is about to + // write an env (#60). It can only still be unresolved on a from-nothing run: + // runProject filled every ref whose database already existed at plan time, so + // what is left is a database THIS apply created moments ago (apply acts + // databases-before-applications, #45, so it exists by now). Read its URL back + // from the same environment_details route `capture --generated-only` uses, and + // key by name — a from-nothing box has no aliases, so the live name IS the + // manifest name the ref carries. + // + // Refuses to write a ref that resolved to nothing: an empty DATABASE_URL boots + // the app pointed at nothing, the exact fill fetchGeneratedSources forbids. + // Coolify mints a database's credentials at CREATE time and internal_db_url is + // a model accessor built from them (not from a running container), so the URL + // is expected the moment the create returns. If a given Coolify build only + // populates it once the container is up, this refuses with a re-run instruction + // rather than writing a blank — and the re-run resolves it as an ordinary + // update, because by then the database is live and the diff fills it. + const resolveDerivedEnv = async (env: ResolvedEnv): Promise => { + if (unresolvedDerived(env).length === 0) return env; + const { sources } = await fetchGeneratedSources( + client, + ctx.projectName, + ctx.envName, + ); + const urls = Object.fromEntries(sources.map((s) => [s.resource, s.url])); + const filled = fillDerivedEnv(env, urls); + const missing = unresolvedDerived(filled); + if (missing.length > 0) { + throw new Error( + [ + "cannot resolve derived env var(s) after creating the database:", + ...missing.map((m) => ` ${m.key} — from database ${m.resource}`), + "", + "cast created the database this run, but Coolify has not published its", + "internal URL yet (the resource may still be starting). Nothing was written", + "— an empty URL would boot the app pointed at nothing. Re-run `cast apply`", + "once the database is up; the second run resolves it as an ordinary update.", + ].join("\n"), + ); + } + return filled; + }; // The two instance-wide constraints a create can die on, both of them invisible // from cast's project-scoped view, both of them arriving at the FIRST create — // after apply has already made the project and the environment. One wrapper, and @@ -3136,6 +3209,12 @@ export function buildExecutor( } }, async syncEnv(uuid, kind, env) { + // Fill any ${resource:.url} still carrying the unresolved sentinel + // before anything is written — the from-nothing case, where the database + // was created earlier in this same apply (#60). A no-op read-wise for an + // env with no derived vars (the common case), and for one already resolved + // at plan time. + const resolved = await resolveDerivedEnv(env); // The reserved-name rule at the wire (reserved.ts). Nothing can reach here // carrying one — resolve.ts refuses the manifest long before a diff, let // alone an apply — and the check is here anyway, because this is the single @@ -3144,7 +3223,7 @@ export function buildExecutor( // caller of buildExecutor will not have read resolve.ts; the guard it needs // is the one standing where the write happens. assertNoReservedEnvNames( - reservedHits(`${kind} ${uuid}`, Object.keys(env.vars)), + reservedHits(`${kind} ${uuid}`, Object.keys(resolved.vars)), ); // Bulk env update is an UPSERT of listed keys, not a full replace — // verified against app/Http/Controllers/Api/{Applications,Databases, @@ -3160,7 +3239,7 @@ export function buildExecutor( ? "databases" : "services"; await client.patch(`/${base}/${uuid}/envs/bulk`, { - data: Object.entries(env.vars).map(([key, v]) => ({ + data: Object.entries(resolved.vars).map(([key, v]) => ({ key, value: v.value, is_buildtime: false, diff --git a/src/diff.ts b/src/diff.ts index 82e1bdd..b7cd8c1 100644 --- a/src/diff.ts +++ b/src/diff.ts @@ -36,6 +36,12 @@ export type Live = { // drift nor a clean bill; it produces a line on the report. computeDiff is // where that is enforced. backupNotCompared?: string; + // The internal URL Coolify minted for this database (`internal_db_url`), when + // it is a database and the read carried one. NOT in `fields`: it is never + // written or compared as a database field — it is what an APPLICATION's + // ${resource:.url} derives from (#60). Absent on applications/services, + // and on a database whose URL the read could not see. + internalDbUrl?: string; }; export type FieldDiff = { field: string; @@ -52,6 +58,13 @@ export type EnvDiff = { // Coolify already did. See diffEnv, renderDiff and applyPlan's refusal. state: "add" | "change" | "remove-candidate" | "placeholder-conflict"; secret: boolean; + // Set to the RESOURCE NAME this var's value is derived from — its + // ${resource:.url} — when it is a derived value rather than an authored + // one. Rendered as "derived from database " rather than "secret differs", + // so a routine URL change (a rotation the derivation is meant to follow) never + // reads as an unexplained secret drift. Still `secret`, so the value itself is + // never printed either way (#60). + derived?: string; }; export type Change = { kind: ResourceKind; @@ -130,7 +143,10 @@ function diffEnv( ): EnvDiff[] { const diffs: EnvDiff[] = []; for (const [key, v] of Object.entries(desired.vars)) { - if (!(key in live)) diffs.push({ key, state: "add", secret: v.secret }); + const derived = + v.derived !== undefined ? { derived: v.derived.resource } : {}; + if (!(key in live)) + diffs.push({ key, state: "add", secret: v.secret, ...derived }); else if (live[key] !== v.value) { // The second pass of the two-pass bootstrap, which for years only ever // ran once. The store's value for a provider-generated secret is the @@ -166,6 +182,7 @@ function diffEnv( key, state: placeheld ? "placeholder-conflict" : "change", secret: v.secret, + ...derived, }); } } @@ -246,6 +263,9 @@ export function computeDiff( key, state: "add" as const, secret: v.secret, + ...(v.derived !== undefined + ? { derived: v.derived.resource } + : {}), })) : [], }); @@ -361,6 +381,18 @@ export function renderDiff(report: DiffReport): string { lines.push( ` secret ${e.key}: store holds the generated-secret PLACEHOLDER, live holds a real value — apply would OVERWRITE it`, ); + // A derived value, said in words that are not a rotation's: it is not a + // secret that "differs", it is a URL cast reads back from the database it + // created and keeps the app pointed at. `add` = the app does not carry it + // yet (a first apply, or a database made this run); `change` = the live + // value has drifted from the database's current URL and apply will follow + // it. Never the value — same rule as any secret. + else if (e.derived) + lines.push( + e.state === "add" + ? ` ${e.key}: derived from database ${e.derived} — apply will set it` + : ` ${e.key}: derived from database ${e.derived} — live differs, apply will follow it`, + ); else if (e.secret) lines.push(` secret ${e.key} differs`); else lines.push(` env ${e.key}: ${e.state}`); } diff --git a/src/envtemplate.ts b/src/envtemplate.ts index 7e3ba9f..e4fa47a 100644 --- a/src/envtemplate.ts +++ b/src/envtemplate.ts @@ -1,10 +1,48 @@ +// A ${resource:.url} reference — the internal URL of a database THIS +// manifest declares, read back from the resource Coolify created. It is not a +// secret: a secret is something a human authored and the age store holds, while +// this is a FACT about a resource cast itself made, readable from the API that +// made it, at any time, for free (#60). So it never enters the store, never +// enters a terminal, and needs no age key to read back — it is resolved from the +// live box, not decrypted. +export type ResourceRef = { resource: string; attr: string }; + +// The value a derived var carries until it is resolved against the live +// resource. NEVER a legal thing to WRITE — the executor refuses it, the same +// fail-closed shape as the generated-secret placeholder — because a database URL +// that resolved to nothing boots every consumer pointed at nothing (the exact +// trap fetchGeneratedSources guards with its `never a fill of ""` rule). +// +// Collision-safe by construction, not by exotic bytes: this string only ever +// occupies a `derived` var's value, and such a value is only ever this sentinel +// or a real `postgres://…`/`redis://…` URL that fillDerivedEnv put there — so it +// cannot be mistaken for a resolved value, and `unresolvedDerived` gates on the +// `derived` flag anyway. (A plain string, no NUL byte: a NUL makes git treat the +// whole source file as binary and its diff unreviewable.) +export const DERIVED_UNRESOLVED = "cast:unresolved-derived-resource-url"; + +// `secret` is true for both a ${SECRET} and a resolved ${resource:…} — both are +// values that must never be printed. `derived` is set (and stays set after +// resolution) so the diff can say "derived from database X" rather than mistaking +// a routine URL change for a secret rotation, and so the executor knows which +// vars to resolve from the live resource before it writes. export type ResolvedEnv = { - vars: Record; + vars: Record< + string, + { value: string; secret: boolean; derived?: ResourceRef } + >; }; -// A template line, parsed but not resolved: `ref` is set when the whole RHS is -// a single ${NAME} placeholder. -export type TemplateVar = { key: string; rhs: string; ref?: string }; +// A template line, parsed but not resolved: `ref` is set when the whole RHS is a +// single ${NAME} placeholder (a store secret); `resourceRef` when it is a single +// ${resource:.attr} placeholder (a derived value). The two are mutually +// exclusive — a secret name is UPPER_SNAKE, a resource ref starts `resource:`. +export type TemplateVar = { + key: string; + rhs: string; + ref?: string; + resourceRef?: ResourceRef; +}; // ONE grammar, shared by both readers of a template — resolveTemplate (which // needs the values) and templateRefs (which needs only the names). Keeping @@ -25,7 +63,22 @@ function parseTemplate(text: string): TemplateVar[] { ); const [, key, rhs] = m; const placeholder = rhs.match(/^\$\{([A-Z][A-Z0-9_]*)\}$/); - vars.push({ key, rhs, ...(placeholder ? { ref: placeholder[1] } : {}) }); + // A resource ref names a Coolify resource (lower-kebab/snake, as the manifest + // and Coolify spell it) and one dotted attribute. Only `.url` is derivable + // today; an unknown attr is not rejected HERE — parsing stays dumb — but at + // validation time (resolve.ts), where the message can name the declared + // resources, exactly as an undeclared-resource ref is. + const resource = rhs.match( + /^\$\{resource:([a-z0-9][a-z0-9_-]*)\.([a-z_]+)\}$/, + ); + vars.push({ + key, + rhs, + ...(placeholder ? { ref: placeholder[1] } : {}), + ...(resource + ? { resourceRef: { resource: resource[1], attr: resource[2] } } + : {}), + }); } return vars; } @@ -35,7 +88,22 @@ export function resolveTemplate( secrets: Record, ): ResolvedEnv { const vars: ResolvedEnv["vars"] = {}; - for (const { key, rhs, ref } of parseTemplate(text)) { + for (const { key, rhs, ref, resourceRef } of parseTemplate(text)) { + // A derived value is resolved from the live resource, not the store, and not + // here — the resource may not exist yet (a from-nothing apply creates it in + // the same run). So it lands UNRESOLVED, carrying the ref; fillDerivedEnv + // fills it once a URL map is known, and the executor refuses to write one + // that never resolved. This is BEFORE the ref/store branch on purpose: a + // resource ref's RHS is not UPPER_SNAKE, so it would otherwise be mistaken + // for a literal and written through as the text "${resource:…}". + if (resourceRef) { + vars[key] = { + value: DERIVED_UNRESOLVED, + secret: true, + derived: resourceRef, + }; + continue; + } if (ref === undefined) { vars[key] = { value: rhs, secret: false }; continue; @@ -49,6 +117,49 @@ export function resolveTemplate( return { vars }; } +// Resolve every derived var whose resource now has a known URL, leaving the rest +// unresolved (a from-nothing apply has no live URL to resolve against until the +// database is created — the executor fills those, post-create). Pure, and the +// single place a derived value is turned into a real one, so the diff-time fill +// (against pre-existing live) and the apply-time fill (against a just-created +// resource) can never resolve the same ref two different ways. +// +// `urls` is keyed by the resource's MANIFEST name — the same name the ref +// carries. The caller keys it that way (see fillDesiredDerived / the executor): +// aliasing has already renamed live resources to the manifest's vocabulary by +// the time a URL map is built. +export function fillDerivedEnv( + env: ResolvedEnv, + urls: Record, +): ResolvedEnv { + const vars: ResolvedEnv["vars"] = {}; + for (const [key, v] of Object.entries(env.vars)) { + const url = + v.derived && v.derived.attr === "url" + ? urls[v.derived.resource] + : undefined; + vars[key] = + url !== undefined && url !== "" + ? { value: url, secret: true, derived: v.derived } + : v; + } + return { vars }; +} + +// The derived vars still holding the unresolved sentinel — what the executor +// must resolve from the live box before it can write this env, and what it +// refuses on if it cannot. Names the env key and the resource it derives from; +// never a value (there is none yet, and there never will be one to print). +export function unresolvedDerived( + env: ResolvedEnv, +): Array<{ key: string; resource: string }> { + return Object.entries(env.vars).flatMap(([key, v]) => + v.derived && v.value === DERIVED_UNRESOLVED + ? [{ key, resource: v.derived.resource }] + : [], + ); +} + // The ${NAME} refs a template declares: the secret names the manifest requires, // paired with the env var each one lands on. `capture` reads these to learn // what to go and fetch — at capture time there is no store to resolve against @@ -61,6 +172,21 @@ export function templateRefs( ); } +// The ${resource:.attr} refs a template declares — the derived edges. NOT +// returned by templateRefs above, and that separation is the point: a derived +// value is not a secret to be captured, so `capture` must never go looking for a +// store name called `resource:postgres.url`. resolve.ts reads these to validate +// each edge against the databases the manifest actually declares. +export function templateResourceRefs( + text: string, +): Array<{ key: string; resource: string; attr: string }> { + return parseTemplate(text).flatMap(({ key, resourceRef }) => + resourceRef === undefined + ? [] + : [{ key, resource: resourceRef.resource, attr: resourceRef.attr }], + ); +} + // Every env var key a template declares — refs and literals alike. `capture` // only cares about the ${...} refs (the secrets); `inventory` needs all of them, // because the question it answers is "what does the manifest put on this diff --git a/src/resolve.ts b/src/resolve.ts index 103bd26..c71dd64 100644 --- a/src/resolve.ts +++ b/src/resolve.ts @@ -5,9 +5,11 @@ import { join } from "node:path"; import type { Desired } from "./diff.js"; import { type ResolvedEnv, + fillDerivedEnv, resolveTemplate, templateKeys, templateRefs, + templateResourceRefs, } from "./envtemplate.js"; import { loadManifest } from "./manifest.js"; import { @@ -205,6 +207,51 @@ export function resolveCheckout( // store is keyed by REF, but the live box knows it as `resource.key`. export type RequiredSecret = { ref: string; resource: string; key: string }; +// The dead-reference check, pointed at derived edges instead of generated +// secrets (#60). The same failure the generated_secrets check below catches — a +// name that resolves against nothing, dressed up as config that guards or +// derives something — and refused in the same voice, wherever a template is +// opened: `apply`, `diff`, and `capture` all validate before they act, because a +// `${resource:X.url}` naming a database the manifest does not declare is broken +// for all three, not just the verb about to write. +// +// - an attr other than `.url`: nothing else is derivable, so it can only be a +// mistake — named here rather than resolved to `undefined` and written blank. +// - a resource the manifest does not declare: the URL would resolve against +// nothing on every box, forever. The likeliest cause is a typo. +function assertResourceRefs( + envName: string, + databases: Set, + refs: Array<{ key: string; resource: string; attr: string }>, +): void { + for (const r of refs) { + if (r.attr !== "url") { + throw new Error( + [ + `manifest environment ${envName}: ${r.key} refers to \${resource:${r.resource}.${r.attr}}, an unknown resource attribute`, + "", + "Only `.url` is derivable — the internal URL of a database the manifest", + "declares. Fix the attribute, or make it a plain ${SECRET} the store holds.", + ].join("\n"), + ); + } + if (!databases.has(r.resource)) { + throw new Error( + [ + `manifest environment ${envName}: ${r.key} refers to \${resource:${r.resource}.url}, but the manifest declares no database named ${r.resource}`, + "", + ` declares: ${[...databases].sort().join(", ") || "(no databases)"}`, + "", + "A ${resource:…} ref derives the URL of a database this manifest creates. One", + "that names a database the manifest does not declare derives nothing, on every", + "box, forever — the likeliest cause is a typo. Fix the name, or declare the", + "database.", + ].join("\n"), + ); + } + } +} + // Exactly the set of secret names an environment's manifest demands — the same // set `apply` will later insist on, read from the same templates by the same // parser. `capture` uses this to know what to go and fetch; nothing else has to @@ -224,6 +271,8 @@ export function requiredSecrets( ); } const required: RequiredSecret[] = []; + const resourceRefs: Array<{ key: string; resource: string; attr: string }> = + []; // Reserved names are checked HERE, and in manifestResources, and in // desiredFromManifest — every function in this file that opens an env // template, rather than once in the verb that writes. The rule is a property @@ -244,6 +293,7 @@ export function requiredSecrets( ); const text = readFileSync(file, "utf8"); reserved.push(...reservedHits(resource, templateKeys(text))); + resourceRefs.push(...templateResourceRefs(text)); for (const { key, ref } of templateRefs(text)) { required.push({ ref, resource, key }); } @@ -255,6 +305,11 @@ export function requiredSecrets( collect(name, svc.env_template); } assertNoReservedEnvNames(reserved); + assertResourceRefs( + envName, + new Set(Object.keys(envSpec.databases ?? {})), + resourceRefs, + ); const generated = envSpec.generated_secrets ?? []; // A generated_secrets entry naming something no template refs is dead // config — and dead config in THIS list is not merely untidy, it is @@ -352,6 +407,8 @@ export function desiredFromManifest( const desired: Desired[] = []; const resolvedEnvs: Record = {}; const reserved: ReservedHit[] = []; + const resourceRefs: Array<{ key: string; resource: string; attr: string }> = + []; const resolveEnvFile = ( name: string, template?: string, @@ -360,8 +417,10 @@ export function desiredFromManifest( const file = join(checkoutDir, ".infra", "env", template); if (!existsSync(file)) throw new Error(`env template missing: ${file} (referenced by ${name})`); - const env = resolveTemplate(readFileSync(file, "utf8"), secrets); + const text = readFileSync(file, "utf8"); + const env = resolveTemplate(text, secrets); reserved.push(...reservedHits(name, Object.keys(env.vars))); + resourceRefs.push(...templateResourceRefs(text)); resolvedEnvs[name] = env; return env; }; @@ -511,5 +570,28 @@ export function desiredFromManifest( // resolved env that carries a reserved name is not desired state, it is a // suppression of the platform's own value dressed up as one. See reserved.ts. assertNoReservedEnvNames(reserved); + assertResourceRefs( + envName, + new Set(Object.keys(envSpec.databases ?? {})), + resourceRefs, + ); return { desired, resolvedEnvs }; } + +// Fill the derived vars in a desired set against a URL map keyed by MANIFEST +// resource name. Used twice, with two different maps, and that is the whole +// point of it being one function: diff/apply fills first against the resources +// that already exist on the box (so an app whose DATABASE_URL already equals the +// live database's URL shows no drift — the "differs on every plan" noise #60 +// deletes), and the executor fills again against a database it has just created +// (the from-nothing case, where nothing existed to resolve against at plan +// time). A ref whose resource is in neither map stays unresolved; the executor +// is the one place that refuses to WRITE one that never resolved. +export function fillDesiredDerived( + desired: Desired[], + urls: Record, +): Desired[] { + return desired.map((d) => + d.env ? { ...d, env: fillDerivedEnv(d.env, urls) } : d, + ); +} diff --git a/test/diff.test.ts b/test/diff.test.ts index 87b66a9..821d461 100644 --- a/test/diff.test.ts +++ b/test/diff.test.ts @@ -217,6 +217,66 @@ describe("computeDiff generated-secret placeholder", () => { }); }); +describe("derived resource refs (#60)", () => { + const DERIVED_URL = "postgres://u:p@pg-uuid:5432/widget"; + const derivedApp = { + kind: "application" as const, + name: "core-api", + fields: { build_pack: "nixpacks", domains: ["https://api.example.com"] }, + env: { + vars: { + DATABASE_URL: { + value: DERIVED_URL, + secret: true, + derived: { resource: "postgres", attr: "url" }, + }, + }, + }, + }; + const liveApp = (env: Record) => [ + { + kind: "application" as const, + name: "core-api", + uuid: "u1", + fields: { ...derivedApp.fields }, + env, + }, + ]; + + it("is clean when the app's live URL already equals the database's — no perpetual drift", () => { + const r = computeDiff( + [derivedApp], + liveApp({ DATABASE_URL: DERIVED_URL }), + "full", + ); + expect(r.clean).toBe(true); + }); + + it("renders a drift as derived — not as a secret rotation — and never prints the value", () => { + const out = renderDiff( + computeDiff( + [derivedApp], + liveApp({ DATABASE_URL: "postgres://old" }), + "full", + ), + ); + expect(out).toContain( + "DATABASE_URL: derived from database postgres — live differs, apply will follow it", + ); + expect(out).not.toContain("secret DATABASE_URL differs"); + expect(out).not.toContain(DERIVED_URL); + expect(out).not.toContain("postgres://old"); + }); + + it("renders a create as a derived add", () => { + const out = renderDiff(computeDiff([derivedApp], [], "full")); + expect(out).toContain( + "DATABASE_URL: derived from database postgres — apply will set it", + ); + expect(out).not.toContain(DERIVED_URL); + }); +}); + describe("renderDiff generated-secret placeholder", () => { it("says it in words no rotation prints, and never prints the live value", () => { const out = renderDiff( diff --git a/test/envtemplate.test.ts b/test/envtemplate.test.ts index 108c10b..78d9af4 100644 --- a/test/envtemplate.test.ts +++ b/test/envtemplate.test.ts @@ -1,5 +1,13 @@ import { describe, expect, it } from "vitest"; -import { assertEnvVarPolicy, resolveTemplate } from "../src/envtemplate.js"; +import { + DERIVED_UNRESOLVED, + assertEnvVarPolicy, + fillDerivedEnv, + resolveTemplate, + templateRefs, + templateResourceRefs, + unresolvedDerived, +} from "../src/envtemplate.js"; describe("resolveTemplate", () => { it("classifies literals as non-secret and ${…} as secret", () => { @@ -53,3 +61,56 @@ describe("assertEnvVarPolicy", () => { ).not.toThrow(); }); }); + +describe("derived resource refs (#60)", () => { + const template = + "PORT=3000\nMAILGUN_KEY=${MAILGUN_KEY}\nDATABASE_URL=${resource:postgres.url}\n"; + + it("resolveTemplate marks a ${resource:…} var derived and UNRESOLVED, not secret-missing", () => { + // No store entry for it, and yet it does not throw the way a missing secret + // does: a derived value is not in the store to be missing FROM. + const r = resolveTemplate(template, { MAILGUN_KEY: "mk" }); + expect(r.vars.DATABASE_URL).toEqual({ + value: DERIVED_UNRESOLVED, + secret: true, + derived: { resource: "postgres", attr: "url" }, + }); + // The secret and the literal are untouched by the new branch. + expect(r.vars.MAILGUN_KEY).toEqual({ value: "mk", secret: true }); + expect(r.vars.PORT).toEqual({ value: "3000", secret: false }); + }); + + it("templateResourceRefs reports the edge; templateRefs does NOT treat it as a secret", () => { + expect(templateResourceRefs(template)).toEqual([ + { key: "DATABASE_URL", resource: "postgres", attr: "url" }, + ]); + // capture reads templateRefs — a derived edge must never appear there, or it + // would go hunting for a store name called `resource:postgres.url`. + expect(templateRefs(template).map((r) => r.key)).toEqual(["MAILGUN_KEY"]); + }); + + it("fillDerivedEnv resolves against a URL map and leaves the rest alone", () => { + const env = resolveTemplate(template, { MAILGUN_KEY: "mk" }); + const filled = fillDerivedEnv(env, { + postgres: "postgres://u:p@uuid:5432/db", + }); + expect(filled.vars.DATABASE_URL).toEqual({ + value: "postgres://u:p@uuid:5432/db", + secret: true, + derived: { resource: "postgres", attr: "url" }, + }); + expect(unresolvedDerived(filled)).toEqual([]); + }); + + it("fillDerivedEnv leaves a ref whose resource is absent (or empty) unresolved", () => { + const env = resolveTemplate(template, { MAILGUN_KEY: "mk" }); + // Absent from the map, and present-but-empty, are both non-resolutions — an + // empty URL must never be written (it boots the app pointed at nothing). + expect(unresolvedDerived(fillDerivedEnv(env, {}))).toEqual([ + { key: "DATABASE_URL", resource: "postgres" }, + ]); + expect(unresolvedDerived(fillDerivedEnv(env, { postgres: "" }))).toEqual([ + { key: "DATABASE_URL", resource: "postgres" }, + ]); + }); +}); diff --git a/test/live-lookup.test.ts b/test/live-lookup.test.ts index 2bbb931..0e78a30 100644 --- a/test/live-lookup.test.ts +++ b/test/live-lookup.test.ts @@ -85,6 +85,35 @@ describe("fetchLive", () => { const r = await fetchLive(client, "incubator", "prod"); expect(r).toEqual({ found: true, live: [] }); }); + + // #60: a database carries its internal_db_url onto Live (what an app's + // ${resource:.url} derives from); an application does not. This is the + // plumbing runProject reads to build its URL map, so it is worth pinning. + it("plumbs a database's internal_db_url onto Live, and only for databases", async () => { + const client = coolify([{ uuid: "p1", name: "incubator" }], { + "p1/prod": { + applications: [ + { name: "core", uuid: "a1", internal_db_url: "nonsense" }, + ], + postgresqls: [ + { + name: "db", + uuid: "d1", + internal_db_url: "postgres://u:p@d1:5432/app", + }, + ], + redis: [{ name: "cache", uuid: "r1" }], + }, + }); + const r = await fetchLive(client, "incubator", "prod"); + if (!r.found) throw new Error("unreachable"); + const byName = Object.fromEntries(r.live.map((l) => [l.name, l])); + expect(byName.db.internalDbUrl).toBe("postgres://u:p@d1:5432/app"); + // An application never carries it — even if the raw record has the key. + expect(byName.core.internalDbUrl).toBeUndefined(); + // A database whose read carried no URL simply has none (not ""). + expect(byName.cache.internalDbUrl).toBeUndefined(); + }); }); describe("renderAbsentTarget", () => { diff --git a/test/resolve.test.ts b/test/resolve.test.ts index 84ec09f..9b292a0 100644 --- a/test/resolve.test.ts +++ b/test/resolve.test.ts @@ -3,9 +3,12 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, it, vi } from "vitest"; import { computeDiff } from "../src/diff.js"; +import { DERIVED_UNRESOLVED } from "../src/envtemplate.js"; import { cloneFailureMessage, desiredFromManifest, + fillDesiredDerived, + requiredSecrets, resolveCheckout, resolveGitAuth, } from "../src/resolve.js"; @@ -472,3 +475,81 @@ environments: ); }); }); + +describe("derived resource refs (#60)", () => { + // A manifest with one app whose template derives a DB URL, plus the database + // the ref names. `dbName` and `attr` are knobs the validation cases turn. + const write = ( + ref = "${resource:postgres.url}", + dbBlock = " databases:\n postgres: { type: postgresql }\n", + ): string => { + const dir = mkdtempSync(join(tmpdir(), "infra-co-")); + mkdirSync(join(dir, ".infra", "env"), { recursive: true }); + writeFileSync( + join(dir, ".infra", "manifest.yaml"), + `project: widget +environments: + staging: + applications: + core: + source: { repo: acme/widget, branch: main } + build: { pack: nixpacks, base_directory: /apps/core } + domains: ["http://api.example.com"] + env_template: core.staging.env.template +${dbBlock}`, + ); + writeFileSync( + join(dir, ".infra", "env", "core.staging.env.template"), + `DATABASE_URL=${ref}\n`, + ); + return dir; + }; + + it("emits a derived var (unresolved) and does not demand it as a secret", () => { + const dir = write(); + const { desired } = desiredFromManifest(dir, "staging", {}); + const app = desired.find((d) => d.name === "core"); + expect(app?.env?.vars.DATABASE_URL).toEqual({ + value: DERIVED_UNRESOLVED, + secret: true, + derived: { resource: "postgres", attr: "url" }, + }); + // capture's view: it is NOT a required store secret. + const req = requiredSecrets(dir, "staging"); + expect(req.required.map((r) => r.ref)).not.toContain( + "resource:postgres.url", + ); + expect(req.required).toHaveLength(0); + }); + + it("fillDesiredDerived fills it from a URL map keyed by manifest name", () => { + const dir = write(); + const { desired } = desiredFromManifest(dir, "staging", {}); + const filled = fillDesiredDerived(desired, { + postgres: "postgres://u:p@uuid:5432/db", + }); + const app = filled.find((d) => d.name === "core"); + expect(app?.env?.vars.DATABASE_URL.value).toBe( + "postgres://u:p@uuid:5432/db", + ); + }); + + it("hard-refuses a ref naming a database the manifest does not declare", () => { + // No databases block at all — the ref points at nothing. + const dir = write("${resource:postgres.url}", ""); + expect(() => desiredFromManifest(dir, "staging", {})).toThrow( + /no database named postgres/, + ); + // capture refuses it too, in the same voice — every verb that opens a template. + expect(() => requiredSecrets(dir, "staging")).toThrow( + /no database named postgres/, + ); + }); + + it("hard-refuses an attribute other than .url", () => { + const dir = write("${resource:postgres.password}"); + expect(() => desiredFromManifest(dir, "staging", {})).toThrow( + /unknown resource attribute/, + ); + }); +}); diff --git a/test/wire.test.ts b/test/wire.test.ts index 43ffa1d..aad03ba 100644 --- a/test/wire.test.ts +++ b/test/wire.test.ts @@ -10,6 +10,7 @@ import { } from "../src/cli.js"; import { CoolifyClient } from "../src/coolify.js"; import { computeDiff } from "../src/diff.js"; +import { DERIVED_UNRESOLVED } from "../src/envtemplate.js"; // Pure wire-translation helpers (Desired vocabulary <-> Coolify API // vocabulary). Importing src/cli.ts here must not run the CLI — see the @@ -1134,3 +1135,111 @@ describe("buildExecutor backup schedules", () => { ).rejects.toThrow(/no s3_destination UUID/); }); }); + +// The from-nothing half of derived resource URLs (#60): at plan time the +// database did not exist, so the app's ${resource:postgres.url} is still +// unresolved when apply goes to write its env. syncEnv reads the URL back from +// the environment_details route (the database was created earlier in the same +// apply) and writes the real value — or refuses, rather than writing a blank. +describe("buildExecutor syncEnv (derived resource URLs, #60)", () => { + const ctx = { + projectName: "widget", + envName: "prod", + serverUuid: "srv-1", + githubAppUuid: "gh-1", + serverName: "prod-box", + orgRepo: "acme/widget", + bindingEnv: "prod", + }; + const derivedEnv = { + vars: { + DATABASE_URL: { + value: DERIVED_UNRESOLVED, + secret: true, + derived: { resource: "postgres", attr: "url" as const }, + }, + }, + }; + const URL_ = "postgres://u:p@pg-uuid:5432/widget"; + + // Serves /projects and the environment_details route; records the envs/bulk + // write so a test can assert what value landed (or that none did). + function box(internalDbUrl?: string) { + const bulkBodies: unknown[] = []; + const fetchImpl = vi.fn(async (url: string | URL, init?: RequestInit) => { + const path = new URL(String(url)).pathname; + if (path === "/api/v1/projects") + return new Response( + JSON.stringify([{ uuid: "proj-1", name: "widget" }]), + { status: 200 }, + ); + if (path === "/api/v1/projects/proj-1/prod") + return new Response( + JSON.stringify({ + postgresqls: [ + internalDbUrl === undefined + ? { name: "postgres" } + : { name: "postgres", internal_db_url: internalDbUrl }, + ], + }), + { status: 200 }, + ); + if (path === "/api/v1/applications/app-1/envs/bulk") { + bulkBodies.push(JSON.parse(String(init?.body))); + return new Response(JSON.stringify([]), { status: 200 }); + } + return new Response("not found", { status: 404 }); + }) as unknown as typeof fetch; + return { fetchImpl, bulkBodies }; + } + + it("resolves the ref from the live database and writes the real URL", async () => { + const { fetchImpl, bulkBodies } = box(URL_); + const exec = buildExecutor( + new CoolifyClient("https://coolify.test", "tok", fetchImpl), + ctx, + ); + await exec.syncEnv("app-1", "application", derivedEnv); + expect(bulkBodies).toHaveLength(1); + expect(bulkBodies[0]).toEqual({ + data: [ + { + key: "DATABASE_URL", + value: URL_, + is_buildtime: false, + is_preview: false, + }, + ], + }); + }); + + it("refuses — and writes nothing — when the database has no URL yet", async () => { + const { fetchImpl, bulkBodies } = box(undefined); + const exec = buildExecutor( + new CoolifyClient("https://coolify.test", "tok", fetchImpl), + ctx, + ); + await expect( + exec.syncEnv("app-1", "application", derivedEnv), + ).rejects.toThrow(/cannot resolve derived env var/); + expect(bulkBodies).toHaveLength(0); + }); + + it("does not read the box at all when there is nothing derived to resolve", async () => { + const { fetchImpl, bulkBodies } = box(URL_); + const exec = buildExecutor( + new CoolifyClient("https://coolify.test", "tok", fetchImpl), + ctx, + ); + await exec.syncEnv("app-1", "application", { + vars: { PORT: { value: "3000", secret: false } }, + }); + // Straight to the write — no /projects lookup for a derived URL. + expect(bulkBodies).toHaveLength(1); + expect( + ( + fetchImpl as unknown as { mock: { calls: unknown[][] } } + ).mock.calls.some(([u]) => String(u).endsWith("/projects")), + ).toBe(false); + }); +});