diff --git a/CHANGELOG.md b/CHANGELOG.md index 447939b..83a1a93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,23 @@ actually cutting it, and this file starts there. contains an application; a manifest that does declare one still refuses on a missing binding exactly as before, clean plan or not, because that binding is state the next create will need. +- **A manifest with no `${…}` refs applies without a store** (#104) — the + greenfield manifest-first bootstrap was a chicken-and-egg with no exit, + found by the 2026-07-19 release drill against two fresh Coolify 4.1.2 + instances: a registered project whose manifest declared databases only + (zero `${…}` refs) could not take its first `apply` — apply refused with + `no secret store for / in `, and `capture`, the documented + way to get a store, rightly refuses a project that is absent on the box, + because apply is the verb that would create it. The drill unblocked with a + hand-rolled empty store (`printf '' | age -r … -o secrets/….env.age`), + documented nowhere. Now `diff`/`apply` gate that refusal on the manifest + actually *referencing* a secret, asked via the same parser resolution + uses: when the templates resolve zero `${…}` refs, an absent store is + treated as empty and the run proceeds, printing a loud one-line note + naming the path the store would live at — and since there is nothing to + decrypt, the age key is not demanded either. The moment any template + gains a `${…}` ref, the refusal returns byte-identical to before. + `capture` and `destroy` are untouched. ### Added diff --git a/README.md b/README.md index 81ab177..4f24bc8 100644 --- a/README.md +++ b/README.md @@ -431,6 +431,13 @@ apply-from-nothing, so every generated secret in every store is a placeholder again. This used to be a hand `age` re-encrypt against production, with the prod key in a process substitution. +**Greenfield needs zero passes** (#104): a manifest whose templates hold no +`${…}` refs at all — databases only, or apps whose env is pure literals — +applies from nothing. No store, no age key: `diff` and `apply` say out loud that +the store is absent and was not needed, and proceed. The store appears the first +time `capture` writes it, or the first time a template gains a placeholder — +from then on, an absent store refuses exactly as before. + `--generated-only` **inverts** capture's rule and changes nothing else: it fills the `generated_secrets` names and leaves every other name in the store **exactly as it is, byte for byte** — never re-read from the box, so a secret you rotated by diff --git a/src/cli.ts b/src/cli.ts index db86494..dd280b7 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1158,7 +1158,24 @@ async function runProject( // store was never written is a project a fleet run cannot read — and under // --all the headline of this message is what the summary carries, so it has to // say which project and which file rather than "Command failed: age -d". - if (!existsSync(store)) { + // + // Unless nothing would ever be read from it (#104). A manifest whose templates + // hold no ${…} refs resolves against an absent store exactly as it would + // against an empty one — and the greenfield first apply (fresh box, databases + // only, no secret authored yet) is precisely that manifest. Refusing there is + // a chicken-and-egg with no exit: apply demands a store, and capture — the + // documented way to get one — rightly refuses a project that does not exist on + // the box yet, because apply is the verb that would create it. So the refusal + // is gated on the manifest actually REFERENCING a secret, asked via + // requiredSecrets — the same parser resolution itself uses, so the two cannot + // disagree about what "no refs" means — and the zero-refs case proceeds on {} + // with a loud note. No store also means no decrypt, so the age key is not + // demanded either: nothing to open, nothing to protect yet. The store appears + // the first time capture writes it, or a template gains a placeholder. + let secrets: Record; + if (existsSync(store)) { + secrets = decryptSecrets(store, keyFileFor(ctx.envName)); + } else if (requiredSecrets(checkout, ctx.envName).required.length > 0) { throw new Error( [ `no secret store for ${orgRepo} in ${ctx.envName}`, @@ -1169,8 +1186,12 @@ async function runProject( "diff or apply without it. `cast capture` writes one from a live box.", ].join("\n"), ); + } else { + console.log( + `NOTE: no secret store for ${orgRepo} in ${ctx.envName} (looked for ${store}) — and none is needed: this manifest's templates hold no \${…} refs, so cast proceeds without a store or an age key. The store appears the first time capture writes it, or a template gains a placeholder.`, + ); + secrets = {}; } - const secrets = decryptSecrets(store, keyFileFor(ctx.envName)); let { desired, resolvedEnvs } = desiredFromManifest( checkout, ctx.envName, diff --git a/test/fleet-cli.test.ts b/test/fleet-cli.test.ts index 01ea316..2f68941 100644 --- a/test/fleet-cli.test.ts +++ b/test/fleet-cli.test.ts @@ -114,7 +114,7 @@ afterEach(async () => { await Promise.all(stubs.splice(0).map((s) => s.close())); }); -const manifest = (repo: string) => `project: ${repo} +const manifest = (repo: string, withRef = false) => `project: ${repo} environments: staging: applications: @@ -122,20 +122,33 @@ environments: source: { repo: heavy-duty/${repo}, branch: main } build: { pack: nixpacks, base_directory: / } domains: ["http://${repo}.example.com"] -`; +${withRef ? " env_template: core.env\n" : ""}`; // A state dir + three clonable product repos. `registry` is the knob: which // slugs the `projects:` block registers for staging — undefined writes no // `projects:` block at all (a state file from before the registry existed). +// `refIn` gives ONE repo a template with a ${…} ref: since #104 gated the +// missing-store refusal on the manifest actually referencing a secret, a +// fixture that wants to exercise that refusal has to reference one. function fixture( url: string, - opts: { registry?: string[]; registryEnv?: string } = {}, + opts: { registry?: string[]; registryEnv?: string; refIn?: string } = {}, ) { const root = mkdtempSync(join(tmpdir(), "cast-fleet-")); for (const repo of REPOS) { const dir = join(root, "repos", "heavy-duty", `${repo}.git`); mkdirSync(join(dir, ".infra"), { recursive: true }); - writeFileSync(join(dir, ".infra", "manifest.yaml"), manifest(repo)); + writeFileSync( + join(dir, ".infra", "manifest.yaml"), + manifest(repo, repo === opts.refIn), + ); + if (repo === opts.refIn) { + mkdirSync(join(dir, ".infra", "env")); + writeFileSync( + join(dir, ".infra", "env", "core.env"), + "API_KEY=${API_KEY}\n", + ); + } const git = (...args: string[]) => execFileSync("git", args, { cwd: dir, stdio: "pipe" }); git("init", "-q"); @@ -157,9 +170,10 @@ function fixture( `COOLIFY_BASE_URL="${url}"\nCOOLIFY_ACCESS_TOKEN="t"\n`, ); for (const repo of REPOS) { - // No template refs a secret, but the store still has to exist and open — - // a project whose store is missing is a project cast cannot read, which is - // a fleet ERROR, not a fleet skip (asserted below). + // No template refs a secret, so since #104 none of these stores is strictly + // required — but the pre-greenfield shape (store on disk, key injected) is + // the shape most fleets are in, and it has to keep working unchanged. The + // missing-store-with-a-ref refusal is exercised below by deleting one. execFileSync("age", ["-r", recipient, "-o", `${repo}.staging.env.age`], { input: "\n", cwd: join(state, "secrets"), @@ -292,8 +306,11 @@ describe("cast diff --all (#26)", () => { expect(r.output).toContain("500"); }); - it("treats a missing secret store as unreachable, naming the file", async () => { - const f = fixture((await stubCoolify()).url); + // `refIn` matters: since #104 the refusal is gated on the manifest actually + // referencing a secret, so the missing store is only an error because beta's + // template holds a ${…} ref. The gate's other half is the test after this one. + it("treats a missing secret store as unreachable when a template refs a secret, naming the file", async () => { + const f = fixture((await stubCoolify()).url, { refIn: "beta" }); execFileSync("rm", [join(f.state, "secrets", "beta.staging.env.age")]); const r = await run("diff", fleet(f), f); expect(r.code).toBe(2); @@ -302,6 +319,22 @@ describe("cast diff --all (#26)", () => { ); expect(r.output).toContain("UNREACHABLE: 1 heavy-duty/beta"); }); + + // The greenfield gate (#104) under --all: a zero-refs project whose store was + // never written is READ, not failed — the note prints, the fleet stays whole. + it("reads a zero-refs project with no store, noting the absence instead of failing it", async () => { + const f = fixture((await stubCoolify()).url); + execFileSync("rm", [join(f.state, "secrets", "beta.staging.env.age")]); + const r = await run("diff", fleet(f), f); + expect(r.code).toBe(0); + expect(r.output).toContain( + "NOTE: no secret store for heavy-duty/beta in staging", + ); + expect(r.output).toContain("read: 3 of 3"); + expect(r.output).toContain( + "all 3 registered project(s) were read, and every one is clean.", + ); + }); }); describe("cast diff --all — an empty fleet is not a clean fleet", () => { diff --git a/test/greenfield-cli.test.ts b/test/greenfield-cli.test.ts new file mode 100644 index 0000000..b395fbe --- /dev/null +++ b/test/greenfield-cli.test.ts @@ -0,0 +1,237 @@ +import { execFileSync, spawn } from "node:child_process"; +import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs"; +import { createServer } from "node:http"; +import type { AddressInfo } from "node:net"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { afterEach, beforeAll, describe, expect, it } from "vitest"; + +// The greenfield manifest-first bootstrap (#104), end to end: fresh box, +// registered project, a manifest that declares databases only and refs no +// secret — the exact shape the 2026-07-19 release drill hit. There used to be +// no path to the first apply: `apply` refused on the absent store, `capture` +// rightly refused the absent project, and the drill unblocked with a hand-made +// empty store nothing documented. +// +// The rule under test: the "no secret store" refusal is gated on the manifest +// actually REFERENCING a secret. Zero ${…} refs → an absent store is treated +// as empty, a loud note names the path, and no age key is demanded (nothing to +// decrypt, nothing to protect yet). One ${…} ref → the refusal, byte-identical +// to what it always said. A present store keeps decrypting exactly as before. + +let recipient: string; +let keyFile: string; + +beforeAll(() => { + const dir = mkdtempSync(join(tmpdir(), "cast-age-")); + keyFile = join(dir, "age.key"); + execFileSync("age-keygen", ["-o", keyFile], { stdio: "pipe" }); + recipient = execFileSync("age-keygen", ["-y", keyFile], { + encoding: "utf8", + }).trim(); +}); + +type Stub = { url: string; close: () => Promise }; +const stubs: Stub[] = []; + +// A fresh box: the project is registered on Coolify but its environment holds +// nothing — the state one API call after `cast project create`, and the state +// the drill's first apply ran against. +async function stubCoolify(): Promise { + const server = createServer((req, res) => { + const path = (req.url ?? "").replace("/api/v1", ""); + const json = (body: unknown) => { + res.writeHead(200, { "content-type": "application/json" }); + res.end(JSON.stringify(body)); + }; + if (path === "/teams/current") return json({ id: 0, name: "Root Team" }); + if (path === "/projects") return json([{ uuid: "p1", name: "fresh" }]); + if (path === "/projects/p1/staging") return json({ applications: [] }); + res.writeHead(404); + res.end("{}"); + }); + await new Promise((r) => { + server.listen(0, "127.0.0.1", r); + }); + const stub: Stub = { + url: `http://127.0.0.1:${(server.address() as AddressInfo).port}`, + close: () => + new Promise((r) => { + server.close(() => r()); + }), + }; + stubs.push(stub); + return stub; +} + +afterEach(async () => { + await Promise.all(stubs.splice(0).map((s) => s.close())); +}); + +// The drill's manifest, minimized: databases only, zero ${…} refs anywhere. +const ZERO_REFS_MANIFEST = `project: fresh +environments: + staging: + applications: {} + databases: + fresh-db: + type: postgresql +`; + +// The same environment the moment one template gains a placeholder — the +// boundary at which the old refusal must return, word for word. +const ONE_REF_MANIFEST = `project: fresh +environments: + staging: + applications: + core: + source: { repo: heavy-duty/fresh, branch: main } + build: { pack: nixpacks, base_directory: / } + domains: ["http://core.example.com"] + env_template: core.env + databases: + fresh-db: + type: postgresql +`; + +function fixture( + url: string, + opts: { manifest: string; store?: boolean } = { + manifest: ZERO_REFS_MANIFEST, + }, +) { + const checkout = mkdtempSync(join(tmpdir(), "cast-co-")); + mkdirSync(join(checkout, ".infra", "env"), { recursive: true }); + writeFileSync(join(checkout, ".infra", "manifest.yaml"), opts.manifest); + writeFileSync( + join(checkout, ".infra", "env", "core.env"), + "API_KEY=${API_KEY}\n", + ); + + const state = mkdtempSync(join(tmpdir(), "cast-state-")); + mkdirSync(join(state, "secrets")); + writeFileSync( + join(state, ".coolify.env"), + `COOLIFY_BASE_URL="${url}"\nCOOLIFY_ACCESS_TOKEN="t"\n`, + ); + if (opts.store) { + execFileSync("age", ["-r", recipient, "-o", "fresh.staging.env.age"], { + input: "\n", + cwd: join(state, "secrets"), + stdio: ["pipe", "pipe", "pipe"], + }); + } + writeFileSync( + join(state, "environments.yaml"), + [ + "environments:", + " staging:", + " server: fresh-box", + " team: { id: 0, name: Root Team }", + "github_apps:", + " fresh: hdb-coolify", + "", + ].join("\n"), + ); + return { checkout, state }; +} + +// `withKey: false` is the greenfield claim itself: the run is spawned with no +// CAST_AGE_KEY_FILE_STAGING and a HOME that holds no standing key, so if cast +// so much as ASKS for the age key, the run dies "no age key for staging" and +// the assertion on the output catches it. +function run( + args: string[], + opts: { withKey: boolean }, +): Promise<{ code: number; output: string }> { + return new Promise((resolve) => { + const { CAST_AGE_KEY_FILE_STAGING: _dropped, ...inherited } = process.env; + const env = opts.withKey + ? { ...inherited, CAST_AGE_KEY_FILE_STAGING: keyFile } + : { ...inherited, HOME: mkdtempSync(join(tmpdir(), "cast-home-")) }; + const child = spawn("node", ["dist/cli.js", "diff", ...args], { + stdio: ["pipe", "pipe", "pipe"], + env, + }); + let output = ""; + child.stdout.on("data", (d) => { + output += String(d); + }); + child.stderr.on("data", (d) => { + output += String(d); + }); + child.on("close", (code) => resolve({ code: code ?? 0, output })); + }); +} + +const base = (f: { checkout: string; state: string }) => [ + "heavy-duty/fresh", + "--env", + "staging", + "--path", + f.checkout, + "--state", + f.state, +]; + +describe("greenfield: zero ${…} refs and no store (#104)", () => { + it("proceeds to a full plan, says the store was absent and unneeded, and never asks for an age key", async () => { + const f = fixture((await stubCoolify()).url, { + manifest: ZERO_REFS_MANIFEST, + }); + const r = await run(base(f), { withKey: false }); + // The plan, not a refusal: the database is there to create, so this is an + // ordinary drift exit — which is the whole point, the first apply's plan. + expect(r.code).toBe(1); + expect(r.output).toContain("fresh-db"); + expect(r.output).toContain( + "NOTE: no secret store for heavy-duty/fresh in staging", + ); + expect(r.output).toContain( + join(f.state, "secrets", "fresh.staging.env.age"), + ); + expect(r.output).toContain("proceeds without a store or an age key"); + // The two ways the old behavior would have surfaced, both absent: the + // refusal (whose body, unlike the note, tells you what the store is FOR), + // and — with no key in the spawn env at all — the key demand. + expect(r.output).not.toContain("resolved from that store"); + expect(r.output).not.toContain("no age key for staging"); + }); + + it("keeps the original refusal, word for word, the moment a template holds a ${…} ref", async () => { + const f = fixture((await stubCoolify()).url, { + manifest: ONE_REF_MANIFEST, + }); + const r = await run(base(f), { withKey: false }); + // Exit 1 is what a single-project refusal has always exited with: the + // throw lands in main()'s rejection handler, same as before #104. The + // fleet flavor of this refusal (exit 2, UNREACHABLE) is fleet-cli.test.ts. + expect(r.code).toBe(1); + expect(r.output).toContain( + "no secret store for heavy-duty/fresh in staging", + ); + expect(r.output).toContain( + `looked for: ${join(f.state, "secrets", "fresh.staging.env.age")}`, + ); + expect(r.output).toContain( + "The manifest's ${…} refs are resolved from that store", + ); + expect(r.output).toContain("`cast capture` writes one from a live box."); + // The refusal, not the plan and not the note. + expect(r.output).not.toContain("NOTE:"); + expect(r.output).not.toContain("fresh-db"); + }); + + it("still opens a store that DOES exist — zero refs or not, a written store is decrypted as before", async () => { + const f = fixture((await stubCoolify()).url, { + manifest: ZERO_REFS_MANIFEST, + store: true, + }); + // The pre-#104 shape: store on disk, key injected. Same plan as the + // greenfield run, and no note — the store was there, so nothing to say. + const r = await run(base(f), { withKey: true }); + expect(r.code).toBe(1); + expect(r.output).toContain("fresh-db"); + expect(r.output).not.toContain("NOTE: no secret store"); + }); +});