fix clone auth (#13); named Coolify instances (#14); cast capture (#15) #16

Merged
dan-claude-bot merged 3 commits from fix/clone-auth-instances-capture into main 2026-07-13 16:54:16 +00:00
11 changed files with 1478 additions and 50 deletions
Showing only changes of commit 3eee70fa77 - Show all commits

117
README.md
View file

@ -52,6 +52,7 @@ environments.yaml # bindings: the team each env's token must belon
# GitHub App name, smoke target, guards
secrets/<repo>.<env>.env.age # age-encrypted values for the ${…} placeholders
.coolify.env # COOLIFY_BASE_URL + COOLIFY_ACCESS_TOKEN (never commit)
.coolify/<name>.env # …the same, for a NAMED instance (see below)
```
Pass it with `--state <dir>`, or set `CAST_STATE`. Defaults to the cwd.
@ -61,6 +62,7 @@ Pass it with `--state <dir>`, or set `CAST_STATE`. Defaults to the cwd.
```sh
cast apply <org>/<repo> --env <env> [--path <dir>] [--hostname-overlay <file>]
cast diff <org>/<repo> --env <env> [--full]
cast capture <org>/<repo> --env <env> [--generated <NAME>] [--override <NAME>]
cast server add <name> --ip <ip> --key <file> --env <env> [--user root] [--port 22]
cast smoke --env <env>
cast team [--env <env>]
@ -73,6 +75,9 @@ cast team [--env <env>]
default branch).
- **`diff`** — reports drift, manifest → Coolify. Structural by default; `--full`
also compares env vars. Exits non-zero when dirty, so CI can gate on it.
- **`capture`** — the adoption path: reads a hand-built instance's live env and
writes the environment's age store from it. See *Adopting a hand-built
instance* below.
- **`server add`** — uploads a server's private key and registers it with Coolify.
- **`smoke`** — contract test against `smoke_target`: proves Coolify's bulk env
endpoint still *upserts* rather than replacing. Run it after every Coolify
@ -89,6 +94,118 @@ them first asserts the token's team (below).
`--hostname-overlay` swaps domains for a pre-flight run against temporary
hostnames; re-applying **without** it is the cutover.
## Cloning: cast authenticates, and never prompts
`apply`, `diff` and `capture` clone the product repo (unless `--path` points at a
local checkout — refused for prod, which always reads the default branch). For a
private repo that needs credentials, and cast resolves them itself:
1. **`gh`**, borrowed as a credential helper for that one invocation — it does
not touch your global git config.
2. **`GITHUB_TOKEN` / `GH_TOKEN`** from the environment (the CI path).
3. Whatever git's own credential helper does, if you have one.
Being logged into `gh` is enough. You do **not** need `gh auth setup-git`
that separate act is what wires git's helper, and not running it is exactly how
you end up at git's interactive username/password prompt, which GitHub no longer
accepts. cast sets `GIT_TERMINAL_PROMPT=0` on every path, so it can never hang
there or hide a credentials failure behind an error about *the repository*. With
no credentials at all it says so, and names the fix.
The token is never put in the clone URL or in `http.extraheader` — both leak it
into `ps`, and the latter persists it into the clone's git config.
## Many Coolifys
`--instance <name>` reads `<state>/.coolify/<name>.env` instead of
`<state>/.coolify.env`. Every verb that reaches Coolify takes it.
```sh
cast diff heavy-duty/incubator --env prod --full --instance legacy
```
An environment can bind one, so `--env` selects the right control plane with no
flag at all:
```yaml
environments:
prod:
server: prod-box
team: { id: 1, name: heavy-duty }
instance: prod-cp # → <state>/.coolify/prod-cp.env
```
An explicit `--instance` still wins, so a one-off read against a legacy box needs
no edit to that file either. **With no flag and no binding, nothing changes**
`.coolify.env` is read exactly as before.
Two properties, both deliberate:
- **An unknown `--instance` refuses**, and names the instances that do exist.
Falling back to the default is how a diff meant for a legacy box gets run
against production.
- **An instance may declare `COOLIFY_READ_ONLY=true`**, and then `apply`,
`smoke` and `server add` refuse it — *before their first call*, and even
though the token itself would permit the writes. That turns "I pointed the
wrong token at the wrong box" from a live incident into an exit code.
Every command that reaches a Coolify now says which one, next to the team
assert. It is the most consequential input to any run, and the least visible.
## Adopting a hand-built instance
cast is otherwise scoped to the steady state: manifest → Coolify, forever.
`capture` is the one-way-in — it bootstraps an environment's age store from an
instance that was built by hand, before any manifest existed.
```sh
CAST_CAPTURE_ADMIN_EMAIL=me@example.com \
cast capture heavy-duty/incubator --env prod --instance legacy \
--override ADMIN_EMAIL
```
It reads the required secret **names** from the manifest's own env templates (the
`${…}` refs — the manifest already declares exactly this set), reads the live
values off the instance, and classifies every name:
| | |
| --- | --- |
| **captured** | found live, value taken |
| **generated** | the manifest's `generated_secrets` declares it provider-made → written as the literal `pending-coolify-generated`, never the live value |
| **overridden** | supplied by you, for a value that must *not* be carried over |
| **missing** | required by a template, absent live → **refuses** |
Then it prints a plan of **names and provenance — never values** — and waits for
you to type the environment's name.
The mapping is not mechanical, and that is the whole design. A `DATABASE_URL`
copied off the source box points at the *source box's* Postgres: confidently
wrong, entirely plausible, and the target's real URL does not exist until Coolify
creates the resource. So the manifest declares those names, and cast placeholds
them:
```yaml
environments:
prod:
generated_secrets: [DATABASE_URL_PROD, REDIS_URL_PROD, UMAMI_DATABASE_URL]
```
It is a manifest property rather than a flag you have to remember, because the
manifest is what knows `DATABASE_URL` comes from a database it declares. (A
`generated_secrets` entry no template refers to is a schema error — a guard
standing over nothing is worse than no guard, because it reads like one.
`--generated <NAME>` covers a manifest that hasn't declared them yet.)
An **`--override`**'s value is read from `$CAST_CAPTURE_<NAME>`, never from the
command line: argv is visible in `ps` to every process on the box. It exists for
values that must not survive the copy — staging and prod sharing a Mailgun
domain means a staging box carrying the real `ADMIN_EMAIL` can mail real users.
The store is encrypted to the environment's `age_recipient` (add it to
`environments.yaml` — it's the public half, safe to commit). Plaintext goes to
`age` on stdin: it is never a temp file, never on stdout, never in your shell
history. An existing store is not overwritten without `--force`.
**[docs/semantics.md](docs/semantics.md)** is the contract behind those
commands: what `apply` guarantees (never deletes, never recreates a database,
fails loudly rather than recreating on un-updatable drift), the `dockercompose`

View file

@ -158,6 +158,132 @@ softened by an implementation detail):
refuses `--path` combined with `--env prod`: prod always reads the default
branch, so a feature-branch checkout can never reach it.
## Instance selection
**The Coolify a command talks to is an explicit, named value** — not a property
of whatever `<state>/.coolify.env` happens to contain at the moment. Resolution
order, highest first:
1. `--instance <name>``<state>/.coolify/<name>.env`
2. the environment's `instance:` binding in `environments.yaml`
3. `<state>/.coolify.env` (the default; unchanged when neither of the above is
used)
Two refusals, both fail-closed:
- **An unknown `--instance` aborts**, naming the instances that do exist. It
does *not* fall back to the default — that fallback is how a `--full` diff
meant for a legacy box gets run against production.
- **`COOLIFY_READ_ONLY=true` in an instance file makes it read-only**, and
`apply` / `smoke` / `server add` refuse it *before their first call*. The
guard is the **declaration**, not the token's scope: an instance configured
for inspection must not be writable even when the token it holds would permit
the writes. `diff`, `team` and `capture` still work against it — they read.
Every command that reaches a live Coolify prints which one, next to the team
assert.
## Adoption (`capture`)
`capture` is the only verb that writes *into* the state directory rather than
into Coolify, and the only one that reads a hand-built instance as a **source**
rather than as a target. It exists because cast is otherwise scoped to the
steady state and has no bootstrap path for a box that predates its manifest.
**The required set comes from the manifest, not from the box.** The names are
the `${…}` refs in that environment's env templates, read by the same parser
`apply` uses to demand them (`parseTemplate`, shared by `resolveTemplate` and
`templateRefs` — deliberately one grammar, because a drift between the two
would mean `capture` collects a different set than `apply` will later require,
which is the "a name silently missed" failure it exists to remove). So the store
it writes contains **exactly** the names the manifest requires: a live var
nobody asked for is not the store's business, and a template literal
(`NODE_ENV=production`) is not a secret.
**The mapping is not mechanical, and must not be.** Some entries encode
migration decisions rather than facts about the source box:
- A `DATABASE_URL` / `REDIS_URL` read off the source points at the **source
box's** Postgres/Redis. Copying it is confidently wrong in a way that looks
entirely plausible, and the target's real URL does not exist until Coolify
creates the resource. These are declared `generated_secrets:` in the manifest
environment and written as the literal `pending-coolify-generated`.
- staging's `ADMIN_EMAIL` must be the operator, not the source's value: staging
and prod share a Mailgun domain, so a staging box carrying the real address
can mail real users. That is `--override`.
A "capture everything" verb would therefore be silently wrong in a handful of
entries out of seventeen — worse than being wrong in all of them. So every
required name is **forced into a disposition**, and two of the four stop the
run:
| disposition | source | outcome |
| --- | --- | --- |
| captured | found live | value taken |
| generated | manifest `generated_secrets` (or `--generated`) | `pending-coolify-generated` |
| overridden | `$CAST_CAPTURE_<NAME>` | operator's value |
| **missing** | required by a template, absent live | **refuses** |
| **conflict** | one name, different live values on two resources | **refuses** |
`generated_secrets` is a **manifest** property, not a flag: the manifest is what
knows `DATABASE_URL` comes from a database it declares. An entry naming
something no template refs is a hard error — dead config here is not untidy but
dangerous, because it reads like a guard standing over a name while standing
over nothing, and the likeliest cause is a typo whose real name is then
*captured* from the source box instead of placeheld.
**Secret hygiene**, all enforced by tests against real values:
- The plan prints **names and provenance, never values**. (The one value-shaped
thing it prints is the `pending-coolify-generated` literal, which carries no
information about the source.)
- An `--override`'s value is read from `$CAST_CAPTURE_<NAME>`, **never from
argv** — a command-line value is visible in `ps` to every process on the box.
- Plaintext is piped to `age` on **stdin**: never a temp file, never stdout,
never shell history. The hand-run recipe this replaces wrote
`/dev/shm/prod.env` and relied on remembering to `shred -u` it.
- An existing store is **not overwritten** without `--force`: it may hold the
only copy of values the source box no longer has. Same disposition as apply's
never-delete.
**`capture` takes `diff`'s position on an absent target** (see `LiveLookup`),
and refuses one: against a project or environment that isn't there it would read
back zero live values and report every required secret as *missing* — an
alarming, meaningless report about the wrong box. It also inherits the team
assert (a wrong-team token reads back `null` for everything, producing the same
lie) and the `--path`-with-`--env prod` refusal (a feature-branch manifest must
not decide which names land in the prod store).
The final gate is a **typed confirmation** — the environment's own name, after
the plan. There is no `--yes`: a store written without someone reading the
provenance column is the outcome the verb exists to prevent. A closed stdin
aborts rather than hanging.
## Cloning a private manifest
`resolveCheckout` resolves git credentials **inside cast**, in a fixed order —
`gh` borrowed as a per-invocation credential helper, then
`GITHUB_TOKEN`/`GH_TOKEN`, then the ambient helper — rather than leaving it to
whatever the workstation's git config happens to do.
It matters because `gh auth login` **does not** wire git's credential helper
(that is `gh auth setup-git`, a separate act most people never run), so a
perfectly logged-in operator still fell through to git's interactive
username/password prompt — which GitHub no longer accepts — and got an error
about *the repository* rather than about the missing credentials. There is no
routing around it for prod: `--path` is refused there, so the clone is the only
path and its auth is mandatory.
`GIT_TERMINAL_PROMPT=0` is set on every path, so cast can never hang on or fall
into that prompt. The token is never placed in the clone URL or in
`http.extraheader` — both leak it into `ps`, and the latter persists it into the
clone's `.git/config`; the helper reads it from the environment at run time, so
what lands in argv is the literal text `$CAST_GIT_TOKEN`. Note that the empty
`credential.helper=` reset clears **URL-scoped** helpers
(`credential.https://github.com.helper`, which is what `gh auth setup-git`
writes) as well as generic ones, so cast's chosen credential is genuinely the
one used — verified against a live private clone.
**Known limitations, not defects:**
- **Backup schedules are create-time only.** A manifest database's `backup`

View file

@ -47,6 +47,12 @@ const BindingsSchema = z
// An explicit --instance still wins, so a one-off read against a
// legacy box needs no change to this file either.
instance: z.string().optional(),
// The age recipient (public key) this environment's secret store is
// encrypted TO. Only `capture` needs it — decryption resolves an
// identity per keyFileFor, and the state repo deliberately holds
// ciphertext but never the identity that opens it. This is the
// public half, so it is safe to commit here next to the bindings.
age_recipient: z.string().optional(),
s3_destination: z.string().optional(),
// Var-name patterns this environment refuses outright (see
// assertEnvVarPolicy). Operator-owned guard: prod typically bans

189
src/capture.ts Normal file
View file

@ -0,0 +1,189 @@
import type { RequiredSecret } from "./resolve.js";
// What the manifest writes for a provider-generated name. Not the source box's
// value — that points at the SOURCE box's Postgres/Redis — and not an empty
// string, which would boot the app misconfigured. A literal that is obviously
// a placeholder, and that Coolify replaces when it creates the resource.
export const GENERATED_PLACEHOLDER = "pending-coolify-generated";
// Live env vars, per resource: resource name -> (env key -> value).
export type LiveEnvs = Record<string, Record<string, string>>;
export type Provenance = "captured" | "generated" | "overridden";
export type Site = { resource: string; key: string };
export type Disposition = {
ref: string;
provenance: Provenance;
// Never rendered. Kept here so the caller can encrypt it, and nowhere else.
value: string;
sites: Site[];
};
export type Classification = {
plan: Disposition[];
// Required by a template, absent from the source, and not dispositioned
// otherwise. Refuses the run: writing an empty value substitutes to nothing
// and the app boots misconfigured — the exact failure capture exists to
// remove, and one that looks entirely plausible from the outside.
missing: Array<{ ref: string; sites: Site[] }>;
// The same ref carrying DIFFERENT live values on two resources. cast cannot
// pick, and picking wrong is silent, so it refuses.
conflicts: Array<{ ref: string; values: Site[] }>;
};
function groupByRef(required: RequiredSecret[]): Map<string, Site[]> {
const byRef = new Map<string, Site[]>();
for (const { ref, resource, key } of required) {
const sites = byRef.get(ref) ?? [];
sites.push({ resource, key });
byRef.set(ref, sites);
}
return byRef;
}
// Force disposition, never guess. Every name the manifest requires lands in
// exactly one of four buckets, and two of them stop the run.
//
// The mapping is deliberately NOT a mechanical dump of the source box: some
// entries encode migration decisions rather than facts about the source. A
// "capture everything" verb would be wrong in a handful of entries out of
// seventeen, silently — which is worse than being wrong in all of them.
export function classify(
required: RequiredSecret[],
generated: string[],
live: LiveEnvs,
overrides: Record<string, string>,
): Classification {
const generatedSet = new Set(generated);
const plan: Disposition[] = [];
const missing: Classification["missing"] = [];
const conflicts: Classification["conflicts"] = [];
for (const [ref, sites] of groupByRef(required)) {
// The operator's word beats both the manifest and the source box: this is
// the escape hatch for a value that must NOT be carried over (staging's
// ADMIN_EMAIL, where the source's value is a real founder and staging
// shares a Mailgun domain with prod).
if (ref in overrides) {
plan.push({
ref,
provenance: "overridden",
value: overrides[ref],
sites,
});
continue;
}
if (generatedSet.has(ref)) {
plan.push({
ref,
provenance: "generated",
value: GENERATED_PLACEHOLDER,
sites,
});
continue;
}
// Captured: read the live value off whichever resources declare it.
const found = sites
.map((s) => ({ site: s, value: live[s.resource]?.[s.key] }))
.filter((f): f is { site: Site; value: string } => f.value !== undefined);
if (found.length === 0) {
missing.push({ ref, sites });
continue;
}
const distinct = new Set(found.map((f) => f.value));
if (distinct.size > 1) {
conflicts.push({ ref, values: found.map((f) => f.site) });
continue;
}
plan.push({
ref,
provenance: "captured",
value: found[0].value,
sites,
});
}
return { plan, missing, conflicts };
}
const site = (s: Site) => `${s.resource}.${s.key}`;
// Names and provenance. NEVER values.
//
// The one thing printed that looks like a value is GENERATED_PLACEHOLDER,
// which is a literal constant in this file and carries no information about
// the source box. Everything else is a name the manifest already declares in
// plaintext, in a committed file.
export function renderCapturePlan(
c: Classification,
ctx: {
orgRepo: string;
env: string;
instance: string;
store: string;
recipient: string;
},
): string {
const lines = [
`capture plan — ${ctx.orgRepo} ${ctx.env}`,
"",
` source: instance ${ctx.instance} (live values read from it)`,
` store: ${ctx.store}`,
` recipient: ${ctx.recipient}`,
"",
];
const width = Math.max(
0,
...[...c.plan, ...c.missing, ...c.conflicts].map((d) => d.ref.length),
);
for (const d of c.plan) {
const where = d.sites.map(site).join(", ");
const note =
d.provenance === "generated"
? `${GENERATED_PLACEHOLDER}`
: d.provenance === "overridden"
? ` (from CAST_CAPTURE_${d.ref})`
: "";
lines.push(
` ${d.ref.padEnd(width)} ${d.provenance.padEnd(10)} ${where}${note}`,
);
}
for (const m of c.missing) {
lines.push(
` ${m.ref.padEnd(width)} MISSING required by ${m.sites.map(site).join(", ")}, absent from the source`,
);
}
for (const c2 of c.conflicts) {
lines.push(
` ${c2.ref.padEnd(width)} CONFLICT differs between ${c2.values.map(site).join(" and ")}`,
);
}
const counts = (["captured", "generated", "overridden"] as const)
.map((p) => [p, c.plan.filter((d) => d.provenance === p).length] as const)
.filter(([, n]) => n > 0)
.map(([p, n]) => `${n} ${p}`)
.join(", ");
lines.push(
"",
`${c.plan.length} name(s) to write${counts ? `: ${counts}` : ""}`,
);
if (c.missing.length > 0) {
lines.push(
"",
`refusing to write the store: ${c.missing.length} name(s) the manifest requires are not`,
"present on the source. An empty value substitutes to nothing and the app boots",
"misconfigured — plausibly, and silently. Supply each one with --override <NAME>",
"(its value is read from CAST_CAPTURE_<NAME>, never from argv), or fix the source.",
);
}
if (c.conflicts.length > 0) {
lines.push(
"",
`refusing to write the store: ${c.conflicts.length} name(s) carry different values on`,
"different resources of the source. The store holds one value per name, and cast",
"will not pick for you. Reconcile them on the source, or pin one with --override.",
);
}
return lines.join("\n");
}

View file

@ -1,10 +1,12 @@
#!/usr/bin/env node
import { readFileSync } from "node:fs";
import { existsSync, readFileSync } from "node:fs";
import { join } from "node:path";
import { createInterface } from "node:readline/promises";
import { parseArgs } from "node:util";
import { parse as parseYaml } from "yaml";
import { type Executor, applyHostnameOverlay, applyPlan } from "./apply.js";
import { githubAppNameFor, loadBindings } from "./bindings.js";
import { type LiveEnvs, classify, renderCapturePlan } from "./capture.js";
import {
type CoolifyInstance,
assertWritable,
@ -19,14 +21,24 @@ import {
renderDiff,
} from "./diff.js";
import { assertEnvVarPolicy } from "./envtemplate.js";
import { desiredFromManifest, resolveCheckout } from "./resolve.js";
import { decryptSecrets, keyFileFor, secretsFileFor } from "./secrets.js";
import {
desiredFromManifest,
requiredSecrets,
resolveCheckout,
} from "./resolve.js";
import {
decryptSecrets,
encryptSecrets,
keyFileFor,
secretsFileFor,
} from "./secrets.js";
import { serverAdd } from "./server.js";
import { smoke } from "./smoke.js";
import { assertTeam, formatTeam } from "./team.js";
const USAGE = `usage: cast apply <org>/<repo> --env <env> [--path <dir>] [--project <name>] [--hostname-overlay <file>]
cast diff <org>/<repo> --env <env> [--full] [--project <name>]
cast capture <org>/<repo> --env <env> [--path <dir>] [--project <name>] [--generated <NAME>] [--override <NAME>] [--force]
cast server add <name> --ip <ip> --key <file> --env <env> [--user root] [--port 22]
cast smoke --env <env>
cast team [--env <env>]
@ -50,7 +62,16 @@ const USAGE = `usage: cast apply <org>/<repo> --env <env> [--path <dir>] [--proj
repo (the default). A project built by hand in the UI is called
whatever someone typed; \`diff\` refuses rather than reporting an
absent project as an empty one, and this is how you point it at
the real name.`;
the real name.
capture (adopt a hand-built instance into the age secret store):
--generated <NAME> force NAME to the \`pending-coolify-generated\` placeholder,
for a manifest that has not declared generated_secrets yet.
Repeatable.
--override <NAME> supply NAME yourself instead of copying the source's value.
The VALUE is read from \$CAST_CAPTURE_<NAME>, never from the
command line argv is visible in \`ps\`. Repeatable.
--force overwrite an existing store (refused by default).`;
// cast is stateless: every instance-scoped input is read from the state
// directory it is pointed at, never from a location the tool itself knows.
@ -290,21 +311,26 @@ export async function fetchLive(
// exists next to it.
export function renderAbsentTarget(
lookup: Extract<LiveLookup, { found: false }>,
ctx: { orgRepo: string; overridden: boolean },
ctx: { orgRepo: string; overridden: boolean; verb?: string },
): string {
// `capture` takes the same position as `diff`, and for the same reason: it
// is only ever a claim about something that already exists. Against an
// absent target it would read back zero live values and call every required
// secret "missing" — an alarming-but-meaningless report about the wrong box.
const verb = ctx.verb ?? "diff";
const origin = ctx.overridden
? "--project"
: `derived from the repo slug ${ctx.orgRepo}`;
const head =
lookup.missing === "project"
? [
`refusing to diff: no project named "${lookup.project}" exists in this team`,
`refusing to ${verb}: no project named "${lookup.project}" exists in this team`,
"",
` looked for: project "${lookup.project}" (${origin})`,
` exists here: ${lookup.available.join(", ") || "(no projects at all)"}`,
]
: [
`refusing to diff: project "${lookup.project}" has no environment "${lookup.environment}"`,
`refusing to ${verb}: project "${lookup.project}" has no environment "${lookup.environment}"`,
"",
` looked for: environment "${lookup.environment}" in project "${lookup.project}"`,
" note: cast names environments after --env, so a project built by",
@ -316,7 +342,7 @@ export function renderAbsentTarget(
"",
"An absent target reads back exactly like an empty one, so continuing would diff",
'it as "nothing exists — create everything": a clean-looking report that verified',
"nothing. `apply` may create a target; `diff` may only ever describe one that is",
`nothing. \`apply\` may create a target; \`${verb}\` may only ever describe one that is`,
"already there.",
"",
lookup.missing === "project"
@ -325,6 +351,79 @@ export function renderAbsentTarget(
].join("\n");
}
// A live resource's env vars, by key. `real_value` is the decrypted one and
// needs a token with read:sensitive; `value` is what a lesser token sees.
//
// A 404 (a resource we just listed no longer having an envs endpoint — not
// expected in practice, but consistent with treating "gone" as "no env vars")
// collapses to {}; anything else (401, 5xx, network) must surface. Swallowing
// it would make a live resource's env look EMPTY, which turns every one of its
// vars into a spurious create in a diff, and into a spurious "missing" in a
// capture.
async function fetchEnv(
client: CoolifyClient,
l: Live,
): Promise<Record<string, string>> {
const base = l.kind === "database" ? "databases" : `${l.kind}s`;
const envs = (await client.get(`/${base}/${l.uuid}/envs`).catch((err) => {
if (err instanceof HttpError && err.status === 404) return [];
throw err;
})) as Array<{ key: string; real_value?: string; value: string }>;
return Object.fromEntries(envs.map((e) => [e.key, e.real_value ?? e.value]));
}
// The value for an --override, read from the ENVIRONMENT rather than argv.
//
// A secret passed as a command-line argument is visible in `ps` to every
// process on the box, and lands in shell history — the same class of leak the
// clone-auth fix (#13) exists to avoid. So --override names the secret and the
// environment carries it.
function readOverrides(names: string[]): Record<string, string> {
const out: Record<string, string> = {};
for (const name of names) {
const varName = `CAST_CAPTURE_${name}`;
const value = process.env[varName];
if (value === undefined) {
throw new Error(
[
`--override ${name}: no value supplied.`,
"",
`cast reads an override's value from ${varName}, never from the command`,
"line — an argv value is visible in `ps` to every process on this box.",
"",
` ${varName}=… cast capture …`,
].join("\n"),
);
}
out[name] = value;
}
return out;
}
// Typed confirmation, and deliberately NOT a --yes flag.
//
// This verb writes an environment's secret store, once, off a box nobody is
// going to rebuild. The entire reason it exists is that the hand-run version
// was easy to get subtly wrong — so the last gate is a human who has read the
// provenance column typing the environment's own name. Nothing shorter counts:
// not "y", not a flag. Automating it means deliberately echoing the
// environment name into cast, which is an explicit act rather than an absent
// one.
//
// EOF (a closed or empty stdin) resolves to `null` and aborts. Without that
// race, a `< /dev/null` run would hang forever on a question nobody can answer.
async function confirmCapture(envName: string): Promise<boolean> {
const rl = createInterface({ input: process.stdin, output: process.stdout });
const answer = await new Promise<string | null>((resolve) => {
rl.question(
`\ntype the environment name to write this store (${envName}): `,
).then(resolve, () => resolve(null));
rl.once("close", () => resolve(null));
});
rl.close();
return answer?.trim() === envName;
}
async function main(): Promise<number> {
const [command, ...rest] = process.argv.slice(2);
if (command === "-h" || command === "--help" || command === "help") {
@ -419,28 +518,7 @@ async function main(): Promise<number> {
const live = lookup.found ? lookup.live : [];
if (mode === "full") {
for (const l of live) {
const envs = (await client
.get(
`/${l.kind === "database" ? "databases" : `${l.kind}s`}/${l.uuid}/envs`,
)
.catch((err) => {
// Same policy as fetchLive's environment fetch: a 404 (a
// resource we just listed no longer having an envs endpoint —
// not expected in practice, but consistent with treating
// "gone" as "no env vars") collapses to []; anything else
// (401, 5xx, network) must surface. Swallowing it here would
// make a live resource's env look empty and turn every one of
// its vars into a spurious create in the diff.
if (err instanceof HttpError && err.status === 404) return [];
throw err;
})) as Array<{
key: string;
real_value?: string;
value: string;
}>;
l.env = Object.fromEntries(
envs.map((e) => [e.key, e.real_value ?? e.value]),
);
l.env = await fetchEnv(client, l);
}
}
const report = computeDiff(desired, live, mode);
@ -466,6 +544,139 @@ async function main(): Promise<number> {
);
return 0;
}
if (command === "capture") {
const { values, positionals } = parseArgs({
args: rest,
allowPositionals: true,
options: {
env: { type: "string" },
state: { type: "string" },
path: { type: "string" },
project: { type: "string" },
instance: { type: "string" },
generated: { type: "string", multiple: true },
override: { type: "string", multiple: true },
force: { type: "boolean", default: false },
},
});
const orgRepo = positionals[0];
const envName = values.env;
if (!orgRepo || !envName) {
console.error(USAGE);
return 2;
}
const stateDir = stateDirFrom(values.state);
const repoShort = orgRepo.split("/")[1];
const projectName = values.project ?? repoShort;
const store = secretsFileFor(stateDir, repoShort, envName);
// Never overwrite a store by accident. `apply` never deletes; the verb
// that WRITES the store gets the same disposition, because the thing it
// would destroy is the only copy of values that may not exist anywhere
// else any more.
if (existsSync(store) && !values.force) {
console.error(
[
`refusing to capture: ${store} already exists`,
"",
"That store may hold the only copy of values the source box no longer has.",
"Pass --force to overwrite it deliberately, or move it aside first.",
].join("\n"),
);
return 2;
}
const bindings = loadBindings(join(stateDir, "environments.yaml"));
const binding = bindings.environments[envName];
if (!binding) {
console.error(`environment ${envName} not in environments.yaml`);
return 2;
}
const recipient = binding.age_recipient;
if (!recipient) {
console.error(
[
`environment ${envName} has no age_recipient in environments.yaml`,
"",
"capture encrypts the store TO that recipient (the public half of the",
"environment's age key — safe to commit next to the bindings). Add it:",
"",
" environments:",
` ${envName}:`,
" age_recipient: age1…",
].join("\n"),
);
return 2;
}
// Same rule as apply (resolveCheckout enforces it): prod always reads the
// default branch. A feature-branch manifest must not be able to decide
// which names land in the prod store.
const checkout = resolveCheckout(orgRepo, {
env: envName,
path: values.path,
});
const { required, generated } = requiredSecrets(checkout, envName);
const overrides = readOverrides(values.override ?? []);
const { client } = openCoolify(stateDir, values.instance, binding);
// capture READS Coolify and writes only to the local store, so it is
// allowed against a read-only instance — inspecting a legacy box is
// precisely what such an instance is for. It still takes the team assert:
// a wrong-team token reads back nothing, and "nothing" here would render
// as "every secret is missing" against a box that is fine.
const team = await assertTeam(client, binding.team, envName);
console.log(`team ${formatTeam(team)}`);
const lookup = await fetchLive(client, projectName, envName);
if (!lookup.found) {
console.error(
renderAbsentTarget(lookup, {
orgRepo,
overridden: values.project !== undefined,
verb: "capture",
}),
);
return 2;
}
const liveEnvs: LiveEnvs = {};
for (const l of lookup.live) {
// Databases hold no manifest-templated env of their own — their URL is
// what the APPS reference, and that name is generated, not captured.
if (l.kind === "database") continue;
liveEnvs[l.name] = await fetchEnv(client, l);
}
const classification = classify(
required,
[...generated, ...(values.generated ?? [])],
liveEnvs,
overrides,
);
console.log(
renderCapturePlan(classification, {
orgRepo,
env: envName,
instance: values.instance ?? binding.instance ?? "default",
store,
recipient,
}),
);
// Refuse, don't write a wrong store. Both of these are stop conditions,
// and the plan above has already named every offending entry.
if (
classification.missing.length > 0 ||
classification.conflicts.length > 0
)
return 2;
if (!(await confirmCapture(envName))) {
console.error("aborted — nothing written");
return 2;
}
encryptSecrets(
recipient,
store,
Object.fromEntries(classification.plan.map((d) => [d.ref, d.value])),
);
console.log(
`wrote ${store}${classification.plan.length} name(s), encrypted to ${recipient}`,
);
return 0;
}
if (command === "server" && rest[0] === "add") {
const { values, positionals } = parseArgs({
args: rest.slice(1),

View file

@ -2,11 +2,18 @@ export type ResolvedEnv = {
vars: Record<string, { value: string; secret: boolean }>;
};
export function resolveTemplate(
text: string,
secrets: Record<string, string>,
): ResolvedEnv {
const vars: ResolvedEnv["vars"] = {};
// A template line, parsed but not resolved: `ref` is set when the whole RHS is
// a single ${NAME} placeholder.
export type TemplateVar = { key: string; rhs: string; ref?: string };
// ONE grammar, shared by both readers of a template — resolveTemplate (which
// needs the values) and templateRefs (which needs only the names). Keeping
// them on separate parsers would let the two drift, and a drift here is not
// cosmetic: `capture` would collect a different set of names than `apply` will
// later demand, which is precisely the "a name silently missed" failure the
// capture verb exists to remove.
function parseTemplate(text: string): TemplateVar[] {
const vars: TemplateVar[] = [];
const lines = text.split("\n");
for (let i = 0; i < lines.length; i++) {
const line = lines[i].trim();
@ -18,21 +25,42 @@ export function resolveTemplate(
);
const [, key, rhs] = m;
const placeholder = rhs.match(/^\$\{([A-Z][A-Z0-9_]*)\}$/);
if (placeholder) {
const value = secrets[placeholder[1]];
vars.push({ key, rhs, ...(placeholder ? { ref: placeholder[1] } : {}) });
}
return vars;
}
export function resolveTemplate(
text: string,
secrets: Record<string, string>,
): ResolvedEnv {
const vars: ResolvedEnv["vars"] = {};
for (const { key, rhs, ref } of parseTemplate(text)) {
if (ref === undefined) {
vars[key] = { value: rhs, secret: false };
continue;
}
const value = secrets[ref];
if (value === undefined) {
throw new Error(
`secret ${placeholder[1]} (for ${key}) missing from the age store`,
);
throw new Error(`secret ${ref} (for ${key}) missing from the age store`);
}
vars[key] = { value, secret: true };
} else {
vars[key] = { value: rhs, secret: false };
}
}
return { vars };
}
// The ${NAME} refs a template declares: the secret names the manifest requires,
// paired with the env var each one lands on. `capture` reads these to learn
// what to go and fetch — at capture time there is no store to resolve against
// yet, which is the whole point of the verb.
export function templateRefs(
text: string,
): Array<{ key: string; ref: string }> {
return parseTemplate(text).flatMap(({ key, ref }) =>
ref === undefined ? [] : [{ key, ref }],
);
}
// An environment may forbid variables by name pattern, declared as
// `environments.<env>.forbidden_var_patterns` in the state repo. The rule is
// PRESENCE, not value: a forbidden var set to "false" still refuses the apply,

View file

@ -82,6 +82,19 @@ const EnvironmentSpecSchema = z
applications: z.record(AppSpecSchema),
databases: z.record(DatabaseSpecSchema).optional(),
services: z.record(ServiceSpecSchema).optional(),
// Secret names whose values the PROVIDER generates — a Coolify-created
// Postgres/Redis URL, a service's own generated credentials. `capture`
// writes these as the literal `pending-coolify-generated` and never copies
// the source box's live value: that value points at the SOURCE box's
// database, so carrying it over would be confidently wrong in a way that
// looks entirely plausible, and the target's real URL does not exist until
// Coolify creates the resource.
//
// It is a manifest property rather than a flag the operator has to
// remember, because the manifest is what knows DATABASE_URL comes from a
// database it declares. Optional: a manifest that names none simply has no
// generated secrets, and `capture` will say so in its plan.
generated_secrets: z.array(z.string()).optional(),
})
.strict();

View file

@ -3,7 +3,11 @@ import { existsSync, mkdtempSync, readFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { Desired } from "./diff.js";
import { type ResolvedEnv, resolveTemplate } from "./envtemplate.js";
import {
type ResolvedEnv,
resolveTemplate,
templateRefs,
} from "./envtemplate.js";
import { loadManifest } from "./manifest.js";
// How cast authenticated (or failed to authenticate) a clone.
@ -132,8 +136,11 @@ export function resolveCheckout(
opts: { env: string; path?: string },
): string {
if (opts.path && opts.env === "prod") {
// Holds for every verb that reads a manifest (apply, and now capture): a
// feature-branch checkout must not be able to decide what prod runs, nor
// which secret names land in prod's store.
throw new Error(
"apply refuses --path with --env prod: prod always reads the default branch",
"refuses --path with --env prod: prod always reads the default branch",
);
}
if (opts.path) return opts.path;
@ -171,6 +178,73 @@ export function resolveCheckout(
return dir;
}
// One secret the manifest requires: the ${REF} a template names (the key it
// gets in the age store), the resource that needs it, and the env var it lands
// on there. That last pair is what `capture` reads the live value from — the
// store is keyed by REF, but the live box knows it as `resource.key`.
export type RequiredSecret = { ref: string; resource: string; key: string };
// Exactly the set of secret names an environment's manifest demands — the same
// set `apply` will later insist on, read from the same templates by the same
// parser. `capture` uses this to know what to go and fetch; nothing else has to
// be told, and nothing can be silently missed.
//
// Deliberately does NOT take a secrets map: at capture time the store does not
// exist yet. That is the whole point of the verb.
export function requiredSecrets(
checkoutDir: string,
envName: string,
): { required: RequiredSecret[]; generated: string[] } {
const manifest = loadManifest(join(checkoutDir, ".infra", "manifest.yaml"));
const envSpec = manifest.environments[envName];
if (!envSpec) {
throw new Error(
`environment ${envName} not in manifest (has: ${Object.keys(manifest.environments).join(", ") || "none"})`,
);
}
const required: RequiredSecret[] = [];
const collect = (resource: string, template?: string) => {
if (!template) return;
const file = join(checkoutDir, ".infra", "env", template);
if (!existsSync(file))
throw new Error(
`env template missing: ${file} (referenced by ${resource})`,
);
for (const { key, ref } of templateRefs(readFileSync(file, "utf8"))) {
required.push({ ref, resource, key });
}
};
for (const [name, app] of Object.entries(envSpec.applications)) {
collect(name, app.env_template);
}
for (const [name, svc] of Object.entries(envSpec.services ?? {})) {
collect(name, svc.env_template);
}
const generated = envSpec.generated_secrets ?? [];
// A generated_secrets entry naming something no template refs is dead
// config — and dead config in THIS list is not merely untidy, it is
// dangerous: it reads like a guard standing over a name while standing over
// nothing. The likeliest cause is a typo, and the consequence of the typo is
// that the real name gets CAPTURED from the source box instead of placeheld.
const refs = new Set(required.map((r) => r.ref));
const dead = generated.filter((g) => !refs.has(g));
if (dead.length > 0) {
throw new Error(
[
`manifest environment ${envName}: generated_secrets names ${dead.join(", ")}, which no env template refers to`,
"",
` declared: ${generated.join(", ")}`,
` templates: ${[...refs].sort().join(", ") || "(no ${...} refs at all)"}`,
"",
"A generated name that matches nothing guards nothing — and if this is a",
"typo, the name it was meant to guard is being captured from the source",
"box instead of placeheld. Fix the spelling, or drop the entry.",
].join("\n"),
);
}
return { required, generated };
}
export function desiredFromManifest(
checkoutDir: string,
envName: string,

View file

@ -22,6 +22,27 @@ export function decryptSecrets(
return secrets;
}
// Write an environment's secret store, encrypted to its recipient.
//
// The plaintext goes to age on STDIN and the ciphertext straight to `file`: it
// is never a temp file, never reaches the terminal, and never lands in shell
// history. The hand-run recipe this replaces assembled /dev/shm/prod.env and
// relied on remembering to `shred -u` it afterwards — a step that is invisible
// when it is skipped.
export function encryptSecrets(
recipient: string,
file: string,
vars: Record<string, string>,
): void {
const plaintext = `${Object.entries(vars)
.map(([k, v]) => `${k}=${v}`)
.join("\n")}\n`;
execFileSync("age", ["-r", recipient, "-o", file], {
input: plaintext,
stdio: ["pipe", "pipe", "pipe"],
});
}
// The age identity for an environment, resolved without cast knowing anything
// about your environment names:
//

334
test/capture-cli.test.ts Normal file
View file

@ -0,0 +1,334 @@
import { execFileSync, spawn } from "node:child_process";
import {
existsSync,
mkdirSync,
mkdtempSync,
readFileSync,
writeFileSync,
} from "node:fs";
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, beforeAll, describe, expect, it } from "vitest";
import { decryptSecrets } from "../src/secrets.js";
// End-to-end: the real CLI, a real age identity, a stub Coolify holding real
// live values. The point is the store that comes out the other side — it is
// decrypted and asserted on, so "exactly the names the manifest requires, no
// more and no fewer" is checked against the actual ciphertext rather than
// against cast's own console output.
const SECRETS = {
// Points at the SOURCE box. Must NOT be carried over.
DATABASE_URL: "postgres://user:pw@SOURCE-BOX-postgres:5432/app",
MAILGUN_API_KEY: "key-REAL-MAILGUN-SECRET",
OPENROUTER_API_KEY: "sk-or-REAL-OPENROUTER-SECRET",
// A real founder. Must NOT be carried over to staging.
ADMIN_EMAIL: "founder@real-company.com",
// Live on the box, but the manifest never asks for it.
UNRELATED_LIVE_VAR: "nobody-asked-for-this",
};
let keyFile: string;
let recipient: string;
beforeAll(() => {
const dir = mkdtempSync(join(tmpdir(), "cast-age-"));
keyFile = join(dir, "age-staging.key");
execFileSync("age-keygen", ["-o", keyFile], { stdio: "pipe" });
const pub = execFileSync("age-keygen", ["-y", keyFile], { encoding: "utf8" });
recipient = pub.trim();
});
type Stub = { url: string; close: () => Promise<void> };
const stubs: Stub[] = [];
// A Coolify with one project, one environment, one application carrying the
// live env above.
async function stubCoolify(): Promise<Stub> {
const server = createServer((req, res) => {
const path = (req.url ?? "").replace("/api/v1", "");
const json = (body: unknown) => {
res.writeHead(200, { "content-type": "application/json" });
res.end(JSON.stringify(body));
};
if (path === "/teams/current") return json({ id: 0, name: "Root Team" });
if (path === "/projects") return json([{ uuid: "p1", name: "incubator" }]);
if (path === "/projects/p1/staging")
return json({ applications: [{ name: "core", uuid: "a1" }] });
if (path === "/applications/a1/envs")
return json(
Object.entries(SECRETS).map(([key, real_value]) => ({
key,
real_value,
value: "REDACTED",
})),
);
res.writeHead(404);
res.end("{}");
});
await new Promise<void>((r) => {
server.listen(0, "127.0.0.1", r);
});
const stub: Stub = {
url: `http://127.0.0.1:${(server.address() as AddressInfo).port}`,
close: () =>
new Promise<void>((r) => {
server.close(() => r());
}),
};
stubs.push(stub);
return stub;
}
afterEach(async () => {
await Promise.all(stubs.splice(0).map((s) => s.close()));
});
const MANIFEST = `project: incubator
environments:
staging:
generated_secrets: [DATABASE_URL_STAGING]
applications:
core:
source: { repo: heavy-duty/incubator, branch: main }
build: { pack: nixpacks, base_directory: / }
domains: ["http://core.example.com"]
env_template: core.staging.env.template
`;
// NODE_ENV is a literal, not a secret — it must not reach the store.
const TEMPLATE = `NODE_ENV=production
DATABASE_URL=\${DATABASE_URL_STAGING}
MAILGUN_API_KEY=\${MAILGUN_API_KEY}
OPENROUTER_API_KEY=\${OPENROUTER_API_KEY}
ADMIN_EMAIL=\${ADMIN_EMAIL}
`;
function fixture(url: string, opts: { template?: string } = {}) {
const checkout = mkdtempSync(join(tmpdir(), "cast-co-"));
mkdirSync(join(checkout, ".infra", "env"), { recursive: true });
writeFileSync(join(checkout, ".infra", "manifest.yaml"), MANIFEST);
writeFileSync(
join(checkout, ".infra", "env", "core.staging.env.template"),
opts.template ?? TEMPLATE,
);
const state = mkdtempSync(join(tmpdir(), "cast-state-"));
mkdirSync(join(state, "secrets"));
writeFileSync(
join(state, ".coolify.env"),
`COOLIFY_BASE_URL="${url}"\nCOOLIFY_ACCESS_TOKEN="t"\n`,
);
writeFileSync(
join(state, "environments.yaml"),
[
"environments:",
" staging:",
" server: staging-box",
" team: { id: 0, name: Root Team }",
` age_recipient: ${recipient}`,
"github_apps:",
" incubator: hdb-coolify",
"",
].join("\n"),
);
return {
checkout,
state,
store: join(state, "secrets", "incubator.staging.env.age"),
};
}
function runCapture(
args: string[],
opts: { stdin?: string; env?: Record<string, string> } = {},
): Promise<{ code: number; output: string }> {
return new Promise((resolve) => {
const child = spawn("node", ["dist/cli.js", "capture", ...args], {
stdio: ["pipe", "pipe", "pipe"],
env: { ...process.env, ...opts.env },
});
let output = "";
child.stdout.on("data", (d) => {
output += String(d);
});
child.stderr.on("data", (d) => {
output += String(d);
});
child.stdin.end(opts.stdin ?? "");
child.on("close", (code) => resolve({ code: code ?? 0, output }));
});
}
const base = (f: ReturnType<typeof fixture>) => [
"heavy-duty/incubator",
"--env",
"staging",
"--state",
f.state,
"--path",
f.checkout,
];
describe("cast capture (end to end)", () => {
it("writes a store with exactly the manifest's names, and the right provenance", async () => {
const f = fixture((await stubCoolify()).url);
const r = await runCapture([...base(f), "--override", "ADMIN_EMAIL"], {
stdin: "staging\n",
env: { CAST_CAPTURE_ADMIN_EMAIL: "operator@example.com" },
});
expect(r.code).toBe(0);
expect(existsSync(f.store)).toBe(true);
const store = decryptSecrets(f.store, keyFile);
// No more, no fewer: the four ${...} refs. NODE_ENV is a literal, and
// UNRELATED_LIVE_VAR is live but unasked-for — neither belongs here.
expect(Object.keys(store).sort()).toEqual([
"ADMIN_EMAIL",
"DATABASE_URL_STAGING",
"MAILGUN_API_KEY",
"OPENROUTER_API_KEY",
]);
// Captured verbatim.
expect(store.MAILGUN_API_KEY).toBe(SECRETS.MAILGUN_API_KEY);
expect(store.OPENROUTER_API_KEY).toBe(SECRETS.OPENROUTER_API_KEY);
// Generated: the placeholder, NEVER the source box's own database URL.
expect(store.DATABASE_URL_STAGING).toBe("pending-coolify-generated");
expect(store.DATABASE_URL_STAGING).not.toContain("SOURCE-BOX");
// Overridden: the operator's value, not the real founder's address.
expect(store.ADMIN_EMAIL).toBe("operator@example.com");
expect(store.ADMIN_EMAIL).not.toBe(SECRETS.ADMIN_EMAIL);
});
// "No secret value is ever written to stdout" — checked against the real
// values the stub served, on the real console output of a real run.
it("never prints a secret value to the console", async () => {
const f = fixture((await stubCoolify()).url);
const r = await runCapture([...base(f), "--override", "ADMIN_EMAIL"], {
stdin: "staging\n",
env: { CAST_CAPTURE_ADMIN_EMAIL: "operator@example.com" },
});
expect(r.code).toBe(0);
for (const value of Object.values(SECRETS)) {
expect(r.output).not.toContain(value);
}
expect(r.output).not.toContain("operator@example.com");
// It did print the NAMES, though — that is the plan.
expect(r.output).toContain("MAILGUN_API_KEY");
expect(r.output).toContain("captured");
expect(r.output).toContain("generated");
expect(r.output).toContain("overridden");
});
// The plaintext exists only in memory and on age's stdin.
it("leaves no plaintext behind — the store is real ciphertext", async () => {
const f = fixture((await stubCoolify()).url);
await runCapture([...base(f), "--override", "ADMIN_EMAIL"], {
stdin: "staging\n",
env: { CAST_CAPTURE_ADMIN_EMAIL: "operator@example.com" },
});
const raw = readFileSync(f.store, "utf8");
expect(raw).toContain("age-encryption.org");
for (const value of Object.values(SECRETS)) {
expect(raw).not.toContain(value);
}
});
// A name required by the template but absent from the source refuses the run
// — writing an empty would boot the app misconfigured, plausibly.
it("refuses when a required name is absent from the source", async () => {
const f = fixture((await stubCoolify()).url, {
template: `${TEMPLATE}TURNSTILE_SECRET=\${TURNSTILE_SECRET}\n`,
});
const r = await runCapture([...base(f), "--override", "ADMIN_EMAIL"], {
stdin: "staging\n",
env: { CAST_CAPTURE_ADMIN_EMAIL: "operator@example.com" },
});
expect(r.code).not.toBe(0);
expect(r.output).toMatch(/TURNSTILE_SECRET\s+MISSING/);
expect(r.output).toMatch(/refusing to write the store/);
expect(existsSync(f.store)).toBe(false);
});
// The confirmation is the last gate, and it is not "y".
it("aborts, writing nothing, when the confirmation does not name the env", async () => {
const f = fixture((await stubCoolify()).url);
const r = await runCapture([...base(f), "--override", "ADMIN_EMAIL"], {
stdin: "y\n",
env: { CAST_CAPTURE_ADMIN_EMAIL: "operator@example.com" },
});
expect(r.code).not.toBe(0);
expect(r.output).toMatch(/aborted/);
expect(existsSync(f.store)).toBe(false);
});
it("aborts on a closed stdin rather than hanging", async () => {
const f = fixture((await stubCoolify()).url);
const r = await runCapture([...base(f), "--override", "ADMIN_EMAIL"], {
stdin: "",
env: { CAST_CAPTURE_ADMIN_EMAIL: "operator@example.com" },
});
expect(r.code).not.toBe(0);
expect(r.output).toMatch(/aborted/);
});
// An override's VALUE never comes from argv — argv is visible in `ps`.
it("refuses an --override whose CAST_CAPTURE_<NAME> is unset", async () => {
const f = fixture((await stubCoolify()).url);
const r = await runCapture([...base(f), "--override", "ADMIN_EMAIL"], {
stdin: "staging\n",
});
expect(r.code).not.toBe(0);
expect(r.output).toMatch(/CAST_CAPTURE_ADMIN_EMAIL/);
expect(r.output).toMatch(/never from the command/);
});
// The store may hold the only copy of values the source box no longer has.
it("refuses to overwrite an existing store without --force", async () => {
const f = fixture((await stubCoolify()).url);
writeFileSync(f.store, "PRE-EXISTING");
const r = await runCapture([...base(f), "--override", "ADMIN_EMAIL"], {
stdin: "staging\n",
env: { CAST_CAPTURE_ADMIN_EMAIL: "operator@example.com" },
});
expect(r.code).not.toBe(0);
expect(r.output).toMatch(/already exists/);
expect(r.output).toMatch(/--force/);
expect(readFileSync(f.store, "utf8")).toBe("PRE-EXISTING");
});
it("refuses an environment with no age_recipient", async () => {
const f = fixture((await stubCoolify()).url);
writeFileSync(
join(f.state, "environments.yaml"),
[
"environments:",
" staging:",
" server: staging-box",
" team: { id: 0, name: Root Team }",
"github_apps:",
" incubator: hdb-coolify",
"",
].join("\n"),
);
const r = await runCapture(base(f), { stdin: "staging\n" });
expect(r.code).not.toBe(0);
expect(r.output).toMatch(/age_recipient/);
});
// Same position as diff: capture is only ever a claim about something that
// already exists. Against an absent target it would call every secret
// "missing" — an alarming report about the wrong box.
it("refuses an absent target rather than reporting every secret missing", async () => {
const f = fixture((await stubCoolify()).url);
const r = await runCapture([...base(f), "--project", "typo"], {
stdin: "staging\n",
});
expect(r.code).not.toBe(0);
expect(r.output).toMatch(/refusing to capture/);
expect(r.output).toMatch(/no project named "typo"/);
expect(r.output).not.toMatch(/MISSING/);
});
});

309
test/capture.test.ts Normal file
View file

@ -0,0 +1,309 @@
import { mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { describe, expect, it } from "vitest";
import {
GENERATED_PLACEHOLDER,
classify,
renderCapturePlan,
} from "../src/capture.js";
import { requiredSecrets } from "../src/resolve.js";
const CTX = {
orgRepo: "heavy-duty/incubator",
env: "prod",
instance: "legacy",
store: "/s/secrets/incubator.prod.env.age",
recipient: "age1abc",
};
// The live case, shrunk: an app whose env template needs a generated database
// URL, a carried-over API key, and an address that must NOT be carried over.
const REQUIRED = [
{ ref: "DATABASE_URL_PROD", resource: "core", key: "DATABASE_URL" },
{ ref: "MAILGUN_API_KEY", resource: "core", key: "MAILGUN_API_KEY" },
{ ref: "ADMIN_EMAIL", resource: "core", key: "ADMIN_EMAIL" },
];
const LIVE = {
core: {
DATABASE_URL: "postgres://SOURCE-BOX-INTERNAL/db",
MAILGUN_API_KEY: "key-abc123-REAL-SECRET",
ADMIN_EMAIL: "founder@real-company.com",
},
};
describe("classify", () => {
it("captures a live value, and records where it came from", () => {
const c = classify([REQUIRED[1]], [], LIVE, {});
expect(c.plan).toEqual([
{
ref: "MAILGUN_API_KEY",
provenance: "captured",
value: "key-abc123-REAL-SECRET",
sites: [{ resource: "core", key: "MAILGUN_API_KEY" }],
},
]);
});
// The failure this verb exists to prevent: the source box's DATABASE_URL
// points at the SOURCE box's Postgres. Copying it over is confidently wrong
// in a way that looks entirely plausible.
it("placeholds a generated name, never copying the source's value", () => {
const c = classify([REQUIRED[0]], ["DATABASE_URL_PROD"], LIVE, {});
expect(c.plan[0]).toMatchObject({
ref: "DATABASE_URL_PROD",
provenance: "generated",
value: GENERATED_PLACEHOLDER,
});
expect(c.plan[0].value).not.toContain("SOURCE-BOX");
});
// staging and prod share a Mailgun domain, so a staging box carrying the
// real ADMIN_EMAIL can mail real users.
it("takes an override from the operator, over the source's value", () => {
const c = classify([REQUIRED[2]], [], LIVE, {
ADMIN_EMAIL: "operator@example.com",
});
expect(c.plan[0]).toMatchObject({
provenance: "overridden",
value: "operator@example.com",
});
});
it("an override beats a generated declaration too", () => {
const c = classify([REQUIRED[0]], ["DATABASE_URL_PROD"], LIVE, {
DATABASE_URL_PROD: "postgres://explicit",
});
expect(c.plan[0]).toMatchObject({
provenance: "overridden",
value: "postgres://explicit",
});
});
// Required by the template, absent from the source: refuse rather than write
// an empty. An empty substitutes to nothing and the app boots misconfigured.
it("refuses on a name required by the template but absent from the source", () => {
const c = classify(
[{ ref: "TURNSTILE_SECRET", resource: "core", key: "TURNSTILE_SECRET" }],
[],
LIVE,
{},
);
expect(c.plan).toEqual([]);
expect(c.missing).toEqual([
{
ref: "TURNSTILE_SECRET",
sites: [{ resource: "core", key: "TURNSTILE_SECRET" }],
},
]);
});
it("a missing name can be rescued by an override", () => {
const c = classify(
[{ ref: "TURNSTILE_SECRET", resource: "core", key: "TURNSTILE_SECRET" }],
[],
LIVE,
{ TURNSTILE_SECRET: "supplied" },
);
expect(c.missing).toEqual([]);
expect(c.plan[0].provenance).toBe("overridden");
});
// One name, two resources, two different live values. The store holds one
// value per name; picking wrong would be silent.
it("refuses when one name carries different values on two resources", () => {
const c = classify(
[
{ ref: "SHARED", resource: "core", key: "SHARED" },
{ ref: "SHARED", resource: "worker", key: "SHARED" },
],
[],
{ core: { SHARED: "a" }, worker: { SHARED: "b" } },
{},
);
expect(c.plan).toEqual([]);
expect(c.conflicts).toEqual([
{
ref: "SHARED",
values: [
{ resource: "core", key: "SHARED" },
{ resource: "worker", key: "SHARED" },
],
},
]);
});
it("is fine when one name carries the SAME value on two resources", () => {
const c = classify(
[
{ ref: "SHARED", resource: "core", key: "SHARED" },
{ ref: "SHARED", resource: "worker", key: "SHARED" },
],
[],
{ core: { SHARED: "same" }, worker: { SHARED: "same" } },
{},
);
expect(c.conflicts).toEqual([]);
expect(c.plan[0].value).toBe("same");
});
// The acceptance criterion: exactly the names the manifest requires, no more
// and no fewer. A live var the manifest does not ask for is not the store's
// business.
it("writes exactly the required names — ignoring live vars nobody asked for", () => {
const c = classify(
REQUIRED,
["DATABASE_URL_PROD"],
{
core: { ...LIVE.core, SOME_OTHER_LIVE_VAR: "not in the manifest" },
},
{},
);
expect(c.plan.map((d) => d.ref).sort()).toEqual([
"ADMIN_EMAIL",
"DATABASE_URL_PROD",
"MAILGUN_API_KEY",
]);
});
});
describe("renderCapturePlan", () => {
// THE invariant. "No secret value is ever written to stdout" — so the plan
// is names and provenance, and the test asserts on the actual live values
// rather than on a pattern that could drift away from them.
it("never prints a secret value", () => {
const c = classify(REQUIRED, ["DATABASE_URL_PROD"], LIVE, {
ADMIN_EMAIL: "operator@example.com",
});
const out = renderCapturePlan(c, CTX);
for (const secret of [
"postgres://SOURCE-BOX-INTERNAL/db",
"key-abc123-REAL-SECRET",
"founder@real-company.com",
"operator@example.com",
]) {
expect(out).not.toContain(secret);
}
});
it("shows every name with its provenance and where it lands", () => {
const c = classify(REQUIRED, ["DATABASE_URL_PROD"], LIVE, {
ADMIN_EMAIL: "operator@example.com",
});
const out = renderCapturePlan(c, CTX);
expect(out).toMatch(/MAILGUN_API_KEY\s+captured\s+core\.MAILGUN_API_KEY/);
expect(out).toMatch(/DATABASE_URL_PROD\s+generated/);
expect(out).toContain(GENERATED_PLACEHOLDER);
expect(out).toMatch(/ADMIN_EMAIL\s+overridden/);
expect(out).toContain("CAST_CAPTURE_ADMIN_EMAIL");
expect(out).toMatch(/3 name\(s\) to write/);
expect(out).toContain("/s/secrets/incubator.prod.env.age");
expect(out).toContain("age1abc");
});
it("names what is missing, and says why an empty would be worse", () => {
const c = classify(
[{ ref: "TURNSTILE_SECRET", resource: "core", key: "TURNSTILE_SECRET" }],
[],
LIVE,
{},
);
const out = renderCapturePlan(c, CTX);
expect(out).toMatch(/TURNSTILE_SECRET\s+MISSING/);
expect(out).toMatch(/refusing to write the store/);
expect(out).toMatch(/--override/);
});
it("names a conflict rather than picking a side", () => {
const c = classify(
[
{ ref: "SHARED", resource: "core", key: "SHARED" },
{ ref: "SHARED", resource: "worker", key: "SHARED" },
],
[],
{ core: { SHARED: "a" }, worker: { SHARED: "b" } },
{},
);
const out = renderCapturePlan(c, CTX);
expect(out).toMatch(/SHARED\s+CONFLICT/);
expect(out).toMatch(/core\.SHARED and worker\.SHARED/);
expect(out).not.toMatch(/\ba\b.*\bb\b/);
});
});
// requiredSecrets is what makes "no more, no fewer" true: the set comes from
// the manifest's own templates, read by the same parser apply uses.
describe("requiredSecrets", () => {
function checkout(manifest: string, templates: Record<string, string>) {
const dir = mkdtempSync(join(tmpdir(), "cast-cap-"));
mkdirSync(join(dir, ".infra", "env"), { recursive: true });
writeFileSync(join(dir, ".infra", "manifest.yaml"), manifest);
for (const [name, body] of Object.entries(templates)) {
writeFileSync(join(dir, ".infra", "env", name), body);
}
return dir;
}
const MANIFEST = `project: incubator
environments:
prod:
generated_secrets: [DATABASE_URL_PROD]
applications:
core:
source: { repo: heavy-duty/incubator, branch: main }
build: { pack: dockercompose, base_directory: /, compose_file: docker-compose.yaml }
service_domains:
api: ["https://api.example.com"]
env_template: core.prod.env.template
services:
umami:
type: umami
env_template: umami.prod.env.template
`;
it("collects the ${...} refs from every app and service template", () => {
const dir = checkout(MANIFEST, {
"core.prod.env.template":
"NODE_ENV=production\nDATABASE_URL=${DATABASE_URL_PROD}\nMAILGUN_API_KEY=${MAILGUN_API_KEY}\n",
"umami.prod.env.template": "APP_SECRET=${UMAMI_APP_SECRET}\n",
});
const { required, generated } = requiredSecrets(dir, "prod");
expect(required).toEqual([
{ ref: "DATABASE_URL_PROD", resource: "core", key: "DATABASE_URL" },
{ ref: "MAILGUN_API_KEY", resource: "core", key: "MAILGUN_API_KEY" },
{ ref: "UMAMI_APP_SECRET", resource: "umami", key: "APP_SECRET" },
]);
expect(generated).toEqual(["DATABASE_URL_PROD"]);
});
// A non-placeholder line (NODE_ENV=production) is not a secret and must not
// land in the store — the store holds the ${...} refs, nothing else.
it("ignores literal template values — only ${...} refs are secrets", () => {
const dir = checkout(MANIFEST, {
"core.prod.env.template":
"NODE_ENV=production\nREPORTING_ENABLED=false\nDATABASE_URL=${DATABASE_URL_PROD}\n",
"umami.prod.env.template": "",
});
const { required } = requiredSecrets(dir, "prod");
expect(required.map((r) => r.ref)).toEqual(["DATABASE_URL_PROD"]);
});
// Dead config in THIS list is dangerous, not merely untidy: it reads like a
// guard standing over a name while standing over nothing, and the likeliest
// cause is a typo whose real name is then CAPTURED from the source box.
it("refuses a generated_secrets entry that no template refers to", () => {
const dir = checkout(
MANIFEST.replace(
"generated_secrets: [DATABASE_URL_PROD]",
"generated_secrets: [DATABASE_URL_TYPO]",
),
{
"core.prod.env.template": "DATABASE_URL=${DATABASE_URL_PROD}\n",
"umami.prod.env.template": "",
},
);
expect(() => requiredSecrets(dir, "prod")).toThrow(
/generated_secrets names DATABASE_URL_TYPO/,
);
});
});