cast/src/capture.ts

242 lines
8.8 KiB
TypeScript
Raw Normal View History

feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
import type { RequiredSecret } from "./resolve.js";
// What the manifest writes for a provider-generated name. Not the source box's
// value — that points at the SOURCE box's Postgres/Redis — and not an empty
// string, which would boot the app misconfigured. A literal that is obviously
// a placeholder, and that Coolify replaces when it creates the resource.
export const GENERATED_PLACEHOLDER = "pending-coolify-generated";
// Live env vars, per resource: resource name -> (env key -> value).
export type LiveEnvs = Record<string, Record<string, string>>;
export type Provenance = "captured" | "generated" | "overridden";
export type Site = { resource: string; key: string };
export type Disposition = {
ref: string;
provenance: Provenance;
// Never rendered. Kept here so the caller can encrypt it, and nowhere else.
value: string;
sites: Site[];
};
export type Classification = {
plan: Disposition[];
// Required by a template, absent from the source, and not dispositioned
// otherwise. Refuses the run: writing an empty value substitutes to nothing
// and the app boots misconfigured — the exact failure capture exists to
// remove, and one that looks entirely plausible from the outside.
missing: Array<{ ref: string; sites: Site[] }>;
// The same ref carrying DIFFERENT live values on two resources. cast cannot
// pick, and picking wrong is silent, so it refuses.
conflicts: Array<{ ref: string; values: Site[] }>;
};
feat: read-side coordinates (#17, #18) + cast inventory (#19) Three fixes at one seam: cast could not READ a box it did not build. #17 — the environment had no read-side coordinate. `--project` exists because a hand-built project is called whatever someone typed. The environment has the identical problem and had no flag, so reading a legacy box forced a choice between mutating that box's UI and renaming OUR environment to match it. The second is what happened: `prod` became `production` across the manifest and environments.yaml — a box being deleted next week naming the environment of the box that replaces it, permanently (apply creates the environment from --env), moving the store to incubator.production.env.age and invalidating every runbook. Reverted. `--environment` is now the coordinate. `--env` stays OURS: manifest block, binding, age key, store path, team assert. `--environment` is theirs, on the wire, and nothing else. #18 — an absent RESOURCE reported as N missing secrets. The D-237 lie, one level deeper. A resource that is absent reads back exactly like one present with no env vars, so capture reported all 15 required names as individually MISSING — from a box that was serving production and sending mail at that moment — and offered --override as the remedy. Taking that offer would have "worked": a valid store, hand-carried values, and the real finding (the manifest and the box disagree about what the app is called) buried. capture now refuses on the resource, names what does exist, and only reports per-name MISSING for resources it actually found — where it means what it says. #19 — cast inventory: see the box before you adopt it. The missing first step. cast could describe a box it built, change one, and take values off one for names a manifest declares — but not tell you what is on a box you did not build, which is the first thing adoption needs. Every mismatch above surfaced as a refusal from a verb already committed to a course of action, and the tempting fix for two of them was to bend the manifest toward the legacy box. inventory reads resources and env var KEYS (never values), sorts them into on-both / manifest-only / box-only, and needs no store, no age key and no recipient — it runs before adoption exists. Its output is a document: inventory → human reads → manifest PR → capture → apply That boundary is what lets capture stay strict. inventory may read everything, because a person reads its output. capture may only write what the manifest declares, because `apply` reads its output. Same box, two consumers, two contracts. A manifest-draft emitter is deliberately NOT included: it would be one `cp` away from becoming desired state, which is the failure this design exists to prevent. Zero drift against a hand-built box is reported as suspicious, not as a pass. npm run check + build clean; 164 tests passing, 18 files (was 151/16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 18:20:53 +00:00
// The manifest resources that no resource of that name exists for on the source.
//
// This is the absent-target lie (#12/D-237) one level deeper in the tree, and it
// fails in exactly the same way: a resource that is ABSENT reads back
// identically to one that is PRESENT with no env vars set. Every name it
// declares reports MISSING, the run refuses, and the message tells the operator
// to supply each of them with --override — which would "work", writing a
// perfectly valid store, while the real finding (the manifest and the box
// disagree about what this resource is called) is never discovered and the
// operator hand-carries values that were sitting right there under another name.
//
// So: refuse on the RESOURCE first, and only report per-name MISSING for
// resources that were actually found — where it means what it says.
export function absentResources(
required: RequiredSecret[],
liveNames: Iterable<string>,
): string[] {
const live = new Set(liveNames);
const declared = new Set(required.map((r) => r.resource));
return [...declared].filter((name) => !live.has(name)).sort();
}
export function renderAbsentResources(
absent: string[],
live: Array<{ kind: string; name: string }>,
ctx: { project: string; environment: string },
): string {
const width = Math.max(0, ...live.map((l) => l.kind.length));
return [
`refusing to capture: the manifest declares ${absent.length} resource(s) that do not exist here`,
"",
` looked in: project "${ctx.project}", environment "${ctx.environment}"`,
` looked for: ${absent.join(", ")}`,
" exists here:",
...(live.length > 0
? live.map((l) => ` ${l.kind.padEnd(width)} ${l.name}`)
: [" (nothing at all)"]),
"",
"A resource that is absent reads back exactly like one with no env vars set:",
"every name it declares reports MISSING, and --override would then have you",
"hand-carry values that are sitting right there under a different name. The",
"finding is not that the secrets are missing — it is that the manifest and this",
"box disagree about what these resources are called.",
"",
feat: --resource, the third name a hand-built box does not share with you #20 shipped the refusal without shipping the resolution: capture correctly refuses when a manifest resource does not exist on the source, and then there was no way to say "it's over there, under another name." Found immediately, on the box that motivated it. The manifest says `core`, `landing`, `postgres`, `redis`, `umami`. The box says `Incubator Stack v2`, `Incubator Landing`, `Incubator Database v2`, `Incubator Redis v2`, `Incubator Umami`. Neither is wrong — one names things for a human reading a UI, the other for a machine reading a diff — and neither gets to overwrite the other. --resource core="Incubator Stack v2" (repeatable) Applied at the boundary: live resources are renamed to the manifest's vocabulary once, immediately after the lookup, so computeDiff / classify / reconcile all match by name exactly as before and none of them needs to know a hand-built box was involved. Read-side only, and `apply` refuses it up front — before a clone, a decrypt or a single call. apply CREATES under the manifest's names, so an alias there could only mean "adopt the existing one instead": a different operation nobody has asked for, whose silent failure mode is a duplicate resource created beside the one you were pointing at. diff needed this as much as capture did. Without it, a --full diff against a box whose resources are named differently reports every manifest resource as "to create" and never mentions the live ones — the D-237 lie by another route, a confident full-create plan against a box that has all of it under other names. That diff is the staleness gate of a live migration. Two smaller things, both about not laundering a naming gap into a pass: - inventory, when NOTHING matched and yet the box has resources, now says so and prints the --resource lines to paste. "The box is empty" is exactly the wrong conclusion, and it was the easy one to draw. - the absent-resource refusal prints the same, per absent resource. An alias whose left side names no manifest resource is an error, not a no-op: a typo would otherwise map nothing, leave the real resource looked-up under its own name, and refuse with no hint that the flag had missed. inventory keeps the box's own name beside ours in the report (`core ← "Incubator Stack v2" on the box`) — a document that renamed the box's resources to our vocabulary and never mentioned theirs would be unusable against the UI it describes. npm run check + build clean; 169 tests passing (was 164). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 18:50:06 +00:00
"`cast inventory` shows both sides. Then map them at the call site:",
"",
...absent.map(
(name) => ` --resource ${name}="<what this box calls it>"`,
),
feat: read-side coordinates (#17, #18) + cast inventory (#19) Three fixes at one seam: cast could not READ a box it did not build. #17 — the environment had no read-side coordinate. `--project` exists because a hand-built project is called whatever someone typed. The environment has the identical problem and had no flag, so reading a legacy box forced a choice between mutating that box's UI and renaming OUR environment to match it. The second is what happened: `prod` became `production` across the manifest and environments.yaml — a box being deleted next week naming the environment of the box that replaces it, permanently (apply creates the environment from --env), moving the store to incubator.production.env.age and invalidating every runbook. Reverted. `--environment` is now the coordinate. `--env` stays OURS: manifest block, binding, age key, store path, team assert. `--environment` is theirs, on the wire, and nothing else. #18 — an absent RESOURCE reported as N missing secrets. The D-237 lie, one level deeper. A resource that is absent reads back exactly like one present with no env vars, so capture reported all 15 required names as individually MISSING — from a box that was serving production and sending mail at that moment — and offered --override as the remedy. Taking that offer would have "worked": a valid store, hand-carried values, and the real finding (the manifest and the box disagree about what the app is called) buried. capture now refuses on the resource, names what does exist, and only reports per-name MISSING for resources it actually found — where it means what it says. #19 — cast inventory: see the box before you adopt it. The missing first step. cast could describe a box it built, change one, and take values off one for names a manifest declares — but not tell you what is on a box you did not build, which is the first thing adoption needs. Every mismatch above surfaced as a refusal from a verb already committed to a course of action, and the tempting fix for two of them was to bend the manifest toward the legacy box. inventory reads resources and env var KEYS (never values), sorts them into on-both / manifest-only / box-only, and needs no store, no age key and no recipient — it runs before adoption exists. Its output is a document: inventory → human reads → manifest PR → capture → apply That boundary is what lets capture stay strict. inventory may read everything, because a person reads its output. capture may only write what the manifest declares, because `apply` reads its output. Same box, two consumers, two contracts. A manifest-draft emitter is deliberately NOT included: it would be one `cp` away from becoming desired state, which is the failure this design exists to prevent. Zero drift against a hand-built box is reported as suspicious, not as a pass. npm run check + build clean; 164 tests passing, 18 files (was 151/16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 18:20:53 +00:00
].join("\n");
}
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
function groupByRef(required: RequiredSecret[]): Map<string, Site[]> {
const byRef = new Map<string, Site[]>();
for (const { ref, resource, key } of required) {
const sites = byRef.get(ref) ?? [];
sites.push({ resource, key });
byRef.set(ref, sites);
}
return byRef;
}
// Force disposition, never guess. Every name the manifest requires lands in
// exactly one of four buckets, and two of them stop the run.
//
// The mapping is deliberately NOT a mechanical dump of the source box: some
// entries encode migration decisions rather than facts about the source. A
// "capture everything" verb would be wrong in a handful of entries out of
// seventeen, silently — which is worse than being wrong in all of them.
export function classify(
required: RequiredSecret[],
generated: string[],
live: LiveEnvs,
overrides: Record<string, string>,
): Classification {
const generatedSet = new Set(generated);
const plan: Disposition[] = [];
const missing: Classification["missing"] = [];
const conflicts: Classification["conflicts"] = [];
for (const [ref, sites] of groupByRef(required)) {
// The operator's word beats both the manifest and the source box: this is
// the escape hatch for a value that must NOT be carried over (staging's
// ADMIN_EMAIL, where the source's value is a real founder and staging
// shares a Mailgun domain with prod).
if (ref in overrides) {
plan.push({
ref,
provenance: "overridden",
value: overrides[ref],
sites,
});
continue;
}
if (generatedSet.has(ref)) {
plan.push({
ref,
provenance: "generated",
value: GENERATED_PLACEHOLDER,
sites,
});
continue;
}
// Captured: read the live value off whichever resources declare it.
const found = sites
.map((s) => ({ site: s, value: live[s.resource]?.[s.key] }))
.filter((f): f is { site: Site; value: string } => f.value !== undefined);
if (found.length === 0) {
missing.push({ ref, sites });
continue;
}
const distinct = new Set(found.map((f) => f.value));
if (distinct.size > 1) {
conflicts.push({ ref, values: found.map((f) => f.site) });
continue;
}
plan.push({
ref,
provenance: "captured",
value: found[0].value,
sites,
});
}
return { plan, missing, conflicts };
}
const site = (s: Site) => `${s.resource}.${s.key}`;
// Names and provenance. NEVER values.
//
// The one thing printed that looks like a value is GENERATED_PLACEHOLDER,
// which is a literal constant in this file and carries no information about
// the source box. Everything else is a name the manifest already declares in
// plaintext, in a committed file.
export function renderCapturePlan(
c: Classification,
ctx: {
orgRepo: string;
env: string;
instance: string;
store: string;
recipient: string;
},
): string {
const lines = [
`capture plan — ${ctx.orgRepo} ${ctx.env}`,
"",
` source: instance ${ctx.instance} (live values read from it)`,
` store: ${ctx.store}`,
` recipient: ${ctx.recipient}`,
"",
];
const width = Math.max(
0,
...[...c.plan, ...c.missing, ...c.conflicts].map((d) => d.ref.length),
);
for (const d of c.plan) {
const where = d.sites.map(site).join(", ");
const note =
d.provenance === "generated"
? `${GENERATED_PLACEHOLDER}`
: d.provenance === "overridden"
? ` (from CAST_CAPTURE_${d.ref})`
: "";
lines.push(
` ${d.ref.padEnd(width)} ${d.provenance.padEnd(10)} ${where}${note}`,
);
}
for (const m of c.missing) {
lines.push(
` ${m.ref.padEnd(width)} MISSING required by ${m.sites.map(site).join(", ")}, absent from the source`,
);
}
for (const c2 of c.conflicts) {
lines.push(
` ${c2.ref.padEnd(width)} CONFLICT differs between ${c2.values.map(site).join(" and ")}`,
);
}
const counts = (["captured", "generated", "overridden"] as const)
.map((p) => [p, c.plan.filter((d) => d.provenance === p).length] as const)
.filter(([, n]) => n > 0)
.map(([p, n]) => `${n} ${p}`)
.join(", ");
lines.push(
"",
`${c.plan.length} name(s) to write${counts ? `: ${counts}` : ""}`,
);
if (c.missing.length > 0) {
lines.push(
"",
`refusing to write the store: ${c.missing.length} name(s) the manifest requires are not`,
"present on the source. An empty value substitutes to nothing and the app boots",
"misconfigured — plausibly, and silently. Supply each one with --override <NAME>",
"(its value is read from CAST_CAPTURE_<NAME>, never from argv), or fix the source.",
);
}
if (c.conflicts.length > 0) {
lines.push(
"",
`refusing to write the store: ${c.conflicts.length} name(s) carry different values on`,
"different resources of the source. The store holds one value per name, and cast",
"will not pick for you. Reconcile them on the source, or pin one with --override.",
);
}
return lines.join("\n");
}