heavy-duty/cast
7
0
Fork 0
Code Issues 4 Pull requests Projects Releases 3 Packages Wiki Activity Actions

feat: read-side coordinates (#17, #18) + cast inventory (#19)

Browse source 
Three fixes at one seam: cast could not READ a box it did not build.

#17 — the environment had no read-side coordinate.

`--project` exists because a hand-built project is called whatever someone
typed. The environment has the identical problem and had no flag, so reading
a legacy box forced a choice between mutating that box's UI and renaming OUR
environment to match it. The second is what happened: `prod` became
`production` across the manifest and environments.yaml — a box being deleted
next week naming the environment of the box that replaces it, permanently
(apply creates the environment from --env), moving the store to
incubator.production.env.age and invalidating every runbook. Reverted.

`--environment` is now the coordinate. `--env` stays OURS: manifest block,
binding, age key, store path, team assert. `--environment` is theirs, on the
wire, and nothing else.

#18 — an absent RESOURCE reported as N missing secrets.

The D-237 lie, one level deeper. A resource that is absent reads back exactly
like one present with no env vars, so capture reported all 15 required names
as individually MISSING — from a box that was serving production and sending
mail at that moment — and offered --override as the remedy. Taking that offer
would have "worked": a valid store, hand-carried values, and the real finding
(the manifest and the box disagree about what the app is called) buried.

capture now refuses on the resource, names what does exist, and only reports
per-name MISSING for resources it actually found — where it means what it says.

#19 — cast inventory: see the box before you adopt it.

The missing first step. cast could describe a box it built, change one, and
take values off one for names a manifest declares — but not tell you what is
on a box you did not build, which is the first thing adoption needs. Every
mismatch above surfaced as a refusal from a verb already committed to a course
of action, and the tempting fix for two of them was to bend the manifest toward
the legacy box.

inventory reads resources and env var KEYS (never values), sorts them into
on-both / manifest-only / box-only, and needs no store, no age key and no
recipient — it runs before adoption exists. Its output is a document:

  inventory → human reads → manifest PR → capture → apply

That boundary is what lets capture stay strict. inventory may read everything,
because a person reads its output. capture may only write what the manifest
declares, because `apply` reads its output. Same box, two consumers, two
contracts. A manifest-draft emitter is deliberately NOT included: it would be
one `cp` away from becoming desired state, which is the failure this design
exists to prevent.

Zero drift against a hand-built box is reported as suspicious, not as a pass.

npm run check + build clean; 164 tests passing, 18 files (was 151/16).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
claude-hdb 2026-07-13 18:20:53 +00:00
parent c6e7d44413
commit 18c4743470
8 changed files with 915 additions and 22 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

55
README.md
View file

@ -63,6 +63,7 @@ Pass it with `--state <dir>`, or set `CAST_STATE`. Defaults to the cwd.
cast apply <org>/<repo> --env <env> [--path <dir>] [--hostname-overlay <file>]
cast diff <org>/<repo> --env <env> [--full]
cast capture <org>/<repo> --env <env> [--generated <NAME>] [--override <NAME>]
cast inventory <org>/<repo> --env <env>
cast server add <name> --ip <ip> --key <file> --env <env> [--user root] [--port 22]
cast smoke --env <env>
cast team [--env <env>]
@ -75,6 +76,12 @@ cast team [--env <env>]
default branch).
- **`diff`** — reports drift, manifest → Coolify. Structural by default; `--full`
also compares env vars. Exits non-zero when dirty, so CI can gate on it.
- **`inventory`** — what is actually *on* a box, and how it lines up with the
manifest: resources and env var **keys** (never values), sorted into
on-both / manifest-only / box-only. Needs no store, no age key, and no
recipient — it runs *before* adoption, which is the point of it. A document,
read by a person; nothing here is consumed by `apply`. See *Adopting a
hand-built instance*.
- **`capture`** — the adoption path: reads a hand-built instance's live env and
writes the environment's age store from it. See *Adopting a hand-built
instance* below.
@ -155,8 +162,26 @@ assert. It is the most consequential input to any run, and the least visible.
## Adopting a hand-built instance
cast is otherwise scoped to the steady state: manifest → Coolify, forever.
`capture` is the one-way-in — it bootstraps an environment's age store from an
instance that was built by hand, before any manifest existed.
Adoption is the one way in, and it has two verbs and a fixed order:
**`inventory` → you read it → a manifest PR → `capture` → `apply`**
**Look before you adopt.** A box nobody declared does not use your vocabulary:
its project is called whatever someone typed, its environment is Coolify's
default (`production`, not `prod`), and its resources are named by whoever
clicked *New Resource* that afternoon. `inventory` shows you both sides at once,
so those differences arrive together, as a document — instead of one at a time,
as refusals from a verb that is already halfway through a migration.
```sh
cast inventory heavy-duty/incubator --env prod --instance legacy \
--project Incubator --environment production
```
It never reads a value, needs no store and no key, and its output is **not**
desired state. What you do with it is decide, resource by resource and key by
key, what the manifest should *gain* and what is cruft that must not travel —
and land that as a manifest PR. Only then:
```sh
CAST_CAPTURE_ADMIN_EMAIL=me@example.com \
@ -178,6 +203,32 @@ values off the instance, and classifies every name:
Then it prints a plan of **names and provenance — never values** — and waits for
you to type the environment's name.
Before any of that, it checks that the resources the manifest names **exist**.
An absent resource reads back exactly like one with no env vars set: every name
it declares reports *missing*, and `--override` would then have you hand-carry
values that are sitting right there under a different name — writing a perfectly
valid store while the actual finding (the manifest and the box disagree about
what this thing is called) is never discovered. So a resource that isn't there
refuses, and names what is. `inventory` is how you reconcile it.
### Three names that are not yours
A hand-built box names things without asking you, at three levels, and cast takes
each as a coordinate to *read* with — never as a reason to rename anything of
yours:
| flag | when |
| --- | --- |
| `--project <name>` | the project isn't named after the repo (`Incubator`, not `incubator`) |
| `--environment <name>` | the environment isn't named after `--env` (Coolify's default is `production`, not `prod`) |
| — | a resource isn't named after the manifest's → **refuses**; reconcile with `inventory` first |
`--env` stays **ours**: it selects the manifest block, the `environments.yaml`
binding, the age key, the store path. `--environment` is *theirs*, on the wire,
and nothing else. Collapsing the two lets a box that is being deleted next week
name the environment of the box that replaces it — `apply` creates the
environment from that value, so it would be inherited permanently.
The mapping is not mechanical, and that is the whole design. A `DATABASE_URL`
copied off the source box points at the *source box's* Postgres: confidently
wrong, entirely plausible, and the target's real URL does not exist until Coolify

48
src/capture.ts
View file

@ -33,6 +33,54 @@ export type Classification = {
conflicts: Array<{ ref: string; values: Site[] }>;
};
// The manifest resources that no resource of that name exists for on the source.
//
// This is the absent-target lie (#12/D-237) one level deeper in the tree, and it
// fails in exactly the same way: a resource that is ABSENT reads back
// identically to one that is PRESENT with no env vars set. Every name it
// declares reports MISSING, the run refuses, and the message tells the operator
// to supply each of them with --override — which would "work", writing a
// perfectly valid store, while the real finding (the manifest and the box
// disagree about what this resource is called) is never discovered and the
// operator hand-carries values that were sitting right there under another name.
//
// So: refuse on the RESOURCE first, and only report per-name MISSING for
// resources that were actually found — where it means what it says.
export function absentResources(
required: RequiredSecret[],
liveNames: Iterable<string>,
): string[] {
const live = new Set(liveNames);
const declared = new Set(required.map((r) => r.resource));
return [...declared].filter((name) => !live.has(name)).sort();
}
export function renderAbsentResources(
absent: string[],
live: Array<{ kind: string; name: string }>,
ctx: { project: string; environment: string },
): string {
const width = Math.max(0, ...live.map((l) => l.kind.length));
return [
`refusing to capture: the manifest declares ${absent.length} resource(s) that do not exist here`,
"",
` looked in: project "${ctx.project}", environment "${ctx.environment}"`,
` looked for: ${absent.join(", ")}`,
" exists here:",
...(live.length > 0
? live.map((l) => ` ${l.kind.padEnd(width)} ${l.name}`)
: [" (nothing at all)"]),
"",
"A resource that is absent reads back exactly like one with no env vars set:",
"every name it declares reports MISSING, and --override would then have you",
"hand-carry values that are sitting right there under a different name. The",
"finding is not that the secrets are missing — it is that the manifest and this",
"box disagree about what these resources are called.",
"",
"Reconcile the names first (`cast inventory` shows both sides), then capture.",
].join("\n");
}
function groupByRef(required: RequiredSecret[]): Map<string, Site[]> {
const byRef = new Map<string, Site[]>();
for (const { ref, resource, key } of required) {

177
src/cli.ts
View file

@ -6,9 +6,16 @@ import { parseArgs } from "node:util";
import { parse as parseYaml } from "yaml";
import { type Executor, applyHostnameOverlay, applyPlan } from "./apply.js";
import { githubAppNameFor, loadBindings } from "./bindings.js";
import { type LiveEnvs, classify, renderCapturePlan } from "./capture.js";
import {
type LiveEnvs,
absentResources,
classify,
renderAbsentResources,
renderCapturePlan,
} from "./capture.js";
import {
type CoolifyInstance,
DEFAULT_INSTANCE,
assertWritable,
formatInstance,
loadInstance,
@ -21,8 +28,10 @@ import {
renderDiff,
} from "./diff.js";
import { assertEnvVarPolicy } from "./envtemplate.js";
import { type LiveResource, reconcile, renderInventory } from "./inventory.js";
import {
desiredFromManifest,
manifestResources,
requiredSecrets,
resolveCheckout,
} from "./resolve.js";
@ -36,9 +45,10 @@ import { serverAdd } from "./server.js";
import { smoke } from "./smoke.js";
import { assertTeam, formatTeam } from "./team.js";
const USAGE = `usage: cast apply <org>/<repo> --env <env> [--path <dir>] [--project <name>] [--hostname-overlay <file>]
cast diff <org>/<repo> --env <env> [--full] [--project <name>]
cast capture <org>/<repo> --env <env> [--path <dir>] [--project <name>] [--generated <NAME>] [--override <NAME>] [--force]
const USAGE = `usage: cast apply <org>/<repo> --env <env> [--path <dir>] [--project <name>] [--environment <name>] [--hostname-overlay <file>]
cast diff <org>/<repo> --env <env> [--full] [--project <name>] [--environment <name>]
cast capture <org>/<repo> --env <env> [--path <dir>] [--project <name>] [--environment <name>] [--generated <NAME>] [--override <NAME>] [--force]
cast inventory <org>/<repo> --env <env> [--path <dir>] [--project <name>] [--environment <name>]
cast server add <name> --ip <ip> --key <file> --env <env> [--user root] [--port 22]
cast smoke --env <env>
cast team [--env <env>]
@ -63,6 +73,13 @@ const USAGE = `usage: cast apply <org>/<repo> --env <env> [--path <dir>] [--pr
whatever someone typed; \`diff\` refuses rather than reporting an
absent project as an empty one, and this is how you point it at
the real name.
--environment <name>
the Coolify environment to act on, when it is not named after
--env (the default). Same problem as --project, one level down:
a box built by hand has whatever Coolify defaulted to, which is
\`production\`, not \`prod\`. This changes ONLY the name on the
wire — --env still selects the manifest block, the
environments.yaml binding, the age key and the store path.
capture (adopt a hand-built instance into the age secret store):
--generated <NAME> force NAME to the \`pending-coolify-generated\` placeholder,
@ -311,7 +328,12 @@ export async function fetchLive(
// exists next to it.
export function renderAbsentTarget(
lookup: Extract<LiveLookup, { found: false }>,
ctx: { orgRepo: string; overridden: boolean; verb?: string },
ctx: {
orgRepo: string;
overridden: boolean;
envOverridden?: boolean;
verb?: string;
},
): string {
// `capture` takes the same position as `diff`, and for the same reason: it
// is only ever a claim about something that already exists. Against an
@ -321,6 +343,7 @@ export function renderAbsentTarget(
const origin = ctx.overridden
? "--project"
: `derived from the repo slug ${ctx.orgRepo}`;
const envOrigin = ctx.envOverridden ? "--environment" : "derived from --env";
const head =
lookup.missing === "project"
? [
@ -332,10 +355,10 @@ export function renderAbsentTarget(
: [
`refusing to ${verb}: project "${lookup.project}" has no environment "${lookup.environment}"`,
"",
` looked for: environment "${lookup.environment}" in project "${lookup.project}"`,
" note: cast names environments after --env, so a project built by",
" hand in the Coolify UI may well use a different name for the",
" same tier (Coolify's own default is `production`).",
` looked for: environment "${lookup.environment}" in project "${lookup.project}" (${envOrigin})`,
" note: a project built by hand in the Coolify UI may well use a",
" different name for the same tier — Coolify's own default is",
" `production`, not `prod`.",
];
return [
...head,
@ -347,7 +370,12 @@ export function renderAbsentTarget(
"",
lookup.missing === "project"
? "Pass --project <name> if this instance names it differently."
: "Re-run with --env naming the environment as it exists here.",
: // NOT "rename your environment to match the box". The box does not get to
// name our environments: --env selects the manifest block, the binding, the
// age key and the store path, and a hand-built box being evicted next week
// must not decide any of them. --environment is the coordinate for reading
// it, and it changes nothing on our side of the line.
"Pass --environment <name> if this instance names it differently.",
].join("\n");
}
@ -439,6 +467,7 @@ async function main(): Promise<number> {
path: { type: "string" },
state: { type: "string" },
project: { type: "string" },
environment: { type: "string" },
instance: { type: "string" },
"hostname-overlay": { type: "string" },
full: { type: "boolean", default: false },
@ -458,6 +487,14 @@ async function main(): Promise<number> {
// whatever someone typed. --project overrides that one, and nothing else —
// secrets stay keyed by the repo (a state-repo convention we own).
const projectName = values.project ?? repoShort;
// Exactly the same split, one level down. `--env` is OUR name for the
// environment: it selects the manifest block, the environments.yaml
// binding, the age key, the store path, the team to assert. `--environment`
// is THEIR name for it on the wire, and nothing else. Collapsing the two
// (as cast did until now) means a box built by hand in someone's UI gets to
// name our environment — and since apply creates the environment from this
// value, a legacy box's accident would be inherited by the new one forever.
const coolifyEnv = values.environment ?? envName;
const checkout = resolveCheckout(orgRepo, {
env: envName,
path: values.path,
@ -499,7 +536,7 @@ async function main(): Promise<number> {
const team = await assertTeam(client, binding.team, envName);
console.log(`team ${formatTeam(team)} ✓`);
const mode = command === "apply" || values.full ? "full" : "structural";
const lookup = await fetchLive(client, projectName, envName);
const lookup = await fetchLive(client, projectName, coolifyEnv);
// apply and diff take opposite (and both correct) positions on absence:
// apply is *allowed* to be the thing that brings a project into existence,
// so [] is a legitimate starting point. diff is only ever a claim about
@ -511,6 +548,7 @@ async function main(): Promise<number> {
renderAbsentTarget(lookup, {
orgRepo,
overridden: values.project !== undefined,
envOverridden: values.environment !== undefined,
}),
);
return 2;
@ -530,7 +568,10 @@ async function main(): Promise<number> {
);
const exec = buildExecutor(client, {
projectName,
envName,
// The name the environment gets ON COOLIFY when apply creates it — so an
// apply that adopts an existing hand-named environment writes into that
// one, rather than creating a second environment beside it.
envName: coolifyEnv,
serverUuid,
githubAppUuid,
s3DestinationUuid: binding.s3_destination,
@ -553,6 +594,7 @@ async function main(): Promise<number> {
state: { type: "string" },
path: { type: "string" },
project: { type: "string" },
environment: { type: "string" },
instance: { type: "string" },
generated: { type: "string", multiple: true },
override: { type: "string", multiple: true },
@ -568,6 +610,11 @@ async function main(): Promise<number> {
const stateDir = stateDirFrom(values.state);
const repoShort = orgRepo.split("/")[1];
const projectName = values.project ?? repoShort;
// Their name for the environment, on the wire. The store below stays keyed
// by OUR name (--env) — capture is the verb most likely to be pointed at a
// hand-built box, and the store it writes must not inherit that box's
// vocabulary.
const coolifyEnv = values.environment ?? envName;
const store = secretsFileFor(stateDir, repoShort, envName);
// Never overwrite a store by accident. `apply` never deletes; the verb
// that WRITES the store gets the same disposition, because the thing it
@ -623,22 +670,43 @@ async function main(): Promise<number> {
// as "every secret is missing" against a box that is fine.
const team = await assertTeam(client, binding.team, envName);
console.log(`team ${formatTeam(team)} ✓`);
const lookup = await fetchLive(client, projectName, envName);
const lookup = await fetchLive(client, projectName, coolifyEnv);
if (!lookup.found) {
console.error(
renderAbsentTarget(lookup, {
orgRepo,
overridden: values.project !== undefined,
envOverridden: values.environment !== undefined,
verb: "capture",
}),
);
return 2;
}
// Databases hold no manifest-templated env of their own — their URL is what
// the APPS reference, and that name is generated, not captured.
const envBearing = lookup.live.filter((l) => l.kind !== "database");
// Before reading a single env: does every resource the manifest requires a
// secret FROM actually exist here? An absent resource reads back exactly
// like one with no env vars set — every name it declares reports MISSING —
// and the suggested remedy for MISSING (--override) would then have the
// operator hand-carry values that are sitting right there under a different
// name, burying the real finding. Same lie as the absent project, one level
// deeper. See absentResources.
const absent = absentResources(
required,
envBearing.map((l) => l.name),
);
if (absent.length > 0) {
console.error(
renderAbsentResources(absent, lookup.live, {
project: projectName,
environment: coolifyEnv,
}),
);
return 2;
}
const liveEnvs: LiveEnvs = {};
for (const l of lookup.live) {
// Databases hold no manifest-templated env of their own — their URL is
// what the APPS reference, and that name is generated, not captured.
if (l.kind === "database") continue;
for (const l of envBearing) {
liveEnvs[l.name] = await fetchEnv(client, l);
}
const classification = classify(
@ -677,6 +745,81 @@ async function main(): Promise<number> {
);
return 0;
}
if (command === "inventory") {
const { values, positionals } = parseArgs({
args: rest,
allowPositionals: true,
options: {
env: { type: "string" },
state: { type: "string" },
path: { type: "string" },
project: { type: "string" },
environment: { type: "string" },
instance: { type: "string" },
},
});
const orgRepo = positionals[0];
const envName = values.env;
if (!orgRepo || !envName) {
console.error(USAGE);
return 2;
}
const stateDir = stateDirFrom(values.state);
const repoShort = orgRepo.split("/")[1];
const projectName = values.project ?? repoShort;
const coolifyEnv = values.environment ?? envName;
const bindings = loadBindings(join(stateDir, "environments.yaml"));
const binding = bindings.environments[envName];
if (!binding) {
console.error(`environment ${envName} not in environments.yaml`);
return 2;
}
// Deliberately NOT resolveCheckout's prod ban, no secrets, no age key, no
// age_recipient: inventory runs BEFORE any store exists — that is the whole
// point of it — and it reads nothing it could leak. A read token is enough.
const checkout = resolveCheckout(orgRepo, {
env: envName,
path: values.path,
});
const manifest = manifestResources(checkout, envName);
const { client } = openCoolify(stateDir, values.instance, binding);
// Read-only instances are exactly what this verb is for. It still takes the
// team assert: a wrong-team token reads back nothing, and "nothing" would
// render here as "the box is empty" — the same lie, dressed as a report.
const team = await assertTeam(client, binding.team, envName);
console.log(`team ${formatTeam(team)} ✓`);
const lookup = await fetchLive(client, projectName, coolifyEnv);
if (!lookup.found) {
console.error(
renderAbsentTarget(lookup, {
orgRepo,
overridden: values.project !== undefined,
envOverridden: values.environment !== undefined,
verb: "inventory",
}),
);
return 2;
}
const live: LiveResource[] = [];
for (const l of lookup.live) {
// Keys, never values — see renderInventory. Databases carry no env of
// their own worth reconciling (their URL is what the apps reference).
const envKeys =
l.kind === "database" ? [] : Object.keys(await fetchEnv(client, l));
live.push({ kind: l.kind, name: l.name, envKeys });
}
console.log(
renderInventory(reconcile(manifest, live), {
orgRepo,
env: envName,
instance: values.instance ?? binding.instance ?? DEFAULT_INSTANCE,
project: projectName,
environment: coolifyEnv,
}),
);
// Always 0: this is a report, not a gate. `diff` is the gate.
return 0;
}
if (command === "server" && rest[0] === "add") {
const { values, positionals } = parseArgs({
args: rest.slice(1),

11
src/envtemplate.ts
View file

@ -61,6 +61,17 @@ export function templateRefs(
);
}
// Every env var key a template declares — refs and literals alike. `capture`
// only cares about the ${...} refs (the secrets); `inventory` needs all of them,
// because the question it answers is "what does the manifest put on this
// resource, and what does the box actually have?", and a literal the manifest
// sets (a feature flag, NODE_ENV) is just as much a difference as a secret.
//
// Same parser as everything else in this file — see parseTemplate.
export function templateKeys(text: string): string[] {
return parseTemplate(text).map(({ key }) => key);
}
// An environment may forbid variables by name pattern, declared as
// `environments.<env>.forbidden_var_patterns` in the state repo. The rule is
// PRESENCE, not value: a forbidden var set to "false" still refuses the apply,

173
src/inventory.ts Normal file
View file

@ -0,0 +1,173 @@
import type { ManifestResource } from "./resolve.js";
// `inventory` answers the question every other verb assumes you already
// answered: **what is actually on this box?**
//
// cast can describe a Coolify it built (`diff`), change one (`apply`), and take
// secret values off one for names a manifest already declares (`capture`). None
// of those can tell you what is on a box you did NOT build — and that is the
// first thing anyone needs when adopting an existing deployment. Without it,
// every mismatch surfaces later, one at a time, as a refusal from a verb that is
// already committed to a course of action; and the tempting fix for a refusal is
// to bend the manifest toward the legacy box, which is exactly backwards.
//
// The output is a DOCUMENT, read by a person. It is deliberately not desired
// state, not a store, and not consumed by `apply`:
//
// inventory → human reads → manifest PR → capture → apply
//
// That boundary is what keeps `capture` safe to be strict about. `inventory` may
// read everything, because a person reads its output. `capture` may only ever
// write what the manifest declares, because `apply` reads its output. Same box,
// two consumers, two contracts.
export type LiveResource = {
kind: string;
name: string;
envKeys: string[];
};
export type Matched = {
kind: string;
name: string;
// Declared by the manifest, absent from the box.
manifestOnlyKeys: string[];
// On the box, and the manifest knows nothing about it. Either something the
// manifest must gain, or cruft that must not travel — and only a human can
// say which. That judgment is the whole reason this verb exists.
boxOnlyKeys: string[];
sharedKeys: string[];
};
export type Reconciliation = {
matched: Matched[];
manifestOnly: ManifestResource[];
boxOnly: LiveResource[];
};
const sorted = (xs: Iterable<string>) => [...xs].sort();
export function reconcile(
manifest: ManifestResource[],
live: LiveResource[],
): Reconciliation {
// Matched by NAME, not by kind: a manifest `application` that the box models
// as a `service` is a real and interesting finding, and collapsing it into
// "manifest-only + box-only" would hide the fact that they are the same thing.
const liveByName = new Map(live.map((l) => [l.name, l]));
const matched: Matched[] = [];
const manifestOnly: ManifestResource[] = [];
for (const m of manifest) {
const l = liveByName.get(m.name);
if (!l) {
manifestOnly.push(m);
continue;
}
const boxKeys = new Set(l.envKeys);
const manifestKeys = new Set(m.envKeys);
matched.push({
kind: m.kind === l.kind ? m.kind : `${m.kind} / ${l.kind} on the box`,
name: m.name,
manifestOnlyKeys: sorted(m.envKeys.filter((k) => !boxKeys.has(k))),
boxOnlyKeys: sorted(l.envKeys.filter((k) => !manifestKeys.has(k))),
sharedKeys: sorted(m.envKeys.filter((k) => boxKeys.has(k))),
});
}
const manifestNames = new Set(manifest.map((m) => m.name));
const boxOnly = live.filter((l) => !manifestNames.has(l.name));
return { matched, manifestOnly, boxOnly };
}
// Names and keys. NEVER values — the whole artifact is meant to be read, pasted,
// and committed to a PR discussion, so it must be safe to do all three with.
export function renderInventory(
rec: Reconciliation,
ctx: {
orgRepo: string;
env: string;
instance: string;
project: string;
environment: string;
},
): string {
const lines = [
`inventory — ${ctx.orgRepo} ${ctx.env}`,
"",
` source: instance ${ctx.instance}`,
` project: ${ctx.project}`,
` environment: ${ctx.environment}`,
"",
" Env var KEYS only — no values are read or printed.",
"",
];
const bullet = (kind: string, name: string) =>
` ${kind.padEnd(12)} ${name}`;
lines.push("on the box, and in the manifest");
if (rec.matched.length === 0) {
lines.push(" (nothing matched — see both lists below)");
}
for (const m of rec.matched) {
lines.push(bullet(m.kind, m.name));
if (m.sharedKeys.length > 0) {
lines.push(` both: ${m.sharedKeys.join(", ")}`);
}
if (m.manifestOnlyKeys.length > 0) {
lines.push(` manifest only: ${m.manifestOnlyKeys.join(", ")}`);
}
if (m.boxOnlyKeys.length > 0) {
lines.push(` box only: ${m.boxOnlyKeys.join(", ")}`);
}
}
lines.push("", "in the manifest, NOT on the box");
if (rec.manifestOnly.length === 0) lines.push(" (none)");
for (const m of rec.manifestOnly) {
lines.push(bullet(m.kind, m.name));
if (m.envKeys.length > 0) {
lines.push(` declares: ${sorted(m.envKeys).join(", ")}`);
}
}
lines.push("", "on the box, NOT in the manifest");
if (rec.boxOnly.length === 0) lines.push(" (none)");
for (const l of rec.boxOnly) {
lines.push(bullet(l.kind, l.name));
// A resource the manifest has never heard of: EVERY key on it is box-only,
// and they are the most interesting keys in the report — this is where a
// resource that the manifest calls something else shows up, carrying the
// values `capture` went looking for and could not find.
if (l.envKeys.length > 0) {
lines.push(` carries: ${sorted(l.envKeys).join(", ")}`);
}
}
const drift =
rec.manifestOnly.length +
rec.boxOnly.length +
rec.matched.reduce(
(n, m) => n + m.manifestOnlyKeys.length + m.boxOnlyKeys.length,
0,
);
lines.push(
"",
drift === 0
? "The manifest and this box name the same things. (On a hand-built box, treat"
: `${drift} difference(s) between the manifest and this box.`,
);
if (drift === 0) {
lines.push(
"that with suspicion rather than relief — a box nobody declared agreeing",
"perfectly with a manifest nobody applied is more often a wrong lookup than",
"a true match.)",
);
}
lines.push(
"",
"This is a document, not desired state. Nothing here is read by `apply` — the",
"path from here is: decide what the manifest should GAIN and what is cruft that",
"must not travel, land that as a manifest PR, then `capture` and `apply`.",
);
return lines.join("\n");
}

50
src/resolve.ts
View file

@ -6,6 +6,7 @@ import type { Desired } from "./diff.js";
import {
type ResolvedEnv,
resolveTemplate,
templateKeys,
templateRefs,
} from "./envtemplate.js";
import { loadManifest } from "./manifest.js";
@ -245,6 +246,55 @@ export function requiredSecrets(
return { required, generated };
}
// What the manifest declares for an environment, as names only — no secrets, no
// age key, no store. `inventory` runs BEFORE any of those exist (that is the
// point of it: you read the box before you can possibly have adopted it), so it
// must be able to describe the manifest side without resolving a single value.
export type ManifestResource = {
kind: "application" | "database" | "service";
name: string;
envKeys: string[];
};
export function manifestResources(
checkoutDir: string,
envName: string,
): ManifestResource[] {
const manifest = loadManifest(join(checkoutDir, ".infra", "manifest.yaml"));
const envSpec = manifest.environments[envName];
if (!envSpec) {
throw new Error(
`environment ${envName} not in manifest (has: ${Object.keys(manifest.environments).join(", ") || "none"})`,
);
}
const keysOf = (resource: string, template?: string): string[] => {
if (!template) return [];
const file = join(checkoutDir, ".infra", "env", template);
if (!existsSync(file))
throw new Error(
`env template missing: ${file} (referenced by ${resource})`,
);
return templateKeys(readFileSync(file, "utf8"));
};
return [
...Object.entries(envSpec.applications).map(([name, app]) => ({
kind: "application" as const,
name,
envKeys: keysOf(name, app.env_template),
})),
...Object.entries(envSpec.databases ?? {}).map(([name]) => ({
kind: "database" as const,
name,
envKeys: [],
})),
...Object.entries(envSpec.services ?? {}).map(([name, svc]) => ({
kind: "service" as const,
name,
envKeys: keysOf(name, svc.env_template),
})),
];
}
export function desiredFromManifest(
checkoutDir: string,
envName: string,

124
test/inventory.test.ts Normal file
View file

@ -0,0 +1,124 @@
import { describe, expect, it } from "vitest";
import { absentResources } from "../src/capture.js";
import { reconcile, renderInventory } from "../src/inventory.js";
import type { ManifestResource } from "../src/resolve.js";
const manifest: ManifestResource[] = [
{
kind: "application",
name: "core",
envKeys: ["NODE_ENV", "DATABASE_URL", "MAILGUN_API_KEY"],
},
{ kind: "service", name: "umami", envKeys: ["APP_SECRET"] },
{ kind: "database", name: "postgres", envKeys: [] },
];
describe("reconcile", () => {
it("sorts resources into both / manifest-only / box-only", () => {
const rec = reconcile(manifest, [
{ kind: "application", name: "core", envKeys: ["NODE_ENV"] },
{ kind: "application", name: "landing", envKeys: [] },
]);
expect(rec.matched.map((m) => m.name)).toEqual(["core"]);
expect(rec.manifestOnly.map((m) => m.name)).toEqual(["umami", "postgres"]);
expect(rec.boxOnly.map((l) => l.name)).toEqual(["landing"]);
});
it("splits a matched resource's env KEYS the same three ways", () => {
const rec = reconcile(manifest, [
{
kind: "application",
name: "core",
envKeys: ["NODE_ENV", "MAILGUN_API_KEY", "LEFTOVER_FROM_2019"],
},
]);
const core = rec.matched[0];
expect(core.sharedKeys).toEqual(["MAILGUN_API_KEY", "NODE_ENV"]);
// The manifest wants it; the box has never heard of it.
expect(core.manifestOnlyKeys).toEqual(["DATABASE_URL"]);
// On the box, unknown to the manifest. Either the manifest must gain it or
// it is cruft that must not travel — the judgment this verb exists to serve.
expect(core.boxOnlyKeys).toEqual(["LEFTOVER_FROM_2019"]);
});
it("matches by name across a kind mismatch rather than hiding it", () => {
// A manifest `service` the box models as an `application` is the same thing
// under two vocabularies. Reporting it as manifest-only + box-only would
// read as "two unrelated resources", which is exactly the wrong conclusion.
const rec = reconcile(
[{ kind: "service", name: "umami", envKeys: [] }],
[{ kind: "application", name: "umami", envKeys: [] }],
);
expect(rec.manifestOnly).toEqual([]);
expect(rec.boxOnly).toEqual([]);
expect(rec.matched[0].kind).toContain("service");
expect(rec.matched[0].kind).toContain("application");
});
});
describe("renderInventory", () => {
const ctx = {
orgRepo: "heavy-duty/incubator",
env: "prod",
instance: "box-b",
project: "Incubator",
environment: "production",
};
it("names both sides and every bucket", () => {
const out = renderInventory(
reconcile(manifest, [
{
kind: "application",
name: "incubator-stack",
envKeys: ["MAILGUN_API_KEY"],
},
]),
ctx,
);
// The two names that are NOT ours — the coordinates that made this readable.
expect(out).toContain("Incubator");
expect(out).toContain("production");
// Declared, absent.
expect(out).toContain("core");
// Present, undeclared — the finding that a MISSING-per-name report buries.
expect(out).toContain("incubator-stack");
expect(out).toContain("This is a document, not desired state");
});
it("treats a zero-drift hand-built box as suspicious, not as a pass", () => {
const out = renderInventory(
reconcile(
[{ kind: "application", name: "core", envKeys: [] }],
[{ kind: "application", name: "core", envKeys: [] }],
),
ctx,
);
expect(out).toContain("suspicion rather than relief");
});
});
describe("absentResources", () => {
const required = [
{ ref: "MAILGUN_API_KEY", resource: "core", key: "MAILGUN_API_KEY" },
{ ref: "DATABASE_URL_PROD", resource: "core", key: "DATABASE_URL" },
{ ref: "UMAMI_APP_SECRET", resource: "umami", key: "APP_SECRET" },
];
it("names the manifest resources the box does not have", () => {
expect(absentResources(required, ["incubator-stack", "umami"])).toEqual([
"core",
]);
});
it("is empty when every declaring resource exists", () => {
expect(absentResources(required, ["core", "umami", "landing"])).toEqual([]);
});
it("reports each absent resource once, not once per secret it declares", () => {
// `core` declares two of the three refs. The old failure reported one
// MISSING per NAME (15 of them, in the live incident) and buried the single
// fact that mattered: one resource is called something else here.
expect(absentResources(required, [])).toEqual(["core", "umami"]);
});
});

293
test/read-side-cli.test.ts Normal file
View file

@ -0,0 +1,293 @@
import { execFileSync, spawn } from "node:child_process";
import { existsSync, mkdirSync, mkdtempSync, writeFileSync } from "node:fs";
import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterEach, beforeAll, describe, expect, it } from "vitest";
// The read side, end to end, against a stub Coolify shaped like a box nobody
// declared: its project is `Incubator` (capital I), its environment is
// `production` (Coolify's default, not ours), and its application is called
// `incubator-stack` rather than the manifest's `core`.
//
// Every one of those three is a real name from the live box this work came out
// of, and each one used to fail differently and badly:
//
// project → refused (already fixed, #12)
// environment → refused, and the "obvious" fix was to rename OUR environment
// to match the box — letting a machine due for deletion name
// the new box's environment forever (#17)
// resource → NOT refused: reported every required secret as individually
// MISSING from a box that was serving production at the time,
// and invited --override to hand-carry all of them (#18)
const LIVE_ENV = {
MAILGUN_API_KEY: "key-REAL-MAILGUN-SECRET",
ADMIN_EMAIL: "founder@real-company.com",
LEFTOVER_FROM_2019: "nobody-asked-for-this",
};
let recipient: string;
beforeAll(() => {
const dir = mkdtempSync(join(tmpdir(), "cast-age-"));
const keyFile = join(dir, "age.key");
execFileSync("age-keygen", ["-o", keyFile], { stdio: "pipe" });
recipient = execFileSync("age-keygen", ["-y", keyFile], {
encoding: "utf8",
}).trim();
});
type Stub = { url: string; close: () => Promise<void> };
const stubs: Stub[] = [];
// appName is the knob: `incubator-stack` is the hand-built box (the manifest's
// `core` does not exist on it); `core` is the box whose names happen to line up.
async function stubCoolify(appName: string): Promise<Stub> {
const server = createServer((req, res) => {
const path = (req.url ?? "").replace("/api/v1", "");
const json = (body: unknown) => {
res.writeHead(200, { "content-type": "application/json" });
res.end(JSON.stringify(body));
};
if (path === "/teams/current") return json({ id: 0, name: "Root Team" });
// Named as someone typed it, not as the repo slug would derive it.
if (path === "/projects") return json([{ uuid: "p1", name: "Incubator" }]);
// Coolify's default environment name — ours is `staging`.
if (path === "/projects/p1/production")
return json({ applications: [{ name: appName, uuid: "a1" }] });
if (path === "/applications/a1/envs")
return json(
Object.entries(LIVE_ENV).map(([key, real_value]) => ({
key,
real_value,
value: "REDACTED",
})),
);
res.writeHead(404);
res.end("{}");
});
await new Promise<void>((r) => {
server.listen(0, "127.0.0.1", r);
});
const stub: Stub = {
url: `http://127.0.0.1:${(server.address() as AddressInfo).port}`,
close: () =>
new Promise<void>((r) => {
server.close(() => r());
}),
};
stubs.push(stub);
return stub;
}
afterEach(async () => {
await Promise.all(stubs.splice(0).map((s) => s.close()));
});
const MANIFEST = `project: incubator
environments:
staging:
applications:
core:
source: { repo: heavy-duty/incubator, branch: main }
build: { pack: nixpacks, base_directory: / }
domains: ["http://core.example.com"]
env_template: core.staging.env.template
`;
const TEMPLATE = `NODE_ENV=production
MAILGUN_API_KEY=\${MAILGUN_API_KEY}
ADMIN_EMAIL=\${ADMIN_EMAIL}
`;
function fixture(url: string) {
const checkout = mkdtempSync(join(tmpdir(), "cast-co-"));
mkdirSync(join(checkout, ".infra", "env"), { recursive: true });
writeFileSync(join(checkout, ".infra", "manifest.yaml"), MANIFEST);
writeFileSync(
join(checkout, ".infra", "env", "core.staging.env.template"),
TEMPLATE,
);
const state = mkdtempSync(join(tmpdir(), "cast-state-"));
mkdirSync(join(state, "secrets"));
writeFileSync(
join(state, ".coolify.env"),
`COOLIFY_BASE_URL="${url}"\nCOOLIFY_ACCESS_TOKEN="t"\n`,
);
writeFileSync(
join(state, "environments.yaml"),
[
"environments:",
" staging:",
" server: staging-box",
" team: { id: 0, name: Root Team }",
` age_recipient: ${recipient}`,
"github_apps:",
" incubator: hdb-coolify",
"",
].join("\n"),
);
return {
checkout,
state,
store: join(state, "secrets", "incubator.staging.env.age"),
};
}
function run(
verb: string,
args: string[],
opts: { stdin?: string; env?: Record<string, string> } = {},
): Promise<{ code: number; output: string }> {
return new Promise((resolve) => {
const child = spawn("node", ["dist/cli.js", verb, ...args], {
stdio: ["pipe", "pipe", "pipe"],
env: { ...process.env, ...opts.env },
});
let output = "";
child.stdout.on("data", (d) => {
output += String(d);
});
child.stderr.on("data", (d) => {
output += String(d);
});
child.stdin.end(opts.stdin ?? "");
child.on("close", (code) => resolve({ code: code ?? 0, output }));
});
}
const base = (f: ReturnType<typeof fixture>) => [
"heavy-duty/incubator",
"--env",
"staging",
"--state",
f.state,
"--path",
f.checkout,
];
describe("--environment (the read-side coordinate, #17)", () => {
it("reads the box's environment name while the store keeps OURS", async () => {
const f = fixture((await stubCoolify("core")).url);
const r = await run(
"capture",
[
...base(f),
"--project",
"Incubator",
"--environment",
"production",
"--override",
"ADMIN_EMAIL",
],
{
stdin: "staging\n",
env: { CAST_CAPTURE_ADMIN_EMAIL: "operator@example.com" },
},
);
expect(r.code).toBe(0);
// THE POINT: the box calls it `production`; we call it `staging`. The store
// is keyed by OUR name. A hand-built box does not get to name our
// environments, our store, or our age key — it only gets to be read.
expect(existsSync(f.store)).toBe(true);
expect(
existsSync(join(f.state, "secrets", "incubator.production.env.age")),
).toBe(false);
});
it("refuses an absent environment by naming --environment, not by inviting a rename", async () => {
const f = fixture((await stubCoolify("core")).url);
// No --environment: cast looks for `staging`, the box has `production`.
const r = await run("capture", [...base(f), "--project", "Incubator"]);
expect(r.code).toBe(2);
expect(r.output).toContain('has no environment "staging"');
expect(r.output).toContain("Pass --environment <name>");
// The old message said "re-run with --env naming the environment as it
// exists here" — i.e. adopt the box's vocabulary as our own. That is the
// sentence that cost us a rename across three repos.
expect(r.output).not.toContain("Re-run with --env");
expect(existsSync(f.store)).toBe(false);
});
});
describe("absent resource (#18)", () => {
it("refuses on the RESOURCE, naming what exists, instead of reporting every secret missing", async () => {
const f = fixture((await stubCoolify("incubator-stack")).url);
const r = await run("capture", [
...base(f),
"--project",
"Incubator",
"--environment",
"production",
]);
expect(r.code).toBe(2);
// The finding: the manifest and the box disagree about what this is called.
expect(r.output).toContain("core");
expect(r.output).toContain("incubator-stack");
expect(r.output).toContain("do not exist here");
// NOT the old report — a wall of per-name MISSING against a box that HAS
// every one of those secrets, sitting right there under another name, with
// --override offered as the remedy (which would have written a perfectly
// valid store and buried the real problem forever).
//
// The assertion is that no INDIVIDUAL SECRET is named at all: this is a
// resource-level finding, and reporting it per-secret is what made the
// original failure so convincingly wrong. (Asserting on the word "MISSING"
// would only catch the prose that explains why we are not doing that.)
for (const name of Object.keys(LIVE_ENV)) {
expect(r.output).not.toContain(name);
}
expect(existsSync(f.store)).toBe(false);
});
});
describe("cast inventory (#19)", () => {
it("shows both sides of a hand-built box — and no values", async () => {
const f = fixture((await stubCoolify("incubator-stack")).url);
const r = await run("inventory", [
...base(f),
"--project",
"Incubator",
"--environment",
"production",
]);
expect(r.code).toBe(0);
// Declared, absent from the box.
expect(r.output).toContain("core");
// On the box, undeclared — including a var the manifest never heard of.
expect(r.output).toContain("incubator-stack");
expect(r.output).toContain("LEFTOVER_FROM_2019");
// Keys, never values. This artifact is meant to be pasted into a PR.
for (const value of Object.values(LIVE_ENV)) {
expect(r.output).not.toContain(value);
}
});
it("needs no secret store, no age key, and no age_recipient to run", async () => {
// The whole point: inventory runs BEFORE adoption, when none of those exist.
const f = fixture((await stubCoolify("incubator-stack")).url);
writeFileSync(
join(f.state, "environments.yaml"),
[
"environments:",
" staging:",
" server: staging-box",
" team: { id: 0, name: Root Team }",
"github_apps:",
" incubator: hdb-coolify",
"",
].join("\n"),
);
const r = await run("inventory", [
...base(f),
"--project",
"Incubator",
"--environment",
"production",
]);
expect(r.code).toBe(0);
expect(r.output).toContain("inventory — heavy-duty/incubator staging");
});
});