cast/src/envtemplate.ts

230 lines
9.9 KiB
TypeScript
Raw Normal View History

feat(resolve): derive DATABASE_URL/REDIS_URL from the database cast created (#60) Add a ${resource:<name>.url} env-template ref that resolves to the internal URL of a database the same manifest declares, read back from the live resource's internal_db_url — never stored in the age store, never decrypted, never printed. This deletes the two-pass generated-secret bootstrap for a database's own URL rather than automating it: no placeholder, no stored copy to drift or overwrite, and a rotated password is simply followed on the next apply. Resolution runs in one function (fillDerivedEnv) against two URL maps: at diff time against databases already on the box (so a matching app shows no drift — killing the "secret DATABASE_URL differs" noise that ran on every plan), and in the executor at apply time against a database created earlier in the same run (the from-nothing case; apply acts databases-before-applications, #45). The unresolved sentinel is never written — the executor refuses, rather than write a blank that boots the app pointed at nothing, and re-running once the database is up resolves it as an ordinary update. A ${resource:X.url} naming a database the manifest does not declare, or an attribute other than .url, is a hard plan-time error refused by every verb that opens a template (apply, diff, capture). generated_secrets and the two-pass bootstrap remain for the residual class — a provider-generated value that genuinely is not derivable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 23:46:26 +00:00
// A ${resource:<name>.url} reference — the internal URL of a database THIS
// manifest declares, read back from the resource Coolify created. It is not a
// secret: a secret is something a human authored and the age store holds, while
// this is a FACT about a resource cast itself made, readable from the API that
// made it, at any time, for free (#60). So it never enters the store, never
// enters a terminal, and needs no age key to read back — it is resolved from the
// live box, not decrypted.
export type ResourceRef = { resource: string; attr: string };
// The value a derived var carries until it is resolved against the live
// resource. NEVER a legal thing to WRITE — the executor refuses it, the same
// fail-closed shape as the generated-secret placeholder — because a database URL
// that resolved to nothing boots every consumer pointed at nothing (the exact
// trap fetchGeneratedSources guards with its `never a fill of ""` rule).
//
// Collision-safe by construction, not by exotic bytes: this string only ever
// occupies a `derived` var's value, and such a value is only ever this sentinel
// or a real `postgres://…`/`redis://…` URL that fillDerivedEnv put there — so it
// cannot be mistaken for a resolved value, and `unresolvedDerived` gates on the
// `derived` flag anyway. (A plain string, no NUL byte: a NUL makes git treat the
// whole source file as binary and its diff unreviewable.)
export const DERIVED_UNRESOLVED = "cast:unresolved-derived-resource-url";
// `secret` is true for both a ${SECRET} and a resolved ${resource:…} — both are
// values that must never be printed. `derived` is set (and stays set after
// resolution) so the diff can say "derived from database X" rather than mistaking
// a routine URL change for a secret rotation, and so the executor knows which
// vars to resolve from the live resource before it writes.
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
export type ResolvedEnv = {
feat(resolve): derive DATABASE_URL/REDIS_URL from the database cast created (#60) Add a ${resource:<name>.url} env-template ref that resolves to the internal URL of a database the same manifest declares, read back from the live resource's internal_db_url — never stored in the age store, never decrypted, never printed. This deletes the two-pass generated-secret bootstrap for a database's own URL rather than automating it: no placeholder, no stored copy to drift or overwrite, and a rotated password is simply followed on the next apply. Resolution runs in one function (fillDerivedEnv) against two URL maps: at diff time against databases already on the box (so a matching app shows no drift — killing the "secret DATABASE_URL differs" noise that ran on every plan), and in the executor at apply time against a database created earlier in the same run (the from-nothing case; apply acts databases-before-applications, #45). The unresolved sentinel is never written — the executor refuses, rather than write a blank that boots the app pointed at nothing, and re-running once the database is up resolves it as an ordinary update. A ${resource:X.url} naming a database the manifest does not declare, or an attribute other than .url, is a hard plan-time error refused by every verb that opens a template (apply, diff, capture). generated_secrets and the two-pass bootstrap remain for the residual class — a provider-generated value that genuinely is not derivable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 23:46:26 +00:00
vars: Record<
string,
{ value: string; secret: boolean; derived?: ResourceRef }
>;
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
};
feat(resolve): derive DATABASE_URL/REDIS_URL from the database cast created (#60) Add a ${resource:<name>.url} env-template ref that resolves to the internal URL of a database the same manifest declares, read back from the live resource's internal_db_url — never stored in the age store, never decrypted, never printed. This deletes the two-pass generated-secret bootstrap for a database's own URL rather than automating it: no placeholder, no stored copy to drift or overwrite, and a rotated password is simply followed on the next apply. Resolution runs in one function (fillDerivedEnv) against two URL maps: at diff time against databases already on the box (so a matching app shows no drift — killing the "secret DATABASE_URL differs" noise that ran on every plan), and in the executor at apply time against a database created earlier in the same run (the from-nothing case; apply acts databases-before-applications, #45). The unresolved sentinel is never written — the executor refuses, rather than write a blank that boots the app pointed at nothing, and re-running once the database is up resolves it as an ordinary update. A ${resource:X.url} naming a database the manifest does not declare, or an attribute other than .url, is a hard plan-time error refused by every verb that opens a template (apply, diff, capture). generated_secrets and the two-pass bootstrap remain for the residual class — a provider-generated value that genuinely is not derivable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 23:46:26 +00:00
// A template line, parsed but not resolved: `ref` is set when the whole RHS is a
// single ${NAME} placeholder (a store secret); `resourceRef` when it is a single
// ${resource:<name>.attr} placeholder (a derived value). The two are mutually
// exclusive — a secret name is UPPER_SNAKE, a resource ref starts `resource:`.
export type TemplateVar = {
key: string;
rhs: string;
ref?: string;
resourceRef?: ResourceRef;
};
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
// ONE grammar, shared by both readers of a template — resolveTemplate (which
// needs the values) and templateRefs (which needs only the names). Keeping
// them on separate parsers would let the two drift, and a drift here is not
// cosmetic: `capture` would collect a different set of names than `apply` will
// later demand, which is precisely the "a name silently missed" failure the
// capture verb exists to remove.
function parseTemplate(text: string): TemplateVar[] {
const vars: TemplateVar[] = [];
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
const lines = text.split("\n");
for (let i = 0; i < lines.length; i++) {
const line = lines[i].trim();
if (line === "" || line.startsWith("#")) continue;
const m = line.match(/^([A-Z][A-Z0-9_]*)=(.*)$/);
if (!m)
throw new Error(
`env template line ${i + 1}: expected KEY=value, got "${line}"`,
);
const [, key, rhs] = m;
const placeholder = rhs.match(/^\$\{([A-Z][A-Z0-9_]*)\}$/);
feat(resolve): derive DATABASE_URL/REDIS_URL from the database cast created (#60) Add a ${resource:<name>.url} env-template ref that resolves to the internal URL of a database the same manifest declares, read back from the live resource's internal_db_url — never stored in the age store, never decrypted, never printed. This deletes the two-pass generated-secret bootstrap for a database's own URL rather than automating it: no placeholder, no stored copy to drift or overwrite, and a rotated password is simply followed on the next apply. Resolution runs in one function (fillDerivedEnv) against two URL maps: at diff time against databases already on the box (so a matching app shows no drift — killing the "secret DATABASE_URL differs" noise that ran on every plan), and in the executor at apply time against a database created earlier in the same run (the from-nothing case; apply acts databases-before-applications, #45). The unresolved sentinel is never written — the executor refuses, rather than write a blank that boots the app pointed at nothing, and re-running once the database is up resolves it as an ordinary update. A ${resource:X.url} naming a database the manifest does not declare, or an attribute other than .url, is a hard plan-time error refused by every verb that opens a template (apply, diff, capture). generated_secrets and the two-pass bootstrap remain for the residual class — a provider-generated value that genuinely is not derivable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 23:46:26 +00:00
// A resource ref names a Coolify resource (lower-kebab/snake, as the manifest
// and Coolify spell it) and one dotted attribute. Only `.url` is derivable
// today; an unknown attr is not rejected HERE — parsing stays dumb — but at
// validation time (resolve.ts), where the message can name the declared
// resources, exactly as an undeclared-resource ref is.
const resource = rhs.match(
/^\$\{resource:([a-z0-9][a-z0-9_-]*)\.([a-z_]+)\}$/,
);
vars.push({
key,
rhs,
...(placeholder ? { ref: placeholder[1] } : {}),
...(resource
? { resourceRef: { resource: resource[1], attr: resource[2] } }
: {}),
});
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
}
return vars;
}
export function resolveTemplate(
text: string,
secrets: Record<string, string>,
): ResolvedEnv {
const vars: ResolvedEnv["vars"] = {};
feat(resolve): derive DATABASE_URL/REDIS_URL from the database cast created (#60) Add a ${resource:<name>.url} env-template ref that resolves to the internal URL of a database the same manifest declares, read back from the live resource's internal_db_url — never stored in the age store, never decrypted, never printed. This deletes the two-pass generated-secret bootstrap for a database's own URL rather than automating it: no placeholder, no stored copy to drift or overwrite, and a rotated password is simply followed on the next apply. Resolution runs in one function (fillDerivedEnv) against two URL maps: at diff time against databases already on the box (so a matching app shows no drift — killing the "secret DATABASE_URL differs" noise that ran on every plan), and in the executor at apply time against a database created earlier in the same run (the from-nothing case; apply acts databases-before-applications, #45). The unresolved sentinel is never written — the executor refuses, rather than write a blank that boots the app pointed at nothing, and re-running once the database is up resolves it as an ordinary update. A ${resource:X.url} naming a database the manifest does not declare, or an attribute other than .url, is a hard plan-time error refused by every verb that opens a template (apply, diff, capture). generated_secrets and the two-pass bootstrap remain for the residual class — a provider-generated value that genuinely is not derivable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 23:46:26 +00:00
for (const { key, rhs, ref, resourceRef } of parseTemplate(text)) {
// A derived value is resolved from the live resource, not the store, and not
// here — the resource may not exist yet (a from-nothing apply creates it in
// the same run). So it lands UNRESOLVED, carrying the ref; fillDerivedEnv
// fills it once a URL map is known, and the executor refuses to write one
// that never resolved. This is BEFORE the ref/store branch on purpose: a
// resource ref's RHS is not UPPER_SNAKE, so it would otherwise be mistaken
// for a literal and written through as the text "${resource:…}".
if (resourceRef) {
vars[key] = {
value: DERIVED_UNRESOLVED,
secret: true,
derived: resourceRef,
};
continue;
}
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
if (ref === undefined) {
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
vars[key] = { value: rhs, secret: false };
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
continue;
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
}
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
const value = secrets[ref];
if (value === undefined) {
throw new Error(`secret ${ref} (for ${key}) missing from the age store`);
}
vars[key] = { value, secret: true };
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
}
return { vars };
}
feat(resolve): derive DATABASE_URL/REDIS_URL from the database cast created (#60) Add a ${resource:<name>.url} env-template ref that resolves to the internal URL of a database the same manifest declares, read back from the live resource's internal_db_url — never stored in the age store, never decrypted, never printed. This deletes the two-pass generated-secret bootstrap for a database's own URL rather than automating it: no placeholder, no stored copy to drift or overwrite, and a rotated password is simply followed on the next apply. Resolution runs in one function (fillDerivedEnv) against two URL maps: at diff time against databases already on the box (so a matching app shows no drift — killing the "secret DATABASE_URL differs" noise that ran on every plan), and in the executor at apply time against a database created earlier in the same run (the from-nothing case; apply acts databases-before-applications, #45). The unresolved sentinel is never written — the executor refuses, rather than write a blank that boots the app pointed at nothing, and re-running once the database is up resolves it as an ordinary update. A ${resource:X.url} naming a database the manifest does not declare, or an attribute other than .url, is a hard plan-time error refused by every verb that opens a template (apply, diff, capture). generated_secrets and the two-pass bootstrap remain for the residual class — a provider-generated value that genuinely is not derivable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 23:46:26 +00:00
// Resolve every derived var whose resource now has a known URL, leaving the rest
// unresolved (a from-nothing apply has no live URL to resolve against until the
// database is created — the executor fills those, post-create). Pure, and the
// single place a derived value is turned into a real one, so the diff-time fill
// (against pre-existing live) and the apply-time fill (against a just-created
// resource) can never resolve the same ref two different ways.
//
// `urls` is keyed by the resource's MANIFEST name — the same name the ref
// carries. The caller keys it that way (see fillDesiredDerived / the executor):
// aliasing has already renamed live resources to the manifest's vocabulary by
// the time a URL map is built.
export function fillDerivedEnv(
env: ResolvedEnv,
urls: Record<string, string>,
): ResolvedEnv {
const vars: ResolvedEnv["vars"] = {};
for (const [key, v] of Object.entries(env.vars)) {
const url =
v.derived && v.derived.attr === "url"
? urls[v.derived.resource]
: undefined;
vars[key] =
url !== undefined && url !== ""
? { value: url, secret: true, derived: v.derived }
: v;
}
return { vars };
}
// The derived vars still holding the unresolved sentinel — what the executor
// must resolve from the live box before it can write this env, and what it
// refuses on if it cannot. Names the env key and the resource it derives from;
// never a value (there is none yet, and there never will be one to print).
export function unresolvedDerived(
env: ResolvedEnv,
): Array<{ key: string; resource: string }> {
return Object.entries(env.vars).flatMap(([key, v]) =>
v.derived && v.value === DERIVED_UNRESOLVED
? [{ key, resource: v.derived.resource }]
: [],
);
}
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
// The ${NAME} refs a template declares: the secret names the manifest requires,
// paired with the env var each one lands on. `capture` reads these to learn
// what to go and fetch — at capture time there is no store to resolve against
// yet, which is the whole point of the verb.
export function templateRefs(
text: string,
): Array<{ key: string; ref: string }> {
return parseTemplate(text).flatMap(({ key, ref }) =>
ref === undefined ? [] : [{ key, ref }],
);
}
feat(resolve): derive DATABASE_URL/REDIS_URL from the database cast created (#60) Add a ${resource:<name>.url} env-template ref that resolves to the internal URL of a database the same manifest declares, read back from the live resource's internal_db_url — never stored in the age store, never decrypted, never printed. This deletes the two-pass generated-secret bootstrap for a database's own URL rather than automating it: no placeholder, no stored copy to drift or overwrite, and a rotated password is simply followed on the next apply. Resolution runs in one function (fillDerivedEnv) against two URL maps: at diff time against databases already on the box (so a matching app shows no drift — killing the "secret DATABASE_URL differs" noise that ran on every plan), and in the executor at apply time against a database created earlier in the same run (the from-nothing case; apply acts databases-before-applications, #45). The unresolved sentinel is never written — the executor refuses, rather than write a blank that boots the app pointed at nothing, and re-running once the database is up resolves it as an ordinary update. A ${resource:X.url} naming a database the manifest does not declare, or an attribute other than .url, is a hard plan-time error refused by every verb that opens a template (apply, diff, capture). generated_secrets and the two-pass bootstrap remain for the residual class — a provider-generated value that genuinely is not derivable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 23:46:26 +00:00
// The ${resource:<name>.attr} refs a template declares — the derived edges. NOT
// returned by templateRefs above, and that separation is the point: a derived
// value is not a secret to be captured, so `capture` must never go looking for a
// store name called `resource:postgres.url`. resolve.ts reads these to validate
// each edge against the databases the manifest actually declares.
export function templateResourceRefs(
text: string,
): Array<{ key: string; resource: string; attr: string }> {
return parseTemplate(text).flatMap(({ key, resourceRef }) =>
resourceRef === undefined
? []
: [{ key, resource: resourceRef.resource, attr: resourceRef.attr }],
);
}
feat: read-side coordinates (#17, #18) + cast inventory (#19) Three fixes at one seam: cast could not READ a box it did not build. #17 — the environment had no read-side coordinate. `--project` exists because a hand-built project is called whatever someone typed. The environment has the identical problem and had no flag, so reading a legacy box forced a choice between mutating that box's UI and renaming OUR environment to match it. The second is what happened: `prod` became `production` across the manifest and environments.yaml — a box being deleted next week naming the environment of the box that replaces it, permanently (apply creates the environment from --env), moving the store to incubator.production.env.age and invalidating every runbook. Reverted. `--environment` is now the coordinate. `--env` stays OURS: manifest block, binding, age key, store path, team assert. `--environment` is theirs, on the wire, and nothing else. #18 — an absent RESOURCE reported as N missing secrets. The D-237 lie, one level deeper. A resource that is absent reads back exactly like one present with no env vars, so capture reported all 15 required names as individually MISSING — from a box that was serving production and sending mail at that moment — and offered --override as the remedy. Taking that offer would have "worked": a valid store, hand-carried values, and the real finding (the manifest and the box disagree about what the app is called) buried. capture now refuses on the resource, names what does exist, and only reports per-name MISSING for resources it actually found — where it means what it says. #19 — cast inventory: see the box before you adopt it. The missing first step. cast could describe a box it built, change one, and take values off one for names a manifest declares — but not tell you what is on a box you did not build, which is the first thing adoption needs. Every mismatch above surfaced as a refusal from a verb already committed to a course of action, and the tempting fix for two of them was to bend the manifest toward the legacy box. inventory reads resources and env var KEYS (never values), sorts them into on-both / manifest-only / box-only, and needs no store, no age key and no recipient — it runs before adoption exists. Its output is a document: inventory → human reads → manifest PR → capture → apply That boundary is what lets capture stay strict. inventory may read everything, because a person reads its output. capture may only write what the manifest declares, because `apply` reads its output. Same box, two consumers, two contracts. A manifest-draft emitter is deliberately NOT included: it would be one `cp` away from becoming desired state, which is the failure this design exists to prevent. Zero drift against a hand-built box is reported as suspicious, not as a pass. npm run check + build clean; 164 tests passing, 18 files (was 151/16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 18:20:53 +00:00
// Every env var key a template declares — refs and literals alike. `capture`
// only cares about the ${...} refs (the secrets); `inventory` needs all of them,
// because the question it answers is "what does the manifest put on this
// resource, and what does the box actually have?", and a literal the manifest
// sets (a feature flag, NODE_ENV) is just as much a difference as a secret.
//
// Same parser as everything else in this file — see parseTemplate.
export function templateKeys(text: string): string[] {
return parseTemplate(text).map(({ key }) => key);
}
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
// An environment may forbid variables by name pattern, declared as
// `environments.<env>.forbidden_var_patterns` in the state repo. The rule is
// PRESENCE, not value: a forbidden var set to "false" still refuses the apply,
// because a var that exists can be flipped on later in the Coolify UI without
// touching a manifest — "off" has to mean absent.
//
// The policy lives in the operator's private state, never in a product's
// manifest: a product-side change must not be able to lower its own guard.
export function assertEnvVarPolicy(
envName: string,
resolved: Record<string, ResolvedEnv>,
forbiddenPatterns: string[] | undefined,
): void {
if (!forbiddenPatterns?.length) return;
const patterns = forbiddenPatterns.map((p) => ({
src: p,
re: new RegExp(p),
}));
for (const [resource, env] of Object.entries(resolved)) {
for (const key of Object.keys(env.vars)) {
const hit = patterns.find((p) => p.re.test(key));
if (hit) {
throw new Error(
`refusing ${envName} apply: ${key} is present on ${resource} regardless of value — forbidden by forbidden_var_patterns /${hit.src}/ ("off" means absent, not false)`,
);
}
}
}
}