cast/src/resolve.ts

396 lines
15 KiB
TypeScript
Raw Normal View History

feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
import { execFileSync } from "node:child_process";
import { existsSync, mkdtempSync, readFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import type { Desired } from "./diff.js";
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
import {
type ResolvedEnv,
resolveTemplate,
feat: read-side coordinates (#17, #18) + cast inventory (#19) Three fixes at one seam: cast could not READ a box it did not build. #17 — the environment had no read-side coordinate. `--project` exists because a hand-built project is called whatever someone typed. The environment has the identical problem and had no flag, so reading a legacy box forced a choice between mutating that box's UI and renaming OUR environment to match it. The second is what happened: `prod` became `production` across the manifest and environments.yaml — a box being deleted next week naming the environment of the box that replaces it, permanently (apply creates the environment from --env), moving the store to incubator.production.env.age and invalidating every runbook. Reverted. `--environment` is now the coordinate. `--env` stays OURS: manifest block, binding, age key, store path, team assert. `--environment` is theirs, on the wire, and nothing else. #18 — an absent RESOURCE reported as N missing secrets. The D-237 lie, one level deeper. A resource that is absent reads back exactly like one present with no env vars, so capture reported all 15 required names as individually MISSING — from a box that was serving production and sending mail at that moment — and offered --override as the remedy. Taking that offer would have "worked": a valid store, hand-carried values, and the real finding (the manifest and the box disagree about what the app is called) buried. capture now refuses on the resource, names what does exist, and only reports per-name MISSING for resources it actually found — where it means what it says. #19 — cast inventory: see the box before you adopt it. The missing first step. cast could describe a box it built, change one, and take values off one for names a manifest declares — but not tell you what is on a box you did not build, which is the first thing adoption needs. Every mismatch above surfaced as a refusal from a verb already committed to a course of action, and the tempting fix for two of them was to bend the manifest toward the legacy box. inventory reads resources and env var KEYS (never values), sorts them into on-both / manifest-only / box-only, and needs no store, no age key and no recipient — it runs before adoption exists. Its output is a document: inventory → human reads → manifest PR → capture → apply That boundary is what lets capture stay strict. inventory may read everything, because a person reads its output. capture may only write what the manifest declares, because `apply` reads its output. Same box, two consumers, two contracts. A manifest-draft emitter is deliberately NOT included: it would be one `cp` away from becoming desired state, which is the failure this design exists to prevent. Zero drift against a hand-built box is reported as suspicious, not as a pass. npm run check + build clean; 164 tests passing, 18 files (was 151/16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 18:20:53 +00:00
templateKeys,
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
templateRefs,
} from "./envtemplate.js";
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
import { loadManifest } from "./manifest.js";
fix: authenticate clones via gh / token, never fall into git's prompt (#13) resolveCheckout shelled out to a bare `git clone` and relied entirely on the ambient credential helper. On a workstation with none configured, git falls through to its interactive username/password prompt — which GitHub no longer accepts — and the resulting error talks about *the repository* rather than about cast's missing credentials. Being logged into `gh` does not help: `gh auth login` alone does not wire git's helper (that is `gh auth setup-git`, a separate act most people never run). Not routable around for prod: resolveCheckout refuses --path with --env prod, so the clone is the only path and its auth is mandatory. cast now resolves credentials itself, in order: `gh` borrowed as a per-invocation credential helper (no mutation of the user's global git config), then GITHUB_TOKEN / GH_TOKEN, then the ambient helper. The token is never embedded in the clone URL or in http.extraheader — both leak it into `ps`, and the latter persists it into the clone's git config. The helper reads it from the environment at run time, so what lands in argv is the literal text `$CAST_GIT_TOKEN`, never its value. GIT_TERMINAL_PROMPT=0 on every path: whichever credential was used, git may never fall through to a prompt it cannot satisfy — it can only hang, or hide the real fault. When there were no credentials at all, cast now says so, and names the fix. Note that the empty `credential.helper=` reset clears URL-scoped helpers (`credential.https://github.com.helper`, what `gh auth setup-git` writes) as well as generic ones — verified against a live private clone, along with all three acceptance criteria. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:30:04 +00:00
// How cast authenticated (or failed to authenticate) a clone.
//
// gh — `gh` is installed and holds a token; borrowed as a credential
// helper for this invocation only
// token — GITHUB_TOKEN / GH_TOKEN in the environment (the CI path)
// ambient — neither; whatever git's own credential helper does, if anything
export type GitAuth = {
source: "gh" | "token" | "ambient";
configArgs: string[];
env: Record<string, string>;
};
// A credential helper reads the token from the ENVIRONMENT at run time. The
// alternatives both leak it: a token in the clone URL shows up in `ps` and in
// git's own error messages, and `http.extraheader` additionally persists into
// the clone's .git/config. What lands in argv here is the literal text
// `$CAST_GIT_TOKEN`, never its value.
const TOKEN_HELPER =
'!f() { test "$1" = get || exit 0; echo username=x-access-token; echo "password=$CAST_GIT_TOKEN"; }; f';
// `gh auth login` alone does NOT wire git's credential helper — that is
// `gh auth setup-git`, a separate act most people never run. So being logged
// into `gh` does not make `git clone` work, which is exactly the trap #13
// fell into. Borrowing gh as a helper for this one invocation closes that gap
// without mutating the operator's global git config.
const GH_HELPER = "!gh auth git-credential";
function ghHasToken(): boolean {
try {
// A local keyring/config read, not a network call. We never keep the
// value — the helper re-reads it inside git.
execFileSync("gh", ["auth", "token"], { stdio: "pipe" });
return true;
} catch {
return false;
}
}
// Resolve clone credentials INSIDE cast, in a fixed order, rather than leaving
// it to whatever the ambient git config happens to do. `credential.helper=`
// (empty) first RESETS the inherited helper list — otherwise a helper
// configured globally is consulted before ours and silently decides the
// outcome, which is the same "the connection target is implicit in a file's
// contents" problem #14 is about.
export function resolveGitAuth(
env: NodeJS.ProcessEnv = process.env,
hasGh: () => boolean = ghHasToken,
): GitAuth {
if (hasGh()) {
return {
source: "gh",
configArgs: [
"-c",
"credential.helper=",
"-c",
`credential.helper=${GH_HELPER}`,
],
env: {},
};
}
const token = env.GITHUB_TOKEN || env.GH_TOKEN;
if (token) {
return {
source: "token",
configArgs: [
"-c",
"credential.helper=",
"-c",
`credential.helper=${TOKEN_HELPER}`,
],
env: { CAST_GIT_TOKEN: token },
};
}
return { source: "ambient", configArgs: [], env: {} };
}
// GitHub answers "you cannot see this" with a 404, not a 403 — so a private
// repo you lack access to and a repo that does not exist are the same message
// on the wire. The failure text must not pick one; it has to name both, and
// name the credential cast actually used, or the operator debugs the wrong
// half. (The original bug reported *the repository* when the real fault was
// cast's missing credentials.)
export function cloneFailureMessage(
orgRepo: string,
auth: GitAuth,
stderr: string,
): string {
const detail = stderr.trim();
const tail = detail
? ["", "git said:", ...detail.split("\n").map((l) => ` ${l}`)]
: [];
if (auth.source === "ambient") {
return [
`cannot clone ${orgRepo}: no GitHub credentials.`,
"",
"cast looked for, in order:",
" 1. `gh` — not installed, or not logged in (`gh auth token` failed)",
" 2. GITHUB_TOKEN / GH_TOKEN — not set in the environment",
" 3. git's own credential helper — did not supply credentials either",
"",
"Run `gh auth login`, or set GITHUB_TOKEN. (`gh auth setup-git` also works,",
"but cast borrows `gh` as a credential helper on its own, so logging in is",
"enough — you do not need to change your global git config.)",
...tail,
].join("\n");
}
const used =
auth.source === "gh"
? "`gh` (borrowed as a credential helper for this clone)"
: "GITHUB_TOKEN / GH_TOKEN from the environment";
return [
`cannot clone ${orgRepo}: authenticated with ${used}, and GitHub still refused.`,
"",
"GitHub answers 'you cannot see this' with a 404, so this is one of:",
` - ${orgRepo} does not exist (check the slug)`,
" - it is private and this credential has no access to it",
" - the credential is expired, or lacks the `repo` scope",
...tail,
].join("\n");
}
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
export function resolveCheckout(
orgRepo: string,
opts: { env: string; path?: string },
): string {
if (opts.path && opts.env === "prod") {
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
// Holds for every verb that reads a manifest (apply, and now capture): a
// feature-branch checkout must not be able to decide what prod runs, nor
// which secret names land in prod's store.
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
throw new Error(
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
"refuses --path with --env prod: prod always reads the default branch",
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
);
}
if (opts.path) return opts.path;
const dir = mkdtempSync(join(tmpdir(), "infra-checkout-"));
fix: authenticate clones via gh / token, never fall into git's prompt (#13) resolveCheckout shelled out to a bare `git clone` and relied entirely on the ambient credential helper. On a workstation with none configured, git falls through to its interactive username/password prompt — which GitHub no longer accepts — and the resulting error talks about *the repository* rather than about cast's missing credentials. Being logged into `gh` does not help: `gh auth login` alone does not wire git's helper (that is `gh auth setup-git`, a separate act most people never run). Not routable around for prod: resolveCheckout refuses --path with --env prod, so the clone is the only path and its auth is mandatory. cast now resolves credentials itself, in order: `gh` borrowed as a per-invocation credential helper (no mutation of the user's global git config), then GITHUB_TOKEN / GH_TOKEN, then the ambient helper. The token is never embedded in the clone URL or in http.extraheader — both leak it into `ps`, and the latter persists it into the clone's git config. The helper reads it from the environment at run time, so what lands in argv is the literal text `$CAST_GIT_TOKEN`, never its value. GIT_TERMINAL_PROMPT=0 on every path: whichever credential was used, git may never fall through to a prompt it cannot satisfy — it can only hang, or hide the real fault. When there were no credentials at all, cast now says so, and names the fix. Note that the empty `credential.helper=` reset clears URL-scoped helpers (`credential.https://github.com.helper`, what `gh auth setup-git` writes) as well as generic ones — verified against a live private clone, along with all three acceptance criteria. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:30:04 +00:00
const auth = resolveGitAuth();
try {
execFileSync(
"git",
[
...auth.configArgs,
"clone",
"--depth",
"1",
`https://github.com/${orgRepo}.git`,
dir,
],
{
stdio: "pipe",
env: {
...process.env,
...auth.env,
// Belt and braces: whatever credential path we took, git may NEVER
// fall through to its interactive username/password prompt. GitHub
// stopped accepting passwords there years ago, so it cannot succeed
// — it can only hang cast, or (in the original report) hand back an
// error about the repository that hides the real fault.
GIT_TERMINAL_PROMPT: "0",
},
},
);
} catch (err) {
const stderr = String((err as { stderr?: Buffer | string })?.stderr ?? "");
throw new Error(cloneFailureMessage(orgRepo, auth, stderr));
}
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
return dir;
}
feat: cast capture — adopt a hand-built Coolify into the age secret store (#15) cast was scoped to the steady state: manifest → Coolify, forever. It had no adoption path — no way to bootstrap the age store from an instance built by hand, before any manifest existed. The operator did it by hand: curl the envs, assemble 17 name=value pairs into /dev/shm/prod.env, age -r, shred. Every input to that pipeline is something cast already has, so a human was shuffling cast's own inputs through a terminal, with the leak (scrollback, history, a tmp file that never got shredded) and the silent miss both live. cast capture <org>/<repo> --env <env> [--generated N] [--override N] [--force] The required set comes from the MANIFEST, not the box: the ${...} refs in that environment's env templates, read by the same parser apply uses to demand them. resolveTemplate and templateRefs now share one grammar — a drift between them would mean capture collects a different set than apply later requires, which is exactly the "a name silently missed" failure this verb exists to remove. The mapping is deliberately NOT mechanical. A DATABASE_URL read off the source points at the SOURCE box's Postgres: confidently wrong, entirely plausible, and the target's real URL does not exist until Coolify creates the resource. So the manifest declares `generated_secrets:` and those names are written as the literal `pending-coolify-generated`. staging's ADMIN_EMAIL must be the operator, not the source's — staging and prod share a Mailgun domain, so a staging box carrying the real address can mail real users; that is --override. A "capture everything" verb would be wrong in ~4 of 17 entries, silently — worse than being wrong in all of them. So every name is forced into a disposition, and two of the four stop the run: a name required by a template but absent from the source REFUSES (an empty substitutes to nothing and the app boots misconfigured), as does one name carrying different values on two resources. generated_secrets is a manifest property rather than a flag the operator must remember, because the manifest is what knows DATABASE_URL comes from a database it declares. An entry no template refers to is a hard error: a guard standing over nothing reads like a guard, and the likeliest cause is a typo whose real name is then captured from the source instead of placeheld. Secret hygiene, all covered by tests asserting on real values: - the plan prints names and provenance, NEVER values - an --override's value comes from $CAST_CAPTURE_<NAME>, never argv (`ps`) - plaintext is piped to age on stdin — never a temp file, stdout, or history - an existing store is not overwritten without --force: it may hold the only copy of values the source no longer has (apply's never-delete, applied here) capture inherits diff's absent-target refusal (D-237) — against a project that isn't there it would report every secret as missing, an alarming report about the wrong box — plus the team assert and the --path/--env prod ban. The last gate is a typed confirmation of the environment's name; there is no --yes. The end-to-end test decrypts the store cast wrote and asserts on its contents, so "exactly the names the manifest requires, no more and no fewer" is checked against real ciphertext rather than against cast's own console output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:51:43 +00:00
// One secret the manifest requires: the ${REF} a template names (the key it
// gets in the age store), the resource that needs it, and the env var it lands
// on there. That last pair is what `capture` reads the live value from — the
// store is keyed by REF, but the live box knows it as `resource.key`.
export type RequiredSecret = { ref: string; resource: string; key: string };
// Exactly the set of secret names an environment's manifest demands — the same
// set `apply` will later insist on, read from the same templates by the same
// parser. `capture` uses this to know what to go and fetch; nothing else has to
// be told, and nothing can be silently missed.
//
// Deliberately does NOT take a secrets map: at capture time the store does not
// exist yet. That is the whole point of the verb.
export function requiredSecrets(
checkoutDir: string,
envName: string,
): { required: RequiredSecret[]; generated: string[] } {
const manifest = loadManifest(join(checkoutDir, ".infra", "manifest.yaml"));
const envSpec = manifest.environments[envName];
if (!envSpec) {
throw new Error(
`environment ${envName} not in manifest (has: ${Object.keys(manifest.environments).join(", ") || "none"})`,
);
}
const required: RequiredSecret[] = [];
const collect = (resource: string, template?: string) => {
if (!template) return;
const file = join(checkoutDir, ".infra", "env", template);
if (!existsSync(file))
throw new Error(
`env template missing: ${file} (referenced by ${resource})`,
);
for (const { key, ref } of templateRefs(readFileSync(file, "utf8"))) {
required.push({ ref, resource, key });
}
};
for (const [name, app] of Object.entries(envSpec.applications)) {
collect(name, app.env_template);
}
for (const [name, svc] of Object.entries(envSpec.services ?? {})) {
collect(name, svc.env_template);
}
const generated = envSpec.generated_secrets ?? [];
// A generated_secrets entry naming something no template refs is dead
// config — and dead config in THIS list is not merely untidy, it is
// dangerous: it reads like a guard standing over a name while standing over
// nothing. The likeliest cause is a typo, and the consequence of the typo is
// that the real name gets CAPTURED from the source box instead of placeheld.
const refs = new Set(required.map((r) => r.ref));
const dead = generated.filter((g) => !refs.has(g));
if (dead.length > 0) {
throw new Error(
[
`manifest environment ${envName}: generated_secrets names ${dead.join(", ")}, which no env template refers to`,
"",
` declared: ${generated.join(", ")}`,
` templates: ${[...refs].sort().join(", ") || "(no ${...} refs at all)"}`,
"",
"A generated name that matches nothing guards nothing — and if this is a",
"typo, the name it was meant to guard is being captured from the source",
"box instead of placeheld. Fix the spelling, or drop the entry.",
].join("\n"),
);
}
return { required, generated };
}
feat: read-side coordinates (#17, #18) + cast inventory (#19) Three fixes at one seam: cast could not READ a box it did not build. #17 — the environment had no read-side coordinate. `--project` exists because a hand-built project is called whatever someone typed. The environment has the identical problem and had no flag, so reading a legacy box forced a choice between mutating that box's UI and renaming OUR environment to match it. The second is what happened: `prod` became `production` across the manifest and environments.yaml — a box being deleted next week naming the environment of the box that replaces it, permanently (apply creates the environment from --env), moving the store to incubator.production.env.age and invalidating every runbook. Reverted. `--environment` is now the coordinate. `--env` stays OURS: manifest block, binding, age key, store path, team assert. `--environment` is theirs, on the wire, and nothing else. #18 — an absent RESOURCE reported as N missing secrets. The D-237 lie, one level deeper. A resource that is absent reads back exactly like one present with no env vars, so capture reported all 15 required names as individually MISSING — from a box that was serving production and sending mail at that moment — and offered --override as the remedy. Taking that offer would have "worked": a valid store, hand-carried values, and the real finding (the manifest and the box disagree about what the app is called) buried. capture now refuses on the resource, names what does exist, and only reports per-name MISSING for resources it actually found — where it means what it says. #19 — cast inventory: see the box before you adopt it. The missing first step. cast could describe a box it built, change one, and take values off one for names a manifest declares — but not tell you what is on a box you did not build, which is the first thing adoption needs. Every mismatch above surfaced as a refusal from a verb already committed to a course of action, and the tempting fix for two of them was to bend the manifest toward the legacy box. inventory reads resources and env var KEYS (never values), sorts them into on-both / manifest-only / box-only, and needs no store, no age key and no recipient — it runs before adoption exists. Its output is a document: inventory → human reads → manifest PR → capture → apply That boundary is what lets capture stay strict. inventory may read everything, because a person reads its output. capture may only write what the manifest declares, because `apply` reads its output. Same box, two consumers, two contracts. A manifest-draft emitter is deliberately NOT included: it would be one `cp` away from becoming desired state, which is the failure this design exists to prevent. Zero drift against a hand-built box is reported as suspicious, not as a pass. npm run check + build clean; 164 tests passing, 18 files (was 151/16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 18:20:53 +00:00
// What the manifest declares for an environment, as names only — no secrets, no
// age key, no store. `inventory` runs BEFORE any of those exist (that is the
// point of it: you read the box before you can possibly have adopted it), so it
// must be able to describe the manifest side without resolving a single value.
export type ManifestResource = {
kind: "application" | "database" | "service";
name: string;
envKeys: string[];
};
export function manifestResources(
checkoutDir: string,
envName: string,
): ManifestResource[] {
const manifest = loadManifest(join(checkoutDir, ".infra", "manifest.yaml"));
const envSpec = manifest.environments[envName];
if (!envSpec) {
throw new Error(
`environment ${envName} not in manifest (has: ${Object.keys(manifest.environments).join(", ") || "none"})`,
);
}
const keysOf = (resource: string, template?: string): string[] => {
if (!template) return [];
const file = join(checkoutDir, ".infra", "env", template);
if (!existsSync(file))
throw new Error(
`env template missing: ${file} (referenced by ${resource})`,
);
return templateKeys(readFileSync(file, "utf8"));
};
return [
...Object.entries(envSpec.applications).map(([name, app]) => ({
kind: "application" as const,
name,
envKeys: keysOf(name, app.env_template),
})),
...Object.entries(envSpec.databases ?? {}).map(([name]) => ({
kind: "database" as const,
name,
envKeys: [],
})),
...Object.entries(envSpec.services ?? {}).map(([name, svc]) => ({
kind: "service" as const,
name,
envKeys: keysOf(name, svc.env_template),
})),
];
}
feat: cast — the Coolify executor, extracted from the infra state repo Public tool, private state. cast holds no hostnames, no bindings, no secrets: it joins a product repo's .infra/ manifest with a state directory you point it at, and makes Coolify match. Extracted from heavy-duty/infra, which was half tool and half state — the inconsistency that made it impossible to say whether "infra" named a CLI or a runbook. rig builds the boxes; cast fills them; infra is what they are filled with. Two changes were required to make it genuinely stateless and publishable: - The implicit cwd contract (environments.yaml / secrets/ / .coolify.env resolved against the working directory, silently reading the wrong file from the wrong place) is now an explicit --state <dir> / $CAST_STATE. - BANNED_IN_PROD — a hardcoded list of one product's ALLOW_* flags, the only product knowledge in the executor — becomes the generic, operator- owned environments.<env>.forbidden_var_patterns. The guard now lives in private state, so a product-side change cannot lower its own guard, and it is a pattern rather than a list, so it catches unforeseen siblings. Age identities resolve as $CAST_AGE_KEY_FILE_<ENV> then ~/.config/cast/age-<env>.key — which is the entire attended-vs-unattended apply mechanism, with no environment names known to the tool. Instance identity (org names, the GitHub App name, founder domains) is out of the fixtures and out of register-github-app.sh, which took APP_NAME and ORG as arguments rather than baking them in. 69 tests green; bin/cast + curl installer mirror rig's shape. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 12:25:44 +00:00
export function desiredFromManifest(
checkoutDir: string,
envName: string,
secrets: Record<string, string>,
): {
desired: Desired[];
resolvedEnvs: Record<string, ResolvedEnv>;
backupSchedules: Record<string, { frequency: string; retention: number }>;
} {
const manifest = loadManifest(join(checkoutDir, ".infra", "manifest.yaml"));
const envSpec = manifest.environments[envName];
if (!envSpec) {
throw new Error(
`environment ${envName} not in manifest (has: ${Object.keys(manifest.environments).join(", ") || "none"})`,
);
}
const desired: Desired[] = [];
const resolvedEnvs: Record<string, ResolvedEnv> = {};
const backupSchedules: Record<
string,
{ frequency: string; retention: number }
> = {};
const resolveEnvFile = (
name: string,
template?: string,
): ResolvedEnv | undefined => {
if (!template) return undefined;
const file = join(checkoutDir, ".infra", "env", template);
if (!existsSync(file))
throw new Error(`env template missing: ${file} (referenced by ${name})`);
const env = resolveTemplate(readFileSync(file, "utf8"), secrets);
resolvedEnvs[name] = env;
return env;
};
for (const [name, app] of Object.entries(envSpec.applications)) {
desired.push({
kind: "application",
name,
fields: {
git_repository: app.source.repo,
git_branch: app.source.branch,
build_pack: app.build.pack,
base_directory: app.build.base_directory,
...(app.build.publish_directory
? { publish_directory: app.build.publish_directory }
: {}),
...(app.build.pack === "dockercompose"
? {
docker_compose_location: app.build.compose_file,
docker_compose_domains: app.service_domains,
}
: {
...(app.port !== undefined ? { port: app.port } : {}),
...(app.healthcheck ? { healthcheck: app.healthcheck } : {}),
domains: app.domains,
}),
},
env: resolveEnvFile(name, app.env_template),
});
}
for (const [name, db] of Object.entries(envSpec.databases ?? {})) {
desired.push({
kind: "database",
name,
fields: { type: db.type, ...(db.version ? { version: db.version } : {}) },
});
if (db.backup)
backupSchedules[name] = {
frequency: db.backup.frequency,
retention: db.backup.retention,
};
}
for (const [name, svc] of Object.entries(envSpec.services ?? {})) {
if (svc.domains && svc.domains.length > 0) {
// Coolify 4.1.2's service executor has no flat `domains` concept —
// hostnames live per-container on `urls` (see serviceApiFields in
// cli.ts) — so a manifest-declared service `domains` list is silently
// unhonorable by apply. Warn at build time, once per run, while the
// service name is still in scope.
console.warn(
`service ${name} declares domains (${svc.domains.join(", ")}), but apply cannot set them on Coolify 4.1.2 services — configure hostnames manually in the Coolify UI`,
);
}
desired.push({
kind: "service",
name,
// domains dropped from fields, same as database `backup` above: the
// live side (projectLiveFields in cli.ts) can't read service domains
// and the write side (serviceApiFields) drops them, so keeping
// domains in fields makes every domain-bearing service diff as a
// perpetual update. Hostnames stay a manual Coolify UI act (warned
// above).
fields: { type: svc.type },
env: resolveEnvFile(name, svc.env_template),
});
}
return { desired, resolvedEnvs, backupSchedules };
}