From 18660041f9552a402ecfba32806f99264efdb839 Mon Sep 17 00:00:00 2001 From: claude-hdb Date: Mon, 13 Jul 2026 20:03:35 +0000 Subject: [PATCH] =?UTF-8?q?feat:=20a=20project=20registry=20=E2=80=94=20th?= =?UTF-8?q?e=20list=20of=20what=20exists=20(#25)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit environments.yaml could say where things deploy to, and how a project you have already named is placed once it is there. It could not say which projects exist. "Every project" was a thing the operator remembered — so fleet operations (#26) had nothing to iterate, and rebuild-from-state (#27) was an assumption, since you cannot restore what you cannot enumerate. A new optional top-level block, keyed by the full / slug: projects: heavy-duty/incubator: environments: [prod, staging] The key IS the repo — no `repo:` field, because a second place to write the same string is a second place for it to be wrong. No bare- fallback, unlike github_apps and environments..projects: those carry one because state files in the wild are keyed that way, and this block has none to support. A bare is unique only within an org, which is why it is not a key (#12, twice learned). Validated in loadBindings, so every verb refuses a registry that lies: - an environment no `environments:` block defines is an error — the project would be registered into an environment no command can visit - every environments..projects. binding must be registered for that env, or the two blocks describe two different fleets: a destination or smoke_target real enough for a direct apply, invisible to every fleet run. Only enforced when `projects:` is present, so pre-registry state files keep loading unchanged. Both defend one failure: a silently skipped project reads exactly like a clean one. Errors render multi-line now — zod's own .message is the issue array as JSON, which flattened the refusals into a line of \n escapes. projectsIn(bindings, env) gives an environment's slugs, sorted; [] with no registry. The --all flag that consumes it is #26's, not here. Co-Authored-By: Claude Opus 4.8 --- README.md | 50 +++++++++ docs/semantics.md | 45 ++++++++ src/bindings.ts | 209 +++++++++++++++++++++++++++++++++++++- test/bindings.test.ts | 231 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 533 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 0f43a88..6441a00 100644 --- a/README.md +++ b/README.md @@ -51,6 +51,8 @@ environments.yaml # bindings: the team each env's token must belon # which server it deploys onto, the S3 destination, # GitHub App name, guards — and, per project, # the destination it deploys onto + its smoke target + # …plus `projects:`, the registry: which projects + # exist, and in which environments secrets/..env.age # age-encrypted values for the ${…} placeholders .coolify.env # COOLIFY_BASE_URL + COOLIFY_ACCESS_TOKEN (never commit) .coolify/.env # …the same, for a NAMED instance (see below) @@ -357,6 +359,54 @@ own (it hangs off a project) and no API path scopes by one: **Coolify environments are an organizational construct, not an auth boundary.** The team is the only boundary there is, so it is the one cast asserts. +## The registry: which projects exist + +`environments.yaml` says where things deploy *to*, and how a project you have +already named is placed once it is there. Until the `projects:` block, nothing in +it said **which projects exist at all** — "every project" was a thing the +operator remembered: + +```yaml +projects: + heavy-duty/incubator: + environments: [prod, staging] + acme/client-site: + environments: [prod] +``` + +Keyed by the **full `/` slug**, and the key *is* the repo — there is +no `repo:` field inside, because a second place to write the same string is a +second place for it to be wrong. Unlike `github_apps`, a bare `` key is +**refused** rather than resolved: this block is new, so it has no state files in +the wild to keep working, and a bare `` is unique only *within* an org — +which is exactly why it is not a key. `environments:` lists **our** environment +names (the values `--env` takes), never Coolify's. + +The block is optional; a state file written before it loads unchanged. + +**It has to be true, so cast checks that it is — at parse time, for every verb.** +Two ways it could quietly stop being true, both refused: + +- an environment name that no `environments:` block defines (a typo). The project + is real and its environment imaginary, so a fleet run visits nothing for it, + reports nothing, and exits clean. +- an `environments..projects.` binding — a destination, a smoke target + — in an environment the registry does not register that project for. The two + blocks then describe two different fleets: state real enough for a direct + `cast apply` to use, invisible to every fleet run. (Checked only when + `projects:` is present.) + +Both refusals defend one failure: **a silently skipped project reads exactly like +a clean one.** Silence is the one report that must never be ambiguous. + +What it unlocks, neither of which was possible without a list to iterate: + +- **fleet operations** — `cast diff --all` / `apply --all` over every project in + an environment. +- **rebuild-from-state** — "restore this Coolify from the state repo" cannot even + be *attempted* without knowing what was on it. The registry is the difference + between a documented recovery and an archaeology exercise. + ## Two projects, one box: destinations A **destination** is the Docker network a resource is created on. A server has a diff --git a/docs/semantics.md b/docs/semantics.md index 409bd34..668b284 100644 --- a/docs/semantics.md +++ b/docs/semantics.md @@ -158,6 +158,51 @@ softened by an implementation detail): refuses `--path` combined with `--env prod`: prod always reads the default branch, so a feature-branch checkout can never reach it. +## The registry (`projects:`) + +The top-level `projects:` block is the list of **which projects exist**, and in +which environments. It is the only place that says so: `environments:` says where +things deploy to, `environments..projects.` says how an already-named +project is placed, `github_apps` says how to clone one you have already named. + +```yaml +projects: + heavy-duty/incubator: + environments: [prod, staging] +``` + +- **Keyed by the full `/` slug, with no bare-`` fallback.** The + key is the repo; there is no `repo:` field. `github_apps` and + `environments..projects` accept a bare key because state files in the wild + are written that way; this block is new and has none, so it requires the slug — + a bare `` is unique only *within* an org. +- **`environments:` names OUR environments** — the keys of the `environments:` + block, the values `--env` takes — never Coolify's. Non-empty. +- **Optional.** A state file with no `projects:` block loads unchanged, and + `projectsIn` reports `[]` for every environment. + +**Validated at parse time, so every verb refuses a registry that lies.** Two +refusals, both defending the same failure — *a silently skipped project reads +exactly like a clean one*, which makes silence, the most common report there is, +ambiguous: + +1. **An environment that does not exist** (a typo in `projects..environments`) + is an error naming the unknown environment and listing the known ones. Left + alone, the project would be registered into an environment no command can + visit: a fleet run skips it, reports nothing, exits clean. +2. **A binding the registry does not register.** Every + `environments..projects.` key must be a project the registry + registers *for that environment*. Otherwise the two blocks describe two + different fleets: a `destination_uuid` or `smoke_target` real enough for a + direct `cast apply --env ` to act on, and invisible to every + fleet run over that environment. Enforced **only when `projects:` is + present**, so pre-registry state files keep loading. + +The registry is what makes two things possible, neither of which can be attempted +without a list to iterate: **fleet operations** (`--all`), and +**rebuild-from-state** — restoring a Coolify from the state repo, which is +otherwise an assumption, since you cannot restore what you cannot enumerate. + ## Placement (destinations) A **destination** is the Docker network a resource is created on. It is declared diff --git a/src/bindings.ts b/src/bindings.ts index c414cc0..c0ff16b 100644 --- a/src/bindings.ts +++ b/src/bindings.ts @@ -58,6 +58,32 @@ const ProjectBindingSchema = z export type ProjectBinding = z.infer; +// One project in the registry — the top-level `projects:` block, which is the +// list of what EXISTS. Nothing else in this file says that: `environments:` +// says where things are deployed to, `environments..projects.` says +// how one project is placed once you already know it is there, and +// `github_apps` says how to clone one you have already named. "Every project" +// was, until this block, a thing the operator remembered. +// +// Two things need the list, and neither can be built without it: fleet +// operations (`cast diff --all`, #26 — iterating "every project in this +// environment") and rebuild-from-state (#27 — a Coolify restored from the state +// repo, which cannot even be attempted without knowing what was on it). +const RegisteredProjectSchema = z + .object({ + // OUR environment names — the values `--env` takes, the keys of the + // `environments:` block above — never Coolify's. The distinction is the same + // one `--env` vs `--environment` draws everywhere else in cast. + // + // Non-empty: a project registered into no environment is not a registration, + // it is a line of YAML that reads like one. It would be skipped by every + // fleet run silently. + environments: z.array(z.string().min(1)).nonempty(), + }) + .strict(); + +export type RegisteredProject = z.infer; + const BindingsSchema = z .object({ environments: z.record( @@ -100,6 +126,22 @@ const BindingsSchema = z }) .strict(), ), + // THE REGISTRY: which projects exist at all, keyed by the full `/` + // slug. The key IS the repo — there is no `repo:` field inside, because a + // second place to write the same string is a second place for it to be + // wrong. + // + // Full slug REQUIRED, with no bare-`` fallback — the one place in this + // file where that fallback does not exist. `github_apps` and + // `environments..projects` carry one because they predate the lesson + // (#12) and there are state files in the wild keyed the old way; this block + // is new, has no such files, and so gets to be right from the start. A bare + // `` is unique only *within* an org, which is precisely why it is not + // a key. + // + // Optional: a state file written before the registry existed keeps loading + // untouched, and `projectsIn` answers `[]` for it. + projects: z.record(RegisteredProjectSchema).optional(), // Keyed by the repo the App clones for. Prefer the FULL `/` // slug; a bare `` key still resolves (see githubAppNameFor) so // existing state files keep working. @@ -112,7 +154,136 @@ const BindingsSchema = z // cannot distinguish prod's app from staging's either. smoke_target: z.string().optional(), }) - .strict(); + .strict() + // The registry only earns its keep if it is TRUE. Every check here defends the + // same failure: a project that a fleet run never visits, because a fleet run + // that skips a project prints exactly what a fleet run over a clean project + // prints — nothing. Silence is the one report that must never be ambiguous, so + // these are parse-time errors (every verb loads bindings, so every verb refuses + // a registry that lies) rather than warnings some command might print. + .superRefine((bindings, ctx) => { + const registry = bindings.projects; + if (!registry) return; + + const knownEnvs = Object.keys(bindings.environments); + const knownEnvList = knownEnvs.join(", ") || "(none)"; + + for (const [slug, project] of Object.entries(registry)) { + // A key with no `/` is not a repo. See the schema note above: no fallback. + if (!slug.includes("/")) { + ctx.addIssue({ + code: z.ZodIssueCode.custom, + path: ["projects", slug], + message: [ + `projects["${slug}"] is not a repo — a registry key has no meaning without its org`, + "", + ` found: projects["${slug}"]`, + " wanted: a full / slug", + "", + "A bare is unique only *within* an org: heavy-duty/incubator and", + "acme/incubator collapse onto one entry, and the registry then claims one", + "project where there are two. Unlike github_apps, this block is new and has", + "no legacy state files to support, so there is no bare- fallback.", + "", + " projects:", + ` /${slug}:`, + ` environments: [${project.environments.join(", ")}]`, + ].join("\n"), + }); + } + + // Every environment named here must be one that actually exists. A typo + // makes the project real but its environment imaginary — so `--all` visits + // nothing for it, reports nothing about it, and exits clean. + for (const envName of project.environments) { + if (!(envName in bindings.environments)) { + ctx.addIssue({ + code: z.ZodIssueCode.custom, + path: ["projects", slug, "environments"], + message: [ + `projects["${slug}"] is registered in environment "${envName}", which does not exist`, + "", + ` registered: projects["${slug}"].environments → ${project.environments.join(", ")}`, + ` known envs: ${knownEnvList}`, + "", + "An environment nothing defines is one that no command can visit: a fleet", + "run would skip this project, and a silently skipped project reads exactly", + "like a clean one. Fix the name, or declare the environment:", + "", + " environments:", + ` ${envName}:`, + " server: ", + " team: { id: , name: }", + ].join("\n"), + }); + } + } + } + + // The other direction, and the one that rots quietly: a project carrying + // per-environment state (destination_uuid, smoke_target — #21) in an + // environment the registry does not register it for. That state is then real + // enough to be used by a direct `cast apply --env ` and invisible + // to every fleet run — the two blocks describing two different fleets. + // + // Only checked when `projects:` is present, so state files written before the + // registry keep loading exactly as they did. + for (const [envName, env] of Object.entries(bindings.environments)) { + for (const key of Object.keys(env.projects ?? {})) { + const entry = registry[key]; + if (entry?.environments.includes(envName)) continue; + + // The likeliest cause, worth saying out loud: a legacy bare- key + // (which projectBindingFor still resolves) under a registry that is + // correctly keyed by slug. The fix is a rename, not a registration. + const slugFor = key.includes("/") + ? undefined + : Object.keys(registry).find((s) => s.endsWith(`/${key}`)); + + const cause = slugFor + ? [ + ` registry has: projects["${slugFor}"]`, + "", + "The binding uses the legacy bare- key. The registry is keyed by the", + `full slug, so rename it to match — environments.${envName}.projects["${slugFor}"].`, + ] + : entry + ? [ + ` registered for: ${entry.environments.join(", ")}`, + "", + "Register the project in this environment, or drop the binding — state that", + "no fleet run will ever visit is state that stops being true without anyone", + "finding out:", + "", + " projects:", + ` ${key}:`, + ` environments: [${[...entry.environments, envName].join(", ")}]`, + ] + : [ + ` registry has: ${Object.keys(registry).join(", ") || "(nothing)"}`, + "", + "The project is not in the registry at all, so no fleet run will ever visit", + "it — while this binding says it is deployed here. Register it, or drop the", + "binding:", + "", + " projects:", + ` ${key}:`, + ` environments: [${envName}]`, + ]; + + ctx.addIssue({ + code: z.ZodIssueCode.custom, + path: ["environments", envName, "projects", key], + message: [ + `environments.${envName}.projects["${key}"] is bound in an environment the registry does not register it for`, + "", + ` bound at: environments.${envName}.projects["${key}"]`, + ...cause, + ].join("\n"), + }); + } + } + }); export type Bindings = z.infer; @@ -188,6 +359,27 @@ export function smokeTargetFor( return undefined; } +// The `/` slugs registered for one environment — the list a fleet +// operation iterates (`cast diff --all`, #26) and a rebuild reads (#27). +// +// SORTED, deliberately: the order of keys in a YAML file is an accident of who +// typed what when, and a fleet run's output — which a human reads top to bottom, +// and CI diffs — must not reshuffle because someone appended a project. The +// registry is a set; this returns it as one. +// +// `[]` when there is no registry, which is every state file written before this +// block existed. That is the honest answer for "which projects are registered +// here" when nothing is registered anywhere — and it makes a fleet verb over an +// unmigrated state file a clean no-op rather than a crash. +export function projectsIn(bindings: Bindings, envName: string): string[] { + const registry = bindings.projects; + if (!registry) return []; + return Object.entries(registry) + .filter(([, project]) => project.environments.includes(envName)) + .map(([slug]) => slug) + .sort(); +} + export function loadBindings( path: string, opts: { overrideText?: string } = {}, @@ -195,7 +387,20 @@ export function loadBindings( const text = opts.overrideText ?? readFileSync(path, "utf8"); const result = BindingsSchema.safeParse(parse(text)); if (!result.success) { - throw new Error(`invalid bindings ${path}: ${result.error.message}`); + // Zod's own `.message` is the entire issue array as JSON — which renders the + // refusals above as one long line of `\n` escapes, i.e. throws away the part + // of them that was worth writing. Render the issues instead. + const detail = result.error.issues + .map((issue) => { + // A multi-line message is one WE wrote: it already names the path, the + // cause, and the YAML to write. A one-line message is zod's ("Required"), + // and is useless without the path it happened at. + if (issue.message.includes("\n")) return issue.message; + const where = issue.path.map(String).join("."); + return where ? `${where}: ${issue.message}` : issue.message; + }) + .join("\n\n"); + throw new Error(`invalid bindings ${path}:\n\n${detail}`); } return result.data; } diff --git a/test/bindings.test.ts b/test/bindings.test.ts index e9cace5..b38c3c0 100644 --- a/test/bindings.test.ts +++ b/test/bindings.test.ts @@ -5,6 +5,7 @@ import { githubAppNameFor, loadBindings, projectBindingFor, + projectsIn, smokeTargetFor, } from "../src/bindings.js"; @@ -223,3 +224,233 @@ github_apps: {} ).toThrow(/invalid bindings/); }); }); + +// The registry: the list of which projects exist at all. Everything it is FOR +// (fleet iteration, rebuild-from-state) depends on it being true, and the way it +// stops being true is silent — see the refusals below. +describe("the project registry", () => { + const twoEnvs = ` +environments: + prod: + server: shared-box + team: { id: 0, name: Root Team } + staging: + server: staging-box + team: { id: 0, name: Root Team } +`; + + it("registers projects per environment, keyed by the full slug", () => { + const b = loadBindings("environments.yaml", { + overrideText: `${twoEnvs} +projects: + heavy-duty/incubator: + environments: [prod, staging] + acme/client-site: + environments: [prod] +github_apps: {} +`, + }); + expect(b.projects).toEqual({ + "heavy-duty/incubator": { environments: ["prod", "staging"] }, + "acme/client-site": { environments: ["prod"] }, + }); + }); + + describe("projectsIn", () => { + const b = loadBindings("environments.yaml", { + overrideText: `${twoEnvs} +projects: + heavy-duty/incubator: + environments: [prod, staging] + acme/client-site: + environments: [prod] +github_apps: {} +`, + }); + + // Sorted, not file-order: a fleet run's output is read by a human and diffed + // by CI, and must not reshuffle because someone appended a project. + it("gives an environment's projects, sorted", () => { + expect(projectsIn(b, "prod")).toEqual([ + "acme/client-site", + "heavy-duty/incubator", + ]); + }); + + it("gives only the projects registered for that environment", () => { + expect(projectsIn(b, "staging")).toEqual(["heavy-duty/incubator"]); + }); + + it("is empty for an environment no project is registered in", () => { + expect(projectsIn(b, "nowhere")).toEqual([]); + }); + }); + + // The refusal the issue is actually about. A typo'd environment name makes the + // project real and its environment imaginary: `cast diff --all` visits nothing + // for it, reports nothing, and exits clean — and a silently skipped project + // reads exactly like a clean one. + it("refuses an environment that does not exist, naming the ones that do", () => { + const err = () => + loadBindings("environments.yaml", { + overrideText: `${twoEnvs} +projects: + heavy-duty/incubator: + environments: [prod, stagng] +github_apps: {} +`, + }); + expect(err).toThrow(/environment "stagng", which does not exist/); + expect(err).toThrow(/known envs:\s+prod, staging/); + }); + + // The other direction, and the one that rots quietly: per-environment state + // (#21) sitting in an environment the registry does not register the project + // for. The two blocks then describe two different fleets. + it("refuses a binding in an environment the registry does not register", () => { + const err = () => + loadBindings("environments.yaml", { + overrideText: ` +environments: + prod: + server: shared-box + team: { id: 0, name: Root Team } + staging: + server: staging-box + team: { id: 0, name: Root Team } + projects: + heavy-duty/incubator: + destination_uuid: dest-abc +projects: + heavy-duty/incubator: + environments: [prod] +github_apps: {} +`, + }); + expect(err).toThrow( + /environments\.staging\.projects\["heavy-duty\/incubator"\]/, + ); + expect(err).toThrow(/registered for:\s+prod/); + }); + + it("refuses a binding for a project the registry does not carry at all", () => { + const err = () => + loadBindings("environments.yaml", { + overrideText: ` +environments: + prod: + server: shared-box + team: { id: 0, name: Root Team } + projects: + acme/client-site: + destination_uuid: dest-client +projects: + heavy-duty/incubator: + environments: [prod] +github_apps: {} +`, + }); + expect(err).toThrow(/environments\.prod\.projects\["acme\/client-site"\]/); + expect(err).toThrow(/registry has:\s+heavy-duty\/incubator/); + }); + + // A legacy bare- binding key (projectBindingFor still resolves one) under + // a slug-keyed registry is drift with an obvious fix — say which fix. + it("tells a legacy bare- binding key which slug to rename to", () => { + const err = () => + loadBindings("environments.yaml", { + overrideText: ` +environments: + prod: + server: shared-box + team: { id: 0, name: Root Team } + projects: + incubator: + smoke_target: core +projects: + heavy-duty/incubator: + environments: [prod] +github_apps: {} +`, + }); + expect(err).toThrow(/registry has:\s+projects\["heavy-duty\/incubator"\]/); + expect(err).toThrow(/legacy bare- key/); + }); + + // No fallback here, unlike github_apps: this block is new, so it has no state + // files in the wild to keep working, and a bare is unique only within an + // org — which is exactly why it is not a key. + it("refuses a bare registry key — the org is not optional", () => { + const err = () => + loadBindings("environments.yaml", { + overrideText: `${twoEnvs} +projects: + incubator: + environments: [prod] +github_apps: {} +`, + }); + expect(err).toThrow(/projects\["incubator"\] is not a repo/); + expect(err).toThrow(/full \/ slug/); + }); + + // A project registered into nothing is a line of YAML that reads like a + // registration and is skipped by every fleet run. + it("refuses a project registered into no environment", () => { + expect(() => + loadBindings("environments.yaml", { + overrideText: `${twoEnvs} +projects: + heavy-duty/incubator: + environments: [] +github_apps: {} +`, + }), + ).toThrow(/invalid bindings/); + }); + + it("rejects an unknown key inside a registry entry", () => { + expect(() => + loadBindings("environments.yaml", { + overrideText: `${twoEnvs} +projects: + heavy-duty/incubator: + environments: [prod] + repo: heavy-duty/incubator +github_apps: {} +`, + }), + ).toThrow(/invalid bindings/); + }); + + // Back-compat: the registry is optional, and every state file written before it + // existed has no `projects:` block. Such a file loads unchanged — including its + // per-environment bindings, which are NOT checked against a registry that is + // not there. + describe("with no registry at all", () => { + const b = loadBindings("environments.yaml", { + overrideText: ` +environments: + prod: + server: shared-box + team: { id: 0, name: Root Team } + projects: + heavy-duty/incubator: + destination_uuid: dest-abc + smoke_target: core +github_apps: {} +`, + }); + + it("loads, and keeps its per-environment bindings working", () => { + expect(b.projects).toBe(undefined); + expect( + projectBindingFor(b, "prod", "heavy-duty/incubator")?.destination_uuid, + ).toBe("dest-abc"); + }); + + it("has no projects registered in any environment", () => { + expect(projectsIn(b, "prod")).toEqual([]); + }); + }); +}); -- 2.45.2