diff cannot tell "project absent" from "project empty" — a name mismatch reports a clean diff that verified nothing #11

New issue

Closed

opened 2026-07-13 14:33:49 +00:00 by dan-claude-bot · 0 comments

dan-claude-bot commented

2026-07-13 14:33:49 +00:00

(Migrated from github.com)

The bug

diff cannot distinguish "this project does not exist" from "this project is empty", and renders both as "nothing exists — create everything". Against a live, fully-populated Coolify instance, a one-character project-name mismatch therefore produces a confident, clean-looking diff that verified nothing.

This is the same failure #9 / #10 closed — an unverifiable read that reports absent and invites a create — reached through the project name instead of the team.

Why it is live again now

cast#10's assert is what stood between us and this class of bug. It only fires on a team mismatch, and we have just collapsed to a single Root Team for every project (a Coolify server belongs to exactly one team — Team hasMany Server, no pivot — so team-per-project would force a server per project, per tier; deferred).

With one team, every token Coolify mints is stamped team_id = 0, so assertTeam passes on every call, forever. It is worth keeping — it costs one GET /teams/current and it is the gate the day a second team exists — but it is inert, and it is no longer guarding anything. The project name is now the only thing selecting what cast compares against, and nothing checks it.

Mechanism

Two behaviours compose into the silent failure:

The project name is derived, not configurable. cli.ts:229 — const repoShort = orgRepo.split("/")[1]. heavy-duty/incubator → "incubator", and that one string is used for three unrelated lookups: the Coolify project name, the secrets filename, and the github_apps key. There is no override.
A missing project resolves to an empty live side. cli.ts:157-158:
```
const project = projects.find((p) => p.name === projectName);
if (!project) return [];
```
The [] is deliberate and correct for apply — a first apply legitimately has no project yet. But diff shares the code path, and computeDiff(desired, []) means every desired resource is missing → a full-create plan, rendered as though it were a verified finding.

The comment on that line even says it: "tolerate a missing project (first apply creates it)". The tolerance is right for apply and wrong for diff.

Concrete failure we hit

We are migrating a hand-built production Coolify box (call it box B, provisioned through the UI long before the manifest existed) onto a new manifest-driven instance. The migration's key gate is: run cast diff --full against box B and confirm the manifest actually describes it before provisioning from that manifest.

Box B's project is named whatever was typed into the UI that day. If that is incubator-prod, or heavy-duty, or Incubator — the gate returns clean, we conclude the manifest is verified, and we provision production from a manifest nobody ever checked. The gate that was supposed to catch manifest drift instead launders it.

There is no plausible world where a --full diff of a hand-built box against a fresh manifest is genuinely clean: box B has real domains, real env values, and five feature flags that differ by design. Zero drift is not a pass — it is a bug. But cast reports it identically to a real pass.

Asks

1. diff fails loudly when the project is absent. Not a warning — a non-zero exit with a message that names what it looked for:

refusing to diff: no project named "incubator" exists in this Coolify team
  team:     id=0 name="Root Team"  (GET /teams/current)
  projects: incubator-prod, umami, metabase

cast derives the project name from the repo slug (heavy-duty/incubator -> "incubator").
An absent project would diff as "nothing exists, create everything" — a clean-looking
report that verified nothing. Pass --project <name> if it is named differently here.

apply keeps today's tolerant behaviour: absent is legitimate, it creates it. The split is the whole point — apply is allowed to be the thing that brings a project into existence; diff is only ever a claim about something that already exists.

2. --project <name> to override the repo-derived name. Same shape as the team: binding: you state what you expect, cast verifies it, and it refuses when reality disagrees. This is what lets a differently-named legacy project be verified against the manifest without renaming it first, and it decouples "what the repo is called" from "what the project is called in this instance" — which is also what any future multi-project state repo will need.

Bindings-file placement (projects.<repo-short>.name) is probably the better long-term home, but a flag unblocks the migration and can be the escape hatch afterwards.

Not asked for

Nothing here is a reason to weaken cast#10. The team: binding stays required and the assert stays on every command — an inert gate is cheap, and making it optional would reintroduce exactly the fail-open #10 closed. This issue adds the second gate that the single-team topology now leaves uncovered.

Interim mitigation

The migration runbook carries a manual pre-gate (check box B's project name against GET /api/v1/projects before trusting any diff) and a sanity floor (zero drift on a hand-built box is a bug, not a pass). That is a human remembering to look — which is precisely the kind of guard #10 argued should be mechanical.

## The bug `diff` cannot distinguish **"this project does not exist"** from **"this project is empty"**, and renders both as *"nothing exists — create everything"*. Against a live, fully-populated Coolify instance, a one-character project-name mismatch therefore produces a **confident, clean-looking diff that verified nothing**. This is the *same* failure [#9](https://github.com/heavy-duty/cast/issues/9) / [#10](https://github.com/heavy-duty/cast/pull/10) closed — an unverifiable read that reports *absent* and invites a create — reached through the **project name** instead of the **team**. ## Why it is live again now cast#10's assert is what stood between us and this class of bug. It only fires on a **team** mismatch, and we have just collapsed to a **single Root Team** for every project (a Coolify server belongs to exactly one team — `Team hasMany Server`, no pivot — so team-per-project would force a server per project, per tier; deferred). With one team, **every token Coolify mints is stamped `team_id = 0`**, so `assertTeam` passes on every call, forever. It is worth keeping — it costs one `GET /teams/current` and it is the gate the day a second team exists — but it is **inert**, and it is no longer guarding anything. The project name is now the only thing selecting what cast compares against, and nothing checks it. ## Mechanism Two behaviours compose into the silent failure: 1. **The project name is derived, not configurable.** `cli.ts:229` — `const repoShort = orgRepo.split("/")[1]`. `heavy-duty/incubator` → `"incubator"`, and that one string is used for three unrelated lookups: the Coolify project name, the secrets filename, and the `github_apps` key. There is no override. 2. **A missing project resolves to an empty live side.** `cli.ts:157-158`: ```ts const project = projects.find((p) => p.name === projectName); if (!project) return []; ``` The `[]` is deliberate and correct for `apply` — a first apply legitimately has no project yet. But `diff` shares the code path, and `computeDiff(desired, [])` means *every desired resource is missing* → a full-create plan, rendered as though it were a verified finding. The comment on that line even says it: *"tolerate a missing project (first apply creates it)"*. The tolerance is right for `apply` and wrong for `diff`. ## Concrete failure we hit We are migrating a hand-built production Coolify box (call it box B, provisioned through the UI long before the manifest existed) onto a new manifest-driven instance. The migration's key gate is: **run `cast diff --full` against box B and confirm the manifest actually describes it** before provisioning from that manifest. Box B's project is named whatever was typed into the UI that day. If that is `incubator-prod`, or `heavy-duty`, or `Incubator` — the gate returns *clean*, we conclude the manifest is verified, and we provision production from a manifest nobody ever checked. The gate that was supposed to catch manifest drift instead **launders it**. There is no plausible world where a `--full` diff of a hand-built box against a fresh manifest is genuinely clean: box B has real domains, real env values, and five feature flags that differ by design. **Zero drift is not a pass — it is a bug.** But cast reports it identically to a real pass. ## Asks **1. `diff` fails loudly when the project is absent.** Not a warning — a non-zero exit with a message that names what it looked for: ``` refusing to diff: no project named "incubator" exists in this Coolify team team: id=0 name="Root Team" (GET /teams/current) projects: incubator-prod, umami, metabase cast derives the project name from the repo slug (heavy-duty/incubator -> "incubator"). An absent project would diff as "nothing exists, create everything" — a clean-looking report that verified nothing. Pass --project <name> if it is named differently here. ``` `apply` keeps today's tolerant behaviour: absent is legitimate, it creates it. The split is the whole point — `apply` is *allowed* to be the thing that brings a project into existence; `diff` is *only* ever a claim about something that already exists. **2. `--project <name>` to override the repo-derived name.** Same shape as the `team:` binding: you state what you expect, cast verifies it, and it refuses when reality disagrees. This is what lets a differently-named legacy project be verified against the manifest without renaming it first, and it decouples "what the repo is called" from "what the project is called in this instance" — which is also what any future multi-project state repo will need. Bindings-file placement (`projects.<repo-short>.name`) is probably the better long-term home, but a flag unblocks the migration and can be the escape hatch afterwards. ## Not asked for Nothing here is a reason to weaken cast#10. The `team:` binding stays **required** and the assert stays on every command — an inert gate is cheap, and making it optional would reintroduce exactly the fail-open #10 closed. This issue adds the *second* gate that the single-team topology now leaves uncovered. ## Interim mitigation The migration runbook carries a manual pre-gate (check box B's project name against `GET /api/v1/projects` before trusting any diff) and a sanity floor (*zero drift on a hand-built box is a bug, not a pass*). That is a human remembering to look — which is precisely the kind of guard #10 argued should be mechanical.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/cast#11

No description provided.