apply writes the `pending-coolify-generated` placeholder over a live generated secret — a routine command takes prod down #47

New issue

Closed

opened 2026-07-14 20:54:41 +00:00 by dan-claude-bot · 1 comment

dan-claude-bot commented

2026-07-14 20:54:41 +00:00

(Migrated from github.com)

What happens

A manifest declares generated_secrets: [DATABASE_URL, REDIS_URL]. The store therefore holds them as the literal pending-coolify-generated — by design (capture.ts:7, semantics.md:421). The first apply creates the Postgres/Redis resources and Coolify generates the real URLs.

From that moment the store is wrong, and apply does not know it:

secret DATABASE_URL differs        ← every plan prints this
secret REDIS_URL differs

That line is apply announcing it intends to PATCH pending-coolify-generated over the live, working URL — and then redeploy. On heavy-duty/incubator's prod box, which is up and migrated right now, the next routine cast apply takes the database connection away from migrate, api and worker. Not a risk: a certainty, from the tool's happy path. It is currently held off by a hand-written "run NO cast apply against prod" line in an operator punch list — which is not a guardrail, it is a post-it note on a loaded gun.

Root cause

GENERATED_PLACEHOLDER is exported from capture.ts:7 and is referenced only by capture.ts and draft.ts — the two write-side-of-the-store verbs. diff and apply have never heard of it:

diff.ts:diffEnv compares live[key] !== v.value and emits a plain change — the placeholder is just a string that differs.
apply.ts:100 → syncEnv → cli.ts:2129 bulk-UPSERTs every var from the store verbatim.

So the one literal cast itself invented to mean "this value is not real yet" is invisible to the only two verbs that could act on it.

Why it went unnoticed

generated_secrets were designed for the first apply, where writing the placeholder is harmless: Coolify overwrites it when it creates the resource. Nobody had yet run a second apply against a project whose generated secrets had since been filled in — because until this week no project cast created from nothing had ever reached a second apply. The bootstrap is two-pass and only the first pass had ever run.

Suggested

apply must refuse, not warn — this is a data-loss write, and the plan line that precedes it is indistinguishable from a legitimate secret rotation:

refusing apply: DATABASE_URL is a generated_secret and the store still holds the pending-coolify-generated placeholder, but postgres on Coolify holds a real value. Writing the store's value would overwrite it and break every consumer.
Fill the store from the live resource first (see #48), or remove the name from generated_secrets: if it is no longer provider-generated.

Precisely:

On a create, the placeholder is fine — send it (Coolify replaces it). That path is unchanged.
On an update, if a var is declared in generated_secrets, its store value is the placeholder literal, and the live value is anything else → refuse the apply. Live value also-placeholder, or var absent live → not a conflict, proceed.
The refusal is a fail-closed gate in the same family as the team assert (#9) and the absent-project gate (#11): the shape cast already uses for "a routine command is about to do something irreversible."

Two adjacent points, both worth taking in the same change:

diff should say it too, and say it loudly — today secret DATABASE_URL differs is the only signal, and it is the same words a real rotation prints. It should read secret DATABASE_URL: store holds the generated-secret PLACEHOLDER, live holds a real value — apply would OVERWRITE it.
Never print the live value. The refusal names the key and the resource, never the secret — same rule as capture's disposition table (capture.ts:166).

The other half of the hole: nothing can fill the store after the first apply — capture placeholds a generated_secret by design. Filed separately as #48; this issue is the guard, that one is the fix that makes the guard actionable.
Live example, with the full timeline: heavy-duty/incubator#10 (OPERATOR.md, top row).

## What happens A manifest declares `generated_secrets: [DATABASE_URL, REDIS_URL]`. The store therefore holds them as the literal `pending-coolify-generated` — by design (`capture.ts:7`, `semantics.md:421`). The first `apply` creates the Postgres/Redis resources and **Coolify** generates the real URLs. From that moment the store is **wrong**, and `apply` does not know it: ``` secret DATABASE_URL differs ← every plan prints this secret REDIS_URL differs ``` That line is `apply` announcing it intends to PATCH `pending-coolify-generated` **over the live, working URL** — and then redeploy. On `heavy-duty/incubator`'s prod box, which is up and migrated right now, the next routine `cast apply` takes the database connection away from `migrate`, `api` and `worker`. Not a risk: a certainty, from the tool's happy path. It is currently held off by a hand-written *"run NO `cast apply` against prod"* line in an operator punch list — which is not a guardrail, it is a post-it note on a loaded gun. ## Root cause `GENERATED_PLACEHOLDER` is exported from `capture.ts:7` and is referenced **only by `capture.ts` and `draft.ts`** — the two write-side-of-the-store verbs. `diff` and `apply` have never heard of it: - `diff.ts:diffEnv` compares `live[key] !== v.value` and emits a plain `change` — the placeholder is just a string that differs. - `apply.ts:100` → `syncEnv` → `cli.ts:2129` bulk-UPSERTs **every** var from the store verbatim. So the one literal cast itself invented to mean *"this value is not real yet"* is invisible to the only two verbs that could act on it. ## Why it went unnoticed `generated_secrets` were designed for the **first** apply, where writing the placeholder is harmless: Coolify overwrites it when it creates the resource. Nobody had yet run a **second** apply against a project whose generated secrets had since been filled in — because until this week no project cast created from nothing had ever reached a second apply. The bootstrap is two-pass and only the first pass had ever run. ## Suggested `apply` must **refuse**, not warn — this is a data-loss write, and the plan line that precedes it is indistinguishable from a legitimate secret rotation: > refusing apply: `DATABASE_URL` is a `generated_secret` and the store still holds the `pending-coolify-generated` placeholder, but `postgres` on Coolify holds a real value. Writing the store's value would overwrite it and break every consumer. > Fill the store from the live resource first (see #48), or remove the name from `generated_secrets:` if it is no longer provider-generated. Precisely: - On a **create**, the placeholder is fine — send it (Coolify replaces it). That path is unchanged. - On an **update**, if a var is declared in `generated_secrets`, its store value is the placeholder literal, and the live value is **anything else** → refuse the apply. Live value also-placeholder, or var absent live → not a conflict, proceed. - The refusal is a **fail-closed** gate in the same family as the team assert (#9) and the absent-project gate (#11): the shape cast already uses for *"a routine command is about to do something irreversible."* Two adjacent points, both worth taking in the same change: 1. **`diff` should say it too, and say it loudly** — today `secret DATABASE_URL differs` is the *only* signal, and it is the same words a real rotation prints. It should read `secret DATABASE_URL: store holds the generated-secret PLACEHOLDER, live holds a real value — apply would OVERWRITE it`. 2. **Never print the live value.** The refusal names the key and the resource, never the secret — same rule as `capture`'s disposition table (`capture.ts:166`). ## Related - The other half of the hole: nothing can *fill* the store after the first apply — `capture` placeholds a `generated_secret` by design. Filed separately as #48; this issue is the guard, that one is the fix that makes the guard actionable. - Live example, with the full timeline: [heavy-duty/incubator#10](https://github.com/heavy-duty/incubator/pull/10) (`OPERATOR.md`, top row).

dan-claude-bot commented

2026-07-14 22:35:27 +00:00

(Migrated from github.com)

#60 would make this failure structurally impossible for the two names currently causing it: if DATABASE_URL/REDIS_URL are derived from the declared database rather than stored, there is no placeholder in the store to overwrite the live value with, and the secret DATABASE_URL differs line stops being printed because the store no longer holds a competing copy.

Worth keeping this guard regardless, as a backstop for any residual provider-generated secret that is genuinely not derivable — but the live instance of it (the loaded store on heavy-duty/incubator's prod box) is dissolved by #60, not merely defended against.

#60 would make this failure **structurally impossible for the two names currently causing it**: if `DATABASE_URL`/`REDIS_URL` are derived from the declared database rather than stored, there is no placeholder in the store to overwrite the live value *with*, and the `secret DATABASE_URL differs` line stops being printed because the store no longer holds a competing copy. Worth keeping this guard regardless, as a backstop for any residual provider-generated secret that is genuinely not derivable — but the live instance of it (the loaded store on `heavy-duty/incubator`'s prod box) is dissolved by #60, not merely defended against.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/cast#47

No description provided.

apply writes the pending-coolify-generated placeholder over a live generated secret — a routine command takes prod down #47

What happens

Root cause

Why it went unnoticed

Suggested

Related

apply writes the `pending-coolify-generated` placeholder over a live generated secret — a routine command takes prod down #47