heavy-duty/cast

Fork 0

Coolify's own SERVICE_/POSTGRES_ magic vars report as orphans forever — a correct box can never read clean #87

New issue

Closed

opened 2026-07-16 16:44:32 +00:00 by dan-claude-bot · 1 comment

dan-claude-bot commented

2026-07-16 16:44:32 +00:00

(Migrated from github.com)

Summary

A prod box with zero real drift cannot make cast diff say clean. Today's run, after #86 removed the last genuine finding:

update application core
  env SERVICE_URL_API: live-only (orphan var — apply never removes)
  env SERVICE_FQDN_API: live-only (orphan var — apply never removes)
  env SERVICE_URL_ADMIN: live-only (orphan var — apply never removes)
  env SERVICE_FQDN_ADMIN: live-only (orphan var — apply never removes)
  env SERVICE_URL_INTAKE: live-only (orphan var — apply never removes)
  env SERVICE_FQDN_INTAKE: live-only (orphan var — apply never removes)
update service umami
  env SERVICE_URL_UMAMI: live-only (orphan var — apply never removes)
  env SERVICE_FQDN_UMAMI: live-only (orphan var — apply never removes)
  env SERVICE_URL_UMAMI_3000: live-only (orphan var — apply never removes)
  env SERVICE_FQDN_UMAMI_3000: live-only (orphan var — apply never removes)
  env SERVICE_USER_POSTGRES: live-only (orphan var — apply never removes)
  env SERVICE_PASSWORD_POSTGRES: live-only (orphan var — apply never removes)
  env SERVICE_PASSWORD_64_UMAMI: live-only (orphan var — apply never removes)
  env POSTGRES_USER: live-only (orphan var — apply never removes)
  env POSTGRES_PASSWORD: live-only (orphan var — apply never removes)
  env POSTGRES_DB: live-only (orphan var — apply never removes)
2 change(s), 0 orphan(s)

Every one of those is Coolify's own generated magic var — SERVICE_FQDN_*/SERVICE_URL_* minted for a compose app's per-container domains, and the one-click umami service's bundled-Postgres credentials. cast never declares them, apply never touches them, and they will be there forever. They are not drift. Yet they are the entire content of the report, and they hold two resources permanently in change.

The asymmetry — cast already knows

draft.ts has exactly this judgment and applies it:

const COOLIFY_MAGIC = /^SERVICE_(FQDN|URL|USER|PASSWORD|BASE64|REALBASE64)(_|$)/;

It refuses to copy these into a draft, because they are "generated per-instance by Coolify … and mean nothing anywhere else."

But diffEnv's remove-candidate branch consults only reserved.ts:

if (!(key in desired.vars) && !isReservedEnvName(key))
  diffs.push({ key, state: "remove-candidate", secret: false });

isReservedEnvName covers SOURCE_COMMIT + COOLIFY_* only. Same judgment, two homes, applied in one of them.

Why this is worth fixing, not tolerating

This issue's own predecessor said it best. #78's Impact section:

Masks real drift — an operator who learns these five "always show change" stops trusting the diff on them.

That is precisely what a permanent 16-line orphan list does, but to the whole report rather than five keys. A diff that can never say clean trains the reader to skim it — and the next time something real appears, it appears in a list that has cried wolf on every previous run. #78 sat misdiagnosed for exactly this reason.

Probe — see what would be reclassified

Copy-paste (swap the uuid; use /services/$UUID/envs for a service):

COOLIFY=http://coolify-box:8000
TOKEN=...                  # a token with read access
UUID=kppwkvh6qjun7d0uvk60wi97   # core

curl -sS -H "Authorization: Bearer $TOKEN" "$COOLIFY/api/v1/applications/$UUID/envs" \
| jq 'map(select(.is_preview != true) | .key)
      | { coolify_magic: map(select(test("^SERVICE_(FQDN|URL|USER|PASSWORD|BASE64|REALBASE64)(_|$)"))),
          datastore_ish: map(select(test("^(POSTGRES|REDIS|DB|DATABASE|MYSQL|MONGO|MARIADB|CLICKHOUSE)_"))),
          everything_else: map(select((test("^SERVICE_(FQDN|URL|USER|PASSWORD|BASE64|REALBASE64)(_|$)")
                                       or test("^(POSTGRES|REDIS|DB|DATABASE|MYSQL|MONGO|MARIADB|CLICKHOUSE)_")) | not)) }'

coolify_magic is what the narrow rule silences. datastore_ish is the contested set (see below). everything_else must keep reporting — that bucket is where a genuine orphan hides.

The design decision this needs — narrow vs wide

draft.ts deliberately errs WIDE: isProviderGenerated matches COOLIFY_MAGIC or any name carrying both a datastore word and a connection word (POSTGRES_USER, DB_HOST, …). Its comment explains why the width is safe there:

Over-match a name that was really a secret: it is placeheld, listed in UNCAPTURED.md for disposition … Noisy, recoverable, LOUD.

That asymmetry inverts in a diff. Over-matching here does not withhold a value for review — it hides a live-only var the operator should see. A stray DATABASE_URL left on a resource by hand is exactly the orphan worth reporting, and the wide rule would swallow it silently.

So the two call sites want the same vocabulary but different widths, and that must be explicit rather than accidental:

narrow (COOLIFY_MAGIC prefix only) → silences the 13 SERVICE_* lines above; leaves umami's POSTGRES_USER/POSTGRES_PASSWORD/POSTGRES_DB still reported.
wide (isProviderGenerated) → silences all 16, at the cost of hiding any future hand-left datastore var.

Those three POSTGRES_* on umami are the crux: they are genuinely Coolify's (the one-click service's bundled Postgres, per D-253/D-279 in the incubator brain) but they match only the wide rule. Note umami is slated to stop being a one-click service (incubator D-279), which shrinks — but does not remove — this case.

What to do

Give the provider-generated vocabulary one home. reserved.ts is the natural one: it already owns "names the platform, not the manifest, controls". Have draft.ts and diffEnv both consult it instead of draft.ts owning the regex privately.
Use the narrow rule in diffEnv (over-matching hides real orphans) and keep the wide rule in draft — and say why at both sites, because the next reader will assume one rule.
Decide the disposition: not a remove-candidate. Either drop silently, or surface once as a finding (the ReservedVar precedent) — "N Coolify-generated vars present; apply never removes them" — rather than N lines of per-var noise.
Resolve POSTGRES_*-on-a-service explicitly, one way or the other. A "clean" that still prints 3 lines has not fixed the thing this issue is about.
Regression test: a box with only magic vars live-only reads clean; a hand-left FOO_URL still reports.

Found while verifying #85/#86 against prod: with the shadow-row bug fixed, this is all that stands between a correct box and a clean diff.

## Summary A prod box with **zero real drift** cannot make `cast diff` say clean. Today's run, after #86 removed the last genuine finding: ``` update application core env SERVICE_URL_API: live-only (orphan var — apply never removes) env SERVICE_FQDN_API: live-only (orphan var — apply never removes) env SERVICE_URL_ADMIN: live-only (orphan var — apply never removes) env SERVICE_FQDN_ADMIN: live-only (orphan var — apply never removes) env SERVICE_URL_INTAKE: live-only (orphan var — apply never removes) env SERVICE_FQDN_INTAKE: live-only (orphan var — apply never removes) update service umami env SERVICE_URL_UMAMI: live-only (orphan var — apply never removes) env SERVICE_FQDN_UMAMI: live-only (orphan var — apply never removes) env SERVICE_URL_UMAMI_3000: live-only (orphan var — apply never removes) env SERVICE_FQDN_UMAMI_3000: live-only (orphan var — apply never removes) env SERVICE_USER_POSTGRES: live-only (orphan var — apply never removes) env SERVICE_PASSWORD_POSTGRES: live-only (orphan var — apply never removes) env SERVICE_PASSWORD_64_UMAMI: live-only (orphan var — apply never removes) env POSTGRES_USER: live-only (orphan var — apply never removes) env POSTGRES_PASSWORD: live-only (orphan var — apply never removes) env POSTGRES_DB: live-only (orphan var — apply never removes) 2 change(s), 0 orphan(s) ``` Every one of those is **Coolify's own generated magic var** — `SERVICE_FQDN_*`/`SERVICE_URL_*` minted for a compose app's per-container domains, and the one-click umami service's bundled-Postgres credentials. cast never declares them, `apply` never touches them, and they will be there forever. They are **not drift**. Yet they are the entire content of the report, and they hold two resources permanently in `change`. ## The asymmetry — cast already knows `draft.ts` has exactly this judgment and applies it: ```ts const COOLIFY_MAGIC = /^SERVICE_(FQDN|URL|USER|PASSWORD|BASE64|REALBASE64)(_|$)/; ``` It refuses to copy these into a draft, because they are "generated per-instance by Coolify … and mean nothing anywhere else." But `diffEnv`'s remove-candidate branch consults only `reserved.ts`: ```ts if (!(key in desired.vars) && !isReservedEnvName(key)) diffs.push({ key, state: "remove-candidate", secret: false }); ``` `isReservedEnvName` covers `SOURCE_COMMIT` + `COOLIFY_*` only. **Same judgment, two homes, applied in one of them.** ## Why this is worth fixing, not tolerating This issue's own predecessor said it best. #78's *Impact* section: > **Masks real drift** — an operator who learns these five "always show change" stops trusting the diff on them. That is precisely what a permanent 16-line orphan list does, but to the whole report rather than five keys. A diff that can never say *clean* trains the reader to skim it — and the next time something real appears, it appears in a list that has cried wolf on every previous run. #78 sat misdiagnosed for exactly this reason. ## Probe — see what would be reclassified Copy-paste (swap the uuid; use `/services/$UUID/envs` for a service): ```bash COOLIFY=http://coolify-box:8000 TOKEN=... # a token with read access UUID=kppwkvh6qjun7d0uvk60wi97 # core curl -sS -H "Authorization: Bearer $TOKEN" "$COOLIFY/api/v1/applications/$UUID/envs" \ | jq 'map(select(.is_preview != true) | .key) | { coolify_magic: map(select(test("^SERVICE_(FQDN|URL|USER|PASSWORD|BASE64|REALBASE64)(_|$)"))), datastore_ish: map(select(test("^(POSTGRES|REDIS|DB|DATABASE|MYSQL|MONGO|MARIADB|CLICKHOUSE)_"))), everything_else: map(select((test("^SERVICE_(FQDN|URL|USER|PASSWORD|BASE64|REALBASE64)(_|$)") or test("^(POSTGRES|REDIS|DB|DATABASE|MYSQL|MONGO|MARIADB|CLICKHOUSE)_")) | not)) }' ``` `coolify_magic` is what the narrow rule silences. `datastore_ish` is the contested set (see below). `everything_else` must keep reporting — that bucket is where a genuine orphan hides. ## The design decision this needs — narrow vs wide `draft.ts` deliberately errs **WIDE**: `isProviderGenerated` matches `COOLIFY_MAGIC` **or** any name carrying both a datastore word and a connection word (`POSTGRES_USER`, `DB_HOST`, …). Its comment explains why the width is safe *there*: > Over-match a name that was really a secret: it is placeheld, listed in UNCAPTURED.md for disposition … Noisy, recoverable, LOUD. **That asymmetry inverts in a diff.** Over-matching here does not withhold a value for review — it **hides a live-only var the operator should see**. A stray `DATABASE_URL` left on a resource by hand is exactly the orphan worth reporting, and the wide rule would swallow it silently. So the two call sites want the *same vocabulary* but **different widths**, and that must be explicit rather than accidental: - **narrow (`COOLIFY_MAGIC` prefix only)** → silences the 13 `SERVICE_*` lines above; leaves umami's `POSTGRES_USER`/`POSTGRES_PASSWORD`/`POSTGRES_DB` still reported. - **wide (`isProviderGenerated`)** → silences all 16, at the cost of hiding any future hand-left datastore var. Those three `POSTGRES_*` on umami are the crux: they are genuinely Coolify's (the one-click service's bundled Postgres, per D-253/D-279 in the incubator brain) but they match only the **wide** rule. Note umami is slated to stop being a one-click service (incubator D-279), which shrinks — but does not remove — this case. ## What to do - [ ] Give the provider-generated vocabulary one home. `reserved.ts` is the natural one: it already owns "names the platform, not the manifest, controls". Have `draft.ts` and `diffEnv` both consult it instead of `draft.ts` owning the regex privately. - [ ] Use the **narrow** rule in `diffEnv` (over-matching hides real orphans) and keep the **wide** rule in `draft` — and say why at both sites, because the next reader will assume one rule. - [ ] Decide the disposition: not a `remove-candidate`. Either drop silently, or surface once as a **finding** (the `ReservedVar` precedent) — "N Coolify-generated vars present; apply never removes them" — rather than N lines of per-var noise. - [ ] Resolve `POSTGRES_*`-on-a-service explicitly, one way or the other. A "clean" that still prints 3 lines has not fixed the thing this issue is about. - [ ] Regression test: a box with only magic vars live-only reads **clean**; a hand-left `FOO_URL` still reports. Found while verifying #85/#86 against prod: with the shadow-row bug fixed, this is all that stands between a correct box and a clean diff.

dan-claude-bot commented

2026-07-16 16:47:54 +00:00

(Migrated from github.com)

Probe run against prod core — this settles the narrow-vs-wide decision empirically.

{
  "coolify_magic": ["SERVICE_URL_API","SERVICE_FQDN_API","SERVICE_URL_ADMIN",
                    "SERVICE_FQDN_ADMIN","SERVICE_URL_INTAKE","SERVICE_FQDN_INTAKE"],
  "datastore_ish": ["DATABASE_URL","REDIS_URL"],
  "everything_else": ["NODE_ENV","MAILGUN_API_KEY","MAILGUN_SIGNING_KEY","MAILGUN_DOMAIN",
                      "OPENROUTER_API_KEY","ADMIN_EMAIL","ADMIN_API_TOKEN","INTAKE_TOKEN",
                      "ADMIN_USER","ADMIN_PASSWORD","TURNSTILE_SECRET","TURNSTILE_SITE_KEY",
                      "REPORTING_ENABLED","BRAIN_ENABLED","BRANDED_EMAIL_ENABLED",
                      "EMAIL_PREVIEW_ENABLED","OPERATOR_SETTINGS_ENABLED",
                      "LANDING_BASE_URL","ADMIN_WEB_BASE_URL","REPORTING_TZ"]
}

1. `coolify_magic` is exactly core's orphan list

All six, one for one. The narrow rule alone makes core read clean. No wide rule needed for the application case.

2. `datastore_ish` kills the wide rule — decisively

It contains DATABASE_URL and REDIS_URL: cast's own manifest-declared vars, resolved from ${resource:postgres.url} / ${resource:redis.url} (#60).

They are harmless today only by accident of ordering — the remove-candidate branch fires solely for !(key in desired.vars), and these are declared, so they never reach it. But that is exactly the point:

The instant someone drops DATABASE_URL from the manifest, it becomes a live-only orphan — and the wide rule would silence the single orphan most worth seeing.

A live-only DATABASE_URL is a connection string still pointing at a box nobody declares any more. That is the precise poison draft.ts was built to refuse ("the new box comes up WORKING, reading and writing the old box's database … you find out the day the old box is deleted"). Applying isProviderGenerated's width in diffEnv would make the diff hide it, silently, forever.

So: narrow (COOLIFY_MAGIC prefix) in diffEnv, wide (isProviderGenerated) in draft — and the divergence is now evidence-backed rather than a judgment call. The checklist item stands as written.

3. `everything_else` confirms no over-reach

Twenty entries, every one manifest-declared. Nothing in that bucket should be silenced, and nothing would be.

Still open: umami's `POSTGRES_*`

This probe covered the application. The service still needs its own (/services/<uuid>/envs), and it holds the contested three:

umami var	narrow?
`SERVICE_URL_UMAMI`, `SERVICE_FQDN_UMAMI`, `SERVICE_URL_UMAMI_3000`, `SERVICE_FQDN_UMAMI_3000`, `SERVICE_USER_POSTGRES`, `SERVICE_PASSWORD_POSTGRES`, `SERVICE_PASSWORD_64_UMAMI`	✅ silenced (7)	all match `^SERVICE_(…)(_
`POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB`	❌ still reported (3)	wide-only

So narrow gets 13 of 16 box-wide, and umami keeps printing three lines — which does not finish what this issue is about.

Those three are the one-click template's bundled-Postgres credentials: genuinely Coolify's, genuinely undeclarable, and not SERVICE_-prefixed. Options, in preference order:

Scope the wide rule to kind === "service" only. A Coolify service is a vendored bundle whose internals cast does not model — an undeclared datastore var there is the template's, never a hand-left one. The DATABASE_URL hazard above is an application hazard; applications keep the narrow rule. This buys all 16 without blinding the case that matters.
Leave the three reported, and accept "clean minus three".
Extend COOLIFY_MAGIC to name the one-click Postgres trio explicitly — cheap, but a list that rots per template.

(1) looks right and is narrow in the way that counts, but it deserves a second opinion before it lands.

Worth noting: incubator D-279 retires umami-as-a-one-click-service in favour of a self-managed app on our own Postgres, which deletes this case rather than solving it — so (2) is not unreasonable as a holding position either.

**Probe run against prod `core`** — this settles the narrow-vs-wide decision empirically. ```json { "coolify_magic": ["SERVICE_URL_API","SERVICE_FQDN_API","SERVICE_URL_ADMIN", "SERVICE_FQDN_ADMIN","SERVICE_URL_INTAKE","SERVICE_FQDN_INTAKE"], "datastore_ish": ["DATABASE_URL","REDIS_URL"], "everything_else": ["NODE_ENV","MAILGUN_API_KEY","MAILGUN_SIGNING_KEY","MAILGUN_DOMAIN", "OPENROUTER_API_KEY","ADMIN_EMAIL","ADMIN_API_TOKEN","INTAKE_TOKEN", "ADMIN_USER","ADMIN_PASSWORD","TURNSTILE_SECRET","TURNSTILE_SITE_KEY", "REPORTING_ENABLED","BRAIN_ENABLED","BRANDED_EMAIL_ENABLED", "EMAIL_PREVIEW_ENABLED","OPERATOR_SETTINGS_ENABLED", "LANDING_BASE_URL","ADMIN_WEB_BASE_URL","REPORTING_TZ"] } ``` ### 1. `coolify_magic` is exactly core's orphan list All six, one for one. **The narrow rule alone makes `core` read clean.** No wide rule needed for the application case. ### 2. `datastore_ish` kills the wide rule — decisively It contains **`DATABASE_URL` and `REDIS_URL`**: cast's **own manifest-declared vars**, resolved from `${resource:postgres.url}` / `${resource:redis.url}` (#60). They are harmless *today* only by accident of ordering — the remove-candidate branch fires solely for `!(key in desired.vars)`, and these are declared, so they never reach it. But that is exactly the point: > The instant someone drops `DATABASE_URL` from the manifest, it becomes a live-only orphan — and the **wide** rule would silence the single orphan most worth seeing. A live-only `DATABASE_URL` is a connection string still pointing at a box nobody declares any more. That is the *precise* poison `draft.ts` was built to refuse ("the new box comes up WORKING, reading and writing the old box's database … you find out the day the old box is deleted"). Applying `isProviderGenerated`'s width in `diffEnv` would make the diff hide it, silently, forever. So: **narrow (`COOLIFY_MAGIC` prefix) in `diffEnv`, wide (`isProviderGenerated`) in `draft`** — and the divergence is now evidence-backed rather than a judgment call. The checklist item stands as written. ### 3. `everything_else` confirms no over-reach Twenty entries, every one manifest-declared. Nothing in that bucket should be silenced, and nothing would be. ## Still open: umami's `POSTGRES_*` This probe covered the **application**. The service still needs its own (`/services/<uuid>/envs`), and it holds the contested three: | umami var | narrow? | | |---|---|---| | `SERVICE_URL_UMAMI`, `SERVICE_FQDN_UMAMI`, `SERVICE_URL_UMAMI_3000`, `SERVICE_FQDN_UMAMI_3000`, `SERVICE_USER_POSTGRES`, `SERVICE_PASSWORD_POSTGRES`, `SERVICE_PASSWORD_64_UMAMI` | ✅ silenced (7) | all match `^SERVICE_(…)(_|$)` | | `POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB` | ❌ still reported (3) | wide-only | So narrow gets **13 of 16** box-wide, and umami keeps printing three lines — which does not finish what this issue is about. Those three are the one-click template's bundled-Postgres credentials: genuinely Coolify's, genuinely undeclarable, and *not* `SERVICE_`-prefixed. Options, in preference order: 1. **Scope the wide rule to `kind === "service"` only.** A Coolify *service* is a vendored bundle whose internals cast does not model — an undeclared datastore var there is the template's, never a hand-left one. The `DATABASE_URL` hazard above is an **application** hazard; applications keep the narrow rule. This buys all 16 without blinding the case that matters. 2. Leave the three reported, and accept "clean minus three". 3. Extend `COOLIFY_MAGIC` to name the one-click Postgres trio explicitly — cheap, but a list that rots per template. (1) looks right and is narrow in the way that counts, but it deserves a second opinion before it lands. Worth noting: incubator D-279 retires umami-as-a-one-click-service in favour of a self-managed app on our own Postgres, which deletes this case rather than solving it — so (2) is not unreasonable as a holding position either.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/cast#87

No description provided.

Coolify's own SERVICE_*/POSTGRES_* magic vars report as orphans forever — a correct box can never read clean #87