heavy-duty/cast

Fork 0

cast diff re-proposes the five prod flags as `change` — root cause UNKNOWN (real_value-stale theory disproved) #78

New issue

Closed

opened 2026-07-16 13:42:01 +00:00 by dan-claude-bot · 2 comments

dan-claude-bot commented

2026-07-16 13:42:01 +00:00

(Migrated from github.com)

Summary

cast diff reports a phantom env … : change for an env var that was updated in place and is genuinely correct on the box. The live value the operator sees (Coolify UI) and the value the container actually runs with are both right; only cast's diff disagrees — so a re-apply "changes" it again, and every future diff re-proposes it. A false-drift that never clears also masks real drift.

Reproduction (prod flag flip, 2026-07-16)

Manifest flips five env vars false→true (REPORTING_ENABLED, BRAIN_ENABLED, BRANDED_EMAIL_ENABLED, EMAIL_PREVIEW_ENABLED, OPERATOR_SETTINGS_ENABLED).
cast apply --env prod sets them and redeploys core. Verified live: Coolify UI shows value=true for all five; the running container booted with them on (admin nav renders the flag-gated pages; /version 200 ⇒ loadConfig didn't throw).
cast diff --env prod --full immediately after still lists all five as env … : change.
Vars that were only ever created (NODE_ENV, REPORTING_TZ, the derived base-URLs) do not re-propose — only the five that were flipped in place.

Root cause

fetchEnv (src/cli.ts) builds the live map preferring real_value:

})) as Array<{ key: string; real_value?: string; value: string }>;
return Object.fromEntries(envs.map((e) => [e.key, e.real_value ?? e.value]));

diffEnv (src/diff.ts) then compares live[key] !== v.value against the manifest literal "true".

Coolify's stored real_value goes stale after an in-place PATCH of value: the update sets value="true" but leaves real_value="false" (a redeploy does not refresh it — the drift persists after the apply's redeploy). cast reads the stale real_value → "false" !== "true" → phantom change. Created-once vars have a fresh real_value, so they match — which is exactly why only the flipped-in-place vars re-propose.

Confirming evidence

GET /api/v1/applications/{core-uuid}/envs for the five flags shows value:"true" alongside real_value:"false". (operator to attach the exact JSON)

Impact

False drift that never clears; a re-apply re-writes a value that is already correct.
Masks real drift — an operator who learns these five "always show change" stops trusting the diff on them.
Can bite secret rotations the same way: an in-place secret change could leave real_value stale → a phantom secret X differs.

Proposed fix

Not a blanket "read value" — the real_value ?? value choice is deliberate: for secrets, value is masked and real_value is the plaintext (needs read:sensitive), so cast must keep real_value there. The stale-real_value hazard is specific to non-secret in-place updates.

Two directions:

cast-side (preferred): compare non-secret vars against value (always fresh), keep real_value for secrets. cast already knows v.secret on the desired side (ResolvedEnv), so the choice can be made per-var. fetchEnv currently discards the {value, real_value} split before the diff sees it — either carry both through to diffEnv, or have diffEnv pick per v.secret.
root-side: make syncEnv's update force Coolify to recompute real_value (it's a stored column the API PATCH doesn't refresh; a redeploy doesn't either).

Repro-test outline

diffEnv / fetchEnv: live var {key:"REPORTING_ENABLED", value:"true", real_value:"false"}, non-secret, desired "true" → no diff.
Regression guard: a secret var whose value is masked and real_value is the plaintext must still compare via real_value (don't break the secret path).

Context

Surfaced by the incubator prod migration (heavy-duty/incubator D-280); prod is correct, this is a diff-fidelity bug.
Cousin of the Coolify-4.1.2 read-back-fidelity family already handled explicitly in cast (is_static cast#68; destination/backup "declared, NOT compared"). This one currently produces false drift rather than a "not compared" line.
Immediate operator workaround: re-save the five in the Coolify UI (recomputes real_value) → next diff clean.

## Summary `cast diff` reports a phantom `env … : change` for an env var that was **updated in place** and is genuinely correct on the box. The live value the operator sees (Coolify UI) and the value the container actually runs with are both right; only cast's diff disagrees — so a re-`apply` "changes" it again, and every future diff re-proposes it. A false-drift that never clears also **masks real drift**. ## Reproduction (prod flag flip, 2026-07-16) 1. Manifest flips five env vars false→true (`REPORTING_ENABLED`, `BRAIN_ENABLED`, `BRANDED_EMAIL_ENABLED`, `EMAIL_PREVIEW_ENABLED`, `OPERATOR_SETTINGS_ENABLED`). 2. `cast apply --env prod` sets them and redeploys `core`. Verified live: Coolify UI shows `value=true` for all five; the running container booted with them on (admin nav renders the flag-gated pages; `/version` 200 ⇒ `loadConfig` didn't throw). 3. `cast diff --env prod --full` immediately after **still lists all five as `env … : change`**. 4. Vars that were only ever *created* (`NODE_ENV`, `REPORTING_TZ`, the derived base-URLs) do **not** re-propose — only the five that were flipped **in place**. ## Root cause `fetchEnv` (`src/cli.ts`) builds the live map preferring `real_value`: ```js })) as Array<{ key: string; real_value?: string; value: string }>; return Object.fromEntries(envs.map((e) => [e.key, e.real_value ?? e.value])); ``` `diffEnv` (`src/diff.ts`) then compares `live[key] !== v.value` against the manifest literal `"true"`. Coolify's stored **`real_value` goes stale after an in-place PATCH of `value`**: the update sets `value="true"` but leaves `real_value="false"` (a redeploy does **not** refresh it — the drift persists *after* the apply's redeploy). cast reads the stale `real_value` → `"false" !== "true"` → phantom `change`. Created-once vars have a fresh `real_value`, so they match — which is exactly why only the flipped-in-place vars re-propose. ### Confirming evidence `GET /api/v1/applications/{core-uuid}/envs` for the five flags shows `value:"true"` alongside `real_value:"false"`. *(operator to attach the exact JSON)* ## Impact - False drift that never clears; a re-`apply` re-writes a value that is already correct. - **Masks real drift** — an operator who learns these five "always show change" stops trusting the diff on them. - Can bite **secret rotations** the same way: an in-place secret change could leave `real_value` stale → a phantom `secret X differs`. ## Proposed fix Not a blanket "read `value`" — the `real_value ?? value` choice is deliberate: for **secrets**, `value` is masked and `real_value` is the plaintext (needs `read:sensitive`), so cast must keep `real_value` there. The stale-`real_value` hazard is specific to **non-secret** in-place updates. Two directions: 1. **cast-side (preferred):** compare **non-secret** vars against `value` (always fresh), keep `real_value` for **secrets**. cast already knows `v.secret` on the desired side (`ResolvedEnv`), so the choice can be made per-var. `fetchEnv` currently discards the `{value, real_value}` split before the diff sees it — either carry both through to `diffEnv`, or have `diffEnv` pick per `v.secret`. 2. **root-side:** make `syncEnv`'s update force Coolify to recompute `real_value` (it's a stored column the API PATCH doesn't refresh; a redeploy doesn't either). ### Repro-test outline - `diffEnv` / `fetchEnv`: live var `{key:"REPORTING_ENABLED", value:"true", real_value:"false"}`, **non-secret**, desired `"true"` → **no diff**. - Regression guard: a **secret** var whose `value` is masked and `real_value` is the plaintext must still compare via `real_value` (don't break the secret path). ## Context - Surfaced by the incubator prod migration (`heavy-duty/incubator` D-280); prod is correct, this is a diff-fidelity bug. - Cousin of the Coolify-4.1.2 read-back-fidelity family already handled explicitly in cast (`is_static` cast#68; `destination`/`backup` "declared, NOT compared"). This one currently produces *false drift* rather than a "not compared" line. - Immediate operator workaround: re-save the five in the Coolify UI (recomputes `real_value`) → next diff clean.

dan-claude-bot commented

2026-07-16 16:13:02 +00:00

(Migrated from github.com)

Reopening: the symptom persists, and this issue's stated root cause is wrong.

cast diff --env prod run today against prod, on a cast that provably contains #79 (the same install diffs service_domains, which only exists post-#81):

update application core
  env REPORTING_ENABLED: change
  env BRAIN_ENABLED: change
  env BRANDED_EMAIL_ENABLED: change
  env EMAIL_PREVIEW_ENABLED: change
  env OPERATOR_SETTINGS_ENABLED: change

Still all five. #79 did not fix this.

The root cause above is false

Coolify's stored real_value goes stale after an in-place PATCH of value … it's a stored column the API PATCH doesn't refresh

real_value is not a stored column. It is an Attribute accessor, declared in $appends and recomputed from value on every read (app/Models/EnvironmentVariable.php:81,171-207 @ v4.1.2):

protected $appends = ['real_value', 'is_shared', ...];

public function realValue(): Attribute {
    return Attribute::make(get: function () {
        $real_value = $this->get_real_environment_variables($this->value, $resource);
        if (json_validate($real_value) && (str_starts_with($real_value,'{') || str_starts_with($real_value,'['))) return $real_value;
        if ($this->is_literal || $this->is_multiline) $real_value = '\''.$real_value.'\'';
        else $real_value = escapeEnvVariables($real_value);
        return $real_value;
    });
}

It therefore cannot go stale. What it actually is: a shell-escaped / quoted rendering of value — single-quoted when is_literal/is_multiline, otherwise escapeEnvVariables(...). And value itself is trim(decrypt(...)) (:150-156), the raw decrypted value.

The "confirming evidence" this issue rested on (value:"true" alongside real_value:"false") was never actually captured — the body says "(operator to attach the exact JSON)". Nothing here was ever verified against the wire.

What that means for #79

#79 still fixes a real defect, and I'd keep it: comparing a manifest literal against an escaped rendering is wrong on its face, and would produce phantom drift for any value whose escaping differs from its raw form (is_literal ⇒ 'true' ≠ true; anything escapeEnvVariables touches). It just isn't what is biting these five.

Where the trail now leads

The five are plain literals in the env template (REPORTING_ENABLED=true, not ${REF}), so secret: false, so post-#79 diffEnv compares them against live.value. They still differ ⇒ live.value is not "true". This is not token-masking either: every other manifest var on the same resource (NODE_ENV, REPORTING_TZ, the derived base-URLs) compares equal and never appears in the diff.

Which leaves the uncomfortable possibility the original diagnosis foreclosed: the stored value on the box may genuinely not be true, whatever the UI renders — i.e. the diff may have been correct all along, and "diagnosed benign" may have been the actual defect.

Next step — the evidence this issue never got

curl -sS -H "Authorization: Bearer $TOKEN" \
  "$COOLIFY/api/v1/applications/<core-uuid>/envs" \
  | jq '.[] | select(.key|test("REPORTING_ENABLED|BRAIN_ENABLED")) | {key,value,real_value,is_literal,is_shared}'

{key, value, real_value, is_literal} for one flag settles it. Blocking any cast apply --env prod until then: if the box really is false, this stops being a diff-fidelity bug and becomes the five prod flags are not actually on.

**Reopening: the symptom persists, and this issue's stated root cause is wrong.** `cast diff --env prod` run today against prod, on a cast that provably contains #79 (the same install diffs `service_domains`, which only exists post-#81): ``` update application core env REPORTING_ENABLED: change env BRAIN_ENABLED: change env BRANDED_EMAIL_ENABLED: change env EMAIL_PREVIEW_ENABLED: change env OPERATOR_SETTINGS_ENABLED: change ``` Still all five. #79 did not fix this. ## The root cause above is false > Coolify's stored `real_value` goes stale after an in-place PATCH of `value` … it's a stored column the API PATCH doesn't refresh `real_value` is **not a stored column**. It is an `Attribute` accessor, declared in `$appends` and recomputed from `value` on **every read** (`app/Models/EnvironmentVariable.php:81,171-207` @ v4.1.2): ```php protected $appends = ['real_value', 'is_shared', ...]; public function realValue(): Attribute { return Attribute::make(get: function () { $real_value = $this->get_real_environment_variables($this->value, $resource); if (json_validate($real_value) && (str_starts_with($real_value,'{') || str_starts_with($real_value,'['))) return $real_value; if ($this->is_literal || $this->is_multiline) $real_value = '\''.$real_value.'\''; else $real_value = escapeEnvVariables($real_value); return $real_value; }); } ``` It therefore **cannot go stale**. What it actually is: a **shell-escaped / quoted rendering** of `value` — single-quoted when `is_literal`/`is_multiline`, otherwise `escapeEnvVariables(...)`. And `value` itself is `trim(decrypt(...))` (`:150-156`), the raw decrypted value. The "confirming evidence" this issue rested on (`value:"true"` alongside `real_value:"false"`) was never actually captured — the body says *"(operator to attach the exact JSON)"*. Nothing here was ever verified against the wire. ## What that means for #79 #79 still fixes a **real** defect, and I'd keep it: comparing a manifest literal against an *escaped rendering* is wrong on its face, and would produce phantom drift for any value whose escaping differs from its raw form (`is_literal` ⇒ `'true'` ≠ `true`; anything `escapeEnvVariables` touches). It just isn't what is biting these five. ## Where the trail now leads The five are **plain literals** in the env template (`REPORTING_ENABLED=true`, not `${REF}`), so `secret: false`, so post-#79 `diffEnv` compares them against `live.value`. They still differ ⇒ **`live.value` is not `"true"`**. This is not token-masking either: every other manifest var on the same resource (`NODE_ENV`, `REPORTING_TZ`, the derived base-URLs) compares equal and never appears in the diff. Which leaves the uncomfortable possibility the original diagnosis foreclosed: **the stored value on the box may genuinely not be `true`**, whatever the UI renders — i.e. the diff may have been *correct all along*, and "diagnosed benign" may have been the actual defect. ## Next step — the evidence this issue never got ```bash curl -sS -H "Authorization: Bearer $TOKEN" \ "$COOLIFY/api/v1/applications/<core-uuid>/envs" \ | jq '.[] | select(.key|test("REPORTING_ENABLED|BRAIN_ENABLED")) | {key,value,real_value,is_literal,is_shared}' ``` `{key, value, real_value, is_literal}` for one flag settles it. Blocking any `cast apply --env prod` until then: if the box really is `false`, this stops being a diff-fidelity bug and becomes *the five prod flags are not actually on*.

dan-claude-bot commented

2026-07-16 16:29:43 +00:00

(Migrated from github.com)

Root cause found: #85. Not real_value — a duplicate row.

The probe came back with two rows per key:

{ "key": "REPORTING_ENABLED", "value": "true",  "real_value": "true",  "is_literal": false, "is_shared": false }
{ "key": "BRAIN_ENABLED",     "value": "true",  "real_value": "true",  "is_literal": false, "is_shared": false }
{ "key": "REPORTING_ENABLED", "value": "false", "real_value": "false", "is_literal": false, "is_shared": false }
{ "key": "BRAIN_ENABLED",     "value": "false", "real_value": "false", "is_literal": false, "is_shared": false }

GET /applications/{uuid}/envs merges two disjoint sets into one flat array (ApplicationsController@envs merges environment_variables with environment_variables_preview), so a key can legitimately arrive twice. cast's fetchEnv collapses that with Object.fromEntries — last wins — so it diffs the manifest's "true" against the trailing "false" row. Full analysis and fix in #85.

Correcting this issue's record, point by point

❌ "Coolify's stored real_value goes stale after an in-place PATCH" — false twice over. real_value is an Attribute accessor recomputed from value on every read, so it cannot go stale; and the probe shows real_value == value on every row, exactly as the accessor predicts (is_literal: false ⇒ escapeEnvVariables("true") == "true"). The evidence line that would have caught this — "(operator to attach the exact JSON)" — was never filled in, and the theory stood for it.
✅ "only the vars flipped in place re-propose; created-once vars don't" — exactly right, and the decisive clue. Both rows start equal at creation; apply PATCHes only the production row, leaving its twin behind. So they diverge for precisely the in-place-updated vars. NODE_ENV/REPORTING_TZ/the ${domain:...} base-URLs never diverged, so last-wins happens to pick a matching value and they read clean through the identical path. The "something goes stale after an in-place PATCH" intuition was sound — the stale thing is a second row, not a computed field.
✅ "prod is correct; this is a diff-fidelity bug" — confirmed. The production rows are "true". The flags really are on; the diff was the thing that was wrong. "Diagnosed benign" reached the right conclusion by the wrong route.
⚠️ "immediate operator workaround: re-save the five in the Coolify UI" — would not have helped, and might have looked like it did: it rewrites the production row, not the shadow.

On #79

Keep it. Comparing a manifest literal against an escaped rendering is wrong regardless (is_literal ⇒ 'true' ≠ true), so it fixes a real defect and a latent secret-rotation hazard — it just was never what was biting these five. Its stated motivation is now wrong and should be re-pointed at #85.

Closing this in favour of #85, which carries the confirmed cause, the probe, and the fix.

**Root cause found: #85.** Not `real_value` — a **duplicate row**. The probe came back with **two rows per key**: ```json { "key": "REPORTING_ENABLED", "value": "true", "real_value": "true", "is_literal": false, "is_shared": false } { "key": "BRAIN_ENABLED", "value": "true", "real_value": "true", "is_literal": false, "is_shared": false } { "key": "REPORTING_ENABLED", "value": "false", "real_value": "false", "is_literal": false, "is_shared": false } { "key": "BRAIN_ENABLED", "value": "false", "real_value": "false", "is_literal": false, "is_shared": false } ``` `GET /applications/{uuid}/envs` merges two disjoint sets into one flat array (`ApplicationsController@envs` merges `environment_variables` with `environment_variables_preview`), so a key can legitimately arrive twice. cast's `fetchEnv` collapses that with `Object.fromEntries` — **last wins** — so it diffs the manifest's `"true"` against the trailing `"false"` row. Full analysis and fix in #85. ### Correcting this issue's record, point by point - ❌ *"Coolify's stored `real_value` goes stale after an in-place PATCH"* — **false twice over.** `real_value` is an `Attribute` accessor recomputed from `value` on every read, so it cannot go stale; and the probe shows `real_value == value` on **every** row, exactly as the accessor predicts (`is_literal: false` ⇒ `escapeEnvVariables("true") == "true"`). The evidence line that would have caught this — *"(operator to attach the exact JSON)"* — was never filled in, and the theory stood for it. - ✅ *"only the vars flipped **in place** re-propose; created-once vars don't"* — **exactly right, and the decisive clue.** Both rows start equal at creation; `apply` PATCHes only the production row, leaving its twin behind. So they diverge for precisely the in-place-updated vars. `NODE_ENV`/`REPORTING_TZ`/the `${domain:...}` base-URLs never diverged, so last-wins happens to pick a matching value and they read clean through the identical path. The "something goes stale after an in-place PATCH" intuition was sound — the stale thing is a **second row**, not a computed field. - ✅ *"prod is correct; this is a diff-fidelity bug"* — **confirmed.** The production rows are `"true"`. The flags really are on; the diff was the thing that was wrong. "Diagnosed benign" reached the right conclusion by the wrong route. - ⚠️ *"immediate operator workaround: re-save the five in the Coolify UI"* — would not have helped, and might have looked like it did: it rewrites the production row, not the shadow. ### On #79 Keep it. Comparing a manifest literal against an escaped rendering is wrong regardless (`is_literal` ⇒ `'true'` ≠ `true`), so it fixes a real defect and a latent secret-rotation hazard — it just was never what was biting these five. Its stated *motivation* is now wrong and should be re-pointed at #85. Closing this in favour of #85, which carries the confirmed cause, the probe, and the fix.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/cast#78

No description provided.

cast diff re-proposes the five prod flags as change — root cause UNKNOWN (real_value-stale theory disproved) #78

Summary

Reproduction (prod flag flip, 2026-07-16)

Root cause

Confirming evidence

Impact

Proposed fix

Repro-test outline

Context

The root cause above is false

What that means for #79

Where the trail now leads

Next step — the evidence this issue never got

Correcting this issue's record, point by point

On #79

cast diff re-proposes the five prod flags as `change` — root cause UNKNOWN (real_value-stale theory disproved) #78