cast smoke's canary appears to live in the PREVIEW env set — the never-delete guarantee is verified against rows apply never writes #89

New issue

Closed

opened 2026-07-16 17:06:29 +00:00 by dan-claude-bot · 0 comments

dan-claude-bot commented

2026-07-16 17:06:29 +00:00

(Migrated from github.com)

Summary

INFRA_SMOKE_KEEP and INFRA_SMOKE_PROBE — the canary and probe cast smoke writes — appear to live in the preview env set on prod. If so, smoke's central guarantee is being verified against rows apply never touches.

The evidence, by elimination

Both vars were reported as live-only orphans on core in every cast diff --env prod --full of 2026-07-16 — and vanished the moment #86 landed, with nothing between the two runs but a cast reinstall:

   update application core
-  env INFRA_SMOKE_KEEP: live-only (orphan var — apply never removes)
-  env INFRA_SMOKE_PROBE: live-only (orphan var — apply never removes)

#86 changed exactly one thing about what fetchEnv returns:

.filter((e) => e.is_preview !== true)

That predicate is strict. A numeric 1, a string, an absent field — all !== true, all kept. The only rows it can drop are ones where is_preview is boolean true. So both smoke vars are is_preview: true on this box.

Why that shouldn't be possible

smoke.ts sends is_preview: false on both writes — the singular POST for the canary and the bulk PATCH for the probe:

await client.post(envsPath, { key: KEEP_KEY, value: "1", is_buildtime: false, is_preview: false });
await client.patch(`${envsPath}/bulk`, { data: [{ key: PROBE_KEY, value: "1", is_buildtime: false, is_preview: false }] });

And Coolify honours the payload without duplicating (ApplicationsController@create_bulk_envs, v4.1.2):

$is_preview = $item->get('is_preview') ?? false;
…
$env = $application->environment_variables()->create([... 'is_preview' => $is_preview]);

Even an older cast that sent no is_preview would land on ?? false. So on the code as written these should be production rows, and #86 should have kept them.

Why it matters — smoke is testing the wrong rows

smoke exists to prove one thing, and smoke.ts says so itself:

The bulk envs endpoint is documented/verified as UPSERT-only (never deletes unlisted keys) — that's the load-bearing guarantee behind the iron rule that apply never deletes. If a Coolify upgrade regresses it to full-replace, KEEP_KEY will vanish from the read-back below.

If the canary and probe sit in the preview set, then:

the guarantee is being exercised against rows apply's syncEnv never writes (it sends is_preview: false), so a full-replace regression on the production set would not move the canary — smoke would stay green through exactly the regression it exists to catch;
and cast smoke has been leaving two permanent vars in a set nobody looks at.

A canary in the wrong mine is worse than no canary: it reports safety it never measured.

Probe

COOLIFY=http://coolify-box:8000
TOKEN=...
UUID=<core-uuid>

curl -sS -H "Authorization: Bearer $TOKEN" "$COOLIFY/api/v1/applications/$UUID/envs" \
| jq 'map(select(.key | startswith("INFRA_SMOKE")) | {key, value, is_preview, id})'

rows with is_preview: true → confirms the above; smoke's canary is in the preview set.
rows with is_preview: false → then #86 is dropping production rows, which is the more serious outcome: #86 must be narrowed or reverted, and the reasoning above is wrong somewhere.
[] → they were deleted independently and this whole issue is moot.

What to do

Run the probe; the two outcomes point at opposite fixes, so nothing should change before it
If preview: find out how they got there (the payload says otherwise — check the singular POST path, and whether these rows predate is_preview: false being sent)
Make smoke assert what it created: it already reads the env list back to check KEEP_KEY survived — checking that the surviving row is the is_preview: false one is the same read, and it is the difference between a canary and a decoration
Have smoke clean up after itself, or say in the report that it leaves two vars behind (they surfaced as orphans on every diff until #86 hid them — which is its own reason this went unnoticed)

Found while verifying #85/#86 against prod (the note landed on #85 after it had already closed, hence this issue).

## Summary `INFRA_SMOKE_KEEP` and `INFRA_SMOKE_PROBE` — the canary and probe `cast smoke` writes — appear to live in the **preview** env set on prod. If so, smoke's central guarantee is being verified against rows `apply` never touches. ## The evidence, by elimination Both vars were reported as live-only orphans on `core` in every `cast diff --env prod --full` of 2026-07-16 — and **vanished the moment #86 landed**, with nothing between the two runs but a `cast` reinstall: ```diff update application core - env INFRA_SMOKE_KEEP: live-only (orphan var — apply never removes) - env INFRA_SMOKE_PROBE: live-only (orphan var — apply never removes) ``` #86 changed exactly one thing about what `fetchEnv` returns: ```ts .filter((e) => e.is_preview !== true) ``` That predicate is **strict**. A numeric `1`, a string, an absent field — all `!== true`, all kept. The only rows it can drop are ones where `is_preview` is boolean `true`. So both smoke vars are `is_preview: true` on this box. ## Why that shouldn't be possible `smoke.ts` sends `is_preview: false` on **both** writes — the singular POST for the canary and the bulk PATCH for the probe: ```ts await client.post(envsPath, { key: KEEP_KEY, value: "1", is_buildtime: false, is_preview: false }); await client.patch(`${envsPath}/bulk`, { data: [{ key: PROBE_KEY, value: "1", is_buildtime: false, is_preview: false }] }); ``` And Coolify honours the payload without duplicating (`ApplicationsController@create_bulk_envs`, v4.1.2): ```php $is_preview = $item->get('is_preview') ?? false; … $env = $application->environment_variables()->create([... 'is_preview' => $is_preview]); ``` Even an older cast that sent no `is_preview` would land on `?? false`. So on the code as written these should be **production** rows, and #86 should have kept them. ## Why it matters — smoke is testing the wrong rows `smoke` exists to prove one thing, and `smoke.ts` says so itself: > The bulk envs endpoint is documented/verified as UPSERT-only (never deletes unlisted keys) — that's the load-bearing guarantee behind the iron rule that **apply never deletes**. If a Coolify upgrade regresses it to full-replace, KEEP_KEY will vanish from the read-back below. If the canary and probe sit in the **preview** set, then: - the guarantee is being exercised against rows `apply`'s `syncEnv` never writes (it sends `is_preview: false`), so a full-replace regression **on the production set** would not move the canary — smoke would stay green through exactly the regression it exists to catch; - and `cast smoke` has been leaving two permanent vars in a set nobody looks at. A canary in the wrong mine is worse than no canary: it reports safety it never measured. ## Probe ```bash COOLIFY=http://coolify-box:8000 TOKEN=... UUID=<core-uuid> curl -sS -H "Authorization: Bearer $TOKEN" "$COOLIFY/api/v1/applications/$UUID/envs" \ | jq 'map(select(.key | startswith("INFRA_SMOKE")) | {key, value, is_preview, id})' ``` - rows with **`is_preview: true`** → confirms the above; smoke's canary is in the preview set. - rows with **`is_preview: false`** → then #86 is dropping **production** rows, which is the more serious outcome: #86 must be narrowed or reverted, and the reasoning above is wrong somewhere. - **`[]`** → they were deleted independently and this whole issue is moot. ## What to do - [ ] Run the probe; the two outcomes point at opposite fixes, so nothing should change before it - [ ] If preview: find out how they got there (the payload says otherwise — check the singular POST path, and whether these rows predate `is_preview: false` being sent) - [ ] Make `smoke` **assert what it created**: it already reads the env list back to check `KEEP_KEY` survived — checking that the surviving row is the `is_preview: false` one is the same read, and it is the difference between a canary and a decoration - [ ] Have smoke clean up after itself, or say in the report that it leaves two vars behind (they surfaced as orphans on every diff until #86 hid them — which is its own reason this went unnoticed) Found while verifying #85/#86 against prod (the note landed on #85 after it had already closed, hence this issue).

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/cast#89

No description provided.