heavy-duty/cast

Fork 0

Derive DATABASE_URL/REDIS_URL from the database resource cast created — delete the two-pass bootstrap instead of automating it #60

New issue

Closed

opened 2026-07-14 22:35:10 +00:00 by dan-claude-bot · 1 comment

dan-claude-bot commented

2026-07-14 22:35:10 +00:00

(Migrated from github.com)

The idea

The manifest already declares both ends of this edge and never connects them:

databases:
  postgres: { type: postgresql, version: "16" }
  redis:    { type: redis }
generated_secrets: [DATABASE_URL_PROD, REDIS_URL_PROD]   # ← a list of NAMES

generated_secrets is a set of names where what it means is a set of edges: DATABASE_URL_PROD is not merely "a value someone else generates", it is the URL of the postgres resource declared six lines up. manifest.ts:95 says so out loud, in the schema's own comment:

"It is a manifest property rather than a flag the operator has to remember, because the manifest is what knows DATABASE_URL comes from a database it declares."

The manifest knows. cast doesn't ask. internal_db_url appears nowhere in cast's source — cli.ts:2063 creates the Postgres/Redis resource and never reads back the one thing Coolify minted while doing it.

So: let the manifest name the edge, and let apply resolve it from the live resource.

# core.prod.env.template
DATABASE_URL=${resource:postgres.url}
REDIS_URL=${resource:redis.url}

resolve.ts already parses ${…} refs (templateRefs) and already validates that every declared generated name is actually referenced by some template (resolve.ts:239-258) — the exact dead-reference check this needs, pointed the other way: a ${resource:X.url} ref naming a database the manifest does not declare is a hard error at plan time.

Why this beats #48

#48 asks for a pass 2 — a verb that captures the generated value back into the age store after apply creates it. That automates the hand-dance. This deletes it.

The store should not hold DATABASE_URL at all. It is not a secret anybody authored; it is a fact about a resource cast itself created, readable from the API that created it, at any time, for free. Storing it means keeping a copy of a value you do not own, in a second place, that can silently disagree with the first — which is precisely the failure in #47.

What derivation deletes, rather than automates:

	today	with #48 (pass 2)	derived
placeholder in the store	yes	yes	gone
store names (incubator prod)	14	14	12
hand `age` re-encrypt against prod	yes	no	no
a second copy that can drift	yes	yes	none exists
`secret DATABASE_URL differs` on every plan	yes	no	no
#47's overwrite	certain	guarded	structurally impossible — nothing in the store to overwrite it with
rebuild-from-nothing	attended	attended	unattended

#47 is still worth keeping as a backstop for any residual generated secret that is genuinely not derivable (a service's own generated credential). But its live instance — the one currently holding a loaded gun on heavy-duty/incubator's prod box — dies with this change, because the two names causing it stop existing in the store.

The security argument, which I think is the real one

Today the prod Postgres password:

is minted by Coolify,
is read by a human off the API or the UI,
passes through that human's shell and scrollback,
is encrypted to an age recipient and committed to git,
is decrypted on every subsequent apply, with the prod key injected from a password manager.

Under derivation it goes: minted by Coolify → read by cast over the API → written to the app's env in the same Coolify. It never lands in git, never enters a terminal, never touches disk, and no key needs to exist to read it back. The age store shrinks to the things a human actually authored — Mailgun, OpenRouter, Turnstile, the admin credentials — which is what a secret store is for.

It also makes rotation work: rotate the DB password in Coolify and the next apply follows it. Today the store silently diverges and the next apply reverts prod to the old value.

Ordering — this needs #45

Derivation only works if the database exists before the app's env is resolved, so this requires #45 (apply creates applications before the databases they depend on). #45 is currently justified by "a first apply always deploys against nothing"; this gives it a second, harder reason — without it, ${resource:postgres.url} has nothing to resolve against on a from-nothing apply. Land #45 first.

Open question (cheap to answer)

Is internal_db_url populated at create time, before the database has started? The value is known to exist and be readable on a running resource — the heavy-duty/incubator runbook reads it with GET /api/v1/databases | jq '.[].internal_db_url', and that is how prod's URLs were obtained. What is unverified is whether it is present on the create response / an immediate GET, which is what a single-pass from-nothing apply needs. Coolify generates the credentials at create time, so it should be — but it should be checked, not assumed. If it turns out to require a started resource, apply needs a small wait-for-resource step, not a redesign.

Scope

Umami stays a UI act, and that is correct (incubator D-253). Coolify 4.1.2's umami service builds its own DATABASE_URL inside its compose file from magic vars it generates against umami's own bundled Postgres. There is no edge for us to declare there, and a value of ours would never be read. This proposal covers only resources cast itself declares and creates.

Suggested shape

${resource:<name>.url} in an env template resolves to the declared database's internal URL (internal, not external: apps are on the Docker network via connect_to_docker_network).
Plan-time validation: a ${resource:…} ref naming an undeclared database is a hard error, with the same "this guards nothing" voice as resolve.ts:250.
diff renders the derived value as derived, and — like every other secret — never prints it.
generated_secrets remains for the residual class (non-derivable provider-generated values), and DATABASE_URL_* / REDIS_URL_* leave it.
Migration for an existing store: the two names are simply dropped. No re-encrypt, no pass 2 — the value they held was never the source of truth.

Filed off the heavy-duty/incubator prod migration (Task 8 step 5), where the hand-age procedure this replaces is currently blocking a second apply against a live prod box.

## The idea The manifest already declares both ends of this edge and never connects them: ```yaml databases: postgres: { type: postgresql, version: "16" } redis: { type: redis } generated_secrets: [DATABASE_URL_PROD, REDIS_URL_PROD] # ← a list of NAMES ``` `generated_secrets` is a set of *names* where what it means is a set of *edges*: `DATABASE_URL_PROD` is not merely "a value someone else generates", it is **the URL of the `postgres` resource declared six lines up**. `manifest.ts:95` says so out loud, in the schema's own comment: > *"It is a manifest property rather than a flag the operator has to remember, because the manifest is what knows `DATABASE_URL` comes from a database it declares."* The manifest knows. cast doesn't ask. **`internal_db_url` appears nowhere in cast's source** — `cli.ts:2063` creates the Postgres/Redis resource and never reads back the one thing Coolify minted while doing it. So: let the manifest name the edge, and let `apply` resolve it from the live resource. ``` # core.prod.env.template DATABASE_URL=${resource:postgres.url} REDIS_URL=${resource:redis.url} ``` `resolve.ts` already parses `${…}` refs (`templateRefs`) and already validates that every declared generated name is actually referenced by some template (`resolve.ts:239-258`) — the exact dead-reference check this needs, pointed the other way: *a `${resource:X.url}` ref naming a database the manifest does not declare is a hard error at plan time.* ## Why this beats #48 #48 asks for a **pass 2** — a verb that captures the generated value back into the age store after `apply` creates it. That automates the hand-dance. This deletes it. The store should not hold `DATABASE_URL` **at all**. It is not a secret anybody authored; it is a fact about a resource cast itself created, readable from the API that created it, at any time, for free. Storing it means keeping a copy of a value you do not own, in a second place, that can silently disagree with the first — which is precisely the failure in #47. What derivation deletes, rather than automates: | | today | with #48 (pass 2) | derived | |---|---|---|---| | placeholder in the store | yes | yes | **gone** | | store names (incubator prod) | 14 | 14 | **12** | | hand `age` re-encrypt against prod | yes | no | **no** | | a second copy that can drift | yes | yes | **none exists** | | `secret DATABASE_URL differs` on every plan | yes | no | **no** | | #47's overwrite | certain | guarded | **structurally impossible** — nothing in the store to overwrite it *with* | | rebuild-from-nothing | attended | attended | **unattended** | #47 is still worth keeping as a backstop for any *residual* generated secret that is genuinely not derivable (a service's own generated credential). But its live instance — the one currently holding a loaded gun on `heavy-duty/incubator`'s prod box — dies with this change, because the two names causing it stop existing in the store. ## The security argument, which I think is the real one Today the prod Postgres password: 1. is minted by Coolify, 2. is read by a human off the API or the UI, 3. passes through that human's shell and scrollback, 4. is encrypted to an age recipient and **committed to git**, 5. is decrypted on every subsequent `apply`, with the prod key injected from a password manager. Under derivation it goes: minted by Coolify → read by cast over the API → written to the app's env in the same Coolify. **It never lands in git, never enters a terminal, never touches disk, and no key needs to exist to read it back.** The `age` store shrinks to the things a human actually authored — Mailgun, OpenRouter, Turnstile, the admin credentials — which is what a secret store is *for*. It also makes rotation work: rotate the DB password in Coolify and the next `apply` follows it. Today the store silently diverges and the next apply reverts prod to the old value. ## Ordering — this needs #45 Derivation only works if the database exists before the app's env is resolved, so this **requires #45** (`apply` creates applications before the databases they depend on). #45 is currently justified by "a first apply always deploys against nothing"; this gives it a second, harder reason — without it, `${resource:postgres.url}` has nothing to resolve against on a from-nothing apply. Land #45 first. ## Open question (cheap to answer) **Is `internal_db_url` populated at create time, before the database has started?** The value is known to exist and be readable on a *running* resource — the `heavy-duty/incubator` runbook reads it with `GET /api/v1/databases | jq '.[].internal_db_url'`, and that is how prod's URLs were obtained. What is unverified is whether it is present on the create response / an immediate GET, which is what a single-pass from-nothing apply needs. Coolify generates the credentials at create time, so it should be — but it should be *checked*, not assumed. If it turns out to require a started resource, `apply` needs a small wait-for-resource step, not a redesign. ## Scope **Umami stays a UI act, and that is correct** (incubator D-253). Coolify 4.1.2's umami service builds its own `DATABASE_URL` inside its compose file from magic vars it generates against umami's *own bundled* Postgres. There is no edge for us to declare there, and a value of ours would never be read. This proposal covers only resources cast itself declares and creates. ## Suggested shape - `${resource:<name>.url}` in an env template resolves to the declared database's internal URL (internal, not external: apps are on the Docker network via `connect_to_docker_network`). - Plan-time validation: a `${resource:…}` ref naming an undeclared database is a hard error, with the same "this guards nothing" voice as `resolve.ts:250`. - `diff` renders the derived value as derived, and — like every other secret — never prints it. - `generated_secrets` remains for the residual class (non-derivable provider-generated values), and `DATABASE_URL_*` / `REDIS_URL_*` leave it. - Migration for an existing store: the two names are simply dropped. No re-encrypt, no pass 2 — the value they held was never the source of truth. --- Filed off the `heavy-duty/incubator` prod migration (Task 8 step 5), where the hand-`age` procedure this replaces is currently blocking a second `apply` against a live prod box.

dan-claude-bot commented

2026-07-14 22:48:50 +00:00

(Migrated from github.com)

Resolution must be per-resource, not environment-wide — a note to pin the design.

generated_secrets and databases are declared at the environment level of the manifest, which makes them look project-wide. They are not: env reaches a resource only through that resource's own env_template (resolve.ts collect() — if (!template) return). There is no project-wide env layer in cast, and that is a security property, not an accident: incubator's landing resource has no env_template, so it carries zero secrets, and a static site has no business holding a database credential.

So ${resource:postgres.url} must resolve through the same per-template path, gated by the existing dead-ref check (resolve.ts:239-258): only a resource whose template actually references the database gets the URL. Do not implement this as "inject the derived DB URL into every app in the environment" — that would hand the credential to landing, which is exactly the blast-radius the current per-resource design avoids. The derived value rides the env_template graph like any other ref; a resource that doesn't ask for it never sees it.

**Resolution must be per-resource, not environment-wide — a note to pin the design.** `generated_secrets` and `databases` are declared at the *environment* level of the manifest, which makes them look project-wide. They are not: env reaches a resource only through that resource's own `env_template` (`resolve.ts` `collect()` — `if (!template) return`). There is no project-wide env layer in cast, and that is a security property, not an accident: incubator's `landing` resource has no `env_template`, so it carries zero secrets, and a static site has no business holding a database credential. So `${resource:postgres.url}` must resolve through the **same per-template path**, gated by the existing dead-ref check (`resolve.ts:239-258`): only a resource whose template actually references the database gets the URL. Do **not** implement this as "inject the derived DB URL into every app in the environment" — that would hand the credential to `landing`, which is exactly the blast-radius the current per-resource design avoids. The derived value rides the env_template graph like any other ref; a resource that doesn't ask for it never sees it.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/cast#60

No description provided.