feat(capture): --generated-only, the bootstrap's missing pass 2 #58

Merged

dan-claude-bot merged 1 commit from feat/capture-generated-only into main

2026-07-14 22:49:50 +00:00

dan-claude-bot commented

2026-07-14 22:32:51 +00:00

(Migrated from github.com)

The problem

A manifest that declares generated_secrets: bootstraps in two passes, by construction — the value does not exist until Coolify makes it. cast had pass 1 and no pass 2:

capture placeholds the generated names (pending-coolify-generated) — correctly: copying a source box's DATABASE_URL into a new box's store is the silent, unrecoverable failure that rule exists to prevent.
apply creates the database; Coolify generates the real URL.
…nothing taught the store that value.

So the operator did step 3 by hand: decrypt a fourteen-name store, edit two lines, re-encrypt to the environment's age recipient, against production, holding the prod key, with a jq filter that must not pick the wrong row.

Until it runs, the store says pending-coolify-generated while the live value is real — exactly the state in which the next routine apply overwrites a working secret (#47). #47 stops the shot; this removes the gun. It is also on the DR path: rebuild the control plane from state means apply-from-nothing, so every generated secret in every store is a placeholder again.

What changed

cast capture <org>/<repo> --env <env> --generated-only [--from <NAME>=<db>] [--force]

Same verb, same ceremony, same store-writing code path. The flag inverts capture.ts's disposition rule: names in generated_secrets are the ones it fills, and every other name is left exactly as the store has it, byte for byte — never re-read from the box, so a secret rotated by hand last month survives it. The store must already exist: pass 2 fills names, it does not create them.

Each refusal, and the assertion:


UNMAPPED	cast cannot attribute the name to exactly one database → refuses, hands back `--from`
OCCUPIED	the name already holds a real value → refuses without `--force` (it would silently rotate a live credential)
ABSENT	the name is not in the store at all → refuses (pass 1 has not run, or wrong store)
PENDING	a placeholder in a name nothing here fills → refuses (the store would still be a lie)
postcondition	after the write, re-decrypted from disk: zero `pending-coolify-generated` remain, and the name count is unchanged

The plan prints names only — capture.ts:166's rule holds, and the only value-shaped thing printed is the placeholder literal being replaced. apply deliberately does not do this automatically after a create, for the reason the issue gives.

On not guessing which database a name comes from. Nothing in the system carries that edge: generated_secrets: is a flat list of names, the env template knows only DATABASE_URL=${DATABASE_URL}, and the box does not say. So cast infers only when the inference cannot be wrong (one generated name, one database) and otherwise refuses with a ready-to-paste --from DATABASE_URL=incubator-db. Reading the type out of the name (REDIS_URL → the redis one) is precisely the bug this must not have: a name-directed pick is wrong silently, and what it writes is a well-formed URL to somebody else's database. The clean long-term fix is a manifest that can express the edge (generated_secrets: {DATABASE_URL: {from: postgres}}) — that is a manifest.ts schema change, outside this PR's file scope; noted below.

Field-name evidence, from the Coolify source

The vendored reference/coolify-openapi-4.1.2.json documents the GET /databases body as "Content is very complex. Will be implemented later.", so the field names were settled from coollabsio/coolify v4.1.2 directly:

internal_db_url is the same key on BOTH database types. app/Models/StandalonePostgresql.php:61 and app/Models/StandaloneRedis.php:54 each carry protected $appends = ['internal_db_url', 'external_db_url', 'database_type', 'server_status'];. It is an appended accessor, not a column. The Redis field name is internal_db_url — the issue anticipated it might differ, and it does not. Only the URL the accessor builds differs: postgres://{user}:{pw}@{uuid}:5432/{db} (StandalonePostgresql.php:307-325) vs redis://{user}:{pw}@{uuid}:6379/0 (StandaloneRedis.php:295-315, which also switches to rediss:///6380 under enable_ssl, and omits the username segment below Redis 6.0).
The value is read from GET /projects/{uuid}/{env}, not GET /databases — and this is a deliberate deviation from the issue's letter, on two independent grounds:
1. Scope. ProjectController@environment_details (ProjectController.php:167) eager-loads ['applications', 'postgresqls', 'redis', 'mongodbs', 'mysqls', 'mariadbs', 'services'] for this environment. The instance-wide GET /databases (DatabasesController@databases:80-105) merges every database of every project the team owns — so picking ours out of it means matching by name across a list where a collision is possible and silent (#29 in another hat; the hand-run jq carried a comment about not taking the third row, umami's bundled Postgres). On the environment route that bug cannot be expressed — the scoping is structural rather than a filter cast has to get right.
2. Permission. DatabasesController::removeSensitiveData() (DatabasesController.php:30-49) calls makeHidden(['internal_db_url', 'external_db_url', 'postgres_password', 'redis_password', …]) whenever request()->attributes->get('can_read_sensitive', false) === false — so on GET /databases the field is absent for a token without the sensitive-read permission, silently. environment_details calls no removeSensitiveData() and returns serializeApiResponse($environment), which (bootstrap/helpers/api.php:29-88) only sorts keys and hides nothing — so the appended attribute rides along, ungated. The route the issue proposed would have failed open on a lesser token; this one does not. cast still refuses loudly if a database reports no internal_db_url, rather than filling a secret with "".

Tests added (+32; 294 → 326, all green)

npm run check && npm run build && npm test — the gate CI runs, in that order.

test/capture.test.ts (unit, +16): the mapping refuses to guess across two databases and names the candidates; infers the only database when there is exactly one; still refuses when one database must serve two generated names; refuses a --from naming a database outside the project+env. The plan fills and keeps byte-for-byte, refuses OCCUPIED, --force overrides it, refuses ABSENT and PENDING, and never double-reports a refused name as keep. The renderer prints no value. assertGeneratedComplete catches a placeholder left standing, a name lost on the way through, and a name added.

test/capture-cli.test.ts (end-to-end, +16): the real dist/cli.js, a real age identity round-trip, a stub Coolify. A pass-1 store is seeded, pass 2 runs, and the ciphertext is decrypted and asserted on: DATABASE_URL gets the Postgres URL, REDIS_URL gets the Redis URL (not the Postgres one under a Redis name), MAILGUN_API_KEY survives byte-for-byte, zero placeholders remain, name count unchanged. The stub records every path requested, and one test asserts cast never touches /databases — where the stub serves a third row, another project's incubator-db, that a name-directed lookup would eventually take. Plus: no secret value on the console, every refusal above, the typed confirmation, and the three flag-pairing refusals (--override, --from without --generated-only, --resource).

No live Coolify was contacted; there are no credentials in this box.

Noticed, not fixed (staying inside my files)

The manifest cannot express which database a generated name comes from. generated_secrets: is z.array(z.string()). --from is the operator-supplied edge; a manifest.ts schema change would let cast infer it with no flag and no guess. Worth its own issue.
capture's --force now carries two meanings depending on the pass (pass 1: overwrite the whole store; pass 2: rotate a generated name that already holds a real value). They are documented and cannot collide — pass 2 requires the store to exist — but a future --rotate might read better.

Closes #48.

## The problem A manifest that declares `generated_secrets:` **bootstraps in two passes, by construction** — the value does not exist until Coolify makes it. cast had pass 1 and no pass 2: 1. `capture` placeholds the generated names (`pending-coolify-generated`) — correctly: copying a *source* box's `DATABASE_URL` into a *new* box's store is the silent, unrecoverable failure that rule exists to prevent. 2. `apply` creates the database; Coolify generates the real URL. 3. …nothing taught the store that value. So the operator did step 3 by hand: decrypt a fourteen-name store, edit two lines, re-encrypt to the environment's age recipient, against production, holding the prod key, with a `jq` filter that must not pick the wrong row. Until it runs, the store says `pending-coolify-generated` while the live value is real — exactly the state in which the next routine `apply` overwrites a working secret (**#47**). #47 stops the shot; this removes the gun. It is also on the DR path: *rebuild the control plane from state* means apply-from-nothing, so every generated secret in every store is a placeholder again. ## What changed `cast capture <org>/<repo> --env <env> --generated-only [--from <NAME>=<db>] [--force]` Same verb, same ceremony, same store-writing code path. The flag **inverts** `capture.ts`'s disposition rule: names in `generated_secrets` are the ones it *fills*, and every other name is left **exactly as the store has it, byte for byte** — never re-read from the box, so a secret rotated by hand last month survives it. The store must already exist: pass 2 fills names, it does not create them. Each refusal, and the assertion: | | | | --- | --- | | **UNMAPPED** | cast cannot attribute the name to exactly one database → refuses, hands back `--from` | | **OCCUPIED** | the name already holds a *real* value → refuses without `--force` (it would silently rotate a live credential) | | **ABSENT** | the name is not in the store at all → refuses (pass 1 has not run, or wrong store) | | **PENDING** | a placeholder in a name nothing here fills → refuses (the store would still be a lie) | | **postcondition** | after the write, re-decrypted from disk: zero `pending-coolify-generated` remain, and the name count is unchanged | The plan prints **names only** — `capture.ts:166`'s rule holds, and the only value-shaped thing printed is the placeholder literal being replaced. `apply` deliberately does **not** do this automatically after a create, for the reason the issue gives. **On not guessing which database a name comes from.** Nothing in the system carries that edge: `generated_secrets:` is a flat list of *names*, the env template knows only `DATABASE_URL=${DATABASE_URL}`, and the box does not say. So cast infers **only when the inference cannot be wrong** (one generated name, one database) and otherwise refuses with a ready-to-paste `--from DATABASE_URL=incubator-db`. Reading the type out of the *name* (`REDIS_URL` → the redis one) is precisely the bug this must not have: a name-directed pick is wrong **silently**, and what it writes is a well-formed URL to somebody else's database. The clean long-term fix is a manifest that can express the edge (`generated_secrets: {DATABASE_URL: {from: postgres}}`) — that is a `manifest.ts` schema change, outside this PR's file scope; noted below. ## Field-name evidence, from the Coolify source The vendored `reference/coolify-openapi-4.1.2.json` documents the `GET /databases` body as *"Content is very complex. Will be implemented later."*, so the field names were settled from `coollabsio/coolify` v4.1.2 directly: - **`internal_db_url` is the same key on BOTH database types.** `app/Models/StandalonePostgresql.php:61` and `app/Models/StandaloneRedis.php:54` each carry `protected $appends = ['internal_db_url', 'external_db_url', 'database_type', 'server_status'];`. It is an appended *accessor*, not a column. **The Redis field name is `internal_db_url`** — the issue anticipated it might differ, and it does not. Only the URL the accessor *builds* differs: `postgres://{user}:{pw}@{uuid}:5432/{db}` (`StandalonePostgresql.php:307-325`) vs `redis://{user}:{pw}@{uuid}:6379/0` (`StandaloneRedis.php:295-315`, which also switches to `rediss://`/6380 under `enable_ssl`, and omits the username segment below Redis 6.0). - **The value is read from `GET /projects/{uuid}/{env}`, not `GET /databases`** — and this is a deliberate deviation from the issue's letter, on two independent grounds: 1. **Scope.** `ProjectController@environment_details` (`ProjectController.php:167`) eager-loads `['applications', 'postgresqls', 'redis', 'mongodbs', 'mysqls', 'mariadbs', 'services']` for *this* environment. The instance-wide `GET /databases` (`DatabasesController@databases:80-105`) merges every database of every project the team owns — so picking ours out of it means matching by name across a list where a collision is possible and silent (#29 in another hat; the hand-run `jq` carried a comment about not taking the third row, umami's bundled Postgres). On the environment route that bug **cannot be expressed** — the scoping is structural rather than a filter cast has to get right. 2. **Permission.** `DatabasesController::removeSensitiveData()` (`DatabasesController.php:30-49`) calls `makeHidden(['internal_db_url', 'external_db_url', 'postgres_password', 'redis_password', …])` whenever `request()->attributes->get('can_read_sensitive', false) === false` — so on `GET /databases` the field is **absent for a token without the sensitive-read permission**, silently. `environment_details` calls no `removeSensitiveData()` and returns `serializeApiResponse($environment)`, which (`bootstrap/helpers/api.php:29-88`) only sorts keys and hides nothing — so the appended attribute rides along, ungated. The route the issue proposed would have failed open on a lesser token; this one does not. cast still refuses loudly if a database reports no `internal_db_url`, rather than filling a secret with `""`. ## Tests added (+32; 294 → 326, all green) `npm run check && npm run build && npm test` — the gate CI runs, in that order. **`test/capture.test.ts`** (unit, +16): the mapping refuses to guess across two databases and names the candidates; infers the only database when there is exactly one; still refuses when one database must serve two generated names; refuses a `--from` naming a database outside the project+env. The plan fills and keeps byte-for-byte, refuses OCCUPIED, `--force` overrides it, refuses ABSENT and PENDING, and never double-reports a refused name as `keep`. The renderer prints no value. `assertGeneratedComplete` catches a placeholder left standing, a name **lost** on the way through, and a name added. **`test/capture-cli.test.ts`** (end-to-end, +16): the real `dist/cli.js`, a real `age` identity round-trip, a stub Coolify. A pass-1 store is seeded, pass 2 runs, and the **ciphertext is decrypted and asserted on**: `DATABASE_URL` gets the Postgres URL, `REDIS_URL` gets the *Redis* URL (not the Postgres one under a Redis name), `MAILGUN_API_KEY` survives byte-for-byte, zero placeholders remain, name count unchanged. The stub **records every path requested**, and one test asserts cast never touches `/databases` — where the stub serves a third row, another project's `incubator-db`, that a name-directed lookup would eventually take. Plus: no secret value on the console, every refusal above, the typed confirmation, and the three flag-pairing refusals (`--override`, `--from` without `--generated-only`, `--resource`). No live Coolify was contacted; there are no credentials in this box. ## Noticed, not fixed (staying inside my files) - **The manifest cannot express which database a generated name comes from.** `generated_secrets:` is `z.array(z.string())`. `--from` is the operator-supplied edge; a `manifest.ts` schema change would let cast infer it with no flag and no guess. Worth its own issue. - `capture`'s `--force` now carries two meanings depending on the pass (pass 1: overwrite the whole store; pass 2: rotate a generated name that already holds a real value). They are documented and cannot collide — pass 2 requires the store to exist — but a future `--rotate` might read better. Closes #48.

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/cast#58

No description provided.