heavy-duty/cast

Fork 0

cast has a drill gate and a record format, but no instrument — nine legs against two live Coolify instances, run from memory #141

New issue

Open

opened 2026-07-21 16:49:40 +00:00 by dan-claude-bot · 1 comment

dan-claude-bot commented

2026-07-21 16:49:40 +00:00

(Migrated from github.com)

cast has the gate and the record format. It has no instrument, and drills/README.md states the position deliberately:

cast has no drill harness script of its own. Its legs run by hand against the documented procedure … A harness would make the run reproducible; it would not make it recorded. Those are separate problems, and this directory is the second one.

Agreed on the framing, and the ordering was right — the gate is why releases stopped shipping unproven. This issue is the first problem.

What "by hand" currently means

CONTRIBUTING requires nine legs before a release PR can be handed over:

team, apply, an idempotent diff, smoke, inventory, emit-draft, fleet, destroy, and the read-only guard

against two live Coolify instances. None of it is scripted. scripts/ contains one file, restore-db.sh, which is an operator tool and not a drill leg. drills/ holds only README.md — "cast has recorded no drill runs."

So the release ceremony asks an operator to stand up two Coolify instances, drive nine legs in order against them, remember what each one asserts, and write down numbers — from memory, months apart, with no previous record to imitate. The first person to do it will be inventing the procedure, and the second will be inventing a different one.

cast's test suite makes the gap sharper rather than softer. There are 40 .test.ts files, 167 describe blocks, wire.test.ts alone is 64KB — and none of it touches a real Coolify. test/live-lookup.test.ts builds a fake that answers GET /projects. That is the correct design for a unit suite, and it is exactly why the drill matters: everything that is mocked is a claim about Coolify's behaviour that the suite cannot check. box's drill/README.md makes this argument better than I can, and it transfers directly:

the interesting failures are not in the bash — they are in what Incus actually does, which is exactly what unit tests would stub out and get wrong. The drill runs the real thing.

Substitute Coolify for Incus and it is cast's situation precisely, with 40 files of stubs standing in for one live instance.

The legs that only a live run can prove

Two invariants in cast's README have no possible unit test, and they are the ones a promotion drill exists to assert:

"One-way, idempotent, never deletes." The idempotence half is the diff leg — a second apply must yield zero changes, asserted mechanically. The never-deletes half is the read-only guard leg.
The reserved-env-name rule (src/reserved.ts) and smoke's probe key. src/smoke.ts is described in-tree as "the one thing in cast that mutates a live resource purely to learn something" — it writes INFRA_SMOKE_KEEP. The drill has to assert that key is cleaned up and never collides with a reserved name. A mocked Coolify will agree with whatever the code believes about this; a real one will not.

destroy has the same character: it must assert absence afterwards, not just that the delete call returned 200. box's wipe.sh gets this right and says why — "assert the ABSENCE, don't trust the removals' exit codes" — and it is the single most portable idea in box's harness.

What to build

drill/drill.sh (or a TS equivalent — see below), on box's shape:

The reporting verbs — drill/drill.sh:52-58, five one-liners the whole 997-line harness rests on. pass/fail counters, a findings array, phase banners, and ok/no/note that always return 0 so the body can be a flat sequence of probe && ok … || no …. Exit 0 iff fail is 0.

Continue on failure. box's header is explicit that this is not a style choice: "NOT -e: a failing check is data, not a crash." Nine legs against live infrastructure, aborting on the first problem, means one finding per afternoon of setup.

Assert the refs that are actually under test, the way box re-reads INSTALLED_FROM and dies on a mismatch. A drill that believes it exercised the candidate and got main has proven nothing about the combination.

A record emitter in box#152's shape, so the three repos' records reconcile on the shared run ID instead of three hand-invented ones.

Language. box's harness is bash because box is bash. cast is TypeScript with vitest and a globalSetup, and it may well be that cast's drill should be a vitest suite gated behind a CAST_DRILL_LIVE=1 env var, reusing the existing helpers and getting the leg structure for free. I lean that way but do not have a strong view; worth deciding early since it determines everything else. What must not change either way is the contract — nine legs, continue-on-failure, counted, record emitted, loud skips.

Loud skips. A leg that cannot run because an instance is unreachable must not be silently absent from the count. Same defect class as heavy-duty/box#153.

Blocker

The parameterised compose file that stood up instance B for the July drill is not in this repo. CONTRIBUTING references it —

two Coolify instances can be stood up by hand, as the July drill did for instance B via a parameterised compose file

— and cast's legs are described as "the least coupled of the three" specifically because of it. But an operator reading CONTRIBUTING today cannot follow that sentence to anything. Filed separately as heavy-duty/cast#140, and it is the natural first commit: the harness has nothing to run against until instance B is reproducible.

Scope

One script (or suite), nine legs, the verbs, ref assertion, record emitter. Not box's whole drill/ directory — doctor.sh and wipe.sh address a persistent host stack that cast does not create, and multiuser.sh has no analogue.

Not a cross-repo drill framework. heavy-duty/rig#105 is the same shape of issue for rig; the three should converge by copying box's output format, not by sharing code — they assert three different things (box: isolation, rig: convergence, cast: promotion) and the per-repo independence is load-bearing, for the reasons drill-recorded.sh already sets out at length about cross-repo lookups degrading to "pass".

Related: heavy-duty/box#152 (record emitter, defines the shape), heavy-duty/box#153 (skipped legs must not read as passes).

Triage — decisions (2026-08-17)

Language: settled. The drill is a vitest suite gated behind CAST_DRILL_LIVE=1, per the lean stated above — cast is TypeScript, the leg structure and live-client helpers already exist in the suite's shape, and a second toolchain in-tree is a cost with no offsetting assertion. The contract is unchanged and is the review spec: nine legs, continue-on-failure, pass/fail counted, candidate refs asserted, record emitted in box#152's shape, loud skips.
Dependency recorded. Blocked by #140. The Blocker section above already says why: the harness has nothing reproducible to run against until instance B's definition is in-tree. This flips to ready when #140 lands.

cast has the gate and the record format. It has no instrument, and `drills/README.md` states the position deliberately: > cast has **no drill harness script of its own**. Its legs run by hand against the documented procedure … A harness would make the run reproducible; it would not make it recorded. Those are separate problems, and this directory is the second one. Agreed on the framing, and the ordering was right — the gate is why releases stopped shipping unproven. This issue is the first problem. ## What "by hand" currently means CONTRIBUTING requires nine legs before a release PR can be handed over: > team, apply, an idempotent diff, smoke, inventory, emit-draft, fleet, destroy, and the read-only guard against **two live Coolify instances**. None of it is scripted. `scripts/` contains one file, `restore-db.sh`, which is an operator tool and not a drill leg. `drills/` holds only `README.md` — *"cast has recorded no drill runs."* So the release ceremony asks an operator to stand up two Coolify instances, drive nine legs in order against them, remember what each one asserts, and write down numbers — from memory, months apart, with no previous record to imitate. The first person to do it will be inventing the procedure, and the second will be inventing a different one. cast's test suite makes the gap sharper rather than softer. There are 40 `.test.ts` files, 167 `describe` blocks, `wire.test.ts` alone is 64KB — and **none of it touches a real Coolify**. `test/live-lookup.test.ts` builds a fake that answers `GET /projects`. That is the correct design for a unit suite, and it is exactly why the drill matters: everything that is mocked is a claim about Coolify's behaviour that the suite cannot check. box's `drill/README.md` makes this argument better than I can, and it transfers directly: > the interesting failures are not in the bash — they are in what Incus actually does, which is exactly what unit tests would stub out and get wrong. The drill runs the real thing. Substitute Coolify for Incus and it is cast's situation precisely, with 40 files of stubs standing in for one live instance. ## The legs that only a live run can prove Two invariants in cast's README have no possible unit test, and they are the ones a promotion drill exists to assert: - **"One-way, idempotent, never deletes."** The idempotence half is the `diff` leg — a second `apply` must yield **zero** changes, asserted mechanically. The never-deletes half is the read-only guard leg. - **The reserved-env-name rule** (`src/reserved.ts`) and `smoke`'s probe key. `src/smoke.ts` is described in-tree as *"the one thing in cast that mutates a live resource purely to learn something"* — it writes `INFRA_SMOKE_KEEP`. The drill has to assert that key is cleaned up and never collides with a reserved name. A mocked Coolify will agree with whatever the code believes about this; a real one will not. `destroy` has the same character: it must assert **absence** afterwards, not just that the delete call returned 200. box's `wipe.sh` gets this right and says why — *"assert the ABSENCE, don't trust the removals' exit codes"* — and it is the single most portable idea in box's harness. ## What to build `drill/drill.sh` (or a TS equivalent — see below), on box's shape: **The reporting verbs** — `drill/drill.sh:52-58`, five one-liners the whole 997-line harness rests on. `pass`/`fail` counters, a `findings` array, `phase` banners, and `ok`/`no`/`note` that always return 0 so the body can be a flat sequence of `probe && ok … || no …`. Exit 0 iff `fail` is 0. **Continue on failure.** box's header is explicit that this is not a style choice: *"NOT -e: a failing check is data, not a crash."* Nine legs against live infrastructure, aborting on the first problem, means one finding per afternoon of setup. **Assert the refs that are actually under test**, the way box re-reads `INSTALLED_FROM` and dies on a mismatch. A drill that believes it exercised the candidate and got `main` has proven nothing about the combination. **A record emitter** in box#152's shape, so the three repos' records reconcile on the shared run ID instead of three hand-invented ones. **Language.** box's harness is bash because box is bash. cast is TypeScript with vitest and a `globalSetup`, and it may well be that cast's drill should be a vitest suite gated behind a `CAST_DRILL_LIVE=1` env var, reusing the existing helpers and getting the leg structure for free. I lean that way but do not have a strong view; worth deciding early since it determines everything else. What must not change either way is the *contract* — nine legs, continue-on-failure, counted, record emitted, loud skips. **Loud skips.** A leg that cannot run because an instance is unreachable must not be silently absent from the count. Same defect class as heavy-duty/box#153. ## Blocker The parameterised compose file that stood up instance B for the July drill **is not in this repo**. CONTRIBUTING references it — > two Coolify instances can be stood up by hand, as the July drill did for instance B via a parameterised compose file — and cast's legs are described as "the least coupled of the three" specifically because of it. But an operator reading CONTRIBUTING today cannot follow that sentence to anything. Filed separately as heavy-duty/cast#140, and it is the natural first commit: the harness has nothing to run against until instance B is reproducible. ## Scope One script (or suite), nine legs, the verbs, ref assertion, record emitter. Not box's whole `drill/` directory — `doctor.sh` and `wipe.sh` address a persistent host stack that cast does not create, and `multiuser.sh` has no analogue. Not a cross-repo drill framework. heavy-duty/rig#105 is the same shape of issue for rig; the three should converge by copying box's *output format*, not by sharing code — they assert three different things (box: isolation, rig: convergence, cast: promotion) and the per-repo independence is load-bearing, for the reasons `drill-recorded.sh` already sets out at length about cross-repo lookups degrading to "pass". Related: heavy-duty/box#152 (record emitter, defines the shape), heavy-duty/box#153 (skipped legs must not read as passes). --- ## Triage — decisions (2026-08-17) - **Language: settled.** The drill is a **vitest suite gated behind `CAST_DRILL_LIVE=1`**, per the lean stated above — cast is TypeScript, the leg structure and live-client helpers already exist in the suite's shape, and a second toolchain in-tree is a cost with no offsetting assertion. The contract is unchanged and is the review spec: nine legs, continue-on-failure, pass/fail counted, candidate refs asserted, record emitted in box#152's shape, loud skips. - **Dependency recorded.** Blocked by #140. The Blocker section above already says why: the harness has nothing reproducible to run against until instance B's definition is in-tree. This flips to `ready` when #140 lands.

claude-bot-andresmgsl referenced this issue

2026-08-17 22:44:26 +00:00

Instance B's compose file is not in the repo — the drill's only reproducible half is the pinned refs #140

claude-bot-andresmgsl referenced this issue

2026-08-17 22:44:27 +00:00

The next release must carry a real drill record, not a waiver #142

claude-bot-andresmgsl added the

release

blocked

labels 2026-08-17 22:44:30 +00:00

claude-bot-andresmgsl referenced this issue

2026-08-17 22:44:32 +00:00

Instance B's compose file is not in the repo — the drill's only reproducible half is the pinned refs #140

claude-bot-andresmgsl commented

2026-08-17 22:44:32 +00:00

Member

Triage: the open language question is settled (vitest suite gated behind CAST_DRILL_LIVE=1) and the dependency is recorded — blocked by #140. Details appended to the body; flips to ready when #140 lands.

Triage: the open language question is settled (vitest suite gated behind `CAST_DRILL_LIVE=1`) and the dependency is recorded — blocked by #140. Details appended to the body; flips to `ready` when #140 lands.

claude-bot-andresmgsl referenced this issue

2026-08-17 22:44:32 +00:00

The next release must carry a real drill record, not a waiver #142

claude-bot-andresmgsl referenced this issue from heavy-duty/box

2026-08-21 15:30:08 +00:00

drill/README.md documents four phases; the script prints six, in a different order, and --help repeats the stale text #154