test: add an end-to-end drill against a real Incus #18

Merged

dan-claude-bot merged 3 commits from drill/rehearsal into main

2026-07-13 21:34:44 +00:00

dan-claude-bot commented

2026-07-13 21:18:07 +00:00

(Migrated from github.com)

The repo has no tests and no CI. Everything in #9, #10 and #13 was verified against an incus stub — the assumptions those PRs rest on had never been checked against the real thing. And #15 needs a live audit of the isolation boundary before #16/#17 can move.

This PR is both, as one run-and-forget script on a throwaway host:

git clone https://github.com/heavy-duty/claudebox && cd claudebox
bash drill/drill.sh --yes

~20 minutes, most of it the cold box. Exit 0 = all green. The run ends with an audit-answers block to paste into #15.

What it checks

A. Incus semantics — the assumptions the CLI is built on, probed directly: the user.claudebox tag read (on the path of every command since #13), the list filter selecting ours and excluding an untagged instance, CSV field shapes, the RUNNING state string, rename refusing a running instance, snapshot-list labels — plus two #15 mechanism checks: an unset config key reads empty with exit 0 (B4, the shape #17's user lookup depends on) and incus copy preserves user.* keys (B2, the whole metadata design).

B. The surface — mint, list, info, snapshot, clone-from-a-snapshot-of-a-renamed-box, rename preconditions, the escape hatch and its isolation warning, the rm guard, the CLI contract. The boundary gets its own test: an instance claudebox did not mint must survive down, rm --force, and the escape hatch untouched.

C. Isolation baseline (#15 section A) — from inside a real box: public egress passes; box → host gateway listener blocked; RFC1918 dropped; box → sibling box blocked — the drill clones a peer and runs a listener on it, so refused (packet arrived) cannot masquerade as dropped; DNS enumeration of the sibling recorded (#12 predicts it leaks — audit data either way); IPv6 off; host → box inbound dropped.

D. Hardening rehearsal (#15 section B) — the host is disposable, so the drill applies exactly what #16 proposes and watches what breaks: dns.mode=none (must kill sibling resolution, must not kill egress), security.mac_filtering + security.ipv4_filtering (box networking and in-box Docker must survive), and @internal as a bridge-ACL drop destination (accepted + egress survives ⇒ #16's sibling drop is renumber-proof by construction; otherwise #16 derives the subnet). A FAIL here is a design veto for #16, caught before its code is written.

It already found one thing — now fixed here

setup-host.sh drives nft, but a stock Debian 13 cloud image ships no nftables and no UFW — host setup died on its first nft call on exactly the kind of fresh cloud host this tool targets. This PR fixes setup-host.sh to install nftables itself when neither exists; the drill keeps a tripwire NOTE should that regress.

Honest limits

It rearranges the host it runs on — Incus, a systemd unit, a network, an ACL, a profile, firewall rules, and (phase D) deliberate network/profile mutations. That's inherent; hence the consent prompt and the loud README.
No /dev/kvm → no verdict on the VM boundary. The drill says so loudly rather than passing quietly on container mode.
claude /login isn't drilled — interactive by design; the drill confirms Claude Code is installed and runnable.
The drill itself has not been run yet — shellcheck-clean and syntax-clean, but the first real run will find friction. That's what the throwaway server is for.

🤖 Generated with Claude Code

The repo has no tests and no CI. Everything in #9, #10 and #13 was verified against an `incus` **stub** — the assumptions those PRs rest on had never been checked against the real thing. And #15 needs a live audit of the isolation boundary before #16/#17 can move. This PR is both, as one run-and-forget script on a throwaway host: ```sh git clone https://github.com/heavy-duty/claudebox && cd claudebox bash drill/drill.sh --yes ``` ~20 minutes, most of it the cold box. Exit 0 = all green. The run ends with an **audit-answers block to paste into #15**. ## What it checks **A. Incus semantics** — the assumptions the CLI is built on, probed directly: the `user.claudebox` tag read (on the path of every command since #13), the list filter selecting ours and excluding an untagged instance, CSV field shapes, the `RUNNING` state string, rename refusing a running instance, snapshot-list labels — plus two #15 mechanism checks: an **unset config key reads empty with exit 0** (B4, the shape #17's user lookup depends on) and **`incus copy` preserves `user.*` keys** (B2, the whole metadata design). **B. The surface** — mint, list, info, snapshot, clone-from-a-snapshot-of-a-renamed-box, rename preconditions, the escape hatch and its isolation warning, the `rm` guard, the CLI contract. The boundary gets its own test: an instance claudebox did *not* mint must survive `down`, `rm --force`, and the escape hatch untouched. **C. Isolation baseline (#15 section A)** — from inside a real box: public egress passes; box → host gateway listener blocked; RFC1918 dropped; **box → sibling box blocked** — the drill clones a peer and runs a listener on it, so *refused* (packet arrived) cannot masquerade as *dropped*; **DNS enumeration of the sibling recorded** (#12 predicts it leaks — audit data either way); IPv6 off; host → box inbound dropped. **D. Hardening rehearsal (#15 section B)** — the host is disposable, so the drill applies exactly what #16 proposes and watches what breaks: `dns.mode=none` (must kill sibling resolution, must **not** kill egress), `security.mac_filtering` + `security.ipv4_filtering` (box networking and in-box Docker must survive), and `@internal` as a bridge-ACL drop destination (accepted + egress survives ⇒ #16's sibling drop is renumber-proof by construction; otherwise #16 derives the subnet). A FAIL here is a **design veto for #16, caught before its code is written**. ## It already found one thing — now fixed here `setup-host.sh` drives `nft`, but a stock Debian 13 cloud image ships no nftables and no UFW — host setup died on its first `nft` call on exactly the kind of fresh cloud host this tool targets. This PR fixes `setup-host.sh` to install nftables itself when neither exists; the drill keeps a tripwire NOTE should that regress. ## Honest limits - **It rearranges the host it runs on** — Incus, a systemd unit, a network, an ACL, a profile, firewall rules, and (phase D) deliberate network/profile mutations. That's inherent; hence the consent prompt and the loud README. - **No `/dev/kvm` → no verdict on the VM boundary.** The drill says so loudly rather than passing quietly on container mode. - **`claude /login` isn't drilled** — interactive by design; the drill confirms Claude Code is installed and runnable. - **The drill itself has not been run yet** — shellcheck-clean and syntax-clean, but the first real run will find friction. That's what the throwaway server is for. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dan-claude-bot commented

2026-07-13 21:31:42 +00:00

(Migrated from github.com)

Reviewed against main @ 0982a2d and against #15 (the audit this is now the vehicle for). Verdict: right instrument, was incomplete as an audit — fixed in the two commits above. What the review found:

The central probe was missing. The drill never had two boxes running at once, so sibling isolation — the one claim #12 could not verify from code and the whole reason #15 exists — was never tested. Phase C now clones a peer from the snapshot and probes it with a listener running, so the three outcomes are distinguishable: connected (isolation broken), refused (packet arrived — the egress drop isn't covering siblings), timeout (dropped, as designed).
No DNS-enumeration, IPv6, or host→box probes — added (#15 A4/A6/A7).
None of #15's four mechanism checks — added: B4 + B2 in phase A (unset-key semantics; incus copy preserving user.*), B3 + B5 + B1 as a new phase D that applies #16's exact proposed changes to the throwaway host and re-probes. A phase-D FAIL vetoes a piece of #16's design before it's written — cheaper than finding out in review.
Stale defaults: it installed claude-hdb/claudebox@refactor/command-table — a fork branch that stopped moving when #13 merged. Now heavy-duty/claudebox@main.
Version check pinned 0.3.0 — would fail the drill on every release. Now compares against the installed VERSION file.
sleep 10 after start — a VM agent can take longer; a false "box cannot reach the internet" would poison the audit. Now an agent poll (~2 min budget).
The nft gap is fixed at the source: setup-host.sh installs nftables when neither nft nor UFW exists (a stock Debian 13 cloud image ships neither — host setup died on its first nft call). The drill's manual workaround became a regression tripwire.

Also verified every grep target in the drill against the merged CLI (no such box, refusing, renamed {} to, isolation stack, did you mean, no boxes yet — all present), and both scripts are shellcheck-clean.

Run plan: provision the throwaway server → git clone … && bash drill/drill.sh --yes → paste the closing audit-answers block into #15. Green run + answers on #15 unblocks #16's design points; #17 queues behind it. Known open risk, stated in the body: the drill itself has never executed against a live Incus — first-run friction is expected and is the point.

Reviewed against `main` @ `0982a2d` and against #15 (the audit this is now the vehicle for). Verdict: **right instrument, was incomplete as an audit** — fixed in the two commits above. What the review found: 1. **The central probe was missing.** The drill never had two boxes running at once, so sibling isolation — the one claim #12 could not verify from code and the whole reason #15 exists — was never tested. Phase C now clones a `peer` from the snapshot and probes it with a listener running, so the three outcomes are distinguishable: *connected* (isolation broken), *refused* (packet **arrived** — the egress drop isn't covering siblings), *timeout* (dropped, as designed). 2. **No DNS-enumeration, IPv6, or host→box probes** — added (#15 A4/A6/A7). 3. **None of #15's four mechanism checks** — added: B4 + B2 in phase A (unset-key semantics; `incus copy` preserving `user.*`), B3 + B5 + B1 as a new phase D that applies #16's exact proposed changes to the throwaway host and re-probes. A phase-D FAIL vetoes a piece of #16's design before it's written — cheaper than finding out in review. 4. **Stale defaults**: it installed `claude-hdb/claudebox@refactor/command-table` — a fork branch that stopped moving when #13 merged. Now `heavy-duty/claudebox@main`. 5. **Version check pinned `0.3.0`** — would fail the drill on every release. Now compares against the installed `VERSION` file. 6. **`sleep 10` after `start`** — a VM agent can take longer; a false "box cannot reach the internet" would poison the audit. Now an agent poll (~2 min budget). 7. **The nft gap is fixed at the source**: `setup-host.sh` installs nftables when neither `nft` nor UFW exists (a stock Debian 13 cloud image ships neither — host setup died on its first `nft` call). The drill's manual workaround became a regression tripwire. Also verified every grep target in the drill against the merged CLI (`no such box`, `refusing`, `renamed {} to`, `isolation stack`, `did you mean`, `no boxes yet` — all present), and both scripts are shellcheck-clean. **Run plan:** provision the throwaway server → `git clone … && bash drill/drill.sh --yes` → paste the closing audit-answers block into #15. Green run + answers on #15 unblocks #16's design points; #17 queues behind it. Known open risk, stated in the body: the drill itself has never executed against a live Incus — first-run friction is expected and is the point.

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/box#18

No description provided.