box/drill/RUNS.md

Drill run log

What the drill has actually found, what has broken in the drill itself, and how to diagnose the next stall without starting from zero. Append a section per run; keep the traps table current — it is the part that saves time.

The audit this fed, #15, is complete (run 10, 48/49) — and its headline finding is fixed and re-measured: sibling isolation shipped in #37/#38, and runs 11–13 hold at zero failures, run 13 from a bare host (full teardown → rebuild → drill, 47/47). #16 closed on that evidence.

The audit's answer

Probe	Answer
A1/A5 egress + gateway DNS	PASS
A2 box → host	dropped
A2 box → RFC1918	dropped
A3 sibling isolation	run 10: 🔴 FAIL — tcp REFUSED, i.e. the packet ARRIVED. #12's central claim was wrong; #16 became a fix, not a formalization. Fixed (security.port_isolation + nft bridge drop, #37/#38); runs 11–13: BLOCKED — tcp dropped + no icmp reply
A4 DNS enumeration	run 10: LEAKS. Fixed (dns.mode=none shipped); runs 11–13: blocked — and the resolver is pinned so tailnet/split-DNS names don't resolve either (#33/#45)
A6 IPv6 off	none ✓
A7 inbound host → box	dropped
B1 @internal on a bridge ACL	REJECTED — Unsupported nftables subject ⇒ #16 derives the subnet
B2 incus copy preserves user.*	YES ⇒ #17's metadata design holds
B3 dns.mode=none	VIABLE — closes the leak, egress survives, no outage window
B4 config get unset key	empty + exit 0 ⇒ #17 must use ${var:-}
B5 L2 filtering	🔴 ipv4_filtering BREAKS the box — design veto (measured on a healthy baseline)

The headline, then: the tool's contract — "a box reaches the public internet and nothing else" — was false: a box also reached every other box on the host. The headline, now: the contract holds, and it is measured — 47/47 from a bare host (run 13), every layer probed from inside the boxes.

Findings in claudebox (not in the drill)

Finding	Status
setup-host.sh called nft but a stock Debian 13 cloud image ships neither nftables nor UFW — host setup died on a fresh cloud host	fixed (setup-host installs it)
claudebox exec box -- claude … — the help's own example — failed: the binary is in ~/.local/bin, but cloud-init exported PATH only in .bashrc/.zshrc, which the non-interactive shell behind exec never reads (login shell is zsh, so even sudo -i misses both)	fixed (symlink into /usr/local/bin)
Cold mint takes ~95s, not the ~10 min the docs claim — consistently. Either the host is fast, or cloud-init status --wait returns before runcmd finishes (which would hand over boxes whose installs are still running)	open — worth its own issue if run 6 shows cloud-init mid-flight

Traps this script has already fallen into

Read this before adding a probe. Every one of these cost a run.

1. set -o pipefail breaks refusal checks. Half the drill is claudebox <refusal> 2>&1 | grep -q 'text'. The refusal exits 1/2 by design, and grep -q SIGPIPEs the left side when it matches early. Under pipefail both become false FAILs. The pipeline's verdict must be grep's alone — hence set -u and no pipefail.

2. $( ) waits for stdout to CLOSE, not for the command to exit. A grandchild inheriting an incus exec session's stdout holds the substitution open forever, and timeout does not save you: it kills the wrapper, not the process holding the pipe. Use in_box/box_curl, which talk to incus exec directly, pin stdin to /dev/null, and land output in a file rather than a pipe.

3. Never start a background process inside a box. Same mechanism as (2), and it is why the drill now runs no listener anywhere. It does not need one: curl exit 7 (refused) means the packet arrived, 28 (timeout) means it was dropped. A closed port answers the question.

4. The box's address is hard to read, and every way of getting it wrong was tried. (a) incus list name filters are not regexes — incus list "^peer$" silently matches nothing. (b) Its CSV quotes a multi-address box across lines. (c) The interface is not eth0. The profile names the device eth0, but inside a VM guest predictable naming renames it enp5s0 — so ip addr show dev eth0 finds nothing either. That is the real reason A3 went unprobed for six runs, through two "fixes" of mine that never questioned the interface name. Read it from inside the box and select by subnet (10.87.x), not by interface name: docker0 (172.17.x) is the decoy, and the NIC's name is the guest's business. Lesson: when the same probe fails three different ways, stop patching the probe and go look at the thing itself.

5. incus delete -f a b c aborts at the first MISSING name. One interrupted run then poisons the next: stale boxes survive cleanup and cascade into half a dozen unrelated FAILs. Delete one name at a time.

6. apt-get -qq … >/dev/null hides both a sudo prompt and the apt lock. apt-daily/unattended-upgrades hold the lock on a cloud image and apt waits in complete silence. Pre-authorize sudo, set DPkg::Lock::Timeout, and narrate.

7. claudebox exec is sudo -u claude -i — a login zsh with oh-my-zsh. Fine for a human, needless machinery for a probe, and one more thing that can hold an fd. Probes use incus exec directly.

8. Clean before you set up, not after. setup-host.sh reconfigures the network's ACLs, and a previous run's boxes are still attached to that network — incus network set then has to push the change onto every live NIC. An aborted run also leaves the D-phase mutations (dns.mode=none, NIC filtering) in place, so setup converges against a moving target. Delete the boxes and revert the mutations first.

9. Never render a verdict on a broken baseline. Run 7's box had no network (a clone/source IP collision), and phase D dutifully reported "L2 filtering BREAKS the box — design veto". It did not; the box was already broken. A measurement taken on a broken instrument is not evidence, and #16 would have been redesigned around a fiction. Phase D is now gated on baseline egress passing, and refuses to judge otherwise. This is the same failure as the B3 flip, in a different costume: check that the thing you are measuring with still works before you trust what it tells you.

10. The drill mutates the host, and those mutations outlive an aborted run. Phase D sets dns.mode=none and NIC filtering. If the run dies before reverting them, every box minted afterwards has no DNS — cloud-init fails with Temporary failure resolving deb.debian.org — and the next run reports that breakage as a finding. This is the worst failure mode in the whole list: a poisoned host does not fail honestly, it produces confident wrong answers. Hence the trap-armed revert, the verified (not /dev/null-ed) unset, the refusal to start on a dirty host, and doctor.sh.

11. A network Incus calls Created may have nothing serving it. After an unclean daemon death (a wedge, a SIGKILL, an OOM), Incus can come back without respawning a network's dnsmasq. The bridge is up, incus network show is perfect, status: Created — and no DHCP server exists, so every box minted afterwards gets no lease, no gateway, no DNS, and dies deep in cloud-init blaming Debian's mirrors. Incus's own status does not cover this; the process table does. doctor.sh now checks it, because two cold mints and an hour went into learning it the other way.

12. A curl exit code cannot tell you whether the packet arrived. Exit 7 is "failed to connect", and it means both Connection refused (a RST came back — reachable) and Could not connect / No route to host (nothing came back — isolated). Opposite conclusions, one number. The drill mapped 7 → "it arrived" and reported a working boundary as a broken one for two full runs after the fix had landed, while the kernel had isolated on on the bridge ports the whole time. Read the message. A refusal is instant; an unreachable host burns the timeout. This is the same disease as every other trap here — trusting a proxy for the fact instead of the fact.

13. A TTY on stdin makes incus exec interactive — and then timeout cannot kill it. The doctor's probes ran bare timeout N incus exec … with the operator's terminal as stdin: exec attached it in raw mode, the 20s TERM never took (no -k escalation), and Ctrl-C was forwarded into the box as a keystroke — a 15-minute hang the operator could only end by killing the shell. The drill had already learned this in #22 (exec_in pins stdin, escalates with -k 5); the doctor's probe section was added later and never inherited the cure. Every incus exec in a script gets </dev/null and timeout -k — no exceptions, including the ones that "obviously exit immediately" (#42).

14. The gateway does not answer ping — by design, so pinging it proves nothing. claudebox-firewall.sh drops everything box→host except DNS/DHCP; ICMP to 10.87.0.1 dies in that trailing drop on every healthy host. The doctor used exactly that ping as its routing probe, and declared a host "NOT fit to drill" while the very next line resolved deb.debian.org through the gateway it had just called unreachable. A probe whose failure is the designed state measures the firewall doing its job and calls it a fault. Routing is probed the way the contract states it: the public internet, by address (#43).

15. Do not demand an empty host — assert your boxes, not no boxes. Run 11's only two FAILs were pre-existing operator boxes (t1, t2): the empty-host message check can't pass on a shared host, and the teardown check demanded global emptiness right after an rm loop that deliberately removes only the drill's own names. A check that contradicts the multi-tenant discipline of the code above it teaches operators to "clean up" boxes the tool refuses to touch (#44).

Diagnosing a stall

Start here: bash drill/doctor.sh — it answers "what state is this host actually in?" (network, profile, ACL, leftover boxes, and whether a box can still resolve DNS), and --fix reverts what the drill left behind.

If the drill goes quiet mid-run, open a second terminal:

# what is actually running / blocked?
ps -eo pid,etimes,stat,args | grep -Ev grep | grep -E 'apt|dpkg|incus|setup-host|sudo|curl'

# apt lock held by a background upgrade? (the classic silent stall)
sudo fuser -v /var/lib/dpkg/lock-frontend
systemctl status unattended-upgrades apt-daily.service

# incus itself wedged?
incus list
journalctl -u incus --no-pager -n 30

Running one probe by hand

Nothing in the audit requires the whole drill. To answer A3 on a host that already has two boxes up (archive and peer):

PEER_IP=$(timeout 20 incus exec peer -- ip -4 -o addr show dev eth0 \
          | awk '{split($4,a,"/"); print a[1]}')
timeout 30 incus exec archive -- curl -sS -m 5 -o /dev/null "http://$PEER_IP:8088"
echo "curl exit: $?"    # 28 = dropped (isolated) · 7 = refused (it ARRIVED) · 0 = connected
timeout 30 incus exec archive -- ping -c1 -W2 "$PEER_IP"
echo "ping exit: $?"    # 0 = ICMP replies — isolation is partial at best

No listener is needed, and none should be started: see trap 3.

Run history

Run	Result	What it cost
MU-5	53/1 → 54/0 (containers)	the #75 review round, measured in: criterion (m) — a raw --network boxnet attach keeps egress, the ACL, both sibling-drop directions and dns.mode=none, losing only per-NIC port_isolation (the scoped guarantee, now measured, not prose) — and criterion (n), both grant-failure injections (fresh user backed out with the removal VERIFIED; pre-existing member warned loudly, never stripped; re-runs converge). The one instrument bug: (m) first minted the non-cloud image — no DHCP client, no lease, and a dead NIC passes every negative probe vacuously. The probe now requires the lease before it believes any of its own answers
MU-4	41/1 on CI (ubuntu-latest, incus 6.0.0) → 42/0	the CI job earned its keep on its second-ever run: the attach-refusal probe matched 6.0.4's error wording and 6.0.0 words it differently (and refuses at a different stage) — a version-drift false FAIL the design host could never show. The probe now asserts the OUTCOME (nothing running on the private bridge) and prints the refusal as evidence. Also this round, from adversarial review: revoke survives the live-session case (groups are read at login; --purge now terminates sessions first, else a stale-group process could recreate the project UNHARDENED after the purge), grant backs out its own group-add on failure, and the session-dies-with-the-tier check is criterion 42
MU-3	41/0 (containers) + 41/0 (VMs) — the tier holds	multiuser.sh (#74) from zero on Debian 13 / Incus 6.0.4: grant, confinement, lifecycle, the in-box contract with the cross-user sibling drop, closed escape hatches, re-sync survival, scoped revoke. The VM pass needed right-sized boxes (1GiB): two 2GiB nested VMs on a 7.7GiB host measured swap, not isolation
MU-2	40/41 (VMs)	the egress probe cried broken over a TLS handshake that outlived its 8s budget under nested-KVM memory pressure — every isolation probe (which PASSES on silence) was immune. Probe budgets raised, reachable-probes retry once, and the mint keeps its log for the post-mortem
MU-1	39/41 (containers)	both FAILs were the instrument: a box new rc swallowed with its narration (the drill's oldest sin, relearned), and the enumeration probe resolving the box's OWN name — /etc/hosts answered, not the gateway. Probe a SIBLING's name or you measure cloud-init's hostname hygiene. Found for real along the way: box restore dispatched incus restore, which does not exist in Incus 6 — broken for every tier since the rename
1	hung at C7; ~9 false FAILs	traps 1, 2, 4 — pipefail, the exec-pty hang, the DHCP race
2	42/49	trap 5 — an interrupted run 1 left boxes behind, cascading 5 FAILs. Found the claude-on-PATH bug. Phase D delivered B1 (@internal rejected) and a B3 reading of broken
3	48/49	trap 4 — eth0_ip never matched, so A3 again unprobed. B3 now read intact, contradicting run 2
4	hung at C4	trap 2 again, this time via claudebox exec in a command substitution
5	stalled in host setup	trap 6 — silence through apt/sudo
6	stalled in setup-host.sh	trap 8 — cleanup ran after setup. Recovering the host exposed two real claudebox bugs: setup-host deadlocks the incus daemon when re-run with boxes up (#26), and clones inherit their source's machine-id → same DHCP lease → two boxes, one IP (#27)
13	47/0 — the contract, from zero	full teardown-host.sh → rebuild → drill on a bare host: setup-host built the entire stack from scratch — including the pinned resolver (#45) on a Tailscale host, cold mint resolving through it — and the empty-host check finally got to run. Every layer measured, nothing inherited
12	46/0	first zero-failure run. #44's shared-host fixes verified live (tenant note where the false FAIL was; teardown asserts its own names)
11	45/2	A3 re-answered with the fixed probe (#41): BLOCKED — tcp dropped + no icmp reply. #16's hardening measured end-to-end; both FAILs were trap 15 (the drill demanding an empty host, #44). Before the run, the doctor hung unkillably twice (trap 13 → #42) and cried wolf a fifth time (trap 14 → #43)
10	48/49 — the audit is complete	🔴 A3 answered: sibling isolation DOES NOT HOLD (tcp refused = the packet arrived). B5's ipv4_filtering veto confirmed on a healthy baseline. B3 cleared. Every #15 probe answered
9	aborted: cold mint failed, twice	not the drill and not the host's mutations (doctor was green): claudenet had no dnsmasq — it never respawned after the SIGKILL in run 6's recovery. Boxes got no DHCP lease at all. Trap 11
8	aborted: cold mint failed	cloud-init status: error — the box had no DNS at all. Run 7's phase-D dns.mode=none survived the run and poisoned the host. Trap 10, and the reason doctor.sh exists
7	41/49	the clone-identity fix could not reboot (systemd needs a valid machine-id to shut down cleanly), so it never took effect → the IP collision persisted → the box lost networking → phase D reported a false design veto against #16. Trap 9. Also found: dir storage makes every clone a full disk copy (#29)

The instrument was less reliable than the thing it measured. Of ten runs, four died on drill plumbing and three on bugs in claudebox or the host. It still paid for itself many times over — every one of those seven failures was a real defect, and the audit's headline finding overturned the premise it was written to confirm. But the lesson stands: if a probe can be answered by hand, answer it by hand rather than paying for another full run.

Runs 11–13 close the arc: every remaining failure was the instrument's own false alarm (traps 13–15, all in the doctor or the drill, none in the tool), and the last run measured the whole contract from a bare host with zero. The instrument is finally quieter than the thing it measures — which is the only state in which its green means anything.

The B3 flip — a lesson worth keeping

Run 2 said dns.mode=none broke egress; run 3 said it was intact. Both probed ~2s after setting the key, and setting dns.mode restarts the network's dnsmasq — so run 2 caught the restart window and run 3 missed it. A design veto was posted to #16 on the strength of run 2, then retracted.

A verdict drawn from one observation of a system with restart semantics is not a verdict. The probe now distinguishes transient (recovers within 30s) from broken (does not), and any test #16 ships must tolerate that window rather than race it.