box/drill/RUNS.md

# Drill run log

What the drill has actually found, what has broken *in the drill itself*, and
how to diagnose the next stall without starting from zero. Append a section per
run; keep the traps table current — it is the part that saves time.

The audit this feeds is [#15](https://github.com/heavy-duty/claudebox/issues/15).

## Where the audit stands

| Probe | Answer | Run |
| --- | --- | --- |
| A1/A5 egress + gateway DNS | **PASS** | 2, 3, 4 |
| A2 box → host | **dropped** | 2, 3, 4 |
| A2 box → RFC1918 | **dropped** | 2, 3, 4 |
| **A3 sibling isolation** | ⏳ **still unanswered** — blocked by a drill bug in every run so far | — |
| A4 DNS enumeration | **LEAKS** — a box resolves its sibling by name (as #12 predicted) | 2, 3, 4 |
| A6 IPv6 off | `none` ✓ | 2, 3, 4 |
| A7 inbound host → box | **dropped** | 3, 4 |
| B1 `@internal` on a bridge ACL | **REJECTED** — `Unsupported nftables subject` ⇒ #16 derives the subnet | 2, 3 |
| B2 `incus copy` preserves `user.*` | **YES** ⇒ #17's metadata design holds | 2, 3, 4 |
| B3 `dns.mode=none` | closes the leak; **egress verdict FLIPPED between runs** (see below) | 2, 3 |
| B4 `config get` on unset key | **empty + exit 0** ⇒ #17 must use `${var:-}`, never `\|\|` | 2, 3, 4 |
| B5 L2 filtering | box networking **intact**; in-box docker unverified | 2, 3 |

**A3 is the whole point and it has never fired.** It is the one claim #12 could
not verify from a code reading. Until it is answered, #16 does not know whether
it is writing a *formalization* or a *fix*.

## Findings in claudebox (not in the drill)

| Finding | Status |
| --- | --- |
| `setup-host.sh` called `nft` but a stock Debian 13 cloud image ships neither nftables nor UFW — host setup died on a fresh cloud host | **fixed** (setup-host installs it) |
| `claudebox exec box -- claude …` — the help's own example — failed: the binary is in `~/.local/bin`, but cloud-init exported PATH only in `.bashrc`/`.zshrc`, which the non-interactive shell behind `exec` never reads (login shell is zsh, so even `sudo -i` misses both) | **fixed** (symlink into `/usr/local/bin`) |
| Cold mint takes **~95s**, not the ~10 min the docs claim — consistently. Either the host is fast, or `cloud-init status --wait` returns before `runcmd` finishes (which would hand over boxes whose installs are still running) | **open** — worth its own issue if run 6 shows cloud-init mid-flight |

## Traps this script has already fallen into

Read this before adding a probe. Every one of these cost a run.

1. **`set -o pipefail` breaks refusal checks.** Half the drill is
   `claudebox <refusal> 2>&1 | grep -q 'text'`. The refusal exits 1/2 *by
   design*, and `grep -q` SIGPIPEs the left side when it matches early. Under
   pipefail both become false FAILs. The pipeline's verdict must be grep's
   alone — hence `set -u` and no pipefail.
2. **`$( )` waits for stdout to CLOSE, not for the command to exit.** A
   grandchild inheriting an `incus exec` session's stdout holds the
   substitution open forever, and `timeout` does *not* save you: it kills the
   wrapper, not the process holding the pipe. Use `in_box`/`box_curl`, which
   talk to `incus exec` directly, pin stdin to `/dev/null`, and land output in
   a file rather than a pipe.
3. **Never start a background process inside a box.** Same mechanism as (2),
   and it is why the drill now runs **no listener anywhere**. It does not need
   one: `curl` exit `7` (refused) means the packet *arrived*, `28` (timeout)
   means it was *dropped*. A closed port answers the question.
4. **The box's address is hard to read, and every way of getting it wrong was
   tried.** (a) `incus list` name filters are **not regexes** — `incus list
   "^peer$"` silently matches nothing. (b) Its CSV quotes a multi-address box
   across lines. (c) **The interface is not `eth0`.** The *profile* names the
   device `eth0`, but inside a **VM guest** predictable naming renames it
   **`enp5s0`** — so `ip addr show dev eth0` finds nothing either. That is the
   real reason A3 went unprobed for six runs, through two "fixes" of mine that
   never questioned the interface name. Read it from inside the box and select
   by **subnet** (`10.87.x`), not by interface name: docker0 (`172.17.x`) is
   the decoy, and the NIC's name is the guest's business.
   *Lesson: when the same probe fails three different ways, stop patching the
   probe and go look at the thing itself.*
5. **`incus delete -f a b c` aborts at the first MISSING name.** One interrupted
   run then poisons the next: stale boxes survive cleanup and cascade into
   half a dozen unrelated FAILs. Delete one name at a time.
6. **`apt-get -qq … >/dev/null` hides both a sudo prompt and the apt lock.**
   `apt-daily`/`unattended-upgrades` hold the lock on a cloud image and apt
   waits in complete silence. Pre-authorize sudo, set `DPkg::Lock::Timeout`,
   and narrate.
7. **`claudebox exec` is `sudo -u claude -i`** — a *login zsh* with oh-my-zsh.
   Fine for a human, needless machinery for a probe, and one more thing that
   can hold an fd. Probes use `incus exec` directly.
8. **Clean before you set up, not after.** `setup-host.sh` reconfigures the
   network's ACLs, and a previous run's boxes are still *attached* to that
   network — `incus network set` then has to push the change onto every live
   NIC. An aborted run also leaves the D-phase mutations (`dns.mode=none`, NIC
   filtering) in place, so setup converges against a moving target. Delete the
   boxes and revert the mutations **first**.

## Diagnosing a stall

The drill narrates every long step. If it goes quiet, open a second terminal:

```sh
# what is actually running / blocked?
ps -eo pid,etimes,stat,args | grep -Ev grep | grep -E 'apt|dpkg|incus|setup-host|sudo|curl'

# apt lock held by a background upgrade? (the classic silent stall)
sudo fuser -v /var/lib/dpkg/lock-frontend
systemctl status unattended-upgrades apt-daily.service

# incus itself wedged?
incus list
journalctl -u incus --no-pager -n 30
```

## Running one probe by hand

Nothing in the audit requires the whole drill. To answer **A3** on a host that
already has two boxes up (`archive` and `peer`):

```sh
PEER_IP=$(timeout 20 incus exec peer -- ip -4 -o addr show dev eth0 \
          | awk '{split($4,a,"/"); print a[1]}')
timeout 30 incus exec archive -- curl -sS -m 5 -o /dev/null "http://$PEER_IP:8088"
echo "curl exit: $?"    # 28 = dropped (isolated) · 7 = refused (it ARRIVED) · 0 = connected
timeout 30 incus exec archive -- ping -c1 -W2 "$PEER_IP"
echo "ping exit: $?"    # 0 = ICMP replies — isolation is partial at best
```

No listener is needed, and none should be started: see trap 3.

## Run history

| Run | Result | What it cost |
| --- | --- | --- |
| 1 | hung at C7; ~9 false FAILs | traps 1, 2, 4 — pipefail, the exec-pty hang, the DHCP race |
| 2 | 42/49 | trap 5 — an interrupted run 1 left boxes behind, cascading 5 FAILs. Found the `claude`-on-PATH bug. Phase D delivered B1 (`@internal` rejected) and a B3 reading of *broken* |
| 3 | 48/49 | trap 4 — `eth0_ip` never matched, so A3 again unprobed. B3 now read *intact*, contradicting run 2 |
| 4 | hung at C4 | trap 2 again, this time via `claudebox exec` in a command substitution |
| 5 | stalled in host setup | trap 6 — silence through apt/sudo |
| 6 | stalled in `setup-host.sh` | trap 8 — cleanup ran *after* setup, so setup reconfigured claudenet's ACLs while run 4's boxes were still attached to it |

**The instrument has been less reliable than the thing it measures.** Four of
five runs died on drill plumbing, not on claudebox. That is worth stating
plainly: if a probe can be answered by hand (see above), answer it by hand and
move on rather than paying for another full run.

### The B3 flip — a lesson worth keeping

Run 2 said `dns.mode=none` **broke** egress; run 3 said it was **intact**. Both
probed ~2s after setting the key, and setting `dns.mode` restarts the network's
dnsmasq — so run 2 caught the restart window and run 3 missed it. A design veto
was posted to #16 on the strength of run 2, then retracted.

**A verdict drawn from one observation of a system with restart semantics is not
a verdict.** The probe now distinguishes *transient* (recovers within 30s) from
*broken* (does not), and any test #16 ships must tolerate that window rather
than race it.