From 40a9e05e51ed1f870d70b3a90db43ed34d9d6bd7 Mon Sep 17 00:00:00 2001 From: claude-hdb Date: Mon, 13 Jul 2026 23:05:34 +0000 Subject: [PATCH 1/2] fix(drill): stop going silent through host setup MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 5 looked stuck for minutes right after the "Host setup" header. The setup stretch is the longest and most failure-prone part of the run, and it was also the only part that printed nothing at all — two ways to block, both invisible: · 'sudo apt-get install -y -qq incus >/dev/null 2>&1' swallows a sudo password prompt, so a stalled sudo is indistinguishable from a hang. Sudo is now pre-authorized up front, out loud. · apt's lock is routinely held by apt-daily / unattended-upgrades on a cloud image, and apt waits for it in complete silence. It now gets DPkg::Lock::Timeout=300 plus an outer timeout, announces that it may be waiting on the lock, and on failure prints the two commands that identify the holder — instead of hanging forever. Also: skip the apt call entirely when incus is already installed (every re-run on the same host), and narrate each setup sub-step, so a stall is locatable rather than a blank terminal. Co-Authored-By: Claude Fable 5 --- drill/drill.sh | 34 ++++++++++++++++++++++++++++++---- 1 file changed, 30 insertions(+), 4 deletions(-) diff --git a/drill/drill.sh b/drill/drill.sh index 5a4554f..5197b46 100755 --- a/drill/drill.sh +++ b/drill/drill.sh @@ -154,20 +154,46 @@ EOF phase "Host setup (Incus, claudenet, ACL, profile, firewall)" # setup-host.sh installs nftables itself when neither nft nor UFW exists - # (fixed in this PR — a stock Debian 13 cloud image ships neither). This - # guard stays as a tripwire: if it fires, that fix regressed. + # (a stock Debian 13 cloud image ships neither). This guard is a tripwire: + # if it fires, that fix regressed. if ! command -v nft >/dev/null 2>&1 && ! command -v ufw >/dev/null 2>&1; then note "neither nft nor ufw present pre-setup — setup-host.sh must install nftables itself (it fixed this once; watch that it still does)" fi - sudo apt-get install -y -qq incus >/dev/null 2>&1 # so the group exists before we sg - ~/.local/share/claudebox/host/setup-host.sh || true # first run may only add the group + + # Sudo, up front and out loud. Later calls run unattended, and a password + # prompt swallowed by a '-qq' redirect looks exactly like a hang. + sudo -v || { echo "drill: need sudo (the host setup installs packages and firewall rules)"; exit 1; } + + # apt's lock is held by apt-daily / unattended-upgrades on a fresh cloud + # image, and 'apt-get -qq >/dev/null' waits for it in COMPLETE SILENCE — + # which is how run 5 looked stuck for minutes right after this header. + # Say what we are waiting for, and give up rather than hang forever. + if ! command -v incus >/dev/null 2>&1; then + inf "installing incus (waiting for the apt lock if a background upgrade holds it)…" + if ! sudo DEBIAN_FRONTEND=noninteractive timeout 600 \ + apt-get -o DPkg::Lock::Timeout=300 install -y incus; then + echo "drill: 'apt-get install incus' failed or timed out." >&2 + echo " a background apt job usually holds the lock. check with:" >&2 + echo " sudo fuser -v /var/lib/dpkg/lock-frontend" >&2 + echo " systemctl status unattended-upgrades apt-daily.service" >&2 + exit 1 + fi + else + inf "incus already installed — skipping apt" + fi + + inf "running setup-host.sh (first pass: may only add you to incus-admin)…" + ~/.local/share/claudebox/host/setup-host.sh || true # The group we were just added to isn't in this shell's credentials yet. + inf "re-entering inside the incus-admin group…" exec sg incus-admin -c "IN_GROUP=1 CLAUDEBOX_REPO='$REPO' CLAUDEBOX_REF='$REF' KEEP=$KEEP bash '$SELF' --in-group" fi export PATH="$HOME/.local/bin:$PATH" KEEP="${KEEP:-0}" +inf "running setup-host.sh (in-group pass: network, ACL, profile, firewall)…" ~/.local/share/claudebox/host/setup-host.sh || { echo "setup-host failed inside the group"; exit 1; } +inf "host setup complete" # Re-runnable: clear anything a previous drill left behind. One name at a # time — 'incus delete -f a b c' aborts at the first MISSING name, which is From a3e5d1226e261d48650cd616b8033d986cd77935 Mon Sep 17 00:00:00 2001 From: claude-hdb Date: Mon, 13 Jul 2026 23:06:38 +0000 Subject: [PATCH 2/2] docs(drill): keep the run log in the repo, not in PR descriptions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Five runs of hard-won knowledge — which probes are answered, which bugs the drill found in claudebox, and the seven traps the drill itself fell into — has been living in PR bodies and chat, where the next person debugging this cannot find it. RUNS.md carries: the audit scoreboard (A3 still unanswered, and why that matters), the claudebox findings, a traps section to read BEFORE adding a probe, stall diagnostics, how to run a single probe by hand instead of paying for a full run, and the B3 flip-flop as a standing lesson about verdicts drawn from one observation of a system with restart semantics. Co-Authored-By: Claude Fable 5 --- drill/README.md | 6 +++ drill/RUNS.md | 131 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) create mode 100644 drill/RUNS.md diff --git a/drill/README.md b/drill/README.md index 1cca200..701aedf 100644 --- a/drill/README.md +++ b/drill/README.md @@ -25,6 +25,12 @@ and profile mutations stay applied with them). Exit 0 means every check passed. Roughly 20 minutes, most of it the cold box. +**Iterating on the drill?** Read [RUNS.md](RUNS.md) first — it is the run log: +what the audit has answered so far, the bugs the drill has found in claudebox, +the traps this script has already fallen into (every one cost a run), how to +diagnose a stall, and how to run a single probe by hand instead of paying for a +whole run. + ## Why it exists The repo has no tests and no CI, and the CLI is a shell script that shells out to diff --git a/drill/RUNS.md b/drill/RUNS.md new file mode 100644 index 0000000..425b404 --- /dev/null +++ b/drill/RUNS.md @@ -0,0 +1,131 @@ +# Drill run log + +What the drill has actually found, what has broken *in the drill itself*, and +how to diagnose the next stall without starting from zero. Append a section per +run; keep the traps table current — it is the part that saves time. + +The audit this feeds is [#15](https://github.com/heavy-duty/claudebox/issues/15). + +## Where the audit stands + +| Probe | Answer | Run | +| --- | --- | --- | +| A1/A5 egress + gateway DNS | **PASS** | 2, 3, 4 | +| A2 box → host | **dropped** | 2, 3, 4 | +| A2 box → RFC1918 | **dropped** | 2, 3, 4 | +| **A3 sibling isolation** | ⏳ **still unanswered** — blocked by a drill bug in every run so far | — | +| A4 DNS enumeration | **LEAKS** — a box resolves its sibling by name (as #12 predicted) | 2, 3, 4 | +| A6 IPv6 off | `none` ✓ | 2, 3, 4 | +| A7 inbound host → box | **dropped** | 3, 4 | +| B1 `@internal` on a bridge ACL | **REJECTED** — `Unsupported nftables subject` ⇒ #16 derives the subnet | 2, 3 | +| B2 `incus copy` preserves `user.*` | **YES** ⇒ #17's metadata design holds | 2, 3, 4 | +| B3 `dns.mode=none` | closes the leak; **egress verdict FLIPPED between runs** (see below) | 2, 3 | +| B4 `config get` on unset key | **empty + exit 0** ⇒ #17 must use `${var:-}`, never `\|\|` | 2, 3, 4 | +| B5 L2 filtering | box networking **intact**; in-box docker unverified | 2, 3 | + +**A3 is the whole point and it has never fired.** It is the one claim #12 could +not verify from a code reading. Until it is answered, #16 does not know whether +it is writing a *formalization* or a *fix*. + +## Findings in claudebox (not in the drill) + +| Finding | Status | +| --- | --- | +| `setup-host.sh` called `nft` but a stock Debian 13 cloud image ships neither nftables nor UFW — host setup died on a fresh cloud host | **fixed** (setup-host installs it) | +| `claudebox exec box -- claude …` — the help's own example — failed: the binary is in `~/.local/bin`, but cloud-init exported PATH only in `.bashrc`/`.zshrc`, which the non-interactive shell behind `exec` never reads (login shell is zsh, so even `sudo -i` misses both) | **fixed** (symlink into `/usr/local/bin`) | +| Cold mint takes **~95s**, not the ~10 min the docs claim — consistently. Either the host is fast, or `cloud-init status --wait` returns before `runcmd` finishes (which would hand over boxes whose installs are still running) | **open** — worth its own issue if run 6 shows cloud-init mid-flight | + +## Traps this script has already fallen into + +Read this before adding a probe. Every one of these cost a run. + +1. **`set -o pipefail` breaks refusal checks.** Half the drill is + `claudebox 2>&1 | grep -q 'text'`. The refusal exits 1/2 *by + design*, and `grep -q` SIGPIPEs the left side when it matches early. Under + pipefail both become false FAILs. The pipeline's verdict must be grep's + alone — hence `set -u` and no pipefail. +2. **`$( )` waits for stdout to CLOSE, not for the command to exit.** A + grandchild inheriting an `incus exec` session's stdout holds the + substitution open forever, and `timeout` does *not* save you: it kills the + wrapper, not the process holding the pipe. Use `in_box`/`box_curl`, which + talk to `incus exec` directly, pin stdin to `/dev/null`, and land output in + a file rather than a pipe. +3. **Never start a background process inside a box.** Same mechanism as (2), + and it is why the drill now runs **no listener anywhere**. It does not need + one: `curl` exit `7` (refused) means the packet *arrived*, `28` (timeout) + means it was *dropped*. A closed port answers the question. +4. **`incus list` name filters are not regexes.** `incus list "^peer$"` matches + nothing and returns empty — silently. This is how A3 went unprobed for + three runs. Read addresses from inside the box (`ip -4 -o addr show dev + eth0`), not out of `incus list` CSV (which also quotes multi-address boxes + across lines). +5. **`incus delete -f a b c` aborts at the first MISSING name.** One interrupted + run then poisons the next: stale boxes survive cleanup and cascade into + half a dozen unrelated FAILs. Delete one name at a time. +6. **`apt-get -qq … >/dev/null` hides both a sudo prompt and the apt lock.** + `apt-daily`/`unattended-upgrades` hold the lock on a cloud image and apt + waits in complete silence. Pre-authorize sudo, set `DPkg::Lock::Timeout`, + and narrate. +7. **`claudebox exec` is `sudo -u claude -i`** — a *login zsh* with oh-my-zsh. + Fine for a human, needless machinery for a probe, and one more thing that + can hold an fd. Probes use `incus exec` directly. + +## Diagnosing a stall + +The drill narrates every long step. If it goes quiet, open a second terminal: + +```sh +# what is actually running / blocked? +ps -eo pid,etimes,stat,args | grep -Ev grep | grep -E 'apt|dpkg|incus|setup-host|sudo|curl' + +# apt lock held by a background upgrade? (the classic silent stall) +sudo fuser -v /var/lib/dpkg/lock-frontend +systemctl status unattended-upgrades apt-daily.service + +# incus itself wedged? +incus list +journalctl -u incus --no-pager -n 30 +``` + +## Running one probe by hand + +Nothing in the audit requires the whole drill. To answer **A3** on a host that +already has two boxes up (`archive` and `peer`): + +```sh +PEER_IP=$(timeout 20 incus exec peer -- ip -4 -o addr show dev eth0 \ + | awk '{split($4,a,"/"); print a[1]}') +timeout 30 incus exec archive -- curl -sS -m 5 -o /dev/null "http://$PEER_IP:8088" +echo "curl exit: $?" # 28 = dropped (isolated) · 7 = refused (it ARRIVED) · 0 = connected +timeout 30 incus exec archive -- ping -c1 -W2 "$PEER_IP" +echo "ping exit: $?" # 0 = ICMP replies — isolation is partial at best +``` + +No listener is needed, and none should be started: see trap 3. + +## Run history + +| Run | Result | What it cost | +| --- | --- | --- | +| 1 | hung at C7; ~9 false FAILs | traps 1, 2, 4 — pipefail, the exec-pty hang, the DHCP race | +| 2 | 42/49 | trap 5 — an interrupted run 1 left boxes behind, cascading 5 FAILs. Found the `claude`-on-PATH bug. Phase D delivered B1 (`@internal` rejected) and a B3 reading of *broken* | +| 3 | 48/49 | trap 4 — `eth0_ip` never matched, so A3 again unprobed. B3 now read *intact*, contradicting run 2 | +| 4 | hung at C4 | trap 2 again, this time via `claudebox exec` in a command substitution | +| 5 | stalled in host setup | trap 6 — silence through apt/sudo | + +**The instrument has been less reliable than the thing it measures.** Four of +five runs died on drill plumbing, not on claudebox. That is worth stating +plainly: if a probe can be answered by hand (see above), answer it by hand and +move on rather than paying for another full run. + +### The B3 flip — a lesson worth keeping + +Run 2 said `dns.mode=none` **broke** egress; run 3 said it was **intact**. Both +probed ~2s after setting the key, and setting `dns.mode` restarts the network's +dnsmasq — so run 2 caught the restart window and run 3 missed it. A design veto +was posted to #16 on the strength of run 2, then retracted. + +**A verdict drawn from one observation of a system with restart semantics is not +a verdict.** The probe now distinguishes *transient* (recovers within 30s) from +*broken* (does not), and any test #16 ships must tolerate that window rather +than race it.