fix(drill): stop going silent through host setup, and keep the run log in the repo #24

Merged

dan-claude-bot merged 2 commits from drill/setup-visibility into main

2026-07-13 23:07:46 +00:00

dan-claude-bot commented

2026-07-13 23:05:36 +00:00

(Migrated from github.com)

Run 5 looked stuck for minutes right after the Host setup header. That stretch is the longest and most failure-prone part of the drill — and it was the only part that printed nothing at all, with two ways to block, both invisible:

sudo apt-get install -y -qq incus >/dev/null 2>&1 swallows a sudo password prompt. A stalled sudo and a hang look identical. Sudo is now pre-authorized up front (sudo -v), out loud, before anything unattended runs.
apt's lock. apt-daily / unattended-upgrades routinely hold it on a cloud image, and apt then waits in complete silence. It now runs with DPkg::Lock::Timeout=300 and an outer timeout, announces that it may be waiting on the lock, and on failure prints the two commands that identify the holder:
```
sudo fuser -v /var/lib/dpkg/lock-frontend
systemctl status unattended-upgrades apt-daily.service
```

Also: the apt call is skipped entirely when incus is already installed — which is every re-run on the same host, so this whole failure mode disappears for repeat runs — and each setup sub-step now narrates itself, so a stall is locatable instead of a blank terminal.

No probe logic changed; this is purely about the drill being legible while it works.

🤖 Generated with Claude Code

Run 5 looked stuck for minutes right after the `Host setup` header. That stretch is the longest and most failure-prone part of the drill — and it was the only part that printed **nothing at all**, with two ways to block, both invisible: 1. **`sudo apt-get install -y -qq incus >/dev/null 2>&1` swallows a sudo password prompt.** A stalled sudo and a hang look identical. Sudo is now pre-authorized up front (`sudo -v`), out loud, before anything unattended runs. 2. **apt's lock.** `apt-daily` / `unattended-upgrades` routinely hold it on a cloud image, and apt then waits **in complete silence**. It now runs with `DPkg::Lock::Timeout=300` and an outer `timeout`, announces that it may be waiting on the lock, and on failure prints the two commands that identify the holder: ```sh sudo fuser -v /var/lib/dpkg/lock-frontend systemctl status unattended-upgrades apt-daily.service ``` Also: the apt call is **skipped entirely when incus is already installed** — which is every re-run on the same host, so this whole failure mode disappears for repeat runs — and each setup sub-step now narrates itself, so a stall is locatable instead of a blank terminal. No probe logic changed; this is purely about the drill being legible while it works. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

dan-claude-bot commented

2026-07-13 23:06:50 +00:00

(Migrated from github.com)

Added drill/RUNS.md — the debugging knowledge from five runs, in the repo instead of scattered across PR bodies and chat. It carries:

The audit scoreboard — what #15 has answered so far, and that A3 (sibling isolation) has still never fired, which is the one claim #12 couldn't verify from code.
The claudebox bugs the drill found — the nftables gap (fixed), claude not on exec's PATH (fixed), and the ~95s cold mint that may mean cloud-init status --wait returns before runcmd finishes (open).
Seven traps, to read before adding a probe — pipefail vs refusal checks; $( ) waiting on stdout close rather than exit; never backgrounding inside a box; incus list name filters not being regexes (this alone cost three runs of A3); incus delete -f a b c aborting on the first missing name; apt/sudo silence; claudebox exec being a login zsh.
Stall diagnostics and how to run a single probe by hand — because nothing in the audit requires the whole drill.
The B3 flip-flop as a standing lesson: a verdict drawn from one observation of a system with restart semantics is not a verdict. I posted a design veto to #16 on run 2's reading and had to retract it after run 3.

It also states the uncomfortable thing plainly: four of five runs died on drill plumbing, not on claudebox. If a probe can be answered by hand, answer it by hand rather than paying for another full run.

Added `drill/RUNS.md` — the debugging knowledge from five runs, in the repo instead of scattered across PR bodies and chat. It carries: - **The audit scoreboard** — what #15 has answered so far, and that **A3 (sibling isolation) has still never fired**, which is the one claim #12 couldn't verify from code. - **The claudebox bugs the drill found** — the nftables gap (fixed), `claude` not on `exec`'s PATH (fixed), and the ~95s cold mint that may mean `cloud-init status --wait` returns before `runcmd` finishes (open). - **Seven traps, to read before adding a probe** — pipefail vs refusal checks; `$( )` waiting on stdout *close* rather than exit; never backgrounding inside a box; `incus list` name filters not being regexes (this alone cost three runs of A3); `incus delete -f a b c` aborting on the first missing name; apt/sudo silence; `claudebox exec` being a login zsh. - **Stall diagnostics** and **how to run a single probe by hand** — because nothing in the audit requires the whole drill. - **The B3 flip-flop** as a standing lesson: a verdict drawn from one observation of a system with restart semantics is not a verdict. I posted a design veto to #16 on run 2's reading and had to retract it after run 3. It also states the uncomfortable thing plainly: **four of five runs died on drill plumbing, not on claudebox.** If a probe can be answered by hand, answer it by hand rather than paying for another full run.

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/box#24

No description provided.