heavy-duty/box
7
0
Fork 1
Code Issues 9 Pull requests Projects Releases 4 Packages Wiki Activity Actions 1

Merge pull request #35 from claude-hdb/drill/doctor-dnsmasq

Browse source 
fix(doctor): check that something is actually serving the network
This commit is contained in:
Daniel Marin 2026-07-14 02:23:51 +01:00 committed by GitHub
commit 08b6b2ce90
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 32 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
drill/RUNS.md
View file

@ -102,6 +102,15 @@ Read this before adding a probe. Every one of these cost a run.
`/dev/null`-ed) unset, the refusal to start on a dirty host, and
`doctor.sh`.
11. **A network Incus calls `Created` may have nothing serving it.** After an
unclean daemon death (a wedge, a SIGKILL, an OOM), Incus can come back
without respawning a network's **dnsmasq**. The bridge is up, `incus
network show` is perfect, `status: Created` — and no DHCP server exists, so
every box minted afterwards gets **no lease, no gateway, no DNS**, and dies
deep in cloud-init blaming Debian's mirrors. Incus's own status does not
cover this; the process table does. `doctor.sh` now checks it, because two
cold mints and an hour went into learning it the other way.
## Diagnosing a stall
**Start here: `bash drill/doctor.sh`** — it answers "what state is this host
@ -149,6 +158,7 @@ No listener is needed, and none should be started: see trap 3.
| 4 | hung at C4 | trap 2 again, this time via `claudebox exec` in a command substitution |
| 5 | stalled in host setup | trap 6 — silence through apt/sudo |
| 6 | stalled in `setup-host.sh` | trap 8 — cleanup ran *after* setup. Recovering the host exposed **two real claudebox bugs**: `setup-host` deadlocks the incus daemon when re-run with boxes up (#26), and clones inherit their source's machine-id → same DHCP lease → **two boxes, one IP** (#27) |
| 9 | aborted: cold mint failed, twice | **not** the drill and **not** the host's mutations (doctor was green): `claudenet` had **no dnsmasq** — it never respawned after the SIGKILL in run 6's recovery. Boxes got no DHCP lease at all. Trap 11 |
| 8 | aborted: cold mint failed | `cloud-init status: error` — **the box had no DNS at all**. Run 7's phase-D `dns.mode=none` survived the run and poisoned the host. Trap 10, and the reason `doctor.sh` exists |
| 7 | 41/49 | the clone-identity fix could not reboot (systemd needs a valid machine-id to shut down cleanly), so it never took effect → the IP collision persisted → the box lost networking → **phase D reported a false design veto against #16**. Trap 9. Also found: `dir` storage makes every clone a full disk copy (#29) |

22
drill/doctor.sh
View file

@ -49,6 +49,28 @@ if incus network show claudenet >/dev/null 2>&1; then
[ "$FIX" = 1 ] && { incus network unset claudenet dns.mode && inf "reverted: dns.mode unset"; }
fi
inf "ipv4.address = $(incus network get claudenet ipv4.address 2>/dev/null)"
# Incus reports the network as "Created" whether or not anything is actually
# SERVING it. Kill the daemon uncleanly (a wedge, an OOM, a SIGKILL) and it
# can come back without respawning this network's dnsmasq — the bridge is up,
# the config is perfect, and every box minted afterwards gets NO DHCP lease,
# so it dies deep inside cloud-init with "Temporary failure resolving
# deb.debian.org". Two cold mints and an hour of hunting went into learning
# that Incus's own status does not cover this. Ask the process table instead.
if pgrep -af 'dnsmasq.*--interface=claudenet' >/dev/null 2>&1; then
ok "a dnsmasq is serving claudenet (DHCP + DNS)"
else
no "NO dnsmasq is serving claudenet — the bridge is up and incus says 'Created', but nothing hands out leases"
inf "every box minted now gets no address, no DNS, and dies in cloud-init"
inf "fix: timeout 60 incus delete -f <any boxes>; sudo systemctl restart incus"
inf " (if it does not come back: teardown-host.sh, then re-run the drill)"
[ "$FIX" = 1 ] && {
inf "restarting incus to respawn it…"
sudo systemctl restart incus && sleep 5
pgrep -af 'dnsmasq.*--interface=claudenet' >/dev/null 2>&1 \
&& inf "reverted: dnsmasq is serving claudenet again" \
|| inf "STILL missing — run teardown-host.sh and let the drill rebuild the network"
}
fi
ipv6="$(incus network get claudenet ipv6.address 2>/dev/null)"
[ "$ipv6" = none ] && ok "ipv6.address = none (the isolation contract — every ACL rule is IPv4-only)" \
|| no "ipv6.address = $ipv6 — IPv6 is on and NOT covered by any ACL rule"