diff --git a/drill/RUNS.md b/drill/RUNS.md index c997a90..0aa54a4 100644 --- a/drill/RUNS.md +++ b/drill/RUNS.md @@ -102,6 +102,15 @@ Read this before adding a probe. Every one of these cost a run. `/dev/null`-ed) unset, the refusal to start on a dirty host, and `doctor.sh`. +11. **A network Incus calls `Created` may have nothing serving it.** After an + unclean daemon death (a wedge, a SIGKILL, an OOM), Incus can come back + without respawning a network's **dnsmasq**. The bridge is up, `incus + network show` is perfect, `status: Created` — and no DHCP server exists, so + every box minted afterwards gets **no lease, no gateway, no DNS**, and dies + deep in cloud-init blaming Debian's mirrors. Incus's own status does not + cover this; the process table does. `doctor.sh` now checks it, because two + cold mints and an hour went into learning it the other way. + ## Diagnosing a stall **Start here: `bash drill/doctor.sh`** — it answers "what state is this host @@ -149,6 +158,7 @@ No listener is needed, and none should be started: see trap 3. | 4 | hung at C4 | trap 2 again, this time via `claudebox exec` in a command substitution | | 5 | stalled in host setup | trap 6 — silence through apt/sudo | | 6 | stalled in `setup-host.sh` | trap 8 — cleanup ran *after* setup. Recovering the host exposed **two real claudebox bugs**: `setup-host` deadlocks the incus daemon when re-run with boxes up (#26), and clones inherit their source's machine-id → same DHCP lease → **two boxes, one IP** (#27) | +| 9 | aborted: cold mint failed, twice | **not** the drill and **not** the host's mutations (doctor was green): `claudenet` had **no dnsmasq** — it never respawned after the SIGKILL in run 6's recovery. Boxes got no DHCP lease at all. Trap 11 | | 8 | aborted: cold mint failed | `cloud-init status: error` — **the box had no DNS at all**. Run 7's phase-D `dns.mode=none` survived the run and poisoned the host. Trap 10, and the reason `doctor.sh` exists | | 7 | 41/49 | the clone-identity fix could not reboot (systemd needs a valid machine-id to shut down cleanly), so it never took effect → the IP collision persisted → the box lost networking → **phase D reported a false design veto against #16**. Trap 9. Also found: `dir` storage makes every clone a full disk copy (#29) | diff --git a/drill/doctor.sh b/drill/doctor.sh index ac26f9b..34769cc 100755 --- a/drill/doctor.sh +++ b/drill/doctor.sh @@ -49,6 +49,28 @@ if incus network show claudenet >/dev/null 2>&1; then [ "$FIX" = 1 ] && { incus network unset claudenet dns.mode && inf "reverted: dns.mode unset"; } fi inf "ipv4.address = $(incus network get claudenet ipv4.address 2>/dev/null)" + # Incus reports the network as "Created" whether or not anything is actually + # SERVING it. Kill the daemon uncleanly (a wedge, an OOM, a SIGKILL) and it + # can come back without respawning this network's dnsmasq — the bridge is up, + # the config is perfect, and every box minted afterwards gets NO DHCP lease, + # so it dies deep inside cloud-init with "Temporary failure resolving + # deb.debian.org". Two cold mints and an hour of hunting went into learning + # that Incus's own status does not cover this. Ask the process table instead. + if pgrep -af 'dnsmasq.*--interface=claudenet' >/dev/null 2>&1; then + ok "a dnsmasq is serving claudenet (DHCP + DNS)" + else + no "NO dnsmasq is serving claudenet — the bridge is up and incus says 'Created', but nothing hands out leases" + inf "every box minted now gets no address, no DNS, and dies in cloud-init" + inf "fix: timeout 60 incus delete -f ; sudo systemctl restart incus" + inf " (if it does not come back: teardown-host.sh, then re-run the drill)" + [ "$FIX" = 1 ] && { + inf "restarting incus to respawn it…" + sudo systemctl restart incus && sleep 5 + pgrep -af 'dnsmasq.*--interface=claudenet' >/dev/null 2>&1 \ + && inf "reverted: dnsmasq is serving claudenet again" \ + || inf "STILL missing — run teardown-host.sh and let the drill rebuild the network" + } + fi ipv6="$(incus network get claudenet ipv6.address 2>/dev/null)" [ "$ipv6" = none ] && ok "ipv6.address = none (the isolation contract — every ACL rule is IPv4-only)" \ || no "ipv6.address = $ipv6 — IPv6 is on and NOT covered by any ACL rule"