Merge pull request #35 from claude-hdb/drill/doctor-dnsmasq

fix(doctor): check that something is actually serving the network

drill/RUNS.md

@@ -102,6 +102,15 @@ Read this before adding a probe. Every one of these cost a run.
     `/dev/null`-ed) unset, the refusal to start on a dirty host, and
     `doctor.sh`.
 
+11. **A network Incus calls `Created` may have nothing serving it.** After an
+    unclean daemon death (a wedge, a SIGKILL, an OOM), Incus can come back
+    without respawning a network's **dnsmasq**. The bridge is up, `incus
+    network show` is perfect, `status: Created` — and no DHCP server exists, so
+    every box minted afterwards gets **no lease, no gateway, no DNS**, and dies
+    deep in cloud-init blaming Debian's mirrors. Incus's own status does not
+    cover this; the process table does. `doctor.sh` now checks it, because two
+    cold mints and an hour went into learning it the other way.
+
 ## Diagnosing a stall
 
 **Start here: `bash drill/doctor.sh`** — it answers "what state is this host
@@ -149,6 +158,7 @@ No listener is needed, and none should be started: see trap 3.
 | 4 | hung at C4 | trap 2 again, this time via `claudebox exec` in a command substitution |
 | 5 | stalled in host setup | trap 6 — silence through apt/sudo |
 | 6 | stalled in `setup-host.sh` | trap 8 — cleanup ran *after* setup. Recovering the host exposed **two real claudebox bugs**: `setup-host` deadlocks the incus daemon when re-run with boxes up (#26), and clones inherit their source's machine-id → same DHCP lease → **two boxes, one IP** (#27) |
+| 9 | aborted: cold mint failed, twice | **not** the drill and **not** the host's mutations (doctor was green): `claudenet` had **no dnsmasq** — it never respawned after the SIGKILL in run 6's recovery. Boxes got no DHCP lease at all. Trap 11 |
 | 8 | aborted: cold mint failed | `cloud-init status: error` — **the box had no DNS at all**. Run 7's phase-D `dns.mode=none` survived the run and poisoned the host. Trap 10, and the reason `doctor.sh` exists |
 | 7 | 41/49 | the clone-identity fix could not reboot (systemd needs a valid machine-id to shut down cleanly), so it never took effect → the IP collision persisted → the box lost networking → **phase D reported a false design veto against #16**. Trap 9. Also found: `dir` storage makes every clone a full disk copy (#29) |
 

drill/doctor.sh

@@ -49,6 +49,28 @@ if incus network show claudenet >/dev/null 2>&1; then
     [ "$FIX" = 1 ] && { incus network unset claudenet dns.mode && inf "reverted: dns.mode unset"; }
   fi
   inf "ipv4.address = $(incus network get claudenet ipv4.address 2>/dev/null)"
+  # Incus reports the network as "Created" whether or not anything is actually
+  # SERVING it. Kill the daemon uncleanly (a wedge, an OOM, a SIGKILL) and it
+  # can come back without respawning this network's dnsmasq — the bridge is up,
+  # the config is perfect, and every box minted afterwards gets NO DHCP lease,
+  # so it dies deep inside cloud-init with "Temporary failure resolving
+  # deb.debian.org". Two cold mints and an hour of hunting went into learning
+  # that Incus's own status does not cover this. Ask the process table instead.
+  if pgrep -af 'dnsmasq.*--interface=claudenet' >/dev/null 2>&1; then
+    ok "a dnsmasq is serving claudenet (DHCP + DNS)"
+  else
+    no "NO dnsmasq is serving claudenet — the bridge is up and incus says 'Created', but nothing hands out leases"
+    inf "every box minted now gets no address, no DNS, and dies in cloud-init"
+    inf "fix:  timeout 60 incus delete -f <any boxes>; sudo systemctl restart incus"
+    inf "      (if it does not come back: teardown-host.sh, then re-run the drill)"
+    [ "$FIX" = 1 ] && {
+      inf "restarting incus to respawn it…"
+      sudo systemctl restart incus && sleep 5
+      pgrep -af 'dnsmasq.*--interface=claudenet' >/dev/null 2>&1 \
+        && inf "reverted: dnsmasq is serving claudenet again" \
+        || inf "STILL missing — run teardown-host.sh and let the drill rebuild the network"
+    }
+  fi
   ipv6="$(incus network get claudenet ipv6.address 2>/dev/null)"
   [ "$ipv6" = none ] && ok "ipv6.address = none (the isolation contract — every ACL rule is IPv4-only)" \
                      || no "ipv6.address = $ipv6 — IPv6 is on and NOT covered by any ACL rule"