fix(drill): stop poisoning the host, and add a doctor to prove it

Run 8's cold mint failed with 'cloud-init status: error' — the box could
not resolve deb.debian.org, or claude.ai, or anything. The cause was not
in that run at all: run 7's phase D set dns.mode=none on claudenet, the
run ended before reverting it, and every box minted afterwards came up
with no DNS.

This is the worst failure mode the drill has: a poisoned host does not
fail the next run honestly, it produces confident wrong answers. It is
how a false design veto against #16 got posted, and it wasted a cold
mint plus an hour of diagnosis that had nothing to do with the code
under test.

Three defences:
  · the phase-D revert is armed with a trap BEFORE the first mutation,
    so it fires on any exit, Ctrl-C included;
  · the revert is VERIFIED rather than fired into /dev/null, so a failed
    unset can no longer masquerade as a successful one;
  · the drill refuses to start on a host still carrying the mutations.

And drill/doctor.sh answers the question that kept being answered by
hand: what state is this host actually in? Network, profile, ACL,
leftover boxes, and whether a box can still resolve DNS — with --fix to
revert the leftovers.

RUNS.md gains trap 10.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

drill/README.md

@@ -25,6 +25,12 @@ and profile mutations stay applied with them).
 
 Exit 0 means every check passed. Roughly 20 minutes, most of it the cold box.
 
+**Something wrong with the host?** `bash drill/doctor.sh` — it reports whether
+the host is fit to drill (network, profile, ACL, leftover boxes, whether a box
+can still resolve DNS), and `--fix` reverts what an aborted run left behind.
+The drill mutates the host in phase D; an aborted run can leave a network that
+mints boxes with **no DNS**.
+
 **Iterating on the drill?** Read [RUNS.md](RUNS.md) first — it is the run log:
 what the audit has answered so far, the bugs the drill has found in claudebox,
 the traps this script has already fallen into (every one cost a run), how to

drill/RUNS.md

@@ -92,9 +92,23 @@ Read this before adding a probe. Every one of these cost a run.
    flip, in a different costume: **check that the thing you are measuring with
    still works before you trust what it tells you.**
 
+10. **The drill mutates the host, and those mutations outlive an aborted run.**
+    Phase D sets `dns.mode=none` and NIC filtering. If the run dies before
+    reverting them, **every box minted afterwards has no DNS** — cloud-init
+    fails with `Temporary failure resolving deb.debian.org` — and the next run
+    reports that breakage as a *finding*. This is the worst failure mode in the
+    whole list: a poisoned host does not fail honestly, it produces confident
+    wrong answers. Hence the `trap`-armed revert, the verified (not
+    `/dev/null`-ed) unset, the refusal to start on a dirty host, and
+    `doctor.sh`.
+
 ## Diagnosing a stall
 
-The drill narrates every long step. If it goes quiet, open a second terminal:
+**Start here: `bash drill/doctor.sh`** — it answers "what state is this host
+actually in?" (network, profile, ACL, leftover boxes, and whether a box can
+still resolve DNS), and `--fix` reverts what the drill left behind.
+
+If the drill goes quiet mid-run, open a second terminal:
 
 ```sh
 # what is actually running / blocked?
@@ -135,6 +149,7 @@ No listener is needed, and none should be started: see trap 3.
 | 4 | hung at C4 | trap 2 again, this time via `claudebox exec` in a command substitution |
 | 5 | stalled in host setup | trap 6 — silence through apt/sudo |
 | 6 | stalled in `setup-host.sh` | trap 8 — cleanup ran *after* setup. Recovering the host exposed **two real claudebox bugs**: `setup-host` deadlocks the incus daemon when re-run with boxes up (#26), and clones inherit their source's machine-id → same DHCP lease → **two boxes, one IP** (#27) |
+| 8 | aborted: cold mint failed | `cloud-init status: error` — **the box had no DNS at all**. Run 7's phase-D `dns.mode=none` survived the run and poisoned the host. Trap 10, and the reason `doctor.sh` exists |
 | 7 | 41/49 | the clone-identity fix could not reboot (systemd needs a valid machine-id to shut down cleanly), so it never took effect → the IP collision persisted → the box lost networking → **phase D reported a false design veto against #16**. Trap 9. Also found: `dir` storage makes every clone a full disk copy (#29) |
 
 **The instrument has been less reliable than the thing it measures.** Four of

drill/doctor.sh

@@ -0,0 +1,124 @@
+#!/usr/bin/env bash
+# doctor.sh — is this host in a fit state to drill, and if not, what is wrong?
+#
+#   bash drill/doctor.sh          # report
+#   bash drill/doctor.sh --fix    # report, then revert what the drill left behind
+#
+# The drill MUTATES the host in phase D (dns.mode, NIC filtering, ACL rules) to
+# rehearse the #16 hardening. If a run aborts before it reverts them, those
+# mutations outlive it — and the next run mints boxes on a broken network. That
+# is not a hypothetical: it is how a box came up with no DNS at all
+# ("Temporary failure resolving deb.debian.org" in cloud-init), and how a false
+# design veto against #16 got posted from a poisoned baseline.
+#
+# This script is the answer to "what state is the host actually in?" — the
+# question that kept getting answered by hand.
+set -u
+
+FIX=0
+[ "${1:-}" = "--fix" ] && FIX=1
+
+bad=0
+ok()   { printf '  \033[32mOK\033[0m    %s\n' "$*"; }
+no()   { printf '  \033[31mDIRTY\033[0m %s\n' "$*"; bad=$((bad + 1)); }
+inf()  { printf '        %s\n' "$*"; }
+head_() { printf '\n\033[1m%s\033[0m\n' "$*"; }
+
+command -v incus >/dev/null || { echo "doctor: incus is not installed on this host."; exit 1; }
+timeout 10 incus list >/dev/null 2>&1 || {
+  echo "doctor: the incus daemon is not answering (see issue #26 for recovery):"
+  echo "  sudo pkill -9 -f 'incusd shutdown'"
+  echo "  sudo systemctl stop incus.service incus.socket"
+  echo "  sudo systemctl reset-failed incus.service incus.socket"
+  echo "  sudo systemctl start incus.socket incus.service"
+  exit 1
+}
+
+head_ "Network — claudenet"
+if incus network show claudenet >/dev/null 2>&1; then
+  dns="$(incus network get claudenet dns.mode 2>/dev/null)"
+  if [ -z "$dns" ] || [ "$dns" = managed ]; then
+    ok "dns.mode = ${dns:-<unset, i.e. managed>}"
+  else
+    no "dns.mode = $dns  ← the drill's phase D left this behind. Boxes minted now get NO working DNS."
+    [ "$FIX" = 1 ] && { incus network unset claudenet dns.mode && inf "reverted: dns.mode unset"; }
+  fi
+  inf "ipv4.address = $(incus network get claudenet ipv4.address 2>/dev/null)"
+  ipv6="$(incus network get claudenet ipv6.address 2>/dev/null)"
+  [ "$ipv6" = none ] && ok "ipv6.address = none (the isolation contract — every ACL rule is IPv4-only)" \
+                     || no "ipv6.address = $ipv6 — IPv6 is on and NOT covered by any ACL rule"
+else
+  inf "claudenet does not exist (a fresh host — setup-host.sh will create it)"
+fi
+
+head_ "Profile — claude-dev (the NIC is the isolation contract)"
+if incus profile show claude-dev >/dev/null 2>&1; then
+  for k in security.mac_filtering security.ipv4_filtering; do
+    v="$(incus profile device get claude-dev eth0 "$k" 2>/dev/null)"
+    if [ -z "$v" ]; then
+      ok "$k unset (as shipped)"
+    else
+      no "$k = $v  ← phase D left this behind. A box can fail to get on the network at all."
+      [ "$FIX" = 1 ] && { incus profile device unset claude-dev eth0 "$k" && inf "reverted: $k unset"; }
+    fi
+  done
+  inf "cpu/mem: $(incus profile get claude-dev limits.cpu 2>/dev/null)/$(incus profile get claude-dev limits.memory 2>/dev/null) (the drill lowers these on a small host)"
+else
+  inf "claude-dev does not exist (a fresh host)"
+fi
+
+head_ "ACL — claude-isolate"
+if incus network acl show claude-isolate >/dev/null 2>&1; then
+  n="$(incus network acl show claude-isolate | grep -c 'action:' || true)"
+  inf "$n rules"
+  incus network acl show claude-isolate | grep -E 'action:|destination:' | sed 's/^/        /'
+  if incus network acl show claude-isolate | grep -q '@internal'; then
+    no "an @internal rule survived phase D"
+    [ "$FIX" = 1 ] && { incus network acl rule remove claude-isolate egress action=drop destination=@internal && inf "reverted: @internal rule removed"; }
+  fi
+else
+  inf "claude-isolate does not exist (a fresh host)"
+fi
+
+head_ "Instances"
+left="$(incus list --format csv --columns ns 2>/dev/null)"
+[ -z "$left" ] && inf "(none)" || printf '        %s\n' "$left"
+for b in drill clone archive peer payroll cbprobe cbcopy cbnotours; do
+  if incus config show "$b" >/dev/null 2>&1; then
+    no "leftover drill box: $b"
+    [ "$FIX" = 1 ] && { timeout 60 incus delete -f "$b" >/dev/null 2>&1 && inf "reverted: deleted $b"; }
+  fi
+done
+
+head_ "Can a box actually resolve DNS?"
+probe=""
+for b in drill archive peer clone; do
+  incus config show "$b" >/dev/null 2>&1 && { probe="$b"; break; }
+done
+if [ -n "$probe" ] && [ "$FIX" != 1 ]; then
+  inf "probing inside '$probe' (the cheapest test of a poisoned network):"
+  inf "resolv.conf: $(timeout 20 incus exec "$probe" -- sh -c 'grep -m2 nameserver /etc/resolv.conf' 2>/dev/null | tr '\n' ' ')"
+  if timeout 25 incus exec "$probe" -- getent hosts deb.debian.org >/dev/null 2>&1; then
+    ok "$probe resolves deb.debian.org"
+  else
+    no "$probe CANNOT resolve deb.debian.org — this is what breaks cloud-init on every new box"
+  fi
+  timeout 20 incus exec "$probe" -- ping -c1 -W2 10.87.0.1 >/dev/null 2>&1 \
+    && ok "$probe reaches the gateway (10.87.0.1) — so it is DNS, not routing" \
+    || no "$probe cannot even reach the gateway"
+else
+  inf "no box to probe with (mint one, or run without --fix after a run)"
+fi
+
+head_ "Verdict"
+if [ "$bad" -eq 0 ]; then
+  printf '  \033[32mclean\033[0m — this host is fit to drill.\n\n'
+  exit 0
+fi
+printf '  \033[31m%s problem(s)\033[0m — this host is NOT fit to drill.\n' "$bad"
+if [ "$FIX" = 1 ]; then
+  printf '  reverted what could be reverted; re-run doctor to confirm.\n\n'
+else
+  printf '  run:  bash drill/doctor.sh --fix\n\n'
+fi
+exit 1

drill/drill.sh

@@ -202,6 +202,22 @@ KEEP="${KEEP:-0}"
 # filtering) in place, so setup would be converging against a moving target.
 # Take the boxes down and revert the mutations FIRST; then the host is a
 # clean-ish slate and setup-host is the no-op it should be.
+# A host still carrying a previous run's phase-D mutations mints boxes with no
+# DNS, and then reports the resulting breakage as a finding. Refuse to run.
+dirty=""
+[ -n "$(incus network get claudenet dns.mode 2>/dev/null)" ] && dirty="dns.mode"
+[ -n "$(incus profile device get claude-dev eth0 security.ipv4_filtering 2>/dev/null)" ] && dirty="$dirty ipv4_filtering"
+[ -n "$(incus profile device get claude-dev eth0 security.mac_filtering 2>/dev/null)" ] && dirty="$dirty mac_filtering"
+if [ -n "$dirty" ]; then
+  note "this host still carries a previous run's phase-D mutations:$dirty — reverting them now"
+  incus network unset claudenet dns.mode >/dev/null 2>&1
+  incus profile device unset claude-dev eth0 security.mac_filtering >/dev/null 2>&1
+  incus profile device unset claude-dev eth0 security.ipv4_filtering >/dev/null 2>&1
+  incus network acl rule remove claude-isolate egress action=drop destination=@internal >/dev/null 2>&1
+  still="$(incus network get claudenet dns.mode 2>/dev/null)"
+  [ -n "$still" ] && { echo "drill: could not revert dns.mode ('$still'). run: bash drill/doctor.sh --fix" >&2; exit 1; }
+fi
+
 inf "clearing anything a previous run left behind…"
 # One name at a time — 'incus delete -f a b c' aborts at the first MISSING name,
 # which is how run 2 inherited run 1's boxes and cascaded five false FAILs.
@@ -542,6 +558,21 @@ else
   aud "A7 inbound host→box: NOT PROBED"
 fi
 
+# The mutations below OUTLIVE the run if it dies: dns.mode=none leaves every
+# box minted afterwards with NO DNS at all (cloud-init then fails with
+# "Temporary failure resolving deb.debian.org"), and NIC filtering can stop a
+# box getting on the network. A poisoned host does not fail the NEXT run
+# honestly — it produces confident, wrong answers, which is how a false design
+# veto against #16 got posted. So arm the revert BEFORE making the first
+# mutation, and let it fire on any exit, including Ctrl-C.
+revert_phase_d() {
+  incus network unset claudenet dns.mode >/dev/null 2>&1
+  incus profile device unset claude-dev eth0 security.mac_filtering >/dev/null 2>&1
+  incus profile device unset claude-dev eth0 security.ipv4_filtering >/dev/null 2>&1
+  incus network acl rule remove claude-isolate egress action=drop destination=@internal >/dev/null 2>&1
+}
+trap 'revert_phase_d' EXIT INT TERM
+
 # ===========================================================================
 phase "D. Hardening rehearsal — #16's changes, applied live (#15 section B)"
 # ===========================================================================
@@ -634,6 +665,18 @@ fi
 
 fi   # end of the BASELINE_OK guard around phase D
 
+# Revert now, and CHECK it — the old code fired these into /dev/null and moved
+# on, so a failed unset was indistinguishable from a successful one.
+revert_phase_d
+d="$(incus network get claudenet dns.mode 2>/dev/null)"
+f="$(incus profile device get claude-dev eth0 security.ipv4_filtering 2>/dev/null)"
+if [ -z "$d" ] && [ -z "$f" ]; then
+  ok "phase D reverted: dns.mode and NIC filtering are back to shipped defaults"
+else
+  no "PHASE D DID NOT REVERT (dns.mode='$d' ipv4_filtering='$f') — this host will poison the next run"
+  inf "fix it with: bash drill/doctor.sh --fix"
+fi
+
 # ===========================================================================
 if [ "$KEEP" = 1 ]; then
   phase "Boxes left up (--keep-boxes)"