From b0eefd836957c025ce04a18e577fc051718614d5 Mon Sep 17 00:00:00 2001 From: claude-hdb Date: Tue, 14 Jul 2026 00:32:03 +0000 Subject: [PATCH] fix(drill): stop poisoning the host, and add a doctor to prove it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 8's cold mint failed with 'cloud-init status: error' — the box could not resolve deb.debian.org, or claude.ai, or anything. The cause was not in that run at all: run 7's phase D set dns.mode=none on claudenet, the run ended before reverting it, and every box minted afterwards came up with no DNS. This is the worst failure mode the drill has: a poisoned host does not fail the next run honestly, it produces confident wrong answers. It is how a false design veto against #16 got posted, and it wasted a cold mint plus an hour of diagnosis that had nothing to do with the code under test. Three defences: · the phase-D revert is armed with a trap BEFORE the first mutation, so it fires on any exit, Ctrl-C included; · the revert is VERIFIED rather than fired into /dev/null, so a failed unset can no longer masquerade as a successful one; · the drill refuses to start on a host still carrying the mutations. And drill/doctor.sh answers the question that kept being answered by hand: what state is this host actually in? Network, profile, ACL, leftover boxes, and whether a box can still resolve DNS — with --fix to revert the leftovers. RUNS.md gains trap 10. Co-Authored-By: Claude Fable 5 --- drill/README.md | 6 +++ drill/RUNS.md | 17 ++++++- drill/doctor.sh | 124 ++++++++++++++++++++++++++++++++++++++++++++++++ drill/drill.sh | 43 +++++++++++++++++ 4 files changed, 189 insertions(+), 1 deletion(-) create mode 100755 drill/doctor.sh diff --git a/drill/README.md b/drill/README.md index 701aedf..f5e06c6 100644 --- a/drill/README.md +++ b/drill/README.md @@ -25,6 +25,12 @@ and profile mutations stay applied with them). Exit 0 means every check passed. Roughly 20 minutes, most of it the cold box. +**Something wrong with the host?** `bash drill/doctor.sh` — it reports whether +the host is fit to drill (network, profile, ACL, leftover boxes, whether a box +can still resolve DNS), and `--fix` reverts what an aborted run left behind. +The drill mutates the host in phase D; an aborted run can leave a network that +mints boxes with **no DNS**. + **Iterating on the drill?** Read [RUNS.md](RUNS.md) first — it is the run log: what the audit has answered so far, the bugs the drill has found in claudebox, the traps this script has already fallen into (every one cost a run), how to diff --git a/drill/RUNS.md b/drill/RUNS.md index bbc5e96..c997a90 100644 --- a/drill/RUNS.md +++ b/drill/RUNS.md @@ -92,9 +92,23 @@ Read this before adding a probe. Every one of these cost a run. flip, in a different costume: **check that the thing you are measuring with still works before you trust what it tells you.** +10. **The drill mutates the host, and those mutations outlive an aborted run.** + Phase D sets `dns.mode=none` and NIC filtering. If the run dies before + reverting them, **every box minted afterwards has no DNS** — cloud-init + fails with `Temporary failure resolving deb.debian.org` — and the next run + reports that breakage as a *finding*. This is the worst failure mode in the + whole list: a poisoned host does not fail honestly, it produces confident + wrong answers. Hence the `trap`-armed revert, the verified (not + `/dev/null`-ed) unset, the refusal to start on a dirty host, and + `doctor.sh`. + ## Diagnosing a stall -The drill narrates every long step. If it goes quiet, open a second terminal: +**Start here: `bash drill/doctor.sh`** — it answers "what state is this host +actually in?" (network, profile, ACL, leftover boxes, and whether a box can +still resolve DNS), and `--fix` reverts what the drill left behind. + +If the drill goes quiet mid-run, open a second terminal: ```sh # what is actually running / blocked? @@ -135,6 +149,7 @@ No listener is needed, and none should be started: see trap 3. | 4 | hung at C4 | trap 2 again, this time via `claudebox exec` in a command substitution | | 5 | stalled in host setup | trap 6 — silence through apt/sudo | | 6 | stalled in `setup-host.sh` | trap 8 — cleanup ran *after* setup. Recovering the host exposed **two real claudebox bugs**: `setup-host` deadlocks the incus daemon when re-run with boxes up (#26), and clones inherit their source's machine-id → same DHCP lease → **two boxes, one IP** (#27) | +| 8 | aborted: cold mint failed | `cloud-init status: error` — **the box had no DNS at all**. Run 7's phase-D `dns.mode=none` survived the run and poisoned the host. Trap 10, and the reason `doctor.sh` exists | | 7 | 41/49 | the clone-identity fix could not reboot (systemd needs a valid machine-id to shut down cleanly), so it never took effect → the IP collision persisted → the box lost networking → **phase D reported a false design veto against #16**. Trap 9. Also found: `dir` storage makes every clone a full disk copy (#29) | **The instrument has been less reliable than the thing it measures.** Four of diff --git a/drill/doctor.sh b/drill/doctor.sh new file mode 100755 index 0000000..dc4ec9d --- /dev/null +++ b/drill/doctor.sh @@ -0,0 +1,124 @@ +#!/usr/bin/env bash +# doctor.sh — is this host in a fit state to drill, and if not, what is wrong? +# +# bash drill/doctor.sh # report +# bash drill/doctor.sh --fix # report, then revert what the drill left behind +# +# The drill MUTATES the host in phase D (dns.mode, NIC filtering, ACL rules) to +# rehearse the #16 hardening. If a run aborts before it reverts them, those +# mutations outlive it — and the next run mints boxes on a broken network. That +# is not a hypothetical: it is how a box came up with no DNS at all +# ("Temporary failure resolving deb.debian.org" in cloud-init), and how a false +# design veto against #16 got posted from a poisoned baseline. +# +# This script is the answer to "what state is the host actually in?" — the +# question that kept getting answered by hand. +set -u + +FIX=0 +[ "${1:-}" = "--fix" ] && FIX=1 + +bad=0 +ok() { printf ' \033[32mOK\033[0m %s\n' "$*"; } +no() { printf ' \033[31mDIRTY\033[0m %s\n' "$*"; bad=$((bad + 1)); } +inf() { printf ' %s\n' "$*"; } +head_() { printf '\n\033[1m%s\033[0m\n' "$*"; } + +command -v incus >/dev/null || { echo "doctor: incus is not installed on this host."; exit 1; } +timeout 10 incus list >/dev/null 2>&1 || { + echo "doctor: the incus daemon is not answering (see issue #26 for recovery):" + echo " sudo pkill -9 -f 'incusd shutdown'" + echo " sudo systemctl stop incus.service incus.socket" + echo " sudo systemctl reset-failed incus.service incus.socket" + echo " sudo systemctl start incus.socket incus.service" + exit 1 +} + +head_ "Network — claudenet" +if incus network show claudenet >/dev/null 2>&1; then + dns="$(incus network get claudenet dns.mode 2>/dev/null)" + if [ -z "$dns" ] || [ "$dns" = managed ]; then + ok "dns.mode = ${dns:-}" + else + no "dns.mode = $dns ← the drill's phase D left this behind. Boxes minted now get NO working DNS." + [ "$FIX" = 1 ] && { incus network unset claudenet dns.mode && inf "reverted: dns.mode unset"; } + fi + inf "ipv4.address = $(incus network get claudenet ipv4.address 2>/dev/null)" + ipv6="$(incus network get claudenet ipv6.address 2>/dev/null)" + [ "$ipv6" = none ] && ok "ipv6.address = none (the isolation contract — every ACL rule is IPv4-only)" \ + || no "ipv6.address = $ipv6 — IPv6 is on and NOT covered by any ACL rule" +else + inf "claudenet does not exist (a fresh host — setup-host.sh will create it)" +fi + +head_ "Profile — claude-dev (the NIC is the isolation contract)" +if incus profile show claude-dev >/dev/null 2>&1; then + for k in security.mac_filtering security.ipv4_filtering; do + v="$(incus profile device get claude-dev eth0 "$k" 2>/dev/null)" + if [ -z "$v" ]; then + ok "$k unset (as shipped)" + else + no "$k = $v ← phase D left this behind. A box can fail to get on the network at all." + [ "$FIX" = 1 ] && { incus profile device unset claude-dev eth0 "$k" && inf "reverted: $k unset"; } + fi + done + inf "cpu/mem: $(incus profile get claude-dev limits.cpu 2>/dev/null)/$(incus profile get claude-dev limits.memory 2>/dev/null) (the drill lowers these on a small host)" +else + inf "claude-dev does not exist (a fresh host)" +fi + +head_ "ACL — claude-isolate" +if incus network acl show claude-isolate >/dev/null 2>&1; then + n="$(incus network acl show claude-isolate | grep -c 'action:' || true)" + inf "$n rules" + incus network acl show claude-isolate | grep -E 'action:|destination:' | sed 's/^/ /' + if incus network acl show claude-isolate | grep -q '@internal'; then + no "an @internal rule survived phase D" + [ "$FIX" = 1 ] && { incus network acl rule remove claude-isolate egress action=drop destination=@internal && inf "reverted: @internal rule removed"; } + fi +else + inf "claude-isolate does not exist (a fresh host)" +fi + +head_ "Instances" +left="$(incus list --format csv --columns ns 2>/dev/null)" +[ -z "$left" ] && inf "(none)" || printf ' %s\n' "$left" +for b in drill clone archive peer payroll cbprobe cbcopy cbnotours; do + if incus config show "$b" >/dev/null 2>&1; then + no "leftover drill box: $b" + [ "$FIX" = 1 ] && { timeout 60 incus delete -f "$b" >/dev/null 2>&1 && inf "reverted: deleted $b"; } + fi +done + +head_ "Can a box actually resolve DNS?" +probe="" +for b in drill archive peer clone; do + incus config show "$b" >/dev/null 2>&1 && { probe="$b"; break; } +done +if [ -n "$probe" ] && [ "$FIX" != 1 ]; then + inf "probing inside '$probe' (the cheapest test of a poisoned network):" + inf "resolv.conf: $(timeout 20 incus exec "$probe" -- sh -c 'grep -m2 nameserver /etc/resolv.conf' 2>/dev/null | tr '\n' ' ')" + if timeout 25 incus exec "$probe" -- getent hosts deb.debian.org >/dev/null 2>&1; then + ok "$probe resolves deb.debian.org" + else + no "$probe CANNOT resolve deb.debian.org — this is what breaks cloud-init on every new box" + fi + timeout 20 incus exec "$probe" -- ping -c1 -W2 10.87.0.1 >/dev/null 2>&1 \ + && ok "$probe reaches the gateway (10.87.0.1) — so it is DNS, not routing" \ + || no "$probe cannot even reach the gateway" +else + inf "no box to probe with (mint one, or run without --fix after a run)" +fi + +head_ "Verdict" +if [ "$bad" -eq 0 ]; then + printf ' \033[32mclean\033[0m — this host is fit to drill.\n\n' + exit 0 +fi +printf ' \033[31m%s problem(s)\033[0m — this host is NOT fit to drill.\n' "$bad" +if [ "$FIX" = 1 ]; then + printf ' reverted what could be reverted; re-run doctor to confirm.\n\n' +else + printf ' run: bash drill/doctor.sh --fix\n\n' +fi +exit 1 diff --git a/drill/drill.sh b/drill/drill.sh index 8589319..d21d7d0 100755 --- a/drill/drill.sh +++ b/drill/drill.sh @@ -202,6 +202,22 @@ KEEP="${KEEP:-0}" # filtering) in place, so setup would be converging against a moving target. # Take the boxes down and revert the mutations FIRST; then the host is a # clean-ish slate and setup-host is the no-op it should be. +# A host still carrying a previous run's phase-D mutations mints boxes with no +# DNS, and then reports the resulting breakage as a finding. Refuse to run. +dirty="" +[ -n "$(incus network get claudenet dns.mode 2>/dev/null)" ] && dirty="dns.mode" +[ -n "$(incus profile device get claude-dev eth0 security.ipv4_filtering 2>/dev/null)" ] && dirty="$dirty ipv4_filtering" +[ -n "$(incus profile device get claude-dev eth0 security.mac_filtering 2>/dev/null)" ] && dirty="$dirty mac_filtering" +if [ -n "$dirty" ]; then + note "this host still carries a previous run's phase-D mutations:$dirty — reverting them now" + incus network unset claudenet dns.mode >/dev/null 2>&1 + incus profile device unset claude-dev eth0 security.mac_filtering >/dev/null 2>&1 + incus profile device unset claude-dev eth0 security.ipv4_filtering >/dev/null 2>&1 + incus network acl rule remove claude-isolate egress action=drop destination=@internal >/dev/null 2>&1 + still="$(incus network get claudenet dns.mode 2>/dev/null)" + [ -n "$still" ] && { echo "drill: could not revert dns.mode ('$still'). run: bash drill/doctor.sh --fix" >&2; exit 1; } +fi + inf "clearing anything a previous run left behind…" # One name at a time — 'incus delete -f a b c' aborts at the first MISSING name, # which is how run 2 inherited run 1's boxes and cascaded five false FAILs. @@ -542,6 +558,21 @@ else aud "A7 inbound host→box: NOT PROBED" fi +# The mutations below OUTLIVE the run if it dies: dns.mode=none leaves every +# box minted afterwards with NO DNS at all (cloud-init then fails with +# "Temporary failure resolving deb.debian.org"), and NIC filtering can stop a +# box getting on the network. A poisoned host does not fail the NEXT run +# honestly — it produces confident, wrong answers, which is how a false design +# veto against #16 got posted. So arm the revert BEFORE making the first +# mutation, and let it fire on any exit, including Ctrl-C. +revert_phase_d() { + incus network unset claudenet dns.mode >/dev/null 2>&1 + incus profile device unset claude-dev eth0 security.mac_filtering >/dev/null 2>&1 + incus profile device unset claude-dev eth0 security.ipv4_filtering >/dev/null 2>&1 + incus network acl rule remove claude-isolate egress action=drop destination=@internal >/dev/null 2>&1 +} +trap 'revert_phase_d' EXIT INT TERM + # =========================================================================== phase "D. Hardening rehearsal — #16's changes, applied live (#15 section B)" # =========================================================================== @@ -634,6 +665,18 @@ fi fi # end of the BASELINE_OK guard around phase D +# Revert now, and CHECK it — the old code fired these into /dev/null and moved +# on, so a failed unset was indistinguishable from a successful one. +revert_phase_d +d="$(incus network get claudenet dns.mode 2>/dev/null)" +f="$(incus profile device get claude-dev eth0 security.ipv4_filtering 2>/dev/null)" +if [ -z "$d" ] && [ -z "$f" ]; then + ok "phase D reverted: dns.mode and NIC filtering are back to shipped defaults" +else + no "PHASE D DID NOT REVERT (dns.mode='$d' ipv4_filtering='$f') — this host will poison the next run" + inf "fix it with: bash drill/doctor.sh --fix" +fi + # =========================================================================== if [ "$KEEP" = 1 ]; then phase "Boxes left up (--keep-boxes)" -- 2.45.2