From 769287eb7ebd8582432962d3c2d48f0c8acd5d04 Mon Sep 17 00:00:00 2001 From: claude-hdb Date: Tue, 14 Jul 2026 00:04:01 +0000 Subject: [PATCH] fix(drill): refuse to judge #16 on a broken baseline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Run 7's box had no network — a clone/source IP collision had taken it down before a single hardening change was applied. Phase D measured anyway and reported "L2 filtering BREAKS the box — design veto". It does not. The box was already broken, and #16 would have been redesigned around a fiction. Phase D is now gated on baseline egress passing, and says loudly that it is skipping rather than quietly producing a verdict. The dns.mode rejection also stops swallowing incus's error message — the message IS the finding. RUNS.md gains trap 9: check that the thing you are measuring WITH still works before you trust what it tells you. Same failure as the B3 flip, different costume. Co-Authored-By: Claude Fable 5 --- drill/RUNS.md | 12 +++++++++++- drill/drill.sh | 32 ++++++++++++++++++++++++++------ 2 files changed, 37 insertions(+), 7 deletions(-) diff --git a/drill/RUNS.md b/drill/RUNS.md index ed22e92..bbc5e96 100644 --- a/drill/RUNS.md +++ b/drill/RUNS.md @@ -83,6 +83,15 @@ Read this before adding a probe. Every one of these cost a run. filtering) in place, so setup converges against a moving target. Delete the boxes and revert the mutations **first**. +9. **Never render a verdict on a broken baseline.** Run 7's box had no network + (a clone/source IP collision), and phase D dutifully reported *"L2 filtering + BREAKS the box — design veto"*. It did not; the box was already broken. A + measurement taken on a broken instrument is not evidence, and #16 would have + been redesigned around a fiction. Phase D is now gated on baseline egress + passing, and refuses to judge otherwise. This is the same failure as the B3 + flip, in a different costume: **check that the thing you are measuring with + still works before you trust what it tells you.** + ## Diagnosing a stall The drill narrates every long step. If it goes quiet, open a second terminal: @@ -125,7 +134,8 @@ No listener is needed, and none should be started: see trap 3. | 3 | 48/49 | trap 4 — `eth0_ip` never matched, so A3 again unprobed. B3 now read *intact*, contradicting run 2 | | 4 | hung at C4 | trap 2 again, this time via `claudebox exec` in a command substitution | | 5 | stalled in host setup | trap 6 — silence through apt/sudo | -| 6 | stalled in `setup-host.sh` | trap 8 — cleanup ran *after* setup, so setup reconfigured claudenet's ACLs while run 4's boxes were still attached to it | +| 6 | stalled in `setup-host.sh` | trap 8 — cleanup ran *after* setup. Recovering the host exposed **two real claudebox bugs**: `setup-host` deadlocks the incus daemon when re-run with boxes up (#26), and clones inherit their source's machine-id → same DHCP lease → **two boxes, one IP** (#27) | +| 7 | 41/49 | the clone-identity fix could not reboot (systemd needs a valid machine-id to shut down cleanly), so it never took effect → the IP collision persisted → the box lost networking → **phase D reported a false design veto against #16**. Trap 9. Also found: `dir` storage makes every clone a full disk copy (#29) | **The instrument has been less reliable than the thing it measures.** Four of five runs died on drill plumbing, not on claudebox. That is worth stating diff --git a/drill/drill.sh b/drill/drill.sh index b2129cd..8589319 100755 --- a/drill/drill.sh +++ b/drill/drill.sh @@ -428,9 +428,15 @@ else fi # C1 — public egress (#15 A1; resolving the hostname also proves A5, gateway DNS) -[ "$(box_curl archive https://api.github.com 20)" = 0 ] \ - && { ok "box reaches the public internet (and gateway DNS resolves public names)"; aud "A1/A5 egress + public DNS: PASS"; } \ - || { no "box cannot reach the internet (a box that can't is useless)"; aud "A1/A5 egress: FAIL"; } +BASELINE_OK=1 +if [ "$(box_curl archive https://api.github.com 20)" = 0 ]; then + ok "box reaches the public internet (and gateway DNS resolves public names)" + aud "A1/A5 egress + public DNS: PASS" +else + BASELINE_OK=0 + no "box cannot reach the internet (a box that can't is useless)" + aud "A1/A5 egress: FAIL" +fi # C2 — box → host (#15 A2). The host DOES listen on the gateway: dnsmasq is on # :53 by design (that carve-out is what makes egress DNS work). So probe a port @@ -541,13 +547,25 @@ phase "D. Hardening rehearsal — #16's changes, applied live (#15 section B)" # =========================================================================== # The host is disposable, so rehearse the exact changes #16 proposes and watch # what breaks. A FAIL here vetoes a piece of #16's design before it is written. +# +# But ONLY if the box was healthy to begin with. On run 7 the baseline was +# broken (a clone/source IP collision took the box's networking down), and phase +# D dutifully reported "L2 filtering BREAKS the box — design veto". It did not. +# A verdict measured on a broken box is not a verdict, it is a slander; #16 +# would have been redesigned around a fiction. Refuse to judge instead. +if [ "$BASELINE_OK" -ne 1 ]; then + note "SKIPPING phase D — the box could not reach the internet BEFORE any hardening was applied" + inf "a design verdict measured on a broken baseline is worthless; fix the baseline and re-run" + aud "B1/B3/B5: NOT MEASURED — baseline egress was already broken (see A1)" +fi +if [ "$BASELINE_OK" -eq 1 ]; then # D1 — dns.mode=none (#15 B3): must kill sibling resolution, must NOT kill egress. # Runs 2 and 3 DISAGREED on the egress half (broken, then intact) — a verdict # that flips is a verdict you cannot design on. Setting dns.mode restarts the # network's dnsmasq, so a probe fired immediately can catch it mid-restart. # Distinguish TRANSIENT (recovers) from BROKEN (still dead after 30s), and say so. -if incus network set claudenet dns.mode=none 2>/dev/null; then +if incus network set claudenet dns.mode=none 2>/tmp/dnsmode.err; then sleep 2 [ -n "$(in_box archive getent hosts peer.incus)" ] \ && { no "dns.mode=none did not stop sibling resolution"; aud "B3 dns.mode=none: does NOT close the leak"; } \ @@ -571,8 +589,8 @@ if incus network set claudenet dns.mode=none 2>/dev/null; then fi fi else - no "incus rejected dns.mode=none on claudenet" - aud "B3 dns.mode=none: REJECTED by incus — #16 needs another mechanism" + no "incus rejected dns.mode=none on claudenet: $(head -1 /tmp/dnsmode.err 2>/dev/null)" + aud "B3 dns.mode=none: REJECTED by incus — $(head -1 /tmp/dnsmode.err 2>/dev/null) — #16 needs another mechanism" fi # D2 — L2 filtering (#15 B5): the box must keep working with it on. @@ -614,6 +632,8 @@ else aud "B1 @internal: rejected ⇒ #16 derives the subnet in setup-host.sh (mask the gateway CIDR)" fi +fi # end of the BASELINE_OK guard around phase D + # =========================================================================== if [ "$KEEP" = 1 ]; then phase "Boxes left up (--keep-boxes)" -- 2.45.2