Two cold mints died in cloud-init with "Temporary failure resolving
deb.debian.org", on a host the doctor had just certified clean. The
cause was not DNS forwarding, not leftover mutations, and not the drill:
claudenet had NO dnsmasq. It never respawned after the SIGKILL that
recovered the wedged daemon in run 6, so boxes got no DHCP lease at all
— no address, no gateway, no DNS.
Incus does not surface this. The bridge is up, the config is pristine,
'incus network show' says status: Created. Only the process table knows.
So the doctor now asks the process table, and --fix restarts incus to
respawn it.
An hour of hunting and two dead mints went into learning this. It is a
five-second check.
RUNS.md gains trap 11: a network incus calls "Created" may have nothing
serving it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two cold mints in a row died with cloud-init 'status: error' on a host
the doctor had just certified clean — so the earlier "leftover mutations
poisoned the network" theory is dead, and the DNS failure is
reproducible rather than transient.
'claudebox new' printed four hundred dots and the word "error", leaving
the user with nothing to act on: the reason was in the box's own log and
nobody was told the log existed. It now prints cloud-init's status, the
fetch/resolve errors from the box's log, and how to inspect the box —
which is left running, because a box that failed to build is evidence,
not garbage. It also names the usual culprit: the host's resolver.
doctor.sh gains the diagnosis that keeps being done by hand:
· what the HOST resolves through, and whether that is a CGNAT/Tailscale
resolver the boxes inherit (issue #33);
· whether claudenet's resolver is pinned;
· and inside a box, the question that settles it — DNS is broken, but
can it still reach 1.1.1.1 BY ADDRESS? If yes, egress is fine and the
fault is purely the inherited forwarder.
· --pin-dns applies the #33 fix (raw.dnsmasq: no-resolv + public
servers) so the hypothesis can be TESTED rather than argued.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 8's cold mint failed with 'cloud-init status: error' — the box could
not resolve deb.debian.org, or claude.ai, or anything. The cause was not
in that run at all: run 7's phase D set dns.mode=none on claudenet, the
run ended before reverting it, and every box minted afterwards came up
with no DNS.
This is the worst failure mode the drill has: a poisoned host does not
fail the next run honestly, it produces confident wrong answers. It is
how a false design veto against #16 got posted, and it wasted a cold
mint plus an hour of diagnosis that had nothing to do with the code
under test.
Three defences:
· the phase-D revert is armed with a trap BEFORE the first mutation,
so it fires on any exit, Ctrl-C included;
· the revert is VERIFIED rather than fired into /dev/null, so a failed
unset can no longer masquerade as a successful one;
· the drill refuses to start on a host still carrying the mutations.
And drill/doctor.sh answers the question that kept being answered by
hand: what state is this host actually in? Network, profile, ACL,
leftover boxes, and whether a box can still resolve DNS — with --fix to
revert the leftovers.
RUNS.md gains trap 10.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 7's box had no network — a clone/source IP collision had taken it
down before a single hardening change was applied. Phase D measured
anyway and reported "L2 filtering BREAKS the box — design veto". It does
not. The box was already broken, and #16 would have been redesigned
around a fiction.
Phase D is now gated on baseline egress passing, and says loudly that it
is skipping rather than quietly producing a verdict. The dns.mode
rejection also stops swallowing incus's error message — the message IS
the finding.
RUNS.md gains trap 9: check that the thing you are measuring WITH still
works before you trust what it tells you. Same failure as the B3 flip,
different costume.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reset truncated /etc/machine-id and called 'incus restart'. But
systemd needs a VALID machine-id to shut down cleanly — so the graceful
stop hung, incus timed out ("Failed shutting down instance, status is
Running: context deadline exceeded"), and the reboot never happened. The
clone kept its source's machine-id, hence its DUID, hence its DHCP
lease: two boxes on one address, which is the exact bug the reset exists
to prevent. Worse, the duplicate address then broke the box's networking
outright ("box cannot reach the internet"), and poisoned the isolation
run downstream.
Write a fresh VALID id with systemd-machine-id-setup instead of emptying
the file — in a VM it derives from the DMI product UUID, which Incus
makes unique per instance. Then restart with a real timeout and a forced
fallback: a clone that keeps its source's lease is worse than an unclean
stop of a box that booted thirty seconds ago.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A3, the one probe the whole audit exists for, has never fired in six
runs. It was never the network: the profile names the DEVICE eth0, but
inside a VM guest predictable naming renames it enp5s0, so every address
lookup — first the '(eth0)' CSV match, then 'ip addr show dev eth0' —
was hunting an interface that does not exist. I fixed that symptom twice
without ever questioning the assumption underneath it.
Read the address from inside the box and select by SUBNET (10.87.x, what
claudenet hands out) rather than by interface name. docker0's 172.17.x
is the decoy; the NIC's name is the guest's business, not ours.
A3 also gains a guard it should have had from the start: if the peer and
the source hold the SAME address, refuse to probe. That is not
hypothetical — clones were inheriting their source's machine-id, hence
its DHCP lease, hence its address, so 'archive → peer' was archive
probing itself and would have reported a cheerful 'reachable' as an
isolation failure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two boxes cloned from one snapshot came up holding the SAME IP address —
10.87.0.128, on the same dnsmasq lease, matching to the second on the
lease timer. Incus does regenerate the MAC (they differed), but
/etc/machine-id rides along inside the disk image, and systemd derives
its DHCP client identifier from it. Same client-id, same lease.
This breaks the workflow claudebox exists for: log in once, snapshot,
clone forever. Every clone of a snapshot is, to the network, the same
machine as its source and as its siblings.
Truncating /etc/machine-id makes systemd mint a fresh one at next boot,
so the clone path now resets it and restarts before handing the box
over. Costs one reboot; the alternative is colliding boxes.
Found by the drill on a live host (#26) — and it also explains why the
sibling-isolation probe (#15 A3) could never work: it was aiming box A
at box A's own address.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 6 stalled in setup-host.sh, which should be seconds on a host that
already has incus. The ordering was wrong: cleanup ran AFTER setup, so
setup-host reconfigured claudenet's ACLs while an aborted run's boxes
were still attached to that network — 'incus network set' then has to
push the change onto every live NIC. The same aborted run also left the
D-phase mutations (dns.mode=none, NIC filtering) in place, so setup was
converging against a moving target.
Boxes are now deleted and the mutations reverted first, so setup-host is
the no-op it should be. It is also bounded (5 min) and, on timeout,
prints the three things worth checking instead of hanging: instances
still on the network, the firewall unit, the incus daemon. Every cleanup
call gets its own timeout, so a wedged instance cannot stall the run
before the drill has printed a single line.
RUNS.md gains this as trap 8.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five runs of hard-won knowledge — which probes are answered, which bugs
the drill found in claudebox, and the seven traps the drill itself fell
into — has been living in PR bodies and chat, where the next person
debugging this cannot find it.
RUNS.md carries: the audit scoreboard (A3 still unanswered, and why that
matters), the claudebox findings, a traps section to read BEFORE adding
a probe, stall diagnostics, how to run a single probe by hand instead of
paying for a full run, and the B3 flip-flop as a standing lesson about
verdicts drawn from one observation of a system with restart semantics.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 5 looked stuck for minutes right after the "Host setup" header. The
setup stretch is the longest and most failure-prone part of the run, and
it was also the only part that printed nothing at all — two ways to
block, both invisible:
· 'sudo apt-get install -y -qq incus >/dev/null 2>&1' swallows a sudo
password prompt, so a stalled sudo is indistinguishable from a hang.
Sudo is now pre-authorized up front, out loud.
· apt's lock is routinely held by apt-daily / unattended-upgrades on a
cloud image, and apt waits for it in complete silence. It now gets
DPkg::Lock::Timeout=300 plus an outer timeout, announces that it may
be waiting on the lock, and on failure prints the two commands that
identify the holder — instead of hanging forever.
Also: skip the apt call entirely when incus is already installed (every
re-run on the same host), and narrate each setup sub-step, so a stall is
locatable rather than a blank terminal.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The listener was never load-bearing. curl's exit code already separates
the two cases the audit cares about: 7 (connection refused) means the
packet ARRIVED and a closed port answered with a RST — reachable, not
isolated; 28 (timed out) means it was dropped in flight — isolated. A
closed port is a perfectly good target: it answers, or it doesn't.
So all three listeners go, and with them the whole class of bug that
wedged runs 1 and 4: a backgrounded process inside an 'incus exec'
session holds that session open no matter what you redirect, and the
host-side one leaked a python on every run. The drill now starts no
background process anywhere.
C4 gains ping as a second opinion — tcp dropped but icmp replying is
'partial isolation', which is a finding, not a pass. C2 drops its
gateway listener too: the host already listens there (dnsmasq on :53 by
design), so probing an unserved port and reading refused-vs-dropped is
both safer and more honest about what the firewall must prevent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 4 hung 10+ minutes at C4. The cause is a shell trap, not a network
one: $( ) waits for stdout to CLOSE, not for the command to exit, so a
grandchild inheriting the exec session's stdout holds the substitution
open forever — and 'timeout' does not save you, because it kills the
wrapper, not the process holding the pipe. 'claudebox exec' made it
likelier still: it becomes 'sudo -u claude -i', a login zsh with
oh-my-zsh, which is a lot of machinery to keep a fd alive.
Every in-box read now goes through in_box()/box_curl(): incus exec
directly, stdin pinned to /dev/null, output landed in a file rather than
a pipe, and 'timeout -k' to hard-kill. box_curl returns the curl exit
code as its finding — for A3 the exit code IS the answer (0 connected,
7 refused-so-it-arrived, 28 dropped). C4 also prints what it is probing
before it probes, so a stall is locatable instead of silent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A3 has now gone unprobed for three runs, and the cause was in eth0_ip:
'incus list' name filters are not regexes, so the anchored "^peer$" form
matched nothing — while run 3's own DNS note showed peer resolving to
10.87.0.210, proving the address was there all along. Read it from
inside the box with 'ip -4 -o addr show dev eth0' instead: unambiguous,
no CSV quoting, no filter semantics to get wrong.
Runs 2 and 3 also DISAGREED on B3's egress half — broken, then intact.
Setting dns.mode restarts the network's dnsmasq, so a probe fired
immediately can catch it mid-restart. The probe now distinguishes a
transient outage (recovers within 30s) from a real break (does not), so
#16 gets a verdict it can design on rather than a coin flip.
The docker check now probes the daemon with sudo (a fresh exec session
may lack the docker group) and actually pulls and runs a container,
which is the thing ipv4_filtering could plausibly break.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 2's five cascade FAILs traced to one line: 'incus delete -f a b c'
aborts at the first missing name, so an interrupted run's boxes survived
into the next run — stale 'archive' broke the rename, stale 'peer' broke
the clone (UNIQUE constraint), and the un-renamed 'drill' survived
teardown. Deletion is now one name at a time, and teardown sweeps every
name the drill can have left, whatever branch a partial run took.
The claude-in-box diagnostics also read the wrong line: the escape hatch
prints its 'claudebox: incus exec …' announce before the output, and
'head -1' captured the announce. Diagnostics now filter it, and probe
whether the binary itself runs by full path. Docker's D-phase note gains
a dockerd systemctl readout.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'claudebox exec box -- claude --version' — the exact invocation the help
promises — failed on the first live drill: the binary lands in
~/.local/bin, but cloud-init only exported PATH in .bashrc/.zshrc, which
the non-interactive shell behind exec never reads (the box's login shell
is zsh, so even sudo -i reads neither). A symlink in /usr/local/bin is
on every shell's PATH, interactive or not. Found by the drill (#15).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first live run (issue #15) hit all three: (1) 'set -o pipefail'
turned every 'refusal 2>&1 | grep -q' check into a false FAIL — the
refusals exit 1/2 by design, and 'grep -q' SIGPIPEs the left side on
early match; the pipeline verdict must be grep's alone, so pipefail is
gone with a comment explaining why. (2) The in-box background listener
inherited the exec pty, so 'incus exec' waited on it forever — the run
hung at C7 for 10+ minutes; the listener is now a helper with </dev/null
on the child and a timeout on the client. (3) eth0_ip raced DHCP — the
agent answers before the address exists — so the sibling probe never
ran; it now retries for 30s.
Also: the claude-in-box check now distinguishes 'installed but not on
exec's PATH' (a repo bug — the help promises that exact invocation)
from 'not installed', and dumps cloud-init status when it is the latter.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The drill covered the CLI surface and basic egress but never probed the
one claim everything hinges on: it only ever had one box running, so
sibling isolation — #12's central, never-reproduced finding — went
untested. Phase C now clones a peer and probes box-to-box reachability
(with a listener, so "refused" cannot masquerade as "dropped"), DNS
enumeration, IPv6-off, and host-to-box inbound. A new phase D applies
the exact changes #16 proposes (dns.mode=none, NIC filtering, @internal)
and watches what breaks — a FAIL there is a design veto caught before
the code is written. The run ends with an answers block to paste into
issue #15.
Also: defaults now install heavy-duty/claudebox@main (the old default
pointed at a fork branch that stopped moving when #13 merged), the
version check compares against the installed VERSION file instead of a
hardcoded 0.3.0, and the blind sleep after start became an agent poll.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A stock Debian 13 cloud image ships neither, so the no-UFW firewall path
died on its first nft call — on exactly the kind of fresh cloud host the
tool targets. Found by the drill; the drill's manual workaround becomes a
regression tripwire.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The repo has no tests and no CI, and everything in #9/#10/#13 was verified
against a stubbed incus. The interesting failures aren't in the bash — they're
in what Incus actually does, which is exactly what a stub gets wrong. So the
drill runs the real thing on a host you can format.
A. Incus semantics — the assumptions claudebox is built on, probed directly:
the user.claudebox=1 tag read (now on the path of every box command in
#13), the list filter, the CSV shape, the state string, whether rename
really refuses a running instance, the snapshot-list field order.
B. The claudebox surface — mint, list, info, snapshot, clone-from-snapshot-of-
a-renamed-box, rename, the escape hatch, the rm guard, the CLI contract —
and the boundary: an instance claudebox did NOT mint has down/rm/hatch
aimed at it, must refuse all three, and must still be standing after.
C. Isolation — from inside a real box: public egress works, box → host is
blocked, box → RFC1918 is dropped.
Destructive by design, and it says so before it starts: it asks for consent
(--yes to skip), and refuses outright with no TTY. Without /dev/kvm it reports
that the VM trust boundary was NOT validated rather than passing on a container.
Every incus verb is a candidate feature request, and wrapping them one at a
time grows a worse incus. This lands the rule instead: claudebox owns a command
when it must enforce an invariant incus cannot see — the user.claudebox=1
boundary, the isolation stack, or the creds-free snapshot workflow. Everything
else is incus's job, and now has a door.
- CMDS table: one row per command, carrying synopsis, preconditions, summary,
action and success message. Dispatch AND help are rendered from it, so the
help can no longer describe a command that doesn't exist — the drift that
produced #8 is now impossible, not merely fixed.
- rename, via a table row: it needs the box stopped (incus won't rename a
running instance), so it says so instead of leaking an incus error.
- `claudebox incus <box> -- <args...>`: the escape hatch. Box resolved and
tag-checked, rest passed to incus verbatim, {} substituted, command echoed.
Warns when it can move a box off the isolation stack.
- The boundary is now ENFORCED, not assumed: every box-taking command resolves
through the user.claudebox=1 tag, so claudebox will not stop, rename or delete
an instance it didn't mint.
- The rule, written into docs/claudebox-design.md.
Closes#11
The help was `sed -n '2,9p' "$0"` — the script scraping its own header comment
by line number — with no sections, no per-command help, no `-h`, and no
`--version`. Worse, parts of the interface it described weren't real: `--force`
was parsed and never read, so `claudebox rm work` force-deleted a running box
with no confirmation while advertising a flag that implied the bare form was
the careful one.
- usage()/help_cmd(): NAME/USAGE/COMMANDS/OPTIONS/EXAMPLES/EXIT STATUS, plus a
page per command. `claudebox help <cmd>` and `claudebox <cmd> --help`.
- `-h`/`--help` anywhere; `--version`/`-V` (VERSION file).
- Unknown command → stderr, exit 2, and a did-you-mean by edit distance.
- Unknown options are rejected, not swallowed as positionals: `snapshot work
--labl x` no longer names the snapshot "--labl". `--` still passes everything
through for exec.
- `--force` is real: `rm` confirms on a TTY, refuses without one unless forced.
BEHAVIOR CHANGE — a scripted `claudebox rm` now needs `--force`.
- Exit 2 for usage errors, 1 for runtime failures; accurate per-command usage
strings (`exec` no longer omits `-- <cmd...>`).
- shellcheck is clean: SC2034 (dead `force`) and SC2015 both gone.
Closes#8
`status` was the only lister: misnamed, silently ignoring the `<box>` its own
help advertised, and unable to show a snapshot label — the one thing the
README's snapshot→clone flow requires you to know. Typing the obvious
`claudebox list` printed the help and exited 0.
- `list` — a table of your boxes (name, state, type, snapshot count).
- `info <box>` — detail, the snapshot labels, and the `--from` line to clone one.
- `list <box>` is a wrong guess we can answer: it points at `info`.
- `--json` passthrough; `--remote` honored; a real error on an unknown box.
- `status` stays as a deprecated alias for `list` so muscle memory keeps working.
Reads are filtered by the `user.claudebox=1` tag, so an Incus instance
claudebox didn't mint is never reported on or touched.
Closes#7
The README's auth flow tells operators to run `gh auth login`, but the
cloud-init package set never installed gh — a fresh box couldn't follow
its own instructions. Debian 13 ships gh natively, so a plain packages
entry suffices.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Instance names now equal the box name (claudebox new --name work -> 'work',
not 'claude-work'). To still tell claudebox's instances apart from any other
incus instances, tag each on creation with user.claudebox=1 and filter status
+ teardown by that tag instead of a name prefix. Clones (new --from) inherit
the tag.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The fork claude-hdb/claudebox is private, so an anonymous 'curl | bash' against
it 401s. Point the installer default and the README one-liner at the public
canonical repo so the install flow works without auth.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A CLI that mints trust-less, network-isolated Incus VMs with Claude Code
installed. Boxes are strictly creds-free — the operator logs into Claude
interactively inside; authenticated state is reused via snapshots. The tool
knows nothing about projects; a repo ships an optional agent-facing .claudebox/
runbook that Claude reads.
- bin/claudebox: new/shell/exec/snapshot/restore/down/start/rm/status; creds-free
'new' (fresh launch or clone via --from <src>[/<snap>]).
- cloud-init: global ~/.claude/CLAUDE.md self-describing the box + .claudebox/ runbook.
- install.sh: curl-pipe-bash installer.
- host/: Incus isolation stack (claudenet + claude-isolate ACL + claude-dev
profile + firewall).
- docs/: design + .claudebox/ convention.
Initial canonical import (prototyped separately; re-homed onto the heavy-duty fork).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
