Wall 2 is solved: 'EFI stub: Failed to decompress kernel' was a CORRUPT
IMAGE — the --purge-storage re-download produced a bad blob. Deleting
the cached image and re-pulling booted the box immediately. The storage
pool was innocent (1.29GiB used of 30GiB).
Both walls cost hours to diagnose by hand. The next box to hit them
should be told the answer, not the symptom — so wait_agent now reads
the console log and names the failure:
· 'Failed to decompress kernel' -> the cached image is corrupt; here
is the incus image delete command to re-pull it
· 'bad shim signature' -> Secure Boot rejected the kernel (shouldn't
happen now; box mints with security.secureboot=false)
· GRUB/firmware menu -> never booted; re-pull or pin BOX_IMAGE
The 0.5.0 env-var rename (CLAUDEBOX_* -> BOX_*) created a silent trap:
a STALE local drill.sh passes CLAUDEBOX_REPO/REF, today's install.sh
reads BOX_REPO/REF, the vars are ignored, main gets installed — and the
drill runs to a green summary having drilled the wrong tree entirely.
The same class already cost an hour once via a lagged CDN tarball.
install.sh now records what it installed (/INSTALLED_FROM), and the
drill ASSERTS it matches the requested repo@ref before touching the
host — failing loudly, and naming the stale-checkout cause, instead of
drilling a lie.
Two things:
1. The host scripts are first-class verbs now — 'box setup-host',
'box teardown-host [--purge-incus]', 'box migrate-host --box <n>'.
Nobody should have to run ~/.local/share/box/host/<script>.sh; that
read like an external script and exposed an install path. Each verb
execs the installed script with its flags passed through (same
pattern as 'box doctor'). README, doctor hints, and the uninstall
section point at the verbs now.
2. The repo-runbook convention is '.box/', not '.claudebox/'. Renamed
across docs and the templates' agent briefing; the briefing tells
the agent to read either, and box-recipe.md notes the rename, so
repos still shipping '.claudebox/' keep working through the
transition. (Consuming repos rename their own folder — tracked
separately.)
Also widened the help command column for the longer verb names, and
fixed one sed-casualty where a broad '.claudebox'→'.box' pass had
turned the README's legacy user.claudebox tag into user.box.
The 0.4.0 rename left surface leftovers the user hit through the
CLAUDEBOX_* env vars. Sweep them, drawing a clean line:
box = everything the user touches — env vars (BOX_REPO/REF/HOME/
BIN), installer messages (box-install:), the install tree
(~/.local/share/box, with the installer sweeping the old
~/.local/share/claudebox on upgrade), tool prose, and the
docs (docs/box-{design,recipe}.md).
claudebox = the GitHub repo name (URLs, the claudebox-<ref> tarball
dir, issue refs), the legacy user.claudebox=1 tag, the
old-stack cleanup code (claudenet/claude-dev/claude-isolate/
claudebox-firewall), and the .claudebox/ runbook convention
— a deliberate v1 hold, since renaming it breaks consuming
repos.
Renamed the two doc files and their links; updated drill.sh/doctor.sh
paths and BOX_REPO/BOX_REF; fixed the claude template's in-box briefing
to say 'box'. RUNS.md left as-is (append-only history). No behavior
change beyond the install-dir move, which the installer migrates.
The console log finally showed the real error behind the GRUB-menu hang:
error: prohibited by secure boot policy.
error: bad shim signature.
Failed to boot both default and fallback entries.
Incus defaults VMs to security.secureboot=true. A Debian cloud image
whose shim is signed with a key this host's OVMF does not trust then
fails signature verification, the kernel never loads, and the VM sits
at the GRUB menu forever — which is exactly the 5-min agent timeout on
every box. It worked in runs 11–15 on the old cached image and broke
the moment --purge-storage re-downloaded a build with a different shim.
security.secureboot=false on VM launch (cmd_new, and the drill's legacy
box). Secure Boot inside a throwaway box is not part of its threat
model — the VM boundary is — and off, it boots reliably across image
rebuilds. Container mode has no firmware and is unaffected.
Bare repro that isolated it: 'incus launch images:debian/13/cloud x --vm'
alone reproduced the hang, proving it was never the 0.5.0 code.
The first sanitize stripped only the ESC byte, leaving visible '[1m[37m'
halves as noise. Strip full CSI/escape sequences first (while ESC is
present), then residual control bytes — clean text.
And read the log: a box sitting at 'GNU GRUB / Press enter to boot /
UEFI Firmware Settings' never booted — that is the IMAGE, not box. Say
so, and point at re-pulling the image or pinning BOX_IMAGE. Surfaced
running the 0.5.0 drill after --purge-storage re-downloaded a
debian/13/cloud build that hangs at the GRUB menu on the serial console.
Two bugs surfaced running the 0.5.0 drill with --purge-storage on a
cold btrfs pool:
1. wait_agent dumped the VM's RAW console log on timeout — full of
terminal escape sequences and a firmware menu — which scrambled the
operator's terminal, and doubly so when it landed in a log they were
tail -f'ing ('*Debian GNU/Linux', 'ESC to return previous menu',
^[^[^[). Now: capture to /tmp/box-console-<n>.log, strip everything
but printable ASCII + tab/newline, print only a short sanitized tail.
Nothing raw reaches a terminal.
2. A failed mint left its stuck VM running, starving the NEXT box's boot
and cascading more 5-min timeouts (tpl failed → codex failed). Every
mint-failure branch now tears the box down before continuing.
The 'no inbound path' contract is one notch too absolute for the tool's
own flagship workflow: coding in a box, a dev server on :3000, and no
way to open it in your browser. expose is the deliberate un-screwing.
- Loopback only, always: the host side listens on 127.0.0.1, never
0.0.0.0 — no other machine can reach the box; only this host gets a
door. No flag widens it (that is the escape hatch's job).
- A verb, per-port, reversible, visible: each exposure is a named proxy
device (expose-<port>); --list and box info show it, --remove undoes
it. A box with a hole says so.
- Mechanism (VMs): an Incus proxy device forwards host loopback to the
box's ip:port, plus a SCOPED ingress ACL allow (this box's ip + this
port only) so the forkproxy's connection survives the default drop —
the drill decides whether that allow is needed or redundant. The
in-box server must listen on 0.0.0.0 (a VM's forwarder reaches it over
the network); inside an isolated box that is safe.
Drill phase E: start a detached listener in a box, expose it, prove the
HOST loopback reaches it, prove a NON-exposed port is still dropped (A7
survives), prove --remove shuts the door.
Closes#55
The 0.4.0 transition is zero-ceremony (install + setup-host = a
dual-stack host). This script is the two things that path does not do:
- --box <name> / --all-boxes: re-home a pre-rename box onto the new
stack, PRESERVING its authed state (no re-login). Order is
load-bearing — tag first (additive, reversible), profile-assign last
(the network move), then verify the box actually resolves + reaches
the internet on its 10.88 leg before declaring it migrated. A box
never ends up tagless or profileless.
- --retire-legacy: remove claudenet/claude-dev/claude-isolate and the
old firewall unit + nft tables, but REFUSE while any legacy box still
references them; assert their absence rather than trust exit codes.
Drill phase M builds a faithful legacy stack (claudenet on 10.87, a
claude-dev profile pinned to it, a box on the old tag), then proves:
retire refuses with a legacy box present, re-home flips the tag +
reassigns box-net + lands a 10.88 address + resolves, and retire then
succeeds and leaves nothing. The transition is measured, not asserted.
Closes#53
Two coding-CLI templates mirroring claude's shape: a box.env + verbatim
cloud-init, inheriting the box-net placement contract structurally, no
new design.
- codex: OpenAI Codex CLI via 'npm i -g @openai/codex' (the SCOPED
package; needs Node 22), symlinked onto the non-interactive exec PATH
via 'npm prefix -g' — the same PATH fix the claude template needed.
- grok: xAI Grok Build via the official 'curl x.ai/cli/install.sh',
run AS the grok user (the installer drops into $HOME); the binary is
found and symlinked to /usr/local/bin.
Install commands verified upstream at implementation time, per the
issue's rule (npmjs.com/package/@openai/codex, x.ai/cli). Each gets an
AGENTS.md-style context file telling the agent it lives in a
disposable, isolated, creds-free box.
Drill: templates listing now expects four; a compact per-template smoke
(mint, '<cli> --version' via box exec, remove) validates each payload
installs and lands on the exec PATH — the generic mechanic is already
proven by blank+claude and not repeated.
Closes#54
Operator watched /tmp/new.log through a claude mint and saw not one
message: cloud-init's progress dots are block-buffered the moment
stdout is not a tty, so a redirected mint shows nothing for the whole
install and then one burst — which reads exactly like a hang, on the
very night three real hangs happened.
PYTHONUNBUFFERED=1 on the cloud-init wait makes the dots arrive as
dots; box new also prints how to watch the box's own full narration
(incus exec <box> -- tail -f /var/log/cloud-init-output.log); and the
drill's logs are named for the box being minted (/tmp/mint-drill.log),
not for the verb that mints it.
Run 14: the blank mint — the FIRST VM launch on the fresh btrfs pool,
which unpacks the image into a pool volume and takes the coldest boot —
died at wait_agent's 3-minute window ('Processes: -1' well past it),
while the identical claude mint sixty seconds later booted in the warm
path and passed in 96s. The window was tuned on a dir pool with a
cached, unpacked image.
150×2s now, and on failure box new prints the VM's console log tail
before dying — this run's evidence was torn down with the box before
anyone could read it.
All four mints (blank, claude, clone, peer) now run through mint_box:
box new's narration lands in the log as before, the drill prints where
to tail it, and a dot every 5s on the drill's own terminal proves the
run is alive. A silent multi-minute mint is indistinguishable from a
wedge, and that ambiguity has cost whole evenings — the operator said
so, verbatim.
Run 14's second catch: the blank mint's cloud-init finished, printed
'status: done' — and 'box new' hung for 15+ minutes on an exec session
that never closed ('incus operation list' showed it still RUNNING).
With a TTY on stdin (the drill redirects only stdout/stderr), incus
exec goes interactive, and the session can wedge open after the remote
command has exited. Same disease as drill trap 2 and doctor trap 13;
the CLI's own execs never got the cure.
Every non-interactive exec now pins stdin: wait_agent's probe, the
cloud-init wait, both failure-path reads, and the clone identity
reset. shell/exec/tmux keep the terminal — owning it is their job.
Also: wipe.sh keeps cached images on a plain wipe. An image is
upstream's artifact, content-addressed by fingerprint — deleting it
buys zero cleanliness and costs the next mint a full re-download. It
goes only with --purge-storage, where the pool it lives in goes too
(and it must go first: images block pool deletion).
The profile rename changed the file's name, header and limits but not
the device's 'network:' field; the claudenet→boxnet sed covered
host/*.sh only. On a wiped host (no claudenet to silently latch onto)
'incus profile edit box-net' refused the YAML and setup died — run 14's
first catch, before a single box was minted. The sweep this fix rode in
on found exactly one other stale reference, in the same file's comment.
Follow-up to the rename, per operator direction — the divergence is
reversed and the cut is complete:
- Host stack: boxnet (10.88.0.0/24 — a pre-rename host may still carry
claudenet on 10.87, two bridges must not claim one subnet),
box-isolate, nft tables 'inet box'/'bridge box', box-firewall.{sh,
service}. teardown-host now strips BOTH name generations, so one
script uninstalls a host of any age.
- Default template is blank: 'box new --name x' mints bare Debian;
the claude box is '--template claude'. The login hint follows the
EFFECTIVE template read off the instance, so clones of claude boxes
still get it and blank boxes are not told to run a binary they lack.
- The drill validates templates: listing, unknown-template refusal,
the allowlist rejecting BOX_NETWORK by name, and a full blank mint —
default resolves to blank, metadata stamped, box-net placement, exec
lands in 'dev', no claude binary, and isolation parity (egress +
pinned DNS) on the same contract as every template.
- drill/wipe.sh: scorched earth for drill hosts. Both tag generations,
every drill-named instance, networks/ACLs/profiles/firewall of both
generations, cached images, and (--purge-storage) the default pool.
Ends by asserting the ABSENCE of every artifact rather than trusting
the removals' exit codes.
The tool underneath was already generic: a thin, honest wrapper over
Incus. What was Claude-specific was welded on — one image, one profile,
one cloud-init file, one hardcoded 'sudo -u claude'. The weld is now a
template.
The mechanic: 'box new' stamps the template's identity onto the
instance (user.box=1, user.box.template, user.box.user); shell/exec/
tmux read the user back off the instance, and 'incus copy' carries
user.* keys (audit B2), so a clone knows what it is without consulting
the template. Templates are box.env (parsed against a strict allowlist,
never sourced — no key for a network exists, on purpose) plus a
verbatim cloud-init. Every template launches with the shared box-net
profile: the isolated NIC and root disk, nothing template-controlled —
resources land per-instance from box.env, overridable via BOX_CPU/
BOX_MEMORY/BOX_DISK (which is also how the drill shrinks boxes on a
small host now that profile edits can't).
The three open calls, taken as recommended: clean cut at 0.4.0 (no
claudebox shim; the installer retires the old symlink); default
template = claude (muscle memory survives); repo stays heavy-duty/
claudebox, binary is box.
Compat is the tag, not the name: resolve_box and list honor the legacy
user.claudebox=1 forever, and the legacy tag maps to the claude user —
a pre-rename box lists, shells, clones, unchanged.
Deliberate divergence from #17's table: the host-stack resource names
(claudenet, claude-isolate, nft tables, claudebox-firewall.*) are NOT
renamed — they are host-internal, invisible to users, and renaming
them breaks every provisioned host for zero user-visible gain.
claude-dev is no longer created; setup-host creates box-net, teardown
removes both.
Closes#17
'incus admin init --minimal' picks the 'dir' backend, which has no
copy-on-write: every snapshot and clone is a full copy of the box's
root — several GB and minutes apiece once Docker, Node and Claude Code
are installed. Measured live: one clone took minutes, and a single
drill run pays that three times (snapshot + two clones). The workflow
the tool exists for — log in once, snapshot, clone forever — stops
being attractive at exactly that price.
setup-host.sh now bootstraps via a preseed that mirrors what --minimal
creates (default pool, incusbr0, default profile with root + eth0) with
only the driver deliberate: btrfs on a loop device (CoW — clones share
blocks and diverge on write), installing btrfs-progs if absent, falling
back to dir with a warning if btrfs cannot be had. Existing hosts are
untouched: the block is skipped whenever a 'default' pool exists.
Closes#29
Every fault the drill's doctor diagnoses is a user's fault first: a
wedged Incus daemon, a dnsmasq that silently isn't serving, a VPN
resolver boxes inherit, isolation claimed by config but off in the
kernel — each has killed a cold mint or weakened a boundary, with a
cloud-init error that names none of them. The CLI half-admitted it:
cmd_new's failure path hand-pointed at issue #33, doing one special
case of a doctor's job inline.
The verb delegates to the installed drill/doctor.sh (the tree ships
whole) — one hardened script, two audiences. Its flags pass through;
the verdict now reads 'fit to mint boxes (and to drill)'; the mint-
failure hint ends with 'claudebox doctor' instead of the hand-rolled
diagnosis.
Closes#46
'shell' is a child of the exec connection: drop the terminal and a long
Claude run inside it is SIGHUP'd — which has already cost one run. tmux
is installed in every box but relying on remembering to start it is how
that run was lost.
New verb, not a changed default: 'tmux new-session -A -s <session>'
attaches when the session exists and creates it when it doesn't, so
starting work and reattaching after a disconnect are the same command.
The session name (default: main) gives parallel streams in one box.
'shell' stays bare on purpose — two verbs, two contracts — so tmux's
costs (nesting, scrollback semantics, exit-vs-detach) are only paid
when asked for.
Closes#6
The log stopped at run 10's headline (boxes reach each other). Three
runs later the arc is closed and RUNS.md should say so:
- run 11 (45/2): A3 re-answered with the fixed probe — BLOCKED,
security.port_isolation; both FAILs were the drill's own empty-host
assumption
- run 12 (46/0): first zero-failure run
- run 13 (47/0): full teardown → rebuild → drill from a bare host,
pinned resolver built from scratch on a Tailscale host
Answer table updated (A3/A4 show fixed + re-measured state), headline
flipped to match, traps 13–15 added (TTY stdin under timeout, the
gateway-ping false alarm, demanding an empty host), run history gains
the three rows and a closing note.
The Isolation section predated the #16 arc: it described the ACL and the
host firewall, and none of what that audit shipped — port-isolated NICs
(the actual sibling boundary), dns.mode=none, the pinned resolver. It
also never mentioned that the contract is tested at all.
State the contract layer by layer as it exists on main, and add a
'Measured, not claimed' section: the drill and doctor, what each does,
which one mutates the host, and the standing score — 47/47 from a bare
host (run 13, full teardown → rebuild → drill).
The bridge's dnsmasq forwarded to whatever sat in the host's
/etc/resolv.conf at that moment. On a Tailscale host that is MagicDNS:
box DNS flapped with the tailnet (killing cold mints), and tailnet peer
names and split-DNS zones resolved from inside a box — name-level
reconnaissance of a private network, the same class as the sibling
enumeration dns.mode=none already closes.
setup-host.sh now sets raw.dnsmasq to no-resolv + pinned public
upstreams (BOX_DNS overrides the default 1.1.1.1 8.8.8.8), answering
the issue's three open questions from live measurement: raw.dnsmasq is
the lever (no first-class upstream key on the bridge; verified by
doctor --pin-dns followed by a box resolving), upstreams are a setting
with a sane default, and the pin is unconditional.
The doctor's unpinned-state messages now point at setup-host.sh as the
durable fix, keeping --pin-dns as the quick test.
Closes#33
Run 11: 45 passed, 2 failed — and both failures were the same non-fact.
The host had two operator boxes (t1, t2) minted before the run, so
'claudebox list' could not say 'no boxes yet' at either end of the drill.
The empty-host message is only testable on an actually-empty host: skip
it (with a note) when tenants pre-exist, instead of failing it.
The teardown check contradicted its own rm loop: the loop deliberately
removes only the names the drill minted — the multi-tenant discipline —
and then the assertion demanded a globally empty host, flagging that
very restraint as a failure. Assert the drill's own names are gone;
name the survivor when one is.
claudebox-firewall.sh drops everything from a box to the host except
DNS (53) and DHCP (67); ICMP to 10.87.0.1 dies in that trailing drop on
every healthy host. The doctor used exactly that ping as its routing
probe, so it reported 'cannot even reach the gateway' — and a NOT-fit-
to-drill verdict — on a host whose very next line proved DNS working
through that same gateway.
Probe routing the way the contract states it: a box reaches the public
internet. curl to 1.1.1.1 by address, reusing the one probe for the
DNS-failure diagnosis instead of running it twice.
With a TTY on stdin, 'incus exec' goes interactive and puts the terminal
in raw mode. The DNS probes then hang forever: timeout's bare TERM never
lands (no -k escalation), and ^C is forwarded into the box as a keystroke
instead of killing the script. Observed live: doctor hung 15+ minutes at
'Can a box actually resolve DNS?' and survived Ctrl-C; the operator had
to kill the shell.
The drill already learned this exact lesson in #22 (exec_in pins stdin
and uses 'timeout -k'); the doctor's probe section was added later in
#34/#35 and never got the cure. All four probes now pin stdin to
/dev/null and escalate to SIGKILL.
Boxes are isolated. The kernel says so: 'isolated on' on both live bridge
ports. A box gets 100% packet loss pinging its sibling, and its TCP
connect burns the full timeout and dies with "Could not connect to
server" — not "Connection refused", which is what an arriving packet
gets, instantly.
The drill called that a FAIL. curl exit 7 is "failed to connect", and it
covers BOTH a refusal (a RST came back — the packet ARRIVED) and an
unreachable host (nothing came back — it was DROPPED). Opposite
conclusions, one exit code, and the drill mapped 7 → "arrived". So for
two runs after the isolation fix had landed and was working, the drill
reported a working boundary as a broken one.
The words distinguish what the number cannot. box_probe now returns
reachable | refused | dropped by reading the message, ping corroborates,
and every isolation probe (sibling, box→host, RFC1918, host→box) uses it.
RUNS.md gains trap 12. It is the same disease as all eleven before it:
trusting a proxy for the fact instead of the fact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run the doctor on a healthy host and it reported three problems, all of
them its own:
· dns.mode=none was flagged as leftover rehearsal dirt. It is now the
SHIPPED setting — it is what stops a box enumerating its siblings.
Its absence is the fault; its presence was being "fixed" away.
· the nft check used 'sudo -n', which fails without cached credentials,
so it reported the box-to-box rule MISSING on a host where 'sudo nft
list' plainly shows it.
· the kernel's bridge view — the one fact that settles the isolation
question — was skipped with "'bridge' not installed". It is installed;
it lives in /usr/sbin, which is not on a normal user's PATH.
· and the DNS probe looked only for the drill's own box names, so it
said "no box to probe with" while two boxes sat there RUNNING.
A diagnostic that cries wolf is worse than no diagnostic: it costs the
same trust as a real failure and teaches you to ignore it. All four now
report what is actually true.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two bugs, one in each direction.
setup-host's new assertion ran 'nft list table bridge claudebox' without
sudo. nft needs root, so it failed with permission denied and printed
"the box-to-box drop is NOT active" about a rule that was demonstrably
there. A check that cries wolf is worse than no check.
And the deeper one: every check so far has asked the CONFIG whether
boxes are isolated. The config is a claim. Incus can accept
security.port_isolation and the kernel can still leave 'isolated off' on
the tap — and then boxes reach each other while every config in sight
says they cannot. That is precisely the shape of the original bug: the
ACL looked airtight and never saw the traffic.
So the doctor now reads the kernel's own view — 'bridge -d link show'
on claudenet's ports — and reports the isolated flag as the fact it is.
If the profile says true and the kernel says off, we learn that in a
second instead of after another ten-minute drill.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nft bridge-family rule from the previous commit is LIVE on the host
and boxes still reach each other:
table bridge claudebox {
chain forward { ... meta ibrname "claudenet" meta obrname "claudenet" drop }
}
FAIL BOX A REACHES BOX B — sibling isolation does NOT hold [tcp: refused]
So the rule is not wrong about intent, it is wrong about mechanism —
whatever path these frames take, that hook does not stop them. Rather
than reason harder about netfilter (reasoning is what put the hole there
in the first place), use the mechanism Incus provides for exactly this:
security.port_isolation on the bridged NIC, which sets the kernel bridge
port's isolated flag so two isolated ports cannot exchange frames at all.
The nft rule stays as a second layer — it costs nothing — but the
profile flag is what carries the guarantee. doctor.sh checks it, because
the absence of this one is invisible: everything works and boxes can
simply reach each other.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The box-to-box drop shipped, the drill still found boxes reaching each
other, and the rule was simply not on the host. setup-host.sh ended with
'systemctl enable --now claudebox-firewall.service' — but the unit is
RemainAfterExit, so once it has run it stays "active" forever, and
'--now' does nothing to an active unit. Re-running setup-host after
upgrading claudebox therefore installed the new script to
/usr/local/sbin and never executed it. The host silently kept its old
firewall, and the box-to-box hole stayed open through the release that
claimed to close it.
This is worse than the original bug: every future firewall change would
have landed only on hosts that had never run setup-host before.
Restart the unit instead — the script is idempotent by design. Then
ASSERT the rule is live rather than assume it, because the absence of
this particular rule is invisible: everything keeps working and boxes
can simply reach each other. doctor.sh checks it too.
Also: dns.mode=none is now part of the shipped stack, so the drill must
stop treating it as leftover rehearsal dirt and reverting it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
