The 'no inbound path' contract is one notch too absolute for the tool's
own flagship workflow: coding in a box, a dev server on :3000, and no
way to open it in your browser. expose is the deliberate un-screwing.
- Loopback only, always: the host side listens on 127.0.0.1, never
0.0.0.0 — no other machine can reach the box; only this host gets a
door. No flag widens it (that is the escape hatch's job).
- A verb, per-port, reversible, visible: each exposure is a named proxy
device (expose-<port>); --list and box info show it, --remove undoes
it. A box with a hole says so.
- Mechanism (VMs): an Incus proxy device forwards host loopback to the
box's ip:port, plus a SCOPED ingress ACL allow (this box's ip + this
port only) so the forkproxy's connection survives the default drop —
the drill decides whether that allow is needed or redundant. The
in-box server must listen on 0.0.0.0 (a VM's forwarder reaches it over
the network); inside an isolated box that is safe.
Drill phase E: start a detached listener in a box, expose it, prove the
HOST loopback reaches it, prove a NON-exposed port is still dropped (A7
survives), prove --remove shuts the door.
Closes#55
The 0.4.0 transition is zero-ceremony (install + setup-host = a
dual-stack host). This script is the two things that path does not do:
- --box <name> / --all-boxes: re-home a pre-rename box onto the new
stack, PRESERVING its authed state (no re-login). Order is
load-bearing — tag first (additive, reversible), profile-assign last
(the network move), then verify the box actually resolves + reaches
the internet on its 10.88 leg before declaring it migrated. A box
never ends up tagless or profileless.
- --retire-legacy: remove claudenet/claude-dev/claude-isolate and the
old firewall unit + nft tables, but REFUSE while any legacy box still
references them; assert their absence rather than trust exit codes.
Drill phase M builds a faithful legacy stack (claudenet on 10.87, a
claude-dev profile pinned to it, a box on the old tag), then proves:
retire refuses with a legacy box present, re-home flips the tag +
reassigns box-net + lands a 10.88 address + resolves, and retire then
succeeds and leaves nothing. The transition is measured, not asserted.
Closes#53
Two coding-CLI templates mirroring claude's shape: a box.env + verbatim
cloud-init, inheriting the box-net placement contract structurally, no
new design.
- codex: OpenAI Codex CLI via 'npm i -g @openai/codex' (the SCOPED
package; needs Node 22), symlinked onto the non-interactive exec PATH
via 'npm prefix -g' — the same PATH fix the claude template needed.
- grok: xAI Grok Build via the official 'curl x.ai/cli/install.sh',
run AS the grok user (the installer drops into $HOME); the binary is
found and symlinked to /usr/local/bin.
Install commands verified upstream at implementation time, per the
issue's rule (npmjs.com/package/@openai/codex, x.ai/cli). Each gets an
AGENTS.md-style context file telling the agent it lives in a
disposable, isolated, creds-free box.
Drill: templates listing now expects four; a compact per-template smoke
(mint, '<cli> --version' via box exec, remove) validates each payload
installs and lands on the exec PATH — the generic mechanic is already
proven by blank+claude and not repeated.
Closes#54
Operator watched /tmp/new.log through a claude mint and saw not one
message: cloud-init's progress dots are block-buffered the moment
stdout is not a tty, so a redirected mint shows nothing for the whole
install and then one burst — which reads exactly like a hang, on the
very night three real hangs happened.
PYTHONUNBUFFERED=1 on the cloud-init wait makes the dots arrive as
dots; box new also prints how to watch the box's own full narration
(incus exec <box> -- tail -f /var/log/cloud-init-output.log); and the
drill's logs are named for the box being minted (/tmp/mint-drill.log),
not for the verb that mints it.
Run 14: the blank mint — the FIRST VM launch on the fresh btrfs pool,
which unpacks the image into a pool volume and takes the coldest boot —
died at wait_agent's 3-minute window ('Processes: -1' well past it),
while the identical claude mint sixty seconds later booted in the warm
path and passed in 96s. The window was tuned on a dir pool with a
cached, unpacked image.
150×2s now, and on failure box new prints the VM's console log tail
before dying — this run's evidence was torn down with the box before
anyone could read it.
All four mints (blank, claude, clone, peer) now run through mint_box:
box new's narration lands in the log as before, the drill prints where
to tail it, and a dot every 5s on the drill's own terminal proves the
run is alive. A silent multi-minute mint is indistinguishable from a
wedge, and that ambiguity has cost whole evenings — the operator said
so, verbatim.
Run 14's second catch: the blank mint's cloud-init finished, printed
'status: done' — and 'box new' hung for 15+ minutes on an exec session
that never closed ('incus operation list' showed it still RUNNING).
With a TTY on stdin (the drill redirects only stdout/stderr), incus
exec goes interactive, and the session can wedge open after the remote
command has exited. Same disease as drill trap 2 and doctor trap 13;
the CLI's own execs never got the cure.
Every non-interactive exec now pins stdin: wait_agent's probe, the
cloud-init wait, both failure-path reads, and the clone identity
reset. shell/exec/tmux keep the terminal — owning it is their job.
Also: wipe.sh keeps cached images on a plain wipe. An image is
upstream's artifact, content-addressed by fingerprint — deleting it
buys zero cleanliness and costs the next mint a full re-download. It
goes only with --purge-storage, where the pool it lives in goes too
(and it must go first: images block pool deletion).
The profile rename changed the file's name, header and limits but not
the device's 'network:' field; the claudenet→boxnet sed covered
host/*.sh only. On a wiped host (no claudenet to silently latch onto)
'incus profile edit box-net' refused the YAML and setup died — run 14's
first catch, before a single box was minted. The sweep this fix rode in
on found exactly one other stale reference, in the same file's comment.
Follow-up to the rename, per operator direction — the divergence is
reversed and the cut is complete:
- Host stack: boxnet (10.88.0.0/24 — a pre-rename host may still carry
claudenet on 10.87, two bridges must not claim one subnet),
box-isolate, nft tables 'inet box'/'bridge box', box-firewall.{sh,
service}. teardown-host now strips BOTH name generations, so one
script uninstalls a host of any age.
- Default template is blank: 'box new --name x' mints bare Debian;
the claude box is '--template claude'. The login hint follows the
EFFECTIVE template read off the instance, so clones of claude boxes
still get it and blank boxes are not told to run a binary they lack.
- The drill validates templates: listing, unknown-template refusal,
the allowlist rejecting BOX_NETWORK by name, and a full blank mint —
default resolves to blank, metadata stamped, box-net placement, exec
lands in 'dev', no claude binary, and isolation parity (egress +
pinned DNS) on the same contract as every template.
- drill/wipe.sh: scorched earth for drill hosts. Both tag generations,
every drill-named instance, networks/ACLs/profiles/firewall of both
generations, cached images, and (--purge-storage) the default pool.
Ends by asserting the ABSENCE of every artifact rather than trusting
the removals' exit codes.
The tool underneath was already generic: a thin, honest wrapper over
Incus. What was Claude-specific was welded on — one image, one profile,
one cloud-init file, one hardcoded 'sudo -u claude'. The weld is now a
template.
The mechanic: 'box new' stamps the template's identity onto the
instance (user.box=1, user.box.template, user.box.user); shell/exec/
tmux read the user back off the instance, and 'incus copy' carries
user.* keys (audit B2), so a clone knows what it is without consulting
the template. Templates are box.env (parsed against a strict allowlist,
never sourced — no key for a network exists, on purpose) plus a
verbatim cloud-init. Every template launches with the shared box-net
profile: the isolated NIC and root disk, nothing template-controlled —
resources land per-instance from box.env, overridable via BOX_CPU/
BOX_MEMORY/BOX_DISK (which is also how the drill shrinks boxes on a
small host now that profile edits can't).
The three open calls, taken as recommended: clean cut at 0.4.0 (no
claudebox shim; the installer retires the old symlink); default
template = claude (muscle memory survives); repo stays heavy-duty/
claudebox, binary is box.
Compat is the tag, not the name: resolve_box and list honor the legacy
user.claudebox=1 forever, and the legacy tag maps to the claude user —
a pre-rename box lists, shells, clones, unchanged.
Deliberate divergence from #17's table: the host-stack resource names
(claudenet, claude-isolate, nft tables, claudebox-firewall.*) are NOT
renamed — they are host-internal, invisible to users, and renaming
them breaks every provisioned host for zero user-visible gain.
claude-dev is no longer created; setup-host creates box-net, teardown
removes both.
Closes#17
'incus admin init --minimal' picks the 'dir' backend, which has no
copy-on-write: every snapshot and clone is a full copy of the box's
root — several GB and minutes apiece once Docker, Node and Claude Code
are installed. Measured live: one clone took minutes, and a single
drill run pays that three times (snapshot + two clones). The workflow
the tool exists for — log in once, snapshot, clone forever — stops
being attractive at exactly that price.
setup-host.sh now bootstraps via a preseed that mirrors what --minimal
creates (default pool, incusbr0, default profile with root + eth0) with
only the driver deliberate: btrfs on a loop device (CoW — clones share
blocks and diverge on write), installing btrfs-progs if absent, falling
back to dir with a warning if btrfs cannot be had. Existing hosts are
untouched: the block is skipped whenever a 'default' pool exists.
Closes#29
Every fault the drill's doctor diagnoses is a user's fault first: a
wedged Incus daemon, a dnsmasq that silently isn't serving, a VPN
resolver boxes inherit, isolation claimed by config but off in the
kernel — each has killed a cold mint or weakened a boundary, with a
cloud-init error that names none of them. The CLI half-admitted it:
cmd_new's failure path hand-pointed at issue #33, doing one special
case of a doctor's job inline.
The verb delegates to the installed drill/doctor.sh (the tree ships
whole) — one hardened script, two audiences. Its flags pass through;
the verdict now reads 'fit to mint boxes (and to drill)'; the mint-
failure hint ends with 'claudebox doctor' instead of the hand-rolled
diagnosis.
Closes#46
'shell' is a child of the exec connection: drop the terminal and a long
Claude run inside it is SIGHUP'd — which has already cost one run. tmux
is installed in every box but relying on remembering to start it is how
that run was lost.
New verb, not a changed default: 'tmux new-session -A -s <session>'
attaches when the session exists and creates it when it doesn't, so
starting work and reattaching after a disconnect are the same command.
The session name (default: main) gives parallel streams in one box.
'shell' stays bare on purpose — two verbs, two contracts — so tmux's
costs (nesting, scrollback semantics, exit-vs-detach) are only paid
when asked for.
Closes#6
The log stopped at run 10's headline (boxes reach each other). Three
runs later the arc is closed and RUNS.md should say so:
- run 11 (45/2): A3 re-answered with the fixed probe — BLOCKED,
security.port_isolation; both FAILs were the drill's own empty-host
assumption
- run 12 (46/0): first zero-failure run
- run 13 (47/0): full teardown → rebuild → drill from a bare host,
pinned resolver built from scratch on a Tailscale host
Answer table updated (A3/A4 show fixed + re-measured state), headline
flipped to match, traps 13–15 added (TTY stdin under timeout, the
gateway-ping false alarm, demanding an empty host), run history gains
the three rows and a closing note.
The Isolation section predated the #16 arc: it described the ACL and the
host firewall, and none of what that audit shipped — port-isolated NICs
(the actual sibling boundary), dns.mode=none, the pinned resolver. It
also never mentioned that the contract is tested at all.
State the contract layer by layer as it exists on main, and add a
'Measured, not claimed' section: the drill and doctor, what each does,
which one mutates the host, and the standing score — 47/47 from a bare
host (run 13, full teardown → rebuild → drill).
The bridge's dnsmasq forwarded to whatever sat in the host's
/etc/resolv.conf at that moment. On a Tailscale host that is MagicDNS:
box DNS flapped with the tailnet (killing cold mints), and tailnet peer
names and split-DNS zones resolved from inside a box — name-level
reconnaissance of a private network, the same class as the sibling
enumeration dns.mode=none already closes.
setup-host.sh now sets raw.dnsmasq to no-resolv + pinned public
upstreams (BOX_DNS overrides the default 1.1.1.1 8.8.8.8), answering
the issue's three open questions from live measurement: raw.dnsmasq is
the lever (no first-class upstream key on the bridge; verified by
doctor --pin-dns followed by a box resolving), upstreams are a setting
with a sane default, and the pin is unconditional.
The doctor's unpinned-state messages now point at setup-host.sh as the
durable fix, keeping --pin-dns as the quick test.
Closes#33
Run 11: 45 passed, 2 failed — and both failures were the same non-fact.
The host had two operator boxes (t1, t2) minted before the run, so
'claudebox list' could not say 'no boxes yet' at either end of the drill.
The empty-host message is only testable on an actually-empty host: skip
it (with a note) when tenants pre-exist, instead of failing it.
The teardown check contradicted its own rm loop: the loop deliberately
removes only the names the drill minted — the multi-tenant discipline —
and then the assertion demanded a globally empty host, flagging that
very restraint as a failure. Assert the drill's own names are gone;
name the survivor when one is.
claudebox-firewall.sh drops everything from a box to the host except
DNS (53) and DHCP (67); ICMP to 10.87.0.1 dies in that trailing drop on
every healthy host. The doctor used exactly that ping as its routing
probe, so it reported 'cannot even reach the gateway' — and a NOT-fit-
to-drill verdict — on a host whose very next line proved DNS working
through that same gateway.
Probe routing the way the contract states it: a box reaches the public
internet. curl to 1.1.1.1 by address, reusing the one probe for the
DNS-failure diagnosis instead of running it twice.
With a TTY on stdin, 'incus exec' goes interactive and puts the terminal
in raw mode. The DNS probes then hang forever: timeout's bare TERM never
lands (no -k escalation), and ^C is forwarded into the box as a keystroke
instead of killing the script. Observed live: doctor hung 15+ minutes at
'Can a box actually resolve DNS?' and survived Ctrl-C; the operator had
to kill the shell.
The drill already learned this exact lesson in #22 (exec_in pins stdin
and uses 'timeout -k'); the doctor's probe section was added later in
#34/#35 and never got the cure. All four probes now pin stdin to
/dev/null and escalate to SIGKILL.
Boxes are isolated. The kernel says so: 'isolated on' on both live bridge
ports. A box gets 100% packet loss pinging its sibling, and its TCP
connect burns the full timeout and dies with "Could not connect to
server" — not "Connection refused", which is what an arriving packet
gets, instantly.
The drill called that a FAIL. curl exit 7 is "failed to connect", and it
covers BOTH a refusal (a RST came back — the packet ARRIVED) and an
unreachable host (nothing came back — it was DROPPED). Opposite
conclusions, one exit code, and the drill mapped 7 → "arrived". So for
two runs after the isolation fix had landed and was working, the drill
reported a working boundary as a broken one.
The words distinguish what the number cannot. box_probe now returns
reachable | refused | dropped by reading the message, ping corroborates,
and every isolation probe (sibling, box→host, RFC1918, host→box) uses it.
RUNS.md gains trap 12. It is the same disease as all eleven before it:
trusting a proxy for the fact instead of the fact.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run the doctor on a healthy host and it reported three problems, all of
them its own:
· dns.mode=none was flagged as leftover rehearsal dirt. It is now the
SHIPPED setting — it is what stops a box enumerating its siblings.
Its absence is the fault; its presence was being "fixed" away.
· the nft check used 'sudo -n', which fails without cached credentials,
so it reported the box-to-box rule MISSING on a host where 'sudo nft
list' plainly shows it.
· the kernel's bridge view — the one fact that settles the isolation
question — was skipped with "'bridge' not installed". It is installed;
it lives in /usr/sbin, which is not on a normal user's PATH.
· and the DNS probe looked only for the drill's own box names, so it
said "no box to probe with" while two boxes sat there RUNNING.
A diagnostic that cries wolf is worse than no diagnostic: it costs the
same trust as a real failure and teaches you to ignore it. All four now
report what is actually true.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two bugs, one in each direction.
setup-host's new assertion ran 'nft list table bridge claudebox' without
sudo. nft needs root, so it failed with permission denied and printed
"the box-to-box drop is NOT active" about a rule that was demonstrably
there. A check that cries wolf is worse than no check.
And the deeper one: every check so far has asked the CONFIG whether
boxes are isolated. The config is a claim. Incus can accept
security.port_isolation and the kernel can still leave 'isolated off' on
the tap — and then boxes reach each other while every config in sight
says they cannot. That is precisely the shape of the original bug: the
ACL looked airtight and never saw the traffic.
So the doctor now reads the kernel's own view — 'bridge -d link show'
on claudenet's ports — and reports the isolated flag as the fact it is.
If the profile says true and the kernel says off, we learn that in a
second instead of after another ten-minute drill.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nft bridge-family rule from the previous commit is LIVE on the host
and boxes still reach each other:
table bridge claudebox {
chain forward { ... meta ibrname "claudenet" meta obrname "claudenet" drop }
}
FAIL BOX A REACHES BOX B — sibling isolation does NOT hold [tcp: refused]
So the rule is not wrong about intent, it is wrong about mechanism —
whatever path these frames take, that hook does not stop them. Rather
than reason harder about netfilter (reasoning is what put the hole there
in the first place), use the mechanism Incus provides for exactly this:
security.port_isolation on the bridged NIC, which sets the kernel bridge
port's isolated flag so two isolated ports cannot exchange frames at all.
The nft rule stays as a second layer — it costs nothing — but the
profile flag is what carries the guarantee. doctor.sh checks it, because
the absence of this one is invisible: everything works and boxes can
simply reach each other.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The box-to-box drop shipped, the drill still found boxes reaching each
other, and the rule was simply not on the host. setup-host.sh ended with
'systemctl enable --now claudebox-firewall.service' — but the unit is
RemainAfterExit, so once it has run it stays "active" forever, and
'--now' does nothing to an active unit. Re-running setup-host after
upgrading claudebox therefore installed the new script to
/usr/local/sbin and never executed it. The host silently kept its old
firewall, and the box-to-box hole stayed open through the release that
claimed to close it.
This is worse than the original bug: every future firewall change would
have landed only on hosts that had never run setup-host before.
Restart the unit instead — the script is idempotent by design. Then
ASSERT the rule is live rather than assume it, because the absence of
this particular rule is invisible: everything keeps working and boxes
can simply reach each other. doctor.sh checks it too.
Also: dns.mode=none is now part of the shipped stack, so the drill must
stop treating it as leftover rehearsal dirt and reverting it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A live probe (drill run 10) found box A's SYN arriving at box B, and B
answering with a RST. Boxes were not isolated from each other at all,
and the README's contract — "a box reaches the public internet and
nothing else" — was false.
The ACL was not wrong; it simply never saw the traffic. Two boxes on one
bridge share an L2 segment, so their frames are SWITCHED between bridge
ports and never traverse the netfilter path where an L3 rule lives. The
drop on 10.0.0.0/8 (which contains claudenet) and the default ingress
drop both looked airtight and neither ever fired. This is why the
original reasoning — "belt and braces" — was plausible and wrong.
The bridge family does see it. Its forward hook fires exactly when a
frame passes from one bridge port to another, which on claudenet means
box→box and nothing else: frames for the gateway are delivered locally,
and so is anything routed out to the internet. Dropping every forwarded
frame on the bridge isolates the boxes and costs them nothing — DHCP and
ARP are unaffected, being broadcast and delivered on INPUT.
Also: dns.mode=none, so a box can no longer ENUMERATE its siblings
through the gateway's dnsmasq. Blocked connections with open
reconnaissance is not isolation.
security.ipv4_filtering is deliberately NOT used: it breaks the box's
networking (dockerd comes up but cannot pull or run a container).
The drill now ASSERTS all of this in phase C against the real stack;
phase D's rehearsal is retired, its findings recorded.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 10, on a doctor-clean host: 48/49, every #15 probe answered.
A3, the probe the whole exercise existed for and which nine runs failed
to fire, says sibling isolation DOES NOT HOLD. Box A's SYN reaches box
B and B answers with a RST. #12 reasoned from the code that the
10.0.0.0/8 egress drop covered box-to-box traffic "belt and braces";
two boxes on one bridge share an L2 segment, so their traffic is
switched, never routed, and never meets that rule. The tool's contract —
"a box reaches the public internet and nothing else" — is false: it also
reaches every other box on the host.
Also settled: ipv4_filtering breaks the box (measured on a baseline that
had already passed egress, unlike run 7's void verdict); dns.mode=none is
viable after all, retracting a veto I posted on one bad reading;
@internal is unsupported on bridge ACLs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two cold mints died in cloud-init with "Temporary failure resolving
deb.debian.org", on a host the doctor had just certified clean. The
cause was not DNS forwarding, not leftover mutations, and not the drill:
claudenet had NO dnsmasq. It never respawned after the SIGKILL that
recovered the wedged daemon in run 6, so boxes got no DHCP lease at all
— no address, no gateway, no DNS.
Incus does not surface this. The bridge is up, the config is pristine,
'incus network show' says status: Created. Only the process table knows.
So the doctor now asks the process table, and --fix restarts incus to
respawn it.
An hour of hunting and two dead mints went into learning this. It is a
five-second check.
RUNS.md gains trap 11: a network incus calls "Created" may have nothing
serving it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two cold mints in a row died with cloud-init 'status: error' on a host
the doctor had just certified clean — so the earlier "leftover mutations
poisoned the network" theory is dead, and the DNS failure is
reproducible rather than transient.
'claudebox new' printed four hundred dots and the word "error", leaving
the user with nothing to act on: the reason was in the box's own log and
nobody was told the log existed. It now prints cloud-init's status, the
fetch/resolve errors from the box's log, and how to inspect the box —
which is left running, because a box that failed to build is evidence,
not garbage. It also names the usual culprit: the host's resolver.
doctor.sh gains the diagnosis that keeps being done by hand:
· what the HOST resolves through, and whether that is a CGNAT/Tailscale
resolver the boxes inherit (issue #33);
· whether claudenet's resolver is pinned;
· and inside a box, the question that settles it — DNS is broken, but
can it still reach 1.1.1.1 BY ADDRESS? If yes, egress is fine and the
fault is purely the inherited forwarder.
· --pin-dns applies the #33 fix (raw.dnsmasq: no-resolv + public
servers) so the hypothesis can be TESTED rather than argued.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Run 8's cold mint failed with 'cloud-init status: error' — the box could
not resolve deb.debian.org, or claude.ai, or anything. The cause was not
in that run at all: run 7's phase D set dns.mode=none on claudenet, the
run ended before reverting it, and every box minted afterwards came up
with no DNS.
This is the worst failure mode the drill has: a poisoned host does not
fail the next run honestly, it produces confident wrong answers. It is
how a false design veto against #16 got posted, and it wasted a cold
mint plus an hour of diagnosis that had nothing to do with the code
under test.
Three defences:
· the phase-D revert is armed with a trap BEFORE the first mutation,
so it fires on any exit, Ctrl-C included;
· the revert is VERIFIED rather than fired into /dev/null, so a failed
unset can no longer masquerade as a successful one;
· the drill refuses to start on a host still carrying the mutations.
And drill/doctor.sh answers the question that kept being answered by
hand: what state is this host actually in? Network, profile, ACL,
leftover boxes, and whether a box can still resolve DNS — with --fix to
revert the leftovers.
RUNS.md gains trap 10.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
