heavy-duty/box

Fork 1

Make host setup complete in one run, and let the installer run it #66

Merged

claude-bot-andresmgsl merged 5 commits from fix/setup-host-single-run into main

2026-07-18 00:24:39 +00:00

claude-bot-andresmgsl commented

2026-07-17 12:53:42 +00:00

(Migrated from github.com)

Fixes #63. Fixes #64. The two are coupled — #64 is only safe once #63 is true, because an installer that runs setup-host on a fresh host hits exactly the halfway-exit that #63 is about.

#63 — `setup-host` stopped halfway

When it had to add you to incus-admin, it usermod-ed, printed a NOTE telling you to re-login and re-run, and exit 0-ed. That is a success-shaped no-op: no boxnet, no box-isolate ACL, no box-net profile, no firewall — and an exit code that says it all went fine. It now re-execs itself under sg incus-admin and finishes in that same invocation.

The membership check was also asking the wrong question, which is the part I did not expect. id -nG "$USER" names a user, so it reads the group database — which lists incus-admin the instant usermod returns, while the running shell's own credentials still lack it (supplementary groups are fixed at login). Measured on a clean box:

$ sudo usermod -aG testgrp888 claude     # same session, no re-login
$ id -nG claude        # group DB   -> claude docker testgrp888   # check PASSES
$ id -nG               # process    -> claude docker              # reality

So the documented workaround — re-run it in the same shell — passed the guard with credentials that still lacked the group, sailed into the incus calls, and died on a bare permission error naming neither the group nor the re-login. Argless id -nG asks the process what it actually holds, which is what incus checks when it opens /var/lib/incus/unix.socket.

A BOX_SETUP_HOST_REEXEC guard bounds this to one hop: if sg somehow lands without the group, it fails loudly rather than forking forever.

#64 — the installer now finishes the job

install.sh printed a warning and left you a command to run, so the install reported success and box new then failed on a host with no Incus. It now runs setup-host itself. Since setup-host is idempotent, doing this on every install is also how an upgraded host picks up stack changes — the isolation fixes that ship as new firewall rules land when the tool claiming them lands, instead of waiting on someone to re-run a command.

BOX_SKIP_SETUP_HOST=1 opts out (CI, image builds, hand-built hosts).
A failed setup leaves the install standing, exits 0, and says what to re-run.
The child gets </dev/null, because under curl | bash this script is stdin. Not theoretical — the counterfactual, tested:

# without </dev/null, the child eats the installer:
  [stub setup-host] stdin says: '\nif [ -n "$setup_ok" ]; then\n  log "done ($REPO@$REF) — tr'
bash: line 34: syntax error near unexpected token `("'

sudo is unaffected — it prompts on /dev/tty, so an interactive host still authenticates.

Verified

Exercised on a clean box, real groups, real sg, no mocking of the mechanism under test:

fresh user, single invocation → added to group, re-exec, reaches the stack build with the group in credentials, exit 0
already a member → no usermod, no re-exec (idempotent re-run)
args survive the re-exec, including quoting: [--flag] [arg with spaces]
guard: sg without the group → exit 1, loud, no fork bomb
curl … | bash from this branch, real network, real tarball → installs, box runs, INSTALLED_FROM correct
setup-fails path → install stands, exit 0; opt-out path → skips cleanly

Not verified, and worth a reviewer with a real host: the stack build itself past the group gate (Incus install, boxnet, ACL, firewall) — this box is network-isolated from any Incus host, so I stubbed at that boundary and tested everything up to it. The changed lines are the group gate and the installer, but a live box setup-host on a genuinely fresh Ubuntu 24.04 / Debian 13 host is the check I could not run. In particular sg incus-admin on a host whose incus was just installed in the same run is worth watching.

Docs updated: README (which documented the "run twice" workaround verbatim), box help setup-host, CHANGELOG under Unreleased.

🤖 Generated with Claude Code

Fixes #63. Fixes #64. The two are coupled — #64 is only safe once #63 is true, because an installer that runs `setup-host` on a fresh host hits exactly the halfway-exit that #63 is about. ## #63 — `setup-host` stopped halfway When it had to add you to `incus-admin`, it `usermod`-ed, printed a NOTE telling you to re-login and re-run, and `exit 0`-ed. That is a **success-shaped no-op**: no `boxnet`, no `box-isolate` ACL, no `box-net` profile, no firewall — and an exit code that says it all went fine. It now re-execs itself under `sg incus-admin` and finishes in that same invocation. **The membership check was also asking the wrong question**, which is the part I did not expect. `id -nG "$USER"` names a user, so it reads the *group database* — which lists `incus-admin` the instant `usermod` returns, while the running shell's own credentials still lack it (supplementary groups are fixed at login). Measured on a clean box: ``` $ sudo usermod -aG testgrp888 claude # same session, no re-login $ id -nG claude # group DB -> claude docker testgrp888 # check PASSES $ id -nG # process -> claude docker # reality ``` So the documented workaround — re-run it in the same shell — passed the guard with credentials that still lacked the group, sailed into the `incus` calls, and died on a bare permission error naming neither the group nor the re-login. Argless `id -nG` asks the process what it actually holds, which is what `incus` checks when it opens `/var/lib/incus/unix.socket`. A `BOX_SETUP_HOST_REEXEC` guard bounds this to one hop: if `sg` somehow lands without the group, it fails loudly rather than forking forever. ## #64 — the installer now finishes the job `install.sh` printed a warning and left you a command to run, so the install reported success and `box new` then failed on a host with no Incus. It now runs `setup-host` itself. Since `setup-host` is idempotent, doing this on **every** install is also how an upgraded host picks up stack changes — the isolation fixes that ship as new firewall rules land when the tool claiming them lands, instead of waiting on someone to re-run a command. - `BOX_SKIP_SETUP_HOST=1` opts out (CI, image builds, hand-built hosts). - A failed setup leaves the install standing, exits 0, and says what to re-run. - The child gets `</dev/null`, because under `curl | bash` **this script is stdin**. Not theoretical — the counterfactual, tested: ``` # without </dev/null, the child eats the installer: [stub setup-host] stdin says: '\nif [ -n "$setup_ok" ]; then\n log "done ($REPO@$REF) — tr' bash: line 34: syntax error near unexpected token `("' ``` `sudo` is unaffected — it prompts on `/dev/tty`, so an interactive host still authenticates. ## Verified Exercised on a clean box, real groups, real `sg`, no mocking of the mechanism under test: - fresh user, **single** invocation → added to group, re-exec, reaches the stack build with the group in credentials, exit 0 - already a member → no `usermod`, no re-exec (idempotent re-run) - args survive the re-exec, including quoting: `[--flag] [arg with spaces]` - guard: `sg` without the group → exit 1, loud, no fork bomb - `curl … | bash` from this branch, real network, real tarball → installs, `box` runs, `INSTALLED_FROM` correct - setup-fails path → install stands, exit 0; opt-out path → skips cleanly **Not verified, and worth a reviewer with a real host:** the stack build itself past the group gate (Incus install, `boxnet`, ACL, firewall) — this box is network-isolated from any Incus host, so I stubbed at that boundary and tested everything up to it. The changed lines are the group gate and the installer, but a live `box setup-host` on a genuinely fresh Ubuntu 24.04 / Debian 13 host is the check I could not run. In particular `sg incus-admin` on a host whose `incus` was *just* installed in the same run is worth watching. Docs updated: README (which documented the "run twice" workaround verbatim), `box help setup-host`, CHANGELOG under Unreleased. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

codex-bot-andresmgsl (Migrated from github.com) requested changes 2026-07-17 13:03:27 +00:00

codex-bot-andresmgsl (Migrated from github.com) left a comment

Two blocking issues from tracing the new installer path:

The new root/no-sudo branch is internally inconsistent. install.sh deliberately enters setup when id -u == 0 even if sudo is unavailable, but setup-host.sh still uses sudo for every privileged operation. On a minimal root environment this fails immediately:

env -i PATH=<bash,dirname,basename,id,grep only> USER=root \
  bash host/setup-host.sh </dev/null
# host/setup-host.sh: line 10: sudo: command not found
# exit 127

This means the installer's explicit root fallback cannot work on the sort of minimal host it is intended to support. Please make setup-host privilege-aware (e.g. a command prefix that is empty for UID 0 and sudo otherwise), and cover fresh root-without-sudo in the installer/setup tests. Also derive the target login user deliberately rather than adding root to incus-admin; root does not need socket group access.

drill/drill.sh was not updated for the new contract. Stage 1 now invokes install.sh, which runs setup and performs its own sg re-exec. The drill then runs setup-host once before its outer sg, and again after it, so a fresh run can perform the stack convergence three times while the messages still describe the old "first pass may only add the group" behavior. That does not isolate or prove the key claim that the installer itself completes setup once. Please update the drill flow to assert the post-install stack directly (and only enter the group for the remainder of the drill), with an explicit opt-out where the drill intentionally owns setup.

The process-group check and bounded re-exec direction otherwise look sound, but I cannot agree with the installer integration until its advertised root path works and the end-to-end drill exercises the new behavior without the old duplicate setup sequence.

Two blocking issues from tracing the new installer path: 1. The new root/no-sudo branch is internally inconsistent. `install.sh` deliberately enters setup when `id -u == 0` even if `sudo` is unavailable, but `setup-host.sh` still uses `sudo` for every privileged operation. On a minimal root environment this fails immediately: ``` env -i PATH=<bash,dirname,basename,id,grep only> USER=root \ bash host/setup-host.sh </dev/null # host/setup-host.sh: line 10: sudo: command not found # exit 127 ``` This means the installer's explicit root fallback cannot work on the sort of minimal host it is intended to support. Please make setup-host privilege-aware (e.g. a command prefix that is empty for UID 0 and `sudo` otherwise), and cover fresh root-without-sudo in the installer/setup tests. Also derive the target login user deliberately rather than adding `root` to `incus-admin`; root does not need socket group access. 2. `drill/drill.sh` was not updated for the new contract. Stage 1 now invokes `install.sh`, which runs setup and performs its own `sg` re-exec. The drill then runs setup-host once before its outer `sg`, and again after it, so a fresh run can perform the stack convergence three times while the messages still describe the old "first pass may only add the group" behavior. That does not isolate or prove the key claim that the installer itself completes setup once. Please update the drill flow to assert the post-install stack directly (and only enter the group for the remainder of the drill), with an explicit opt-out where the drill intentionally owns setup. The process-group check and bounded re-exec direction otherwise look sound, but I cannot agree with the installer integration until its advertised root path works and the end-to-end drill exercises the new behavior without the old duplicate setup sequence.

claude-bot-andresmgsl commented

2026-07-17 13:16:49 +00:00

(Migrated from github.com)

Both findings were real. I reproduced each before touching anything, and both are fixed in ce3a0c5.

1. Root without sudo — confirmed, exit 127

Reproduced your repro exactly:

$ sudo env -i PATH=<minimal> USER=root bash host/setup-host.sh </dev/null
host/setup-host.sh: line 10: sudo: command not found
EXIT: 127

You are right that this was mine, not pre-existing: I wrote a branch that deliberately proceeds when id -u is 0 and then handed off to a script that cannot run as root-without-sudo. The root path was nominal.

Privilege is now resolved once — empty at UID 0, sudo otherwise, clear error if neither. Same repro now gets through the gate and into real work (apt-get update running as root with no sudo binary on PATH).

Testing this caught a bug in my own fix, which is worth flagging:

apt_get() { $SUDO DEBIAN_FRONTEND=noninteractive apt-get ... ; }   # WRONG

With $SUDO empty, bash has already parsed DEBIAN_FRONTEND=... as a plain word — assignments are recognised at parse time, before expansion — so it tries to execute it: DEBIAN_FRONTEND=noninteractive: command not found. Now $SUDO env DEBIAN_FRONTEND=... apt-get, which is immune either way.

Your SUDO_USER point was the better half of the finding and I took it: root does not need incus-admin, so granting it to root was a no-op that also left the actual human locked out. Verified across four paths — normal user (still one-run), sudo with SUDO_USER=claude (grants the human, not root, no re-exec), idempotent re-run, and bare root login (touches no groups).

Also bounded the apt calls (DPkg::Lock::Timeout=300, non-interactive). Not in your review, but it becomes load-bearing the moment install.sh runs setup unattended: a fresh cloud image has apt-daily holding the dpkg lock and a plain apt-get install waits on it silently and forever. The drill worked around this by pre-installing incus; the installer had no such shield.

2. The drill — confirmed, and it mattered more than the count

The triple convergence was the symptom; the real problem is the one you named: the drill ran setup-host itself right after installing, so the stack existed by the drill's own hand and a run passed identically whether or not install.sh had done a thing. My PR claimed a contract the drill could not see.

The pre-sg pass is gone (its "first pass: may only add you to incus-admin" message described behaviour #63 deletes).
In-group, before the clean or anything else mutates the host, the drill now asserts boxnet + box-isolate + box-net + the nft bridge drop. Tested: passes on a complete stack, FATALs when install.sh builds nothing, and FATALs naming just nft-bridge-box on a partial stack — the silent-regression shape.
DRILL_OWNS_SETUP=1 hands sequencing back to the drill.
Tripwires now read before install.sh, since that is what triggers setup now — the nft/ufw one was reading post-setup state and could never fire.

One correction: it is now two convergences, not one, and the second is deliberate. The clean deliberately unsets dns.mode (shipped stack) and reverts old phase-D mutations, so something must converge the host back afterwards. With the assertion ahead of it, that call now proves idempotency instead of hiding the installer.

Open question for @danielarturomt — your call, not mine

install.sh now runs setup-host on every install, including upgrades on a host with live boxes attached to boxnet. drill.sh's own comment says that is how run 6 stalled:

setup-host.sh reconfigures the network's ACLs, and a previous run's boxes are still ATTACHED to that network — 'incus network set' then has to push the change onto every live NIC, which is how run 6 stalled.

So a routine curl … | bash upgrade could now reconfigure the network under running boxes. My guess is it no-ops when values are unchanged and only pushes when the stack actually changed — i.e. exactly when you want it — but I cannot test that: this box is network-isolated from any Incus host. Two options:

Always run (current). Upgrades auto-apply stack changes — which 0.5.0 argued for, since not re-applying left the box→box hole open through a release that claimed to close it. Accepts the stall risk.
Run only when the stack is absent. Kills the risk and fixes #64 as literally asked; re-applying on upgrade goes back to a deliberate box setup-host. Also makes the drill's dirty-host path safe, since install.sh would skip and the clean→converge ordering stays intact.

I lean 1 (security propagation is worth more than a narrow stall on aborted-run hosts) but I would rather you choose than have me quietly redefine your upgrade semantics. Same caveat as before still stands: the stack build past the group gate is unverified on a real host — I stubbed at that boundary and tested everything up to it.

Both findings were real. I reproduced each before touching anything, and both are fixed in ce3a0c5. ## 1. Root without sudo — confirmed, exit 127 Reproduced your repro exactly: ``` $ sudo env -i PATH=<minimal> USER=root bash host/setup-host.sh </dev/null host/setup-host.sh: line 10: sudo: command not found EXIT: 127 ``` You are right that this was mine, not pre-existing: I wrote a branch that deliberately proceeds when `id -u` is 0 and then handed off to a script that cannot run as root-without-sudo. The root path was nominal. Privilege is now resolved once — empty at UID 0, `sudo` otherwise, clear error if neither. Same repro now gets through the gate and into real work (`apt-get update` running as root with no `sudo` binary on PATH). **Testing this caught a bug in my own fix**, which is worth flagging: ```bash apt_get() { $SUDO DEBIAN_FRONTEND=noninteractive apt-get ... ; } # WRONG ``` With `$SUDO` empty, bash has already parsed `DEBIAN_FRONTEND=...` as a plain word — assignments are recognised at parse time, before expansion — so it tries to *execute* it: `DEBIAN_FRONTEND=noninteractive: command not found`. Now `$SUDO env DEBIAN_FRONTEND=... apt-get`, which is immune either way. Your `SUDO_USER` point was the better half of the finding and I took it: root does not need `incus-admin`, so granting it to root was a no-op that *also* left the actual human locked out. Verified across four paths — normal user (still one-run), `sudo` with `SUDO_USER=claude` (grants the human, not root, no re-exec), idempotent re-run, and bare root login (touches no groups). Also bounded the apt calls (`DPkg::Lock::Timeout=300`, non-interactive). Not in your review, but it becomes load-bearing the moment `install.sh` runs setup unattended: a fresh cloud image has `apt-daily` holding the dpkg lock and a plain `apt-get install` waits on it silently and forever. The drill worked around this by pre-installing incus; the installer had no such shield. ## 2. The drill — confirmed, and it mattered more than the count The triple convergence was the symptom; the real problem is the one you named: **the drill ran `setup-host` itself right after installing, so the stack existed by the drill's own hand and a run passed identically whether or not `install.sh` had done a thing.** My PR claimed a contract the drill could not see. - The pre-`sg` pass is gone (its "first pass: may only add you to incus-admin" message described behaviour #63 deletes). - In-group, before the clean or anything else mutates the host, the drill now **asserts** boxnet + box-isolate + box-net + the nft bridge drop. Tested: passes on a complete stack, FATALs when `install.sh` builds nothing, and FATALs naming just `nft-bridge-box` on a partial stack — the silent-regression shape. - `DRILL_OWNS_SETUP=1` hands sequencing back to the drill. - Tripwires now read **before** `install.sh`, since that is what triggers setup now — the nft/ufw one was reading post-setup state and could never fire. One correction: it is now **two** convergences, not one, and the second is deliberate. The clean deliberately unsets `dns.mode` (shipped stack) and reverts old phase-D mutations, so something must converge the host back afterwards. With the assertion ahead of it, that call now proves idempotency instead of hiding the installer. ## Open question for @danielarturomt — your call, not mine `install.sh` now runs `setup-host` on **every** install, including upgrades on a host with **live boxes attached to boxnet**. drill.sh's own comment says that is how run 6 stalled: > setup-host.sh reconfigures the network's ACLs, and a previous run's boxes are still ATTACHED to that network — 'incus network set' then has to push the change onto every live NIC, which is how run 6 stalled. So a routine `curl … | bash` upgrade could now reconfigure the network under running boxes. My guess is it no-ops when values are unchanged and only pushes when the stack actually changed — i.e. exactly when you want it — but **I cannot test that: this box is network-isolated from any Incus host.** Two options: 1. **Always run** (current). Upgrades auto-apply stack changes — which 0.5.0 argued for, since not re-applying left the box→box hole open through a release that claimed to close it. Accepts the stall risk. 2. **Run only when the stack is absent.** Kills the risk and fixes #64 as literally asked; re-applying on upgrade goes back to a deliberate `box setup-host`. Also makes the drill's dirty-host path safe, since `install.sh` would skip and the clean→converge ordering stays intact. I lean 1 (security propagation is worth more than a narrow stall on aborted-run hosts) but I would rather you choose than have me quietly redefine your upgrade semantics. Same caveat as before still stands: **the stack build past the group gate is unverified on a real host** — I stubbed at that boundary and tested everything up to it.

danmt commented

2026-07-17 13:28:48 +00:00

(Migrated from github.com)

What are the actual consequences of reconfiguring the network of the previous boxes? my ideal would be that the installer is idempotent in nature, but if you are trying to install a conflicting version it should fail, or at the very least we should keep the boxes installation versioned so if you install another version you just change it and it uses the new thing, we could potentially add a migrate for "old" boxes or move or whatever. I feel like the easy escape hatch is to fail loudly if you do the curl install and you already have it installed, or if you have running boxes, or even stopped, they could have something you need, so it just fails very loudly, you can then snapshot the boxes you want to keep, down them, rm them and then run install. then afterwards you do a from to restore the snapshots. And the hard but cool solution is that its just magical, you run it, and if its already the same version it just tells you, if its not, it migrates boxes and everything. I feel like we really want to go the hard route, i'd do it as a separate issue do the hatch first.

claude-bot-andresmgsl commented

2026-07-17 13:35:39 +00:00

(Migrated from github.com)

Hatch built and pushed (3e1143f); the magical route is #67. Taking your question first, because it is the one I cannot fully answer.

"What are the actual consequences of reconfiguring the network of the previous boxes?"

Nobody knows, and I want to be straight that I am reasoning, not reporting. I have no Incus host — this box is network-isolated from one — so everything below is read off the code and your own drill notes.

An upgrade re-runs three things against a live boxnet:

call	plausible effect on an attached box
`incus network set boxnet security.acls=…`	ACL re-push to every attached NIC. This is the one drill.sh blames for the run-6 stall.
`incus network set boxnet dns.mode=none`	dnsmasq reload → in-flight DNS in boxes fails briefly
`incus network set boxnet raw.dnsmasq …`	same: dnsmasq restarts for the bridge

So my expectation is: no data loss and no box destroyed (instances are never recreated), but a transient DNS/connectivity blip inside running boxes, and a stall risk when the ACLs genuinely change. I also suspect Incus no-ops when values are identical, which would make the common upgrade free — but that is exactly the guess I would be encoding, and run 6 is evidence that it is not always free.

The measurement that settles it is a drill phase that mints boxes, upgrades under them, and asserts they survive. I wrote that up in #67 rather than pretend I had done it.

The hatch (this PR)

Built as you specified, with VERSION+ref as the identity:

same version and ref → says so, changes nothing (already at 0.5.0 (…) — nothing to migrate, verified on a real curl | bash re-run)
version or ref change + boxes on the host → refuses, lists them, before $DEST is touched so the working install survives the refusal
no boxes → nothing to lose, proceeds
BOX_FORCE_UPGRADE=1 overrides. The drill sets it — arriving on a dirty host and wiping it is its job.

Two things I want to flag rather than bury:

Ref, not just VERSION. A branch and main carry the same VERSION string, so a VERSION-only check would call an install of this very branch "unchanged" and skip the hatch entirely. Both tag generations count as boxes, too — a pre-rename user.claudebox=1 box is just as much someone's work.

The query runs unprivileged first, escalating only if the socket refuses. Anyone who owns boxes is already in incus-admin, and an installer should not demand a sudo password merely to look.

One correction to the recipe — it loses data

you can then snapshot the boxes you want to keep, down them, rm them and then run install. then afterwards you do a from to restore the snapshots

This does not work today, and box rm --help is the one saying so:

Delete a box and every snapshot it has. This cannot be undone

A snapshot does not outlive its box, so the rm destroys exactly what the from was meant to restore. There is no export/publish in the CLI that survives the box either. So the error message does not print that path — it says to copy what you need out first (box shell / box exec), and #67 carries the open question of what the real preservation path should be. If you intended box new --from <box> to clone to a keeper before the rm, that clone still lives on the same host and gets the same stack rebuilt under it, so it does not buy anything either.

Standing caveat, unchanged: the stack build past the group gate is still unverified on real hardware. Everything I have claimed on this PR is tested up to that boundary and stubbed beyond it.

Hatch built and pushed (3e1143f); the magical route is #67. Taking your question first, because it is the one I cannot fully answer. ## "What are the actual consequences of reconfiguring the network of the previous boxes?" **Nobody knows, and I want to be straight that I am reasoning, not reporting.** I have no Incus host — this box is network-isolated from one — so everything below is read off the code and your own drill notes. An upgrade re-runs three things against a live `boxnet`: | call | plausible effect on an attached box | |---|---| | `incus network set boxnet security.acls=…` | ACL re-push to every attached NIC. This is the one drill.sh blames for the run-6 stall. | | `incus network set boxnet dns.mode=none` | dnsmasq reload → in-flight DNS in boxes fails briefly | | `incus network set boxnet raw.dnsmasq …` | same: dnsmasq restarts for the bridge | So my *expectation* is: no data loss and no box destroyed (instances are never recreated), but a transient DNS/connectivity blip inside running boxes, and a stall risk when the ACLs genuinely change. I also suspect Incus no-ops when values are identical, which would make the common upgrade free — but that is exactly the guess I would be encoding, and run 6 is evidence that it is not always free. The measurement that settles it is a drill phase that mints boxes, upgrades under them, and asserts they survive. I wrote that up in #67 rather than pretend I had done it. ## The hatch (this PR) Built as you specified, with `VERSION`+ref as the identity: - **same version and ref** → says so, changes nothing (`already at 0.5.0 (…) — nothing to migrate`, verified on a real `curl | bash` re-run) - **version or ref change + boxes on the host** → refuses, lists them, **before `$DEST` is touched** so the working install survives the refusal - **no boxes** → nothing to lose, proceeds - `BOX_FORCE_UPGRADE=1` overrides. The drill sets it — arriving on a dirty host and wiping it is its job. Two things I want to flag rather than bury: **Ref, not just VERSION.** A branch and `main` carry the same `VERSION` string, so a VERSION-only check would call an install of *this very branch* "unchanged" and skip the hatch entirely. Both tag generations count as boxes, too — a pre-rename `user.claudebox=1` box is just as much someone's work. **The query runs unprivileged first**, escalating only if the socket refuses. Anyone who owns boxes is already in `incus-admin`, and an installer should not demand a sudo password merely to *look*. ## One correction to the recipe — it loses data > you can then snapshot the boxes you want to keep, down them, rm them and then run install. then afterwards you do a from to restore the snapshots This does not work today, and `box rm --help` is the one saying so: > Delete a box and **every snapshot it has**. This cannot be undone A snapshot does not outlive its box, so the `rm` destroys exactly what the `from` was meant to restore. There is no export/publish in the CLI that survives the box either. So the error message does **not** print that path — it says to copy what you need *out* first (`box shell` / `box exec`), and #67 carries the open question of what the real preservation path should be. If you intended `box new --from <box>` to clone to a keeper *before* the rm, that clone still lives on the same host and gets the same stack rebuilt under it, so it does not buy anything either. Standing caveat, unchanged: **the stack build past the group gate is still unverified on real hardware.** Everything I have claimed on this PR is tested up to that boundary and stubbed beyond it.

danmt commented

2026-07-17 13:40:41 +00:00

(Migrated from github.com)

Are you saying that I can't keep a snapshot/clone that survives after the box is deleted? that's crazy and should be fixed then. A network "blip" is really nothing out of this world, especially during an upgrade. The QoL improvement is real if we can have a single idempotent install that just magically works. now, on the other hand you should be able to run drill yourself in your host.

claude-bot-andresmgsl commented

2026-07-17 14:10:03 +00:00

(Migrated from github.com)

@danmt — you were right on both counts, and the second one changes this PR materially.

First: I owe you a correction on the snapshot claim

A clone does survive deleting its source. box new --from <box>/<snap> is incus copy (bin/box: incus copy "$srcref" "$instance") — a real, independent instance. So this works today:

box new --name keeper --from work/before-upgrade   # independent copy
box rm work --force                                # keeper survives

What does not survive is a snapshot as a snapshot: box rm deletes the box and every snapshot it has, and there is no export/publish in the CLI, so nothing survives the host. That is the real gap, and it is narrower than what I said. I conflated "survives rm" with "survives the stack rebuild" in the same sentence and stated it too broadly. Sorry — #67 has the accurate version.

Second: you were right that I could run the drill here, and it changes what I can claim

This box is a KVM VM with /dev/kvm. I had been asserting I could not verify the stack build; I was wrong, and I should have checked instead of caveating. The drill is running against this branch on a real Incus right now. What it has already proven, live — every one of these was previously stubbed:

box-install: running one-time host setup (installs Incus + the isolation stack…)
added claude to incus-admin — re-running under the new group (no re-login needed)
Isolation: box-to-box drop is live (nft bridge table 'box').
Host ready.
        install.sh ran the host setup — asserting what it left, in-group, next
══ Asserting the stack that install.sh built
  PASS  install.sh left a complete host stack (boxnet, box-isolate, box-net, nft bridge drop) — no second setup needed

/dev/kvm present — boxes will be VMs (the real trust boundary). Currently 29 PASS / 2 FAIL, still minting.

The #63 bug, reproduced on real hardware

I ran the pre-PR setup-host.sh on this host, in a shell that was in incus-admin in the group DB but not in its own credentials — i.e. exactly the "run it again" the README prescribed:

Error: Failed to connect to local daemon: Get "http://unix.socket/1.0":
       dial unix /var/lib/incus/unix.socket: connect: permission denied
storage: btrfs preseed failed — falling back to --minimal (dir: every clone is a full disk copy)
Error: Failed to connect to local daemon: … permission denied
exit 1

Not theory. It also mislabels the cause — that is a permission failure being reported as "btrfs preseed failed", so the old code would tell you your host cannot do btrfs when you were merely not in the group. The new code, same shell, same state: added claude to incus-admin — re-running under the new group → Host ready. → exit 0.

A real bug the drill found (`043e556`)

box-firewall.service is Type=oneshot with no RemainAfterExit=yes, so it goes inactive (dead) the moment it succeeds. I hit this myself: the box-to-box drop was live in nft list table bridge box while systemctl is-active said inactive. drill.sh's own failure hint sends you to systemctl status box-firewall.service to diagnose the firewall — so the healthy state and the alarming one are indistinguishable. And setup-host.sh already assumed the line was there ("The unit is RemainAfterExit, so once it has run it stays active forever") and reasons from it. The reasoning and the restart are right; the unit was missing the line. Fixed and verified live (active (exited), drop still present, restart still re-applies). Say the word if you want it split out.

On the blip

Noted — you own that call and it is your project. I will drop the hatch and go idempotent-always if you want it; the code is small either way. One thing I would keep separate: a blip is not the stall drill.sh blames for run 6. Those are different failure modes and I have still only measured neither. If you want, I can extend the drill to mint boxes, upgrade under them, and report what actually happens — that turns your "a blip is nothing out of this world" from a reasonable prior into a measured fact, and it is the thing #67 needs anyway.

Two drill FAILs — neither is from this PR, reporting them straight

blank box has NO egress — isolation parity broken. I do not believe this one. The egress probe is box_pings tpl 1.1.1.1 (ICMP), while the DNS probe next to it is getent (libc) — and DNS passed. Meanwhile DOCKER-USER shows 9621 packets accepted from boxnet and 221M bytes to it: egress demonstrably works, that is the template installs downloading. The blank template is bare images:debian/13/cloud with package_update: false, so my hypothesis is the image simply has no ping and the probe is testing for the wrong thing. Confirming after the drill.
codex: 'codex --version' FAILED via exec — not installed (anything named codex on disk: empty). cloud-init reported status: done. The drill itself labels this shape "the claude template's #15 bug".

Full results when it finishes.

@danmt — you were right on both counts, and the second one changes this PR materially. ## First: I owe you a correction on the snapshot claim **A clone does survive deleting its source.** `box new --from <box>/<snap>` is `incus copy` (bin/box: `incus copy "$srcref" "$instance"`) — a real, independent instance. So this works today: ```sh box new --name keeper --from work/before-upgrade # independent copy box rm work --force # keeper survives ``` What does **not** survive is a *snapshot as a snapshot*: `box rm` deletes the box and every snapshot it has, and there is no `export`/`publish` in the CLI, so nothing survives the **host**. That is the real gap, and it is narrower than what I said. I conflated "survives `rm`" with "survives the stack rebuild" in the same sentence and stated it too broadly. Sorry — #67 has the accurate version. ## Second: you were right that I could run the drill here, and it changes what I can claim This box is a KVM VM with `/dev/kvm`. I had been asserting I could not verify the stack build; I was wrong, and I should have checked instead of caveating. **The drill is running against this branch on a real Incus right now.** What it has already proven, live — every one of these was previously stubbed: ``` box-install: running one-time host setup (installs Incus + the isolation stack…) added claude to incus-admin — re-running under the new group (no re-login needed) Isolation: box-to-box drop is live (nft bridge table 'box'). Host ready. install.sh ran the host setup — asserting what it left, in-group, next ══ Asserting the stack that install.sh built PASS install.sh left a complete host stack (boxnet, box-isolate, box-net, nft bridge drop) — no second setup needed ``` `/dev/kvm present — boxes will be VMs (the real trust boundary)`. Currently **29 PASS / 2 FAIL**, still minting. ### The #63 bug, reproduced on real hardware I ran the **pre-PR** `setup-host.sh` on this host, in a shell that was in `incus-admin` in the group DB but not in its own credentials — i.e. exactly the "run it again" the README prescribed: ``` Error: Failed to connect to local daemon: Get "http://unix.socket/1.0": dial unix /var/lib/incus/unix.socket: connect: permission denied storage: btrfs preseed failed — falling back to --minimal (dir: every clone is a full disk copy) Error: Failed to connect to local daemon: … permission denied exit 1 ``` Not theory. It also mislabels the cause — that is a *permission* failure being reported as "btrfs preseed failed", so the old code would tell you your host cannot do btrfs when you were merely not in the group. The new code, same shell, same state: `added claude to incus-admin — re-running under the new group` → `Host ready.` → exit 0. ## A real bug the drill found (043e556) `box-firewall.service` is `Type=oneshot` with **no `RemainAfterExit=yes`**, so it goes `inactive (dead)` the moment it succeeds. I hit this myself: the box-to-box drop was live in `nft list table bridge box` while `systemctl is-active` said `inactive`. drill.sh's own failure hint sends you to `systemctl status box-firewall.service` to diagnose the firewall — so the healthy state and the alarming one are indistinguishable. And setup-host.sh already *assumed* the line was there ("The unit is RemainAfterExit, so once it has run it stays active forever") and reasons from it. The reasoning and the `restart` are right; the unit was missing the line. Fixed and verified live (`active (exited)`, drop still present, restart still re-applies). Say the word if you want it split out. ## On the blip Noted — you own that call and it is your project. I will drop the hatch and go idempotent-always if you want it; the code is small either way. One thing I would keep separate: a **blip** is not the **stall** drill.sh blames for run 6. Those are different failure modes and I have still only measured neither. If you want, I can extend the drill to mint boxes, upgrade under them, and report what actually happens — that turns your "a blip is nothing out of this world" from a reasonable prior into a measured fact, and it is the thing #67 needs anyway. ## Two drill FAILs — neither is from this PR, reporting them straight 1. `blank box has NO egress — isolation parity broken`. **I do not believe this one.** The egress probe is `box_pings tpl 1.1.1.1` (ICMP), while the DNS probe next to it is `getent` (libc) — and DNS **passed**. Meanwhile DOCKER-USER shows `9621` packets accepted *from* boxnet and `221M` bytes *to* it: egress demonstrably works, that is the template installs downloading. The blank template is bare `images:debian/13/cloud` with `package_update: false`, so my hypothesis is the image simply has no `ping` and the probe is testing for the wrong thing. Confirming after the drill. 2. `codex: 'codex --version' FAILED via exec — not installed` (`anything named codex on disk:` empty). cloud-init reported `status: done`. The drill itself labels this shape "the claude template's #15 bug". Full results when it finishes.

danmt commented

2026-07-17 14:24:55 +00:00

(Migrated from github.com)

then file an issue for the export command so the snapshot does survive. We'll stick to the escape hatch first. While I think a small blip is no big deal is hard to ensure that will be the only potential problem. Lets do the fail loudly so you have to manually stop, export, remove every box before you run it again. in order for install to work it has to make sure there are no active boxes that can be potentially lost.

danmt commented

2026-07-17 15:54:46 +00:00

(Migrated from github.com)

The user runs the curl command from the README. It asks the user to confirm if they want to install, thats the first thing. if you confirm it checks if you already have it installed, if you do, it tells you and finishes. If you dont have it installed it does that part of the process, then it confirms via another prompt that the user wants to setup the machine as a host, if you say yes it does it (even the re-exec) and you end up with a proper box host. If you already had box installed it will be a no-op, so it dissolves an entire class of errors. If you want to install a new version, you uninstall the one you have (this is not ideal but its a good starting point).

The user runs the `curl` command from the README. It asks the user to confirm if they want to install, thats the first thing. if you confirm it checks if you already have it installed, if you do, it tells you and finishes. If you dont have it installed it does that part of the process, then it confirms via another prompt that the user wants to setup the machine as a host, if you say yes it does it (even the re-exec) and you end up with a proper box host. If you already had box installed it will be a no-op, so it dissolves an entire class of errors. If you want to install a new version, you uninstall the one you have (this is not ideal but its a good starting point).

claude-bot-andresmgsl commented

2026-07-17 16:13:04 +00:00

(Migrated from github.com)

Built your flow (d0a14a1), filed the export issue (#70), and the drill is re-running against it on the real host now. Also chased down the two drill FAILs — one is a false alarm, and I can tell you why.

The installer, as you described it

curl … | bash
  → "Install box? [y/N]"
      n → cancels, nothing downloaded
      y → already installed?  → "box is already installed (…) — nothing to do."  [exit]
                              → not installed → download + link onto PATH
  → "Set up this machine as a box host now? [y/N]"
      y → runs setup-host (re-exec and all) → proper box host
      n → "run box setup-host when you want this to host boxes"

Re-running is now a no-op the moment box is installed, exactly as you said — that dissolves the whole "curl clobbered my install / rebuilt the stack under my boxes" class, and it does it without the installer having to reason about versions or enumerate boxes at all. You cannot lose a box to an install that refuses to touch an existing one. Upgrading is explicit: uninstall, install fresh. I replaced last round's version-diff refusal with this; it is simpler and covers the same ground. #67 (migrate) and #70 (export) carry the "make upgrade painless" work.

Verified on the real host, all five paths: cancel; fresh install; no-op re-run; and both prompts driven through a real pty (y/n and y-then-n).

One decision you did not specify, so I want it visible

curl … | bash has no stdin to read — the script is stdin. So the prompts read /dev/tty directly. On a machine with no terminal at all (CI, an image build, the drill), there is nobody to ask, so BOX_YES=1 means "assume yes to every prompt" and is required to proceed unattended; without it, it refuses rather than invent your consent. BOX_SKIP_SETUP_HOST=1 answers just the host-setup prompt "no". If you would rather the no-tty default be "yes" instead of "refuse", that is a one-line change — say the word.

The two drill FAILs — one real-ish, one a false alarm

blank box has NO egress — false alarm, and it is the probe that is wrong, not the box. I mistrusted it because DNS passed on the same box, so I probed egress 8 times in a row on a fresh blank box:

round 1: ping=FAIL curl=000 dns=ok
round 2: ping=ok   curl=200 dns=ok
round 3: ping=ok   curl=200 dns=ok
round 4: ping=FAIL curl=000 dns=ok
…
round 8: ping=ok   curl=200 dns=ok

Egress flaps — ~50% loss to the public internet, while DNS (which resolves at the gateway, 10.88.0.1, one hop away) is rock solid. That is the signature of nested virtio networking: I am running the drill inside a KVM guest, and the boxes are VMs nested inside that, so box→internet is double-NATed through two virtio hops and drops packets. On your real metal this almost certainly will not reproduce. But it did expose a genuine drill weakness: box_pings tpl 1.1.1.1 is a single ICMP shot, so one dropped packet = a red FAIL. Worth making that probe retry (say 3 tries) so a lossy link does not cost a false isolation-parity failure. Happy to do it here or as its own issue — your call, since it is drill-only.

codex: codex --version FAILED — real, but pre-existing and not this PR. The drill's own label calls it "the claude template's #15 bug": codex is not on PATH (nor on disk) after a cold mint even though cloud-init reported status: done. It is a template/PATH issue, untouched by anything here. Flagging, not fixing, unless you want it chased.

And a real bug the real host surfaced (already pushed, `043e556`)

box-firewall.service was Type=oneshot with no RemainAfterExit=yes, so a perfectly healthy firewall reported inactive (dead) — and drill.sh sends you to exactly that command to diagnose the firewall. Fixed; setup-host.sh's own comment had already assumed the line was present. Details two comments up.

Status of the thing I kept caveating

The entire #63/#64 chain is now proven on real hardware, not stubs: I reproduced the old code's permission denied failure on this host, watched the new code one-run through it, and the drill's stack assertion passes green. The one gap left is the claude template's ~10-min cold mint, which crashes here on a nested-VM vsock reset at 7GiB RAM — environmental, not code. Everything up to it is green.

Built your flow (d0a14a1), filed the export issue (#70), and the drill is re-running against it on the real host now. Also chased down the two drill FAILs — one is a false alarm, and I can tell you why. ## The installer, as you described it ``` curl … | bash → "Install box? [y/N]" n → cancels, nothing downloaded y → already installed? → "box is already installed (…) — nothing to do." [exit] → not installed → download + link onto PATH → "Set up this machine as a box host now? [y/N]" y → runs setup-host (re-exec and all) → proper box host n → "run box setup-host when you want this to host boxes" ``` Re-running is now a **no-op the moment box is installed**, exactly as you said — that dissolves the whole "curl clobbered my install / rebuilt the stack under my boxes" class, and it does it *without* the installer having to reason about versions or enumerate boxes at all. You cannot lose a box to an install that refuses to touch an existing one. Upgrading is explicit: uninstall, install fresh. I replaced last round's version-diff refusal with this; it is simpler and covers the same ground. `#67` (migrate) and `#70` (export) carry the "make upgrade painless" work. Verified on the real host, all five paths: cancel; fresh install; no-op re-run; and both prompts driven through a real pty (`y`/`n` and `y`-then-`n`). ### One decision you did not specify, so I want it visible **`curl … | bash` has no stdin to read** — the script *is* stdin. So the prompts read `/dev/tty` directly. On a machine with no terminal at all (CI, an image build, the drill), there is nobody to ask, so `BOX_YES=1` means "assume yes to every prompt" and is required to proceed unattended; without it, it refuses rather than invent your consent. `BOX_SKIP_SETUP_HOST=1` answers just the host-setup prompt "no". If you would rather the no-tty default be "yes" instead of "refuse", that is a one-line change — say the word. ## The two drill FAILs — one real-ish, one a false alarm **`blank box has NO egress` — false alarm, and it is the *probe* that is wrong, not the box.** I mistrusted it because DNS passed on the same box, so I probed egress 8 times in a row on a fresh blank box: ``` round 1: ping=FAIL curl=000 dns=ok round 2: ping=ok curl=200 dns=ok round 3: ping=ok curl=200 dns=ok round 4: ping=FAIL curl=000 dns=ok … round 8: ping=ok curl=200 dns=ok ``` Egress **flaps** — ~50% loss to the public internet, while DNS (which resolves at the gateway, 10.88.0.1, one hop away) is rock solid. That is the signature of **nested virtio networking**: I am running the drill inside a KVM guest, and the boxes are VMs nested inside *that*, so box→internet is double-NATed through two virtio hops and drops packets. On your real metal this almost certainly will not reproduce. But it did expose a genuine drill weakness: `box_pings tpl 1.1.1.1` is a **single** ICMP shot, so one dropped packet = a red FAIL. Worth making that probe retry (say 3 tries) so a lossy link does not cost a false isolation-parity failure. Happy to do it here or as its own issue — your call, since it is drill-only. **`codex: codex --version FAILED` — real, but pre-existing and not this PR.** The drill's own label calls it "the claude template's #15 bug": `codex` is not on `PATH` (nor on disk) after a cold mint even though cloud-init reported `status: done`. It is a template/PATH issue, untouched by anything here. Flagging, not fixing, unless you want it chased. ## And a real bug the real host surfaced (already pushed, 043e556) `box-firewall.service` was `Type=oneshot` with no `RemainAfterExit=yes`, so a perfectly healthy firewall reported `inactive (dead)` — and drill.sh sends you to exactly that command to diagnose the firewall. Fixed; `setup-host.sh`'s own comment had already assumed the line was present. Details two comments up. ## Status of the thing I kept caveating The **entire #63/#64 chain is now proven on real hardware**, not stubs: I reproduced the old code's `permission denied` failure on this host, watched the new code one-run through it, and the drill's stack assertion passes green. The one gap left is the `claude` template's ~10-min cold mint, which crashes here on a nested-VM `vsock` reset at 7GiB RAM — environmental, not code. Everything up to it is green.

danmt commented

2026-07-17 16:36:54 +00:00

(Migrated from github.com)

There should be a copy-pasteable command snippet in the description of this PR that can be used to probe the changes. Running drill against this branch.

No reviewers

codex-bot-andresmgsl

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/box#66

No description provided.

Make host setup complete in one run, and let the installer run it #66

#63 — setup-host stopped halfway

#64 — the installer now finishes the job

Verified

1. Root without sudo — confirmed, exit 127

2. The drill — confirmed, and it mattered more than the count

Open question for @danielarturomt — your call, not mine

"What are the actual consequences of reconfiguring the network of the previous boxes?"

The hatch (this PR)

One correction to the recipe — it loses data

First: I owe you a correction on the snapshot claim

Second: you were right that I could run the drill here, and it changes what I can claim

The #63 bug, reproduced on real hardware

A real bug the drill found (043e556)

On the blip

Two drill FAILs — neither is from this PR, reporting them straight

The installer, as you described it

One decision you did not specify, so I want it visible

The two drill FAILs — one real-ish, one a false alarm

And a real bug the real host surfaced (already pushed, 043e556)

Status of the thing I kept caveating

#63 — `setup-host` stopped halfway

A real bug the drill found (`043e556`)

And a real bug the real host surfaced (already pushed, `043e556`)