box/host/migrate-host.sh

164 lines
8.2 KiB
Bash
Raw Normal View History

#!/usr/bin/env bash
# migrate-host.sh — move a host from the pre-0.4.0 'claudebox' stack to 'box'.
#
# The zero-ceremony transition is just install.sh + setup-host.sh: that leaves
# a DUAL-STACK host where legacy boxes (tag user.claudebox=1, claudenet/10.87,
feat(setup-host): refuse a claimed subnet, and BOX_SUBNET to move off one (#80) Run inside a box, setup-host built a nested boxnet claiming the exact subnet and gateway of the guest's own uplink: the guest then held 10.88.0.1 as a LOCAL address while its default route pointed at 10.88.0.1 as its GATEWAY, carried two connected routes for the subnet, and suffered intermittent, self-recovering egress blackouts (~24-36 s, roughly hourly) that no host-side check could attribute. The flagship use case funnels agents toward exactly this — working on box, in a box — so the guard, not the operator, has to catch it. setup-host now scans the target subnet BEFORE any mutation: the default route's gateway inside it (the smoking gun — that is this machine's own uplink), or any non-boxnet interface holding an address in it, refuses with the way out named (BOX_SUBNET). A prior boxnet owning the subnet is the one claimant that is NOT a collision — the legitimate re-run — and an existing bridge on a DIFFERENT subnet refuses too: setup-host converges a bridge, it never re-addresses one under live leases. BOX_SUBNET=<a.b.c.0/24> (validated at the gate, alongside BOX_DNS) moves the whole stack from one place: · the bridge address derives (ipv4.address=<gw>/24) · the ACL's gateway carve-out derives — and the ruleset is now converged via 'network acl edit', not created once: the #80 escape hatch (moving the bridge) left the old /32 behind, stranding box DNS inside the 10.0.0.0/8 drop, which presents as a dead resolver, never as a stale ACL · box-firewall reads the gateway off the live bridge ('|| true' because under pipefail an absent bridge would kill the script instead of answering "no bridge yet") · the drill, multiuser rehearsal and migrate-host probes derive the prefix from 'incus network get boxnet ipv4.address' instead of hardcoding 10.88 — a BOX_SUBNET host must not fail its own rehearsals Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 19:46:58 +00:00
# claude-dev) keep working while new mints land on boxnet. This script is
# the two things that path does not do:
#
# migrate-host.sh --box <name> re-home ONE legacy box onto the new stack
# migrate-host.sh --all-boxes re-home every legacy box
# migrate-host.sh --retire-legacy remove the legacy stack (refuses while any
# legacy box still exists)
#
# One action per invocation, idempotent, loud about what it did. Re-homing
# PRESERVES the box's authed state (Claude login, git creds — the expensive
# thing); it does not re-mint. The order is load-bearing: tag first (additive,
# reversible), profile last, and verify the box works on its new leg BEFORE
# calling it migrated — a box must never end up tagless or profileless.
#
# NOT 'set -e' around the per-box work: a box that fails one step is reported
# and skipped, not a crash that abandons the rest mid-migration.
#
# The report idiom is 'action && say "did X" || warn/die': say and warn always
# return 0, so the C-may-run-when-A-is-true trap SC2015 warns about cannot fire
# on those lines (same reasoning as drill.sh's ok/no).
# shellcheck disable=SC2015
set -u
say() { printf 'migrate: %s\n' "$*"; }
warn() { printf 'migrate: WARNING: %s\n' "$*" >&2; }
die() { printf 'migrate: ERROR: %s\n' "$*" >&2; exit 1; }
mode=""
target=""
while [ $# -gt 0 ]; do
case "$1" in
--box) mode=box; target="${2:-}"; shift 2 || die "--box needs a name" ;;
--all-boxes) mode=all; shift ;;
--retire-legacy) mode=retire; shift ;;
-h|--help) sed -n '2,20p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
*) die "unknown argument: $1 (see --help)" ;;
esac
done
[ -n "$mode" ] || die "pick one: --box <name> | --all-boxes | --retire-legacy"
command -v incus >/dev/null || die "incus is not installed on this host"
# The new stack must exist before any box can be re-homed onto it. setup-host
# creates it; refuse rather than move a box onto a network that isn't there.
require_new_stack() {
incus network show boxnet >/dev/null 2>&1 || die "boxnet does not exist — run host/setup-host.sh first"
incus profile show box-net >/dev/null 2>&1 || die "box-net profile does not exist — run host/setup-host.sh first"
}
fix: run 17's four real findings — migrate retire, expose proxy, wait_box, grok PATH The first drill run where every failure was the RELEASE CODE, not the environment. 71 passed, 5 failed; all five traced to four bugs: 1. migrate-host --retire-legacy could NEVER succeed. Re-homing ADDS user.box=1 but never removed user.claudebox=1, and legacy_boxes() counted the old tag — so retire saw its own freshly-migrated box as un-migrated and refused forever ('legacy boxes still exist: legacybox'), leaving claudenet + claude-dev behind. Now: a verified re-home drops the legacy tag LAST (after the move is proven, so a failure anywhere above still leaves the box valid under one tag or the other), and legacy_boxes() ignores boxes already carrying user.box=1. 2. box expose died with a bare 'could not add the proxy device' — it swallowed incus's reason, exactly the sin this repo keeps punishing. Now it prints incus's error. And the mechanism is corrected: a VM's proxy needs NAT mode, which requires a static NIC address, so expose pins the box's current lease first (which also fixes the restart caveat — the exposure no longer points at a lease the box may lose). 3. wait_box's 2-minute window was too short: the legacy box was declared dead and then every migration check against it passed. 4 minutes. 4. The grok template hunted for a regular file named exactly 'grok-build' under /home/grok and found nothing — an installer's drop may be a SYMLINK, and its binary name is upstream's to choose. Now it tries the plausible names and paths, falls back to any executable grok*, links both names, and SAYS what it found — or dumps what the installer actually left when it finds nothing. The drill likewise dumps the on-disk evidence and the cloud-init log on a --version failure instead of discarding the box.
2026-07-14 19:56:17 +00:00
# A box is LEGACY only if it still carries the old tag and has NOT been
# re-homed. Counting every user.claudebox=1 box was a bug: re-homing ADDS
# user.box=1 without removing the old tag, so --retire-legacy saw its own
# migrated boxes as un-migrated and refused forever.
legacy_boxes() {
local b
for b in $(incus list "user.claudebox=1" -f csv -c n 2>/dev/null); do
[ "$(incus config get "$b" user.box 2>/dev/null)" = 1 ] && continue # already re-homed
echo "$b"
done
}
# Re-home one box. Legacy boxes are all claude boxes (the only template the old
refactor(templates): the tenant seeds carry rig's -box family suffix rig is growing a second family of roles, and once a 'staging' role can mean either a fleet machine or a box tenant, the bare name stops naming anything. rig's answer is a suffix on the role (heavy-duty/rig#76): '-server' for fleet machines, '-box' for box tenants. box's answer is that a template keeps being named for the role it converges, so the tenant templates move with it: claude -> claude-box codex -> codex-box grok -> grok-box staging -> staging-box Templates are the only surface that spells a rig role out loud (BOX_BOOTSTRAP_ROLE, auto-run at mint since #81), so a directory that says one thing and a role key that says another is a trap with a 15-minute fuse: it mints clean and dies at convergence. Renamed with 'git mv' so the history of each seed follows it. 'blank' keeps its name. It seeds no tenant role and sets no BOX_BOOTSTRAP_ROLE, so it has nothing to agree with — renaming it would only churn the default template's name for symmetry's sake. Two namespaces move apart here and only one of them moved: the template name and the role are now claude-box, while the seed USER stays 'claude' — that is the user rig's role converges and the one 'box shell' lands in. test/cli.sh pins the pair per tenant rather than each half alone, because a later rename that moves one and forgets the other mints a box whose role dies looking for a user nobody created. drill.sh keeps its bare box NAMES ('codex', 'grok' — what the pre-flight banner announces and what teardown deletes) and only moves the --template it passes. The mint-time hints in cmd_new match both spellings of user.box.template, and that is not an alias for the role: 'rig bootstrap claude' is gone and nothing here softens the cut. The stamp is a fact about an INSTANCE, written at its own mint time and carried forward by every clone; refusing the old spelling would cut nothing over and only drop the login hint on boxes that predate today — the same reason user.claudebox is honored everywhere else. migrate-host.sh stamps re-homed legacy boxes claude-box, the name the template has today, so a re-homed box looks like a fresh mint rather than a fossil. Ordered AFTER rig's rename, and that is not a preference. The seeds install rig from RIG_REPO/RIG_REF, defaulting to heavy-duty/rig@main and unpinned until rig#32's releases, so these templates ask whatever main happens to be for 'rig bootstrap claude-box'. Against a pre-rename rig that role does not exist and cmd_new refuses to call the box ready. Merged in the other order the window closes instead of opening: rig's cut is hard, with no aliases, so the day it lands every unmerged box seed naming a bare role is the broken one. Closes #123 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 00:01:29 +00:00
# tool minted), so the new metadata is the claude-box template's. The stamp
# names the template as it is called TODAY, not as it was called when the box
# was minted: it is what 'box shell' and the mint hints read, so a re-homed
# box should look like a fresh claude-box mint, not like a fossil (rig#76's
# family suffix — the template is named for the role it converges).
rehome_one() {
local b="$1" st
incus config get "$b" user.claudebox >/dev/null 2>&1 || { warn "$b is not a legacy box (no user.claudebox tag) — skipping"; return 1; }
if [ "$(incus config get "$b" user.box 2>/dev/null)" = 1 ]; then
say "$b already carries user.box=1 — already re-homed, skipping"; return 0
fi
say "re-homing $b"
# 1. TAG FIRST — additive and reversible. A box that stops here is still a
# valid legacy box (the old tag is untouched) AND now a new one.
refactor(templates): the tenant seeds carry rig's -box family suffix rig is growing a second family of roles, and once a 'staging' role can mean either a fleet machine or a box tenant, the bare name stops naming anything. rig's answer is a suffix on the role (heavy-duty/rig#76): '-server' for fleet machines, '-box' for box tenants. box's answer is that a template keeps being named for the role it converges, so the tenant templates move with it: claude -> claude-box codex -> codex-box grok -> grok-box staging -> staging-box Templates are the only surface that spells a rig role out loud (BOX_BOOTSTRAP_ROLE, auto-run at mint since #81), so a directory that says one thing and a role key that says another is a trap with a 15-minute fuse: it mints clean and dies at convergence. Renamed with 'git mv' so the history of each seed follows it. 'blank' keeps its name. It seeds no tenant role and sets no BOX_BOOTSTRAP_ROLE, so it has nothing to agree with — renaming it would only churn the default template's name for symmetry's sake. Two namespaces move apart here and only one of them moved: the template name and the role are now claude-box, while the seed USER stays 'claude' — that is the user rig's role converges and the one 'box shell' lands in. test/cli.sh pins the pair per tenant rather than each half alone, because a later rename that moves one and forgets the other mints a box whose role dies looking for a user nobody created. drill.sh keeps its bare box NAMES ('codex', 'grok' — what the pre-flight banner announces and what teardown deletes) and only moves the --template it passes. The mint-time hints in cmd_new match both spellings of user.box.template, and that is not an alias for the role: 'rig bootstrap claude' is gone and nothing here softens the cut. The stamp is a fact about an INSTANCE, written at its own mint time and carried forward by every clone; refusing the old spelling would cut nothing over and only drop the login hint on boxes that predate today — the same reason user.claudebox is honored everywhere else. migrate-host.sh stamps re-homed legacy boxes claude-box, the name the template has today, so a re-homed box looks like a fresh mint rather than a fossil. Ordered AFTER rig's rename, and that is not a preference. The seeds install rig from RIG_REPO/RIG_REF, defaulting to heavy-duty/rig@main and unpinned until rig#32's releases, so these templates ask whatever main happens to be for 'rig bootstrap claude-box'. Against a pre-rename rig that role does not exist and cmd_new refuses to call the box ready. Merged in the other order the window closes instead of opening: rig's cut is hard, with no aliases, so the day it lands every unmerged box seed naming a bare role is the broken one. Closes #123 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 00:01:29 +00:00
incus config set "$b" user.box=1 user.box.template=claude-box user.box.user=claude \
|| { warn "$b: could not set new metadata — left untouched"; return 1; }
# 2. Stop, reassign the profile (this is the network move), restart. Incus
# won't reassign a profile on a running instance's NIC cleanly, and the
# box needs a fresh DHCP lease on boxnet anyway.
st="$(incus list "$b" -f csv -c s 2>/dev/null | head -1)"
case "$st" in RUNNING|Running|running) incus stop "$b" >/dev/null 2>&1 || warn "$b: stop was not clean" ;; esac
incus profile assign "$b" box-net \
|| { warn "$b: profile assign failed — it still has user.box=1 but is on the OLD network; fix by hand"; return 1; }
incus start "$b" >/dev/null 2>&1 || { warn "$b: did not restart — start it by hand"; return 1; }
# 3. VERIFY THE EFFECT, not the exit codes (the whole repo's lesson). The box
feat(setup-host): refuse a claimed subnet, and BOX_SUBNET to move off one (#80) Run inside a box, setup-host built a nested boxnet claiming the exact subnet and gateway of the guest's own uplink: the guest then held 10.88.0.1 as a LOCAL address while its default route pointed at 10.88.0.1 as its GATEWAY, carried two connected routes for the subnet, and suffered intermittent, self-recovering egress blackouts (~24-36 s, roughly hourly) that no host-side check could attribute. The flagship use case funnels agents toward exactly this — working on box, in a box — so the guard, not the operator, has to catch it. setup-host now scans the target subnet BEFORE any mutation: the default route's gateway inside it (the smoking gun — that is this machine's own uplink), or any non-boxnet interface holding an address in it, refuses with the way out named (BOX_SUBNET). A prior boxnet owning the subnet is the one claimant that is NOT a collision — the legitimate re-run — and an existing bridge on a DIFFERENT subnet refuses too: setup-host converges a bridge, it never re-addresses one under live leases. BOX_SUBNET=<a.b.c.0/24> (validated at the gate, alongside BOX_DNS) moves the whole stack from one place: · the bridge address derives (ipv4.address=<gw>/24) · the ACL's gateway carve-out derives — and the ruleset is now converged via 'network acl edit', not created once: the #80 escape hatch (moving the bridge) left the old /32 behind, stranding box DNS inside the 10.0.0.0/8 drop, which presents as a dead resolver, never as a stale ACL · box-firewall reads the gateway off the live bridge ('|| true' because under pipefail an absent bridge would kill the script instead of answering "no bridge yet") · the drill, multiuser rehearsal and migrate-host probes derive the prefix from 'incus network get boxnet ipv4.address' instead of hardcoding 10.88 — a BOX_SUBNET host must not fail its own rehearsals Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 19:46:58 +00:00
# must be on boxnet's subnet (read off the network — BOX_SUBNET moves it,
# #80) and actually resolve+reach the internet on its new leg before we
# call it migrated.
local _i ip pfx
pfx="$(incus network get boxnet ipv4.address 2>/dev/null | cut -d/ -f1)"; pfx="${pfx%.*}."
ip=""
for _i in $(seq 1 30); do
ip="$(incus exec "$b" -- ip -4 -o addr show scope global </dev/null 2>/dev/null \
feat(setup-host): refuse a claimed subnet, and BOX_SUBNET to move off one (#80) Run inside a box, setup-host built a nested boxnet claiming the exact subnet and gateway of the guest's own uplink: the guest then held 10.88.0.1 as a LOCAL address while its default route pointed at 10.88.0.1 as its GATEWAY, carried two connected routes for the subnet, and suffered intermittent, self-recovering egress blackouts (~24-36 s, roughly hourly) that no host-side check could attribute. The flagship use case funnels agents toward exactly this — working on box, in a box — so the guard, not the operator, has to catch it. setup-host now scans the target subnet BEFORE any mutation: the default route's gateway inside it (the smoking gun — that is this machine's own uplink), or any non-boxnet interface holding an address in it, refuses with the way out named (BOX_SUBNET). A prior boxnet owning the subnet is the one claimant that is NOT a collision — the legitimate re-run — and an existing bridge on a DIFFERENT subnet refuses too: setup-host converges a bridge, it never re-addresses one under live leases. BOX_SUBNET=<a.b.c.0/24> (validated at the gate, alongside BOX_DNS) moves the whole stack from one place: · the bridge address derives (ipv4.address=<gw>/24) · the ACL's gateway carve-out derives — and the ruleset is now converged via 'network acl edit', not created once: the #80 escape hatch (moving the bridge) left the old /32 behind, stranding box DNS inside the 10.0.0.0/8 drop, which presents as a dead resolver, never as a stale ACL · box-firewall reads the gateway off the live bridge ('|| true' because under pipefail an absent bridge would kill the script instead of answering "no bridge yet") · the drill, multiuser rehearsal and migrate-host probes derive the prefix from 'incus network get boxnet ipv4.address' instead of hardcoding 10.88 — a BOX_SUBNET host must not fail its own rehearsals Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 19:46:58 +00:00
| awk -v p="$pfx" '{for(i=1;i<NF;i++) if($i=="inet" && index($(i+1),p)==1){split($(i+1),a,"/"); print a[1]; exit}}')"
[ -n "$ip" ] && break
sleep 2
done
feat(setup-host): refuse a claimed subnet, and BOX_SUBNET to move off one (#80) Run inside a box, setup-host built a nested boxnet claiming the exact subnet and gateway of the guest's own uplink: the guest then held 10.88.0.1 as a LOCAL address while its default route pointed at 10.88.0.1 as its GATEWAY, carried two connected routes for the subnet, and suffered intermittent, self-recovering egress blackouts (~24-36 s, roughly hourly) that no host-side check could attribute. The flagship use case funnels agents toward exactly this — working on box, in a box — so the guard, not the operator, has to catch it. setup-host now scans the target subnet BEFORE any mutation: the default route's gateway inside it (the smoking gun — that is this machine's own uplink), or any non-boxnet interface holding an address in it, refuses with the way out named (BOX_SUBNET). A prior boxnet owning the subnet is the one claimant that is NOT a collision — the legitimate re-run — and an existing bridge on a DIFFERENT subnet refuses too: setup-host converges a bridge, it never re-addresses one under live leases. BOX_SUBNET=<a.b.c.0/24> (validated at the gate, alongside BOX_DNS) moves the whole stack from one place: · the bridge address derives (ipv4.address=<gw>/24) · the ACL's gateway carve-out derives — and the ruleset is now converged via 'network acl edit', not created once: the #80 escape hatch (moving the bridge) left the old /32 behind, stranding box DNS inside the 10.0.0.0/8 drop, which presents as a dead resolver, never as a stale ACL · box-firewall reads the gateway off the live bridge ('|| true' because under pipefail an absent bridge would kill the script instead of answering "no bridge yet") · the drill, multiuser rehearsal and migrate-host probes derive the prefix from 'incus network get boxnet ipv4.address' instead of hardcoding 10.88 — a BOX_SUBNET host must not fail its own rehearsals Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 19:46:58 +00:00
[ -n "$ip" ] || { warn "$b: never got a boxnet address after restart — re-home INCOMPLETE, inspect: incus console $b"; return 1; }
if incus exec "$b" -- getent hosts deb.debian.org </dev/null >/dev/null 2>&1; then
fix: run 17's four real findings — migrate retire, expose proxy, wait_box, grok PATH The first drill run where every failure was the RELEASE CODE, not the environment. 71 passed, 5 failed; all five traced to four bugs: 1. migrate-host --retire-legacy could NEVER succeed. Re-homing ADDS user.box=1 but never removed user.claudebox=1, and legacy_boxes() counted the old tag — so retire saw its own freshly-migrated box as un-migrated and refused forever ('legacy boxes still exist: legacybox'), leaving claudenet + claude-dev behind. Now: a verified re-home drops the legacy tag LAST (after the move is proven, so a failure anywhere above still leaves the box valid under one tag or the other), and legacy_boxes() ignores boxes already carrying user.box=1. 2. box expose died with a bare 'could not add the proxy device' — it swallowed incus's reason, exactly the sin this repo keeps punishing. Now it prints incus's error. And the mechanism is corrected: a VM's proxy needs NAT mode, which requires a static NIC address, so expose pins the box's current lease first (which also fixes the restart caveat — the exposure no longer points at a lease the box may lose). 3. wait_box's 2-minute window was too short: the legacy box was declared dead and then every migration check against it passed. 4 minutes. 4. The grok template hunted for a regular file named exactly 'grok-build' under /home/grok and found nothing — an installer's drop may be a SYMLINK, and its binary name is upstream's to choose. Now it tries the plausible names and paths, falls back to any executable grok*, links both names, and SAYS what it found — or dumps what the installer actually left when it finds nothing. The drill likewise dumps the on-disk evidence and the cloud-init log on a --version failure instead of discarding the box.
2026-07-14 19:56:17 +00:00
# LAST, and only once the move is VERIFIED: drop the legacy tag. Until this
# point the box wears both tags, so a failure anywhere above leaves it a
# valid box under one name or the other — never orphaned. Now it is simply
# a box, and --retire-legacy can see the old stack is empty.
incus config unset "$b" user.claudebox >/dev/null 2>&1 \
|| warn "$b: migrated, but the legacy tag could not be removed — retire-legacy will still see it"
say "$b re-homed: on boxnet ($ip), resolves + reachable, authed state preserved"
return 0
fi
warn "$b is on boxnet ($ip) but cannot resolve — check the new stack's resolver (box doctor)"
return 1
}
case "$mode" in
box)
[ -n "$target" ] || die "--box needs a name"
require_new_stack
rehome_one "$target"
;;
all)
require_new_stack
boxes="$(legacy_boxes)"
[ -n "$boxes" ] || { say "no legacy boxes to re-home"; exit 0; }
rc=0
for b in $boxes; do rehome_one "$b" || rc=1; done
[ "$rc" = 0 ] && say "all legacy boxes re-homed" || warn "some boxes need attention (above)"
exit "$rc"
;;
retire)
# Refuse while any legacy box still references the old stack — removing an
# in-use profile/network fails anyway, and a half-removed stack is worse
# than an intact one.
remaining="$(legacy_boxes)"
if [ -n "$remaining" ]; then
die "legacy boxes still exist: $(echo "$remaining" | tr '\n' ' ')
re-home them first (--all-boxes), or delete them, then retire."
fi
say "no legacy boxes remain — removing the legacy stack"
incus profile delete claude-dev >/dev/null 2>&1 && say "deleted profile claude-dev"
incus network delete claudenet >/dev/null 2>&1 && say "deleted network claudenet"
incus network acl delete claude-isolate >/dev/null 2>&1 && say "deleted ACL claude-isolate"
sudo systemctl disable --now claudebox-firewall.service >/dev/null 2>&1 && say "disabled claudebox-firewall.service"
sudo rm -f /etc/systemd/system/claudebox-firewall.service /usr/local/sbin/claudebox-firewall
sudo systemctl daemon-reload
sudo nft delete table inet claudebox >/dev/null 2>&1 && say "deleted nft table inet claudebox"
sudo nft delete table bridge claudebox >/dev/null 2>&1 && say "deleted nft table bridge claudebox"
# Assert the absence — don't trust the removals' exit codes.
left=""
incus network show claudenet >/dev/null 2>&1 && left="$left claudenet"
incus profile show claude-dev >/dev/null 2>&1 && left="$left claude-dev"
sudo nft list table bridge claudebox >/dev/null 2>&1 && left="$left nft-bridge"
[ -z "$left" ] && say "legacy stack retired — this host is now single-stack (box only)" \
|| die "legacy stack NOT fully removed:$left"
;;
esac