From 8ab9b38ba26e3f982d00b4bac34ea71679d0ac54 Mon Sep 17 00:00:00 2001 From: dan-claude-bot Date: Sun, 19 Jul 2026 12:28:22 +0000 Subject: [PATCH 1/4] =?UTF-8?q?fix:=20narrate=20and=20time-box=20the=20inc?= =?UTF-8?q?us=20launch=20=E2=80=94=20a=20wedge=20fails=20loudly,=20not=20f?= =?UTF-8?q?orever=20(#93)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Twice in the 2026-07-19 release drill (Debian 13, Incus 6.x, /dev/kvm present, images cached), the child 'incus launch' under 'box new' wedged with no server-side operation: 'incus operation list' empty, the instance never created, the daemon journal quiet — one wedge ran 56 minutes before being killed by hand, the other was killed by a 540s wrapper. An immediate retry of the identical command succeeded in ~2-3 minutes, both times. box inherited that as an indefinite silent hang, indistinguishable from a cold mint working. The mint now prints "launching instance ..." before the call, and the call rides 'timeout -k 5 $BOX_LAUNCH_TIMEOUT' (seconds, default 600 — generous: the coldest measured mint is minutes, never an hour; overridable the same way BOX_CPU/BOX_MEMORY are), with stdin pinned per drill/RUNS.md trap 13. When the budget fires (124, or 137 when the KILL was needed) the failure says exactly what was measured — the client wedged with no server-side operation, an immediate retry has been observed to succeed — and points at 'box doctor' for host state. A non-timeout launch failure still surfaces incus's own stderr. The --from clone path is untouched: 'incus copy' of a local instance is a different operation and has never been observed to wedge this way. Proven the way the other mint-path guards are (a daemon-free run cannot mint): test/cli.sh greps that the narration orders before the launch, that the launch sits under 'timeout -k' with the BOX_LAUNCH_TIMEOUT budget and pinned stdin, and that the wedge message carries the retry hint, the doctor, and #93 — plus a live shim-incus drive of all three exits (wedge, plain refusal, success) during development. Fixes #93 Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 18 ++++++++++++++++++ bin/box | 33 +++++++++++++++++++++++++++++++-- test/cli.sh | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 81 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 642a2d6..034b727 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -204,6 +204,24 @@ which records not just what changed but what each drill run proved. ### Fixed +- **A wedged `incus launch` fails loudly, not forever — the mint's launch + phase is narrated and time-boxed** (#93) — twice in the 2026-07-19 + release drill (Debian 13, Incus 6.x, /dev/kvm present, images cached), + the child `incus launch` under `box new` hung with *no server-side + operation*: `incus operation list` empty, the instance never created, the + daemon journal quiet — one wedge ran 56 minutes before being killed by + hand, and an immediate retry of the identical command succeeded in + minutes, both times. `box new` inherited that as an indefinite silent + hang, indistinguishable from a cold mint working. It now prints + `launching instance …` before the call, and the call rides + `timeout -k 5 $BOX_LAUNCH_TIMEOUT` (seconds, default 600 — generous: the + coldest measured mint is minutes, never an hour; the same scripting-knob + shape as `BOX_CPU`/`BOX_MEMORY`), with stdin pinned per the drill's own + trap list. On the budget firing it says exactly what was measured — the + client wedged with no server-side operation, an immediate retry has been + observed to succeed — and points at `box doctor` for the host. The + `--from` clone path is untouched: `incus copy` of a local instance is a + different operation and has never been observed to wedge this way. - **UFW's gateway carve-out converges with the bridge, and the doctor can see it** (the #86 review's blind spot) — `box-firewall` gated its whole UFW block behind "a `DENY on boxnet` rule exists", pinning every UFW host diff --git a/bin/box b/bin/box index a669144..7c03554 100755 --- a/bin/box +++ b/bin/box @@ -1017,14 +1017,43 @@ cmd_new() { # The template's identity is stamped ONTO the instance: which template, # which user. 'incus copy' preserves user.* keys (audit B2), so a clone # knows what it is without ever consulting the template again. - incus launch "$T_IMAGE" "$instance" --profile box-net \ + # + # The launch is narrated and TIME-BOXED (#93). Twice in the 2026-07-19 + # release drill the child 'incus launch' wedged before the create was + # even accepted — 'incus operation list' empty, the instance never + # existed, the daemon journal quiet — once for 56 minutes until killed + # by hand. Without a line here that wedge reads exactly like a cold mint + # working; without a budget it lasts forever. Ten minutes is generous — + # the coldest measured mint (first VM on a fresh pool, see wait_agent) + # is minutes, never an hour — and BOX_LAUNCH_TIMEOUT (seconds) overrides + # it, the same scripting knob shape as BOX_CPU / BOX_MEMORY. The drill's + # lore applies verbatim (RUNS.md trap 13): 'timeout -k' so a launch that + # shrugs off TERM still dies, and stdin pinned like every other + # non-interactive incus call — only shell/exec/tmux may own the terminal. + local budget="${BOX_LAUNCH_TIMEOUT:-600}" rc=0 + echo "box: launching instance $instance (incus launch, $m mode)..." + timeout -k 5 "$budget" incus launch "$T_IMAGE" "$instance" --profile box-net \ --config user.box=1 \ --config user.box.template="$t" \ --config user.box.user="$T_USER" \ --config limits.cpu="$T_CPU" \ --config limits.memory="$T_MEMORY" \ --config cloud-init.user-data="$(render_userdata "$root/templates/$t/user-data.yaml")" \ - "${extra[@]}" + "${extra[@]}" &2 + echo "box: 'incus launch' WEDGED — killed after ${budget}s, and the instance was never created." >&2 + echo "box: this is the #93 failure: the incus client hangs with NO server-side operation" >&2 + echo "box: ('incus operation list' is empty, the daemon journal is quiet). An immediate" >&2 + echo "box: retry of the exact same 'box new' has been observed to succeed, both times it" >&2 + echo "box: was measured. If it persists, diagnose the host: box doctor" >&2 + echo "box: (a genuinely slower mint can raise the budget: BOX_LAUNCH_TIMEOUT=)" >&2 + die "incus launch wedged with no server-side operation — retry the same command (#93)" + elif [ "$rc" -ne 0 ]; then + # Not a wedge: incus refused and said why on stderr, right above. + die "incus launch failed (exit $rc)" + fi wait_agent "$instance" echo "box: waiting for phase-1 (cloud-init)..." echo "box: (its full narration, live: incus exec $name -- tail -f /var/log/cloud-init-output.log)" diff --git a/test/cli.sh b/test/cli.sh index a287993..edc3896 100644 --- a/test/cli.sh +++ b/test/cli.sh @@ -373,6 +373,38 @@ check "new: the auto-run sits under the T_BOOTSTRAP_ROLE guard" 0 "" bash -c ' check "new: a failed tenant role names the re-run (the role converges)" 0 "" bash -c ' awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \ | grep -q "sudo rig bootstrap"' + +# The launch phase, narrated and time-boxed (#93) — grepped the way the other +# mint-path guards are (a daemon-free run cannot mint). Twice in the +# 2026-07-19 release drill the child 'incus launch' wedged silently before +# the create was even accepted, once for 56 minutes. The narration must order +# BEFORE the launch call (a wedge after the line is visible at a glance; a +# wedge before it is the old silent hang), the call itself must sit under +# 'timeout -k' with the BOX_LAUNCH_TIMEOUT override and pinned stdin (RUNS.md +# trap 13: bare 'timeout N' cannot kill an incus call that owns a TTY), and +# the budget's failure must be LOUD — no server-side operation, the measured +# retry-succeeds hint, and the doctor as the next move. +# shellcheck disable=SC2016 # the $-strings are literals in the target file +check "new: the launch narration orders before incus launch (#93)" 0 "" bash -c ' + fn="$(awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box")" + say="$(printf "%s\n" "$fn" | grep -n "launching instance" | head -1 | cut -d: -f1)" + run="$(printf "%s\n" "$fn" | grep -n "timeout -k.*incus launch" | head -1 | cut -d: -f1)" + [ -n "$say" ] && [ -n "$run" ] && [ "$say" -lt "$run" ]' +check "new: incus launch is time-boxed (timeout -k on the budget)" 0 "" bash -c ' + awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \ + | grep "timeout -k" | grep "budget" | grep -q "incus launch"' +check "new: the budget is BOX_LAUNCH_TIMEOUT, default 600s (the BOX_CPU knob shape)" 0 "" bash -c ' + awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \ + | grep "budget=" | grep -q "BOX_LAUNCH_TIMEOUT:-600"' +check "new: the launch pins stdin (RUNS.md trap 13)" 0 "" bash -c ' + awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \ + | grep -F "extra[@]" | grep -qF " Date: Sun, 19 Jul 2026 12:52:41 +0000 Subject: [PATCH 2/4] fix: the timeout path probes the instance, tells the two stories apart, and cleans up MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Round-1 consensus on #94: timeout only proves the CLIENT overran the budget. incus launch is create-then-start, so a slow-but-progressing launch may already have registered the instance — the old message claimed 'never created' unconditionally and the advised retry would collide with 'Instance already exists'. The 124/137 path now probes 'incus info', narrates the branch it found (true #93 wedge vs slow-launch overrun), best-effort 'incus delete --force's either way so the retry is clean in both worlds, hedges 137 as possibly an outside kill, and BOX_LAUNCH_TIMEOUT is documented in 'box help new' beside the other knobs. Both branches driven live against a shim incus; four new grep-proof checks pin the probe, the cleanup, the overrun story, and the help text. Co-Authored-By: Claude Fable 5 --- bin/box | 36 +++++++++++++++++++++++++++++------- test/cli.sh | 21 +++++++++++++++++++-- 2 files changed, 48 insertions(+), 9 deletions(-) diff --git a/bin/box b/bin/box index 7c03554..ad26a7a 100755 --- a/bin/box +++ b/bin/box @@ -267,6 +267,9 @@ template's box.env, then defaults. Flags shape a fresh mint only — a --from clone carries its source's resources. Resources are all a flag can touch: there is no flag for a network or a security key, on purpose. +BOX_LAUNCH_TIMEOUT= (default 600) bounds the 'incus launch' call — +a launch that overruns it fails loudly instead of hanging forever (#93). + box new --name scratch # blank, the default box new --name work --template claude box new --name lean --template claude --cpu 2 --memory 3GiB @@ -1040,16 +1043,35 @@ cmd_new() { --config limits.memory="$T_MEMORY" \ --config cloud-init.user-data="$(render_userdata "$root/templates/$t/user-data.yaml")" \ "${extra[@]}" &2 - echo "box: 'incus launch' WEDGED — killed after ${budget}s, and the instance was never created." >&2 - echo "box: this is the #93 failure: the incus client hangs with NO server-side operation" >&2 - echo "box: ('incus operation list' is empty, the daemon journal is quiet). An immediate" >&2 - echo "box: retry of the exact same 'box new' has been observed to succeed, both times it" >&2 - echo "box: was measured. If it persists, diagnose the host: box doctor" >&2 + # timeout only proves the CLIENT overran the budget. 'incus launch' is + # create-then-start, so a slow-but-progressing launch (first mint + # pulling an uncached image, say) may already have REGISTERED the + # instance — in which case "never created, retry" would be exactly + # wrong: the retry collides with 'Instance already exists'. Probe, say + # which case this is, and best-effort delete either way (a no-op on + # the true #93 wedge, the cleanup on an overrun; also covers a create + # that lands in the race between the probe and the delete) so the + # retry advice below is safe in BOTH worlds. Review consensus on the + # first round of #94: all three reviewers converged on this hole. + if timeout -k 5 15 incus info "$instance" /dev/null 2>&1; then + echo "box: 'incus launch' OVERRAN its ${budget}s budget (killed) — but the instance WAS" >&2 + echo "box: registered: this looks like a slow launch, not the #93 client wedge. Removing" >&2 + echo "box: the partial instance so a retry starts clean..." >&2 + else + echo "box: 'incus launch' WEDGED — killed after ${budget}s (or killed from outside), and" >&2 + echo "box: the instance was never created. This is the #93 failure: the incus client" >&2 + echo "box: hangs with NO server-side operation ('incus operation list' is empty, the" >&2 + echo "box: daemon journal is quiet). An immediate retry of the exact same 'box new' has" >&2 + echo "box: been observed to succeed, both times it was measured." >&2 + fi + timeout -k 5 30 incus delete --force "$instance" /dev/null 2>&1 || true + echo "box: if it persists, diagnose the host: box doctor" >&2 echo "box: (a genuinely slower mint can raise the budget: BOX_LAUNCH_TIMEOUT=)" >&2 - die "incus launch wedged with no server-side operation — retry the same command (#93)" + die "incus launch did not finish inside ${budget}s — retry the same command (#93)" elif [ "$rc" -ne 0 ]; then # Not a wedge: incus refused and said why on stderr, right above. die "incus launch failed (exit $rc)" diff --git a/test/cli.sh b/test/cli.sh index edc3896..1a01ead 100644 --- a/test/cli.sh +++ b/test/cli.sh @@ -403,8 +403,25 @@ check "new: the launch pins stdin (RUNS.md trap 13)" 0 "" bash -c ' check "new: the wedge failure is loud — retry hint, the doctor, and #93" 0 "" bash -c ' fn="$(awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box")" printf "%s\n" "$fn" | grep -A6 "WEDGED" | grep -q "observed to succeed" && - printf "%s\n" "$fn" | grep -A6 "WEDGED" | grep -q "box doctor" && - printf "%s\n" "$fn" | grep "incus launch wedged" | grep -q "#93"' + printf "%s\n" "$fn" | grep -q "box doctor" && + printf "%s\n" "$fn" | grep "did not finish inside" | grep -q "#93"' +# timeout proves only that the CLIENT overran the budget: launch is +# create-then-start, so a slow launch may have REGISTERED the instance and a +# blind "never created, retry" would send the operator into 'Instance already +# exists' (#94 round-1, all three reviewers). The timeout path must probe the +# instance, tell the two stories apart, and best-effort delete either way so +# the retry advice is safe in both worlds. +check "new: the timeout path probes before claiming never-created (#94 r1)" 0 "" bash -c ' + awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \ + | grep "incus info" | grep -q "\$instance"' +check "new: the timeout path best-effort deletes, so retry is always clean" 0 "" bash -c ' + awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \ + | grep "incus delete --force" | grep -q "|| true"' +check "new: the overran-but-registered branch says so (not the wedge story)" 0 "" bash -c ' + awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \ + | grep "OVERRAN" | grep -q "budget"' +check "new: BOX_LAUNCH_TIMEOUT is documented in box help new" 0 "" bash -c ' + "'"$ROOT"'/bin/box" help new | grep "BOX_LAUNCH_TIMEOUT" | grep -q 600' # staging's creds-holding join stays OPERATOR-run: cmd_new may print it as a # next step, but no template and no code path auto-runs "rig bootstrap # workload" — the one absence that keeps box creds-free end to end. -- 2.45.2 From 12edb1966fe7a43b2409c637dcaee29e64d9cdee Mon Sep 17 00:00:00 2001 From: dan-claude-bot Date: Sun, 19 Jul 2026 12:59:58 +0000 Subject: [PATCH 3/4] chore: the four new grep-proof checks carry the SC2016 directive their siblings had MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CI shellchecks every shell file; the round-2 checks embed $ROOT in single-quoted bash -c strings exactly like the check above them, but the disable directive only covers the next command — each new check gets its own, same as the repo pattern. Co-Authored-By: Claude Fable 5 --- test/cli.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/test/cli.sh b/test/cli.sh index 1a01ead..3c1c49e 100644 --- a/test/cli.sh +++ b/test/cli.sh @@ -411,15 +411,19 @@ check "new: the wedge failure is loud — retry hint, the doctor, and #93" 0 "" # exists' (#94 round-1, all three reviewers). The timeout path must probe the # instance, tell the two stories apart, and best-effort delete either way so # the retry advice is safe in both worlds. +# shellcheck disable=SC2016 # the $-strings are literals in the target file check "new: the timeout path probes before claiming never-created (#94 r1)" 0 "" bash -c ' awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \ | grep "incus info" | grep -q "\$instance"' +# shellcheck disable=SC2016 # the $-strings are literals in the target file check "new: the timeout path best-effort deletes, so retry is always clean" 0 "" bash -c ' awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \ | grep "incus delete --force" | grep -q "|| true"' +# shellcheck disable=SC2016 # the $-strings are literals in the target file check "new: the overran-but-registered branch says so (not the wedge story)" 0 "" bash -c ' awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \ | grep "OVERRAN" | grep -q "budget"' +# shellcheck disable=SC2016 # the $-strings are literals in the target file check "new: BOX_LAUNCH_TIMEOUT is documented in box help new" 0 "" bash -c ' "'"$ROOT"'/bin/box" help new | grep "BOX_LAUNCH_TIMEOUT" | grep -q 600' # staging's creds-holding join stays OPERATOR-run: cmd_new may print it as a -- 2.45.2 From 6fc5ba4d0adb671ea851ae1471a5d807c9662b5d Mon Sep 17 00:00:00 2001 From: dan-claude-bot Date: Sun, 19 Jul 2026 13:01:49 +0000 Subject: [PATCH 4/4] =?UTF-8?q?docs:=20the=20changelog=20entry=20tells=20t?= =?UTF-8?q?he=20round-2=20story=20=E2=80=94=20probe,=20branch,=20cleanup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The entry ended at round 1's unconditional wedge message; the shipped behavior probes the instance, narrates wedge vs overrun, and best-effort deletes so the retry is clean either way. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 034b727..40b5426 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -217,9 +217,12 @@ which records not just what changed but what each drill run proved. `timeout -k 5 $BOX_LAUNCH_TIMEOUT` (seconds, default 600 — generous: the coldest measured mint is minutes, never an hour; the same scripting-knob shape as `BOX_CPU`/`BOX_MEMORY`), with stdin pinned per the drill's own - trap list. On the budget firing it says exactly what was measured — the - client wedged with no server-side operation, an immediate retry has been - observed to succeed — and points at `box doctor` for the host. The + trap list. On the budget firing it probes whether the instance was ever + registered and tells the two stories apart — the measured #93 wedge (no + server-side operation; an immediate retry has been observed to succeed) + vs a slow launch that overran the budget with the instance already + created — then best-effort deletes either way, so the retry advice is + clean in both worlds, and points at `box doctor` for the host. The `--from` clone path is untouched: `incus copy` of a local instance is a different operation and has never been observed to wedge this way. - **UFW's gateway carve-out converges with the bridge, and the doctor can -- 2.45.2