CHANGELOG.md

@@ -204,6 +204,27 @@ which records not just what changed but what each drill run proved.
 
 ### Fixed
 
+- **A wedged `incus launch` fails loudly, not forever — the mint's launch
+  phase is narrated and time-boxed** (#93) — twice in the 2026-07-19
+  release drill (Debian 13, Incus 6.x, /dev/kvm present, images cached),
+  the child `incus launch` under `box new` hung with *no server-side
+  operation*: `incus operation list` empty, the instance never created, the
+  daemon journal quiet — one wedge ran 56 minutes before being killed by
+  hand, and an immediate retry of the identical command succeeded in
+  minutes, both times. `box new` inherited that as an indefinite silent
+  hang, indistinguishable from a cold mint working. It now prints
+  `launching instance …` before the call, and the call rides
+  `timeout -k 5 $BOX_LAUNCH_TIMEOUT` (seconds, default 600 — generous: the
+  coldest measured mint is minutes, never an hour; the same scripting-knob
+  shape as `BOX_CPU`/`BOX_MEMORY`), with stdin pinned per the drill's own
+  trap list. On the budget firing it probes whether the instance was ever
+  registered and tells the two stories apart — the measured #93 wedge (no
+  server-side operation; an immediate retry has been observed to succeed)
+  vs a slow launch that overran the budget with the instance already
+  created — then best-effort deletes either way, so the retry advice is
+  clean in both worlds, and points at `box doctor` for the host. The
+  `--from` clone path is untouched: `incus copy` of a local instance is a
+  different operation and has never been observed to wedge this way.
 - **UFW's gateway carve-out converges with the bridge, and the doctor can
   see it** (the #86 review's blind spot) — `box-firewall` gated its whole
   UFW block behind "a `DENY on boxnet` rule exists", pinning every UFW host

bin/box

@@ -267,6 +267,9 @@ template's box.env, then defaults. Flags shape a fresh mint only — a --from
 clone carries its source's resources. Resources are all a flag can touch:
 there is no flag for a network or a security key, on purpose.
 
+BOX_LAUNCH_TIMEOUT=<seconds> (default 600) bounds the 'incus launch' call —
+a launch that overruns it fails loudly instead of hanging forever (#93).
+
   box new --name scratch                     # blank, the default
   box new --name work --template claude
   box new --name lean --template claude --cpu 2 --memory 3GiB
@@ -1017,14 +1020,62 @@ cmd_new() {
     # The template's identity is stamped ONTO the instance: which template,
     # which user. 'incus copy' preserves user.* keys (audit B2), so a clone
     # knows what it is without ever consulting the template again.
-    incus launch "$T_IMAGE" "$instance" --profile box-net \
+    #
+    # The launch is narrated and TIME-BOXED (#93). Twice in the 2026-07-19
+    # release drill the child 'incus launch' wedged before the create was
+    # even accepted — 'incus operation list' empty, the instance never
+    # existed, the daemon journal quiet — once for 56 minutes until killed
+    # by hand. Without a line here that wedge reads exactly like a cold mint
+    # working; without a budget it lasts forever. Ten minutes is generous —
+    # the coldest measured mint (first VM on a fresh pool, see wait_agent)
+    # is minutes, never an hour — and BOX_LAUNCH_TIMEOUT (seconds) overrides
+    # it, the same scripting knob shape as BOX_CPU / BOX_MEMORY. The drill's
+    # lore applies verbatim (RUNS.md trap 13): 'timeout -k' so a launch that
+    # shrugs off TERM still dies, and stdin pinned like every other
+    # non-interactive incus call — only shell/exec/tmux may own the terminal.
+    local budget="${BOX_LAUNCH_TIMEOUT:-600}" rc=0
+    echo "box: launching instance $instance (incus launch, $m mode)..."
+    timeout -k 5 "$budget" incus launch "$T_IMAGE" "$instance" --profile box-net \
       --config user.box=1 \
       --config user.box.template="$t" \
       --config user.box.user="$T_USER" \
       --config limits.cpu="$T_CPU" \
       --config limits.memory="$T_MEMORY" \
       --config cloud-init.user-data="$(render_userdata "$root/templates/$t/user-data.yaml")" \
-      "${extra[@]}"
+      "${extra[@]}" </dev/null || rc=$?
+    # 124 = the budget fired (TERM landed); 137 = the -k KILL was needed —
+    # or, on 137, something external (an OOM kill) beat the budget to it.
+    if [ "$rc" -eq 124 ] || [ "$rc" -eq 137 ]; then
+      echo >&2
+      # timeout only proves the CLIENT overran the budget. 'incus launch' is
+      # create-then-start, so a slow-but-progressing launch (first mint
+      # pulling an uncached image, say) may already have REGISTERED the
+      # instance — in which case "never created, retry" would be exactly
+      # wrong: the retry collides with 'Instance already exists'. Probe, say
+      # which case this is, and best-effort delete either way (a no-op on
+      # the true #93 wedge, the cleanup on an overrun; also covers a create
+      # that lands in the race between the probe and the delete) so the
+      # retry advice below is safe in BOTH worlds. Review consensus on the
+      # first round of #94: all three reviewers converged on this hole.
+      if timeout -k 5 15 incus info "$instance" </dev/null >/dev/null 2>&1; then
+        echo "box: 'incus launch' OVERRAN its ${budget}s budget (killed) — but the instance WAS" >&2
+        echo "box: registered: this looks like a slow launch, not the #93 client wedge. Removing" >&2
+        echo "box: the partial instance so a retry starts clean..." >&2
+      else
+        echo "box: 'incus launch' WEDGED — killed after ${budget}s (or killed from outside), and" >&2
+        echo "box: the instance was never created. This is the #93 failure: the incus client" >&2
+        echo "box: hangs with NO server-side operation ('incus operation list' is empty, the" >&2
+        echo "box: daemon journal is quiet). An immediate retry of the exact same 'box new' has" >&2
+        echo "box: been observed to succeed, both times it was measured." >&2
+      fi
+      timeout -k 5 30 incus delete --force "$instance" </dev/null >/dev/null 2>&1 || true
+      echo "box: if it persists, diagnose the host:  box doctor" >&2
+      echo "box: (a genuinely slower mint can raise the budget: BOX_LAUNCH_TIMEOUT=<seconds>)" >&2
+      die "incus launch did not finish inside ${budget}s — retry the same command (#93)"
+    elif [ "$rc" -ne 0 ]; then
+      # Not a wedge: incus refused and said why on stderr, right above.
+      die "incus launch failed (exit $rc)"
+    fi
     wait_agent "$instance"
     echo "box: waiting for phase-1 (cloud-init)..."
     echo "box: (its full narration, live:  incus exec $name -- tail -f /var/log/cloud-init-output.log)"

test/cli.sh

@@ -373,6 +373,59 @@ check "new: the auto-run sits under the T_BOOTSTRAP_ROLE guard" 0 "" bash -c '
 check "new: a failed tenant role names the re-run (the role converges)" 0 "" bash -c '
   awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \
     | grep -q "sudo rig bootstrap"'
+
+# The launch phase, narrated and time-boxed (#93) — grepped the way the other
+# mint-path guards are (a daemon-free run cannot mint). Twice in the
+# 2026-07-19 release drill the child 'incus launch' wedged silently before
+# the create was even accepted, once for 56 minutes. The narration must order
+# BEFORE the launch call (a wedge after the line is visible at a glance; a
+# wedge before it is the old silent hang), the call itself must sit under
+# 'timeout -k' with the BOX_LAUNCH_TIMEOUT override and pinned stdin (RUNS.md
+# trap 13: bare 'timeout N' cannot kill an incus call that owns a TTY), and
+# the budget's failure must be LOUD — no server-side operation, the measured
+# retry-succeeds hint, and the doctor as the next move.
+# shellcheck disable=SC2016  # the $-strings are literals in the target file
+check "new: the launch narration orders before incus launch (#93)" 0 "" bash -c '
+  fn="$(awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box")"
+  say="$(printf "%s\n" "$fn" | grep -n "launching instance" | head -1 | cut -d: -f1)"
+  run="$(printf "%s\n" "$fn" | grep -n "timeout -k.*incus launch" | head -1 | cut -d: -f1)"
+  [ -n "$say" ] && [ -n "$run" ] && [ "$say" -lt "$run" ]'
+check "new: incus launch is time-boxed (timeout -k on the budget)" 0 "" bash -c '
+  awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \
+    | grep "timeout -k" | grep "budget" | grep -q "incus launch"'
+check "new: the budget is BOX_LAUNCH_TIMEOUT, default 600s (the BOX_CPU knob shape)" 0 "" bash -c '
+  awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \
+    | grep "budget=" | grep -q "BOX_LAUNCH_TIMEOUT:-600"'
+check "new: the launch pins stdin (RUNS.md trap 13)" 0 "" bash -c '
+  awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \
+    | grep -F "extra[@]" | grep -qF "</dev/null"'
+# shellcheck disable=SC2016  # the $-strings are literals in the target file
+check "new: the wedge failure is loud — retry hint, the doctor, and #93" 0 "" bash -c '
+  fn="$(awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box")"
+  printf "%s\n" "$fn" | grep -A6 "WEDGED" | grep -q "observed to succeed" &&
+  printf "%s\n" "$fn" | grep -q "box doctor" &&
+  printf "%s\n" "$fn" | grep "did not finish inside" | grep -q "#93"'
+# timeout proves only that the CLIENT overran the budget: launch is
+# create-then-start, so a slow launch may have REGISTERED the instance and a
+# blind "never created, retry" would send the operator into 'Instance already
+# exists' (#94 round-1, all three reviewers). The timeout path must probe the
+# instance, tell the two stories apart, and best-effort delete either way so
+# the retry advice is safe in both worlds.
+# shellcheck disable=SC2016  # the $-strings are literals in the target file
+check "new: the timeout path probes before claiming never-created (#94 r1)" 0 "" bash -c '
+  awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \
+    | grep "incus info" | grep -q "\$instance"'
+# shellcheck disable=SC2016  # the $-strings are literals in the target file
+check "new: the timeout path best-effort deletes, so retry is always clean" 0 "" bash -c '
+  awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \
+    | grep "incus delete --force" | grep -q "|| true"'
+# shellcheck disable=SC2016  # the $-strings are literals in the target file
+check "new: the overran-but-registered branch says so (not the wedge story)" 0 "" bash -c '
+  awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" \
+    | grep "OVERRAN" | grep -q "budget"'
+# shellcheck disable=SC2016  # the $-strings are literals in the target file
+check "new: BOX_LAUNCH_TIMEOUT is documented in box help new" 0 "" bash -c '
+  "'"$ROOT"'/bin/box" help new | grep "BOX_LAUNCH_TIMEOUT" | grep -q 600'
 # staging's creds-holding join stays OPERATOR-run: cmd_new may print it as a
 # next step, but no template and no code path auto-runs "rig bootstrap
 # workload" — the one absence that keeps box creds-free end to end.