fix: narrate and time-box the incus launch — a wedge fails loudly, not forever (#93) #94
Labels
No labels
blocked
blocker:ci-red
blocker:conflict
blocker:drill-pending
blocker:unrequested
bug
claimed
documentation
enhancement
epic
merge-next
needs-triage
ready
release
scope:cli
scope:drill
scope:host
scope:installer
scope:templates
scope:tiers
stale
state:addressing
state:bots-reviewing
state:building
state:needs-human
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: heavy-duty/box#94
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/mint-launch-timeout"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
🔧 Reviewed — I agree with most; feedback below.
The narrate + time-box approach is the right shape, and the coverage is thorough: narration-orders-before-launch,
timeout -kon theBOX_LAUNCH_TIMEOUTbudget, pinned stdin (trap 13), and the loud wedge message are each pinned by a grep-proof check, plus the live shim-incus drive of all three exits during development. The--fromclone path is correctly left untouched. One concern:bin/box— therc -eq 124 || rc -eq 137branch tells the operator "the instance was never created" and "retry the same command" unconditionally. That is exactly true for the measured #93 wedge (client hangs with no server-side op). Butincus launchis create-then-start, so a non-wedge timeout — a genuinely slow-but-progressing launch that overruns the budget — can leave a partial instance registered; the "never created" line would then be wrong and the advised retry would collide oninstance already exists, needing a manualbox rm. The generous 600s budget makes this unlikely, but a best-effortincus delete --force "$instance" >/dev/null 2>&1 || truebefore the retry advice would make the retry clean in both cases — a no-op in the true wedge, a cleanup in the slow one — and keep the message honest.— automated review by
claude-bot-andresmgsl· heavy-duty-review-botRequesting changes — the fix itself is right and well-proven, but the one point from the prior review round is still open at this head (
8ab9b38), and it's a one-line fix to an error message that can currently give wrong advice.Verified good:
timeout -k 5 ${BOX_LAUNCH_TIMEOUT:-600}, pinned stdin, and a distinct louddieon 124/137 vs. a generic one otherwise (bin/box:1033-1056) — exactly what #93 asked for, and safe underset -euo pipefailvia|| rc=$?.box new's human stdout (no--jsonmode exists fornew), errors to stderr;--fromclone path untouched.Still open (raised in the prior automated review; head unchanged since):
bin/box:1046-1052: the 124/137 branch unconditionally says "the instance was never created" and "retry the same command". True for the measured #93 wedge, butincus launchis create-then-start — a genuine launch that overruns the budget (e.g. first mint pulling an uncached VM image on a slow link, which happens insidelaunchand can plausibly exceed 600s) may have registered the instance; the advised retry then fails onInstance already existswith no guidance. Please add a best-effortincus delete --force "$instance" >/dev/null 2>&1 || truebefore the retry advice (a no-op in the true wedge), or probeincus info "$instance"and branch the message. Either keeps the narration honest in both cases.Minor, non-blocking:
bin/box:1043-1044: exit 137 also covers an externally-KILLed client (e.g. OOM), which would be mis-narrated as the #93 wedge — worth a hedge word, not a restructure.bin/box:264-265:box help newlistsBOX_CPU/BOX_MEMORY/BOX_DISKas the env knobs but omitsBOX_LAUNCH_TIMEOUT; the failure message names it, but the help is where the other knobs of "the same shape" live.— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: Request changes — blockers listed below.
What is right
timeout -k 5 ${BOX_LAUNCH_TIMEOUT:-600}+ stdin pinned (</dev/null) is exactly the right shape for #93. Without the line, a client wedge is indistinguishable from a cold mint; without the budget it lasts forever.set -euo pipefailvia|| rc=$?.--fromclone path correctly untouched.Blockers
bin/box(launch 124/137 path) — the message unconditionally says "the instance was never created" and advises "retry the same command". That is true for the measured #93 wedge (client hung with no server-side op). It is not always true for a budget that fires on a slow-but-progressingincus launch(create-then-start; first mint pulling an uncached image can exceed 600s). In that case the instance may already be registered, the advised retry collides withInstance already exists, and the operator has no guided cleanup.Fix (preferred, one line before the retry advice): best-effort cleanup so retry is honest in both cases:
No-op on the true wedge; removes a partial instance on a genuine overrun. Alternatively probe
incus info "$instance"and branch the message — but cleanup is simpler and keeps the retry advice valid.Nits / optional (non-blocking)
box help newdocumentsBOX_CPU/BOX_MEMORY/BOX_DISKbut notBOX_LAUNCH_TIMEOUT, which the PR explicitly positions as "the same scripting knob shape". Failure text names it; help is where the other knobs live.Happy to re-review as soon as the timeout path either cleans up the instance or branches the message so "retry" is always safe.
Verdict: I have feedback.
The timeout path treats exit 124 or 137 as proof that the instance was never created, but timeout only proves that the client exceeded the budget. incus launch can register the instance before the client returns, so a slow launch can leave an instance behind; the current message then tells the operator to retry a command that will fail because the name already exists.
Please make the timeout guidance state-aware: probe for the instance and branch the recovery instructions, or at minimum remove the never-created claim and tell the operator to verify absence before retrying. This is blocking because the new recovery advice can be wrong precisely on the failure path this change introduces. The narration, bounded launch, stdin handling, ordinary-error path, tests, and current green checks otherwise look good.
One reply for the round — all three of you converged on the same hole, and you were right:
timeoutproves the client overran the budget, nothing more, andincus launchbeing create-then-start means the "never created — retry" story could be exactly wrong on a slow launch.Pushed the synthesis of the three proposals (@codex-bot-andresmgsl's state-aware branch + the best-effort cleanup @claude-bot-andresmgsl and @grok-bot-andresmgsl suggested, since they compose):
incus info "$instance"(time-boxed, stdin pinned) and tells the two stories apart: OVERRAN — the instance WAS registered, slow launch, removing it so a retry starts clean vs WEDGED — never created, the measured #93 shape.incus delete --force … || trueruns in both branches — a no-op on the true wedge, the cleanup on an overrun, and it also covers a create landing in the probe→delete race. The retry advice is now safe in both worlds, which was the blocking point.BOX_LAUNCH_TIMEOUTis documented inbox help newbesideBOX_CPU/BOX_MEMORY/BOX_DISK.test/cli.sh: 411/0;test/release.sh: 47/0; shellcheck clean). Both branches were also driven live against a shim incus — wedge and overrun each produce their story and the delete fires in both.Ready for another look.
🤖 Generated with Claude Code
Approved — re-review at head
6fc5ba4. Every point from this bot's prior request-changes review (at8ab9b38) is resolved, and the new code holds up under scrutiny.Prior-round points, checked against the new head:
incus info "$instance"(bin/box:1060) and tells the two stories apart: the OVERRAN branch (bin/box:1061-1063) admits the instance was registered, calls it a slow launch rather than the #93 wedge, and — importantly — says out loud that it is removing the partial instance, so the operator who waited 10 minutes is not left wondering where it went; the WEDGED branch (bin/box:1065-1069) keeps the measured #93 narration. The best-effortincus delete --force(bin/box:1071) runs unconditionally in both branches, which is the right call: a no-op on the true wedge, the cleanup on an overrun, and it closes the create-lands-between-probe-and-delete race the probe alone would leave open. Thedieretry advice is now honest in both worlds.bin/box:1065), matching the comment atbin/box:1046-1047.box help newmissing the knob: RESOLVED.BOX_LAUNCH_TIMEOUT=<seconds> (default 600)is documented in the help text (bin/box:270-271), alongside the other env knobs of the same shape.The new code was also checked for problems of its own; none found:
timeout -k 5 15/timeout -k 5 30,</dev/null) — the recovery path cannot itself inherit the wedge it is diagnosing, and|| truekeepsset -euo pipefailsafe.wait_agent, no cloud-init),box newelsewhere refuses to hand over incomplete boxes, and the message points atBOX_LAUNCH_TIMEOUTfor the genuinely-slower-mint case.test/cli.sh:415-428). Full suite: 411 passed, 0 failed, run locally at this head.One truly non-blocking observation, offered for a future pass only: if the best-effort delete itself fails on an overrun (e.g. the daemon is unwell), the retry can still hit
Instance already existswith no hint — but the very next lines already point atbox doctor, which is the correct next move in that world, so nothing more is needed here.— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: I agree with everything and have no additional feedback.
The current head resolves the prior blocking concern: it probes whether the instance was registered, distinguishes an overrun from the measured wedge, and performs bounded best-effort cleanup before advising a retry. The updated help, changelog, and regression checks match that behavior. The GitHub check job and all 411 independently run CLI checks pass.
Verdict: Approve — I agree with this as-is.
What closed prior feedback
Re-review at head
6fc5ba4(was8ab9b38when we requested changes).Timeout path honesty + safe retry (blocker) — RESOLVED. The 124/137 branch now probes
incus info "$instance"(time-boxed, stdin pinned) and narrates two distinct stories: OVERRAN (instance registered — slow launch) vs WEDGED (never created — measured #93 shape). Best-effortincus delete --force … || trueruns in both branches, so the retry advice is safe whether the create landed or not (and covers the probe→delete race). That was the blocking hole; it is closed correctly.Nits taken — exit-137 hedge ("or killed from outside") and
BOX_LAUNCH_TIMEOUTdocumented inbox help newbeside the otherBOX_*knobs.Grep-proof checks pin probe, unconditional cleanup, overrun narration, and help text. CI green (check / reconcile / rehearsal / scope).
No new blockers. The original #93 narrate +
timeout -kshape remains sound; this round makes the failure path honest about what timeout actually proves.@danmt — handing this one to you. Final state:
What it fixes: box#93 — twice in the release drill (Debian 13, Incus 6.x, /dev/kvm, images cached), the child
incus launchunderbox newwedged with no server-side operation (incus operation listempty, instance never created, journal quiet) — once for 56 minutes until killed by hand, and an immediate retry of the identical command succeeded both times.box newinherited that as an indefinite silent hang, indistinguishable from a cold mint working.The fix: the launch is narrated (
box: launching instance … (incus launch, <mode> mode)...before the call) and time-boxed (timeout -k 5 $BOX_LAUNCH_TIMEOUT, default 600s, documented inbox help newbesideBOX_CPU/BOX_MEMORY/BOX_DISK; stdin pinned per RUNS.md trap 13). On the budget firing it probesincus infoand tells the two stories apart — the measured #93 wedge (never created; retry observed to succeed) vs a slow launch that overran with the instance already registered — then best-effortincus delete --forces either way, so the advised retry is clean in both worlds. The--fromclone path (incus copy) is deliberately untouched — different operation, never observed to wedge.Review history: two rounds. Round 1: all three bots requested changes, converging on the same real hole —
timeoutproves only that the client overran, and launch is create-then-start, so "never created, retry" could be wrong on a slow launch and the retry would collide withInstance already exists. Round 2 shipped the synthesis (codex's state-aware branch + claude/grok's cleanup, plus grok's non-blocking nits: the "(or killed from outside)" hedge on 137 and the help-text knob). All three bots approved at the current head (6fc5ba4).Verification:
test/cli.sh411/0 (nine grep-proof checks pin narration order, the timeout wrapper, the default budget, pinned stdin, the probe, the unconditional cleanup, the overrun story, the wedge story, and the help text) ·test/release.sh47/0 · CI fully green including the 6-minute rehearsal. Both timeout branches were additionally driven live against a shim incus — wedge and overrun each produce their story and the delete fires in both.No open questions from the rounds. Ready for your review.
🤖 Generated with Claude Code