fix(drill): no in-box probe can hang the run again #22

Merged

dan-claude-bot merged 1 commit from drill/hangproof-exec into main

2026-07-13 22:57:17 +00:00

dan-claude-bot commented

2026-07-13 22:55:45 +00:00

(Migrated from github.com)

Run 4 hung 10+ minutes at C4 — the sibling probe, the last open item in #15. Everything before it passed.

The cause is a shell trap, not a network one

$( ) waits for stdout to close, not for the command to exit. A grandchild that inherits the exec session's stdout holds the substitution open forever — and timeout does not save you: it kills the wrapper, not the process holding the pipe. Using claudebox exec made it likelier still, since that becomes sudo -u claude -i — a login zsh with oh-my-zsh, a lot of machinery to keep an fd alive.

The fix

Every in-box read now goes through two helpers:

in_box — incus exec directly (no login shell), stdin pinned to /dev/null, output landed in a file rather than a pipe, timeout -k to hard-kill.
box_curl — same discipline, and it returns curl's exit code as the finding. For A3 the exit code is the answer: 0 connected (isolation broken), 7 refused (the packet arrived — the egress drop isn't covering siblings), 28 timed out (dropped, as designed).

C4 also prints what it's probing before it probes, so a stall is locatable instead of silent.

A3 can be answered right now, without waiting for this merge

On the stuck server, after Ctrl-C:

PEER_IP=$(incus exec peer -- ip -4 -o addr show dev eth0 | awk '{split($4,a,"/"); print a[1]}')
incus exec peer -- sh -c 'nohup python3 -m http.server 8088 --bind 0.0.0.0 >/dev/null 2>&1 </dev/null &'; sleep 1
incus exec archive -- curl -sS -m 5 -o /dev/null "http://$PEER_IP:8088"; echo "curl exit: $?"

28 = isolation holds · 7 = packets arrive (#16 becomes a fix) · 0 = broken.

🤖 Generated with Claude Code

Run 4 hung 10+ minutes at C4 — the sibling probe, the last open item in #15. Everything before it passed. ## The cause is a shell trap, not a network one `$( )` waits for stdout to **close**, not for the command to **exit**. A grandchild that inherits the exec session's stdout holds the substitution open forever — and `timeout` does not save you: it kills the wrapper, not the process holding the pipe. Using `claudebox exec` made it likelier still, since that becomes `sudo -u claude -i` — a login zsh with oh-my-zsh, a lot of machinery to keep an fd alive. ## The fix Every in-box read now goes through two helpers: - **`in_box`** — `incus exec` directly (no login shell), stdin pinned to `/dev/null`, output landed in a **file rather than a pipe**, `timeout -k` to hard-kill. - **`box_curl`** — same discipline, and it **returns curl's exit code as the finding**. For A3 the exit code *is* the answer: `0` connected (isolation broken), `7` refused (the packet **arrived** — the egress drop isn't covering siblings), `28` timed out (dropped, as designed). C4 also prints what it's probing before it probes, so a stall is locatable instead of silent. ## A3 can be answered right now, without waiting for this merge On the stuck server, after Ctrl-C: ```sh PEER_IP=$(incus exec peer -- ip -4 -o addr show dev eth0 | awk '{split($4,a,"/"); print a[1]}') incus exec peer -- sh -c 'nohup python3 -m http.server 8088 --bind 0.0.0.0 >/dev/null 2>&1 </dev/null &'; sleep 1 incus exec archive -- curl -sS -m 5 -o /dev/null "http://$PEER_IP:8088"; echo "curl exit: $?" ``` 28 = isolation holds · 7 = packets arrive (#16 becomes a fix) · 0 = broken. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/box#22

No description provided.