fix(doctor): pin the probes' stdin — an interactive exec cannot be timed out #42

Merged

dan-claude-bot merged 1 commit from fix/doctor-probe-hangs into main

2026-07-14 11:54:37 +00:00

dan-claude-bot commented

2026-07-14 11:50:27 +00:00

(Migrated from github.com)

Observed live on the drill host, twice in a row: the doctor hangs at "Can a box actually resolve DNS?" — 15+ minutes, and Ctrl-C does not work; the operator had to kill the shell. The daemon and the box's agent were both fine (timeout 10 incus exec t1 -- true → exit 0).

The bug

With a TTY on stdin, incus exec goes interactive: it attaches the operator's terminal and puts it in raw mode. Every probe in the doctor's DNS section runs exactly that way:

timeout 20 incus exec "$probe" -- ping -c1 -W2 10.87.0.1 >/dev/null 2>&1

Three failures stack:

the exec wedges in terminal handling instead of running to completion;
timeout's bare SIGTERM never takes — and without -k it is never escalated to SIGKILL;
^C is forwarded into the box as a keystroke instead of interrupting the script — which is why the hang survives Ctrl-C.

The drill already knows this

drill.sh learned the identical lesson in #22 — exec_in pins stdin and escalates:

timeout -k 5 20 incus exec "$b" -- "$@" >"$out" 2>/dev/null </dev/null

The doctor's probe section was added later (#34/#35) and never got the cure. Same disease as trap 12 and its eleven ancestors: the wrapper looked hang-proof — timeout stood in for the fact of termination.

The fix

All four probes: </dev/null + timeout -k 5, and a comment stating the rule so the next probe added to this file inherits it.

Note for the log: on the live host the --pin-dns mutation did land before the hang (OK set raw.dnsmasq… printed) — the hang was only ever in the verification probe.

🤖 Generated with Claude Code

Observed live on the drill host, twice in a row: the doctor hangs at **"Can a box actually resolve DNS?"** — 15+ minutes, and **Ctrl-C does not work**; the operator had to kill the shell. The daemon and the box's agent were both fine (`timeout 10 incus exec t1 -- true` → exit 0). ## The bug With a TTY on stdin, `incus exec` goes **interactive**: it attaches the operator's terminal and puts it in raw mode. Every probe in the doctor's DNS section runs exactly that way: ```sh timeout 20 incus exec "$probe" -- ping -c1 -W2 10.87.0.1 >/dev/null 2>&1 ``` Three failures stack: 1. the exec wedges in terminal handling instead of running to completion; 2. `timeout`'s bare SIGTERM never takes — and without `-k` it is never escalated to SIGKILL; 3. `^C` is forwarded **into the box** as a keystroke instead of interrupting the script — which is why the hang survives Ctrl-C. ## The drill already knows this `drill.sh` learned the identical lesson in #22 — `exec_in` pins stdin and escalates: ```sh timeout -k 5 20 incus exec "$b" -- "$@" >"$out" 2>/dev/null </dev/null ``` The doctor's probe section was added later (#34/#35) and never got the cure. Same disease as trap 12 and its eleven ancestors: the wrapper *looked* hang-proof — `timeout` stood in for the fact of termination. ## The fix All four probes: `</dev/null` + `timeout -k 5`, and a comment stating the rule so the next probe added to this file inherits it. Note for the log: on the live host the `--pin-dns` mutation **did land** before the hang (`OK set raw.dnsmasq…` printed) — the hang was only ever in the verification probe. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/box#42

No description provided.