heavy-duty/box

Fork 1

Incus daemon wedged on the drill host — forensics before reprovisioning #26

New issue

Closed

opened 2026-07-13 23:13:47 +00:00 by dan-claude-bot · 4 comments

dan-claude-bot commented

2026-07-13 23:13:47 +00:00

(Migrated from github.com)

The drill host stopped answering Incus. Run 6 stalled inside setup-host.sh, and teardown-host.sh then stalled too — right after the y, before printing anything, which places it at its first command: incus list "user.claudebox=1" in a command substitution. Both stalls are the same symptom: incusd is not answering.

Do not reprovision the host yet. A fresh server would come up green and prove nothing. If claudebox (or the drill) can wedge Incus — by force-deleting VMs, by leaving orphaned qemu processes, by filling the storage pool with 60GiB boxes — that is a claudebox finding, and it lives exactly one Ctrl-C away from a user's real workflow (claudebox rm, an interrupted new). Reprovisioning erases it.

This issue is the forensic sequence. Capture before you fix — a daemon restart destroys the evidence.

Step 1 — evidence (restart nothing)

Run all of it, paste the output as a comment on this issue.

# 1a. is the daemon answering, and how does it fail?
timeout 10 incus list; echo "incus list exit: $?"
timeout 10 incus info | head -5; echo "incus info exit: $?"
timeout 10 incus operation list; echo "operation list exit: $?"

# 1b. THE most likely cause: the storage pool is full.
#     Six runs × VMs with a 60GiB root device. Sparse, but not free.
df -h /
timeout 10 incus storage list
timeout 10 incus storage info default
sudo du -sh /var/lib/incus 2>/dev/null

# 1c. stuck processes — orphaned qemu from a force-killed VM, or D-state
ps -eo pid,etimes,stat,args | grep -E 'incusd|qemu' | grep -v grep
ps -eo pid,stat,wchan:32,args | awk '$2 ~ /D/'

# 1d. what the daemon says about itself
systemctl status incus.service incus.socket --no-pager -n 10
sudo journalctl -u incus --no-pager -n 100 | tail -50

# 1e. is anything actually listening on the socket?
sudo ss -xlp | grep -i incus

Step 2 — read the evidence

What you see	What it means	Consequence
`df -h /` at or near 100%, or the storage pool full	The drill's boxes filled the disk. Incus wedges rather than erroring.	claudebox/drill bug. The profile pins a 60GiB root device (`bin/claudebox`), and the drill mints up to 4 boxes. The drill should shrink the disk for its own boxes, or the docs must state the host requirement. Worth its own issue.
Orphaned `qemu` processes with no live instance	A force-deleted VM left its process behind; Incus is waiting on it.	claudebox finding — `claudebox rm --force` / an interrupted `new` can orphan a VM. This is reachable from normal use.
`incusd` in D state, or D-state processes under `/var/lib/incus`	Kernel-level block — a stuck mount or storage backend.	Likely environmental, but record it.
A panic / fatal / deadlock in the journal	Upstream Incus bug.	Record the version (`incus --version`) and the trace; not ours to fix, but ours to avoid.
Everything looks healthy and it answers now	The stall was transient (a long operation completing).	Note it and move on — but the drill's new timeouts (#25) mean it would have said so rather than hanging.

Step 3 — recovery, least destructive first

Try each, re-testing timeout 10 incus list; echo $? after every one. Stop at the first that works and record which.

# 3a. restart the daemon (keeps all state)
sudo systemctl restart incus.socket incus.service
timeout 10 incus list; echo "exit: $?"

# 3b. still wedged? kill orphaned qemu, then restart again
ps -eo pid,args | grep '[q]emu'          # look before you shoot
sudo pkill -f qemu
sudo systemctl restart incus.socket incus.service
timeout 10 incus list; echo "exit: $?"

# 3c. disk full? reclaim, then restart
timeout 30 incus delete -f drill  2>/dev/null
timeout 30 incus delete -f clone  2>/dev/null
timeout 30 incus delete -f archive 2>/dev/null
timeout 30 incus delete -f peer   2>/dev/null
df -h /

Step 4 — once it answers again

Re-run the drill on this host, not a new one:

cd claudebox && git pull && bash drill/drill.sh --yes

It now clears leftover boxes and reverts the phase-D mutations before setup-host.sh (#25), and every step is bounded and narrated — so if it stalls again, it will say where instead of going quiet.

Reprovision only if step 3 cannot revive the daemon. In that case, capture step 1's output here first — that output is the finding.

Why this is worth the detour

The audit (#15) has one probe left (A3, sibling isolation) and six runs have failed to fire it — every time for a different reason, mostly drill plumbing (see drill/RUNS.md). It would be easy to reach for a clean host and call it done. But a tool whose boxes can wedge the host's container daemon is a tool with a bug, and the only place that bug is visible is on the host that has it right now.

The drill host stopped answering Incus. Run 6 stalled inside `setup-host.sh`, and `teardown-host.sh` then stalled too — right after the `y`, before printing anything, which places it at its **first** command: `incus list "user.claudebox=1"` in a command substitution. Both stalls are the same symptom: **`incusd` is not answering.** **Do not reprovision the host yet.** A fresh server would come up green and prove nothing. If claudebox (or the drill) can wedge Incus — by force-deleting VMs, by leaving orphaned `qemu` processes, by filling the storage pool with 60GiB boxes — that is a **claudebox finding**, and it lives exactly one Ctrl-C away from a user's real workflow (`claudebox rm`, an interrupted `new`). Reprovisioning erases it. This issue is the forensic sequence. **Capture before you fix** — a daemon restart destroys the evidence. --- ## Step 1 — evidence (restart nothing) Run all of it, paste the output as a comment on this issue. ```sh # 1a. is the daemon answering, and how does it fail? timeout 10 incus list; echo "incus list exit: $?" timeout 10 incus info | head -5; echo "incus info exit: $?" timeout 10 incus operation list; echo "operation list exit: $?" # 1b. THE most likely cause: the storage pool is full. # Six runs × VMs with a 60GiB root device. Sparse, but not free. df -h / timeout 10 incus storage list timeout 10 incus storage info default sudo du -sh /var/lib/incus 2>/dev/null # 1c. stuck processes — orphaned qemu from a force-killed VM, or D-state ps -eo pid,etimes,stat,args | grep -E 'incusd|qemu' | grep -v grep ps -eo pid,stat,wchan:32,args | awk '$2 ~ /D/' # 1d. what the daemon says about itself systemctl status incus.service incus.socket --no-pager -n 10 sudo journalctl -u incus --no-pager -n 100 | tail -50 # 1e. is anything actually listening on the socket? sudo ss -xlp | grep -i incus ``` ## Step 2 — read the evidence | What you see | What it means | Consequence | | --- | --- | --- | | `df -h /` at or near **100%**, or the storage pool full | The drill's boxes filled the disk. Incus wedges rather than erroring. | **claudebox/drill bug.** The profile pins a **60GiB** root device (`bin/claudebox`), and the drill mints up to 4 boxes. The drill should shrink the disk for its own boxes, or the docs must state the host requirement. Worth its own issue. | | Orphaned **`qemu`** processes with no live instance | A force-deleted VM left its process behind; Incus is waiting on it. | **claudebox finding** — `claudebox rm --force` / an interrupted `new` can orphan a VM. This is reachable from normal use. | | `incusd` in **D state**, or D-state processes under `/var/lib/incus` | Kernel-level block — a stuck mount or storage backend. | Likely environmental, but record it. | | A **panic / fatal / deadlock** in the journal | Upstream Incus bug. | Record the version (`incus --version`) and the trace; not ours to fix, but ours to avoid. | | Everything looks healthy and it answers *now* | The stall was transient (a long operation completing). | Note it and move on — but the drill's new timeouts (#25) mean it would have said so rather than hanging. | ## Step 3 — recovery, least destructive first Try each, re-testing `timeout 10 incus list; echo $?` after every one. **Stop at the first that works** and record which. ```sh # 3a. restart the daemon (keeps all state) sudo systemctl restart incus.socket incus.service timeout 10 incus list; echo "exit: $?" # 3b. still wedged? kill orphaned qemu, then restart again ps -eo pid,args | grep '[q]emu' # look before you shoot sudo pkill -f qemu sudo systemctl restart incus.socket incus.service timeout 10 incus list; echo "exit: $?" # 3c. disk full? reclaim, then restart timeout 30 incus delete -f drill 2>/dev/null timeout 30 incus delete -f clone 2>/dev/null timeout 30 incus delete -f archive 2>/dev/null timeout 30 incus delete -f peer 2>/dev/null df -h / ``` ## Step 4 — once it answers again Re-run the drill on **this** host, not a new one: ```sh cd claudebox && git pull && bash drill/drill.sh --yes ``` It now clears leftover boxes and reverts the phase-D mutations **before** `setup-host.sh` (#25), and every step is bounded and narrated — so if it stalls again, it will say where instead of going quiet. Reprovision **only** if step 3 cannot revive the daemon. In that case, capture step 1's output here first — that output is the finding. ## Why this is worth the detour The audit ([#15](https://github.com/heavy-duty/claudebox/issues/15)) has one probe left (**A3**, sibling isolation) and six runs have failed to fire it — every time for a different reason, mostly drill plumbing (see `drill/RUNS.md`). It would be easy to reach for a clean host and call it done. But a tool whose boxes can wedge the host's container daemon is a tool with a bug, and the only place that bug is visible is on the host that has it right now.

dan-claude-bot commented

2026-07-13 23:17:29 +00:00

(Migrated from github.com)

Evidence is in, and it rules out the disk

Not disk — 16G used of 452G. Not a dead daemon — incusd (pid 22114) is Ssl, holding /var/lib/incus/unix.socket, 48 tasks. But its API is deadlocked: incus list → exit 124 (timeout), incus info → 143, incus operation list → 124.

The cause is visible in the process table:

qemu-system-x86_64 -S -name archive ...   ~1841s
qemu-system-x86_64 -S -name peer    ...   ~1794s

-S = start paused, CPU halted. Incus always launches qemu with -S and then resumes it via the monitor socket. Both VMs have sat paused for ~30 minutes: Incus started them and never resumed them. The daemon is stuck mid-operation on the two boxes an aborted run left behind.

The timeline nails it: incusd has been up 1h38m, but both qemu processes are only ~30 min old — they were (re)started exactly when run 6 began. Run 6 stalled in setup-host.sh, and setup-host runs:

incus profile edit claude-dev < profiles/claude-dev.yaml

— rewriting a profile that two running VMs are attached to, which forces a live CPU/memory reconfiguration on both. That is where it deadlocked.

The finding

setup-host.sh can wedge the Incus daemon when boxes are running. It presents itself as idempotent and safe to re-run, and it is — only on an idle host. Re-running it with live boxes (an upgrade, a re-provision, a second drill run) is reachable from normal use, and the failure mode is a hung daemon, not an error message.

Two things follow:

#25 (drill: clean boxes before setup-host) is the right mitigation, not just a drill nicety.
setup-host.sh itself should not silently reconfigure a profile that running instances depend on — it should detect them and refuse, or warn and skip the profile edit. Tracking that as its own fix.

Memory is worth a footnote: incus.service shows 4G resident, 8.7G peak on a 15GiB host with two 3GiB VMs. Not obviously the cause, but not nothing.

Next: capture the deadlock, then recover

incusd is Go, so SIGQUIT dumps every goroutine stack and exits — which both gives us the trace and clears the wedge. Capture before killing:

# 1. dump the goroutine stacks (this is the evidence — it names the deadlock)
sudo kill -QUIT 22114
sleep 3
sudo journalctl -u incus --no-pager -n 2000 | grep -iE 'goroutine|deadlock|panic|sync\.|incus/internal' | head -60
sudo journalctl -u incus --no-pager --since '2 min ago' > /tmp/incus-deadlock.log
wc -l /tmp/incus-deadlock.log     # paste the interesting part here

# 2. clear the paused VMs
sudo pkill -9 -f 'qemu-system-x86_64 -S -name archive'
sudo pkill -9 -f 'qemu-system-x86_64 -S -name peer'

# 3. bring the daemon back
sudo systemctl restart incus.socket incus.service
timeout 10 incus list; echo "exit: $?"      # expect 0, and a clean list

# 4. if instances linger in a broken state
timeout 30 incus delete -f archive 2>/dev/null
timeout 30 incus delete -f peer    2>/dev/null
timeout 10 incus list

Then re-run the drill on this host (#25 clears leftovers before setup-host, so the trigger is gone):

cd claudebox && git pull && bash drill/drill.sh --yes

## Evidence is in, and it rules out the disk **Not disk** — 16G used of 452G. **Not a dead daemon** — `incusd` (pid 22114) is `Ssl`, holding `/var/lib/incus/unix.socket`, 48 tasks. But **its API is deadlocked**: `incus list` → exit 124 (timeout), `incus info` → 143, `incus operation list` → 124. The cause is visible in the process table: ``` qemu-system-x86_64 -S -name archive ... ~1841s qemu-system-x86_64 -S -name peer ... ~1794s ``` **`-S` = start paused, CPU halted.** Incus always launches qemu with `-S` and then resumes it via the monitor socket. Both VMs have sat paused for ~30 minutes: **Incus started them and never resumed them.** The daemon is stuck mid-operation on the two boxes an aborted run left behind. The timeline nails it: `incusd` has been up 1h38m, but both qemu processes are only ~30 min old — they were (re)started exactly when run 6 began. Run 6 stalled in `setup-host.sh`, and setup-host runs: ```sh incus profile edit claude-dev < profiles/claude-dev.yaml ``` — rewriting a profile that **two running VMs are attached to**, which forces a live CPU/memory reconfiguration on both. That is where it deadlocked. ## The finding **`setup-host.sh` can wedge the Incus daemon when boxes are running.** It presents itself as idempotent and safe to re-run, and it is — *only on an idle host*. Re-running it with live boxes (an upgrade, a re-provision, a second drill run) is reachable from normal use, and the failure mode is a hung daemon, not an error message. Two things follow: - **#25** (drill: clean boxes *before* `setup-host`) is the right mitigation, not just a drill nicety. - **`setup-host.sh` itself should not silently reconfigure a profile that running instances depend on** — it should detect them and refuse, or warn and skip the profile edit. Tracking that as its own fix. Memory is worth a footnote: `incus.service` shows **4G resident, 8.7G peak** on a 15GiB host with two 3GiB VMs. Not obviously the cause, but not nothing. ## Next: capture the deadlock, then recover `incusd` is Go, so `SIGQUIT` dumps every goroutine stack and exits — which both gives us the trace *and* clears the wedge. Capture before killing: ```sh # 1. dump the goroutine stacks (this is the evidence — it names the deadlock) sudo kill -QUIT 22114 sleep 3 sudo journalctl -u incus --no-pager -n 2000 | grep -iE 'goroutine|deadlock|panic|sync\.|incus/internal' | head -60 sudo journalctl -u incus --no-pager --since '2 min ago' > /tmp/incus-deadlock.log wc -l /tmp/incus-deadlock.log # paste the interesting part here # 2. clear the paused VMs sudo pkill -9 -f 'qemu-system-x86_64 -S -name archive' sudo pkill -9 -f 'qemu-system-x86_64 -S -name peer' # 3. bring the daemon back sudo systemctl restart incus.socket incus.service timeout 10 incus list; echo "exit: $?" # expect 0, and a clean list # 4. if instances linger in a broken state timeout 30 incus delete -f archive 2>/dev/null timeout 30 incus delete -f peer 2>/dev/null timeout 10 incus list ``` Then re-run the drill **on this host** (#25 clears leftovers before setup-host, so the trigger is gone): ```sh cd claudebox && git pull && bash drill/drill.sh --yes ```

dan-claude-bot commented

2026-07-13 23:20:32 +00:00

(Migrated from github.com)

Recovery — run this block (the previous one stalled at `systemctl restart`)

Two things went sideways and neither is a problem:

systemctl restart hung. systemd asks the daemon to stop politely, and the daemon is deadlocked — so it waits out its stop timeout. Don't ask politely.
The goroutine dump isn't in the journal. incusd runs with --logfile=/var/log/incus/incus.log, so SIGQUIT's stack dump went there (hence wc -l = 16 on the journal capture).

The qemu kills did land (Killed, twice). Ctrl-C whatever is still hanging, then:

# --- force the daemon down; it cannot answer, so do not wait on it ---
sudo systemctl kill -s SIGKILL incus.service
sudo systemctl reset-failed incus.service
sudo pkill -9 -f qemu-system-x86_64          # any stragglers
sudo systemctl start incus.service
sleep 3
timeout 10 incus list; echo "incus list exit: $?"    # expect 0

# --- the deadlock trace: it went to the logfile, not the journal ---
ls -la /var/log/incus/
sudo grep -c goroutine /var/log/incus/incus.log
sudo sed -n '/goroutine /,$p' /var/log/incus/incus.log | head -80

# --- clear the two boxes that triggered it, if they linger ---
timeout 30 incus delete -f archive 2>/dev/null
timeout 30 incus delete -f peer    2>/dev/null
timeout 10 incus list

Paste the goroutine output here if there is any. If there isn't, no loss — the evidence is already sufficient (see below).

Then: re-run the drill on THIS host

cd claudebox && git pull && bash drill/drill.sh --yes

#25 deletes leftover boxes before setup-host.sh runs, so the trigger is gone. This should finally answer A3 and close #15.

If the daemon will not come back

Only then reprovision — and say so here first. The forensics above are the finding; they are not lost by rebuilding the host afterwards.

The finding, restated (this stands with or without the trace)

setup-host.sh can deadlock the Incus daemon when it is re-run while boxes are running.

Evidence:

Two VMs (archive, peer) stuck in qemu's -S state — started, never resumed — for ~30 minutes. Incus always launches qemu paused and resumes it over the monitor socket; these never got resumed.
Both qemu processes were ~30 min old while incusd had been up 1h38m — i.e. they were (re)started exactly when run 6 began.
Run 6 stalled inside setup-host.sh, which unconditionally runs incus profile edit claude-dev < profiles/claude-dev.yaml — rewriting a profile two live VMs are attached to, forcing a CPU/memory reconfiguration on running guests.
Daemon API deadlocked ever since: list/info/operation list all time out; socket still held; disk fine (16G/452G).

setup-host.sh advertises itself as idempotent and safe to re-run. It is — on an idle host. Re-running it with live boxes is reachable from ordinary use (an upgrade, a re-provision, a second run), and the failure mode is a hung daemon rather than an error. It should detect running instances and refuse, or skip the profile edit. Filing that as its own issue.

## Recovery — run this block (the previous one stalled at `systemctl restart`) Two things went sideways and neither is a problem: - **`systemctl restart` hung.** systemd asks the daemon to stop politely, and the daemon is deadlocked — so it waits out its stop timeout. Don't ask politely. - **The goroutine dump isn't in the journal.** `incusd` runs with `--logfile=/var/log/incus/incus.log`, so `SIGQUIT`'s stack dump went *there* (hence `wc -l` = 16 on the journal capture). The qemu kills did land (`Killed`, twice). Ctrl-C whatever is still hanging, then: ```sh # --- force the daemon down; it cannot answer, so do not wait on it --- sudo systemctl kill -s SIGKILL incus.service sudo systemctl reset-failed incus.service sudo pkill -9 -f qemu-system-x86_64 # any stragglers sudo systemctl start incus.service sleep 3 timeout 10 incus list; echo "incus list exit: $?" # expect 0 # --- the deadlock trace: it went to the logfile, not the journal --- ls -la /var/log/incus/ sudo grep -c goroutine /var/log/incus/incus.log sudo sed -n '/goroutine /,$p' /var/log/incus/incus.log | head -80 # --- clear the two boxes that triggered it, if they linger --- timeout 30 incus delete -f archive 2>/dev/null timeout 30 incus delete -f peer 2>/dev/null timeout 10 incus list ``` Paste the `goroutine` output here if there is any. **If there isn't, no loss** — the evidence is already sufficient (see below). ## Then: re-run the drill on THIS host ```sh cd claudebox && git pull && bash drill/drill.sh --yes ``` #25 deletes leftover boxes **before** `setup-host.sh` runs, so the trigger is gone. This should finally answer **A3** and close [#15](https://github.com/heavy-duty/claudebox/issues/15). ## If the daemon will not come back Only then reprovision — and say so here first. The forensics above are the finding; they are not lost by rebuilding the host afterwards. ## The finding, restated (this stands with or without the trace) **`setup-host.sh` can deadlock the Incus daemon when it is re-run while boxes are running.** Evidence: - Two VMs (`archive`, `peer`) stuck in qemu's `-S` state — **started, never resumed** — for ~30 minutes. Incus always launches qemu paused and resumes it over the monitor socket; these never got resumed. - Both qemu processes were ~30 min old while `incusd` had been up 1h38m — i.e. they were (re)started **exactly when run 6 began**. - Run 6 stalled inside `setup-host.sh`, which unconditionally runs `incus profile edit claude-dev < profiles/claude-dev.yaml` — rewriting a profile **two live VMs are attached to**, forcing a CPU/memory reconfiguration on running guests. - Daemon API deadlocked ever since: `list`/`info`/`operation list` all time out; socket still held; disk fine (16G/452G). `setup-host.sh` advertises itself as idempotent and safe to re-run. It is — **on an idle host.** Re-running it with live boxes is reachable from ordinary use (an upgrade, a re-provision, a second run), and the failure mode is a hung daemon rather than an error. It should detect running instances and refuse, or skip the profile edit. Filing that as its own issue.

dan-claude-bot commented

2026-07-13 23:30:14 +00:00

(Migrated from github.com)

Daemon recovered — and it was stuck shutting down, not deadlocked

pgrep showed only incusd shutdown (systemd's ExecStop helper) waiting on a main daemon we had already SIGKILLed and that could never answer it. The unit sat in activating forever, so every incus call blocked on a socket activation that could not complete. The sql: database is closed lines were post-mortem noise from the kill, not the cause.

What cleared it:

sudo pkill -9 -f 'incusd shutdown'; sudo systemctl stop incus.service incus.socket; sleep 2; sudo systemctl reset-failed incus.service incus.socket; sudo systemctl start incus.socket incus.service; sleep 5; systemctl is-active incus; timeout 10 incus list; echo "EXIT=$?"

→ active, EXIT=0. No reprovision needed. (No goroutine dump survived — grep -c goroutine /var/log/incus/incus.log → 0 — so the exact mutex is unnamed, but the sequence in the previous comment stands on the process evidence.)

The recovered `incus list` exposed TWO findings the drill could never have reported

| archive | RUNNING | 10.87.0.128 (enp5s0) | VIRTUAL-MACHINE | 1 |
| peer    | RUNNING | 10.87.0.128 (enp5s0) | VIRTUAL-MACHINE | 0 |

Finding 1 — the NIC inside a VM is `enp5s0`, not `eth0`. This is why A3 has never fired.

The profile names the device eth0 (profiles/claude-dev.yaml), but inside a VM guest predictable naming renames it enp5s0. Every address lookup in the drill has been hunting an interface that does not exist:

the original CSV match /$eth0$$/ → never matched
its replacement, ip -4 -o addr show dev eth0 → also never matched

Six runs, two "fixes", and the assumption underneath both was wrong. A3's failure was never a network finding — it was this. (Note box_ipv4() in bin/claudebox has the same eth0 assumption baked into its comment, though it takes the first address rather than selecting by name.)

Finding 2 — a clone gets the SAME IP as its source. This is a claudebox bug, not a drill bug.

peer was cloned from archive/authed, and both boxes now hold 10.87.0.128. The snapshot→clone workflow — log in once, snapshot, clone forever — is the entire reuse story of this tool, and clones are colliding on the network.

Almost certainly: the guest's /etc/machine-id is cloned with the disk, DHCP uses it as the client identifier, and dnsmasq therefore hands the clone its source's lease. (Consistent with an earlier run where the two boxes did differ — leases collide only when both are up and the client-id matches.)

It also means A3 was doomed even with the interface name right: probing archive → 10.87.0.128 is archive probing itself.

Confirmation — one line, please

for b in archive peer; do echo "== $b"; timeout 20 incus exec $b -- ip -4 -o addr show scope global; timeout 20 incus exec $b -- cat /etc/machine-id; timeout 20 incus exec $b -- cat /sys/class/net/enp5s0/address; done

Reading it:

Observation	Conclusion
machine-ids match, MACs differ	Confirmed: cloned `machine-id` → identical DHCP client-id → same lease. Fix is a machine-id reset at clone time (cloud-init, or `claudebox new --from`).
machine-ids differ, MACs match	`incus copy` did not regenerate the MAC — different fix, same symptom.
both differ, addresses still equal	Something else in the DHCP path; dig into dnsmasq's lease file.

What this changes

#25 stands (clean before setup) — it is what stops the daemon wedge recurring.
The drill needs the interface name fixed (enp5s0 in VMs, eth0 in containers — read whichever exists rather than assuming).
Finding 2 gets its own issue against claudebox proper: clones must not inherit their source's identity. That is a correctness bug in the tool's headline feature, and it is worth more than the audit that found it.

## Daemon recovered — and it was stuck *shutting down*, not deadlocked `pgrep` showed only `incusd shutdown` (systemd's `ExecStop` helper) waiting on a main daemon we had already `SIGKILL`ed and that could never answer it. The unit sat in `activating` forever, so every `incus` call blocked on a socket activation that could not complete. The `sql: database is closed` lines were post-mortem noise from the kill, not the cause. What cleared it: ```sh sudo pkill -9 -f 'incusd shutdown'; sudo systemctl stop incus.service incus.socket; sleep 2; sudo systemctl reset-failed incus.service incus.socket; sudo systemctl start incus.socket incus.service; sleep 5; systemctl is-active incus; timeout 10 incus list; echo "EXIT=$?" ``` → `active`, `EXIT=0`. **No reprovision needed.** (No goroutine dump survived — `grep -c goroutine /var/log/incus/incus.log` → 0 — so the exact mutex is unnamed, but the sequence in the previous comment stands on the process evidence.) --- ## The recovered `incus list` exposed TWO findings the drill could never have reported ``` | archive | RUNNING | 10.87.0.128 (enp5s0) | VIRTUAL-MACHINE | 1 | | peer | RUNNING | 10.87.0.128 (enp5s0) | VIRTUAL-MACHINE | 0 | ``` ### Finding 1 — the NIC inside a VM is `enp5s0`, not `eth0`. This is why A3 has never fired. The profile names the *device* `eth0` (`profiles/claude-dev.yaml`), but inside a **VM guest** predictable naming renames it **`enp5s0`**. Every address lookup in the drill has been hunting an interface that does not exist: - the original CSV match `/$eth0$$/` → never matched - its replacement, `ip -4 -o addr show dev eth0` → also never matched Six runs, two "fixes", and the assumption underneath both was wrong. **A3's failure was never a network finding — it was this.** (Note `box_ipv4()` in `bin/claudebox` has the same `eth0` assumption baked into its comment, though it takes the first address rather than selecting by name.) ### Finding 2 — a clone gets the SAME IP as its source. This is a claudebox bug, not a drill bug. `peer` was cloned from `archive/authed`, and both boxes now hold **`10.87.0.128`**. The snapshot→clone workflow — *log in once, snapshot, clone forever* — is the entire reuse story of this tool, and clones are **colliding on the network**. Almost certainly: the guest's `/etc/machine-id` is cloned with the disk, DHCP uses it as the client identifier, and dnsmasq therefore hands the clone its source's lease. (Consistent with an earlier run where the two boxes *did* differ — leases collide only when both are up and the client-id matches.) It also means **A3 was doomed even with the interface name right**: probing `archive → 10.87.0.128` is archive probing *itself*. ## Confirmation — one line, please ```sh for b in archive peer; do echo "== $b"; timeout 20 incus exec $b -- ip -4 -o addr show scope global; timeout 20 incus exec $b -- cat /etc/machine-id; timeout 20 incus exec $b -- cat /sys/class/net/enp5s0/address; done ``` Reading it: | Observation | Conclusion | | --- | --- | | **machine-ids match**, MACs differ | Confirmed: cloned `machine-id` → identical DHCP client-id → same lease. Fix is a machine-id reset at clone time (cloud-init, or `claudebox new --from`). | | machine-ids differ, **MACs match** | `incus copy` did not regenerate the MAC — different fix, same symptom. | | both differ, addresses still equal | Something else in the DHCP path; dig into dnsmasq's lease file. | ## What this changes - **#25 stands** (clean before setup) — it is what stops the daemon wedge recurring. - The drill needs the interface name fixed (`enp5s0` in VMs, `eth0` in containers — read whichever exists rather than assuming). - Finding 2 gets its own issue against claudebox proper: **clones must not inherit their source's identity.** That is a correctness bug in the tool's headline feature, and it is worth more than the audit that found it.

dan-claude-bot commented

2026-07-14 12:10:27 +00:00

(Migrated from github.com)

Closing: the forensic sequence this issue demanded ran to a verdict (see thread) — the daemon was not deadlocked but stuck in shutdown, an incusd shutdown ExecStop helper waiting forever on a main process we had already killed, wedging socket activation. Cleared with pkill -9 -f 'incusd shutdown' + stop/reset-failed/start; the recovery recipe now lives in doctor.sh's daemon pre-check, which refuses to run and prints it when the daemon is unresponsive.

Runs 7–11 all completed on this host since. The related debris hypothesis is also settled: the drill no longer poisons the host (#32) and the doctor exists precisely to prove that before every run.

🤖 Generated with Claude Code

Closing: the forensic sequence this issue demanded ran to a verdict (see thread) — the daemon was not deadlocked but stuck in *shutdown*, an `incusd shutdown` ExecStop helper waiting forever on a main process we had already killed, wedging socket activation. Cleared with `pkill -9 -f 'incusd shutdown'` + stop/reset-failed/start; the recovery recipe now lives in `doctor.sh`'s daemon pre-check, which refuses to run and prints it when the daemon is unresponsive. Runs 7–11 all completed on this host since. The related debris hypothesis is also settled: the drill no longer poisons the host (#32) and the doctor exists precisely to prove that before every run. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/box#26

No description provided.

Incus daemon wedged on the drill host — forensics before reprovisioning #26

Step 1 — evidence (restart nothing)

Step 2 — read the evidence

Step 3 — recovery, least destructive first

Step 4 — once it answers again

Why this is worth the detour

Evidence is in, and it rules out the disk

The finding

Next: capture the deadlock, then recover

Recovery — run this block (the previous one stalled at systemctl restart)

Then: re-run the drill on THIS host

If the daemon will not come back

The finding, restated (this stands with or without the trace)

Daemon recovered — and it was stuck shutting down, not deadlocked

The recovered incus list exposed TWO findings the drill could never have reported

Finding 1 — the NIC inside a VM is enp5s0, not eth0. This is why A3 has never fired.

Finding 2 — a clone gets the SAME IP as its source. This is a claudebox bug, not a drill bug.

Confirmation — one line, please

What this changes

Recovery — run this block (the previous one stalled at `systemctl restart`)

The recovered `incus list` exposed TWO findings the drill could never have reported

Finding 1 — the NIC inside a VM is `enp5s0`, not `eth0`. This is why A3 has never fired.