fix(doctor): check that something is actually serving the network #35

Merged

dan-claude-bot merged 1 commit from drill/doctor-dnsmasq into main

2026-07-14 01:23:51 +00:00

dan-claude-bot commented

2026-07-14 01:13:40 +00:00

(Migrated from github.com)

Two cold mints died in cloud-init with Temporary failure resolving deb.debian.org, on a host doctor.sh had just certified clean.

It was not DNS forwarding (#33), not leftover phase-D mutations, and not the drill. claudenet had no dnsmasq. It never respawned after the SIGKILL that recovered the wedged daemon back in run 6 — so every box since got no DHCP lease at all: no address, no gateway, no DNS. The Temporary failure resolving errors pointed at Debian's mirrors, and the actual fault was that the box had never been on the network.

Incus does not surface this

$ incus network show claudenet
config:
  ipv4.address: 10.87.0.1/24
  ...
status: Created          ← "Created"
managed: true

Bridge up. Config pristine. Nothing serving it. Only the process table knows:

$ pgrep -af 'dnsmasq.*--interface=claudenet'
(nothing)

So the doctor now asks the process table, and --fix restarts Incus to respawn it (falling back to "teardown and let the drill rebuild" if that doesn't work).

The repair, for the record

timeout 60 incus delete -f <boxes>; sudo systemctl restart incus

That brought it back on the live host.

Why this matters beyond the drill

Any claudebox host that survives an unclean Incus death — a wedge (#26), an OOM, a hard reboot mid-operation — can be left in this state. Every subsequent claudebox new then fails several minutes deep in cloud-init with an error that blames the internet. Nothing in claudebox notices, and nothing tells the user. It is a five-second check that cost two dead mints and an hour.

RUNS.md gains trap 11: a network Incus calls "Created" may have nothing serving it.

🤖 Generated with Claude Code

Two cold mints died in cloud-init with `Temporary failure resolving deb.debian.org`, on a host `doctor.sh` had **just certified clean**. It was not DNS forwarding (#33), not leftover phase-D mutations, and not the drill. **`claudenet` had no dnsmasq.** It never respawned after the `SIGKILL` that recovered the wedged daemon back in run 6 — so every box since got **no DHCP lease at all**: no address, no gateway, no DNS. The `Temporary failure resolving` errors pointed at Debian's mirrors, and the actual fault was that the box had never been on the network. ## Incus does not surface this ``` $ incus network show claudenet config: ipv4.address: 10.87.0.1/24 ... status: Created ← "Created" managed: true ``` Bridge up. Config pristine. Nothing serving it. **Only the process table knows:** ``` $ pgrep -af 'dnsmasq.*--interface=claudenet' (nothing) ``` So the doctor now asks the process table, and `--fix` restarts Incus to respawn it (falling back to "teardown and let the drill rebuild" if that doesn't work). ## The repair, for the record ```sh timeout 60 incus delete -f <boxes>; sudo systemctl restart incus ``` That brought it back on the live host. ## Why this matters beyond the drill Any claudebox host that survives an unclean Incus death — a wedge (#26), an OOM, a hard reboot mid-operation — can be left in this state. Every subsequent `claudebox new` then fails several minutes deep in cloud-init with an error that blames the internet. Nothing in claudebox notices, and nothing tells the user. It is a five-second check that cost two dead mints and an hour. `RUNS.md` gains **trap 11: a network Incus calls "Created" may have nothing serving it.** 🤖 Generated with [Claude Code](https://claude.com/claude-code)

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/box#35

No description provided.