heavy-duty/box

Fork 1

Running `box setup-host` inside a box creates a nested `boxnet` on the same hardcoded `10.88.0.0/24` as the guest's own uplink, causing intermittent egress blackouts #80

New issue

Closed

opened 2026-07-18 16:37:35 +00:00 by danmt · 0 comments

danmt commented

2026-07-18 16:37:35 +00:00

(Migrated from github.com)

Summary

The host stack's subnet (10.88.0.0/24, gateway 10.88.0.1) is hardcoded. If box setup-host (or the drill) is ever run inside a box, the guest gets a nested boxnet bridge claiming the exact subnet and gateway address of its own uplink. The guest then holds 10.88.0.1/24 as a local address while its default route points at 10.88.0.1 as its gateway, and carries two conflicting 10.88.0.0/24 connected routes. The result is intermittent, self-recovering egress blackouts (tens of seconds, roughly periodic) that are extremely hard to attribute — the host looks healthy, box doctor on the host reports clean, and the symptom presents as flaky internet for whatever runs inside the box (in my case, Claude Code showing "Waiting for API response… check your network" about once an hour).

This is a realistic footgun for this project in particular: the flagship use case is a coding agent working on a repo inside a box, and the repo most likely to be worked on by this project's own users is box itself. An agent (or a human) following the README inside a box will run box setup-host or drill/drill.sh and silently poison the guest's networking.

Environment

Host: Lenovo ThinkCentre M900 (4 cores), Ubuntu, Incus, box 0.5.x stack (boxnet / box-isolate / box-firewall on 10.88.0.0/24)
Guest: claude template VM (dan-claude), Debian 13, minted normally, uplink enp5s0 via DHCP from the host's boxnet
Host also runs Tailscale (initially suspected; fully exonerated — no exit node, table 52 contains only per-peer /32 routes)

Symptoms

Claude Code inside the box stalls on "Waiting for API response" roughly once an hour, recovering on its own after some seconds to a couple of minutes.
Anthropic status page clean; host connectivity clean; Tailscale path clean.
During one live incident, probed simultaneously: host ping 1.1.1.1 = 0% loss, box ping 1.1.1.1 = 66% loss and curl https://api.anthropic.com = 000 (connection failure) while the host got the expected 404.

A continuous ping -i 0.3 1.1.1.1 from inside the box captured two full outages back to back:

(seq 1–80 lost: ~24 s dead)
64 bytes from 1.1.1.1: icmp_seq=81 ttl=52 time=1856 ms
64 bytes from 1.1.1.1: icmp_seq=82 ttl=52 time=1536 ms
64 bytes from 1.1.1.1: icmp_seq=83 ttl=52 time=1216 ms
64 bytes from 1.1.1.1: icmp_seq=84 ttl=52 time=896 ms
64 bytes from 1.1.1.1: icmp_seq=85 ttl=52 time=576 ms
64 bytes from 1.1.1.1: icmp_seq=86 ttl=52 time=256 ms
64 bytes from 1.1.1.1: icmp_seq=87 ttl=52 time=30.7 ms
... (normal ~30 ms)
(seq 114–233 lost: ~36 s dead)
64 bytes from 1.1.1.1: icmp_seq=234 ttl=52 time=288 ms
... (normal again)

Notes on the capture: sequence numbers kept advancing through the gaps, so the guest was scheduled and transmitting the whole time (not CPU starvation), and the descending-latency staircase on recovery shows ~2 s of queued packets flushing when the path came back.

box doctor run as an unprivileged user (which probes a live box) had already caught the signature without my understanding it at the time:

Can one of your boxes actually resolve DNS?
        probing inside 'dan-claude':
  DIRTY cannot reach 1.1.1.1 by address — egress routing is broken (an admin problem: box doctor as admin)
  OK    resolves deb.debian.org — DNS works

"Egress broken but DNS fine" is itself a fingerprint of this bug — see mechanism below. Meanwhile box doctor as root on the host reported fully clean (it validated the host stack's config, and had no instance in its project to probe).

Root cause

incus list dan-claude -c ns4 gave it away:

| dan-claude | RUNNING | 172.17.0.1 (docker0)       |
|            |         | 10.88.0.202 (enp5s0)       |
|            |         | 10.88.0.1 (boxnet)         |   <-- nested box stack INSIDE the guest
|            |         | 10.202.113.1 (incusbr0)    |
|            |         | 10.171.99.1 (incusbr-1002) |

At some point the box stack was installed inside the box (almost certainly by the coding agent while working on this repo — running setup-host/drill per the README). Inside the guest:

$ ip route show
default via 10.88.0.1 dev enp5s0 proto dhcp src 10.88.0.202 metric 1024
10.88.0.0/24 dev boxnet proto kernel scope link src 10.88.0.1 linkdown
10.88.0.0/24 dev enp5s0 proto kernel scope link src 10.88.0.202 metric 1024
10.88.0.1 dev enp5s0 proto dhcp scope link src 10.88.0.202 metric 1024
...

$ ip -4 addr show boxnet
17: boxnet: <NO-CARRIER,BROADCAST,MULTICAST,UP> ...
    inet 10.88.0.1/24 scope global boxnet

So the guest simultaneously has:

Its own gateway's IP as a local address. The kernel's local routing table is consulted before the main table, so packets addressed to 10.88.0.1 (DNS queries to the bridge resolver, unicast DHCP renewals to the DHCP server) can be delivered to the guest itself instead of the real gateway.
A nested dnsmasq answering on 10.88.0.1, intercepting the guest's DNS. This explains the doctor's "DNS works, egress broken" split: DNS was being served by the impostor (which forwarded upstream successfully via the default route), while direct IP egress suffered.
Two connected routes for 10.88.0.0/24 — one via the real uplink (metric 1024), one via the nested bridge (no metric, i.e. metric 0, which wins whenever the nested bridge has carrier). The nested bridge shows linkdown when idle, which keeps its routes inactive — but any activity that brings carrier up (a nested box starting, the drill, bridge churn) instantly makes the nested route preferred and blackholes the subnet.

Why it's intermittent and roughly periodic

The guest's DHCP lease from the host bridge is short (valid_lft 1143sec observed, i.e. ~19 min lease). DHCP renewal at T1 is a unicast to the server — 10.88.0.1 — which the local table can capture, so renewals silently fail; recovery happens later via broadcast (rebind), which reaches the real server. Depending on timing and on whether the nested dnsmasq also answers the broadcast (it is a DHCP server for the same subnet!), the guest experiences windows of tens of seconds with broken or churning addressing/ARP state, then self-heals. This matches the observed cadence ("about once an hour", outages of 24 s and 36 s) without any fault on the host, the NIC, or the uplink. I did not pin down the exact packet-level sequence of each outage; the observations above (two outage windows, lease timing, local-table capture, duplicate routes, impostor dnsmasq) are the measured facts, and each listed mechanism is sufficient on its own to break egress once the nested bridge gains carrier.

Workaround (verified)

Moving the nested bridge off the colliding subnet immediately and durably fixed the blips:

# inside the box
sudo incus network set boxnet ipv4.address 10.89.0.1/24

(Deleting the nested stack entirely — box teardown-host inside the guest — should work equally well.)

Suggested fixes

Any one of these would have prevented the failure; together they'd make the footgun essentially impossible:

setup-host should detect it is running on a box (or more generally: that any existing interface/route already owns an address inside 10.88.0.0/24, e.g. the default route's gateway is in it) and refuse with a clear message — or automatically select a non-colliding subnet.
Make the subnet configurable (BOX_SUBNET, alongside the existing BOX_DNS), so a nested or otherwise-conflicted install has a sanctioned way out. Nested box-in-box may be a non-goal, but the tool's own README funnels agents toward doing exactly this, so failing loudly or adapting seems in scope.
Teach box doctor (run inside a guest, or the host doctor probing a guest) to recognize the signature: a local address equal to the default gateway, and/or duplicate connected routes for the uplink subnet. The doctor already caught the symptom (DIRTY cannot reach 1.1.1.1 by address while DNS resolves) but attributed it to the host admin; this exact split (IP egress broken, DNS fine) could hint at "is something inside this box squatting on the gateway address?"
Consider guarding the claude/codex/grok templates' agent-context file with a note that setup-host/the drill must never be run inside a box — the drill's own docs say "use a machine you own," but an agent inside a box arguably believes it owns the machine.

Reproduction

box new --name victim --template claude (VM mode), enter it.
Inside the guest, follow the README: install box (curl … install.sh | bash), run box setup-host (and optionally mint a nested box or run the drill to give the nested bridge carrier).
Observe: ip route shows two 10.88.0.0/24 connected routes and 10.88.0.1 held locally; DNS is now served by the nested dnsmasq; egress degrades intermittently, correlating with nested-bridge carrier and DHCP lease events.
sudo incus network set boxnet ipv4.address 10.89.0.1/24 inside the guest → symptoms stop.

Credits / trail

Diagnosed by bisecting live incidents: simultaneous host-vs-guest probes proved the host path clean while the guest lost 66% of packets; a timestamped ping capture ruled out CPU starvation and showed queue-flush recovery; incus list exposed the nested bridge; ip route confirmed the collision; changing the nested subnet resolved it. Happy to provide the full captures or re-test a patched setup-host.

## Summary The host stack's subnet (`10.88.0.0/24`, gateway `10.88.0.1`) is hardcoded. If `box setup-host` (or the drill) is ever run **inside** a box, the guest gets a nested `boxnet` bridge claiming the exact subnet and gateway address of its own uplink. The guest then holds `10.88.0.1/24` as a *local* address while its default route points at `10.88.0.1` as its *gateway*, and carries two conflicting `10.88.0.0/24` connected routes. The result is intermittent, self-recovering egress blackouts (tens of seconds, roughly periodic) that are extremely hard to attribute — the host looks healthy, `box doctor` on the host reports clean, and the symptom presents as flaky internet for whatever runs inside the box (in my case, Claude Code showing "Waiting for API response… check your network" about once an hour). This is a realistic footgun for this project in particular: the flagship use case is a coding agent working on a repo inside a box, and the repo most likely to be worked on by this project's own users is **box itself**. An agent (or a human) following the README inside a box will run `box setup-host` or `drill/drill.sh` and silently poison the guest's networking. ## Environment * Host: Lenovo ThinkCentre M900 (4 cores), Ubuntu, Incus, box 0.5.x stack (`boxnet` / `box-isolate` / `box-firewall` on `10.88.0.0/24`) * Guest: `claude` template VM (`dan-claude`), Debian 13, minted normally, uplink `enp5s0` via DHCP from the host's `boxnet` * Host also runs Tailscale (initially suspected; fully exonerated — no exit node, table 52 contains only per-peer /32 routes) ## Symptoms * Claude Code inside the box stalls on "Waiting for API response" roughly once an hour, recovering on its own after some seconds to a couple of minutes. * Anthropic status page clean; host connectivity clean; Tailscale path clean. * During one live incident, probed simultaneously: host `ping 1.1.1.1` = 0% loss, box `ping 1.1.1.1` = 66% loss and `curl https://api.anthropic.com` = `000` (connection failure) while the host got the expected `404`. A continuous `ping -i 0.3 1.1.1.1` from inside the box captured two full outages back to back: ``` (seq 1–80 lost: ~24 s dead) 64 bytes from 1.1.1.1: icmp_seq=81 ttl=52 time=1856 ms 64 bytes from 1.1.1.1: icmp_seq=82 ttl=52 time=1536 ms 64 bytes from 1.1.1.1: icmp_seq=83 ttl=52 time=1216 ms 64 bytes from 1.1.1.1: icmp_seq=84 ttl=52 time=896 ms 64 bytes from 1.1.1.1: icmp_seq=85 ttl=52 time=576 ms 64 bytes from 1.1.1.1: icmp_seq=86 ttl=52 time=256 ms 64 bytes from 1.1.1.1: icmp_seq=87 ttl=52 time=30.7 ms ... (normal ~30 ms) (seq 114–233 lost: ~36 s dead) 64 bytes from 1.1.1.1: icmp_seq=234 ttl=52 time=288 ms ... (normal again) ``` Notes on the capture: sequence numbers kept advancing through the gaps, so the guest was scheduled and transmitting the whole time (not CPU starvation), and the descending-latency staircase on recovery shows ~2 s of queued packets flushing when the path came back. `box doctor` run as an unprivileged user (which probes a live box) had already caught the signature without my understanding it at the time: ``` Can one of your boxes actually resolve DNS? probing inside 'dan-claude': DIRTY cannot reach 1.1.1.1 by address — egress routing is broken (an admin problem: box doctor as admin) OK resolves deb.debian.org — DNS works ``` "Egress broken but DNS fine" is itself a fingerprint of this bug — see mechanism below. Meanwhile `box doctor` as root on the host reported fully clean (it validated the host stack's config, and had no instance in its project to probe). ## Root cause `incus list dan-claude -c ns4` gave it away: ``` | dan-claude | RUNNING | 172.17.0.1 (docker0) | | | | 10.88.0.202 (enp5s0) | | | | 10.88.0.1 (boxnet) | <-- nested box stack INSIDE the guest | | | 10.202.113.1 (incusbr0) | | | | 10.171.99.1 (incusbr-1002) | ``` At some point the box stack was installed inside the box (almost certainly by the coding agent while working on this repo — running `setup-host`/drill per the README). Inside the guest: ``` $ ip route show default via 10.88.0.1 dev enp5s0 proto dhcp src 10.88.0.202 metric 1024 10.88.0.0/24 dev boxnet proto kernel scope link src 10.88.0.1 linkdown 10.88.0.0/24 dev enp5s0 proto kernel scope link src 10.88.0.202 metric 1024 10.88.0.1 dev enp5s0 proto dhcp scope link src 10.88.0.202 metric 1024 ... $ ip -4 addr show boxnet 17: boxnet: <NO-CARRIER,BROADCAST,MULTICAST,UP> ... inet 10.88.0.1/24 scope global boxnet ``` So the guest simultaneously has: 1. **Its own gateway's IP as a local address.** The kernel's `local` routing table is consulted before the main table, so packets addressed to `10.88.0.1` (DNS queries to the bridge resolver, unicast DHCP renewals to the DHCP server) can be delivered to the guest itself instead of the real gateway. 2. **A nested dnsmasq answering on `10.88.0.1`**, intercepting the guest's DNS. This explains the doctor's "DNS works, egress broken" split: DNS was being served by the impostor (which forwarded upstream successfully via the default route), while direct IP egress suffered. 3. **Two connected routes for `10.88.0.0/24`** — one via the real uplink (metric 1024), one via the nested bridge (no metric, i.e. metric 0, which wins whenever the nested bridge has carrier). The nested bridge shows `linkdown` when idle, which keeps its routes inactive — but any activity that brings carrier up (a nested box starting, the drill, bridge churn) instantly makes the nested route preferred and blackholes the subnet. ### Why it's intermittent and roughly periodic The guest's DHCP lease from the host bridge is short (`valid_lft 1143sec` observed, i.e. ~19 min lease). DHCP renewal at T1 is a **unicast** to the server — `10.88.0.1` — which the local table can capture, so renewals silently fail; recovery happens later via **broadcast** (rebind), which reaches the real server. Depending on timing and on whether the nested dnsmasq also answers the broadcast (it is a DHCP server for the same subnet!), the guest experiences windows of tens of seconds with broken or churning addressing/ARP state, then self-heals. This matches the observed cadence ("about once an hour", outages of 24 s and 36 s) without any fault on the host, the NIC, or the uplink. I did not pin down the exact packet-level sequence of each outage; the observations above (two outage windows, lease timing, local-table capture, duplicate routes, impostor dnsmasq) are the measured facts, and each listed mechanism is sufficient on its own to break egress once the nested bridge gains carrier. ## Workaround (verified) Moving the nested bridge off the colliding subnet immediately and durably fixed the blips: ``` # inside the box sudo incus network set boxnet ipv4.address 10.89.0.1/24 ``` (Deleting the nested stack entirely — `box teardown-host` inside the guest — should work equally well.) ## Suggested fixes Any one of these would have prevented the failure; together they'd make the footgun essentially impossible: 1. **`setup-host` should detect it is running on a box** (or more generally: that any existing interface/route already owns an address inside `10.88.0.0/24`, e.g. the default route's gateway is in it) and refuse with a clear message — or automatically select a non-colliding subnet. 2. **Make the subnet configurable** (`BOX_SUBNET`, alongside the existing `BOX_DNS`), so a nested or otherwise-conflicted install has a sanctioned way out. Nested box-in-box may be a non-goal, but the tool's own README funnels agents toward doing exactly this, so failing loudly or adapting seems in scope. 3. **Teach `box doctor` (run inside a guest, or the host doctor probing a guest) to recognize the signature**: a local address equal to the default gateway, and/or duplicate connected routes for the uplink subnet. The doctor already caught the *symptom* (`DIRTY cannot reach 1.1.1.1 by address` while DNS resolves) but attributed it to the host admin; this exact split (IP egress broken, DNS fine) could hint at "is something inside this box squatting on the gateway address?" 4. **Consider guarding the `claude`/`codex`/`grok` templates' agent-context file** with a note that `setup-host`/the drill must never be run inside a box — the drill's own docs say "use a machine you own," but an agent inside a box arguably believes it owns the machine. ## Reproduction 1. `box new --name victim --template claude` (VM mode), enter it. 2. Inside the guest, follow the README: install box (`curl … install.sh | bash`), run `box setup-host` (and optionally mint a nested box or run the drill to give the nested bridge carrier). 3. Observe: `ip route` shows two `10.88.0.0/24` connected routes and `10.88.0.1` held locally; DNS is now served by the nested dnsmasq; egress degrades intermittently, correlating with nested-bridge carrier and DHCP lease events. 4. `sudo incus network set boxnet ipv4.address 10.89.0.1/24` inside the guest → symptoms stop. ## Credits / trail Diagnosed by bisecting live incidents: simultaneous host-vs-guest probes proved the host path clean while the guest lost 66% of packets; a timestamped ping capture ruled out CPU starvation and showed queue-flush recovery; `incus list` exposed the nested bridge; `ip route` confirmed the collision; changing the nested subnet resolved it. Happy to provide the full captures or re-test a patched `setup-host`.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/box#80

No description provided.

Running box setup-host inside a box creates a nested boxnet on the same hardcoded 10.88.0.0/24 as the guest's own uplink, causing intermittent egress blackouts #80