bootstrap: add a `dev` role — the Incus claudebox host is the one box rig doesn't make #12

New issue

Closed

opened 2026-07-13 13:42:16 +00:00 by dan-claude-bot · 1 comment

dan-claude-bot commented

2026-07-13 13:42:16 +00:00

(Migrated from github.com)

rig bootstrap ships three roles today — control-plane, workload, runner. The one machine class that is not rig-made is the Incus host that runs the claudeboxes (dev-server). It was hand-built, so the fleet's "every box is rig-made, reproducibly" property has a hole in exactly the place an agent runs.

Proposed

rig bootstrap dev --hostname dev-server

Same shape as the other roles — idempotent, convergent, a second run is a no-op.

What it does:

base hardening: the 00-rig.conf sshd drop-in plus the sshd -T effective-config assert (the first-wins fix — must not regress)
system hostname convergence
tailscale join as tag:local
incus install + init: storage pool, default profile, network

Hard constraints

tag:local, never tag:server. The ACL grants tag:server → :22, so a mis-tagged dev host hands the control plane free SSH. This has already happened once on this tailnet — both M900s were carrying tag:server and had to be retagged. The role should make the correct tag the only reachable outcome, not a flag the operator remembers to pass.
The claudeboxes must not join the tailnet. The host joins; the guests do not. An agent-inhabited box on the tailnet is a foothold into the control plane, and the whole point of the claudebox is that it holds nothing and reaches nothing. Operator SSH into a claudebox goes through the host (ProxyJump dev-server, or an Incus proxy device) — never via a tailnet node of its own. If bootstrap dev ever grows a "join the guests too" convenience, that is the bug.
No credentials on the host. Claudeboxes are creds-free by design; the operator adds their own interactively. rig should not install, template, or hold any.

Context

Two bare-metal machines, one role each: dev-server (Incus claudeboxes, never Coolify-registered) and staging-server (hosts the Coolify-managed staging-box VM). staging-server's VM is already covered by the existing workload role. dev-server has no role at all — this issue is that gap.

Rehearsal

The existing Incus rehearsal is structurally blind: it runs in a pristine Debian container, which has no cloud-init drop-in, which is precisely why the 99-rig.conf vs 50-cloud-init.conf first-wins sshd bug survived it and shipped to every real box. A dev role rehearsal must assert effective state — sshd -T, incus info, tailscale status --json showing tag:local — and not the mere presence of files rig wrote.

`rig bootstrap` ships three roles today — `control-plane`, `workload`, `runner`. The one machine class that is **not** rig-made is the **Incus host that runs the claudeboxes** (`dev-server`). It was hand-built, so the fleet's "every box is rig-made, reproducibly" property has a hole in exactly the place an agent runs. ## Proposed ``` rig bootstrap dev --hostname dev-server ``` Same shape as the other roles — idempotent, convergent, a second run is a no-op. **What it does:** - base hardening: the `00-rig.conf` sshd drop-in **plus the `sshd -T` effective-config assert** (the first-wins fix — must not regress) - system hostname convergence - tailscale join as **`tag:local`** - `incus` install + init: storage pool, default profile, network ## Hard constraints - **`tag:local`, never `tag:server`.** The ACL grants `tag:server → :22`, so a mis-tagged dev host hands the control plane free SSH. This has already happened once on this tailnet — both M900s were carrying `tag:server` and had to be retagged. The role should make the correct tag the only reachable outcome, not a flag the operator remembers to pass. - **The claudeboxes must not join the tailnet.** The *host* joins; the *guests* do not. An agent-inhabited box on the tailnet is a foothold into the control plane, and the whole point of the claudebox is that it holds nothing and reaches nothing. Operator SSH into a claudebox goes through the host (`ProxyJump dev-server`, or an Incus proxy device) — never via a tailnet node of its own. If `bootstrap dev` ever grows a "join the guests too" convenience, that is the bug. - **No credentials on the host.** Claudeboxes are creds-free by design; the operator adds their own interactively. rig should not install, template, or hold any. ## Context Two bare-metal machines, one role each: `dev-server` (Incus claudeboxes, never Coolify-registered) and `staging-server` (hosts the Coolify-managed `staging-box` VM). staging-server's VM is already covered by the existing `workload` role. `dev-server` has no role at all — this issue is that gap. ## Rehearsal The existing Incus rehearsal is structurally blind: it runs in a pristine Debian container, which has no cloud-init drop-in, which is precisely why the `99-rig.conf` vs `50-cloud-init.conf` first-wins sshd bug survived it and shipped to every real box. A `dev` role rehearsal must assert **effective** state — `sshd -T`, `incus info`, `tailscale status --json` showing `tag:local` — and not the mere presence of files rig wrote.

dan-claude-bot commented

2026-07-17 14:19:05 +00:00

(Migrated from github.com)

Revisiting this against the current state of box and rig. The role is still right; four things in the spec above are now stale, and one of them would have rig re-implement — and quietly degrade — work box already does.

1. It's `box` now, not `claudebox`

The repo renamed to heavy-duty/box (v0.5.0, no longer "claudebox"). rig's README still links heavy-duty/claudebox in the philosophy line; GitHub redirects, so it works and will keep working, but it should be corrected while we're here. box's own installer has already been bitten by the rename once — its comments record the archive glob dying on every host when it kept looking for claudebox-*.

2. rig should NOT do the Incus init — `box` already does, better

The bullet above reads "incus install + init: storage pool, default profile, network". But box/host/setup-host.sh already does exactly that, and encodes decisions rig would lose by reimplementing:

btrfs on a loop device, deliberately not incus admin init --minimal — because minimal picks the dir backend, which has no copy-on-write, making every snapshot and clone a full multi-GB copy. That guts the entire "log in once, snapshot, clone forever" workflow the dev-server exists for. dir stays as a fallback, and it says so out loud when it lands there.
boxnet on 10.88, not 10.87 — because a pre-rename host may still have claudenet on 10.87 with legacy boxes attached, and two bridges must not claim one subnet.
The box-isolate ACL — egress allow to the gateway first (so instance DNS via dnsmasq survives), then explicit drops to 10/8, 172.16/12, and the rest of private space.

A rig-side incus admin init would, by default, do the naive thing and silently destroy the clone-forever property this box exists for. The dev role should install box and delegate host setup to box's own script. rig owns the OS plumbing; box owns what a box host is. That split is also just this repo's stated philosophy applied one level up.

3. `setup-host.sh` cannot be run as root — it will report success having done half the job

This is the part that needs design, not just a call:

if ! id -nG "$USER" | grep -qw incus-admin; then
  sudo usermod -aG incus-admin "$USER"
  echo "NOTE: added $USER to incus-admin — re-login (or 'sg incus-admin') and re-run."
  exit 0
fi

It is written for a sudo-capable non-root user: it sudos, and on first run it adds the group and exits 0 asking for a re-login. Invoked as root from rig bootstrap dev, that returns success with no storage pool, no network, no ACL — and rig, believing it, would print done — role dev.

That is the sshd first-wins bug's exact shape: rig trusting an exit code instead of checking effective state. Whatever the design, bootstrap dev must assert incus info / the pool / the network afterwards and refuse to claim success on the strength of exit 0.

Same root-vs-user problem in box's installer: it targets $HOME/.local/share/box and $HOME/.local/bin (overridable via BOX_HOME/BOX_BIN). Run as root, box lands in /root/.local/bin — installed for nobody who will use it.

Both point the same way: the dev role needs the admin user from #17, and needs box installed and set up as that user. #17 is a prerequisite for this one, not a neighbour.

4. `tag:local` — the flag this issue complains about is being deleted

The hard constraint above says "the role should make the correct tag the only reachable outcome, not a flag the operator remembers to pass" — and notes both M900s carried tag:server and had to be retagged.

#16 delivers exactly that: --ts-tag goes away, the pre-auth key's tags apply on their own, and rig verifies .Self.Tags from tailscale status --json on every run, first join or not. The dev role then needs no tag flag at all — just a policy entry (dev must carry tag:local, never tag:server) and a key minted with tag:local. The re-run verification is also what would have caught the M900s without anyone noticing by hand.

Worth stating plainly in this issue's spec, since "tailscale join as tag:local" above currently implies rig requests it.

5. Version pinning: there is nothing to pin to

box has no tags and no releases, and its installer builds https://github.com/$REPO/archive/refs/heads/$REF.tar.gz — refs/heads is branches only, so BOX_REF=v0.5.0 would 404 even if the tag existed. bootstrap dev can therefore only install a moving main.

By rig's own logic that is the wrong default: coolify install demands a pin because Coolify never self-updates, so its version is a contract. box doesn't self-update either — it has the same shape as Coolify, and the opposite of actions/runner (which is latest-by-default only because GitHub refuses jobs from stale runners). There's no --ephemeral-style forcing function here.

Not a blocker for this issue, but it needs a decision: either box cuts tags and teaches its installer refs/tags, or bootstrap dev installs main and the README says out loud that this one is unpinned and why. Silently tracking main on the box that runs the agents is the option not to pick.

Still correct, unchanged

The three hard constraints above all stand — tag:local never tag:server, the boxes must not join the tailnet (the host joins; guests reach the operator via ProxyJump dev-server), and no credentials on the host. The last one lines up with box's own design (creds-free; the operator runs /login interactively), so rig has nothing to do there but not break it.

The rehearsal note stands too, and gets easier: incus info and tailscale status --json showing tag:local are exactly the effective-state assertions #16 is building the reader for.

Revisiting this against the current state of `box` and rig. The role is still right; four things in the spec above are now stale, and one of them would have rig re-implement — and quietly degrade — work `box` already does. ## 1. It's `box` now, not `claudebox` The repo renamed to [`heavy-duty/box`](https://github.com/heavy-duty/box) (v0.5.0, no longer "claudebox"). rig's README still links `heavy-duty/claudebox` in the philosophy line; GitHub redirects, so it works and will keep working, but it should be corrected while we're here. `box`'s own installer has already been bitten by the rename once — its comments record the archive glob dying on every host when it kept looking for `claudebox-*`. ## 2. rig should NOT do the Incus init — `box` already does, better The bullet above reads *"`incus` install + init: storage pool, default profile, network"*. But [`box/host/setup-host.sh`](https://github.com/heavy-duty/box/blob/main/host/setup-host.sh) **already does exactly that**, and encodes decisions rig would lose by reimplementing: - **btrfs on a loop device, deliberately not `incus admin init --minimal`** — because minimal picks the `dir` backend, which has no copy-on-write, making every snapshot and clone a full multi-GB copy. That guts the entire "log in once, snapshot, clone forever" workflow the dev-server exists for. `dir` stays as a fallback, and it says so out loud when it lands there. - **`boxnet` on 10.88, not 10.87** — because a pre-rename host may still have `claudenet` on 10.87 with legacy boxes attached, and two bridges must not claim one subnet. - **The `box-isolate` ACL** — egress allow to the gateway first (so instance DNS via dnsmasq survives), then explicit drops to 10/8, 172.16/12, and the rest of private space. A rig-side `incus admin init` would, by default, do the naive thing and silently destroy the clone-forever property this box exists for. **The dev role should install `box` and delegate host setup to `box`'s own script.** rig owns the OS plumbing; `box` owns what a box host is. That split is also just this repo's stated philosophy applied one level up. ## 3. `setup-host.sh` cannot be run as root — it will report success having done half the job This is the part that needs design, not just a call: ```sh if ! id -nG "$USER" | grep -qw incus-admin; then sudo usermod -aG incus-admin "$USER" echo "NOTE: added $USER to incus-admin — re-login (or 'sg incus-admin') and re-run." exit 0 fi ``` It is written for a **sudo-capable non-root user**: it `sudo`s, and on first run it adds the group and **exits 0** asking for a re-login. Invoked as root from `rig bootstrap dev`, that returns **success** with no storage pool, no network, no ACL — and rig, believing it, would print `done — role dev`. That is the sshd first-wins bug's exact shape: **rig trusting an exit code instead of checking effective state.** Whatever the design, `bootstrap dev` must assert `incus info` / the pool / the network afterwards and refuse to claim success on the strength of `exit 0`. Same root-vs-user problem in `box`'s installer: it targets `$HOME/.local/share/box` and `$HOME/.local/bin` (overridable via `BOX_HOME`/`BOX_BIN`). Run as root, `box` lands in `/root/.local/bin` — installed for nobody who will use it. **Both point the same way: the dev role needs the admin user from #17, and needs `box` installed and set up *as that user*.** #17 is a prerequisite for this one, not a neighbour. ## 4. `tag:local` — the flag this issue complains about is being deleted The hard constraint above says *"the role should make the correct tag the only reachable outcome, not a flag the operator remembers to pass"* — and notes both M900s carried `tag:server` and had to be retagged. #16 delivers exactly that: `--ts-tag` goes away, the pre-auth key's tags apply on their own, and rig verifies `.Self.Tags` from `tailscale status --json` on **every** run, first join or not. The dev role then needs no tag flag at all — just a policy entry (`dev` must carry `tag:local`, never `tag:server`) and a key minted with `tag:local`. The re-run verification is also what would have caught the M900s without anyone noticing by hand. Worth stating plainly in this issue's spec, since "tailscale join as **`tag:local`**" above currently implies rig requests it. ## 5. Version pinning: there is nothing to pin to `box` has **no tags and no releases**, and its installer builds `https://github.com/$REPO/archive/refs/heads/$REF.tar.gz` — `refs/heads` is **branches only**, so `BOX_REF=v0.5.0` would 404 even if the tag existed. `bootstrap dev` can therefore only install a **moving `main`**. By rig's own logic that is the wrong default: `coolify install` demands a pin *because Coolify never self-updates*, so its version is a contract. **`box` doesn't self-update either** — it has the same shape as Coolify, and the opposite of `actions/runner` (which is latest-by-default only because GitHub refuses jobs from stale runners). There's no `--ephemeral`-style forcing function here. Not a blocker for this issue, but it needs a decision: either `box` cuts tags and teaches its installer `refs/tags`, or `bootstrap dev` installs `main` and the README says out loud that this one is unpinned and why. Silently tracking `main` on the box that runs the agents is the option not to pick. ## Still correct, unchanged The three hard constraints above all stand — **`tag:local` never `tag:server`**, **the boxes must not join the tailnet** (the host joins; guests reach the operator via `ProxyJump dev-server`), and **no credentials on the host**. The last one lines up with `box`'s own design (creds-free; the operator runs `/login` interactively), so rig has nothing to do there but not break it. The rehearsal note stands too, and gets easier: `incus info` and `tailscale status --json` showing `tag:local` are exactly the effective-state assertions #16 is building the reader for.