From 2097982ac237256315804ca0838e9d65ad5a571e Mon Sep 17 00:00:00 2001 From: dan-claude-bot Date: Sat, 18 Jul 2026 04:09:48 +0000 Subject: [PATCH] =?UTF-8?q?docs:=20the=20restricted=20tier=20=E2=80=94=20R?= =?UTF-8?q?EADME,=20design=20doc,=20plan=20doc=20with=20measured=20results?= =?UTF-8?q?=20(#74)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The plan doc records what was measured and why each decision fell where it did: the private bridge is worse than unhardened (a live NAT bridge with IPv6 on), incus-user blocks snapshots, no daemon-level template exists (read in incus-user's source), widening survives re-sync (same source, then measured live). box-design gains the access-tiers section — including why narrowing to boxnet-only is the load-bearing decision and why the nft bridge drop is the layer a restricted user cannot strip. RUNS.md logs MU-1..3. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 28 +++++ README.md | 43 +++++++ docs/box-design.md | 53 ++++++++- docs/plans/2026-07-18-restricted-tier.md | 141 +++++++++++++++++++---- drill/README.md | 24 ++++ drill/RUNS.md | 3 + 6 files changed, 271 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ed2eded..ea191a7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,28 @@ which records not just what changed but what each drill run proved. ### Added +- **The restricted tier: multi-user hosts** (#74, redesigning #72) — an admin + runs `box grant ` and that user gets their own boxes on the same + hardened `boxnet`, seeing nobody else's; `box revoke ` takes it back + (`--purge` deletes their world, and asserts the absence). The tier rides + incus-user, whose defaults miss box's contract three measured ways (Debian + 13 / Incus 6.0.4): a private *unhardened* NAT bridge per user, snapshots + blocked, the `box-net` profile invisible — so grant is an idempotent + convergence: project narrowed to `boxnet` **and only boxnet** (listing the + private bridge too, the obvious fix, would keep an unhardened network one + `--network` flag away), snapshots allowed, the shipped profile installed + into their project. `box_tier()` (live credentials, argless `id -nG`) + drives the tier-aware surface: `expose` refuses honestly before any daemon + call, `setup-host` and `doctor` answer at the caller's tier. Rehearsed + end-to-end by `drill/multiuser.sh` (criteria a–l: confinement, lifecycle, + cross-user visibility, name collisions, the in-box isolation contract, + escape hatches, re-sync survival, revoke) — 41/41 on the design host, in + both container and VM mode. +- **CI runs the multi-user rehearsal on a real Incus** — a second `rehearsal` + job stands up the full stack on the runner (setup-host, doctor, then + `multiuser.sh --container`), so every PR proves the tier's semantics + against a live daemon, not a mock. The VM trust boundary itself remains a + real-hardware ritual, like the full drill. - **Global / root install** (#71) — run as root, box installs *once* to `/opt/box` (world-readable) with the `box` symlink on `/usr/local/bin`, so every operator on a shared host runs the same tree. Per-user installs are @@ -24,6 +46,12 @@ which records not just what changed but what each drill run proved. ### Fixed +- **`box restore` never worked against Incus 6** — the command table + dispatched `incus restore`, a subcommand that does not exist (Incus 6 + spells it `incus snapshot restore`), so every restore died on "unknown + command". Found by #74's rehearsal exercising the full lifecycle as a + restricted user; fixed for every tier, and the rehearsal + a grep-guard in + `test/cli.sh` now hold it. - **`box tmux` works on every template** (#65) — `box tmux` runs `tmux new-session` *inside* the box, but the templates did not install tmux, so it failed with `tmux: command not found`. `tmux` is now in each template's diff --git a/README.md b/README.md index 8317fe0..50e3d67 100644 --- a/README.md +++ b/README.md @@ -106,6 +106,49 @@ re-homes each legacy box onto `boxnet` (authed state preserved), and `box migrate-host --retire-legacy` removes the old bridge and profile once no legacy box remains. +## Multi-user hosts: the restricted tier + +One host, several people, and not everyone should hold the daemon. Incus's +socket is all-or-nothing — `incus-admin` group members own every instance on +the machine — so box layers a second tier on +[incus-user](https://linuxcontainers.org/incus/docs/main/projects/): + +| tier | who | what they hold | +|---|---|---| +| **admin** | root, or the `incus-admin` group | everything: all boxes, the stack, `setup-host`, `expose`, `grant` | +| **restricted** | the `incus` group | their **own** boxes only, on the same hardened network | +| none | everyone else | no socket, nothing | + +An admin hands the tier out per user, and takes it back: + +```sh +box grant dev1 # dev1 can now: box new / list / shell / snapshot / rm — their boxes only +box revoke dev1 # locked out; their boxes survive (grant again restores) +box revoke dev1 --purge # ...or delete everything they had +``` + +`grant` is an idempotent convergence, not a flag flip, because incus-user's +defaults miss box's contract three ways (measured on Debian 13 / Incus 6.0.4, +see [the plan doc](docs/plans/2026-07-18-restricted-tier.md)): it pins each +user to a private *unhardened* NAT bridge, it blocks snapshots, and it cannot +see the `box-net` profile. Granting rewires all three: the user's project is +restricted to `boxnet` **and only boxnet** — the hardened network is not their +default placement but the only one their certificate can express — snapshots +are allowed, and the shipped profile is installed into their project. Re-run +`box grant ` after upgrading box to refresh the profile, like +`setup-host` for the stack. + +What a restricted user gets is the full contract: same ACL, same DNS +isolation, same pinned resolver, same port isolation, same box↔box drop — +and their boxes cannot reach another user's box, which is the same +box↔box drop doing its one job. What they can't do stays honest: `box +expose` (daemon-global state) says to ask an admin, `box setup-host` and +`box doctor` answer at their tier instead of failing at it. + +`drill/multiuser.sh` rehearses all of it live — two users, real grants, real +boxes, probes from inside — and CI runs it on every PR (container mode; the +VM boundary itself is proven on real hardware, like the rest of the drill). + ## Quick start ```sh diff --git a/docs/box-design.md b/docs/box-design.md index 834165a..ceda452 100644 --- a/docs/box-design.md +++ b/docs/box-design.md @@ -127,7 +127,58 @@ The rule that keeps this honest: **isolation claims are tested, never reasoned about.** The box→box hole existed because a plausible code reading said it could not. See `drill/RUNS.md`. +## Multi-user hosts: access tiers + +The daemon socket is binary — `incus-admin` holds everything on the machine — +so a shared host needs a second tier, and Incus ships one: **incus-user** +confines an `incus`-group member to an auto-created project `user-`, +behind a restricted certificate that cannot name any other project. The tier +is decided once, from the process's live credentials (`box_tier()`: UID 0 or +`incus-admin` → admin; `incus` alone → restricted; neither → none), and every +tier-aware verb reads that one function. + +What incus-user does *not* do is honor box's contract — measured on Debian 13 +/ Incus 6.0.4 (#74), after the design that assumed it (#72) was vetoed by its +own Task-0 rehearsal: + +- it pins each user's project to a private auto-created bridge + (`incusbr-`) — a stock NAT bridge with **none** of the hardening: no + ACL, no `dns.mode=none`, no resolver pin, IPv6 on; +- it blocks snapshots — box's entire reuse workflow; +- the `box-net` profile lives in the default project, invisible to theirs. + +So the tier is an **admin-run convergence** (`box grant `), not a +group membership: put them in `incus`, touch incus-user once as them (the +project is created lazily; nothing exists to converge until it does), then +rewire the project — network access narrowed to `boxnet` **and only +`boxnet`**, snapshots allowed, the shipped profile installed. Narrowing is +the load-bearing decision: granting `boxnet,incusbr-` (the obvious fix) +would leave an unhardened NAT bridge one `--network` flag away from any box +they mint. With the private bridge unreferenced (its `eth0` is removed from +their default profile) and outside `restricted.networks.access`, the hardened +network is not their default placement — it is the only placement their +certificate can express. The grant survives incus-user restarts by that +tool's own design (it configures a project only at creation), and a restricted +certificate cannot widen its own project — both measured, not read. + +Cross-USER isolation is the same mechanism as cross-box isolation, on +purpose: their instances share `boxnet` with everyone's, and the bridge-family +drop + port isolation + `dns.mode=none` already make any two boxes strangers. +A restricted user CAN strip `security.port_isolation` from the profile copy +in their own project — which is why the nft bridge drop, which they cannot +touch, exists as the second, host-owned layer. Defense in depth, both layers +measured (`drill/multiuser.sh`, criteria a–l). + +`box revoke` is two strengths: bare, it removes the group — the socket +closes, their boxes keep *running* (revoking a person does not kill their +workloads) and `grant` restores everything; `--purge` deletes their world +(boxes, images, project, private bridge, trust-store certificate) and asserts +the absence afterwards. + ## Non-goals -- No unattended/CI bring-up — the flow is interactive. +- Interactive-first: install and setup prompt by default (`BOX_YES=1` and the + CI rehearsal job are the sanctioned unattended paths). - No credential storage or injection by the tool. +- No per-user resource quotas on the restricted tier (Incus's + `limits.*`/`restricted.*` project keys exist when someone needs them). diff --git a/docs/plans/2026-07-18-restricted-tier.md b/docs/plans/2026-07-18-restricted-tier.md index bade330..d6ace55 100644 --- a/docs/plans/2026-07-18-restricted-tier.md +++ b/docs/plans/2026-07-18-restricted-tier.md @@ -1,27 +1,128 @@ -# Restricted incus tier — implementation plan (issue #74) +# Restricted incus tier — design and measured results (#74) -> **Status: placeholder.** This PR is a work in progress; the design below is being -> validated by a live multi-user rehearsal before the implementation lands. Do not -> review yet — the PR stays in draft until the rehearsal passes. +**Status: implemented and rehearsed.** 41/41 rehearsal criteria green on the +design host (Debian 13 trixie, Incus 6.0.4, nested KVM), in container mode +and VM mode. This doc records the design, what was measured, and why each +decision fell where it did. It supersedes the vetoed #72 design +(`docs/plans/2026-07-17-multiuser-hosts.md` on `feat/restricted-tier-wip`). -## Scope +## What #74 asked -Deliver the restricted (`incus`-group) tier described in #74: +A restricted (`incus`-group) user can `box new/list/shell/snapshot/rm` their +own boxes, on a network carrying box's **full** isolation contract, seeing no +one else's; the convergence is a documented, idempotent path; the rehearsal +passes criteria (a)–(f); the admin tier is unchanged. -- A restricted user can `box new/list/shell/snapshot/rm` **their own** boxes. -- Their boxes ride a network carrying box's full isolation contract - (ACL, `dns.mode=none`, resolver pin, `security.port_isolation`, nft box↔box drop). -- No cross-user visibility. Admin tier unchanged. -- The admin-side convergence is a documented, idempotent command — not manual - per-user `incus project set`. +## The three facts that shaped the design -## Planned shape (subject to rehearsal) +Task-0 (#72) measured one: incus-user confines users to `user-` projects +(sound), but pins them to a private auto-created bridge `incusbr-` and +`restricted.networks.access: incusbr-` — they cannot even see `boxnet`. -- `box grant ` / `box revoke ` — admin convergence hook per #74 - option 1: widen `restricted.networks.access` to include `boxnet` and install - the `box-net` profile into the user's `user-` project. -- CLI awareness of running inside a restricted project. -- `drill/multiuser.sh` rehearsal criteria (a)–(f) green on a real multi-user host. -- Test suite expansion + CI wiring. +This round measured two more: -Tracking issue: heavy-duty/box#74. +1. **The private bridge is worse than unhardened.** `incusbr-` is a + fully functional NAT bridge — `ipv4.nat=true`, **`ipv6.nat=true`** — with + no ACL, no `dns.mode=none`, no resolver pin, no port isolation, and IPv6 + egress box's contract explicitly forbids. Any instance placed on it holds + a door to the host's LAN. +2. **incus-user projects block snapshots** (`Project "user-" doesn't + allow for snapshot creation`) — box's entire reuse workflow. + +And two open questions from #74, answered from incus-user's own source +(`cmd/incus-user/server.go`, stable-6.0) and then confirmed live: + +- **Daemon-level project template?** None exists — the project config is + hardcoded in `serverSetupUser()`. A per-user admin hook is the only path. +- **Does widening survive a re-sync?** Yes. Setup runs only when the project + does not exist (and early-outs when the user's certificate is already + trusted); incus-user never rewrites an existing project. Confirmed live: + `systemctl restart incus-user.socket` leaves the convergence intact. + +## The decision: option 1, tightened + +#74 offered (1) converge users onto the shared `boxnet` or (2) harden each +private bridge. Option 2 multiplies every mechanism per user (ACL, resolver +pin, dnsmasq, nft rules, firewall coexistence) and turns the shipped static +profile into N generated ones. Option 1 keeps one hardened network and one +shipped profile — and the existing box↔box mechanisms already make +cross-user isolation free: the nft bridge-family drop and `dns.mode=none` +are host/network-owned, so they bind every instance on `boxnet` no matter +whose project it lives in. + +One tightening beyond the issue's sketch: the issue proposed +`restricted.networks.access boxnet,incusbr-` ("must list both" — true +as long as the default profile still references the private bridge). Listing +both leaves fact 1's unhardened bridge one `--network` flag away, forever. +Instead, `box grant`: + +- removes `eth0` from the project's default profile (nothing references the + private bridge anymore, so the narrowing validates), and +- sets `restricted.networks.access boxnet` — **only**. + +After which the hardened network is not the user's default placement but the +only placement their certificate can express. Measured: `incus launch +--network incusbr-` as the user → `Network not found`; the user cannot +widen their own project (`Error: Certificate is restricted`); they cannot +touch `boxnet`'s config or the ACL (`no permission for project "default"`). + +A restricted user CAN edit the `box-net` profile copy in their own project +(they own project profiles — `features.profiles=true`), including stripping +`security.port_isolation`. That is why the host-owned nft bridge drop is the +second layer: `meta ibrname boxnet obrname boxnet drop` fires on every +port-to-port frame regardless of per-NIC flags. Cross-user sibling probes are +dropped either way — measured from inside the boxes. + +## What `box grant ` converges (idempotent, re-run to refresh) + +1. `usermod -aG incus` (not `incus-admin` — that is the tier) +2. first-touch incus-user as the user (`runuser`/`sudo -u`, stdin pinned) — + the project is created lazily and cannot be pre-created by an admin +3. remove the default profile's private-bridge `eth0` +4. `restricted.networks.access boxnet` +5. `restricted.snapshots allow` +6. install/refresh the shipped `box-net` profile into the project +7. verify from the user's side of the socket + +`box revoke ` is the inverse, two strengths: bare = group removal (the +socket closes; boxes keep running; re-grant restores), `--purge` = boxes, +images, project, private bridge, trust-store certificate, incus-user state — +then asserts the absence (the wipe.sh discipline). + +## The tier in the CLI + +`box_tier()` — UID 0 / `incus-admin` → admin, `incus` alone → restricted, +neither → none — decided from live process credentials (argless `id -nG`), +byte-identical in `bin/box` and `host/setup-host.sh` (diffed by a test). +Tier-aware surface: `new` pre-flights the profile and names the right fix per +tier; `expose` refuses before any daemon call (its plumbing is daemon-global; +without the guard the failure is a lie — a restricted user cannot read +boxnet's redacted config, so box_net_ip would claim their running box has no +address); `setup-host` exits 0 with the honest note; `doctor` runs a +restricted check-set (is the tier granted, does their box resolve/route) +instead of judging host state they cannot see. + +Found along the way, fixed for every tier: `box restore` dispatched +`incus restore`, which does not exist in Incus 6 (`incus snapshot restore`). +It had never worked. + +## Rehearsal and CI + +`drill/multiuser.sh` (root, opt-in via `BOX_MULTIUSER_REHEARSAL=1`) proves +criteria (a)–(f) from #74 plus the measured extensions (g)–(l): the in-box +isolation contract (egress, DNS, box→host, RFC1918, cross-user sibling drop, +name enumeration, IPv6-off), the closed escape hatches, re-sync survival, and +scoped revoke. Two real users, real grants, real mints, probes from inside; +`--container` for CI, VM mode on real hardware; cleanup deletes everything it +made. + +CI gains a `rehearsal` job (ubuntu-latest): install incus, stage the checkout +at `/opt/box` (the #71 layout), `setup-host`, `doctor`, then the rehearsal in +container mode. The tier's semantics are proven on every PR against a live +daemon; the VM trust boundary stays a real-hardware ritual, like the drill. + +## Environment + +Debian 13 (trixie), Incus 6.0.4, `incus-user.socket` shipped in the incus +package (Debian 13 and Ubuntu 24.04 both), `/dev/kvm` present (VM-mode run), +btrfs pool. Companion work: rig#24 (`box` role), rig#12/#25 (host-class). diff --git a/drill/README.md b/drill/README.md index d7a06cf..b5443ef 100644 --- a/drill/README.md +++ b/drill/README.md @@ -91,3 +91,27 @@ installed and runnable; authenticating is yours. If the host has no `/dev/kvm`, box falls back to container mode. The drill still runs, but it says loudly that **the VM trust boundary was not validated** rather than passing quietly on a weaker one. + +## The multi-user rehearsal (`multiuser.sh`) + +The restricted tier (#74) has its own rehearsal — the drill proves one +operator's host; this proves a *shared* one: + +```sh +sudo BOX_MULTIUSER_REHEARSAL=1 bash drill/multiuser.sh --yes +``` + +Root only, opt-in twice (it creates system users and edits the group +database). It creates two throwaway users, grants them the tier through the +real `box grant`, mints real boxes as them, and measures — from inside those +boxes — that each user is confined to their own project (a), the full +lifecycle works (b), no cross-user visibility (c), names don't collide (d), +`expose`/`setup-host`/`doctor` answer honestly at the tier (e/f), the boxes +ride `boxnet` under the full isolation contract including the cross-user +sibling drop (g), the private-bridge escape hatches are closed (h), the grant +survives an incus-user restart (k), and `revoke --purge` erases one user +without touching the other (l). Everything it makes, it deletes. + +`--container` skips VM mints (CI runs it this way on every PR — the tier's +semantics are instance-type-independent); on real hardware run it bare so the +boxes are VMs. `--keep` leaves the users and boxes up for inspection. diff --git a/drill/RUNS.md b/drill/RUNS.md index 72d94a0..c13e016 100644 --- a/drill/RUNS.md +++ b/drill/RUNS.md @@ -195,6 +195,9 @@ No listener is needed, and none should be started: see trap 3. | Run | Result | What it cost | | --- | --- | --- | +| **MU-3** | **41/0 (containers) + 41/0 (VMs) — the tier holds** | `multiuser.sh` (#74) from zero on Debian 13 / Incus 6.0.4: grant, confinement, lifecycle, the in-box contract with the cross-user sibling drop, closed escape hatches, re-sync survival, scoped revoke. The VM pass needed right-sized boxes (1GiB): two 2GiB nested VMs on a 7.7GiB host measured swap, not isolation | +| MU-2 | 40/41 (VMs) | the egress probe cried broken over a TLS handshake that outlived its 8s budget under nested-KVM memory pressure — every isolation probe (which PASSES on silence) was immune. Probe budgets raised, reachable-probes retry once, and the mint keeps its log for the post-mortem | +| MU-1 | 39/41 (containers) | both FAILs were the instrument: a `box new` rc swallowed with its narration (the drill's oldest sin, relearned), and the enumeration probe resolving the box's OWN name — /etc/hosts answered, not the gateway. Probe a SIBLING's name or you measure cloud-init's hostname hygiene. Found for real along the way: `box restore` dispatched `incus restore`, which does not exist in Incus 6 — broken for every tier since the rename | | 1 | hung at C7; ~9 false FAILs | traps 1, 2, 4 — pipefail, the exec-pty hang, the DHCP race | | 2 | 42/49 | trap 5 — an interrupted run 1 left boxes behind, cascading 5 FAILs. Found the `claude`-on-PATH bug. Phase D delivered B1 (`@internal` rejected) and a B3 reading of *broken* | | 3 | 48/49 | trap 4 — `eth0_ip` never matched, so A3 again unprobed. B3 now read *intact*, contradicting run 2 |