box/docs/plans/2026-07-18-restricted-tier.md

Restricted incus tier — design and measured results (#74)

Status: implemented and rehearsed. 41/41 rehearsal criteria green on the design host (Debian 13 trixie, Incus 6.0.4, nested KVM), in container mode and VM mode. This doc records the design, what was measured, and why each decision fell where it did. It supersedes the vetoed #72 design (docs/plans/2026-07-17-multiuser-hosts.md on feat/restricted-tier-wip).

What #74 asked

A restricted (incus-group) user can box new/list/shell/snapshot/rm their own boxes, on a network carrying box's full isolation contract, seeing no one else's; the convergence is a documented, idempotent path; the rehearsal passes criteria (a)–(f); the admin tier is unchanged.

The three facts that shaped the design

Task-0 (#72) measured one: incus-user confines users to user-<uid> projects (sound), but pins them to a private auto-created bridge incusbr-<uid> and restricted.networks.access: incusbr-<uid> — they cannot even see boxnet.

This round measured two more:

1. The private bridge is worse than unhardened. incusbr-<uid> is a fully functional NAT bridge — ipv4.nat=true, ipv6.nat=true — with no ACL, no dns.mode=none, no resolver pin, no port isolation, and IPv6 egress box's contract explicitly forbids. Any instance placed on it holds a door to the host's LAN.
2. incus-user projects block snapshots (Project "user-<uid>" doesn't allow for snapshot creation) — box's entire reuse workflow.

And two open questions from #74, answered from incus-user's own source (cmd/incus-user/server.go, stable-6.0) and then confirmed live:

• Daemon-level project template? None exists — the project config is hardcoded in serverSetupUser(). A per-user admin hook is the only path.
• Does widening survive a re-sync? Yes. Setup runs only when the project does not exist (and early-outs when the user's certificate is already trusted); incus-user never rewrites an existing project. Confirmed live: systemctl restart incus-user.socket leaves the convergence intact.

The decision: option 1, tightened

#74 offered (1) converge users onto the shared boxnet or (2) harden each private bridge. Option 2 multiplies every mechanism per user (ACL, resolver pin, dnsmasq, nft rules, firewall coexistence) and turns the shipped static profile into N generated ones. Option 1 keeps one hardened network and one shipped profile — and the existing box↔box mechanisms already make cross-user isolation free: the nft bridge-family drop and dns.mode=none are host/network-owned, so they bind every instance on boxnet no matter whose project it lives in.

One tightening beyond the issue's sketch: the issue proposed restricted.networks.access boxnet,incusbr-<uid> ("must list both" — true as long as the default profile still references the private bridge). Listing both leaves fact 1's unhardened bridge one --network flag away, forever. Instead, box grant:

• removes eth0 from the project's default profile (nothing references the private bridge anymore, so the narrowing validates), and
• sets restricted.networks.access boxnet — only.

After which the hardened network is not the user's default placement but the only placement their certificate can express. Measured: incus launch --network incusbr-<uid> as the user → Network not found; the user cannot widen their own project (Error: Certificate is restricted); they cannot touch boxnet's config or the ACL (no permission for project "default").

A restricted user CAN edit the box-net profile copy in their own project (they own project profiles — features.profiles=true), including stripping security.port_isolation. That is why the host-owned nft bridge drop is the second layer: meta ibrname boxnet obrname boxnet drop fires on every port-to-port frame regardless of per-NIC flags. Cross-user sibling probes are dropped either way — measured from inside the boxes.

What box grant <user> converges (idempotent, re-run to refresh)

1. usermod -aG incus (not incus-admin — that is the tier)
2. first-touch incus-user as the user (runuser/sudo -u, stdin pinned) — the project is created lazily and cannot be pre-created by an admin
3. remove the default profile's private-bridge eth0
4. restricted.networks.access boxnet
5. restricted.snapshots allow
6. install/refresh the shipped box-net profile into the project
7. verify from the user's side of the socket

box revoke <user> is the inverse, two strengths: bare = group removal (the socket closes; boxes keep running; re-grant restores), --purge = boxes, images, project, private bridge, trust-store certificate, incus-user state — then asserts the absence (the wipe.sh discipline).

The tier in the CLI

box_tier() — UID 0 / incus-admin → admin, incus alone → restricted, neither → none — decided from live process credentials (argless id -nG), byte-identical in bin/box and host/setup-host.sh (diffed by a test). Tier-aware surface: new pre-flights the profile and names the right fix per tier; expose refuses before any daemon call (its plumbing is daemon-global; without the guard the failure is a lie — a restricted user cannot read boxnet's redacted config, so box_net_ip would claim their running box has no address); setup-host exits 0 with the honest note; doctor runs a restricted check-set (is the tier granted, does their box resolve/route) instead of judging host state they cannot see.

Found along the way, fixed for every tier: box restore dispatched incus restore, which does not exist in Incus 6 (incus snapshot restore). It had never worked.

Rehearsal and CI

drill/multiuser.sh (root, opt-in via BOX_MULTIUSER_REHEARSAL=1) proves criteria (a)–(f) from #74 plus the measured extensions (g)–(l): the in-box isolation contract (egress, DNS, box→host, RFC1918, cross-user sibling drop, name enumeration, IPv6-off), the closed escape hatches, re-sync survival, and scoped revoke. Two real users, real grants, real mints, probes from inside; --container for CI, VM mode on real hardware; cleanup deletes everything it made.

CI gains a rehearsal job (ubuntu-latest): install incus, stage the checkout at /opt/box (the #71 layout), setup-host, doctor, then the rehearsal in container mode. The tier's semantics are proven on every PR against a live daemon; the VM trust boundary stays a real-hardware ritual, like the drill.

Environment

Debian 13 (trixie), Incus 6.0.4, incus-user.socket shipped in the incus package (Debian 13 and Ubuntu 24.04 both), /dev/kvm present (VM-mode run), btrfs pool. Companion work: rig#24 (box role), rig#12/#25 (host-class).