box/docs/plans/2026-07-18-restricted-tier.md

# Restricted incus tier — design and measured results (#74)

**Status: implemented and rehearsed.** 54/54 rehearsal criteria green on the
design host (Debian 13 trixie, Incus 6.0.4, nested KVM), in container mode
and VM mode — and green in CI on ubuntu-latest / incus 6.0.0 (whose one
version-drift false FAIL is MU-4 in `drill/RUNS.md`). This doc records the design, what was measured, and why each
decision fell where it did. It supersedes the vetoed #72 design
(`docs/plans/2026-07-17-multiuser-hosts.md` on `feat/restricted-tier-wip`).

## What #74 asked

A restricted (`incus`-group) user can `box new/list/shell/snapshot/rm` their
own boxes, on a network carrying box's **full** isolation contract, seeing no
one else's; the convergence is a documented, idempotent path; the rehearsal
passes criteria (a)–(f); the admin tier is unchanged.

## The three facts that shaped the design

Task-0 (#72) measured one: incus-user confines users to `user-<uid>` projects
(sound), but pins them to a private auto-created bridge `incusbr-<uid>` and
`restricted.networks.access: incusbr-<uid>` — they cannot even see `boxnet`.

This round measured two more:

1. **The private bridge is worse than unhardened.** `incusbr-<uid>` is a
   fully functional NAT bridge — `ipv4.nat=true`, **`ipv6.nat=true`** — with
   no ACL, no `dns.mode=none`, no resolver pin, no port isolation, and IPv6
   egress box's contract explicitly forbids. Any instance placed on it holds
   a door to the host's LAN.
2. **incus-user projects block snapshots** (`Project "user-<uid>" doesn't
   allow for snapshot creation`) — box's entire reuse workflow.

And two open questions from #74, answered from incus-user's own source
(`cmd/incus-user/server.go`, stable-6.0) and then confirmed live:

- **Daemon-level project template?** None exists — the project config is
  hardcoded in `serverSetupUser()`. A per-user admin hook is the only path.
- **Does widening survive a re-sync?** Yes. Setup runs only when the project
  does not exist (and early-outs when the user's certificate is already
  trusted); incus-user never rewrites an existing project. Confirmed live:
  `systemctl restart incus-user.socket` leaves the convergence intact.

## The decision: option 1, tightened

#74 offered (1) converge users onto the shared `boxnet` or (2) harden each
private bridge. Option 2 multiplies every mechanism per user (ACL, resolver
pin, dnsmasq, nft rules, firewall coexistence) and turns the shipped static
profile into N generated ones. Option 1 keeps one hardened network and one
shipped profile — and the existing box↔box mechanisms already make
cross-user isolation free: the nft bridge-family drop and `dns.mode=none`
are host/network-owned, so they bind every instance on `boxnet` no matter
whose project it lives in.

One tightening beyond the issue's sketch: the issue proposed
`restricted.networks.access boxnet,incusbr-<uid>` ("must list both" — true
as long as the default profile still references the private bridge). Listing
both leaves fact 1's unhardened bridge one `--network` flag away, forever.
Instead, `box grant`:

- removes `eth0` from the project's default profile (nothing references the
  private bridge anymore, so the narrowing validates), and
- sets `restricted.networks.access boxnet` — **only**.

After which the hardened network is not the user's default placement but the
only placement their certificate can express. Measured: `incus launch
--network incusbr-<uid>` as the user → `Network not found`; the user cannot
widen their own project (`Error: Certificate is restricted`); they cannot
touch `boxnet`'s config or the ACL (`no permission for project "default"`).

A restricted user CAN edit the `box-net` profile copy in their own project
(they own project profiles — `features.profiles=true`), including stripping
`security.port_isolation` — and CAN attach `boxnet` raw with `--network
boxnet`, no profile at all (the network must be in
`restricted.networks.access` for the profile to work; there is no
allow-via-profile-only lever). That is why the host-owned nft bridge drop is
the second layer: `meta ibrname boxnet obrname boxnet drop` fires on every
port-to-port frame regardless of per-NIC flags. The documented guarantee is
scoped accordingly (see box-design.md): box-minted instances carry per-NIC
port isolation; raw attachments keep every network- and host-owned control,
losing only that redundant L2 layer. Both shapes are measured from inside
the instances (rehearsal criteria g and m).

Two grant-failure contracts, both injected in the rehearsal (criterion n):
a fresh user is backed out of the group with the removal VERIFIED against
the live group database (and any session begun mid-grant is named, with the
loginctl remedy — the one window the database cannot close); a pre-existing
member is never stripped by a failed re-grant, but the failure states out
loud that they retain socket access on part-converged policy, with both
remediations. The default-profile eth0 removal is deliberately NOT restored
on failure: that mutation only reduces capability, and restoring it would
move the failure state away from fail-closed. Every step is check-then-
converge, which is what makes re-run-to-repair deterministic.

## What `box grant <user>` converges (idempotent, re-run to refresh)

1. `usermod -aG incus` (not `incus-admin` — that is the tier)
2. first-touch incus-user as the user (`runuser`/`sudo -u`, stdin pinned) —
   the project is created lazily and cannot be pre-created by an admin
3. remove the default profile's private-bridge `eth0`
4. `restricted.networks.access boxnet`
5. `restricted.snapshots allow`
6. install/refresh the shipped `box-net` profile into the project
7. verify from the user's side of the socket

`box revoke <user>` is the inverse, two strengths: bare = group removal (the
socket closes; boxes keep running; re-grant restores), `--purge` = boxes,
images, project, private bridge, trust-store certificate, incus-user state —
then asserts the absence (the wipe.sh discipline).

## The tier in the CLI

`box_tier()` — UID 0 / `incus-admin` → admin, `incus` alone → restricted,
neither → none — decided from live process credentials (argless `id -nG`),
byte-identical in `bin/box` and `host/setup-host.sh` (diffed by a test).
Tier-aware surface: `new` pre-flights the profile and names the right fix per
tier; `expose` refuses before any daemon call (its plumbing is daemon-global;
without the guard the failure is a lie — a restricted user cannot read
boxnet's redacted config, so box_net_ip would claim their running box has no
address); `setup-host` exits 0 with the honest note; `doctor` runs a
restricted check-set (is the tier granted, does their box resolve/route)
instead of judging host state they cannot see.

Found along the way, fixed for every tier: `box restore` dispatched
`incus restore`, which does not exist in Incus 6 (`incus snapshot restore`).
It had never worked.

## Rehearsal and CI

`drill/multiuser.sh` (root, opt-in via `BOX_MULTIUSER_REHEARSAL=1`) proves
criteria (a)–(f) from #74 plus the measured extensions (g)–(n): the in-box
isolation contract (egress, DNS, box→host, RFC1918, cross-user sibling drop,
name enumeration, IPv6-off), the closed escape hatches, re-sync survival, and
scoped revoke, the raw-attach scoped guarantee (m) and the grant-failure
injections (n). Real users, real grants, real mints, probes from inside;
`--container` for CI, VM mode on real hardware; cleanup deletes everything it
made.

CI gains a `rehearsal` job (ubuntu-latest): install incus, stage the checkout
at `/opt/box` (the #71 layout), `setup-host`, `doctor`, then the rehearsal in
container mode. The tier's semantics are proven on every PR against a live
daemon; the VM trust boundary stays a real-hardware ritual, like the drill.

## Environment

Debian 13 (trixie), Incus 6.0.4, `incus-user.socket` shipped in the incus
package (Debian 13 and Ubuntu 24.04 both), `/dev/kvm` present (VM-mode run),
btrfs pool. Companion work: rig#24 (`box` role), rig#12/#25 (host-class).