8598cf96fe
The previous revision skipped `usermod -aG incus` for an incus-admin member, reasoning that 'incus' is a strict subset of what incus-admin already opens. That is true of the daemon API and false of the filesystem. On Debian 13 / Incus 6.0.4 the two sockets are two files with two owning groups: /var/lib/incus/unix.socket group incus-admin 0660 /var/lib/incus/unix.socket.user group incus 0660 incus-admin opens the first and not the second, and only the second provisions a user-<uid> project. So for the incus-admin-ONLY user — the canonical #99 case — the pinned provisioning touch took EACCES, the `|| true` swallowed it, no project appeared, and the grant died blaming a healthy incus-user. Both reviewers converged on this independently and were right. The membership is now granted for everyone, with output carrying the concern the old no-op was built around (it is the key to a file, not a privilege; box_tier still reads them as admin). Everything downstream moves with it: - the backout rolls that membership back and verified, while refusing to call the rollback a lockout — incus-admin is untouched and still opens the host - revoke's bare path takes the membership back and reports `partial:` instead of "no-op, nothing was taken", still declining to call them "out" - grant's closing "gpasswd -d <user> incus-admin (no re-grant needed)" is now a true promise: they keep 'incus', so the drop lands them in their project - the socket existence probe goes through $SUDO, matching revoke's measured discipline about /var/lib/incus lying to a non-root admin Tests: the cli.sh assertions that encoded the old no-op design are flipped and the decision is pinned at the seam that broke; the sudo shim now runs `test` for real in both directions. Because the shims model neither INCUS_SOCKET nor permissions and so cannot reproduce the EACCES, drill/multiuser.sh gains criterion (o): an incus-admin-only member granted on real Incus in CI, with the membership, the project, a live connect() to unix.socket.user, and the post-drop landing all measured. Mutation-checked: 11 of the new/flipped assertions fail against the previous implementation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
8.1 KiB
Restricted incus tier — design and measured results (#74)
Status: implemented and rehearsed. 54/54 rehearsal criteria green on the
design host (Debian 13 trixie, Incus 6.0.4, nested KVM), in container mode
and VM mode — and green in CI on ubuntu-latest / incus 6.0.0 (whose one
version-drift false FAIL is MU-4 in drill/RUNS.md). This doc records the design, what was measured, and why each
decision fell where it did. It supersedes the vetoed #72 design
(docs/plans/2026-07-17-multiuser-hosts.md on feat/restricted-tier-wip).
What #74 asked
A restricted (incus-group) user can box new/list/shell/snapshot/rm their
own boxes, on a network carrying box's full isolation contract, seeing no
one else's; the convergence is a documented, idempotent path; the rehearsal
passes criteria (a)–(f); the admin tier is unchanged.
The three facts that shaped the design
Task-0 (#72) measured one: incus-user confines users to user-<uid> projects
(sound), but pins them to a private auto-created bridge incusbr-<uid> and
restricted.networks.access: incusbr-<uid> — they cannot even see boxnet.
This round measured two more:
- The private bridge is worse than unhardened.
incusbr-<uid>is a fully functional NAT bridge —ipv4.nat=true,ipv6.nat=true— with no ACL, nodns.mode=none, no resolver pin, no port isolation, and IPv6 egress box's contract explicitly forbids. Any instance placed on it holds a door to the host's LAN. - incus-user projects block snapshots (
Project "user-<uid>" doesn't allow for snapshot creation) — box's entire reuse workflow.
And two open questions from #74, answered from incus-user's own source
(cmd/incus-user/server.go, stable-6.0) and then confirmed live:
- Daemon-level project template? None exists — the project config is
hardcoded in
serverSetupUser(). A per-user admin hook is the only path. - Does widening survive a re-sync? Yes. Setup runs only when the project
does not exist (and early-outs when the user's certificate is already
trusted); incus-user never rewrites an existing project. Confirmed live:
systemctl restart incus-user.socketleaves the convergence intact.
The decision: option 1, tightened
#74 offered (1) converge users onto the shared boxnet or (2) harden each
private bridge. Option 2 multiplies every mechanism per user (ACL, resolver
pin, dnsmasq, nft rules, firewall coexistence) and turns the shipped static
profile into N generated ones. Option 1 keeps one hardened network and one
shipped profile — and the existing box↔box mechanisms already make
cross-user isolation free: the nft bridge-family drop and dns.mode=none
are host/network-owned, so they bind every instance on boxnet no matter
whose project it lives in.
One tightening beyond the issue's sketch: the issue proposed
restricted.networks.access boxnet,incusbr-<uid> ("must list both" — true
as long as the default profile still references the private bridge). Listing
both leaves fact 1's unhardened bridge one --network flag away, forever.
Instead, box grant:
- removes
eth0from the project's default profile (nothing references the private bridge anymore, so the narrowing validates), and - sets
restricted.networks.access boxnet— only.
After which the hardened network is not the user's default placement but the
only placement their certificate can express. Measured: incus launch --network incusbr-<uid> as the user → Network not found; the user cannot
widen their own project (Error: Certificate is restricted); they cannot
touch boxnet's config or the ACL (no permission for project "default").
A restricted user CAN edit the box-net profile copy in their own project
(they own project profiles — features.profiles=true), including stripping
security.port_isolation — and CAN attach boxnet raw with --network boxnet, no profile at all (the network must be in
restricted.networks.access for the profile to work; there is no
allow-via-profile-only lever). That is why the host-owned nft bridge drop is
the second layer: meta ibrname boxnet obrname boxnet drop fires on every
port-to-port frame regardless of per-NIC flags. The documented guarantee is
scoped accordingly (see box-design.md): box-minted instances carry per-NIC
port isolation; raw attachments keep every network- and host-owned control,
losing only that redundant L2 layer. Both shapes are measured from inside
the instances (rehearsal criteria g and m).
Two grant-failure contracts, both injected in the rehearsal (criterion n): a fresh user is backed out of the group with the removal VERIFIED against the live group database (and any session begun mid-grant is named, with the loginctl remedy — the one window the database cannot close); a pre-existing member is never stripped by a failed re-grant, but the failure states out loud that they retain socket access on part-converged policy, with both remediations. The default-profile eth0 removal is deliberately NOT restored on failure: that mutation only reduces capability, and restoring it would move the failure state away from fail-closed. Every step is check-then- converge, which is what makes re-run-to-repair deterministic.
What box grant <user> converges (idempotent, re-run to refresh)
usermod -aG incus(notincus-admin— that is the tier)- first-touch incus-user as the user (
runuser/sudo -u, stdin pinned) — the project is created lazily and cannot be pre-created by an admin - remove the default profile's private-bridge
eth0 restricted.networks.access boxnetrestricted.snapshots allow- install/refresh the shipped
box-netprofile into the project - verify from the user's side of the socket
box revoke <user> is the inverse, two strengths: bare = group removal (the
socket closes; boxes keep running; re-grant restores), --purge = boxes,
images, project, private bridge, trust-store certificate, incus-user state —
then asserts the absence (the wipe.sh discipline).
The tier in the CLI
box_tier() — UID 0 / incus-admin → admin, incus alone → restricted,
neither → none — decided from live process credentials (argless id -nG),
byte-identical in bin/box and host/setup-host.sh (diffed by a test).
Tier-aware surface: new pre-flights the profile and names the right fix per
tier; expose refuses before any daemon call (its plumbing is daemon-global;
without the guard the failure is a lie — a restricted user cannot read
boxnet's redacted config, so box_net_ip would claim their running box has no
address); setup-host exits 0 with the honest note; doctor runs a
restricted check-set (is the tier granted, does their box resolve/route)
instead of judging host state they cannot see.
Found along the way, fixed for every tier: box restore dispatched
incus restore, which does not exist in Incus 6 (incus snapshot restore).
It had never worked.
Rehearsal and CI
drill/multiuser.sh (root, opt-in via BOX_MULTIUSER_REHEARSAL=1) proves
criteria (a)–(f) from #74 plus the measured extensions (g)–(o): the in-box
isolation contract (egress, DNS, box→host, RFC1918, cross-user sibling drop,
name enumeration, IPv6-off), the closed escape hatches, re-sync survival, and
scoped revoke, the raw-attach scoped guarantee (m), the grant-failure
injections (n) and the incus-admin-only grant (o) — the last of these
because the shim suite models neither INCUS_SOCKET nor socket permissions,
so the group gate on unix.socket.user can only be measured live (#99, #101). Real users, real grants, real mints, probes from inside;
--container for CI, VM mode on real hardware; cleanup deletes everything it
made.
CI gains a rehearsal job (ubuntu-latest): install incus, stage the checkout
at /opt/box (the #71 layout), setup-host, doctor, then the rehearsal in
container mode. The tier's semantics are proven on every PR against a live
daemon; the VM trust boundary stays a real-hardware ritual, like the drill.
Environment
Debian 13 (trixie), Incus 6.0.4, incus-user.socket shipped in the incus
package (Debian 13 and Ubuntu 24.04 both), /dev/kvm present (VM-mode run),
btrfs pool. Companion work: rig#24 (box role), rig#12/#25 (host-class).