2026-07-18 04:09:48 +00:00
|
|
|
|
# Restricted incus tier — design and measured results (#74)
|
2026-07-18 01:06:34 +00:00
|
|
|
|
|
grant/rehearsal: the codex round — verified rollback, loud partial states, and the raw-attach guarantee measured (#75)
Review 4727756972 (A2): the backout no longer trusts gpasswd — it re-reads
the live group database after removal; verified-absent gets the safe
message, anything else screams ROLLBACK INCOMPLETE, exits nonzero, and
names the exact remediation. The concurrent-login window (a session begun
between usermod and backout keeps the group) is CLOSED to the extent the
database can't reach: the backout detects live processes and names
loginctl terminate-user, and the success wording claims only what was
verified.
Review 4727641752 (A1): a failed grant for a user whose membership predates
the run (the hand-added-user scenario) now fails LOUDLY — they retain
socket access on part-converged policy, and the message says so with both
remediations (box revoke now, or fix and re-run). Their membership is not
stripped: breaking a working user over a failed re-grant is its own hazard.
The default-profile eth0 removal is deliberately not restored on failure —
that mutation only reduces capability, and restoring it would move the
failure state AWAY from fail-closed. Injected-failure coverage is criterion
(n), both flavors: fresh-user backout (fault at the LAST mutation, so the
rollback runs after every earlier one) with the group's absence verified
and a converging re-run; blocked narrowing staged for real with an
instance-local NIC parked on the private bridge.
Review A3, resolution 3 with the measurement demanded: criterion (m)
launches exactly 'incus launch --network boxnet' as the restricted user and
probes the raw NIC from inside — egress works, RFC1918 dropped (the ACL is
the network's), sibling probes dropped BOTH directions (the nft drop is the
host's), name enumeration blocked. The scoped guarantee is now stated in
box-design.md and measured on every run: box-minted instances carry per-NIC
port_isolation; raw attachments keep every network- and host-owned control,
losing only that redundant L2 layer. Instrument lesson kept as MU-5: the
probe's first cut minted the non-cloud image — no DHCP client, no lease,
and a dead NIC passes every negative probe vacuously; it now requires the
lease before believing its own answers.
Rehearsal: 54/54 (containers). test/cli.sh: 82 checks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 06:41:05 +00:00
|
|
|
|
**Status: implemented and rehearsed.** 54/54 rehearsal criteria green on the
|
2026-07-18 04:09:48 +00:00
|
|
|
|
design host (Debian 13 trixie, Incus 6.0.4, nested KVM), in container mode
|
2026-07-18 05:38:04 +00:00
|
|
|
|
and VM mode — and green in CI on ubuntu-latest / incus 6.0.0 (whose one
|
|
|
|
|
|
version-drift false FAIL is MU-4 in `drill/RUNS.md`). This doc records the design, what was measured, and why each
|
2026-07-18 04:09:48 +00:00
|
|
|
|
decision fell where it did. It supersedes the vetoed #72 design
|
|
|
|
|
|
(`docs/plans/2026-07-17-multiuser-hosts.md` on `feat/restricted-tier-wip`).
|
2026-07-18 01:06:34 +00:00
|
|
|
|
|
2026-07-18 04:09:48 +00:00
|
|
|
|
## What #74 asked
|
2026-07-18 01:06:34 +00:00
|
|
|
|
|
2026-07-18 04:09:48 +00:00
|
|
|
|
A restricted (`incus`-group) user can `box new/list/shell/snapshot/rm` their
|
|
|
|
|
|
own boxes, on a network carrying box's **full** isolation contract, seeing no
|
|
|
|
|
|
one else's; the convergence is a documented, idempotent path; the rehearsal
|
|
|
|
|
|
passes criteria (a)–(f); the admin tier is unchanged.
|
2026-07-18 01:06:34 +00:00
|
|
|
|
|
2026-07-18 04:09:48 +00:00
|
|
|
|
## The three facts that shaped the design
|
2026-07-18 01:06:34 +00:00
|
|
|
|
|
2026-07-18 04:09:48 +00:00
|
|
|
|
Task-0 (#72) measured one: incus-user confines users to `user-<uid>` projects
|
|
|
|
|
|
(sound), but pins them to a private auto-created bridge `incusbr-<uid>` and
|
|
|
|
|
|
`restricted.networks.access: incusbr-<uid>` — they cannot even see `boxnet`.
|
2026-07-18 01:06:34 +00:00
|
|
|
|
|
2026-07-18 04:09:48 +00:00
|
|
|
|
This round measured two more:
|
2026-07-18 01:06:34 +00:00
|
|
|
|
|
2026-07-18 04:09:48 +00:00
|
|
|
|
1. **The private bridge is worse than unhardened.** `incusbr-<uid>` is a
|
|
|
|
|
|
fully functional NAT bridge — `ipv4.nat=true`, **`ipv6.nat=true`** — with
|
|
|
|
|
|
no ACL, no `dns.mode=none`, no resolver pin, no port isolation, and IPv6
|
|
|
|
|
|
egress box's contract explicitly forbids. Any instance placed on it holds
|
|
|
|
|
|
a door to the host's LAN.
|
|
|
|
|
|
2. **incus-user projects block snapshots** (`Project "user-<uid>" doesn't
|
|
|
|
|
|
allow for snapshot creation`) — box's entire reuse workflow.
|
|
|
|
|
|
|
|
|
|
|
|
And two open questions from #74, answered from incus-user's own source
|
|
|
|
|
|
(`cmd/incus-user/server.go`, stable-6.0) and then confirmed live:
|
|
|
|
|
|
|
|
|
|
|
|
- **Daemon-level project template?** None exists — the project config is
|
|
|
|
|
|
hardcoded in `serverSetupUser()`. A per-user admin hook is the only path.
|
|
|
|
|
|
- **Does widening survive a re-sync?** Yes. Setup runs only when the project
|
|
|
|
|
|
does not exist (and early-outs when the user's certificate is already
|
|
|
|
|
|
trusted); incus-user never rewrites an existing project. Confirmed live:
|
|
|
|
|
|
`systemctl restart incus-user.socket` leaves the convergence intact.
|
|
|
|
|
|
|
|
|
|
|
|
## The decision: option 1, tightened
|
|
|
|
|
|
|
|
|
|
|
|
#74 offered (1) converge users onto the shared `boxnet` or (2) harden each
|
|
|
|
|
|
private bridge. Option 2 multiplies every mechanism per user (ACL, resolver
|
|
|
|
|
|
pin, dnsmasq, nft rules, firewall coexistence) and turns the shipped static
|
|
|
|
|
|
profile into N generated ones. Option 1 keeps one hardened network and one
|
|
|
|
|
|
shipped profile — and the existing box↔box mechanisms already make
|
|
|
|
|
|
cross-user isolation free: the nft bridge-family drop and `dns.mode=none`
|
|
|
|
|
|
are host/network-owned, so they bind every instance on `boxnet` no matter
|
|
|
|
|
|
whose project it lives in.
|
|
|
|
|
|
|
|
|
|
|
|
One tightening beyond the issue's sketch: the issue proposed
|
|
|
|
|
|
`restricted.networks.access boxnet,incusbr-<uid>` ("must list both" — true
|
|
|
|
|
|
as long as the default profile still references the private bridge). Listing
|
|
|
|
|
|
both leaves fact 1's unhardened bridge one `--network` flag away, forever.
|
|
|
|
|
|
Instead, `box grant`:
|
|
|
|
|
|
|
|
|
|
|
|
- removes `eth0` from the project's default profile (nothing references the
|
|
|
|
|
|
private bridge anymore, so the narrowing validates), and
|
|
|
|
|
|
- sets `restricted.networks.access boxnet` — **only**.
|
|
|
|
|
|
|
|
|
|
|
|
After which the hardened network is not the user's default placement but the
|
|
|
|
|
|
only placement their certificate can express. Measured: `incus launch
|
|
|
|
|
|
--network incusbr-<uid>` as the user → `Network not found`; the user cannot
|
|
|
|
|
|
widen their own project (`Error: Certificate is restricted`); they cannot
|
|
|
|
|
|
touch `boxnet`'s config or the ACL (`no permission for project "default"`).
|
|
|
|
|
|
|
|
|
|
|
|
A restricted user CAN edit the `box-net` profile copy in their own project
|
|
|
|
|
|
(they own project profiles — `features.profiles=true`), including stripping
|
grant/rehearsal: the codex round — verified rollback, loud partial states, and the raw-attach guarantee measured (#75)
Review 4727756972 (A2): the backout no longer trusts gpasswd — it re-reads
the live group database after removal; verified-absent gets the safe
message, anything else screams ROLLBACK INCOMPLETE, exits nonzero, and
names the exact remediation. The concurrent-login window (a session begun
between usermod and backout keeps the group) is CLOSED to the extent the
database can't reach: the backout detects live processes and names
loginctl terminate-user, and the success wording claims only what was
verified.
Review 4727641752 (A1): a failed grant for a user whose membership predates
the run (the hand-added-user scenario) now fails LOUDLY — they retain
socket access on part-converged policy, and the message says so with both
remediations (box revoke now, or fix and re-run). Their membership is not
stripped: breaking a working user over a failed re-grant is its own hazard.
The default-profile eth0 removal is deliberately not restored on failure —
that mutation only reduces capability, and restoring it would move the
failure state AWAY from fail-closed. Injected-failure coverage is criterion
(n), both flavors: fresh-user backout (fault at the LAST mutation, so the
rollback runs after every earlier one) with the group's absence verified
and a converging re-run; blocked narrowing staged for real with an
instance-local NIC parked on the private bridge.
Review A3, resolution 3 with the measurement demanded: criterion (m)
launches exactly 'incus launch --network boxnet' as the restricted user and
probes the raw NIC from inside — egress works, RFC1918 dropped (the ACL is
the network's), sibling probes dropped BOTH directions (the nft drop is the
host's), name enumeration blocked. The scoped guarantee is now stated in
box-design.md and measured on every run: box-minted instances carry per-NIC
port_isolation; raw attachments keep every network- and host-owned control,
losing only that redundant L2 layer. Instrument lesson kept as MU-5: the
probe's first cut minted the non-cloud image — no DHCP client, no lease,
and a dead NIC passes every negative probe vacuously; it now requires the
lease before believing its own answers.
Rehearsal: 54/54 (containers). test/cli.sh: 82 checks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 06:41:05 +00:00
|
|
|
|
`security.port_isolation` — and CAN attach `boxnet` raw with `--network
|
|
|
|
|
|
boxnet`, no profile at all (the network must be in
|
|
|
|
|
|
`restricted.networks.access` for the profile to work; there is no
|
|
|
|
|
|
allow-via-profile-only lever). That is why the host-owned nft bridge drop is
|
|
|
|
|
|
the second layer: `meta ibrname boxnet obrname boxnet drop` fires on every
|
|
|
|
|
|
port-to-port frame regardless of per-NIC flags. The documented guarantee is
|
|
|
|
|
|
scoped accordingly (see box-design.md): box-minted instances carry per-NIC
|
|
|
|
|
|
port isolation; raw attachments keep every network- and host-owned control,
|
|
|
|
|
|
losing only that redundant L2 layer. Both shapes are measured from inside
|
|
|
|
|
|
the instances (rehearsal criteria g and m).
|
|
|
|
|
|
|
|
|
|
|
|
Two grant-failure contracts, both injected in the rehearsal (criterion n):
|
|
|
|
|
|
a fresh user is backed out of the group with the removal VERIFIED against
|
|
|
|
|
|
the live group database (and any session begun mid-grant is named, with the
|
|
|
|
|
|
loginctl remedy — the one window the database cannot close); a pre-existing
|
|
|
|
|
|
member is never stripped by a failed re-grant, but the failure states out
|
|
|
|
|
|
loud that they retain socket access on part-converged policy, with both
|
|
|
|
|
|
remediations. The default-profile eth0 removal is deliberately NOT restored
|
|
|
|
|
|
on failure: that mutation only reduces capability, and restoring it would
|
|
|
|
|
|
move the failure state away from fail-closed. Every step is check-then-
|
|
|
|
|
|
converge, which is what makes re-run-to-repair deterministic.
|
2026-07-18 04:09:48 +00:00
|
|
|
|
|
|
|
|
|
|
## What `box grant <user>` converges (idempotent, re-run to refresh)
|
|
|
|
|
|
|
|
|
|
|
|
1. `usermod -aG incus` (not `incus-admin` — that is the tier)
|
|
|
|
|
|
2. first-touch incus-user as the user (`runuser`/`sudo -u`, stdin pinned) —
|
|
|
|
|
|
the project is created lazily and cannot be pre-created by an admin
|
|
|
|
|
|
3. remove the default profile's private-bridge `eth0`
|
|
|
|
|
|
4. `restricted.networks.access boxnet`
|
|
|
|
|
|
5. `restricted.snapshots allow`
|
|
|
|
|
|
6. install/refresh the shipped `box-net` profile into the project
|
|
|
|
|
|
7. verify from the user's side of the socket
|
|
|
|
|
|
|
|
|
|
|
|
`box revoke <user>` is the inverse, two strengths: bare = group removal (the
|
|
|
|
|
|
socket closes; boxes keep running; re-grant restores), `--purge` = boxes,
|
|
|
|
|
|
images, project, private bridge, trust-store certificate, incus-user state —
|
|
|
|
|
|
then asserts the absence (the wipe.sh discipline).
|
|
|
|
|
|
|
|
|
|
|
|
## The tier in the CLI
|
|
|
|
|
|
|
|
|
|
|
|
`box_tier()` — UID 0 / `incus-admin` → admin, `incus` alone → restricted,
|
|
|
|
|
|
neither → none — decided from live process credentials (argless `id -nG`),
|
|
|
|
|
|
byte-identical in `bin/box` and `host/setup-host.sh` (diffed by a test).
|
|
|
|
|
|
Tier-aware surface: `new` pre-flights the profile and names the right fix per
|
|
|
|
|
|
tier; `expose` refuses before any daemon call (its plumbing is daemon-global;
|
|
|
|
|
|
without the guard the failure is a lie — a restricted user cannot read
|
|
|
|
|
|
boxnet's redacted config, so box_net_ip would claim their running box has no
|
|
|
|
|
|
address); `setup-host` exits 0 with the honest note; `doctor` runs a
|
|
|
|
|
|
restricted check-set (is the tier granted, does their box resolve/route)
|
|
|
|
|
|
instead of judging host state they cannot see.
|
|
|
|
|
|
|
|
|
|
|
|
Found along the way, fixed for every tier: `box restore` dispatched
|
|
|
|
|
|
`incus restore`, which does not exist in Incus 6 (`incus snapshot restore`).
|
|
|
|
|
|
It had never worked.
|
|
|
|
|
|
|
|
|
|
|
|
## Rehearsal and CI
|
|
|
|
|
|
|
|
|
|
|
|
`drill/multiuser.sh` (root, opt-in via `BOX_MULTIUSER_REHEARSAL=1`) proves
|
fix: grant the 'incus' membership to incus-admin members too (#101 review)
The previous revision skipped `usermod -aG incus` for an incus-admin member,
reasoning that 'incus' is a strict subset of what incus-admin already opens.
That is true of the daemon API and false of the filesystem. On Debian 13 /
Incus 6.0.4 the two sockets are two files with two owning groups:
/var/lib/incus/unix.socket group incus-admin 0660
/var/lib/incus/unix.socket.user group incus 0660
incus-admin opens the first and not the second, and only the second
provisions a user-<uid> project. So for the incus-admin-ONLY user — the
canonical #99 case — the pinned provisioning touch took EACCES, the `|| true`
swallowed it, no project appeared, and the grant died blaming a healthy
incus-user. Both reviewers converged on this independently and were right.
The membership is now granted for everyone, with output carrying the concern
the old no-op was built around (it is the key to a file, not a privilege;
box_tier still reads them as admin). Everything downstream moves with it:
- the backout rolls that membership back and verified, while refusing to call
the rollback a lockout — incus-admin is untouched and still opens the host
- revoke's bare path takes the membership back and reports `partial:` instead
of "no-op, nothing was taken", still declining to call them "out"
- grant's closing "gpasswd -d <user> incus-admin (no re-grant needed)" is now
a true promise: they keep 'incus', so the drop lands them in their project
- the socket existence probe goes through $SUDO, matching revoke's measured
discipline about /var/lib/incus lying to a non-root admin
Tests: the cli.sh assertions that encoded the old no-op design are flipped and
the decision is pinned at the seam that broke; the sudo shim now runs `test`
for real in both directions. Because the shims model neither INCUS_SOCKET nor
permissions and so cannot reproduce the EACCES, drill/multiuser.sh gains
criterion (o): an incus-admin-only member granted on real Incus in CI, with
the membership, the project, a live connect() to unix.socket.user, and the
post-drop landing all measured.
Mutation-checked: 11 of the new/flipped assertions fail against the previous
implementation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 16:50:42 +00:00
|
|
|
|
criteria (a)–(f) from #74 plus the measured extensions (g)–(o): the in-box
|
2026-07-18 04:09:48 +00:00
|
|
|
|
isolation contract (egress, DNS, box→host, RFC1918, cross-user sibling drop,
|
|
|
|
|
|
name enumeration, IPv6-off), the closed escape hatches, re-sync survival, and
|
fix: grant the 'incus' membership to incus-admin members too (#101 review)
The previous revision skipped `usermod -aG incus` for an incus-admin member,
reasoning that 'incus' is a strict subset of what incus-admin already opens.
That is true of the daemon API and false of the filesystem. On Debian 13 /
Incus 6.0.4 the two sockets are two files with two owning groups:
/var/lib/incus/unix.socket group incus-admin 0660
/var/lib/incus/unix.socket.user group incus 0660
incus-admin opens the first and not the second, and only the second
provisions a user-<uid> project. So for the incus-admin-ONLY user — the
canonical #99 case — the pinned provisioning touch took EACCES, the `|| true`
swallowed it, no project appeared, and the grant died blaming a healthy
incus-user. Both reviewers converged on this independently and were right.
The membership is now granted for everyone, with output carrying the concern
the old no-op was built around (it is the key to a file, not a privilege;
box_tier still reads them as admin). Everything downstream moves with it:
- the backout rolls that membership back and verified, while refusing to call
the rollback a lockout — incus-admin is untouched and still opens the host
- revoke's bare path takes the membership back and reports `partial:` instead
of "no-op, nothing was taken", still declining to call them "out"
- grant's closing "gpasswd -d <user> incus-admin (no re-grant needed)" is now
a true promise: they keep 'incus', so the drop lands them in their project
- the socket existence probe goes through $SUDO, matching revoke's measured
discipline about /var/lib/incus lying to a non-root admin
Tests: the cli.sh assertions that encoded the old no-op design are flipped and
the decision is pinned at the seam that broke; the sudo shim now runs `test`
for real in both directions. Because the shims model neither INCUS_SOCKET nor
permissions and so cannot reproduce the EACCES, drill/multiuser.sh gains
criterion (o): an incus-admin-only member granted on real Incus in CI, with
the membership, the project, a live connect() to unix.socket.user, and the
post-drop landing all measured.
Mutation-checked: 11 of the new/flipped assertions fail against the previous
implementation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-19 16:50:42 +00:00
|
|
|
|
scoped revoke, the raw-attach scoped guarantee (m), the grant-failure
|
|
|
|
|
|
injections (n) and the `incus-admin`-only grant (o) — the last of these
|
|
|
|
|
|
because the shim suite models neither `INCUS_SOCKET` nor socket permissions,
|
|
|
|
|
|
so the group gate on `unix.socket.user` can only be measured live (#99, #101). Real users, real grants, real mints, probes from inside;
|
2026-07-18 04:09:48 +00:00
|
|
|
|
`--container` for CI, VM mode on real hardware; cleanup deletes everything it
|
|
|
|
|
|
made.
|
|
|
|
|
|
|
|
|
|
|
|
CI gains a `rehearsal` job (ubuntu-latest): install incus, stage the checkout
|
|
|
|
|
|
at `/opt/box` (the #71 layout), `setup-host`, `doctor`, then the rehearsal in
|
|
|
|
|
|
container mode. The tier's semantics are proven on every PR against a live
|
|
|
|
|
|
daemon; the VM trust boundary stays a real-hardware ritual, like the drill.
|
|
|
|
|
|
|
|
|
|
|
|
## Environment
|
|
|
|
|
|
|
|
|
|
|
|
Debian 13 (trixie), Incus 6.0.4, `incus-user.socket` shipped in the incus
|
|
|
|
|
|
package (Debian 13 and Ubuntu 24.04 both), `/dev/kvm` present (VM-mode run),
|
|
|
|
|
|
btrfs pool. Companion work: rig#24 (`box` role), rig#12/#25 (host-class).
|