forked from heavy-duty/box
docs: the restricted tier — README, design doc, plan doc with measured results (#74)
The plan doc records what was measured and why each decision fell where it did: the private bridge is worse than unhardened (a live NAT bridge with IPv6 on), incus-user blocks snapshots, no daemon-level template exists (read in incus-user's source), widening survives re-sync (same source, then measured live). box-design gains the access-tiers section — including why narrowing to boxnet-only is the load-bearing decision and why the nft bridge drop is the layer a restricted user cannot strip. RUNS.md logs MU-1..3. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
dan-claude-bot
parent
56eed6250e
commit
2097982ac2
6 changed files with 271 additions and 21 deletions
28
CHANGELOG.md
28
CHANGELOG.md
|
|
@ -7,6 +7,28 @@ which records not just what changed but what each drill run proved.
|
|||
|
||||
### Added
|
||||
|
||||
- **The restricted tier: multi-user hosts** (#74, redesigning #72) — an admin
|
||||
runs `box grant <user>` and that user gets their own boxes on the same
|
||||
hardened `boxnet`, seeing nobody else's; `box revoke <user>` takes it back
|
||||
(`--purge` deletes their world, and asserts the absence). The tier rides
|
||||
incus-user, whose defaults miss box's contract three measured ways (Debian
|
||||
13 / Incus 6.0.4): a private *unhardened* NAT bridge per user, snapshots
|
||||
blocked, the `box-net` profile invisible — so grant is an idempotent
|
||||
convergence: project narrowed to `boxnet` **and only boxnet** (listing the
|
||||
private bridge too, the obvious fix, would keep an unhardened network one
|
||||
`--network` flag away), snapshots allowed, the shipped profile installed
|
||||
into their project. `box_tier()` (live credentials, argless `id -nG`)
|
||||
drives the tier-aware surface: `expose` refuses honestly before any daemon
|
||||
call, `setup-host` and `doctor` answer at the caller's tier. Rehearsed
|
||||
end-to-end by `drill/multiuser.sh` (criteria a–l: confinement, lifecycle,
|
||||
cross-user visibility, name collisions, the in-box isolation contract,
|
||||
escape hatches, re-sync survival, revoke) — 41/41 on the design host, in
|
||||
both container and VM mode.
|
||||
- **CI runs the multi-user rehearsal on a real Incus** — a second `rehearsal`
|
||||
job stands up the full stack on the runner (setup-host, doctor, then
|
||||
`multiuser.sh --container`), so every PR proves the tier's semantics
|
||||
against a live daemon, not a mock. The VM trust boundary itself remains a
|
||||
real-hardware ritual, like the full drill.
|
||||
- **Global / root install** (#71) — run as root, box installs *once* to
|
||||
`/opt/box` (world-readable) with the `box` symlink on `/usr/local/bin`, so
|
||||
every operator on a shared host runs the same tree. Per-user installs are
|
||||
|
|
@ -24,6 +46,12 @@ which records not just what changed but what each drill run proved.
|
|||
|
||||
### Fixed
|
||||
|
||||
- **`box restore` never worked against Incus 6** — the command table
|
||||
dispatched `incus restore`, a subcommand that does not exist (Incus 6
|
||||
spells it `incus snapshot restore`), so every restore died on "unknown
|
||||
command". Found by #74's rehearsal exercising the full lifecycle as a
|
||||
restricted user; fixed for every tier, and the rehearsal + a grep-guard in
|
||||
`test/cli.sh` now hold it.
|
||||
- **`box tmux` works on every template** (#65) — `box tmux` runs
|
||||
`tmux new-session` *inside* the box, but the templates did not install tmux, so
|
||||
it failed with `tmux: command not found`. `tmux` is now in each template's
|
||||
|
|
|
|||
43
README.md
43
README.md
|
|
@ -106,6 +106,49 @@ re-homes each legacy box onto `boxnet` (authed state preserved), and
|
|||
`box migrate-host --retire-legacy` removes the old bridge and profile once no
|
||||
legacy box remains.
|
||||
|
||||
## Multi-user hosts: the restricted tier
|
||||
|
||||
One host, several people, and not everyone should hold the daemon. Incus's
|
||||
socket is all-or-nothing — `incus-admin` group members own every instance on
|
||||
the machine — so box layers a second tier on
|
||||
[incus-user](https://linuxcontainers.org/incus/docs/main/projects/):
|
||||
|
||||
| tier | who | what they hold |
|
||||
|---|---|---|
|
||||
| **admin** | root, or the `incus-admin` group | everything: all boxes, the stack, `setup-host`, `expose`, `grant` |
|
||||
| **restricted** | the `incus` group | their **own** boxes only, on the same hardened network |
|
||||
| none | everyone else | no socket, nothing |
|
||||
|
||||
An admin hands the tier out per user, and takes it back:
|
||||
|
||||
```sh
|
||||
box grant dev1 # dev1 can now: box new / list / shell / snapshot / rm — their boxes only
|
||||
box revoke dev1 # locked out; their boxes survive (grant again restores)
|
||||
box revoke dev1 --purge # ...or delete everything they had
|
||||
```
|
||||
|
||||
`grant` is an idempotent convergence, not a flag flip, because incus-user's
|
||||
defaults miss box's contract three ways (measured on Debian 13 / Incus 6.0.4,
|
||||
see [the plan doc](docs/plans/2026-07-18-restricted-tier.md)): it pins each
|
||||
user to a private *unhardened* NAT bridge, it blocks snapshots, and it cannot
|
||||
see the `box-net` profile. Granting rewires all three: the user's project is
|
||||
restricted to `boxnet` **and only boxnet** — the hardened network is not their
|
||||
default placement but the only one their certificate can express — snapshots
|
||||
are allowed, and the shipped profile is installed into their project. Re-run
|
||||
`box grant <user>` after upgrading box to refresh the profile, like
|
||||
`setup-host` for the stack.
|
||||
|
||||
What a restricted user gets is the full contract: same ACL, same DNS
|
||||
isolation, same pinned resolver, same port isolation, same box↔box drop —
|
||||
and their boxes cannot reach another user's box, which is the same
|
||||
box↔box drop doing its one job. What they can't do stays honest: `box
|
||||
expose` (daemon-global state) says to ask an admin, `box setup-host` and
|
||||
`box doctor` answer at their tier instead of failing at it.
|
||||
|
||||
`drill/multiuser.sh` rehearses all of it live — two users, real grants, real
|
||||
boxes, probes from inside — and CI runs it on every PR (container mode; the
|
||||
VM boundary itself is proven on real hardware, like the rest of the drill).
|
||||
|
||||
## Quick start
|
||||
|
||||
```sh
|
||||
|
|
|
|||
|
|
@ -127,7 +127,58 @@ The rule that keeps this honest: **isolation claims are tested, never reasoned
|
|||
about.** The box→box hole existed because a plausible code reading said it could
|
||||
not. See `drill/RUNS.md`.
|
||||
|
||||
## Multi-user hosts: access tiers
|
||||
|
||||
The daemon socket is binary — `incus-admin` holds everything on the machine —
|
||||
so a shared host needs a second tier, and Incus ships one: **incus-user**
|
||||
confines an `incus`-group member to an auto-created project `user-<uid>`,
|
||||
behind a restricted certificate that cannot name any other project. The tier
|
||||
is decided once, from the process's live credentials (`box_tier()`: UID 0 or
|
||||
`incus-admin` → admin; `incus` alone → restricted; neither → none), and every
|
||||
tier-aware verb reads that one function.
|
||||
|
||||
What incus-user does *not* do is honor box's contract — measured on Debian 13
|
||||
/ Incus 6.0.4 (#74), after the design that assumed it (#72) was vetoed by its
|
||||
own Task-0 rehearsal:
|
||||
|
||||
- it pins each user's project to a private auto-created bridge
|
||||
(`incusbr-<uid>`) — a stock NAT bridge with **none** of the hardening: no
|
||||
ACL, no `dns.mode=none`, no resolver pin, IPv6 on;
|
||||
- it blocks snapshots — box's entire reuse workflow;
|
||||
- the `box-net` profile lives in the default project, invisible to theirs.
|
||||
|
||||
So the tier is an **admin-run convergence** (`box grant <user>`), not a
|
||||
group membership: put them in `incus`, touch incus-user once as them (the
|
||||
project is created lazily; nothing exists to converge until it does), then
|
||||
rewire the project — network access narrowed to `boxnet` **and only
|
||||
`boxnet`**, snapshots allowed, the shipped profile installed. Narrowing is
|
||||
the load-bearing decision: granting `boxnet,incusbr-<uid>` (the obvious fix)
|
||||
would leave an unhardened NAT bridge one `--network` flag away from any box
|
||||
they mint. With the private bridge unreferenced (its `eth0` is removed from
|
||||
their default profile) and outside `restricted.networks.access`, the hardened
|
||||
network is not their default placement — it is the only placement their
|
||||
certificate can express. The grant survives incus-user restarts by that
|
||||
tool's own design (it configures a project only at creation), and a restricted
|
||||
certificate cannot widen its own project — both measured, not read.
|
||||
|
||||
Cross-USER isolation is the same mechanism as cross-box isolation, on
|
||||
purpose: their instances share `boxnet` with everyone's, and the bridge-family
|
||||
drop + port isolation + `dns.mode=none` already make any two boxes strangers.
|
||||
A restricted user CAN strip `security.port_isolation` from the profile copy
|
||||
in their own project — which is why the nft bridge drop, which they cannot
|
||||
touch, exists as the second, host-owned layer. Defense in depth, both layers
|
||||
measured (`drill/multiuser.sh`, criteria a–l).
|
||||
|
||||
`box revoke` is two strengths: bare, it removes the group — the socket
|
||||
closes, their boxes keep *running* (revoking a person does not kill their
|
||||
workloads) and `grant` restores everything; `--purge` deletes their world
|
||||
(boxes, images, project, private bridge, trust-store certificate) and asserts
|
||||
the absence afterwards.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No unattended/CI bring-up — the flow is interactive.
|
||||
- Interactive-first: install and setup prompt by default (`BOX_YES=1` and the
|
||||
CI rehearsal job are the sanctioned unattended paths).
|
||||
- No credential storage or injection by the tool.
|
||||
- No per-user resource quotas on the restricted tier (Incus's
|
||||
`limits.*`/`restricted.*` project keys exist when someone needs them).
|
||||
|
|
|
|||
|
|
@ -1,27 +1,128 @@
|
|||
# Restricted incus tier — implementation plan (issue #74)
|
||||
# Restricted incus tier — design and measured results (#74)
|
||||
|
||||
> **Status: placeholder.** This PR is a work in progress; the design below is being
|
||||
> validated by a live multi-user rehearsal before the implementation lands. Do not
|
||||
> review yet — the PR stays in draft until the rehearsal passes.
|
||||
**Status: implemented and rehearsed.** 41/41 rehearsal criteria green on the
|
||||
design host (Debian 13 trixie, Incus 6.0.4, nested KVM), in container mode
|
||||
and VM mode. This doc records the design, what was measured, and why each
|
||||
decision fell where it did. It supersedes the vetoed #72 design
|
||||
(`docs/plans/2026-07-17-multiuser-hosts.md` on `feat/restricted-tier-wip`).
|
||||
|
||||
## Scope
|
||||
## What #74 asked
|
||||
|
||||
Deliver the restricted (`incus`-group) tier described in #74:
|
||||
A restricted (`incus`-group) user can `box new/list/shell/snapshot/rm` their
|
||||
own boxes, on a network carrying box's **full** isolation contract, seeing no
|
||||
one else's; the convergence is a documented, idempotent path; the rehearsal
|
||||
passes criteria (a)–(f); the admin tier is unchanged.
|
||||
|
||||
- A restricted user can `box new/list/shell/snapshot/rm` **their own** boxes.
|
||||
- Their boxes ride a network carrying box's full isolation contract
|
||||
(ACL, `dns.mode=none`, resolver pin, `security.port_isolation`, nft box↔box drop).
|
||||
- No cross-user visibility. Admin tier unchanged.
|
||||
- The admin-side convergence is a documented, idempotent command — not manual
|
||||
per-user `incus project set`.
|
||||
## The three facts that shaped the design
|
||||
|
||||
## Planned shape (subject to rehearsal)
|
||||
Task-0 (#72) measured one: incus-user confines users to `user-<uid>` projects
|
||||
(sound), but pins them to a private auto-created bridge `incusbr-<uid>` and
|
||||
`restricted.networks.access: incusbr-<uid>` — they cannot even see `boxnet`.
|
||||
|
||||
- `box grant <user>` / `box revoke <user>` — admin convergence hook per #74
|
||||
option 1: widen `restricted.networks.access` to include `boxnet` and install
|
||||
the `box-net` profile into the user's `user-<uid>` project.
|
||||
- CLI awareness of running inside a restricted project.
|
||||
- `drill/multiuser.sh` rehearsal criteria (a)–(f) green on a real multi-user host.
|
||||
- Test suite expansion + CI wiring.
|
||||
This round measured two more:
|
||||
|
||||
Tracking issue: heavy-duty/box#74.
|
||||
1. **The private bridge is worse than unhardened.** `incusbr-<uid>` is a
|
||||
fully functional NAT bridge — `ipv4.nat=true`, **`ipv6.nat=true`** — with
|
||||
no ACL, no `dns.mode=none`, no resolver pin, no port isolation, and IPv6
|
||||
egress box's contract explicitly forbids. Any instance placed on it holds
|
||||
a door to the host's LAN.
|
||||
2. **incus-user projects block snapshots** (`Project "user-<uid>" doesn't
|
||||
allow for snapshot creation`) — box's entire reuse workflow.
|
||||
|
||||
And two open questions from #74, answered from incus-user's own source
|
||||
(`cmd/incus-user/server.go`, stable-6.0) and then confirmed live:
|
||||
|
||||
- **Daemon-level project template?** None exists — the project config is
|
||||
hardcoded in `serverSetupUser()`. A per-user admin hook is the only path.
|
||||
- **Does widening survive a re-sync?** Yes. Setup runs only when the project
|
||||
does not exist (and early-outs when the user's certificate is already
|
||||
trusted); incus-user never rewrites an existing project. Confirmed live:
|
||||
`systemctl restart incus-user.socket` leaves the convergence intact.
|
||||
|
||||
## The decision: option 1, tightened
|
||||
|
||||
#74 offered (1) converge users onto the shared `boxnet` or (2) harden each
|
||||
private bridge. Option 2 multiplies every mechanism per user (ACL, resolver
|
||||
pin, dnsmasq, nft rules, firewall coexistence) and turns the shipped static
|
||||
profile into N generated ones. Option 1 keeps one hardened network and one
|
||||
shipped profile — and the existing box↔box mechanisms already make
|
||||
cross-user isolation free: the nft bridge-family drop and `dns.mode=none`
|
||||
are host/network-owned, so they bind every instance on `boxnet` no matter
|
||||
whose project it lives in.
|
||||
|
||||
One tightening beyond the issue's sketch: the issue proposed
|
||||
`restricted.networks.access boxnet,incusbr-<uid>` ("must list both" — true
|
||||
as long as the default profile still references the private bridge). Listing
|
||||
both leaves fact 1's unhardened bridge one `--network` flag away, forever.
|
||||
Instead, `box grant`:
|
||||
|
||||
- removes `eth0` from the project's default profile (nothing references the
|
||||
private bridge anymore, so the narrowing validates), and
|
||||
- sets `restricted.networks.access boxnet` — **only**.
|
||||
|
||||
After which the hardened network is not the user's default placement but the
|
||||
only placement their certificate can express. Measured: `incus launch
|
||||
--network incusbr-<uid>` as the user → `Network not found`; the user cannot
|
||||
widen their own project (`Error: Certificate is restricted`); they cannot
|
||||
touch `boxnet`'s config or the ACL (`no permission for project "default"`).
|
||||
|
||||
A restricted user CAN edit the `box-net` profile copy in their own project
|
||||
(they own project profiles — `features.profiles=true`), including stripping
|
||||
`security.port_isolation`. That is why the host-owned nft bridge drop is the
|
||||
second layer: `meta ibrname boxnet obrname boxnet drop` fires on every
|
||||
port-to-port frame regardless of per-NIC flags. Cross-user sibling probes are
|
||||
dropped either way — measured from inside the boxes.
|
||||
|
||||
## What `box grant <user>` converges (idempotent, re-run to refresh)
|
||||
|
||||
1. `usermod -aG incus` (not `incus-admin` — that is the tier)
|
||||
2. first-touch incus-user as the user (`runuser`/`sudo -u`, stdin pinned) —
|
||||
the project is created lazily and cannot be pre-created by an admin
|
||||
3. remove the default profile's private-bridge `eth0`
|
||||
4. `restricted.networks.access boxnet`
|
||||
5. `restricted.snapshots allow`
|
||||
6. install/refresh the shipped `box-net` profile into the project
|
||||
7. verify from the user's side of the socket
|
||||
|
||||
`box revoke <user>` is the inverse, two strengths: bare = group removal (the
|
||||
socket closes; boxes keep running; re-grant restores), `--purge` = boxes,
|
||||
images, project, private bridge, trust-store certificate, incus-user state —
|
||||
then asserts the absence (the wipe.sh discipline).
|
||||
|
||||
## The tier in the CLI
|
||||
|
||||
`box_tier()` — UID 0 / `incus-admin` → admin, `incus` alone → restricted,
|
||||
neither → none — decided from live process credentials (argless `id -nG`),
|
||||
byte-identical in `bin/box` and `host/setup-host.sh` (diffed by a test).
|
||||
Tier-aware surface: `new` pre-flights the profile and names the right fix per
|
||||
tier; `expose` refuses before any daemon call (its plumbing is daemon-global;
|
||||
without the guard the failure is a lie — a restricted user cannot read
|
||||
boxnet's redacted config, so box_net_ip would claim their running box has no
|
||||
address); `setup-host` exits 0 with the honest note; `doctor` runs a
|
||||
restricted check-set (is the tier granted, does their box resolve/route)
|
||||
instead of judging host state they cannot see.
|
||||
|
||||
Found along the way, fixed for every tier: `box restore` dispatched
|
||||
`incus restore`, which does not exist in Incus 6 (`incus snapshot restore`).
|
||||
It had never worked.
|
||||
|
||||
## Rehearsal and CI
|
||||
|
||||
`drill/multiuser.sh` (root, opt-in via `BOX_MULTIUSER_REHEARSAL=1`) proves
|
||||
criteria (a)–(f) from #74 plus the measured extensions (g)–(l): the in-box
|
||||
isolation contract (egress, DNS, box→host, RFC1918, cross-user sibling drop,
|
||||
name enumeration, IPv6-off), the closed escape hatches, re-sync survival, and
|
||||
scoped revoke. Two real users, real grants, real mints, probes from inside;
|
||||
`--container` for CI, VM mode on real hardware; cleanup deletes everything it
|
||||
made.
|
||||
|
||||
CI gains a `rehearsal` job (ubuntu-latest): install incus, stage the checkout
|
||||
at `/opt/box` (the #71 layout), `setup-host`, `doctor`, then the rehearsal in
|
||||
container mode. The tier's semantics are proven on every PR against a live
|
||||
daemon; the VM trust boundary stays a real-hardware ritual, like the drill.
|
||||
|
||||
## Environment
|
||||
|
||||
Debian 13 (trixie), Incus 6.0.4, `incus-user.socket` shipped in the incus
|
||||
package (Debian 13 and Ubuntu 24.04 both), `/dev/kvm` present (VM-mode run),
|
||||
btrfs pool. Companion work: rig#24 (`box` role), rig#12/#25 (host-class).
|
||||
|
|
|
|||
|
|
@ -91,3 +91,27 @@ installed and runnable; authenticating is yours.
|
|||
If the host has no `/dev/kvm`, box falls back to container mode. The drill
|
||||
still runs, but it says loudly that **the VM trust boundary was not validated**
|
||||
rather than passing quietly on a weaker one.
|
||||
|
||||
## The multi-user rehearsal (`multiuser.sh`)
|
||||
|
||||
The restricted tier (#74) has its own rehearsal — the drill proves one
|
||||
operator's host; this proves a *shared* one:
|
||||
|
||||
```sh
|
||||
sudo BOX_MULTIUSER_REHEARSAL=1 bash drill/multiuser.sh --yes
|
||||
```
|
||||
|
||||
Root only, opt-in twice (it creates system users and edits the group
|
||||
database). It creates two throwaway users, grants them the tier through the
|
||||
real `box grant`, mints real boxes as them, and measures — from inside those
|
||||
boxes — that each user is confined to their own project (a), the full
|
||||
lifecycle works (b), no cross-user visibility (c), names don't collide (d),
|
||||
`expose`/`setup-host`/`doctor` answer honestly at the tier (e/f), the boxes
|
||||
ride `boxnet` under the full isolation contract including the cross-user
|
||||
sibling drop (g), the private-bridge escape hatches are closed (h), the grant
|
||||
survives an incus-user restart (k), and `revoke --purge` erases one user
|
||||
without touching the other (l). Everything it makes, it deletes.
|
||||
|
||||
`--container` skips VM mints (CI runs it this way on every PR — the tier's
|
||||
semantics are instance-type-independent); on real hardware run it bare so the
|
||||
boxes are VMs. `--keep` leaves the users and boxes up for inspection.
|
||||
|
|
|
|||
|
|
@ -195,6 +195,9 @@ No listener is needed, and none should be started: see trap 3.
|
|||
|
||||
| Run | Result | What it cost |
|
||||
| --- | --- | --- |
|
||||
| **MU-3** | **41/0 (containers) + 41/0 (VMs) — the tier holds** | `multiuser.sh` (#74) from zero on Debian 13 / Incus 6.0.4: grant, confinement, lifecycle, the in-box contract with the cross-user sibling drop, closed escape hatches, re-sync survival, scoped revoke. The VM pass needed right-sized boxes (1GiB): two 2GiB nested VMs on a 7.7GiB host measured swap, not isolation |
|
||||
| MU-2 | 40/41 (VMs) | the egress probe cried broken over a TLS handshake that outlived its 8s budget under nested-KVM memory pressure — every isolation probe (which PASSES on silence) was immune. Probe budgets raised, reachable-probes retry once, and the mint keeps its log for the post-mortem |
|
||||
| MU-1 | 39/41 (containers) | both FAILs were the instrument: a `box new` rc swallowed with its narration (the drill's oldest sin, relearned), and the enumeration probe resolving the box's OWN name — /etc/hosts answered, not the gateway. Probe a SIBLING's name or you measure cloud-init's hostname hygiene. Found for real along the way: `box restore` dispatched `incus restore`, which does not exist in Incus 6 — broken for every tier since the rename |
|
||||
| 1 | hung at C7; ~9 false FAILs | traps 1, 2, 4 — pipefail, the exec-pty hang, the DHCP race |
|
||||
| 2 | 42/49 | trap 5 — an interrupted run 1 left boxes behind, cascading 5 FAILs. Found the `claude`-on-PATH bug. Phase D delivered B1 (`@internal` rejected) and a B3 reading of *broken* |
|
||||
| 3 | 48/49 | trap 4 — `eth0_ip` never matched, so A3 again unprobed. B3 now read *intact*, contradicting run 2 |
|
||||
|
|
|
|||
Loading…
Reference in a new issue