Restricted incus tier: per-user projects converged onto hardened boxnet (#74) #75
14 changed files with 1517 additions and 11 deletions
46
.github/workflows/ci.yml
vendored
46
.github/workflows/ci.yml
vendored
|
|
@ -22,11 +22,41 @@ jobs:
|
|||
- name: cli tests
|
||||
run: bash test/cli.sh
|
||||
|
||||
# NOT run here: the drill (drill/drill.sh) and the multi-user rehearsal
|
||||
# (drill/multiuser.sh). Both stand up a real Incus, rearrange the host's
|
||||
# network/firewall, and — for the rehearsal — create throwaway system users.
|
||||
# None of that belongs in CI: it needs a disposable host and root, and it is
|
||||
# the answer to #72 Task 0's substrate question, which a container runner
|
||||
# cannot answer. CI stays static + dependency-free (shellcheck + cli.sh),
|
||||
# exactly as this repo's design demands ("isolation claims are tested on a
|
||||
# real host, never reasoned about" — docs/box-design.md).
|
||||
# The multi-user rehearsal, on a REAL incus — a GitHub runner is root on a
|
||||
# disposable VM, which is exactly the substrate the rehearsal needs. It runs
|
||||
# in container mode: the tier's mechanics (grant, confinement, the network
|
||||
# contract, revoke) are identical for containers and VMs — the nft bridge
|
||||
# drop, the ACL, dns.mode=none and port_isolation all bind to boxnet, not
|
||||
# to the instance type. What container mode canNOT validate is the VM trust
|
||||
# boundary itself; that stays a real-hardware ritual (drill/RUNS.md), same
|
||||
# as the full drill. So: every PR proves the tier's semantics, and a
|
||||
# release still proves the boundary.
|
||||
rehearsal:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: install incus
|
||||
run: |
|
||||
sudo apt-get update
|
||||
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y incus
|
||||
- name: global install, from this checkout (the #71 layout)
|
||||
# Not install.sh: it installs a RELEASE (REPO@REF), and CI must prove
|
||||
# the code under review. Same tree, same layout, same a+rX.
|
||||
run: |
|
||||
sudo cp -r . /opt/box
|
||||
sudo rm -rf /opt/box/.git
|
||||
sudo chmod -R a+rX /opt/box
|
||||
sudo ln -sf /opt/box/bin/box /usr/local/bin/box
|
||||
- name: setup-host
|
||||
run: sudo bash /opt/box/host/setup-host.sh
|
||||
- name: doctor — the baseline is provable before anything is judged
|
||||
run: sudo BOX_TIER=admin bash /opt/box/drill/doctor.sh
|
||||
- name: multi-user rehearsal (criteria a-l, container mode)
|
||||
run: sudo BOX_MULTIUSER_REHEARSAL=1 bash /opt/box/drill/multiuser.sh --yes --container
|
||||
|
||||
# NOT run here: the full drill (drill/drill.sh). It rehearses the whole
|
||||
# surface — cold template mints, expose, migration — and wants a real host
|
||||
# and the better part of an hour. The rehearsal job above is the CI-shaped
|
||||
# slice of the same discipline: isolation claims are still tested on a real
|
||||
# daemon, never reasoned about (docs/box-design.md).
|
||||
|
|
|
|||
28
CHANGELOG.md
28
CHANGELOG.md
|
|
@ -7,6 +7,28 @@ which records not just what changed but what each drill run proved.
|
|||
|
||||
### Added
|
||||
|
||||
- **The restricted tier: multi-user hosts** (#74, redesigning #72) — an admin
|
||||
runs `box grant <user>` and that user gets their own boxes on the same
|
||||
hardened `boxnet`, seeing nobody else's; `box revoke <user>` takes it back
|
||||
(`--purge` deletes their world, and asserts the absence). The tier rides
|
||||
incus-user, whose defaults miss box's contract three measured ways (Debian
|
||||
13 / Incus 6.0.4): a private *unhardened* NAT bridge per user, snapshots
|
||||
blocked, the `box-net` profile invisible — so grant is an idempotent
|
||||
convergence: project narrowed to `boxnet` **and only boxnet** (listing the
|
||||
private bridge too, the obvious fix, would keep an unhardened network one
|
||||
`--network` flag away), snapshots allowed, the shipped profile installed
|
||||
into their project. `box_tier()` (live credentials, argless `id -nG`)
|
||||
drives the tier-aware surface: `expose` refuses honestly before any daemon
|
||||
call, `setup-host` and `doctor` answer at the caller's tier. Rehearsed
|
||||
end-to-end by `drill/multiuser.sh` (criteria a–n: confinement, lifecycle,
|
||||
cross-user visibility, name collisions, the in-box isolation contract,
|
||||
escape hatches, re-sync survival, revoke incl. the live-session case) —
|
||||
54/54 on the design host (container and VM mode), including the raw-attach scoped-guarantee measurement and both grant-failure injections demanded by #75's review.
|
||||
- **CI runs the multi-user rehearsal on a real Incus** — a second `rehearsal`
|
||||
job stands up the full stack on the runner (setup-host, doctor, then
|
||||
`multiuser.sh --container`), so every PR proves the tier's semantics
|
||||
against a live daemon, not a mock. The VM trust boundary itself remains a
|
||||
real-hardware ritual, like the full drill.
|
||||
- **Global / root install** (#71) — run as root, box installs *once* to
|
||||
`/opt/box` (world-readable) with the `box` symlink on `/usr/local/bin`, so
|
||||
every operator on a shared host runs the same tree. Per-user installs are
|
||||
|
|
@ -24,6 +46,12 @@ which records not just what changed but what each drill run proved.
|
|||
|
||||
### Fixed
|
||||
|
||||
- **`box restore` never worked against Incus 6** — the command table
|
||||
dispatched `incus restore`, a subcommand that does not exist (Incus 6
|
||||
spells it `incus snapshot restore`), so every restore died on "unknown
|
||||
command". Found by #74's rehearsal exercising the full lifecycle as a
|
||||
restricted user; fixed for every tier, and the rehearsal + a grep-guard in
|
||||
`test/cli.sh` now hold it.
|
||||
- **`box tmux` works on every template** (#65) — `box tmux` runs
|
||||
`tmux new-session` *inside* the box, but the templates did not install tmux, so
|
||||
it failed with `tmux: command not found`. `tmux` is now in each template's
|
||||
|
|
|
|||
45
README.md
45
README.md
|
|
@ -106,6 +106,51 @@ re-homes each legacy box onto `boxnet` (authed state preserved), and
|
|||
`box migrate-host --retire-legacy` removes the old bridge and profile once no
|
||||
legacy box remains.
|
||||
|
||||
## Multi-user hosts: the restricted tier
|
||||
|
||||
One host, several people, and not everyone should hold the daemon. Incus's
|
||||
socket is all-or-nothing — `incus-admin` group members own every instance on
|
||||
the machine — so box layers a second tier on
|
||||
[incus-user](https://linuxcontainers.org/incus/docs/main/projects/):
|
||||
|
||||
| tier | who | what they hold |
|
||||
|---|---|---|
|
||||
| **admin** | root, or the `incus-admin` group | everything: all boxes, the stack, `setup-host`, `expose`, `grant` |
|
||||
| **restricted** | the `incus` group | their **own** boxes only, on the same hardened network |
|
||||
| none | everyone else | no socket, nothing |
|
||||
|
||||
An admin hands the tier out per user, and takes it back:
|
||||
|
||||
```sh
|
||||
box grant dev1 # dev1 can now: box new / list / shell / snapshot / rm — their boxes only
|
||||
box revoke dev1 # tier removed; their boxes survive (grant again restores).
|
||||
# a session they already hold keeps the socket until it
|
||||
# ends — revoke warns and names the loginctl command
|
||||
box revoke dev1 --purge # ...or end their sessions and delete everything they had
|
||||
```
|
||||
|
||||
`grant` is an idempotent convergence, not a flag flip, because incus-user's
|
||||
defaults miss box's contract three ways (measured on Debian 13 / Incus 6.0.4,
|
||||
see [the plan doc](docs/plans/2026-07-18-restricted-tier.md)): it pins each
|
||||
user to a private *unhardened* NAT bridge, it blocks snapshots, and it cannot
|
||||
see the `box-net` profile. Granting rewires all three: the user's project is
|
||||
restricted to `boxnet` **and only boxnet** — the hardened network is not their
|
||||
default placement but the only one their certificate can express — snapshots
|
||||
are allowed, and the shipped profile is installed into their project. Re-run
|
||||
`box grant <user>` after upgrading box to refresh the profile, like
|
||||
`setup-host` for the stack.
|
||||
|
||||
What a restricted user gets is the full contract: same ACL, same DNS
|
||||
isolation, same pinned resolver, same port isolation, same box↔box drop —
|
||||
and their boxes cannot reach another user's box, which is the same
|
||||
box↔box drop doing its one job. What they can't do stays honest: `box
|
||||
expose` (daemon-global state) says to ask an admin, `box setup-host` and
|
||||
`box doctor` answer at their tier instead of failing at it.
|
||||
|
||||
`drill/multiuser.sh` rehearses all of it live — two users, real grants, real
|
||||
boxes, probes from inside — and CI runs it on every PR (container mode; the
|
||||
VM boundary itself is proven on real hardware, like the rest of the drill).
|
||||
|
||||
## Quick start
|
||||
|
||||
```sh
|
||||
|
|
|
|||
100
bin/box
100
bin/box
|
|
@ -14,6 +14,24 @@ die() { echo "box: $*" >&2; exit 1; } # 1 = it went wrong
|
|||
usage_error() { echo "box: $*" >&2; echo "try 'box help'." >&2; exit 2; } # 2 = you asked wrong
|
||||
version() { echo "box $(cat "$root/VERSION" 2>/dev/null || echo unknown) ($root)"; }
|
||||
|
||||
# Which tier is THIS PROCESS? Decided from live credentials (argless 'id -nG':
|
||||
# what the kernel will present when incus opens the socket), never from the
|
||||
# group database — the two disagree for exactly as long as a re-login is
|
||||
# pending, and that window is where every wrong answer lives.
|
||||
# UID 0 / incus-admin -> admin (the full daemon socket)
|
||||
# incus (only) -> restricted (incus-user: your own project, nothing else)
|
||||
# neither -> none (no socket at all)
|
||||
# host/setup-host.sh carries a byte-identical copy (it runs before any install
|
||||
# tree exists); test/cli.sh diffs the two so they cannot drift.
|
||||
box_tier() {
|
||||
[ "$(id -u)" -eq 0 ] && { printf 'admin\n'; return; }
|
||||
local groups; groups="$(id -nG 2>/dev/null | tr ' ' '\n')"
|
||||
if printf '%s\n' "$groups" | grep -qx incus-admin; then printf 'admin\n'
|
||||
elif printf '%s\n' "$groups" | grep -qx incus; then printf 'restricted\n'
|
||||
else printf 'none\n'
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The command table.
|
||||
#
|
||||
|
|
@ -46,7 +64,7 @@ CMDS=(
|
|||
"exec^<box> -- <cmd...>^box^Run a command inside a box^fn:cmd_exec^"
|
||||
"tmux^<box> [<session>]^box^Attach or create a tmux session in a box — survives disconnects^fn:cmd_tmux^"
|
||||
"snapshot^<box> [<label>]^box^Checkpoint a box (label defaults to manual-<epoch>)^fn:cmd_snapshot^"
|
||||
"restore^<box> <snapshot>^box,arg2^Roll a box back to one of its snapshots^incus:restore^restored {} to {1}"
|
||||
"restore^<box> <snapshot>^box,arg2^Roll a box back to one of its snapshots^incus:snapshot restore^restored {} to {1}"
|
||||
"rename^<box> <new-name>^box,arg2,stopped^Rename a box (it must be stopped first)^incus:rename^renamed {} to {1}"
|
||||
"down^<box>^box^Stop a box, keeping its state ('start' resumes it)^incus:stop^stopped {}"
|
||||
"start^<box>^box^Start a stopped box^incus:start^started {}"
|
||||
|
|
@ -55,6 +73,8 @@ CMDS=(
|
|||
"incus^<box> -- <args...>^box^Escape hatch: run any incus command against a box^fn:cmd_incus^"
|
||||
"doctor^[--fix | --pin-dns]^^Is this host fit to mint boxes? Diagnose the daemon, network, DNS, isolation^fn:cmd_doctor^"
|
||||
"setup-host^^^One-time host setup: Incus, the boxnet stack, the profile, the firewall^fn:cmd_setup_host^"
|
||||
"grant^<user>^^Admin: give a host user the restricted tier — their own boxes, on the hardened boxnet^fn:cmd_grant^"
|
||||
"revoke^<user> [--purge]^^Admin: take the restricted tier back (--purge also deletes their boxes)^fn:cmd_revoke^"
|
||||
"teardown-host^[--purge-incus]^^Remove the box host stack (both name generations)^fn:cmd_teardown_host^"
|
||||
"migrate-host^--box <n> | --all-boxes | --retire-legacy^^Move a host from the pre-0.4.0 stack onto box^fn:cmd_migrate_host^"
|
||||
"status^^^Deprecated alias for 'list'^fn:cmd_status^"
|
||||
|
|
@ -180,6 +200,10 @@ THE MODEL
|
|||
user.box=1 boundary, the isolation stack, or the creds-free snapshot
|
||||
workflow. Everything else is Incus's job — and 'box incus' is the door.
|
||||
|
||||
On a shared host, an admin hands out the restricted tier per user
|
||||
('box grant <user>'): their own boxes, the same hardened network, and
|
||||
no view of anyone else's. 'box help grant' has the contract.
|
||||
|
||||
Docs: https://github.com/heavy-duty/box
|
||||
EOF
|
||||
}
|
||||
|
|
@ -394,6 +418,48 @@ One run is enough. If it has to add you to the incus-admin group it re-runs
|
|||
itself under that group — no re-login, no second invocation.
|
||||
|
||||
box setup-host
|
||||
|
||||
Multi-user hosts: setup-host builds the stack once, for everyone. An admin
|
||||
then hands individual users the restricted tier with 'box grant <user>' —
|
||||
their own boxes, on this same hardened network, seeing nobody else's.
|
||||
EOF
|
||||
;;
|
||||
grant) cat <<'EOF'
|
||||
Give a host user the restricted tier. They get their own Incus project (via
|
||||
incus-user), and every box they mint lands on the SAME hardened boxnet as an
|
||||
admin's — the full isolation contract (ACL, DNS isolation, resolver pin,
|
||||
port isolation, the box-to-box drop), with no view of anyone else's boxes.
|
||||
|
||||
What it converges, idempotently (safe to re-run, and re-run after upgrades):
|
||||
· puts the user in the 'incus' group (not incus-admin — that is the point)
|
||||
· creates their user-<uid> project by touching incus-user for them
|
||||
· points the project at boxnet and ONLY boxnet — the private incusbr-<uid>
|
||||
bridge incus-user auto-creates carries none of box's hardening, so it is
|
||||
unreferenced and unreachable, not just unused
|
||||
· allows snapshots (incus-user blocks them; box's clone workflow needs them)
|
||||
· installs the box-net profile into their project
|
||||
|
||||
The user's surface: new/list/info/shell/exec/tmux/snapshot/restore/rm on
|
||||
their own boxes. Not theirs: expose (edits daemon-global state), setup-host,
|
||||
grant. Admin boxes and other users' boxes are invisible to them, and the
|
||||
existing box-to-box drop means even their instances cannot reach a sibling.
|
||||
|
||||
box grant dev1
|
||||
EOF
|
||||
;;
|
||||
revoke) cat <<'EOF'
|
||||
Take the restricted tier back from a user. Without --purge, this removes
|
||||
them from the 'incus' group: their project and boxes stay (still running!)
|
||||
and 'box grant' restores access untouched. Group membership is read at
|
||||
LOGIN, so a session they already hold keeps the socket until it ends —
|
||||
revoke says so and names the loginctl command when it happens. With
|
||||
--purge, their sessions are terminated first (a stale session could quietly
|
||||
recreate the project, unhardened, afterwards — measured, not theoretical),
|
||||
then their boxes, images, project, private bridge and trust-store
|
||||
certificate are removed — irreversible, so it asks first.
|
||||
|
||||
box revoke dev1 # take the tier; their boxes keep running
|
||||
box revoke dev1 --purge # ...or end their sessions and delete everything
|
||||
EOF
|
||||
;;
|
||||
teardown-host) cat <<'EOF'
|
||||
|
|
@ -471,7 +537,7 @@ while [ $# -gt 0 ]; do
|
|||
# expose's own flags (--list, --remove) are positional to it, not box's
|
||||
if [ "$cmd" = expose ]; then args+=("$1"); shift; continue; fi
|
||||
# the host verbs delegate their flags to the scripts they wrap
|
||||
case "$cmd" in setup-host|teardown-host|migrate-host) args+=("$1"); shift; continue ;; esac
|
||||
case "$cmd" in setup-host|teardown-host|migrate-host|grant|revoke) args+=("$1"); shift; continue ;; esac
|
||||
if [ "$cmd" = exec ] || [ "$cmd" = incus ]; then
|
||||
usage_error "unknown option: $1 — a command's own flags go after --, as in '$(synopsis_of "$cmd")'"
|
||||
fi
|
||||
|
|
@ -688,6 +754,21 @@ cmd_templates() {
|
|||
|
||||
cmd_new() {
|
||||
[ -n "$name" ] || usage_error "usage: $(synopsis_of new)"
|
||||
# The placement contract must exist before any mint — in the DEFAULT project
|
||||
# for an admin (setup-host builds it), in YOUR project for a restricted user
|
||||
# (box grant converges it). Its absence has a different fix per tier, and
|
||||
# incus's own "Profile not found" at launch time names neither.
|
||||
if [ -z "$remote" ] && ! timeout 10 incus profile show box-net >/dev/null 2>&1 </dev/null; then
|
||||
# A missing profile and a daemon that is not answering are different
|
||||
# faults with different fixes — "run setup-host" at a wedged daemon
|
||||
# (the #26 shape) is wrong advice. Separate them before diagnosing.
|
||||
timeout 10 incus list >/dev/null 2>&1 </dev/null \
|
||||
|| die "the incus daemon is not answering — diagnose it: box doctor"
|
||||
if [ "$(box_tier)" = restricted ]; then
|
||||
die "your project has no box-net profile — the restricted tier is granted per user, by an admin: box grant $(id -un)"
|
||||
fi
|
||||
die "no box-net profile — the host stack is missing. Build it: box setup-host"
|
||||
fi
|
||||
local instance; instance="$(iname_of "$name")"
|
||||
if [ -n "$from" ]; then
|
||||
[ -z "$template" ] || usage_error "--from clones an existing box; its template rides along (drop --template)"
|
||||
|
|
@ -942,6 +1023,10 @@ cmd_status() {
|
|||
cmd_doctor() {
|
||||
local script="$root/drill/doctor.sh"
|
||||
[ -f "$script" ] || die "doctor script not found at $script — re-run install.sh"
|
||||
# The doctor's verdict depends on who is asking: a restricted user cannot
|
||||
# see the nft tables or the kernel's bridge state, and telling them the
|
||||
# host is broken because THEY cannot read it would be a wrong diagnosis.
|
||||
export BOX_TIER; BOX_TIER="$(box_tier)"
|
||||
exec bash "$script" "${args[@]}"
|
||||
}
|
||||
|
||||
|
|
@ -957,6 +1042,8 @@ host_script() { # $1 = script basename under host/
|
|||
cmd_setup_host() { host_script setup-host.sh; }
|
||||
cmd_teardown_host() { host_script teardown-host.sh; }
|
||||
cmd_migrate_host() { host_script migrate-host.sh; }
|
||||
cmd_grant() { host_script grant-user.sh; }
|
||||
cmd_revoke() { host_script revoke-user.sh; }
|
||||
|
||||
cmd_help() { show_help "${args[0]:-}"; }
|
||||
|
||||
|
|
@ -1015,6 +1102,15 @@ exposure_dev() { echo "expose-$1"; } # device name for a port
|
|||
cmd_expose() {
|
||||
local box="${args[0]}" a2="${args[1]:-}" a3="${args[2]:-}"
|
||||
|
||||
# Before ANY incus call: the door's plumbing (the box-isolate ACL, the
|
||||
# host firewall's route_localnet + masquerade) is daemon-global state a
|
||||
# restricted certificate cannot touch. Without this guard the failure is
|
||||
# a lie — box_net_ip cannot read boxnet's (redacted) config, so the
|
||||
# restricted user is told their running box "has no boxnet address yet".
|
||||
if [ "$(box_tier)" = restricted ]; then
|
||||
die "box expose edits the daemon-global ACL and firewall, which the restricted tier cannot modify — ask an incus-admin. (see #74)"
|
||||
fi
|
||||
|
||||
# --list
|
||||
if [ "$a2" = "--list" ]; then
|
||||
local found=0 d listen connect
|
||||
|
|
|
|||
|
|
@ -127,7 +127,71 @@ The rule that keeps this honest: **isolation claims are tested, never reasoned
|
|||
about.** The box→box hole existed because a plausible code reading said it could
|
||||
not. See `drill/RUNS.md`.
|
||||
|
||||
## Multi-user hosts: access tiers
|
||||
|
||||
The daemon socket is binary — `incus-admin` holds everything on the machine —
|
||||
so a shared host needs a second tier, and Incus ships one: **incus-user**
|
||||
confines an `incus`-group member to an auto-created project `user-<uid>`,
|
||||
behind a restricted certificate that cannot name any other project. The tier
|
||||
is decided once, from the process's live credentials (`box_tier()`: UID 0 or
|
||||
`incus-admin` → admin; `incus` alone → restricted; neither → none), and every
|
||||
tier-aware verb reads that one function.
|
||||
|
||||
What incus-user does *not* do is honor box's contract — measured on Debian 13
|
||||
/ Incus 6.0.4 (#74), after the design that assumed it (#72) was vetoed by its
|
||||
own Task-0 rehearsal:
|
||||
|
||||
- it pins each user's project to a private auto-created bridge
|
||||
(`incusbr-<uid>`) — a stock NAT bridge with **none** of the hardening: no
|
||||
ACL, no `dns.mode=none`, no resolver pin, IPv6 on;
|
||||
- it blocks snapshots — box's entire reuse workflow;
|
||||
- the `box-net` profile lives in the default project, invisible to theirs.
|
||||
|
||||
So the tier is an **admin-run convergence** (`box grant <user>`), not a
|
||||
group membership: put them in `incus`, touch incus-user once as them (the
|
||||
project is created lazily; nothing exists to converge until it does), then
|
||||
rewire the project — network access narrowed to `boxnet` **and only
|
||||
`boxnet`**, snapshots allowed, the shipped profile installed. Narrowing is
|
||||
the load-bearing decision: granting `boxnet,incusbr-<uid>` (the obvious fix)
|
||||
would leave an unhardened NAT bridge one `--network` flag away from any box
|
||||
they mint. With the private bridge unreferenced (its `eth0` is removed from
|
||||
their default profile) and outside `restricted.networks.access`, the hardened
|
||||
network is not their default placement — it is the only placement their
|
||||
certificate can express. The grant survives incus-user restarts by that
|
||||
tool's own design (it configures a project only at creation), and a restricted
|
||||
certificate cannot widen its own project — both measured, not read.
|
||||
|
||||
Cross-USER isolation is the same mechanism as cross-box isolation, on
|
||||
purpose: their instances share `boxnet` with everyone's, and the bridge-family
|
||||
drop + port isolation + `dns.mode=none` already make any two boxes strangers.
|
||||
A restricted user CAN strip `security.port_isolation` from the profile copy
|
||||
in their own project — or skip the profile entirely and attach `boxnet` raw
|
||||
(`--network boxnet`); the network must be usable for the profile to work, and
|
||||
Incus has no allow-via-profile-only lever. So the guarantee is scoped, and
|
||||
said plainly: **per-NIC port isolation is guaranteed for box-minted
|
||||
instances; a raw attachment keeps every network-owned control (the ACL,
|
||||
`dns.mode=none`, the resolver pin) and every host-owned one (the nft bridge
|
||||
drop) — losing only the redundant per-NIC L2 layer.** Scoped, and measured:
|
||||
`drill/multiuser.sh` criterion (m) launches exactly that raw instance and
|
||||
probes egress, RFC1918, both sibling directions and name enumeration from
|
||||
inside it. Defense in depth, every layer measured (criteria a–n).
|
||||
|
||||
`box revoke` is two strengths: bare, it removes the group — their boxes keep
|
||||
*running* (revoking a person does not kill their workloads), `grant` restores
|
||||
everything, and because supplementary groups are read at login, revoke warns
|
||||
when live sessions keep the socket until they end (and names the `loginctl`
|
||||
command). `--purge` terminates those sessions *first* — a stale-group process
|
||||
could otherwise touch incus-user after the purge and lazily recreate the
|
||||
project with stock, unhardened defaults, undoing the grant's whole point —
|
||||
then deletes their world (boxes, images, project, private bridge, trust-store
|
||||
certificate) and asserts the absence afterwards. A failed `grant` backs its
|
||||
own group-add out on exit for the same reason: no half-granted user holding
|
||||
an un-narrowed socket.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No unattended/CI bring-up — the flow is interactive.
|
||||
- Interactive-first: install and setup prompt by default (`BOX_YES=1` and the
|
||||
CI rehearsal job are the sanctioned unattended paths).
|
||||
- No credential storage or injection by the tool.
|
||||
- No per-user resource quotas on the restricted tier (Incus's
|
||||
`limits.*`/`restricted.*` project keys exist when someone needs them).
|
||||
|
|
|
|||
147
docs/plans/2026-07-18-restricted-tier.md
Normal file
147
docs/plans/2026-07-18-restricted-tier.md
Normal file
|
|
@ -0,0 +1,147 @@
|
|||
# Restricted incus tier — design and measured results (#74)
|
||||
|
|
||||
|
||||
**Status: implemented and rehearsed.** 54/54 rehearsal criteria green on the
|
||||
design host (Debian 13 trixie, Incus 6.0.4, nested KVM), in container mode
|
||||
and VM mode — and green in CI on ubuntu-latest / incus 6.0.0 (whose one
|
||||
version-drift false FAIL is MU-4 in `drill/RUNS.md`). This doc records the design, what was measured, and why each
|
||||
decision fell where it did. It supersedes the vetoed #72 design
|
||||
(`docs/plans/2026-07-17-multiuser-hosts.md` on `feat/restricted-tier-wip`).
|
||||
|
||||
## What #74 asked
|
||||
|
||||
A restricted (`incus`-group) user can `box new/list/shell/snapshot/rm` their
|
||||
own boxes, on a network carrying box's **full** isolation contract, seeing no
|
||||
one else's; the convergence is a documented, idempotent path; the rehearsal
|
||||
passes criteria (a)–(f); the admin tier is unchanged.
|
||||
|
||||
## The three facts that shaped the design
|
||||
|
||||
Task-0 (#72) measured one: incus-user confines users to `user-<uid>` projects
|
||||
(sound), but pins them to a private auto-created bridge `incusbr-<uid>` and
|
||||
`restricted.networks.access: incusbr-<uid>` — they cannot even see `boxnet`.
|
||||
|
||||
This round measured two more:
|
||||
|
||||
1. **The private bridge is worse than unhardened.** `incusbr-<uid>` is a
|
||||
fully functional NAT bridge — `ipv4.nat=true`, **`ipv6.nat=true`** — with
|
||||
no ACL, no `dns.mode=none`, no resolver pin, no port isolation, and IPv6
|
||||
egress box's contract explicitly forbids. Any instance placed on it holds
|
||||
a door to the host's LAN.
|
||||
2. **incus-user projects block snapshots** (`Project "user-<uid>" doesn't
|
||||
allow for snapshot creation`) — box's entire reuse workflow.
|
||||
|
||||
And two open questions from #74, answered from incus-user's own source
|
||||
(`cmd/incus-user/server.go`, stable-6.0) and then confirmed live:
|
||||
|
||||
- **Daemon-level project template?** None exists — the project config is
|
||||
hardcoded in `serverSetupUser()`. A per-user admin hook is the only path.
|
||||
- **Does widening survive a re-sync?** Yes. Setup runs only when the project
|
||||
does not exist (and early-outs when the user's certificate is already
|
||||
trusted); incus-user never rewrites an existing project. Confirmed live:
|
||||
`systemctl restart incus-user.socket` leaves the convergence intact.
|
||||
|
||||
## The decision: option 1, tightened
|
||||
|
||||
#74 offered (1) converge users onto the shared `boxnet` or (2) harden each
|
||||
private bridge. Option 2 multiplies every mechanism per user (ACL, resolver
|
||||
pin, dnsmasq, nft rules, firewall coexistence) and turns the shipped static
|
||||
profile into N generated ones. Option 1 keeps one hardened network and one
|
||||
shipped profile — and the existing box↔box mechanisms already make
|
||||
cross-user isolation free: the nft bridge-family drop and `dns.mode=none`
|
||||
are host/network-owned, so they bind every instance on `boxnet` no matter
|
||||
whose project it lives in.
|
||||
|
||||
One tightening beyond the issue's sketch: the issue proposed
|
||||
`restricted.networks.access boxnet,incusbr-<uid>` ("must list both" — true
|
||||
as long as the default profile still references the private bridge). Listing
|
||||
both leaves fact 1's unhardened bridge one `--network` flag away, forever.
|
||||
Instead, `box grant`:
|
||||
|
||||
- removes `eth0` from the project's default profile (nothing references the
|
||||
private bridge anymore, so the narrowing validates), and
|
||||
- sets `restricted.networks.access boxnet` — **only**.
|
||||
|
||||
After which the hardened network is not the user's default placement but the
|
||||
only placement their certificate can express. Measured: `incus launch
|
||||
--network incusbr-<uid>` as the user → `Network not found`; the user cannot
|
||||
widen their own project (`Error: Certificate is restricted`); they cannot
|
||||
touch `boxnet`'s config or the ACL (`no permission for project "default"`).
|
||||
|
||||
A restricted user CAN edit the `box-net` profile copy in their own project
|
||||
(they own project profiles — `features.profiles=true`), including stripping
|
||||
`security.port_isolation` — and CAN attach `boxnet` raw with `--network
|
||||
boxnet`, no profile at all (the network must be in
|
||||
`restricted.networks.access` for the profile to work; there is no
|
||||
allow-via-profile-only lever). That is why the host-owned nft bridge drop is
|
||||
the second layer: `meta ibrname boxnet obrname boxnet drop` fires on every
|
||||
port-to-port frame regardless of per-NIC flags. The documented guarantee is
|
||||
scoped accordingly (see box-design.md): box-minted instances carry per-NIC
|
||||
port isolation; raw attachments keep every network- and host-owned control,
|
||||
losing only that redundant L2 layer. Both shapes are measured from inside
|
||||
the instances (rehearsal criteria g and m).
|
||||
|
||||
Two grant-failure contracts, both injected in the rehearsal (criterion n):
|
||||
a fresh user is backed out of the group with the removal VERIFIED against
|
||||
the live group database (and any session begun mid-grant is named, with the
|
||||
loginctl remedy — the one window the database cannot close); a pre-existing
|
||||
member is never stripped by a failed re-grant, but the failure states out
|
||||
loud that they retain socket access on part-converged policy, with both
|
||||
remediations. The default-profile eth0 removal is deliberately NOT restored
|
||||
on failure: that mutation only reduces capability, and restoring it would
|
||||
move the failure state away from fail-closed. Every step is check-then-
|
||||
converge, which is what makes re-run-to-repair deterministic.
|
||||
|
||||
## What `box grant <user>` converges (idempotent, re-run to refresh)
|
||||
|
||||
1. `usermod -aG incus` (not `incus-admin` — that is the tier)
|
||||
2. first-touch incus-user as the user (`runuser`/`sudo -u`, stdin pinned) —
|
||||
the project is created lazily and cannot be pre-created by an admin
|
||||
3. remove the default profile's private-bridge `eth0`
|
||||
4. `restricted.networks.access boxnet`
|
||||
5. `restricted.snapshots allow`
|
||||
6. install/refresh the shipped `box-net` profile into the project
|
||||
7. verify from the user's side of the socket
|
||||
|
||||
`box revoke <user>` is the inverse, two strengths: bare = group removal (the
|
||||
socket closes; boxes keep running; re-grant restores), `--purge` = boxes,
|
||||
images, project, private bridge, trust-store certificate, incus-user state —
|
||||
then asserts the absence (the wipe.sh discipline).
|
||||
|
||||
## The tier in the CLI
|
||||
|
||||
`box_tier()` — UID 0 / `incus-admin` → admin, `incus` alone → restricted,
|
||||
neither → none — decided from live process credentials (argless `id -nG`),
|
||||
byte-identical in `bin/box` and `host/setup-host.sh` (diffed by a test).
|
||||
Tier-aware surface: `new` pre-flights the profile and names the right fix per
|
||||
tier; `expose` refuses before any daemon call (its plumbing is daemon-global;
|
||||
without the guard the failure is a lie — a restricted user cannot read
|
||||
boxnet's redacted config, so box_net_ip would claim their running box has no
|
||||
address); `setup-host` exits 0 with the honest note; `doctor` runs a
|
||||
restricted check-set (is the tier granted, does their box resolve/route)
|
||||
instead of judging host state they cannot see.
|
||||
|
||||
Found along the way, fixed for every tier: `box restore` dispatched
|
||||
`incus restore`, which does not exist in Incus 6 (`incus snapshot restore`).
|
||||
It had never worked.
|
||||
|
||||
## Rehearsal and CI
|
||||
|
||||
`drill/multiuser.sh` (root, opt-in via `BOX_MULTIUSER_REHEARSAL=1`) proves
|
||||
criteria (a)–(f) from #74 plus the measured extensions (g)–(n): the in-box
|
||||
isolation contract (egress, DNS, box→host, RFC1918, cross-user sibling drop,
|
||||
name enumeration, IPv6-off), the closed escape hatches, re-sync survival, and
|
||||
scoped revoke, the raw-attach scoped guarantee (m) and the grant-failure
|
||||
injections (n). Real users, real grants, real mints, probes from inside;
|
||||
`--container` for CI, VM mode on real hardware; cleanup deletes everything it
|
||||
made.
|
||||
|
||||
CI gains a `rehearsal` job (ubuntu-latest): install incus, stage the checkout
|
||||
at `/opt/box` (the #71 layout), `setup-host`, `doctor`, then the rehearsal in
|
||||
container mode. The tier's semantics are proven on every PR against a live
|
||||
daemon; the VM trust boundary stays a real-hardware ritual, like the drill.
|
||||
|
||||
## Environment
|
||||
|
||||
Debian 13 (trixie), Incus 6.0.4, `incus-user.socket` shipped in the incus
|
||||
package (Debian 13 and Ubuntu 24.04 both), `/dev/kvm` present (VM-mode run),
|
||||
btrfs pool. Companion work: rig#24 (`box` role), rig#12/#25 (host-class).
|
||||
|
|
@ -91,3 +91,27 @@ installed and runnable; authenticating is yours.
|
|||
If the host has no `/dev/kvm`, box falls back to container mode. The drill
|
||||
still runs, but it says loudly that **the VM trust boundary was not validated**
|
||||
rather than passing quietly on a weaker one.
|
||||
|
||||
## The multi-user rehearsal (`multiuser.sh`)
|
||||
|
||||
The restricted tier (#74) has its own rehearsal — the drill proves one
|
||||
operator's host; this proves a *shared* one:
|
||||
|
||||
```sh
|
||||
sudo BOX_MULTIUSER_REHEARSAL=1 bash drill/multiuser.sh --yes
|
||||
```
|
||||
|
||||
Root only, opt-in twice (it creates system users and edits the group
|
||||
database). It creates two throwaway users, grants them the tier through the
|
||||
real `box grant`, mints real boxes as them, and measures — from inside those
|
||||
boxes — that each user is confined to their own project (a), the full
|
||||
lifecycle works (b), no cross-user visibility (c), names don't collide (d),
|
||||
`expose`/`setup-host`/`doctor` answer honestly at the tier (e/f), the boxes
|
||||
ride `boxnet` under the full isolation contract including the cross-user
|
||||
sibling drop (g), the private-bridge escape hatches are closed (h), the grant
|
||||
survives an incus-user restart (k), and `revoke --purge` erases one user
|
||||
without touching the other (l). Everything it makes, it deletes.
|
||||
|
||||
`--container` skips VM mints (CI runs it this way on every PR — the tier's
|
||||
semantics are instance-type-independent); on real hardware run it bare so the
|
||||
boxes are VMs. `--keep` leaves the users and boxes up for inspection.
|
||||
|
|
|
|||
|
|
@ -195,6 +195,11 @@ No listener is needed, and none should be started: see trap 3.
|
|||
|
||||
| Run | Result | What it cost |
|
||||
| --- | --- | --- |
|
||||
| MU-5 | 53/1 → 54/0 (containers) | the #75 review round, measured in: criterion (m) — a raw `--network boxnet` attach keeps egress, the ACL, both sibling-drop directions and dns.mode=none, losing only per-NIC port_isolation (the scoped guarantee, now measured, not prose) — and criterion (n), both grant-failure injections (fresh user backed out with the removal VERIFIED; pre-existing member warned loudly, never stripped; re-runs converge). The one instrument bug: (m) first minted the non-cloud image — no DHCP client, no lease, and a dead NIC passes every negative probe vacuously. The probe now requires the lease before it believes any of its own answers |
|
||||
| MU-4 | 41/1 on CI (ubuntu-latest, incus 6.0.0) → 42/0 | the CI job earned its keep on its second-ever run: the attach-refusal probe matched 6.0.4's error wording and 6.0.0 words it differently (and refuses at a different stage) — a version-drift false FAIL the design host could never show. The probe now asserts the OUTCOME (nothing running on the private bridge) and prints the refusal as evidence. Also this round, from adversarial review: revoke survives the live-session case (groups are read at login; --purge now terminates sessions first, else a stale-group process could recreate the project UNHARDENED after the purge), grant backs out its own group-add on failure, and the session-dies-with-the-tier check is criterion 42 |
|
||||
| **MU-3** | **41/0 (containers) + 41/0 (VMs) — the tier holds** | `multiuser.sh` (#74) from zero on Debian 13 / Incus 6.0.4: grant, confinement, lifecycle, the in-box contract with the cross-user sibling drop, closed escape hatches, re-sync survival, scoped revoke. The VM pass needed right-sized boxes (1GiB): two 2GiB nested VMs on a 7.7GiB host measured swap, not isolation |
|
||||
| MU-2 | 40/41 (VMs) | the egress probe cried broken over a TLS handshake that outlived its 8s budget under nested-KVM memory pressure — every isolation probe (which PASSES on silence) was immune. Probe budgets raised, reachable-probes retry once, and the mint keeps its log for the post-mortem |
|
||||
| MU-1 | 39/41 (containers) | both FAILs were the instrument: a `box new` rc swallowed with its narration (the drill's oldest sin, relearned), and the enumeration probe resolving the box's OWN name — /etc/hosts answered, not the gateway. Probe a SIBLING's name or you measure cloud-init's hostname hygiene. Found for real along the way: `box restore` dispatched `incus restore`, which does not exist in Incus 6 — broken for every tier since the rename |
|
||||
| 1 | hung at C7; ~9 false FAILs | traps 1, 2, 4 — pipefail, the exec-pty hang, the DHCP race |
|
||||
| 2 | 42/49 | trap 5 — an interrupted run 1 left boxes behind, cascading 5 FAILs. Found the `claude`-on-PATH bug. Phase D delivered B1 (`@internal` rejected) and a B3 reading of *broken* |
|
||||
| 3 | 48/49 | trap 4 — `eth0_ip` never matched, so A3 again unprobed. B3 now read *intact*, contradicting run 2 |
|
||||
|
|
|
|||
|
|
@ -45,6 +45,65 @@ timeout 10 incus list >/dev/null 2>&1 || {
|
|||
exit 1
|
||||
}
|
||||
|
||||
# The tier changes what this doctor can SEE, so it changes what it may JUDGE.
|
||||
# bin/box exports BOX_TIER; unset means a hand-run, which was always admin.
|
||||
# A restricted (incus-group) user cannot read the nft tables, the kernel's
|
||||
# bridge ports, or boxnet's (redacted) config — reporting those as DIRTY
|
||||
# would blame the host for the reader's own, correct, confinement. They get
|
||||
# the checks that are theirs: is the tier granted, is the contract in their
|
||||
# project, do their boxes actually resolve and route.
|
||||
TIER="${BOX_TIER:-admin}"
|
||||
if [ "$TIER" = restricted ]; then
|
||||
head_ "Access tier — restricted (the incus group: your own boxes, nothing else)"
|
||||
inf "the host stack (network, ACL, firewall, kernel state) is admin-owned;"
|
||||
inf "this doctor judges only what is yours to see"
|
||||
if [ "$FIX" = 1 ] || [ "$PIN" = 1 ]; then
|
||||
inf "--fix / --pin-dns are admin levers — ignored on this tier"
|
||||
FIX=0; PIN=0
|
||||
fi
|
||||
|
||||
head_ "Your project — is the tier granted?"
|
||||
if incus profile show box-net >/dev/null 2>&1 </dev/null; then
|
||||
ok "the box-net profile is in your project — 'box new' lands on the hardened boxnet"
|
||||
iso="$(incus profile device get box-net eth0 security.port_isolation </dev/null 2>/dev/null)"
|
||||
[ "$iso" = "true" ] \
|
||||
&& ok "security.port_isolation = true (as shipped)" \
|
||||
|| no "security.port_isolation is NOT set in your box-net profile — re-grant refreshes it: ask an admin to re-run 'box grant $(id -un)'"
|
||||
else
|
||||
no "no box-net profile in your project — the restricted tier is granted per user"
|
||||
inf "fix: an admin runs: box grant $(id -un)"
|
||||
fi
|
||||
if incus network show boxnet >/dev/null 2>&1 </dev/null; then
|
||||
ok "boxnet is reachable from your project"
|
||||
else
|
||||
no "boxnet is not visible from your project — ask an admin to re-run 'box grant $(id -un)'"
|
||||
fi
|
||||
|
||||
head_ "Can one of your boxes actually resolve DNS?"
|
||||
probe="$({ incus list "user.box=1" --format csv --columns ns 2>/dev/null
|
||||
incus list "user.claudebox=1" --format csv --columns ns 2>/dev/null; } \
|
||||
| awk -F, '$2 == "RUNNING" { print $1; exit }')"
|
||||
if [ -n "$probe" ]; then
|
||||
inf "probing inside '$probe':"
|
||||
timeout -k 5 25 incus exec "$probe" -- curl -sS -m 10 -o /dev/null https://1.1.1.1 </dev/null 2>/dev/null \
|
||||
&& ok "reaches 1.1.1.1 by address — egress routing is fine" \
|
||||
|| no "cannot reach 1.1.1.1 by address — egress routing is broken (an admin problem: box doctor as admin)"
|
||||
timeout -k 5 25 incus exec "$probe" -- getent hosts deb.debian.org </dev/null >/dev/null 2>&1 \
|
||||
&& ok "resolves deb.debian.org — DNS works" \
|
||||
|| no "CANNOT resolve deb.debian.org — an admin problem (the resolver pin lives on the host): box doctor as admin"
|
||||
else
|
||||
inf "no running box to probe with (mint one: box new --name work)"
|
||||
fi
|
||||
|
||||
head_ "Verdict"
|
||||
if [ "$bad" -eq 0 ]; then
|
||||
printf ' \033[32mclean\033[0m — your tier is granted and your boxes are fit.\n\n'
|
||||
exit 0
|
||||
fi
|
||||
printf ' \033[31m%s problem(s)\033[0m — see the fixes above (most need an admin).\n\n' "$bad"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# A FRESH host (no boxnet) is not a DIRTY one. Everything below that would
|
||||
# scream about a missing piece must first ask: missing from a stack, or never
|
||||
# set up? setup-host creates all of it, and the drill runs setup-host itself —
|
||||
|
|
@ -191,6 +250,15 @@ for b in drill clone archive peer payroll cbprobe cbcopy cbnotours; do
|
|||
[ "$FIX" = 1 ] && { timeout 60 incus delete -f "$b" >/dev/null 2>&1 && inf "reverted: deleted $b"; }
|
||||
fi
|
||||
done
|
||||
# An interrupted multiuser.sh leaves its users (and their projects) behind —
|
||||
# and nothing else on this host will ever mention them. Its own cleanup is
|
||||
# 'box revoke --purge + userdel'; say so rather than absorbing them silently.
|
||||
for u in boxdrill1 boxdrill2 boxdrill3 boxdrill4; do
|
||||
if getent passwd "$u" >/dev/null 2>&1; then
|
||||
no "leftover rehearsal user: $u (an interrupted drill/multiuser.sh run)"
|
||||
inf "fix: sudo BOX_YES=1 box revoke $u --purge && sudo userdel -r $u"
|
||||
fi
|
||||
done
|
||||
|
||||
# --- the box's DNS comes from the HOST's resolver. See issue #33. -----------
|
||||
head_ "Host resolver — a box's DNS is forwarded through this"
|
||||
|
|
|
|||
472
drill/multiuser.sh
Normal file
472
drill/multiuser.sh
Normal file
|
|
@ -0,0 +1,472 @@
|
|||
#!/usr/bin/env bash
|
||||
# multiuser.sh — the restricted-tier rehearsal (#74): does 'box grant' give a
|
||||
# plain incus-group user their own boxes, on the hardened boxnet, with no view
|
||||
# of anyone else's — measured live, from both sides of the socket?
|
||||
#
|
||||
# sudo BOX_MULTIUSER_REHEARSAL=1 bash drill/multiuser.sh --yes
|
||||
#
|
||||
# It creates two throwaway system users, grants them the tier through the real
|
||||
# 'box grant', mints real boxes as them, probes the isolation contract from
|
||||
# INSIDE those boxes, revokes one user, and deletes everything it made. Root
|
||||
# only, and opt-in twice (the env gate and the prompt): it edits the group
|
||||
# database and the daemon's project list, which is nothing to do casually.
|
||||
#
|
||||
# Criteria (a)-(f) are issue #74's acceptance gate, kept under their letters;
|
||||
# (g)-(l) are what the Task-0 findings added: the network CONTRACT is the
|
||||
# tier's whole point, so it is measured, never assumed.
|
||||
#
|
||||
# a. an incus-group user is auto-confined to their own project
|
||||
# b. they can box new / list / exec / snapshot / restore / clone their own box
|
||||
# c. they cannot see another user's boxes
|
||||
# d. the same box name in two projects does not collide
|
||||
# e. box expose refuses for them (daemon-global state)
|
||||
# f. box setup-host / box doctor answer with the honest restricted note
|
||||
# g. their boxes ride boxnet and carry the full isolation contract
|
||||
# h. the private incusbr-<uid> escape hatch is closed (boxnet is the ONLY network)
|
||||
# i. (folded into b: snapshot / restore / clone)
|
||||
# k. the grant survives an incus-user restart
|
||||
# l. box revoke --purge removes the user's world and touches nobody else's
|
||||
# m. a RAW attach to boxnet (no box-net profile) keeps every network- and
|
||||
# host-owned control — the scoped guarantee, measured (#75 review)
|
||||
# n. a grant that fails is fail-closed: fresh user backed out (verified),
|
||||
# pre-existing member warned loudly, re-run converges (#75 review)
|
||||
#
|
||||
# ok/no/note return 0 by design — the 'A && ok || no' idiom below is the
|
||||
# same one drill.sh is built on (and the reason for the SC2015 disable).
|
||||
# shellcheck disable=SC2015
|
||||
set -u
|
||||
|
||||
YES=0; KEEP=0; MODE=()
|
||||
for a in "$@"; do
|
||||
case "$a" in
|
||||
--yes) YES=1 ;;
|
||||
--keep) KEEP=1 ;;
|
||||
--container) MODE=(--container) ;; # CI / kvm-less hosts: don't wait on a VM
|
||||
*) echo "usage: sudo BOX_MULTIUSER_REHEARSAL=1 bash drill/multiuser.sh [--yes] [--container] [--keep]" >&2; exit 2 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
[ "${BOX_MULTIUSER_REHEARSAL:-}" = 1 ] || {
|
||||
echo "multiuser.sh: this rehearsal creates system users and edits the group database." >&2
|
||||
echo "opt in explicitly: sudo BOX_MULTIUSER_REHEARSAL=1 bash drill/multiuser.sh --yes" >&2
|
||||
exit 2
|
||||
}
|
||||
[ "$(id -u)" -eq 0 ] || { echo "multiuser.sh: root only (it creates users and grants tiers)." >&2; exit 1; }
|
||||
command -v box >/dev/null || { echo "multiuser.sh: no 'box' on PATH — the tier needs the global install (#71): sudo bash install.sh" >&2; exit 1; }
|
||||
command -v incus >/dev/null || { echo "multiuser.sh: incus is not installed — box setup-host first." >&2; exit 1; }
|
||||
incus network show boxnet >/dev/null 2>&1 || { echo "multiuser.sh: no boxnet — box setup-host first." >&2; exit 1; }
|
||||
|
||||
if [ "$YES" -ne 1 ]; then
|
||||
[ -t 0 ] || { echo "multiuser.sh: no terminal to confirm on — pass --yes." >&2; exit 2; }
|
||||
printf 'multiuser.sh: create users %s/%s, grant them the tier, mint boxes as them, then delete it all? [y/N] ' boxdrill1 boxdrill2
|
||||
read -r reply
|
||||
case "$reply" in y|Y|yes|YES|Yes) : ;; *) echo "aborted."; exit 1 ;; esac
|
||||
fi
|
||||
|
||||
pass=0; fail=0; findings=(); audit=()
|
||||
ok() { printf ' \033[32mPASS\033[0m %s\n' "$*"; pass=$((pass+1)); }
|
||||
no() { printf ' \033[31mFAIL\033[0m %s\n' "$*"; fail=$((fail+1)); findings+=("FAIL: $*"); }
|
||||
note() { printf ' \033[33mNOTE\033[0m %s\n' "$*"; findings+=("NOTE: $*"); }
|
||||
inf() { printf ' %s\n' "$*"; }
|
||||
phase(){ printf '\n\033[1m══ %s\033[0m\n' "$*"; }
|
||||
aud() { audit+=("$*"); }
|
||||
|
||||
U1=boxdrill1; U2=boxdrill2
|
||||
|
||||
# Run as a rehearsal user. runuser resets HOME/USER to the target (we are
|
||||
# root), and stdin is pinned: an incus with a terminal can go interactive and
|
||||
# wedge the run forever — the drill's oldest trap, honored here.
|
||||
as_u() { local u="$1"; shift; runuser -u "$u" -- "$@" </dev/null; }
|
||||
|
||||
# Probe a TCP door from INSIDE a box and answer reachable/refused/dropped by
|
||||
# reading curl's MESSAGE, never its exit code (drill.sh's box_probe, scoped
|
||||
# down): "refused" means a packet ARRIVED and was answered — which, for an
|
||||
# isolation probe, is a failure wearing polite clothes. Silence is the pass.
|
||||
probe_from() { # probe_from <user> <box> <url>
|
||||
local u="$1" b="$2" url="$3" out rc
|
||||
out="$(as_u "$u" timeout -k 5 30 incus exec "$b" -- curl -sS -m 15 -o /dev/null "$url" 2>&1)"; rc=$?
|
||||
# Silence + success is the only 'reachable': when the OUTER timeout kills a
|
||||
# wedged exec (the #26 shape), curl never spoke — empty output with rc 124
|
||||
# must not read as an open door.
|
||||
case "$rc:$out" in
|
||||
0:) echo reachable ;;
|
||||
*Connection\ refused*) echo refused ;;
|
||||
*) echo dropped ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# For probes whose PASS is "reachable": one retry. A nested-virt host can
|
||||
# blow a first TLS handshake on timing alone, and a rehearsal that cries
|
||||
# broken-egress over that teaches people to ignore it. Isolation probes never
|
||||
# retry — for them silence is the pass, and silence is not flaky.
|
||||
probe_up() { # probe_up <user> <box> <url>
|
||||
local r; r="$(probe_from "$@")"
|
||||
[ "$r" = reachable ] || r="$(probe_from "$@")"
|
||||
echo "$r"
|
||||
}
|
||||
|
||||
cleanup() {
|
||||
[ "$KEEP" = 1 ] && { echo "(--keep: users and boxes left for inspection)"; return; }
|
||||
echo
|
||||
echo "── cleanup"
|
||||
for u in "$U1" "$U2" boxdrill3 boxdrill4; do
|
||||
id "$u" >/dev/null 2>&1 || continue
|
||||
# A half-failed purge followed by userdel leaves a project owned by
|
||||
# nobody — and doctor's leftover check keys on the USER existing. Keep
|
||||
# the user when the purge fails, and name what survived.
|
||||
if BOX_YES=1 box revoke "$u" --purge >/dev/null 2>&1; then
|
||||
userdel -r "$u" >/dev/null 2>&1
|
||||
id "$u" >/dev/null 2>&1 && echo " WARNING: user $u still exists" || echo " removed $u (tier, boxes, account)"
|
||||
else
|
||||
echo " WARNING: purge FAILED for $u — kept the account so 'box doctor' can name it; project user-$(id -u "$u") may survive"
|
||||
fi
|
||||
done
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
phase "G. the grant — box grant is the convergence, and it converges"
|
||||
for u in "$U1" "$U2"; do
|
||||
id "$u" >/dev/null 2>&1 || useradd -m -s /bin/bash "$u"
|
||||
done
|
||||
# Before any grant: a bare user has NO tier — the socket refuses them. This is
|
||||
# the baseline that makes everything after it meaningful.
|
||||
as_u "$U1" incus list >/dev/null 2>&1 \
|
||||
&& no "(a) $U1 reached the daemon BEFORE any grant — the socket is not confining" \
|
||||
|| ok "(a) before the grant, $U1 cannot reach the daemon at all"
|
||||
|
||||
box grant "$U1" >/dev/null 2>&1 && ok "box grant $U1 exits 0" || no "box grant $U1 failed"
|
||||
box grant "$U2" >/dev/null 2>&1 && ok "box grant $U2 exits 0" || no "box grant $U2 failed"
|
||||
box grant "$U1" >/dev/null 2>&1 && ok "box grant is idempotent (re-run exits 0)" || no "box grant re-run failed"
|
||||
|
||||
uid1="$(id -u "$U1")"; uid2="$(id -u "$U2")"
|
||||
p1="user-$uid1"; p2="user-$uid2"
|
||||
|
||||
acc="$(incus project get "$p1" restricted.networks.access 2>/dev/null)"
|
||||
[ "$acc" = boxnet ] \
|
||||
&& ok "(h) $p1 is restricted to boxnet and ONLY boxnet" \
|
||||
|| no "(h) $p1 restricted.networks.access = '$acc' — the unhardened private bridge is still reachable"
|
||||
aud "h. restricted.networks.access after grant: '$acc' (the private incusbr-$uid1 is unreferenced)"
|
||||
|
||||
snaps="$(incus project get "$p1" restricted.snapshots 2>/dev/null)"
|
||||
[ "$snaps" = allow ] && ok "snapshots allowed in $p1 (the clone workflow exists)" \
|
||||
|| no "restricted.snapshots = '$snaps' — box snapshot will refuse"
|
||||
|
||||
incus --project "$p1" profile device get default eth0 type >/dev/null 2>&1 \
|
||||
&& no "(h) $p1's default profile still carries the private-bridge eth0" \
|
||||
|| ok "(h) $p1's default profile places no network — box-net is the only door"
|
||||
|
||||
iso="$(incus --project "$p1" profile device get box-net eth0 security.port_isolation 2>/dev/null)"
|
||||
[ "$iso" = true ] && ok "box-net profile is in $p1, port_isolation true" \
|
||||
|| no "box-net profile in $p1 is wrong (port_isolation='$iso')"
|
||||
|
||||
phase "a. confinement — each user lands in their own project, and only theirs"
|
||||
projects="$(as_u "$U1" incus project list --format csv 2>/dev/null | cut -d, -f1)"
|
||||
if [ "$(printf '%s\n' "$projects" | grep -c .)" = 1 ] && printf '%s' "$projects" | grep -q "$p1"; then
|
||||
ok "(a) $U1 sees exactly one project: their own ($p1)"
|
||||
else
|
||||
no "(a) $U1 sees: $(printf '%s' "$projects" | tr '\n' ' ') — confinement LEAKED, which vetoes the tier"
|
||||
fi
|
||||
as_u "$U1" incus list --project default >/dev/null 2>&1 \
|
||||
&& no "(a) $U1 can list the DEFAULT project — admin boxes are visible" \
|
||||
|| ok "(a) the default project refuses $U1"
|
||||
aud "a. incus-user confines: project list as $U1 = '$(printf '%s' "$projects" | tr '\n' ' ')'"
|
||||
|
||||
phase "b. the lifecycle, as a restricted user — new/list/exec/snapshot/restore/clone/rm"
|
||||
# The mint's narration is kept and shown on failure — a FAIL that names
|
||||
# nothing is the drill's oldest sin.
|
||||
mintlog="$(mktemp)"
|
||||
# 1GiB / 2 cpus: a blank Debian needs no more, and the rehearsal runs TWO
|
||||
# boxes at once — on a small (or nested) rehearsal host, 2GiB apiece is the
|
||||
# difference between measuring isolation and measuring swap.
|
||||
if as_u "$U1" box new --name mine --template blank --cpu 2 --memory 1GiB "${MODE[@]}" >"$mintlog" 2>&1; then
|
||||
ok "(b) box new mine — minted"
|
||||
else
|
||||
no "(b) box new failed for $U1 (rc≠0) — its last words:"
|
||||
grep -v '^\.*$' "$mintlog" | tail -6 | sed 's/^/ /'
|
||||
fi
|
||||
rm -f "$mintlog"
|
||||
as_u "$U1" box list 2>/dev/null | grep -q '^mine ' && ok "(b) box list shows mine" || no "(b) box list does not show mine"
|
||||
as_u "$U1" box exec mine -- true >/dev/null 2>&1 && ok "(b) box exec mine -- true" || no "(b) box exec failed"
|
||||
as_u "$U1" box info mine 2>/dev/null | grep -q '10\.88\.' \
|
||||
&& ok "(g) box info shows a boxnet (10.88.x) address — placed on the hardened network" \
|
||||
|| no "(g) mine has no boxnet address in box info"
|
||||
as_u "$U1" box snapshot mine s1 >/dev/null 2>&1 && ok "(b) box snapshot mine s1" || no "(b) snapshot refused"
|
||||
as_u "$U1" box restore mine s1 >/dev/null 2>&1 && ok "(b) box restore mine s1 (the incus 6 'snapshot restore' spelling)" || no "(b) restore failed"
|
||||
as_u "$U1" box new --name c1 --from mine/s1 >/dev/null 2>&1 && ok "(b) box new --from mine/s1 — the clone workflow" || no "(b) clone failed"
|
||||
# c1 stays alive through phase g: it is the distinctly-NAMED sibling the
|
||||
# enumeration probe needs (both users' primaries are 'mine' by design of d).
|
||||
aud "b. full lifecycle exercised as $U1 through the real CLI"
|
||||
|
||||
phase "d. same name, two users — projects mean no collision"
|
||||
as_u "$U2" box new --name mine --template blank --cpu 2 --memory 1GiB "${MODE[@]}" >/dev/null 2>&1 \
|
||||
&& ok "(d) $U2 minted their own 'mine' beside $U1's" \
|
||||
|| no "(d) $U2 could not mint 'mine' — names collide across users"
|
||||
|
||||
phase "c. cross-visibility — each sees exactly their own"
|
||||
n1="$(as_u "$U1" box list 2>/dev/null | grep -c '^mine ')"
|
||||
n2="$(as_u "$U2" box list 2>/dev/null | grep -c '^mine ')"
|
||||
[ "$n1" = 1 ] && [ "$n2" = 1 ] \
|
||||
&& ok "(c) both users see exactly one 'mine' — their own" \
|
||||
|| no "(c) visibility leaked: $U1 sees $n1, $U2 sees $n2 — vetoes the tier"
|
||||
as_u "$U1" incus list --project "$p2" >/dev/null 2>&1 \
|
||||
&& no "(c) $U1 can list $U2's project" \
|
||||
|| ok "(c) $U2's project refuses $U1"
|
||||
aud "c. cross-user visibility: $U1=$n1 'mine', $U2=$n2 'mine', foreign project listing refused"
|
||||
|
||||
phase "g. the isolation contract, measured from INSIDE the boxes"
|
||||
ip1="$(incus --project "$p1" list mine --format csv --columns 4 2>/dev/null | tr -d '"' | sed 's/ (.*//' | head -n1)"
|
||||
ip2="$(incus --project "$p2" list mine --format csv --columns 4 2>/dev/null | tr -d '"' | sed 's/ (.*//' | head -n1)"
|
||||
inf "$U1's mine: ${ip1:-<no ip>} $U2's mine: ${ip2:-<no ip>}"
|
||||
case "$ip1" in 10.88.*) ok "(g) $U1's box holds a boxnet lease" ;; *) no "(g) $U1's box is NOT on boxnet: '$ip1'" ;; esac
|
||||
|
||||
r="$(probe_up "$U1" mine https://1.1.1.1)"
|
||||
[ "$r" = reachable ] && ok "(g) egress to the public internet works (curl 1.1.1.1: $r)" || no "(g) public egress broken: $r"
|
||||
as_u "$U1" timeout -k 5 20 incus exec mine -- getent hosts deb.debian.org >/dev/null 2>&1 \
|
||||
&& ok "(g) public DNS resolves (via the pinned resolver)" || no "(g) DNS broken inside the box"
|
||||
|
||||
r="$(probe_from "$U1" mine "http://10.88.0.1:22")"
|
||||
[ "$r" = dropped ] && ok "(g) box → host is dropped (gateway :22: $r)" || no "(g) box can reach the HOST: $r"
|
||||
r="$(probe_from "$U1" mine "http://192.168.0.1")"
|
||||
[ "$r" = dropped ] && ok "(g) box → RFC1918 is dropped ($r)" || no "(g) box reaches private space: $r"
|
||||
|
||||
if [ -n "$ip2" ]; then
|
||||
r="$(probe_from "$U1" mine "http://$ip2:9")"
|
||||
[ "$r" = dropped ] && ok "(g) $U1's box → $U2's box is DROPPED (cross-user sibling isolation)" \
|
||||
|| no "(g) cross-user box→box answered ($r) — a packet crossed the user boundary"
|
||||
aud "g. cross-user sibling probe $ip1 → $ip2: $r (silence is the pass; 'refused' would mean arrival)"
|
||||
else
|
||||
note "(g) no ip for $U2's box — sibling probe skipped"
|
||||
fi
|
||||
|
||||
# Enumeration is probed with a SIBLING's name, never the box's own — a box
|
||||
# always resolves itself from /etc/hosts (cloud-init writes it), and reading
|
||||
# that as a gateway leak was this rehearsal's first false FAIL. c1 (U1's
|
||||
# clone, still alive) is the distinctly-named instance; U2's box asks.
|
||||
as_u "$U2" timeout -k 5 20 incus exec mine -- getent hosts c1 >/dev/null 2>&1 \
|
||||
&& no "(g) a box can resolve a sibling's name through the gateway (dns.mode leak)" \
|
||||
|| ok "(g) a sibling's name does not resolve (dns.mode=none holds for the tier)"
|
||||
as_u "$U1" box rm c1 --force >/dev/null 2>&1 && ok "(b) box rm c1" || no "(b) rm failed"
|
||||
v6="$(as_u "$U1" timeout -k 5 20 incus exec mine -- sh -c 'ip -6 addr show dev eth0 scope global 2>/dev/null' 2>/dev/null)"
|
||||
[ -z "$v6" ] && ok "(g) no global IPv6 inside the box (the IPv4-only contract)" \
|
||||
|| no "(g) the box holds a global IPv6 address — an uncovered egress path"
|
||||
|
||||
phase "h. the escape hatches, tried and refused"
|
||||
# Each probe asserts more than a nonzero exit — an image hiccup or a name
|
||||
# collision also exits nonzero, and reading that as "the escape is closed"
|
||||
# is a false verdict wearing a green light (the drill has relearned this
|
||||
# enough times to earn a rule). For the attach, the incus ERROR WORDING
|
||||
# drifts between 6.0.x releases (6.0.4 refuses before "Launching", 6.0.0
|
||||
# after — MU-4), so the assertion is the OUTCOME: nothing may end up running
|
||||
# on the private bridge, and the refusal line is printed as evidence.
|
||||
out="$(as_u "$U1" incus launch images:debian/13 esc --network "incusbr-$uid1" 2>&1)"; rc=$?
|
||||
st="$(incus --project "$p1" list esc --format csv --columns s 2>/dev/null | head -n1)"
|
||||
if [ "$rc" -eq 0 ] || [ "$st" = RUNNING ]; then
|
||||
no "(h) the private-bridge attach was NOT refused (rc=$rc, esc state: ${st:-none}):"
|
||||
printf '%s\n' "$out" | tail -3 | sed 's/^/ /'
|
||||
else
|
||||
ok "(h) attaching the private incusbr-$uid1 is refused (rc=$rc, nothing running on it)"
|
||||
inf "refusal: $(printf '%s\n' "$out" | grep -m1 -i 'error' || printf '%s\n' "$out" | tail -1)"
|
||||
fi
|
||||
as_u "$U1" incus delete -f esc >/dev/null 2>&1
|
||||
out="$(as_u "$U1" incus project set "$p1" restricted.networks.access "boxnet,incusbr-$uid1" 2>&1)"; rc=$?
|
||||
if [ "$rc" -ne 0 ] && printf '%s' "$out" | grep -qi 'restricted'; then
|
||||
ok "(h) a restricted certificate cannot widen its own project"
|
||||
else
|
||||
no "(h) project-widen attempt: rc=$rc, said: $(printf '%s' "$out" | head -1)"
|
||||
fi
|
||||
out="$(as_u "$U1" incus network set boxnet dns.mode=managed 2>&1)"; rc=$?
|
||||
if [ "$rc" -ne 0 ] && printf '%s' "$out" | grep -qiE 'permission|restricted'; then
|
||||
ok "(h) boxnet's config refuses a restricted certificate"
|
||||
else
|
||||
no "(h) boxnet edit attempt: rc=$rc, said: $(printf '%s' "$out" | head -1)"
|
||||
fi
|
||||
|
||||
phase "m. a raw attach to boxnet — the scoped guarantee, measured"
|
||||
# A restricted user CAN 'incus launch --network boxnet' without the box-net
|
||||
# profile: boxnet must be in restricted.networks.access for the profile to
|
||||
# work at all, and Incus has no allow-via-profile-only lever. What the raw
|
||||
# NIC loses is per-NIC security.port_isolation — the deliberately redundant
|
||||
# L2 twin of the host-owned nft bridge drop. Everything else binds to the
|
||||
# NETWORK (ACL, dns.mode=none, resolver pin) or the HOST (nft drop), so the
|
||||
# contract's claim for raw attachments is "every control except the
|
||||
# redundant per-NIC layer" — and a claim is a measurement here, not prose.
|
||||
# Same image the blank template mints (the /cloud variant): the plain image
|
||||
# has no DHCP client, so its raw instance holds NO lease — and against a
|
||||
# dead NIC every negative probe below "passes" vacuously while the contract
|
||||
# goes unmeasured. Caught on this criterion's first run (MU-5).
|
||||
rawout="$(as_u "$U1" incus launch images:debian/13/cloud esc2 --network boxnet 2>&1)"; rawrc=$?
|
||||
if [ "$rawrc" -ne 0 ]; then
|
||||
# Version fork, measured: 6.0.4 permits a restricted cert a raw --network
|
||||
# reference to an allowed network; 6.0.0 refuses it at the permission
|
||||
# layer. A refusal is not a broken probe — it is the STRONGEST of the
|
||||
# three resolutions (prevention): on such a daemon the bypass this
|
||||
# criterion measures cannot be expressed at all. Anything else (an image
|
||||
# error, a name collision) is a broken probe and says so with its log.
|
||||
if printf '%s' "$rawout" | grep -qiE 'permission|not allowed|restricted'; then
|
||||
ok "(m) raw attach to boxnet is REFUSED outright by this incus — prevention, the strongest resolution"
|
||||
inf "refusal: $(printf '%s\n' "$rawout" | grep -m1 -i 'error' || printf '%s\n' "$rawout" | tail -1)"
|
||||
aud "m. this incus version refuses raw --network for restricted certs; where permitted (6.0.4, MU-5) the raw NIC keeps every network- and host-owned control — both worlds measured"
|
||||
else
|
||||
no "(m) raw attach failed for a reason that is neither refusal nor success — unmeasured:"
|
||||
printf '%s\n' "$rawout" | tail -3 | sed 's/^/ /'
|
||||
fi
|
||||
else
|
||||
ok "(m) raw attach to boxnet launches (expected: the network must be usable for the profile to work)"
|
||||
ip_raw=""
|
||||
for _ in $(seq 1 45); do
|
||||
ip_raw="$(incus --project "$p1" list esc2 --format csv --columns 4 2>/dev/null | tr -d '"' | sed 's/ (.*//' | grep . | head -n1)"
|
||||
[ -n "$ip_raw" ] && as_u "$U1" timeout -k 5 15 incus exec esc2 -- true >/dev/null 2>&1 && break
|
||||
sleep 2
|
||||
done
|
||||
inf "raw instance esc2: ${ip_raw:-<no ip>}"
|
||||
if [ -z "$ip_raw" ]; then
|
||||
# Without an address the negative probes below would all pass vacuously
|
||||
# — a dead NIC drops everything, including the truth.
|
||||
no "(m) the raw instance never got a boxnet lease — the scoped guarantee went UNMEASURED"
|
||||
else
|
||||
r="$(probe_up "$U1" esc2 https://1.1.1.1)"
|
||||
[ "$r" = reachable ] && ok "(m) raw NIC: public egress works ($r)" || no "(m) raw NIC: egress broken: $r"
|
||||
r="$(probe_from "$U1" esc2 "http://192.168.0.1")"
|
||||
[ "$r" = dropped ] && ok "(m) raw NIC: RFC1918 still dropped (the ACL binds to the network, not the profile)" \
|
||||
|| no "(m) raw NIC: reaches private space ($r) — the ACL did not cover a raw attach"
|
||||
if [ -n "$ip2" ]; then
|
||||
r="$(probe_from "$U1" esc2 "http://$ip2:9")"
|
||||
[ "$r" = dropped ] && ok "(m) raw → another user's box is DROPPED (the nft drop is host-owned)" \
|
||||
|| no "(m) raw instance reached a sibling ($r) — the host drop did not cover it"
|
||||
fi
|
||||
r="$(probe_from "$U2" mine "http://$ip_raw:9")"
|
||||
[ "$r" = dropped ] && ok "(m) another user's box → raw is DROPPED (both directions hold)" \
|
||||
|| no "(m) a sibling reached the raw instance ($r)"
|
||||
as_u "$U1" timeout -k 5 20 incus exec esc2 -- getent hosts mine >/dev/null 2>&1 \
|
||||
&& no "(m) raw NIC can enumerate instance names (dns.mode leak)" \
|
||||
|| ok "(m) raw NIC: name enumeration still blocked (dns.mode=none is the network's)"
|
||||
fi
|
||||
as_u "$U1" incus delete -f esc2 >/dev/null 2>&1
|
||||
aud "m. raw boxnet attach keeps ACL + nft drop + dns.mode (measured); loses only per-NIC port_isolation — the scoped guarantee in box-design.md"
|
||||
fi
|
||||
|
||||
phase "e/f. the honest refusals — expose, setup-host, doctor"
|
||||
out="$(as_u "$U1" box expose mine 3000 2>&1)"; rc=$?
|
||||
[ "$rc" -ne 0 ] && printf '%s' "$out" | grep -qi restricted \
|
||||
&& ok "(e) box expose refuses with the restricted message (rc=$rc)" \
|
||||
|| no "(e) box expose: rc=$rc, said: $(printf '%s' "$out" | head -1)"
|
||||
out="$(as_u "$U1" box setup-host 2>&1)"; rc=$?
|
||||
[ "$rc" -eq 0 ] && printf '%s' "$out" | grep -q "restricted tier" \
|
||||
&& ok "(f) box setup-host: the honest note, exit 0" \
|
||||
|| no "(f) box setup-host as $U1: rc=$rc"
|
||||
out="$(as_u "$U1" box doctor 2>&1)"; rc=$?
|
||||
printf '%s' "$out" | grep -q "Access tier — restricted" \
|
||||
&& ok "(f) box doctor answers at the restricted tier (rc=$rc)" \
|
||||
|| no "(f) box doctor did not honor the tier (rc=$rc)"
|
||||
aud "e/f. expose refused; setup-host and doctor answer honestly at the tier"
|
||||
|
||||
phase "k. the grant survives an incus-user restart (the re-sync question, #74)"
|
||||
systemctl restart incus-user.socket 2>/dev/null
|
||||
sleep 1
|
||||
as_u "$U1" box list 2>/dev/null | grep -q '^mine ' \
|
||||
&& ok "(k) after restarting incus-user, $U1 still reaches their boxes" \
|
||||
|| no "(k) the restart broke $U1's access"
|
||||
acc="$(incus project get "$p1" restricted.networks.access 2>/dev/null)"
|
||||
[ "$acc" = boxnet ] && ok "(k) restricted.networks.access is still boxnet — nothing re-widened" \
|
||||
|| no "(k) the restart rewrote network access to '$acc'"
|
||||
aud "k. incus-user re-sync: convergence intact (matches its source: setup runs only at project creation)"
|
||||
|
||||
phase "l. revoke — one user out, the other untouched"
|
||||
# Revocation's hard case is a user who is LOGGED IN: groups are read at
|
||||
# login, so a held session keeps the socket — and after a purge it could
|
||||
# touch incus-user and recreate the project with stock, unhardened defaults.
|
||||
# Hold a session open across the purge and demand it dies with the tier.
|
||||
runuser -u "$U2" -- sleep 300 </dev/null >/dev/null 2>&1 &
|
||||
sleep 1
|
||||
BOX_YES=1 box revoke "$U2" --purge >/dev/null 2>&1 && ok "(l) box revoke $U2 --purge exits 0" || no "(l) revoke failed"
|
||||
pgrep -u "$U2" >/dev/null 2>&1 \
|
||||
&& no "(l) $U2 still has live processes after the purge — a stale session could recreate their project, unhardened" \
|
||||
|| ok "(l) the purge terminated $U2's held session (no stale-group path back in)"
|
||||
as_u "$U2" incus list >/dev/null 2>&1 \
|
||||
&& no "(l) $U2 still reaches the daemon after revoke" \
|
||||
|| ok "(l) $U2 is locked out"
|
||||
incus project show "$p2" >/dev/null 2>&1 \
|
||||
&& no "(l) $U2's project survived the purge" \
|
||||
|| ok "(l) $U2's project, boxes and bridge are gone"
|
||||
st="$(incus --project "$p1" list mine --format csv --columns s 2>/dev/null | head -n1)"
|
||||
[ "$st" = RUNNING ] && ok "(l) $U1's box is untouched and RUNNING through it all" \
|
||||
|| no "(l) $U1's box state after $U2's purge: '$st'"
|
||||
aud "l. revoke --purge is scoped: $U2 erased, $U1 unmoved"
|
||||
|
||||
phase "n. a grant that fails is fail-closed — injected, both flavors"
|
||||
U3=boxdrill3; U4=boxdrill4
|
||||
BOXROOT="$(dirname "$(dirname "$(readlink -f "$(command -v box)")")")"
|
||||
|
||||
# Flavor 1: a FRESH user, fault injected at the LAST mutation (the profile
|
||||
# edit) — so the backout runs after every earlier mutation has landed. The
|
||||
# contract: nonzero exit, the group's absence VERIFIED, and a clean re-run
|
||||
# converges the partial state (which is what makes re-run-to-repair real).
|
||||
useradd -m -s /bin/bash "$U3" 2>/dev/null
|
||||
badroot="$(mktemp -d)"
|
||||
cp -r "$BOXROOT/." "$badroot/"
|
||||
echo 'devices: {' > "$badroot/profiles/box-net.yaml" # yaml that cannot load
|
||||
out="$(bash "$badroot/host/grant-user.sh" "$U3" 2>&1)"; rc=$?
|
||||
rm -rf "$badroot"
|
||||
if [ "$rc" -ne 0 ] && ! id -nG "$U3" | tr ' ' '\n' | grep -qx incus; then
|
||||
ok "(n) fresh-user grant failed at the last mutation → backed out, group absence verified (rc=$rc)"
|
||||
else
|
||||
no "(n) injected failure: rc=$rc, in-group=$(id -nG "$U3" | tr ' ' '\n' | grep -cx incus) — not fail-closed:"
|
||||
printf '%s\n' "$out" | tail -3 | sed 's/^/ /'
|
||||
fi
|
||||
printf '%s' "$out" | grep -q "verified against the group database" \
|
||||
&& ok "(n) the backout message claims only what it verified" \
|
||||
|| no "(n) the backout message is not the verified one"
|
||||
box grant "$U3" >/dev/null 2>&1 \
|
||||
&& ok "(n) a clean re-run converges the partial state left by the failure" \
|
||||
|| no "(n) re-run after injected failure did NOT converge"
|
||||
|
||||
# Flavor 2: a PRE-EXISTING member (hand-added before box, the review's named
|
||||
# scenario) with an instance parked on the private bridge by an
|
||||
# instance-local NIC — narrowing must fail, the grant must fail LOUDLY
|
||||
# saying they retain socket access, and must NOT strip the membership this
|
||||
# run did not add. Unblock, re-run, converge.
|
||||
useradd -m -s /bin/bash "$U4" 2>/dev/null
|
||||
usermod -aG incus "$U4"
|
||||
as_u "$U4" incus project list >/dev/null 2>&1 # materialize their project
|
||||
# Stage the blocker with an INSTANCE-LOCAL NIC, the shape that actually
|
||||
# blocks narrowing (a profile-inherited NIC is detached by grant's own
|
||||
# eth0 removal — no conflict). A raw --network flag would do it on 6.0.4
|
||||
# but 6.0.0 refuses that spelling for restricted certs (see criterion m);
|
||||
# 'device override' lifts their own stock profile NIC into the instance —
|
||||
# their instance, their config, permitted on both — same resulting state.
|
||||
stage="$(as_u "$U4" incus launch images:debian/13 blocker 2>&1)"; stagerc=$?
|
||||
[ "$stagerc" -eq 0 ] && { stage="$(as_u "$U4" incus config device override blocker eth0 2>&1)"; stagerc=$?; }
|
||||
if [ "$stagerc" -eq 0 ]; then
|
||||
out="$(box grant "$U4" 2>&1)"; rc=$?
|
||||
if [ "$rc" -ne 0 ] && printf '%s' "$out" | grep -q "still holding socket access"; then
|
||||
ok "(n) blocked narrowing fails LOUDLY, naming the retained access (rc=$rc)"
|
||||
else
|
||||
no "(n) blocked narrowing: rc=$rc — the loud contract is missing:"
|
||||
printf '%s\n' "$out" | tail -3 | sed 's/^/ /'
|
||||
fi
|
||||
id -nG "$U4" | tr ' ' '\n' | grep -qx incus \
|
||||
&& ok "(n) the pre-existing membership was NOT stripped by the failed re-grant" \
|
||||
|| no "(n) the failed grant stripped a membership it did not add"
|
||||
as_u "$U4" incus delete -f blocker >/dev/null 2>&1
|
||||
box grant "$U4" >/dev/null 2>&1 \
|
||||
&& ok "(n) unblocked re-run converges" \
|
||||
|| no "(n) re-run after unblocking failed"
|
||||
else
|
||||
no "(n) could not stage the private-bridge blocker — the blocked-narrowing contract went unmeasured:"
|
||||
printf '%s\n' "$stage" | tail -3 | sed 's/^/ /'
|
||||
fi
|
||||
aud "n. fail-closed injections: fresh-user backout verified; pre-existing member warned, not stripped; re-runs converge"
|
||||
|
||||
echo
|
||||
echo "════════════════════════════════════════════"
|
||||
echo " $pass passed, $fail failed"
|
||||
if [ "${#findings[@]}" -gt 0 ]; then
|
||||
echo; printf ' %s\n' "${findings[@]}"
|
||||
fi
|
||||
echo
|
||||
echo " #74 audit answers:"
|
||||
printf ' · %s\n' "${audit[@]}"
|
||||
echo
|
||||
[ "$fail" -eq 0 ] && echo " VERDICT: the restricted tier HOLDS — grant/confine/isolate/revoke, measured." \
|
||||
|| echo " VERDICT: the tier does NOT hold — see the findings."
|
||||
[ "$fail" -eq 0 ]
|
||||
202
host/grant-user.sh
Normal file
202
host/grant-user.sh
Normal file
|
|
@ -0,0 +1,202 @@
|
|||
#!/usr/bin/env bash
|
||||
# box grant <user> — give a host user the restricted tier (#74).
|
||||
#
|
||||
# The tier rides incus-user: the user lands in an auto-created project
|
||||
# user-<uid> and can only ever see their own instances. What incus-user does
|
||||
# NOT do is put them on box's hardened network — it auto-creates a private
|
||||
# bridge incusbr-<uid> (a plain NAT bridge: no ACL, no DNS isolation, IPv6 on,
|
||||
# none of box's contract) and pins the project to it. Measured on Debian 13 /
|
||||
# Incus 6.0.4; the full write-up is in docs/plans/2026-07-18-restricted-tier.md.
|
||||
#
|
||||
# So granting is a per-user CONVERGENCE, and it must be run by an admin:
|
||||
# 1. put the user in the 'incus' group (not incus-admin — that is the tier)
|
||||
# 2. touch incus-user AS the user, so the lazy project exists to converge
|
||||
# 3. unpin the private bridge (drop eth0 from the project's default profile)
|
||||
# 4. restrict the project's network access to boxnet and ONLY boxnet —
|
||||
# "boxnet,incusbr-<uid>" would leave an unhardened NAT bridge one
|
||||
# '--network' flag away from any box they mint
|
||||
# 5. allow snapshots (incus-user blocks them; box's clone workflow is built
|
||||
# on them)
|
||||
# 6. install the shipped box-net profile into their project
|
||||
#
|
||||
# Idempotent: every step converges, so re-running (including after a box
|
||||
# upgrade, to refresh the profile) is safe. incus-user never rewrites a
|
||||
# project it already created (verified against its source: setup is skipped
|
||||
# once the project exists and the user's certificate is trusted), so nothing
|
||||
# here is fighting a re-sync.
|
||||
set -euo pipefail
|
||||
|
||||
self="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
|
||||
here="$(dirname "$(dirname "$self")")"
|
||||
|
||||
usage() { echo "usage: box grant <user>" >&2; exit 2; }
|
||||
|
||||
[ $# -eq 1 ] || usage
|
||||
user="$1"
|
||||
case "$user" in -*) usage ;; esac
|
||||
|
||||
# Root, or sudo — same decision, same reasons as setup-host.sh: granting
|
||||
# needs usermod and a run-as-the-user touch, both root's to give.
|
||||
if [ "$(id -u)" -eq 0 ]; then
|
||||
SUDO=""
|
||||
elif command -v sudo >/dev/null 2>&1; then
|
||||
SUDO="sudo"
|
||||
else
|
||||
echo "ERROR: box grant needs root and 'sudo' was not found." >&2
|
||||
echo " re-run as root: $self $user" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Run a command as the granted user. 'runuser' when we are root (sudo may not
|
||||
# exist there), sudo -u otherwise. -H so incus's client state lands in THEIR
|
||||
# home, not the admin's. stdin pinned: an incus client with a terminal on
|
||||
# stdin can go interactive and wedge a script that will never answer it.
|
||||
run_as() {
|
||||
local u="$1"; shift
|
||||
if [ -n "$SUDO" ]; then $SUDO -u "$u" -H -- "$@" </dev/null
|
||||
else runuser -u "$u" -- "$@" </dev/null
|
||||
fi
|
||||
}
|
||||
|
||||
getent passwd "$user" >/dev/null || { echo "box grant: no such user: $user" >&2; exit 1; }
|
||||
uid="$(id -u "$user")"
|
||||
[ "$uid" -eq 0 ] && { echo "box grant: root does not need a tier — UID 0 owns the daemon socket outright." >&2; exit 1; }
|
||||
|
||||
# An incus-admin member already holds the full socket; "granting" them the
|
||||
# restricted tier would not restrict anything (admin membership wins at the
|
||||
# socket), it would only mislead whoever reads the group list later.
|
||||
if id -nG "$user" | tr ' ' '\n' | grep -qx incus-admin; then
|
||||
echo "box grant: $user is in incus-admin — they already have the admin tier; there is nothing tighter to grant." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The stack the tier converges ONTO must exist first. Checked via the daemon,
|
||||
# not config files: setup-host is the only thing that builds boxnet.
|
||||
incus network show boxnet >/dev/null 2>&1 </dev/null \
|
||||
|| { echo "box grant: no boxnet on this host — build the stack first: box setup-host" >&2; exit 1; }
|
||||
|
||||
# incus-user is the mechanism under the whole tier. Debian 13 and Ubuntu 24.04
|
||||
# ship it in the incus package; a host without it cannot hold this tier at all.
|
||||
if ! systemctl is-active --quiet incus-user.socket; then
|
||||
$SUDO systemctl enable --now incus-user.socket 2>/dev/null \
|
||||
|| { echo "box grant: incus-user.socket is not available — this Incus cannot serve the restricted tier (see #74)." >&2; exit 1; }
|
||||
fi
|
||||
|
||||
# 1. The group. 'incus' is the restricted socket; membership takes effect at
|
||||
# the user's next login, but run_as below starts a fresh process with the
|
||||
# database's groups, so the grant itself never waits on a re-login.
|
||||
#
|
||||
# If THIS run granted the group and a later step fails, take it back on the
|
||||
# way out: a half-granted user would otherwise hold live socket access to an
|
||||
# UN-NARROWED project — the stock unhardened bridge attachable — until an
|
||||
# admin re-runs. Backing out the group closes that window completely for a
|
||||
# fresh grant (their existing sessions predate the membership, so no process
|
||||
# holds it yet). A user who was already in the group keeps it: not ours to
|
||||
# take on a re-run's failure.
|
||||
added_group=0; was_member=0
|
||||
backout() {
|
||||
if [ "$added_group" -eq 1 ]; then
|
||||
$SUDO gpasswd -d "$user" incus >/dev/null 2>&1 || true
|
||||
# VERIFY the removal — an unverified rollback printing a security
|
||||
# guarantee is a lie waiting for its day. Exact-token match, live DB.
|
||||
if id -nG "$user" 2>/dev/null | tr ' ' '\n' | grep -qx incus; then
|
||||
echo "box grant: ROLLBACK INCOMPLETE — the grant failed AND $user is still in the 'incus' group." >&2
|
||||
echo " remove it by hand NOW: gpasswd -d $user incus (then fix the cause and re-run)" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "box grant: FAILED — removed $user from 'incus' again (verified against the group database); fix the cause and re-run" >&2
|
||||
# The one window the database cannot close: a login STARTED between our
|
||||
# usermod and this backout keeps the group in its session credentials.
|
||||
# For a fresh grant that is a rare race, but rare is not never — name it
|
||||
# and the remedy instead of overclaiming.
|
||||
if pgrep -u "$user" >/dev/null 2>&1; then
|
||||
echo "box grant: NOTE — $user has live processes; a session begun during this grant would still hold" >&2
|
||||
echo " the group until it ends: sudo loginctl terminate-user $user" >&2
|
||||
fi
|
||||
elif [ "$was_member" -eq 1 ]; then
|
||||
# A user who was ALREADY in the group keeps it — stripping a membership
|
||||
# this run did not add could break a working user over a failed re-run.
|
||||
# But silence here would leave them holding a socket onto part-converged
|
||||
# policy without the admin being told. Loud, with both remediations.
|
||||
echo "box grant: FAILED with $user still holding socket access (their membership predates this run)." >&2
|
||||
echo " their project may be part-converged — harmless in itself, and a re-run converges the rest." >&2
|
||||
echo " if their access is not acceptable while you fix the cause: box revoke $user" >&2
|
||||
fi
|
||||
}
|
||||
trap backout EXIT
|
||||
|
||||
if id -nG "$user" | tr ' ' '\n' | grep -qx incus; then
|
||||
was_member=1
|
||||
echo "group: $user already in 'incus'"
|
||||
else
|
||||
$SUDO usermod -aG incus "$user"
|
||||
added_group=1
|
||||
echo "group: added $user to 'incus' (their next login picks it up; the grant does not wait)"
|
||||
fi
|
||||
|
||||
project="user-$uid"
|
||||
|
||||
# 2. The project is created LAZILY, on the user's first contact with
|
||||
# incus-user — an admin cannot pre-create it (incus-user would fight over
|
||||
# it), so make that first contact happen now, as the user.
|
||||
if ! incus project show "$project" >/dev/null 2>&1 </dev/null; then
|
||||
echo "project: touching incus-user as $user to create $project..."
|
||||
run_as "$user" timeout 60 incus project list >/dev/null 2>&1 || true
|
||||
incus project show "$project" >/dev/null 2>&1 </dev/null \
|
||||
|| { echo "box grant: incus-user did not create $project — is incus-user.socket healthy? (journalctl -u incus-user)" >&2; exit 1; }
|
||||
echo "project: $project created"
|
||||
else
|
||||
echo "project: $project already exists"
|
||||
fi
|
||||
|
||||
# 3. Unpin the private bridge. incus-user's default profile carries an eth0
|
||||
# on incusbr-<uid>; while ANY profile references that bridge, the narrowing
|
||||
# below is rejected by incus's own validation. Removing the device is also
|
||||
# what it looks like: the default profile in this project places no network —
|
||||
# box-net is the only door, which is the placement contract working.
|
||||
if incus --project "$project" profile device get default eth0 type >/dev/null 2>&1 </dev/null; then
|
||||
incus --project "$project" profile device remove default eth0 >/dev/null </dev/null
|
||||
echo "profile: removed the private-bridge eth0 from $project's default profile"
|
||||
fi
|
||||
|
||||
# 4. boxnet, and ONLY boxnet. The auto-created incusbr-<uid> is a stock NAT
|
||||
# bridge with none of box's hardening — listing it here would keep a
|
||||
# one-flag escape from the isolation contract open forever. Narrowed, the
|
||||
# hardened network is not the default placement but the only one possible.
|
||||
# This can fail honestly: an instance the user already parked on the private
|
||||
# bridge blocks the narrowing, and incus's error names it.
|
||||
if ! err="$(incus project set "$project" restricted.networks.access boxnet 2>&1 </dev/null)"; then
|
||||
echo "box grant: could not restrict $project to boxnet:" >&2
|
||||
echo " $err" >&2
|
||||
echo " (an instance still on the private bridge blocks this — move or delete it, then re-run)" >&2
|
||||
exit 1
|
||||
fi
|
||||
# The private bridge's name follows incus-user's own rule (revoke-user.sh
|
||||
# mirrors it too): incusbr-<uid>, or user-<uid> when that would not fit an
|
||||
# interface name — naming the wrong one here would be a true claim with the
|
||||
# wrong noun on big-uid (SSSD/AD) hosts.
|
||||
bridge="incusbr-$uid"; [ "${#bridge}" -gt 15 ] && bridge="user-$uid"
|
||||
echo "network: $project restricted to boxnet (the private $bridge is unreferenced and unreachable)"
|
||||
|
||||
# 5. Snapshots. incus-user projects block them by default, and box's whole
|
||||
# reuse story — log in once, snapshot, clone forever — is snapshots.
|
||||
incus project set "$project" restricted.snapshots allow </dev/null
|
||||
echo "snapshots: allowed"
|
||||
|
||||
# 6. The placement contract itself, installed into their project. Created if
|
||||
# missing, refreshed unconditionally — same convergence discipline as
|
||||
# setup-host's own profile handling, so a box upgrade propagates by re-run.
|
||||
incus --project "$project" profile show box-net >/dev/null 2>&1 </dev/null \
|
||||
|| incus --project "$project" profile create box-net >/dev/null </dev/null
|
||||
incus --project "$project" profile edit box-net < "$here/profiles/box-net.yaml"
|
||||
echo "profile: box-net installed in $project"
|
||||
|
||||
# Prove the grant from the USER's side of the socket — the only side that
|
||||
# matters. This catches the failure the steps above cannot see one at a time:
|
||||
# a converged project the user still cannot reach.
|
||||
run_as "$user" timeout 30 incus profile show box-net >/dev/null 2>&1 \
|
||||
|| { echo "box grant: converged, but $user cannot see the box-net profile through incus-user — check journalctl -u incus-user" >&2; exit 1; }
|
||||
|
||||
trap - EXIT # converged and verified: the grant stands
|
||||
echo "granted: $user has the restricted tier — their 'box new' lands on the hardened boxnet."
|
||||
echo " (their boxes are theirs alone; 'box revoke $user' takes the tier back)"
|
||||
156
host/revoke-user.sh
Normal file
156
host/revoke-user.sh
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
#!/usr/bin/env bash
|
||||
# box revoke <user> [--purge] — take the restricted tier back (#74).
|
||||
#
|
||||
# Two strengths, deliberately:
|
||||
# · bare revoke removes the user from the 'incus' group. That closes the
|
||||
# socket — the only path their certificate can travel — so access ends at
|
||||
# their next login, while their project and boxes stay intact (and their
|
||||
# boxes stay RUNNING: revoking a person does not kill their workloads).
|
||||
# 'box grant' restores everything untouched.
|
||||
# · --purge also deletes what the tier created: their boxes, their images,
|
||||
# their project, the private bridge, the trust-store certificate, the
|
||||
# incus-user state. Irreversible, so it asks first.
|
||||
set -euo pipefail
|
||||
|
||||
usage() { echo "usage: box revoke <user> [--purge]" >&2; exit 2; }
|
||||
|
||||
user=""; purge=0
|
||||
for a in "$@"; do
|
||||
case "$a" in
|
||||
--purge) purge=1 ;;
|
||||
-*) usage ;;
|
||||
*) [ -z "$user" ] || usage; user="$a" ;;
|
||||
esac
|
||||
done
|
||||
[ -n "$user" ] || usage
|
||||
|
||||
if [ "$(id -u)" -eq 0 ]; then
|
||||
SUDO=""
|
||||
elif command -v sudo >/dev/null 2>&1; then
|
||||
SUDO="sudo"
|
||||
else
|
||||
echo "ERROR: box revoke needs root and 'sudo' was not found." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
getent passwd "$user" >/dev/null || { echo "box revoke: no such user: $user" >&2; exit 1; }
|
||||
uid="$(id -u "$user")"
|
||||
project="user-$uid"
|
||||
# incus-user's own naming rule, mirrored exactly: the bridge is incusbr-<uid>
|
||||
# unless that would not fit in an interface name (15 chars), then user-<uid>.
|
||||
bridge="incusbr-$uid"
|
||||
[ "${#bridge}" -gt 15 ] && bridge="user-$uid"
|
||||
|
||||
if [ "$purge" -eq 1 ]; then
|
||||
# Destructive and irreversible: a TTY to ask on, or BOX_YES=1, or refuse —
|
||||
# the same non-interactive contract as install.sh.
|
||||
if [ -z "${BOX_YES:-}" ]; then
|
||||
if [ -t 0 ]; then
|
||||
printf 'box revoke: delete ALL of %s'\''s boxes, images and their project %s? this cannot be undone. [y/N] ' "$user" "$project"
|
||||
read -r reply
|
||||
case "$reply" in y|Y|yes|YES|Yes) : ;; *) echo "box revoke: aborted." >&2; exit 1 ;; esac
|
||||
else
|
||||
echo "box revoke: refusing to --purge without a terminal to confirm on. BOX_YES=1 means yes." >&2
|
||||
exit 2
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# The group, first — access ends even if a purge step below trips.
|
||||
if id -nG "$user" | tr ' ' '\n' | grep -qx incus; then
|
||||
$SUDO gpasswd -d "$user" incus >/dev/null
|
||||
echo "group: removed $user from 'incus'"
|
||||
else
|
||||
echo "group: $user was not in 'incus'"
|
||||
fi
|
||||
|
||||
# Supplementary groups are fixed AT LOGIN: the database change above does
|
||||
# nothing to a session the user already holds — a leftover tmux keeps the
|
||||
# socket until it dies. For a bare revoke that is an honest warning. For
|
||||
# --purge it is a hole: a stale-group process can touch incus-user AFTER the
|
||||
# purge and lazily recreate the project with incus-user's stock defaults —
|
||||
# the unhardened NAT bridge, un-narrowed — which is strictly worse than the
|
||||
# granted state this script is unwinding. So --purge terminates the user's
|
||||
# sessions first (it is already the destructive, confirmed path), and a bare
|
||||
# revoke says out loud what it did not do.
|
||||
if pgrep -u "$user" >/dev/null 2>&1; then
|
||||
if [ "$purge" -eq 1 ]; then
|
||||
echo "sessions: $user has live processes — terminating them (a stale session could recreate the project, unhardened, after the purge)"
|
||||
$SUDO loginctl terminate-user "$user" 2>/dev/null || true
|
||||
$SUDO pkill -u "$user" 2>/dev/null || true
|
||||
sleep 1
|
||||
$SUDO pkill -9 -u "$user" 2>/dev/null || true
|
||||
if pgrep -u "$user" >/dev/null 2>&1; then
|
||||
echo "box revoke: could not terminate $user's processes — refusing to purge under them" >&2
|
||||
echo " (they retain the socket until those sessions end, and could recreate the project)" >&2
|
||||
exit 1
|
||||
fi
|
||||
else
|
||||
echo "WARNING: $user has live sessions, and group membership is read at login —"
|
||||
echo " those sessions keep the socket until they end. To end them now:"
|
||||
echo " sudo loginctl terminate-user $user"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "$purge" -eq 0 ]; then
|
||||
if incus project show "$project" >/dev/null 2>&1 </dev/null; then
|
||||
echo "kept: project $project and its boxes (still running — revoking a person does not kill their workloads)"
|
||||
echo " 'box revoke $user --purge' deletes them; 'box grant $user' restores access"
|
||||
fi
|
||||
echo "revoked: $user no longer has the restricted tier."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# --purge: unmake what the tier made. Instances one at a time — a wildcard
|
||||
# delete that half-fails leaves a state nobody can name; a loop that fails
|
||||
# names the box it failed on (the wipe.sh discipline).
|
||||
if incus project show "$project" >/dev/null 2>&1 </dev/null; then
|
||||
while IFS=, read -r inst _; do
|
||||
[ -n "$inst" ] || continue
|
||||
echo "purge: deleting instance $inst"
|
||||
incus --project "$project" delete -f "$inst" </dev/null
|
||||
done < <(incus --project "$project" list --format csv --columns n 2>/dev/null)
|
||||
|
||||
while IFS=, read -r fp _; do
|
||||
[ -n "$fp" ] || continue
|
||||
incus --project "$project" image delete "$fp" </dev/null
|
||||
done < <(incus --project "$project" image list --format csv --columns f 2>/dev/null)
|
||||
|
||||
incus --project "$project" profile delete box-net >/dev/null 2>&1 </dev/null || true
|
||||
incus project delete "$project" </dev/null \
|
||||
|| { echo "box revoke: could not delete $project — something is still in it (incus --project $project list / image list / storage volume list)" >&2; exit 1; }
|
||||
echo "purge: project $project removed"
|
||||
fi
|
||||
|
||||
if incus network delete "$bridge" >/dev/null 2>&1 </dev/null; then
|
||||
echo "purge: private bridge $bridge removed"
|
||||
fi
|
||||
|
||||
# The trust-store certificate incus-user minted for them. Named, not guessed:
|
||||
# incus-user calls it incus-user-<uid>.
|
||||
while IFS=, read -r name fp _; do
|
||||
[ "$name" = "incus-user-$uid" ] || continue
|
||||
incus config trust remove "$fp" </dev/null && echo "purge: trust-store certificate $name removed"
|
||||
done < <(incus config trust list --format csv --columns nf 2>/dev/null)
|
||||
|
||||
# incus-user's per-user client state (their key pair). Removed so a future
|
||||
# re-grant starts clean instead of trusting a key the purge revoked.
|
||||
if [ -d "/var/lib/incus/users/$uid" ]; then
|
||||
$SUDO rm -rf "/var/lib/incus/users/$uid"
|
||||
echo "purge: incus-user state for uid $uid removed"
|
||||
fi
|
||||
|
||||
# Assert absence rather than trusting exit codes — the wipe.sh discipline.
|
||||
# The certificate included: its removal above is set -e-exempt (left of &&),
|
||||
# and a promise the header makes is a promise this block checks.
|
||||
leftover=""
|
||||
incus project show "$project" >/dev/null 2>&1 </dev/null && leftover="$leftover $project"
|
||||
incus network show "$bridge" >/dev/null 2>&1 </dev/null && leftover="$leftover $bridge"
|
||||
incus config trust list --format csv --columns nf 2>/dev/null | grep -q "^incus-user-$uid," \
|
||||
&& leftover="$leftover cert:incus-user-$uid"
|
||||
if [ -n "$leftover" ]; then
|
||||
echo "box revoke: purge INCOMPLETE — still present:$leftover" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "revoked: $user is out, and everything the tier created is gone."
|
||||
|
|
@ -6,6 +6,31 @@ set -euo pipefail
|
|||
self="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/$(basename "${BASH_SOURCE[0]}")"
|
||||
here="$(dirname "$(dirname "$self")")"
|
||||
|
||||
# Byte-identical copy of bin/box's box_tier() — this script must know the
|
||||
# tier before any install tree exists, and test/cli.sh diffs the two copies
|
||||
# so they cannot drift.
|
||||
box_tier() {
|
||||
[ "$(id -u)" -eq 0 ] && { printf 'admin\n'; return; }
|
||||
local groups; groups="$(id -nG 2>/dev/null | tr ' ' '\n')"
|
||||
if printf '%s\n' "$groups" | grep -qx incus-admin; then printf 'admin\n'
|
||||
elif printf '%s\n' "$groups" | grep -qx incus; then printf 'restricted\n'
|
||||
else printf 'none\n'
|
||||
fi
|
||||
}
|
||||
|
||||
# A restricted (incus-group) user cannot build daemon-global state, and
|
||||
# telling them to escalate would be wrong twice: the stack is the admin's to
|
||||
# own, and if 'box new' works for them it already exists. Say so and succeed —
|
||||
# this must sit BEFORE the sudo resolution below, which would otherwise bury
|
||||
# the honest answer under a privilege error. Gated on the tier, not on
|
||||
# 'command -v sudo': having the sudo binary is not the same as holding a grant.
|
||||
if [ "$(id -u)" -ne 0 ] && [ "$(box_tier)" = restricted ]; then
|
||||
echo "You are in the 'incus' group (restricted tier): you manage your own boxes," >&2
|
||||
echo "but the host's daemon-global stack is built by an admin. It is already set" >&2
|
||||
echo "up if 'box new' works. Nothing for you to do here." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# How we reach root, decided once. 'sudo' cannot be hardcoded: at UID 0 it is
|
||||
# unnecessary, and on a minimal root image it is not installed at all — this
|
||||
# script died on 'sudo: command not found' before doing anything, which made
|
||||
|
|
@ -195,6 +220,13 @@ $SUDO systemctl enable box-firewall.service
|
|||
# claimed to close it. Restart re-runs the script, which is idempotent by design.
|
||||
$SUDO systemctl restart box-firewall.service
|
||||
|
||||
# incus-user is what serves the restricted tier (box grant). Debian 13 and
|
||||
# Ubuntu 24.04 ship it inside the incus package; enabling it here makes the
|
||||
# host tier-ready, and costs a host that never grants anyone nothing. Failure
|
||||
# is a NOTE, not an error: the admin tier does not depend on it.
|
||||
$SUDO systemctl enable --now incus-user.socket 2>/dev/null \
|
||||
|| echo "NOTE: could not enable incus-user.socket — 'box grant' (the restricted tier) needs it; this Incus may not ship incus-user (#74)." >&2
|
||||
|
||||
# Profile — box-net, the placement contract: the isolated NIC and the root
|
||||
# disk, nothing a template controls (resources are stamped per-instance from
|
||||
# the template at mint time). A legacy claude-dev profile is left alone:
|
||||
|
|
|
|||
137
test/cli.sh
137
test/cli.sh
|
|
@ -139,6 +139,143 @@ for t in blank claude codex grok; do
|
|||
grep -qE '^[[:space:]]*-[[:space:]]+tmux$' "$ROOT/templates/$t/user-data.yaml"
|
||||
done
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# The restricted tier (#74). box_tier() is the decision the whole tier hangs
|
||||
# on, so it is DRIVEN, not grepped: extracted from bin/box, sourced, and run
|
||||
# against a shim id for every case — including the one that bites (a user in
|
||||
# BOTH groups is admin: membership wins at the socket, and the function must
|
||||
# not substring-match 'incus' inside 'incus-admin').
|
||||
# ---------------------------------------------------------------------------
|
||||
TIERFN="$(mktemp)"
|
||||
awk '/^box_tier\(\) \{/,/^\}/' "$ROOT/bin/box" > "$TIERFN"
|
||||
check "box_tier: extracted from bin/box (guards the awk)" 0 "incus-admin" cat "$TIERFN"
|
||||
check "box_tier: the extracted function is valid bash" 0 "" bash -n "$TIERFN"
|
||||
|
||||
tier() { # tier <uid> <groups...>
|
||||
local uid="$1"; shift
|
||||
FAKE_UID="$uid" FAKE_GROUPS="$*" PATH="$SHIMDIR:$PATH" \
|
||||
bash -c ". '$TIERFN'; box_tier"
|
||||
}
|
||||
check "box_tier: uid 0 → admin" 0 "admin" tier 0
|
||||
check "box_tier: incus-admin → admin" 0 "admin" tier 1000 "users incus-admin"
|
||||
check "box_tier: incus only → restricted" 0 "restricted" tier 1000 "users incus"
|
||||
check "box_tier: both groups → admin (membership wins at the socket)" \
|
||||
0 "admin" tier 1000 "users incus incus-admin"
|
||||
check "box_tier: neither → none" 0 "none" tier 1000 "users dialout"
|
||||
rm -f "$TIERFN"
|
||||
|
||||
# setup-host.sh must decide the tier BEFORE any install tree exists, so it
|
||||
# carries its own copy — and a drifted copy is two tiers pretending to be one.
|
||||
# Byte-identical, asserted.
|
||||
BINFN="$(mktemp)"; HOSTFN="$(mktemp)"
|
||||
awk '/^box_tier\(\) \{/,/^\}/' "$ROOT/bin/box" > "$BINFN"
|
||||
awk '/^box_tier\(\) \{/,/^\}/' "$ROOT/host/setup-host.sh" > "$HOSTFN"
|
||||
check "box_tier: bin/box and setup-host.sh copies are byte-identical" 0 "" \
|
||||
diff "$BINFN" "$HOSTFN"
|
||||
rm -f "$BINFN" "$HOSTFN"
|
||||
|
||||
# The tier scripts parse and refuse bad usage without a daemon — drive them.
|
||||
check "grant: no argument is a usage error" 2 "usage: box grant" bash "$ROOT/host/grant-user.sh"
|
||||
check "grant: a flag is not a user" 2 "usage: box grant" bash "$ROOT/host/grant-user.sh" --frob
|
||||
check "revoke: no argument is a usage error" 2 "usage: box revoke" bash "$ROOT/host/revoke-user.sh"
|
||||
check "revoke: two users is a usage error" 2 "usage: box revoke" bash "$ROOT/host/revoke-user.sh" a b
|
||||
check "box grant with no user exits 2 (via the CLI table)" 2 "usage: box grant" "$BOX" grant
|
||||
check "box revoke with no user exits 2 (via the CLI table)" 2 "usage: box revoke" "$BOX" revoke
|
||||
check "help grant names the hardened network" 0 "boxnet" "$BOX" help grant
|
||||
check "help revoke names --purge" 0 "purge" "$BOX" help revoke
|
||||
|
||||
# Load-bearing lines a daemon-free run cannot exercise — grepped so a deleted
|
||||
# guard cannot ship green (the house test discipline).
|
||||
# The expose guard must fire before ANY incus call in cmd_expose: line order.
|
||||
# shellcheck disable=SC2016 # the $-strings are literals in the target file
|
||||
check "expose: the restricted guard precedes the first incus call" 0 "" bash -c '
|
||||
fn="$(awk "/^cmd_expose\(\) \{/,/^\}/" "'"$ROOT"'/bin/box")"
|
||||
guard="$(printf "%s\n" "$fn" | grep -n "box_tier" | head -1 | cut -d: -f1)"
|
||||
first="$(printf "%s\n" "$fn" | grep -n "incus config" | head -1 | cut -d: -f1)"
|
||||
[ -n "$guard" ] && [ -n "$first" ] && [ "$guard" -lt "$first" ]'
|
||||
# cmd_new refuses before minting when the placement contract is absent, and
|
||||
# the message is tier-aware (a restricted user is sent to 'box grant', not
|
||||
# to setup-host they cannot run).
|
||||
check "new: pre-flights the box-net profile" 0 "" bash -c '
|
||||
awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" | grep -q "incus profile show box-net"'
|
||||
check "new: the restricted fix names box grant" 0 "" bash -c '
|
||||
awk "/^cmd_new\(\) \{/,/^\}/" "'"$ROOT"'/bin/box" | grep -q "box grant"'
|
||||
# grant converges to boxnet and ONLY boxnet — "boxnet,incusbr" would keep the
|
||||
# unhardened private bridge one --network flag away (the #74 measured hole).
|
||||
check "grant: narrows access to boxnet alone" 0 "" \
|
||||
grep -qE 'restricted\.networks\.access boxnet($| )' "$ROOT/host/grant-user.sh"
|
||||
check "grant: never grants the private bridge" 1 "" \
|
||||
grep -qE 'networks\.access[^#]*incusbr' "$ROOT/host/grant-user.sh"
|
||||
check "grant: allows snapshots (the clone workflow)" 0 "" \
|
||||
grep -qF 'restricted.snapshots allow' "$ROOT/host/grant-user.sh"
|
||||
# shellcheck disable=SC2016 # the $-strings are literals in the target file
|
||||
check "grant: installs the SHIPPED profile into the project" 0 "" \
|
||||
grep -qF 'profile edit box-net < "$here/profiles/box-net.yaml"' "$ROOT/host/grant-user.sh"
|
||||
check "grant: unpins the private-bridge eth0 from the default profile" 0 "" \
|
||||
grep -qF 'profile device remove default eth0' "$ROOT/host/grant-user.sh"
|
||||
check "grant: refuses an incus-admin member (nothing tighter to grant)" 0 "" \
|
||||
grep -qF 'incus-admin' "$ROOT/host/grant-user.sh"
|
||||
check "revoke: group removal is the lockout" 0 "" \
|
||||
grep -qF 'gpasswd -d' "$ROOT/host/revoke-user.sh"
|
||||
# Group membership is read at login: purge must terminate live sessions (a
|
||||
# stale-group process could recreate the project unhardened AFTER the purge),
|
||||
# and a bare revoke must say the socket survives in held sessions.
|
||||
check "revoke: purge terminates live sessions first" 0 "" \
|
||||
grep -qF 'loginctl terminate-user' "$ROOT/host/revoke-user.sh"
|
||||
check "revoke: purge refuses under unkillable sessions" 0 "" \
|
||||
grep -qF 'refusing to purge under them' "$ROOT/host/revoke-user.sh"
|
||||
check "revoke: bare revoke warns about held sessions" 0 "" \
|
||||
grep -qF 'live sessions' "$ROOT/host/revoke-user.sh"
|
||||
check "revoke: the purge asserts the certificate's absence too" 0 "" \
|
||||
bash -c 'grep -A6 "Assert absence" "'"$ROOT"'/host/revoke-user.sh" | grep -q "config trust list"'
|
||||
# A failed grant must not leave a half-granted user: if THIS run added the
|
||||
# group, the exit path takes it back (and the trap disarms only on success).
|
||||
check "grant: backs out its own group-add on failure" 0 "" \
|
||||
grep -qF 'trap backout EXIT' "$ROOT/host/grant-user.sh"
|
||||
check "grant: the back-out disarms on success" 0 "" \
|
||||
grep -qF 'trap - EXIT' "$ROOT/host/grant-user.sh"
|
||||
# The backout must VERIFY the removal and scream when it cannot — an
|
||||
# unverified rollback printing a security guarantee is the review's A2.
|
||||
check "grant: the backout verifies against the group database" 0 "" \
|
||||
bash -c 'awk "/^backout\(\) \{/,/^\}/" "'"$ROOT"'/host/grant-user.sh" | grep -q "id -nG"'
|
||||
check "grant: an unverifiable rollback screams" 0 "" \
|
||||
grep -qF 'ROLLBACK INCOMPLETE' "$ROOT/host/grant-user.sh"
|
||||
check "grant: a failed re-grant warns the pre-existing member is untouched" 0 "" \
|
||||
grep -qF 'still holding socket access' "$ROOT/host/grant-user.sh"
|
||||
check "grant: the mid-grant login window is named" 0 "" \
|
||||
bash -c 'awk "/^backout\(\) \{/,/^\}/" "'"$ROOT"'/host/grant-user.sh" | grep -q "loginctl terminate-user"'
|
||||
# The scoped guarantee (raw --network boxnet) is measured, not prose:
|
||||
check "rehearsal: measures the raw boxnet attach (criterion m)" 0 "" \
|
||||
grep -qF -- '--network boxnet' "$ROOT/drill/multiuser.sh"
|
||||
# shellcheck disable=SC2016 # the $-string is a literal in the target file
|
||||
check "rehearsal: injects grant failures (criterion n)" 0 "" \
|
||||
grep -qF 'grant-user.sh" "$U3"' "$ROOT/drill/multiuser.sh"
|
||||
# shellcheck disable=SC2016 # the $-strings are literals in the target file
|
||||
check "revoke: purge deletes instances one at a time" 0 "" \
|
||||
grep -qF 'delete -f "$inst"' "$ROOT/host/revoke-user.sh"
|
||||
check "revoke: purge removes the trust-store certificate" 0 "" \
|
||||
grep -qF 'config trust remove' "$ROOT/host/revoke-user.sh"
|
||||
# shellcheck disable=SC2016 # the $-strings are literals in the target file
|
||||
check "setup-host: the restricted gate precedes the sudo resolution" 0 "" bash -c '
|
||||
gate="$(grep -n "restricted tier" "'"$ROOT"'/host/setup-host.sh" | head -1 | cut -d: -f1)"
|
||||
sudo="$(grep -n "^elif command -v sudo" "'"$ROOT"'/host/setup-host.sh" | head -1 | cut -d: -f1)"
|
||||
[ -n "$gate" ] && [ -n "$sudo" ] && [ "$gate" -lt "$sudo" ]'
|
||||
check "setup-host: enables incus-user.socket for the tier" 0 "" \
|
||||
grep -qF 'incus-user.socket' "$ROOT/host/setup-host.sh"
|
||||
check "doctor: honors BOX_TIER" 0 "" \
|
||||
grep -qF 'BOX_TIER' "$ROOT/drill/doctor.sh"
|
||||
check "box exports BOX_TIER to the doctor" 0 "" \
|
||||
grep -qF 'export BOX_TIER' "$ROOT/bin/box"
|
||||
# 'box restore' must speak incus 6 ('snapshot restore'); bare 'incus restore'
|
||||
# does not exist and the verb was broken for everyone until #74's rehearsal hit it.
|
||||
check "restore: dispatches 'incus snapshot restore'" 0 "" \
|
||||
grep -qF '^incus:snapshot restore^' "$ROOT/bin/box"
|
||||
# The rehearsal itself stays runnable: syntax-checked here, run on real hosts.
|
||||
check "multiuser.sh is valid bash" 0 "" bash -n "$ROOT/drill/multiuser.sh"
|
||||
check "multiuser.sh refuses without the env gate" 2 "opt in" \
|
||||
bash "$ROOT/drill/multiuser.sh" --yes
|
||||
check "grant-user.sh is valid bash" 0 "" bash -n "$ROOT/host/grant-user.sh"
|
||||
check "revoke-user.sh is valid bash" 0 "" bash -n "$ROOT/host/revoke-user.sh"
|
||||
|
||||
echo "---"
|
||||
echo "$PASS passed, $FAIL failed"
|
||||
|
|
|
|||
Loading…
Reference in a new issue
On a shared
boxnetthis contract is now doing tenant isolation, not just defense-in-depth. Note that of these four, onlysecurity.port_isolation(L2) and the nft bridge drop actually stop box→box — the ACL is L3/L4 and, perprofiles/box-net.yaml's own comment, was silently bypassed at L2. So the multi-tenant boundary rests on those two mechanisms. Worth stating that explicitly here.This widens access to include
boxnetbut (per #74)incusbr-<uid>stays in the access list too. Does the convergence also remove/repoint the auto default profile so a restricted user can'tincus launchonto their unhardenedincusbr-<uid>, skipping the box-net contract entirely? If not, the isolation stack is opt-in for anyone with rawincusin their project. Please spell out how the per-user bridge is neutralized — and whether the widenedrestricted.networks.accesssurvives an incus-user re-sync.For criteria (a)–(f), please include an active cross-user probe, not just "user A can't see user B's instances": from restricted user A's box, ARP/SYN/broadcast at user B's box on the same
boxnetsubnet, and assert silence. The single-owner drill already trustsport_isolation; multi-tenant is exactly where an L2 gap would bite, and #74's own reachability claim is the thing under test.CI (
.github/workflows/ci.yml) explicitly does not run the drill or the multiuser rehearsal —ubuntu-latestcan't reliably nest Incus +/dev/kvm. So "Test suite expansion + CI wiring" needs a concrete story: a self-hosted runner, or gated/manual drill jobs with shellcheck-only on GH-hosted. Otherwise "wired into CI" overstates what actually gates a merge.