diff --git a/CHANGELOG.md b/CHANGELOG.md index e528a91..de1c3d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,10 +20,10 @@ which records not just what changed but what each drill run proved. into their project. `box_tier()` (live credentials, argless `id -nG`) drives the tier-aware surface: `expose` refuses honestly before any daemon call, `setup-host` and `doctor` answer at the caller's tier. Rehearsed - end-to-end by `drill/multiuser.sh` (criteria a–l: confinement, lifecycle, + end-to-end by `drill/multiuser.sh` (criteria a–n: confinement, lifecycle, cross-user visibility, name collisions, the in-box isolation contract, escape hatches, re-sync survival, revoke incl. the live-session case) — - 42/42 on the design host, in both container and VM mode. + 54/54 on the design host (container and VM mode), including the raw-attach scoped-guarantee measurement and both grant-failure injections demanded by #75's review. - **CI runs the multi-user rehearsal on a real Incus** — a second `rehearsal` job stands up the full stack on the runner (setup-host, doctor, then `multiuser.sh --container`), so every PR proves the tier's semantics diff --git a/docs/box-design.md b/docs/box-design.md index d527f7e..037e929 100644 --- a/docs/box-design.md +++ b/docs/box-design.md @@ -165,9 +165,16 @@ Cross-USER isolation is the same mechanism as cross-box isolation, on purpose: their instances share `boxnet` with everyone's, and the bridge-family drop + port isolation + `dns.mode=none` already make any two boxes strangers. A restricted user CAN strip `security.port_isolation` from the profile copy -in their own project — which is why the nft bridge drop, which they cannot -touch, exists as the second, host-owned layer. Defense in depth, both layers -measured (`drill/multiuser.sh`, criteria a–l). +in their own project — or skip the profile entirely and attach `boxnet` raw +(`--network boxnet`); the network must be usable for the profile to work, and +Incus has no allow-via-profile-only lever. So the guarantee is scoped, and +said plainly: **per-NIC port isolation is guaranteed for box-minted +instances; a raw attachment keeps every network-owned control (the ACL, +`dns.mode=none`, the resolver pin) and every host-owned one (the nft bridge +drop) — losing only the redundant per-NIC L2 layer.** Scoped, and measured: +`drill/multiuser.sh` criterion (m) launches exactly that raw instance and +probes egress, RFC1918, both sibling directions and name enumeration from +inside it. Defense in depth, every layer measured (criteria a–n). `box revoke` is two strengths: bare, it removes the group — their boxes keep *running* (revoking a person does not kill their workloads), `grant` restores diff --git a/docs/plans/2026-07-18-restricted-tier.md b/docs/plans/2026-07-18-restricted-tier.md index e25bf30..8fbec3d 100644 --- a/docs/plans/2026-07-18-restricted-tier.md +++ b/docs/plans/2026-07-18-restricted-tier.md @@ -1,6 +1,6 @@ # Restricted incus tier — design and measured results (#74) -**Status: implemented and rehearsed.** 42/42 rehearsal criteria green on the +**Status: implemented and rehearsed.** 54/54 rehearsal criteria green on the design host (Debian 13 trixie, Incus 6.0.4, nested KVM), in container mode and VM mode — and green in CI on ubuntu-latest / incus 6.0.0 (whose one version-drift false FAIL is MU-4 in `drill/RUNS.md`). This doc records the design, what was measured, and why each @@ -69,10 +69,27 @@ touch `boxnet`'s config or the ACL (`no permission for project "default"`). A restricted user CAN edit the `box-net` profile copy in their own project (they own project profiles — `features.profiles=true`), including stripping -`security.port_isolation`. That is why the host-owned nft bridge drop is the -second layer: `meta ibrname boxnet obrname boxnet drop` fires on every -port-to-port frame regardless of per-NIC flags. Cross-user sibling probes are -dropped either way — measured from inside the boxes. +`security.port_isolation` — and CAN attach `boxnet` raw with `--network +boxnet`, no profile at all (the network must be in +`restricted.networks.access` for the profile to work; there is no +allow-via-profile-only lever). That is why the host-owned nft bridge drop is +the second layer: `meta ibrname boxnet obrname boxnet drop` fires on every +port-to-port frame regardless of per-NIC flags. The documented guarantee is +scoped accordingly (see box-design.md): box-minted instances carry per-NIC +port isolation; raw attachments keep every network- and host-owned control, +losing only that redundant L2 layer. Both shapes are measured from inside +the instances (rehearsal criteria g and m). + +Two grant-failure contracts, both injected in the rehearsal (criterion n): +a fresh user is backed out of the group with the removal VERIFIED against +the live group database (and any session begun mid-grant is named, with the +loginctl remedy — the one window the database cannot close); a pre-existing +member is never stripped by a failed re-grant, but the failure states out +loud that they retain socket access on part-converged policy, with both +remediations. The default-profile eth0 removal is deliberately NOT restored +on failure: that mutation only reduces capability, and restoring it would +move the failure state away from fail-closed. Every step is check-then- +converge, which is what makes re-run-to-repair deterministic. ## What `box grant ` converges (idempotent, re-run to refresh) @@ -110,10 +127,11 @@ It had never worked. ## Rehearsal and CI `drill/multiuser.sh` (root, opt-in via `BOX_MULTIUSER_REHEARSAL=1`) proves -criteria (a)–(f) from #74 plus the measured extensions (g)–(l): the in-box +criteria (a)–(f) from #74 plus the measured extensions (g)–(n): the in-box isolation contract (egress, DNS, box→host, RFC1918, cross-user sibling drop, name enumeration, IPv6-off), the closed escape hatches, re-sync survival, and -scoped revoke. Two real users, real grants, real mints, probes from inside; +scoped revoke, the raw-attach scoped guarantee (m) and the grant-failure +injections (n). Real users, real grants, real mints, probes from inside; `--container` for CI, VM mode on real hardware; cleanup deletes everything it made. diff --git a/drill/RUNS.md b/drill/RUNS.md index f85ca4f..e17771f 100644 --- a/drill/RUNS.md +++ b/drill/RUNS.md @@ -195,6 +195,7 @@ No listener is needed, and none should be started: see trap 3. | Run | Result | What it cost | | --- | --- | --- | +| MU-5 | 53/1 → 54/0 (containers) | the #75 review round, measured in: criterion (m) — a raw `--network boxnet` attach keeps egress, the ACL, both sibling-drop directions and dns.mode=none, losing only per-NIC port_isolation (the scoped guarantee, now measured, not prose) — and criterion (n), both grant-failure injections (fresh user backed out with the removal VERIFIED; pre-existing member warned loudly, never stripped; re-runs converge). The one instrument bug: (m) first minted the non-cloud image — no DHCP client, no lease, and a dead NIC passes every negative probe vacuously. The probe now requires the lease before it believes any of its own answers | | MU-4 | 41/1 on CI (ubuntu-latest, incus 6.0.0) → 42/0 | the CI job earned its keep on its second-ever run: the attach-refusal probe matched 6.0.4's error wording and 6.0.0 words it differently (and refuses at a different stage) — a version-drift false FAIL the design host could never show. The probe now asserts the OUTCOME (nothing running on the private bridge) and prints the refusal as evidence. Also this round, from adversarial review: revoke survives the live-session case (groups are read at login; --purge now terminates sessions first, else a stale-group process could recreate the project UNHARDENED after the purge), grant backs out its own group-add on failure, and the session-dies-with-the-tier check is criterion 42 | | **MU-3** | **41/0 (containers) + 41/0 (VMs) — the tier holds** | `multiuser.sh` (#74) from zero on Debian 13 / Incus 6.0.4: grant, confinement, lifecycle, the in-box contract with the cross-user sibling drop, closed escape hatches, re-sync survival, scoped revoke. The VM pass needed right-sized boxes (1GiB): two 2GiB nested VMs on a 7.7GiB host measured swap, not isolation | | MU-2 | 40/41 (VMs) | the egress probe cried broken over a TLS handshake that outlived its 8s budget under nested-KVM memory pressure — every isolation probe (which PASSES on silence) was immune. Probe budgets raised, reachable-probes retry once, and the mint keeps its log for the post-mortem | diff --git a/drill/doctor.sh b/drill/doctor.sh index 081b342..f8de671 100755 --- a/drill/doctor.sh +++ b/drill/doctor.sh @@ -253,7 +253,7 @@ done # An interrupted multiuser.sh leaves its users (and their projects) behind — # and nothing else on this host will ever mention them. Its own cleanup is # 'box revoke --purge + userdel'; say so rather than absorbing them silently. -for u in boxdrill1 boxdrill2; do +for u in boxdrill1 boxdrill2 boxdrill3 boxdrill4; do if getent passwd "$u" >/dev/null 2>&1; then no "leftover rehearsal user: $u (an interrupted drill/multiuser.sh run)" inf "fix: sudo BOX_YES=1 box revoke $u --purge && sudo userdel -r $u" diff --git a/drill/multiuser.sh b/drill/multiuser.sh index 1a59bda..9143286 100644 --- a/drill/multiuser.sh +++ b/drill/multiuser.sh @@ -26,6 +26,10 @@ # i. (folded into b: snapshot / restore / clone) # k. the grant survives an incus-user restart # l. box revoke --purge removes the user's world and touches nobody else's +# m. a RAW attach to boxnet (no box-net profile) keeps every network- and +# host-owned control — the scoped guarantee, measured (#75 review) +# n. a grant that fails is fail-closed: fresh user backed out (verified), +# pre-existing member warned loudly, re-run converges (#75 review) # # ok/no/note return 0 by design — the 'A && ok || no' idiom below is the # same one drill.sh is built on (and the reason for the SC2015 disable). @@ -105,7 +109,7 @@ cleanup() { [ "$KEEP" = 1 ] && { echo "(--keep: users and boxes left for inspection)"; return; } echo echo "── cleanup" - for u in "$U1" "$U2"; do + for u in "$U1" "$U2" boxdrill3 boxdrill4; do id "$u" >/dev/null 2>&1 || continue # A half-failed purge followed by userdel leaves a project owned by # nobody — and doctor's leftover check keys on the USER existing. Keep @@ -277,6 +281,56 @@ else no "(h) boxnet edit attempt: rc=$rc, said: $(printf '%s' "$out" | head -1)" fi +phase "m. a raw attach to boxnet — the scoped guarantee, measured" +# A restricted user CAN 'incus launch --network boxnet' without the box-net +# profile: boxnet must be in restricted.networks.access for the profile to +# work at all, and Incus has no allow-via-profile-only lever. What the raw +# NIC loses is per-NIC security.port_isolation — the deliberately redundant +# L2 twin of the host-owned nft bridge drop. Everything else binds to the +# NETWORK (ACL, dns.mode=none, resolver pin) or the HOST (nft drop), so the +# contract's claim for raw attachments is "every control except the +# redundant per-NIC layer" — and a claim is a measurement here, not prose. +# Same image the blank template mints (the /cloud variant): the plain image +# has no DHCP client, so its raw instance holds NO lease — and against a +# dead NIC every negative probe below "passes" vacuously while the contract +# goes unmeasured. Caught on this criterion's first run (MU-5). +if as_u "$U1" incus launch images:debian/13/cloud esc2 --network boxnet >/dev/null 2>&1; then + ok "(m) raw attach to boxnet launches (expected: the network must be usable for the profile to work)" + ip_raw="" + for _ in $(seq 1 45); do + ip_raw="$(incus --project "$p1" list esc2 --format csv --columns 4 2>/dev/null | tr -d '"' | sed 's/ (.*//' | grep . | head -n1)" + [ -n "$ip_raw" ] && as_u "$U1" timeout -k 5 15 incus exec esc2 -- true >/dev/null 2>&1 && break + sleep 2 + done + inf "raw instance esc2: ${ip_raw:-}" + if [ -z "$ip_raw" ]; then + # Without an address the negative probes below would all pass vacuously + # — a dead NIC drops everything, including the truth. + no "(m) the raw instance never got a boxnet lease — the scoped guarantee went UNMEASURED" + else + r="$(probe_up "$U1" esc2 https://1.1.1.1)" + [ "$r" = reachable ] && ok "(m) raw NIC: public egress works ($r)" || no "(m) raw NIC: egress broken: $r" + r="$(probe_from "$U1" esc2 "http://192.168.0.1")" + [ "$r" = dropped ] && ok "(m) raw NIC: RFC1918 still dropped (the ACL binds to the network, not the profile)" \ + || no "(m) raw NIC: reaches private space ($r) — the ACL did not cover a raw attach" + if [ -n "$ip2" ]; then + r="$(probe_from "$U1" esc2 "http://$ip2:9")" + [ "$r" = dropped ] && ok "(m) raw → another user's box is DROPPED (the nft drop is host-owned)" \ + || no "(m) raw instance reached a sibling ($r) — the host drop did not cover it" + fi + r="$(probe_from "$U2" mine "http://$ip_raw:9")" + [ "$r" = dropped ] && ok "(m) another user's box → raw is DROPPED (both directions hold)" \ + || no "(m) a sibling reached the raw instance ($r)" + as_u "$U1" timeout -k 5 20 incus exec esc2 -- getent hosts mine >/dev/null 2>&1 \ + && no "(m) raw NIC can enumerate instance names (dns.mode leak)" \ + || ok "(m) raw NIC: name enumeration still blocked (dns.mode=none is the network's)" + fi + as_u "$U1" incus delete -f esc2 >/dev/null 2>&1 + aud "m. raw boxnet attach keeps ACL + nft drop + dns.mode (measured); loses only per-NIC port_isolation — the scoped guarantee in box-design.md" +else + no "(m) raw attach to boxnet failed to launch — the scoped-guarantee measurement could not run" +fi + phase "e/f. the honest refusals — expose, setup-host, doctor" out="$(as_u "$U1" box expose mine 3000 2>&1)"; rc=$? [ "$rc" -ne 0 ] && printf '%s' "$out" | grep -qi restricted \ @@ -325,6 +379,63 @@ st="$(incus --project "$p1" list mine --format csv --columns s 2>/dev/null | hea || no "(l) $U1's box state after $U2's purge: '$st'" aud "l. revoke --purge is scoped: $U2 erased, $U1 unmoved" +phase "n. a grant that fails is fail-closed — injected, both flavors" +U3=boxdrill3; U4=boxdrill4 +BOXROOT="$(dirname "$(dirname "$(readlink -f "$(command -v box)")")")" + +# Flavor 1: a FRESH user, fault injected at the LAST mutation (the profile +# edit) — so the backout runs after every earlier mutation has landed. The +# contract: nonzero exit, the group's absence VERIFIED, and a clean re-run +# converges the partial state (which is what makes re-run-to-repair real). +useradd -m -s /bin/bash "$U3" 2>/dev/null +badroot="$(mktemp -d)" +cp -r "$BOXROOT/." "$badroot/" +echo 'devices: {' > "$badroot/profiles/box-net.yaml" # yaml that cannot load +out="$(bash "$badroot/host/grant-user.sh" "$U3" 2>&1)"; rc=$? +rm -rf "$badroot" +if [ "$rc" -ne 0 ] && ! id -nG "$U3" | tr ' ' '\n' | grep -qx incus; then + ok "(n) fresh-user grant failed at the last mutation → backed out, group absence verified (rc=$rc)" +else + no "(n) injected failure: rc=$rc, in-group=$(id -nG "$U3" | tr ' ' '\n' | grep -cx incus) — not fail-closed:" + printf '%s\n' "$out" | tail -3 | sed 's/^/ /' +fi +printf '%s' "$out" | grep -q "verified against the group database" \ + && ok "(n) the backout message claims only what it verified" \ + || no "(n) the backout message is not the verified one" +box grant "$U3" >/dev/null 2>&1 \ + && ok "(n) a clean re-run converges the partial state left by the failure" \ + || no "(n) re-run after injected failure did NOT converge" + +# Flavor 2: a PRE-EXISTING member (hand-added before box, the review's named +# scenario) with an instance parked on the private bridge by an +# instance-local NIC — narrowing must fail, the grant must fail LOUDLY +# saying they retain socket access, and must NOT strip the membership this +# run did not add. Unblock, re-run, converge. +useradd -m -s /bin/bash "$U4" 2>/dev/null +usermod -aG incus "$U4" +as_u "$U4" incus project list >/dev/null 2>&1 # materialize their project +uid4="$(id -u "$U4")" +br4="incusbr-$uid4"; [ "${#br4}" -gt 15 ] && br4="user-$uid4" +if as_u "$U4" incus launch images:debian/13 blocker --network "$br4" >/dev/null 2>&1; then + out="$(box grant "$U4" 2>&1)"; rc=$? + if [ "$rc" -ne 0 ] && printf '%s' "$out" | grep -q "still holding socket access"; then + ok "(n) blocked narrowing fails LOUDLY, naming the retained access (rc=$rc)" + else + no "(n) blocked narrowing: rc=$rc — the loud contract is missing:" + printf '%s\n' "$out" | tail -3 | sed 's/^/ /' + fi + id -nG "$U4" | tr ' ' '\n' | grep -qx incus \ + && ok "(n) the pre-existing membership was NOT stripped by the failed re-grant" \ + || no "(n) the failed grant stripped a membership it did not add" + as_u "$U4" incus delete -f blocker >/dev/null 2>&1 + box grant "$U4" >/dev/null 2>&1 \ + && ok "(n) unblocked re-run converges" \ + || no "(n) re-run after unblocking failed" +else + no "(n) could not stage the private-bridge blocker — the blocked-narrowing contract went unmeasured" +fi +aud "n. fail-closed injections: fresh-user backout verified; pre-existing member warned, not stripped; re-runs converge" + echo echo "════════════════════════════════════════════" echo " $pass passed, $fail failed" diff --git a/host/grant-user.sh b/host/grant-user.sh index 8f4802c..82ee1ab 100644 --- a/host/grant-user.sh +++ b/host/grant-user.sh @@ -93,16 +93,40 @@ fi # fresh grant (their existing sessions predate the membership, so no process # holds it yet). A user who was already in the group keeps it: not ours to # take on a re-run's failure. -added_group=0 +added_group=0; was_member=0 backout() { if [ "$added_group" -eq 1 ]; then $SUDO gpasswd -d "$user" incus >/dev/null 2>&1 || true - echo "box grant: FAILED — removed $user from 'incus' again (no half-granted access left behind); fix the cause and re-run" >&2 + # VERIFY the removal — an unverified rollback printing a security + # guarantee is a lie waiting for its day. Exact-token match, live DB. + if id -nG "$user" 2>/dev/null | tr ' ' '\n' | grep -qx incus; then + echo "box grant: ROLLBACK INCOMPLETE — the grant failed AND $user is still in the 'incus' group." >&2 + echo " remove it by hand NOW: gpasswd -d $user incus (then fix the cause and re-run)" >&2 + exit 1 + fi + echo "box grant: FAILED — removed $user from 'incus' again (verified against the group database); fix the cause and re-run" >&2 + # The one window the database cannot close: a login STARTED between our + # usermod and this backout keeps the group in its session credentials. + # For a fresh grant that is a rare race, but rare is not never — name it + # and the remedy instead of overclaiming. + if pgrep -u "$user" >/dev/null 2>&1; then + echo "box grant: NOTE — $user has live processes; a session begun during this grant would still hold" >&2 + echo " the group until it ends: sudo loginctl terminate-user $user" >&2 + fi + elif [ "$was_member" -eq 1 ]; then + # A user who was ALREADY in the group keeps it — stripping a membership + # this run did not add could break a working user over a failed re-run. + # But silence here would leave them holding a socket onto part-converged + # policy without the admin being told. Loud, with both remediations. + echo "box grant: FAILED with $user still holding socket access (their membership predates this run)." >&2 + echo " their project may be part-converged — harmless in itself, and a re-run converges the rest." >&2 + echo " if their access is not acceptable while you fix the cause: box revoke $user" >&2 fi } trap backout EXIT if id -nG "$user" | tr ' ' '\n' | grep -qx incus; then + was_member=1 echo "group: $user already in 'incus'" else $SUDO usermod -aG incus "$user" diff --git a/test/cli.sh b/test/cli.sh index feadf96..12e113b 100644 --- a/test/cli.sh +++ b/test/cli.sh @@ -234,6 +234,22 @@ check "grant: backs out its own group-add on failure" 0 "" \ grep -qF 'trap backout EXIT' "$ROOT/host/grant-user.sh" check "grant: the back-out disarms on success" 0 "" \ grep -qF 'trap - EXIT' "$ROOT/host/grant-user.sh" +# The backout must VERIFY the removal and scream when it cannot — an +# unverified rollback printing a security guarantee is the review's A2. +check "grant: the backout verifies against the group database" 0 "" \ + bash -c 'awk "/^backout\(\) \{/,/^\}/" "'"$ROOT"'/host/grant-user.sh" | grep -q "id -nG"' +check "grant: an unverifiable rollback screams" 0 "" \ + grep -qF 'ROLLBACK INCOMPLETE' "$ROOT/host/grant-user.sh" +check "grant: a failed re-grant warns the pre-existing member is untouched" 0 "" \ + grep -qF 'still holding socket access' "$ROOT/host/grant-user.sh" +check "grant: the mid-grant login window is named" 0 "" \ + bash -c 'awk "/^backout\(\) \{/,/^\}/" "'"$ROOT"'/host/grant-user.sh" | grep -q "loginctl terminate-user"' +# The scoped guarantee (raw --network boxnet) is measured, not prose: +check "rehearsal: measures the raw boxnet attach (criterion m)" 0 "" \ + grep -qF -- '--network boxnet' "$ROOT/drill/multiuser.sh" +# shellcheck disable=SC2016 # the $-string is a literal in the target file +check "rehearsal: injects grant failures (criterion n)" 0 "" \ + grep -qF 'grant-user.sh" "$U3"' "$ROOT/drill/multiuser.sh" # shellcheck disable=SC2016 # the $-strings are literals in the target file check "revoke: purge deletes instances one at a time" 0 "" \ grep -qF 'delete -f "$inst"' "$ROOT/host/revoke-user.sh"