grant/rehearsal: the codex round — verified rollback, loud partial states, and the raw-attach guarantee measured (#75)

Review 4727756972 (A2): the backout no longer trusts gpasswd — it re-reads
the live group database after removal; verified-absent gets the safe
message, anything else screams ROLLBACK INCOMPLETE, exits nonzero, and
names the exact remediation. The concurrent-login window (a session begun
between usermod and backout keeps the group) is CLOSED to the extent the
database can't reach: the backout detects live processes and names
loginctl terminate-user, and the success wording claims only what was
verified.

Review 4727641752 (A1): a failed grant for a user whose membership predates
the run (the hand-added-user scenario) now fails LOUDLY — they retain
socket access on part-converged policy, and the message says so with both
remediations (box revoke now, or fix and re-run). Their membership is not
stripped: breaking a working user over a failed re-grant is its own hazard.
The default-profile eth0 removal is deliberately not restored on failure —
that mutation only reduces capability, and restoring it would move the
failure state AWAY from fail-closed. Injected-failure coverage is criterion
(n), both flavors: fresh-user backout (fault at the LAST mutation, so the
rollback runs after every earlier one) with the group's absence verified
and a converging re-run; blocked narrowing staged for real with an
instance-local NIC parked on the private bridge.

Review A3, resolution 3 with the measurement demanded: criterion (m)
launches exactly 'incus launch --network boxnet' as the restricted user and
probes the raw NIC from inside — egress works, RFC1918 dropped (the ACL is
the network's), sibling probes dropped BOTH directions (the nft drop is the
host's), name enumeration blocked. The scoped guarantee is now stated in
box-design.md and measured on every run: box-minted instances carry per-NIC
port_isolation; raw attachments keep every network- and host-owned control,
losing only that redundant L2 layer. Instrument lesson kept as MU-5: the
probe's first cut minted the non-cloud image — no DHCP client, no lease,
and a dead NIC passes every negative probe vacuously; it now requires the
lease before believing its own answers.

Rehearsal: 54/54 (containers). test/cli.sh: 82 checks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -20,10 +20,10 @@ which records not just what changed but what each drill run proved.
   into their project. `box_tier()` (live credentials, argless `id -nG`)
   drives the tier-aware surface: `expose` refuses honestly before any daemon
   call, `setup-host` and `doctor` answer at the caller's tier. Rehearsed
-  end-to-end by `drill/multiuser.sh` (criteria a–l: confinement, lifecycle,
+  end-to-end by `drill/multiuser.sh` (criteria a–n: confinement, lifecycle,
   cross-user visibility, name collisions, the in-box isolation contract,
   escape hatches, re-sync survival, revoke incl. the live-session case) —
-  42/42 on the design host, in both container and VM mode.
+  54/54 on the design host (container and VM mode), including the raw-attach scoped-guarantee measurement and both grant-failure injections demanded by #75's review.
 - **CI runs the multi-user rehearsal on a real Incus** — a second `rehearsal`
   job stands up the full stack on the runner (setup-host, doctor, then
   `multiuser.sh --container`), so every PR proves the tier's semantics

docs/box-design.md

@@ -165,9 +165,16 @@ Cross-USER isolation is the same mechanism as cross-box isolation, on
 purpose: their instances share `boxnet` with everyone's, and the bridge-family
 drop + port isolation + `dns.mode=none` already make any two boxes strangers.
 A restricted user CAN strip `security.port_isolation` from the profile copy
-in their own project — which is why the nft bridge drop, which they cannot
+in their own project — or skip the profile entirely and attach `boxnet` raw
-touch, exists as the second, host-owned layer. Defense in depth, both layers
+(`--network boxnet`); the network must be usable for the profile to work, and
-measured (`drill/multiuser.sh`, criteria a–l).
+Incus has no allow-via-profile-only lever. So the guarantee is scoped, and
+said plainly: **per-NIC port isolation is guaranteed for box-minted
+instances; a raw attachment keeps every network-owned control (the ACL,
+`dns.mode=none`, the resolver pin) and every host-owned one (the nft bridge
+drop) — losing only the redundant per-NIC L2 layer.** Scoped, and measured:
+`drill/multiuser.sh` criterion (m) launches exactly that raw instance and
+probes egress, RFC1918, both sibling directions and name enumeration from
+inside it. Defense in depth, every layer measured (criteria a–n).
 
 `box revoke` is two strengths: bare, it removes the group — their boxes keep
 *running* (revoking a person does not kill their workloads), `grant` restores

docs/plans/2026-07-18-restricted-tier.md

@@ -1,6 +1,6 @@
 # Restricted incus tier — design and measured results (#74)
 
-**Status: implemented and rehearsed.** 42/42 rehearsal criteria green on the
+**Status: implemented and rehearsed.** 54/54 rehearsal criteria green on the
 design host (Debian 13 trixie, Incus 6.0.4, nested KVM), in container mode
 and VM mode — and green in CI on ubuntu-latest / incus 6.0.0 (whose one
 version-drift false FAIL is MU-4 in `drill/RUNS.md`). This doc records the design, what was measured, and why each
@@ -69,10 +69,27 @@ touch `boxnet`'s config or the ACL (`no permission for project "default"`).
 
 A restricted user CAN edit the `box-net` profile copy in their own project
 (they own project profiles — `features.profiles=true`), including stripping
-`security.port_isolation`. That is why the host-owned nft bridge drop is the
+`security.port_isolation` — and CAN attach `boxnet` raw with `--network
-second layer: `meta ibrname boxnet obrname boxnet drop` fires on every
+boxnet`, no profile at all (the network must be in
-port-to-port frame regardless of per-NIC flags. Cross-user sibling probes are
+`restricted.networks.access` for the profile to work; there is no
-dropped either way — measured from inside the boxes.
+allow-via-profile-only lever). That is why the host-owned nft bridge drop is
+the second layer: `meta ibrname boxnet obrname boxnet drop` fires on every
+port-to-port frame regardless of per-NIC flags. The documented guarantee is
+scoped accordingly (see box-design.md): box-minted instances carry per-NIC
+port isolation; raw attachments keep every network- and host-owned control,
+losing only that redundant L2 layer. Both shapes are measured from inside
+the instances (rehearsal criteria g and m).
+
+Two grant-failure contracts, both injected in the rehearsal (criterion n):
+a fresh user is backed out of the group with the removal VERIFIED against
+the live group database (and any session begun mid-grant is named, with the
+loginctl remedy — the one window the database cannot close); a pre-existing
+member is never stripped by a failed re-grant, but the failure states out
+loud that they retain socket access on part-converged policy, with both
+remediations. The default-profile eth0 removal is deliberately NOT restored
+on failure: that mutation only reduces capability, and restoring it would
+move the failure state away from fail-closed. Every step is check-then-
+converge, which is what makes re-run-to-repair deterministic.
 
 ## What `box grant <user>` converges (idempotent, re-run to refresh)
 
@@ -110,10 +127,11 @@ It had never worked.
 ## Rehearsal and CI
 
 `drill/multiuser.sh` (root, opt-in via `BOX_MULTIUSER_REHEARSAL=1`) proves
-criteria (a)–(f) from #74 plus the measured extensions (g)–(l): the in-box
+criteria (a)–(f) from #74 plus the measured extensions (g)–(n): the in-box
 isolation contract (egress, DNS, box→host, RFC1918, cross-user sibling drop,
 name enumeration, IPv6-off), the closed escape hatches, re-sync survival, and
-scoped revoke. Two real users, real grants, real mints, probes from inside;
+scoped revoke, the raw-attach scoped guarantee (m) and the grant-failure
+injections (n). Real users, real grants, real mints, probes from inside;
 `--container` for CI, VM mode on real hardware; cleanup deletes everything it
 made.
 

drill/RUNS.md

@@ -195,6 +195,7 @@ No listener is needed, and none should be started: see trap 3.
 
 | Run | Result | What it cost |
 | --- | --- | --- |
+| MU-5 | 53/1 → 54/0 (containers) | the #75 review round, measured in: criterion (m) — a raw `--network boxnet` attach keeps egress, the ACL, both sibling-drop directions and dns.mode=none, losing only per-NIC port_isolation (the scoped guarantee, now measured, not prose) — and criterion (n), both grant-failure injections (fresh user backed out with the removal VERIFIED; pre-existing member warned loudly, never stripped; re-runs converge). The one instrument bug: (m) first minted the non-cloud image — no DHCP client, no lease, and a dead NIC passes every negative probe vacuously. The probe now requires the lease before it believes any of its own answers |
 | MU-4 | 41/1 on CI (ubuntu-latest, incus 6.0.0) → 42/0 | the CI job earned its keep on its second-ever run: the attach-refusal probe matched 6.0.4's error wording and 6.0.0 words it differently (and refuses at a different stage) — a version-drift false FAIL the design host could never show. The probe now asserts the OUTCOME (nothing running on the private bridge) and prints the refusal as evidence. Also this round, from adversarial review: revoke survives the live-session case (groups are read at login; --purge now terminates sessions first, else a stale-group process could recreate the project UNHARDENED after the purge), grant backs out its own group-add on failure, and the session-dies-with-the-tier check is criterion 42 |
 | **MU-3** | **41/0 (containers) + 41/0 (VMs) — the tier holds** | `multiuser.sh` (#74) from zero on Debian 13 / Incus 6.0.4: grant, confinement, lifecycle, the in-box contract with the cross-user sibling drop, closed escape hatches, re-sync survival, scoped revoke. The VM pass needed right-sized boxes (1GiB): two 2GiB nested VMs on a 7.7GiB host measured swap, not isolation |
 | MU-2 | 40/41 (VMs) | the egress probe cried broken over a TLS handshake that outlived its 8s budget under nested-KVM memory pressure — every isolation probe (which PASSES on silence) was immune. Probe budgets raised, reachable-probes retry once, and the mint keeps its log for the post-mortem |

drill/doctor.sh

@@ -253,7 +253,7 @@ done
 # An interrupted multiuser.sh leaves its users (and their projects) behind —
 # and nothing else on this host will ever mention them. Its own cleanup is
 # 'box revoke --purge + userdel'; say so rather than absorbing them silently.
-for u in boxdrill1 boxdrill2; do
+for u in boxdrill1 boxdrill2 boxdrill3 boxdrill4; do
   if getent passwd "$u" >/dev/null 2>&1; then
     no "leftover rehearsal user: $u (an interrupted drill/multiuser.sh run)"
     inf "fix:  sudo BOX_YES=1 box revoke $u --purge && sudo userdel -r $u"

drill/multiuser.sh

@@ -26,6 +26,10 @@
 #   i. (folded into b: snapshot / restore / clone)
 #   k. the grant survives an incus-user restart
 #   l. box revoke --purge removes the user's world and touches nobody else's
+#   m. a RAW attach to boxnet (no box-net profile) keeps every network- and
+#      host-owned control — the scoped guarantee, measured (#75 review)
+#   n. a grant that fails is fail-closed: fresh user backed out (verified),
+#      pre-existing member warned loudly, re-run converges (#75 review)
 #
 # ok/no/note return 0 by design — the 'A && ok || no' idiom below is the
 # same one drill.sh is built on (and the reason for the SC2015 disable).
@@ -105,7 +109,7 @@ cleanup() {
   [ "$KEEP" = 1 ] && { echo "(--keep: users and boxes left for inspection)"; return; }
   echo
   echo "── cleanup"
-  for u in "$U1" "$U2"; do
+  for u in "$U1" "$U2" boxdrill3 boxdrill4; do
     id "$u" >/dev/null 2>&1 || continue
     # A half-failed purge followed by userdel leaves a project owned by
     # nobody — and doctor's leftover check keys on the USER existing. Keep
@@ -277,6 +281,56 @@ else
   no "(h) boxnet edit attempt: rc=$rc, said: $(printf '%s' "$out" | head -1)"
 fi
 
+phase "m. a raw attach to boxnet — the scoped guarantee, measured"
+# A restricted user CAN 'incus launch --network boxnet' without the box-net
+# profile: boxnet must be in restricted.networks.access for the profile to
+# work at all, and Incus has no allow-via-profile-only lever. What the raw
+# NIC loses is per-NIC security.port_isolation — the deliberately redundant
+# L2 twin of the host-owned nft bridge drop. Everything else binds to the
+# NETWORK (ACL, dns.mode=none, resolver pin) or the HOST (nft drop), so the
+# contract's claim for raw attachments is "every control except the
+# redundant per-NIC layer" — and a claim is a measurement here, not prose.
+# Same image the blank template mints (the /cloud variant): the plain image
+# has no DHCP client, so its raw instance holds NO lease — and against a
+# dead NIC every negative probe below "passes" vacuously while the contract
+# goes unmeasured. Caught on this criterion's first run (MU-5).
+if as_u "$U1" incus launch images:debian/13/cloud esc2 --network boxnet >/dev/null 2>&1; then
+  ok "(m) raw attach to boxnet launches (expected: the network must be usable for the profile to work)"
+  ip_raw=""
+  for _ in $(seq 1 45); do
+    ip_raw="$(incus --project "$p1" list esc2 --format csv --columns 4 2>/dev/null | tr -d '"' | sed 's/ (.*//' | grep . | head -n1)"
+    [ -n "$ip_raw" ] && as_u "$U1" timeout -k 5 15 incus exec esc2 -- true >/dev/null 2>&1 && break
+    sleep 2
+  done
+  inf "raw instance esc2: ${ip_raw:-<no ip>}"
+  if [ -z "$ip_raw" ]; then
+    # Without an address the negative probes below would all pass vacuously
+    # — a dead NIC drops everything, including the truth.
+    no "(m) the raw instance never got a boxnet lease — the scoped guarantee went UNMEASURED"
+  else
+    r="$(probe_up "$U1" esc2 https://1.1.1.1)"
+    [ "$r" = reachable ] && ok "(m) raw NIC: public egress works ($r)" || no "(m) raw NIC: egress broken: $r"
+    r="$(probe_from "$U1" esc2 "http://192.168.0.1")"
+    [ "$r" = dropped ] && ok "(m) raw NIC: RFC1918 still dropped (the ACL binds to the network, not the profile)" \
+                       || no "(m) raw NIC: reaches private space ($r) — the ACL did not cover a raw attach"
+    if [ -n "$ip2" ]; then
+      r="$(probe_from "$U1" esc2 "http://$ip2:9")"
+      [ "$r" = dropped ] && ok "(m) raw → another user's box is DROPPED (the nft drop is host-owned)" \
+                         || no "(m) raw instance reached a sibling ($r) — the host drop did not cover it"
+    fi
+    r="$(probe_from "$U2" mine "http://$ip_raw:9")"
+    [ "$r" = dropped ] && ok "(m) another user's box → raw is DROPPED (both directions hold)" \
+                       || no "(m) a sibling reached the raw instance ($r)"
+    as_u "$U1" timeout -k 5 20 incus exec esc2 -- getent hosts mine >/dev/null 2>&1 \
+      && no "(m) raw NIC can enumerate instance names (dns.mode leak)" \
+      || ok "(m) raw NIC: name enumeration still blocked (dns.mode=none is the network's)"
+  fi
+  as_u "$U1" incus delete -f esc2 >/dev/null 2>&1
+  aud "m. raw boxnet attach keeps ACL + nft drop + dns.mode (measured); loses only per-NIC port_isolation — the scoped guarantee in box-design.md"
+else
+  no "(m) raw attach to boxnet failed to launch — the scoped-guarantee measurement could not run"
+fi
+
 phase "e/f. the honest refusals — expose, setup-host, doctor"
 out="$(as_u "$U1" box expose mine 3000 2>&1)"; rc=$?
 [ "$rc" -ne 0 ] && printf '%s' "$out" | grep -qi restricted \
@@ -325,6 +379,63 @@ st="$(incus --project "$p1" list mine --format csv --columns s 2>/dev/null | hea
                     || no "(l) $U1's box state after $U2's purge: '$st'"
 aud "l. revoke --purge is scoped: $U2 erased, $U1 unmoved"
 
+phase "n. a grant that fails is fail-closed — injected, both flavors"
+U3=boxdrill3; U4=boxdrill4
+BOXROOT="$(dirname "$(dirname "$(readlink -f "$(command -v box)")")")"
+
+# Flavor 1: a FRESH user, fault injected at the LAST mutation (the profile
+# edit) — so the backout runs after every earlier mutation has landed. The
+# contract: nonzero exit, the group's absence VERIFIED, and a clean re-run
+# converges the partial state (which is what makes re-run-to-repair real).
+useradd -m -s /bin/bash "$U3" 2>/dev/null
+badroot="$(mktemp -d)"
+cp -r "$BOXROOT/." "$badroot/"
+echo 'devices: {' > "$badroot/profiles/box-net.yaml"   # yaml that cannot load
+out="$(bash "$badroot/host/grant-user.sh" "$U3" 2>&1)"; rc=$?
+rm -rf "$badroot"
+if [ "$rc" -ne 0 ] && ! id -nG "$U3" | tr ' ' '\n' | grep -qx incus; then
+  ok "(n) fresh-user grant failed at the last mutation → backed out, group absence verified (rc=$rc)"
+else
+  no "(n) injected failure: rc=$rc, in-group=$(id -nG "$U3" | tr ' ' '\n' | grep -cx incus) — not fail-closed:"
+  printf '%s\n' "$out" | tail -3 | sed 's/^/        /'
+fi
+printf '%s' "$out" | grep -q "verified against the group database" \
+  && ok "(n) the backout message claims only what it verified" \
+  || no "(n) the backout message is not the verified one"
+box grant "$U3" >/dev/null 2>&1 \
+  && ok "(n) a clean re-run converges the partial state left by the failure" \
+  || no "(n) re-run after injected failure did NOT converge"
+
+# Flavor 2: a PRE-EXISTING member (hand-added before box, the review's named
+# scenario) with an instance parked on the private bridge by an
+# instance-local NIC — narrowing must fail, the grant must fail LOUDLY
+# saying they retain socket access, and must NOT strip the membership this
+# run did not add. Unblock, re-run, converge.
+useradd -m -s /bin/bash "$U4" 2>/dev/null
+usermod -aG incus "$U4"
+as_u "$U4" incus project list >/dev/null 2>&1   # materialize their project
+uid4="$(id -u "$U4")"
+br4="incusbr-$uid4"; [ "${#br4}" -gt 15 ] && br4="user-$uid4"
+if as_u "$U4" incus launch images:debian/13 blocker --network "$br4" >/dev/null 2>&1; then
+  out="$(box grant "$U4" 2>&1)"; rc=$?
+  if [ "$rc" -ne 0 ] && printf '%s' "$out" | grep -q "still holding socket access"; then
+    ok "(n) blocked narrowing fails LOUDLY, naming the retained access (rc=$rc)"
+  else
+    no "(n) blocked narrowing: rc=$rc — the loud contract is missing:"
+    printf '%s\n' "$out" | tail -3 | sed 's/^/        /'
+  fi
+  id -nG "$U4" | tr ' ' '\n' | grep -qx incus \
+    && ok "(n) the pre-existing membership was NOT stripped by the failed re-grant" \
+    || no "(n) the failed grant stripped a membership it did not add"
+  as_u "$U4" incus delete -f blocker >/dev/null 2>&1
+  box grant "$U4" >/dev/null 2>&1 \
+    && ok "(n) unblocked re-run converges" \
+    || no "(n) re-run after unblocking failed"
+else
+  no "(n) could not stage the private-bridge blocker — the blocked-narrowing contract went unmeasured"
+fi
+aud "n. fail-closed injections: fresh-user backout verified; pre-existing member warned, not stripped; re-runs converge"
+
 echo
 echo "════════════════════════════════════════════"
 echo "  $pass passed, $fail failed"

host/grant-user.sh

@@ -93,16 +93,40 @@ fi
 # fresh grant (their existing sessions predate the membership, so no process
 # holds it yet). A user who was already in the group keeps it: not ours to
 # take on a re-run's failure.
-added_group=0
+added_group=0; was_member=0
 backout() {
   if [ "$added_group" -eq 1 ]; then
     $SUDO gpasswd -d "$user" incus >/dev/null 2>&1 || true
-    echo "box grant: FAILED — removed $user from 'incus' again (no half-granted access left behind); fix the cause and re-run" >&2
+    # VERIFY the removal — an unverified rollback printing a security
+    # guarantee is a lie waiting for its day. Exact-token match, live DB.
+    if id -nG "$user" 2>/dev/null | tr ' ' '\n' | grep -qx incus; then
+      echo "box grant: ROLLBACK INCOMPLETE — the grant failed AND $user is still in the 'incus' group." >&2
+      echo "           remove it by hand NOW:  gpasswd -d $user incus   (then fix the cause and re-run)" >&2
+      exit 1
+    fi
+    echo "box grant: FAILED — removed $user from 'incus' again (verified against the group database); fix the cause and re-run" >&2
+    # The one window the database cannot close: a login STARTED between our
+    # usermod and this backout keeps the group in its session credentials.
+    # For a fresh grant that is a rare race, but rare is not never — name it
+    # and the remedy instead of overclaiming.
+    if pgrep -u "$user" >/dev/null 2>&1; then
+      echo "box grant: NOTE — $user has live processes; a session begun during this grant would still hold" >&2
+      echo "           the group until it ends:  sudo loginctl terminate-user $user" >&2
+    fi
+  elif [ "$was_member" -eq 1 ]; then
+    # A user who was ALREADY in the group keeps it — stripping a membership
+    # this run did not add could break a working user over a failed re-run.
+    # But silence here would leave them holding a socket onto part-converged
+    # policy without the admin being told. Loud, with both remediations.
+    echo "box grant: FAILED with $user still holding socket access (their membership predates this run)." >&2
+    echo "           their project may be part-converged — harmless in itself, and a re-run converges the rest." >&2
+    echo "           if their access is not acceptable while you fix the cause:  box revoke $user" >&2
   fi
 }
 trap backout EXIT
 
 if id -nG "$user" | tr ' ' '\n' | grep -qx incus; then
+  was_member=1
   echo "group: $user already in 'incus'"
 else
   $SUDO usermod -aG incus "$user"

test/cli.sh

@@ -234,6 +234,22 @@ check "grant: backs out its own group-add on failure" 0 "" \
   grep -qF 'trap backout EXIT' "$ROOT/host/grant-user.sh"
 check "grant: the back-out disarms on success" 0 "" \
   grep -qF 'trap - EXIT' "$ROOT/host/grant-user.sh"
+# The backout must VERIFY the removal and scream when it cannot — an
+# unverified rollback printing a security guarantee is the review's A2.
+check "grant: the backout verifies against the group database" 0 "" \
+  bash -c 'awk "/^backout\(\) \{/,/^\}/" "'"$ROOT"'/host/grant-user.sh" | grep -q "id -nG"'
+check "grant: an unverifiable rollback screams" 0 "" \
+  grep -qF 'ROLLBACK INCOMPLETE' "$ROOT/host/grant-user.sh"
+check "grant: a failed re-grant warns the pre-existing member is untouched" 0 "" \
+  grep -qF 'still holding socket access' "$ROOT/host/grant-user.sh"
+check "grant: the mid-grant login window is named" 0 "" \
+  bash -c 'awk "/^backout\(\) \{/,/^\}/" "'"$ROOT"'/host/grant-user.sh" | grep -q "loginctl terminate-user"'
+# The scoped guarantee (raw --network boxnet) is measured, not prose:
+check "rehearsal: measures the raw boxnet attach (criterion m)" 0 "" \
+  grep -qF -- '--network boxnet' "$ROOT/drill/multiuser.sh"
+# shellcheck disable=SC2016  # the $-string is a literal in the target file
+check "rehearsal: injects grant failures (criterion n)" 0 "" \
+  grep -qF 'grant-user.sh" "$U3"' "$ROOT/drill/multiuser.sh"
 # shellcheck disable=SC2016  # the $-strings are literals in the target file
 check "revoke: purge deletes instances one at a time" 0 "" \
   grep -qF 'delete -f "$inst"' "$ROOT/host/revoke-user.sh"