grant/rehearsal: the codex round — verified rollback, loud partial states, and the raw-attach guarantee measured (#75)

Review 4727756972 (A2): the backout no longer trusts gpasswd — it re-reads
the live group database after removal; verified-absent gets the safe
message, anything else screams ROLLBACK INCOMPLETE, exits nonzero, and
names the exact remediation. The concurrent-login window (a session begun
between usermod and backout keeps the group) is CLOSED to the extent the
database can't reach: the backout detects live processes and names
loginctl terminate-user, and the success wording claims only what was
verified.

Review 4727641752 (A1): a failed grant for a user whose membership predates
the run (the hand-added-user scenario) now fails LOUDLY — they retain
socket access on part-converged policy, and the message says so with both
remediations (box revoke now, or fix and re-run). Their membership is not
stripped: breaking a working user over a failed re-grant is its own hazard.
The default-profile eth0 removal is deliberately not restored on failure —
that mutation only reduces capability, and restoring it would move the
failure state AWAY from fail-closed. Injected-failure coverage is criterion
(n), both flavors: fresh-user backout (fault at the LAST mutation, so the
rollback runs after every earlier one) with the group's absence verified
and a converging re-run; blocked narrowing staged for real with an
instance-local NIC parked on the private bridge.

Review A3, resolution 3 with the measurement demanded: criterion (m)
launches exactly 'incus launch --network boxnet' as the restricted user and
probes the raw NIC from inside — egress works, RFC1918 dropped (the ACL is
the network's), sibling probes dropped BOTH directions (the nft drop is the
host's), name enumeration blocked. The scoped guarantee is now stated in
box-design.md and measured on every run: box-minted instances carry per-NIC
port_isolation; raw attachments keep every network- and host-owned control,
losing only that redundant L2 layer. Instrument lesson kept as MU-5: the
probe's first cut minted the non-cloud image — no DHCP client, no lease,
and a dead NIC passes every negative probe vacuously; it now requires the
lease before believing its own answers.

Rehearsal: 54/54 (containers). test/cli.sh: 82 checks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
dan-claude-bot 2026-07-18 06:41:05 +00:00
parent 565e7693e8
commit 8e6f3a4bb8
8 changed files with 193 additions and 16 deletions

View file

@ -20,10 +20,10 @@ which records not just what changed but what each drill run proved.
into their project. `box_tier()` (live credentials, argless `id -nG`)
drives the tier-aware surface: `expose` refuses honestly before any daemon
call, `setup-host` and `doctor` answer at the caller's tier. Rehearsed
end-to-end by `drill/multiuser.sh` (criteria al: confinement, lifecycle,
end-to-end by `drill/multiuser.sh` (criteria an: confinement, lifecycle,
cross-user visibility, name collisions, the in-box isolation contract,
escape hatches, re-sync survival, revoke incl. the live-session case) —
42/42 on the design host, in both container and VM mode.
54/54 on the design host (container and VM mode), including the raw-attach scoped-guarantee measurement and both grant-failure injections demanded by #75's review.
- **CI runs the multi-user rehearsal on a real Incus** — a second `rehearsal`
job stands up the full stack on the runner (setup-host, doctor, then
`multiuser.sh --container`), so every PR proves the tier's semantics

View file

@ -165,9 +165,16 @@ Cross-USER isolation is the same mechanism as cross-box isolation, on
purpose: their instances share `boxnet` with everyone's, and the bridge-family
drop + port isolation + `dns.mode=none` already make any two boxes strangers.
A restricted user CAN strip `security.port_isolation` from the profile copy
in their own project — which is why the nft bridge drop, which they cannot
touch, exists as the second, host-owned layer. Defense in depth, both layers
measured (`drill/multiuser.sh`, criteria al).
in their own project — or skip the profile entirely and attach `boxnet` raw
(`--network boxnet`); the network must be usable for the profile to work, and
Incus has no allow-via-profile-only lever. So the guarantee is scoped, and
said plainly: **per-NIC port isolation is guaranteed for box-minted
instances; a raw attachment keeps every network-owned control (the ACL,
`dns.mode=none`, the resolver pin) and every host-owned one (the nft bridge
drop) — losing only the redundant per-NIC L2 layer.** Scoped, and measured:
`drill/multiuser.sh` criterion (m) launches exactly that raw instance and
probes egress, RFC1918, both sibling directions and name enumeration from
inside it. Defense in depth, every layer measured (criteria an).
`box revoke` is two strengths: bare, it removes the group — their boxes keep
*running* (revoking a person does not kill their workloads), `grant` restores

View file

@ -1,6 +1,6 @@
# Restricted incus tier — design and measured results (#74)
**Status: implemented and rehearsed.** 42/42 rehearsal criteria green on the
**Status: implemented and rehearsed.** 54/54 rehearsal criteria green on the
design host (Debian 13 trixie, Incus 6.0.4, nested KVM), in container mode
and VM mode — and green in CI on ubuntu-latest / incus 6.0.0 (whose one
version-drift false FAIL is MU-4 in `drill/RUNS.md`). This doc records the design, what was measured, and why each
@ -69,10 +69,27 @@ touch `boxnet`'s config or the ACL (`no permission for project "default"`).
A restricted user CAN edit the `box-net` profile copy in their own project
(they own project profiles — `features.profiles=true`), including stripping
`security.port_isolation`. That is why the host-owned nft bridge drop is the
second layer: `meta ibrname boxnet obrname boxnet drop` fires on every
port-to-port frame regardless of per-NIC flags. Cross-user sibling probes are
dropped either way — measured from inside the boxes.
`security.port_isolation` — and CAN attach `boxnet` raw with `--network
boxnet`, no profile at all (the network must be in
`restricted.networks.access` for the profile to work; there is no
allow-via-profile-only lever). That is why the host-owned nft bridge drop is
the second layer: `meta ibrname boxnet obrname boxnet drop` fires on every
port-to-port frame regardless of per-NIC flags. The documented guarantee is
scoped accordingly (see box-design.md): box-minted instances carry per-NIC
port isolation; raw attachments keep every network- and host-owned control,
losing only that redundant L2 layer. Both shapes are measured from inside
the instances (rehearsal criteria g and m).
Two grant-failure contracts, both injected in the rehearsal (criterion n):
a fresh user is backed out of the group with the removal VERIFIED against
the live group database (and any session begun mid-grant is named, with the
loginctl remedy — the one window the database cannot close); a pre-existing
member is never stripped by a failed re-grant, but the failure states out
loud that they retain socket access on part-converged policy, with both
remediations. The default-profile eth0 removal is deliberately NOT restored
on failure: that mutation only reduces capability, and restoring it would
move the failure state away from fail-closed. Every step is check-then-
converge, which is what makes re-run-to-repair deterministic.
## What `box grant <user>` converges (idempotent, re-run to refresh)
@ -110,10 +127,11 @@ It had never worked.
## Rehearsal and CI
`drill/multiuser.sh` (root, opt-in via `BOX_MULTIUSER_REHEARSAL=1`) proves
criteria (a)(f) from #74 plus the measured extensions (g)(l): the in-box
criteria (a)(f) from #74 plus the measured extensions (g)(n): the in-box
isolation contract (egress, DNS, box→host, RFC1918, cross-user sibling drop,
name enumeration, IPv6-off), the closed escape hatches, re-sync survival, and
scoped revoke. Two real users, real grants, real mints, probes from inside;
scoped revoke, the raw-attach scoped guarantee (m) and the grant-failure
injections (n). Real users, real grants, real mints, probes from inside;
`--container` for CI, VM mode on real hardware; cleanup deletes everything it
made.

View file

@ -195,6 +195,7 @@ No listener is needed, and none should be started: see trap 3.
| Run | Result | What it cost |
| --- | --- | --- |
| MU-5 | 53/1 → 54/0 (containers) | the #75 review round, measured in: criterion (m) — a raw `--network boxnet` attach keeps egress, the ACL, both sibling-drop directions and dns.mode=none, losing only per-NIC port_isolation (the scoped guarantee, now measured, not prose) — and criterion (n), both grant-failure injections (fresh user backed out with the removal VERIFIED; pre-existing member warned loudly, never stripped; re-runs converge). The one instrument bug: (m) first minted the non-cloud image — no DHCP client, no lease, and a dead NIC passes every negative probe vacuously. The probe now requires the lease before it believes any of its own answers |
| MU-4 | 41/1 on CI (ubuntu-latest, incus 6.0.0) → 42/0 | the CI job earned its keep on its second-ever run: the attach-refusal probe matched 6.0.4's error wording and 6.0.0 words it differently (and refuses at a different stage) — a version-drift false FAIL the design host could never show. The probe now asserts the OUTCOME (nothing running on the private bridge) and prints the refusal as evidence. Also this round, from adversarial review: revoke survives the live-session case (groups are read at login; --purge now terminates sessions first, else a stale-group process could recreate the project UNHARDENED after the purge), grant backs out its own group-add on failure, and the session-dies-with-the-tier check is criterion 42 |
| **MU-3** | **41/0 (containers) + 41/0 (VMs) — the tier holds** | `multiuser.sh` (#74) from zero on Debian 13 / Incus 6.0.4: grant, confinement, lifecycle, the in-box contract with the cross-user sibling drop, closed escape hatches, re-sync survival, scoped revoke. The VM pass needed right-sized boxes (1GiB): two 2GiB nested VMs on a 7.7GiB host measured swap, not isolation |
| MU-2 | 40/41 (VMs) | the egress probe cried broken over a TLS handshake that outlived its 8s budget under nested-KVM memory pressure — every isolation probe (which PASSES on silence) was immune. Probe budgets raised, reachable-probes retry once, and the mint keeps its log for the post-mortem |

View file

@ -253,7 +253,7 @@ done
# An interrupted multiuser.sh leaves its users (and their projects) behind —
# and nothing else on this host will ever mention them. Its own cleanup is
# 'box revoke --purge + userdel'; say so rather than absorbing them silently.
for u in boxdrill1 boxdrill2; do
for u in boxdrill1 boxdrill2 boxdrill3 boxdrill4; do
if getent passwd "$u" >/dev/null 2>&1; then
no "leftover rehearsal user: $u (an interrupted drill/multiuser.sh run)"
inf "fix: sudo BOX_YES=1 box revoke $u --purge && sudo userdel -r $u"

View file

@ -26,6 +26,10 @@
# i. (folded into b: snapshot / restore / clone)
# k. the grant survives an incus-user restart
# l. box revoke --purge removes the user's world and touches nobody else's
# m. a RAW attach to boxnet (no box-net profile) keeps every network- and
# host-owned control — the scoped guarantee, measured (#75 review)
# n. a grant that fails is fail-closed: fresh user backed out (verified),
# pre-existing member warned loudly, re-run converges (#75 review)
#
# ok/no/note return 0 by design — the 'A && ok || no' idiom below is the
# same one drill.sh is built on (and the reason for the SC2015 disable).
@ -105,7 +109,7 @@ cleanup() {
[ "$KEEP" = 1 ] && { echo "(--keep: users and boxes left for inspection)"; return; }
echo
echo "── cleanup"
for u in "$U1" "$U2"; do
for u in "$U1" "$U2" boxdrill3 boxdrill4; do
id "$u" >/dev/null 2>&1 || continue
# A half-failed purge followed by userdel leaves a project owned by
# nobody — and doctor's leftover check keys on the USER existing. Keep
@ -277,6 +281,56 @@ else
no "(h) boxnet edit attempt: rc=$rc, said: $(printf '%s' "$out" | head -1)"
fi
phase "m. a raw attach to boxnet — the scoped guarantee, measured"
# A restricted user CAN 'incus launch --network boxnet' without the box-net
# profile: boxnet must be in restricted.networks.access for the profile to
# work at all, and Incus has no allow-via-profile-only lever. What the raw
# NIC loses is per-NIC security.port_isolation — the deliberately redundant
# L2 twin of the host-owned nft bridge drop. Everything else binds to the
# NETWORK (ACL, dns.mode=none, resolver pin) or the HOST (nft drop), so the
# contract's claim for raw attachments is "every control except the
# redundant per-NIC layer" — and a claim is a measurement here, not prose.
# Same image the blank template mints (the /cloud variant): the plain image
# has no DHCP client, so its raw instance holds NO lease — and against a
# dead NIC every negative probe below "passes" vacuously while the contract
# goes unmeasured. Caught on this criterion's first run (MU-5).
if as_u "$U1" incus launch images:debian/13/cloud esc2 --network boxnet >/dev/null 2>&1; then
ok "(m) raw attach to boxnet launches (expected: the network must be usable for the profile to work)"
ip_raw=""
for _ in $(seq 1 45); do
ip_raw="$(incus --project "$p1" list esc2 --format csv --columns 4 2>/dev/null | tr -d '"' | sed 's/ (.*//' | grep . | head -n1)"
[ -n "$ip_raw" ] && as_u "$U1" timeout -k 5 15 incus exec esc2 -- true >/dev/null 2>&1 && break
sleep 2
done
inf "raw instance esc2: ${ip_raw:-<no ip>}"
if [ -z "$ip_raw" ]; then
# Without an address the negative probes below would all pass vacuously
# — a dead NIC drops everything, including the truth.
no "(m) the raw instance never got a boxnet lease — the scoped guarantee went UNMEASURED"
else
r="$(probe_up "$U1" esc2 https://1.1.1.1)"
[ "$r" = reachable ] && ok "(m) raw NIC: public egress works ($r)" || no "(m) raw NIC: egress broken: $r"
r="$(probe_from "$U1" esc2 "http://192.168.0.1")"
[ "$r" = dropped ] && ok "(m) raw NIC: RFC1918 still dropped (the ACL binds to the network, not the profile)" \
|| no "(m) raw NIC: reaches private space ($r) — the ACL did not cover a raw attach"
if [ -n "$ip2" ]; then
r="$(probe_from "$U1" esc2 "http://$ip2:9")"
[ "$r" = dropped ] && ok "(m) raw → another user's box is DROPPED (the nft drop is host-owned)" \
|| no "(m) raw instance reached a sibling ($r) — the host drop did not cover it"
fi
r="$(probe_from "$U2" mine "http://$ip_raw:9")"
[ "$r" = dropped ] && ok "(m) another user's box → raw is DROPPED (both directions hold)" \
|| no "(m) a sibling reached the raw instance ($r)"
as_u "$U1" timeout -k 5 20 incus exec esc2 -- getent hosts mine >/dev/null 2>&1 \
&& no "(m) raw NIC can enumerate instance names (dns.mode leak)" \
|| ok "(m) raw NIC: name enumeration still blocked (dns.mode=none is the network's)"
fi
as_u "$U1" incus delete -f esc2 >/dev/null 2>&1
aud "m. raw boxnet attach keeps ACL + nft drop + dns.mode (measured); loses only per-NIC port_isolation — the scoped guarantee in box-design.md"
else
no "(m) raw attach to boxnet failed to launch — the scoped-guarantee measurement could not run"
fi
phase "e/f. the honest refusals — expose, setup-host, doctor"
out="$(as_u "$U1" box expose mine 3000 2>&1)"; rc=$?
[ "$rc" -ne 0 ] && printf '%s' "$out" | grep -qi restricted \
@ -325,6 +379,63 @@ st="$(incus --project "$p1" list mine --format csv --columns s 2>/dev/null | hea
|| no "(l) $U1's box state after $U2's purge: '$st'"
aud "l. revoke --purge is scoped: $U2 erased, $U1 unmoved"
phase "n. a grant that fails is fail-closed — injected, both flavors"
U3=boxdrill3; U4=boxdrill4
BOXROOT="$(dirname "$(dirname "$(readlink -f "$(command -v box)")")")"
# Flavor 1: a FRESH user, fault injected at the LAST mutation (the profile
# edit) — so the backout runs after every earlier mutation has landed. The
# contract: nonzero exit, the group's absence VERIFIED, and a clean re-run
# converges the partial state (which is what makes re-run-to-repair real).
useradd -m -s /bin/bash "$U3" 2>/dev/null
badroot="$(mktemp -d)"
cp -r "$BOXROOT/." "$badroot/"
echo 'devices: {' > "$badroot/profiles/box-net.yaml" # yaml that cannot load
out="$(bash "$badroot/host/grant-user.sh" "$U3" 2>&1)"; rc=$?
rm -rf "$badroot"
if [ "$rc" -ne 0 ] && ! id -nG "$U3" | tr ' ' '\n' | grep -qx incus; then
ok "(n) fresh-user grant failed at the last mutation → backed out, group absence verified (rc=$rc)"
else
no "(n) injected failure: rc=$rc, in-group=$(id -nG "$U3" | tr ' ' '\n' | grep -cx incus) — not fail-closed:"
printf '%s\n' "$out" | tail -3 | sed 's/^/ /'
fi
printf '%s' "$out" | grep -q "verified against the group database" \
&& ok "(n) the backout message claims only what it verified" \
|| no "(n) the backout message is not the verified one"
box grant "$U3" >/dev/null 2>&1 \
&& ok "(n) a clean re-run converges the partial state left by the failure" \
|| no "(n) re-run after injected failure did NOT converge"
# Flavor 2: a PRE-EXISTING member (hand-added before box, the review's named
# scenario) with an instance parked on the private bridge by an
# instance-local NIC — narrowing must fail, the grant must fail LOUDLY
# saying they retain socket access, and must NOT strip the membership this
# run did not add. Unblock, re-run, converge.
useradd -m -s /bin/bash "$U4" 2>/dev/null
usermod -aG incus "$U4"
as_u "$U4" incus project list >/dev/null 2>&1 # materialize their project
uid4="$(id -u "$U4")"
br4="incusbr-$uid4"; [ "${#br4}" -gt 15 ] && br4="user-$uid4"
if as_u "$U4" incus launch images:debian/13 blocker --network "$br4" >/dev/null 2>&1; then
out="$(box grant "$U4" 2>&1)"; rc=$?
if [ "$rc" -ne 0 ] && printf '%s' "$out" | grep -q "still holding socket access"; then
ok "(n) blocked narrowing fails LOUDLY, naming the retained access (rc=$rc)"
else
no "(n) blocked narrowing: rc=$rc — the loud contract is missing:"
printf '%s\n' "$out" | tail -3 | sed 's/^/ /'
fi
id -nG "$U4" | tr ' ' '\n' | grep -qx incus \
&& ok "(n) the pre-existing membership was NOT stripped by the failed re-grant" \
|| no "(n) the failed grant stripped a membership it did not add"
as_u "$U4" incus delete -f blocker >/dev/null 2>&1
box grant "$U4" >/dev/null 2>&1 \
&& ok "(n) unblocked re-run converges" \
|| no "(n) re-run after unblocking failed"
else
no "(n) could not stage the private-bridge blocker — the blocked-narrowing contract went unmeasured"
fi
aud "n. fail-closed injections: fresh-user backout verified; pre-existing member warned, not stripped; re-runs converge"
echo
echo "════════════════════════════════════════════"
echo " $pass passed, $fail failed"

View file

@ -93,16 +93,40 @@ fi
# fresh grant (their existing sessions predate the membership, so no process
# holds it yet). A user who was already in the group keeps it: not ours to
# take on a re-run's failure.
added_group=0
added_group=0; was_member=0
backout() {
if [ "$added_group" -eq 1 ]; then
$SUDO gpasswd -d "$user" incus >/dev/null 2>&1 || true
echo "box grant: FAILED — removed $user from 'incus' again (no half-granted access left behind); fix the cause and re-run" >&2
# VERIFY the removal — an unverified rollback printing a security
# guarantee is a lie waiting for its day. Exact-token match, live DB.
if id -nG "$user" 2>/dev/null | tr ' ' '\n' | grep -qx incus; then
echo "box grant: ROLLBACK INCOMPLETE — the grant failed AND $user is still in the 'incus' group." >&2
echo " remove it by hand NOW: gpasswd -d $user incus (then fix the cause and re-run)" >&2
exit 1
fi
echo "box grant: FAILED — removed $user from 'incus' again (verified against the group database); fix the cause and re-run" >&2
# The one window the database cannot close: a login STARTED between our
# usermod and this backout keeps the group in its session credentials.
# For a fresh grant that is a rare race, but rare is not never — name it
# and the remedy instead of overclaiming.
if pgrep -u "$user" >/dev/null 2>&1; then
echo "box grant: NOTE — $user has live processes; a session begun during this grant would still hold" >&2
echo " the group until it ends: sudo loginctl terminate-user $user" >&2
fi
elif [ "$was_member" -eq 1 ]; then
# A user who was ALREADY in the group keeps it — stripping a membership
# this run did not add could break a working user over a failed re-run.
# But silence here would leave them holding a socket onto part-converged
# policy without the admin being told. Loud, with both remediations.
echo "box grant: FAILED with $user still holding socket access (their membership predates this run)." >&2
echo " their project may be part-converged — harmless in itself, and a re-run converges the rest." >&2
echo " if their access is not acceptable while you fix the cause: box revoke $user" >&2
fi
}
trap backout EXIT
if id -nG "$user" | tr ' ' '\n' | grep -qx incus; then
was_member=1
echo "group: $user already in 'incus'"
else
$SUDO usermod -aG incus "$user"

View file

@ -234,6 +234,22 @@ check "grant: backs out its own group-add on failure" 0 "" \
grep -qF 'trap backout EXIT' "$ROOT/host/grant-user.sh"
check "grant: the back-out disarms on success" 0 "" \
grep -qF 'trap - EXIT' "$ROOT/host/grant-user.sh"
# The backout must VERIFY the removal and scream when it cannot — an
# unverified rollback printing a security guarantee is the review's A2.
check "grant: the backout verifies against the group database" 0 "" \
bash -c 'awk "/^backout\(\) \{/,/^\}/" "'"$ROOT"'/host/grant-user.sh" | grep -q "id -nG"'
check "grant: an unverifiable rollback screams" 0 "" \
grep -qF 'ROLLBACK INCOMPLETE' "$ROOT/host/grant-user.sh"
check "grant: a failed re-grant warns the pre-existing member is untouched" 0 "" \
grep -qF 'still holding socket access' "$ROOT/host/grant-user.sh"
check "grant: the mid-grant login window is named" 0 "" \
bash -c 'awk "/^backout\(\) \{/,/^\}/" "'"$ROOT"'/host/grant-user.sh" | grep -q "loginctl terminate-user"'
# The scoped guarantee (raw --network boxnet) is measured, not prose:
check "rehearsal: measures the raw boxnet attach (criterion m)" 0 "" \
grep -qF -- '--network boxnet' "$ROOT/drill/multiuser.sh"
# shellcheck disable=SC2016 # the $-string is a literal in the target file
check "rehearsal: injects grant failures (criterion n)" 0 "" \
grep -qF 'grant-user.sh" "$U3"' "$ROOT/drill/multiuser.sh"
# shellcheck disable=SC2016 # the $-strings are literals in the target file
check "revoke: purge deletes instances one at a time" 0 "" \
grep -qF 'delete -f "$inst"' "$ROOT/host/revoke-user.sh"