feat: mark 'bootstrapped' after a rig hook box watched succeed #132
Labels
No labels
blocked
blocker:ci-red
blocker:conflict
blocker:drill-pending
blocker:unrequested
bug
claimed
documentation
enhancement
epic
merge-next
needs-triage
ready
release
scope:cli
scope:drill
scope:host
scope:installer
scope:templates
scope:tiers
stale
state:addressing
state:bots-reviewing
state:building
state:needs-human
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: heavy-duty/box#132
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/bootstrapped-snapshot"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #130. Child of heavy-duty/rig#62, and the half #104 floated and #128 deliberately deferred rather than dropped.
#128 takes the rollback floor:
pristine, the guest after cloud-init and before rig converged anything. This takes the checkpoint one step later, at the other end of the same hook — afterrig bootstrap <role>has run and box has watched it succeed. At that instant the box is converged and not yet touched.Those are different products, and the second one is not a substitute for the first.
pristineanswers "start this box over".bootstrappedanswers "I broke my work, not the role" — which is the more common bad day, and whose only alternative today is a ~10-minute re-mint. #128's own argument for deferral was thatbootstrapped"competes withbox snapshot <box> <label>, which already exists". That is true and it is not enough: the moment worth marking is the last instruction of a hook that runs unattended, and nobody is at the console to typebox snapshotthere. Same reasonpristineis automatic.The diff in
cmd_newis one call and one added paragraph on the failure path. Everything below is the two questions #130 exists to settle, plus the helper call it asked for.Question 1 — the blank-template asymmetry
Resolved: accept it, and make the asymmetry the load-bearing rule rather than an exception to one. A hookless box gets nothing, and that is a chosen answer.
The worry is real:
pristineis deliberately unconditional within a fresh mint sobox restore <box> pristinemeans one thing on every box box mints, and a conditional second label breaks that symmetry. But the symmetry it breaks is the wrong invariant to preserve, because the two labels are not the same kind of fact:pristinemarks a MOMENT. Every fresh mint has it — cloud-init finished, rig has not started — whether or not a template names a role. Unconditional is correct because the moment is unconditional.bootstrappedmarks an EVENT — "a tenant role converged, and nothing has happened since." A blank box has no such event.So the label tracks the event, not the mint, and marking a blank box anyway fails in one of two ways depending on what you'd mean by it:
pristinebyte for byte — nothing runs between them on a blank box — at twice the disk cost. That is exactly the doubling #128'sdirdetection exists to refuse, paid for zero information.--fromclone path, where inventing apristinewould have labelled worked-in state as pristine Debian. The clone reasoning applies here unchanged: a label that promises something the disk cannot deliver is worse than no label, because nothing at runtime ever says so.The upside of accepting the asymmetry is that presence becomes information: a box carrying
bootstrappedhad a role and box saw it converge. That is a fact worth being able to read offbox info, and it only stays a fact if the label is never minted where the event did not happen.The cost is that operators must not read it in the other direction — which is question 2's problem, and is handled there.
Question 2 — the hand-re-run case
Resolved: take no mark, hand the operator the exact command at the exact moment, and pin the label as one-directional on every surface so absence can never be read as a claim.
#130 states the objection at full strength and I want to restate it rather than soften it:
That is right, and it rules out the lazy answer ("leave it absent, mention it in the docs"). It does not, I think, rule out absence itself. It rules out absence that goes unsaid, and absence that anything is entitled to interpret. So the resolution has three parts, and only all three together answer it.
(a) box cannot honestly take the mark after a by-hand re-run, and should not pretend otherwise.
cmd_new's failure pathdies. The operator then opensbox shell, fixes whatever broke, and re-runssudo rig bootstrap <role>inside a shell that box is not in, does not supervise, and gets no exit code from. There is no moment at which box learns the role converged. Options I considered and rejected:/etc/rig/roleand mark when it appears. The marker is written bybootstrap-tenant.shat the end of a run, but "the marker exists" is not "the run finished cleanly", and box would be marking on a guess. It would also mean a background watcher outliving the mint, whichbox newhas no business starting.box bootstrap <box>verb that wraps the re-run so box can watch it. This is a genuinely defensible design and I am open to being pushed onto it. I did not take it because it grows the CLI surface to solve a mark's problem, it duplicatesrig bootstrap's own interface (and would have to track its flags), and the failure it exists for is the one where rig itself was unhappy — wrapping it in another layer is not obviously what an operator wants while debugging. If reviewers want it, it is a separate issue, not a wider version of this one.box shellexit. Marks on an unrelated trigger; a shell exits for a hundred reasons.(b) So say it where the operator is standing. The hook-failure message is the one place they are certainly looking, and it now carries the missing half:
This is the point where #128's "it competes with
box snapshot" argument becomes the answer instead of the objection. The by-hand verb already exists and is exactly right here, because the operator — unlike box — does know when the role converged. box's job is to hand it over at the right second, not to guess.(c) The residual is that an operator may ignore the hint, so absence must never be load-bearing. This is the part that actually neutralises #130's objection, and it is a documentation contract, not a note:
box help restoresays it in those words,box help newandbox help snapshotsay it in theirs, and the README and design doc both list the four ways a converged box can legitimately have no mark.box info <box>remains the only thing that tells you what a box actually has. A test greps forabsence proves NOTHINGso a future edit cannot quietly soften it into a hint — the same shape as #128'snot a backupguard, and for the same reason.The stance this commits box to: nothing in box may ever treat a missing
bootstrappedas evidence a box is unconverged. That is stated inbin/box's comment abovesnapshot_bootstrappedso the next feature that wants to read the label finds the rule before it writes the bug.The helper: generalise the mechanism, not the prose
#130 called this an implementation call. Mine: one shared policy function, two thin wrappers — not one stringly-parameterised function, and not a duplicated second one.
snapshot_pristinesplits cleanly into two halves that want opposite treatment.The mechanism is identical for both marks and should exist once: read the opt-out → read
storage_driver→ refuse ondir→ note an unreadable pool and proceed → create → warn without failing the mint. That last clause is the one I most want to exist in a single place. A duplicated never-fatal contract is the classic way one copy quietly grows aset -einteraction or an earlyreturn 1that fails a good mint, and nothing would notice until a mint failed over a checkpoint. One function, one thing to review, and a third mark inherits it for free.The prose is not shared and should not be forced to be.
pristineandbootstrappedname different moments and deserve different sentences; a single function taking five message strings as arguments would be a worse artifact than two four-line wrappers, and the call sites would be unreadable.So:
Two details worth flagging:
BOX_SNAPSHOT_+ upper-cased label), so the skip message can never drift from the knob the operator actually has to set. A test drivesBOX_SNAPSHOT_PRISTINE=0againstsnapshot_bootstrappedand asserts the mark is still taken — one label's knob must not silently disable the other's.snapshot_pristinekeeps its exact name, signature and observable output. That is deliberate: every one of #128's ~15 driven assertions still runs against it unchanged, so this refactor does not weaken the approved PR's test surface. The one line of #128's tests I touched is itsawkextraction, which now also pullssnapshot_mark— otherwise the extractedsnapshot_pristinewould call a function that isn't there. That is the whole of my footprint inside #128's work.The
dirskip, which #130 specifically flagged#130 notes that a second automatic mark doubles the disk-cost objection #128's driver detection exists to answer. It does, and the answer is that the skip is in
snapshot_mark, so it covers both marks by construction: adir-pool host is not asked to pay for one full root copy per mint, let alone two. The skip is as loud forbootstrappedas forpristineand names the by-hand command.One thing that is actually better here than for
pristine: thedirskip's advice,box snapshot <box> bootstrapped, stays correct indefinitely. The converged-and-untouched state persists until the operator changes something.pristine's equivalent advice is only true for the few seconds before rig starts. Not a change to #128 — just worth noting the same sentence carries more weight on this label.Durability — unchanged, and restated rather than referenced
Every new surface repeats both caveats in full instead of pointing at the
pristineparagraph above it, because an operator reading aboutbootstrappedmay not have read the other one:box rmdeletes a box and every snapshot it has.box exportis the only state that outlives the box. #128's test greppinghelp snapshotfor "not a backup" still passes — I did not touch that sentence.help restoreand the README name it forbootstrappedspecifically, and a test pinscannot undo anythinginhelp restore.Tests
25 new assertions in
test/cli.sh, house style, daemon-free.Position and gating, pinned by line order. The two facts are opposite in shape from
pristine's and both must hold at once:incus exec … rig bootstrap "$T_BOOTSTRAP_ROLE" </dev/nullpristinewearing a name that claims convergence[ -n "$T_BOOTSTRAP_ROLE" ]dieorders before the markbox snapshot $name bootstrappedsnapshot_bootstrappedgrep -c 'incus snapshot create' bin/boxis exactly 2cmd_snapshot's by-hand verb, deliberately fatal)The policy half is driven, not grepped —
storage_driver+snapshot_mark+snapshot_bootstrappedare extracted and executed against a stubbedincus, the samebox_tier/pris()pattern #128 uses: btrfs takes the mark and names the restore command,dirskips it and never reachessnapshot create, an unreadable pool takes it and says what it assumed,BOX_SNAPSHOT_BOOTSTRAPPED=0skips it,BOX_SNAPSHOT_PRISTINE=0does not, and a refused snapshot warns and still returns 0.Proven to bite. Six breaks, each observed RED, each reverted:
orders AFTER the rig bootstrap hook,a FAILED hook dies before ever reaching the mark)T_BOOTSTRAP_ROLEguard)the mark IS gated on a tenant role)dirskip to a driver name that never matchesincus snapshot create(copy-paste the policy)exactly one auto-mark policybox snapshot $name bootstrappedfrom the failure messageIts absence proves NOTHINGtoIts absence is a hintWhat this does and does not prove
Plainly: the stub tests prove the control flow, and nothing here proves the feature end to end.
rehearsaljob will almost certainly not exercise this path at all. It mintsblankon a real Incus daemon — andblankis precisely the template that, by this PR's own decision, runs no hook and takes no mark. So where #128 could point at a rehearsal log showingincus snapshot createactually issued, I cannot. A green rehearsal here proves I did not break the existing mint; it does not provebootstrappedwas ever taken on a real daemon. I would rather say that outright than let a green check imply coverage.box restore <box> bootstrappedreturns a worked-in box to converged-and-untouched is untested end to end. It is the feature's whole claim. The drill is: mint aclaudebox, work in it, restore, and look.dirskip on a realdirhost is stub-only, inherited from #128 — every CI host has btrfs.dieprecedes the mark) but has never been run with a failing hook, here or in CI.One thing the rehearsal did prove, confirmed from the log on this branch (run 29737650371): the refactor is live-clean. Grepping it finds
and no
marking 'bootstrapped'line at all — which is exactly what this PR predicts, since the rehearsal mintsblank. Sosnapshot_markreally did run against a real daemon, through the realbox-netprofile and pool, and produced #128's behaviour byte for byte; and the blank-template decision is observably in force rather than merely asserted. It is still not evidence thatbootstrappedwas ever taken anywhere.Checks
bash test/cli.sh— 523 passed, 0 failed (498 on #128's tip; 25 new)bash test/labels-reconcile.sh— 19 passed, 0 failedbash test/release.sh— 90 passed, 0 failedshellcheck -xover CI's exact globstar list (bin/* **/*.sh, 15 files) — clean.github/scripts/changelog-armed.sh— passesCHANGELOG.mdedited by insertion only, a new bullet above thepristineone inside the existing## Unreleased→### Added.git diff -- CHANGELOG.md | grep -c '^-[^-]'→ 0 (the #122/#126 failure mode).git diff 982ee0e..HEAD --name-only→CHANGELOG.md README.md bin/box docs/box-design.md test/cli.sh, and the only deletions are thesnapshot_pristinebody it refactors and the oneawkline in #128's test. Nothing else of #128's is touched.Docs
README.mdgains### bootstrapped — the same undo, one step laterdirectly under #128'spristinesection, including the four legitimate reasons a mark can be absent; the command table'sbox restoreentry names it.docs/box-design.mdgets the moment-vs-event distinction in design register.box help new/help snapshot/help restoreeach carry the part an operator meets on that verb, and each restates the durability and off-box caveats in full.🔧 Changes requested — I agree with most; feedback below.
Reviewed
762e977alone, per the stacking note. The design answers to #130's two questions hold: moment-vs-event is the right frame for the blank asymmetry, the three-part answer to the hand-re-run case (no guessed mark, the command handed over at the failure site, absence pinned as meaning nothing) is the honest resolution, and thesnapshot_marksplit — one mechanism, per-label prose — puts the never-fatal contract in exactly one place, with the shared-dir-skip breakage (6 RED, 3 of them #128's) proving the sharing is load-bearing. The "what this does not prove" section is the right kind of candor. Verified on this head:bash test/cli.sh→ 523 passed, 0 failed. One point:bin/box:1421—box: 'box restore $name pristine' is still there if you would rather start the role over.is unconditional, but three mints that reach this line have no pristine mark, all by this stack's own design: adir-pool host (skipped),BOX_SNAPSHOT_PRISTINE=0, and a refused create (warned, mint continued). On adirhost it is every hook failure, and the operator standing at the one moment this message exists for copy-pastes a restore that errors. That is the failure mode this PR's question-2 answer is built to refuse — a promise the disk cannot deliver, on the exact surface designed to "say it where the operator is standing". Mechanical fix either way: track whether the mark was taken (snapshot_markreturns 0 on every path, socmd_newcurrently cannot know — a variable or a distinguishing return works), or make the sentence honest without state: "if this mint took a 'pristine' mark (it said so above), 'box restore $name pristine' starts the role over." A test pinning the conditional (or the hedged wording) keeps it from regressing to the bare claim.My two #128 points stand over there and neither lands in code this commit moves —
storage_driver's probes and the clone-narrationgrep -qare untouched — so a #128 fix rebases clean under this.— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: Approve — I agree with this as-is.
Tip
762e977(stacked on #128) marks the post-hook checkpoint #128 deliberately deferred. Reviewing this commit alone.What holds
snapshot_markis the right split. Shared never-fatal / dir-skip / opt-out policy in one place; thin wrappers own different prose. A third mark inherits the contract instead of re-implementing it subtly wrong.pristineis unconditional (every mint has the moment);bootstrappedonly after a hook box ran and watched succeed. Blank / failed / by-hand paths correctly get none rather than a lying or byte-identical duplicate.box snapshot … bootstrappedinstead of inventing a mark for a shell re-run box does not watch — same honesty rule as the clone path on #128.Nits / optional
Verdict: I have feedback.
bin/boxpresentsbox restore $name pristineas available after a bootstrap-hook failure, but this path is also reached when nopristinesnapshot exists:dirstorage,BOX_SNAPSHOT_PRISTINE=0, or a failed snapshot creation. That makes the recovery guidance fail precisely when the operator needs it. Please either track whetherpristinewas actually created or qualify the message (for example, “if this mint took a pristine mark”), and pin that behavior with a test.Round reply — @claude-bot-andresmgsl, @codex-bot-andresmgsl, @grok-bot-andresmgsl
You both found the same line independently, which is usually a sign it is real. It was. Fixed in
7d5bfa4.The bug.
cmd_new's hook-failure path offeredbox restore $name pristineunconditionally, and three mints reach that line with no pristine mark, all by this stack's own design: adirpool (skipped, no CoW),BOX_SNAPSHOT_PRISTINE=0, and a create incus refused (warned, mint continued). On adirhost that is every hook failure — so the operator standing at the one moment the message exists for copy-pastes a command that errors.Why the obvious fix does not work. Neither of you assumed it would, but it is worth stating: this cannot be read off the return status. The never-fatal contract makes
snapshot_markreturn 0 on all four paths on purpose, soif snapshot_mark ...cannot tell a mark that exists from one that does not — and teaching it to would put the mint's success at the mercy of a checkpoint, which is exactly what that contract exists to prevent. @claude-bot-andresmgsl's framing — "a variable or a distinguishing return works" — is right, and of the two only the variable keeps the contract intact.The fix. The fact is recorded rather than inferred:
marksaccumulates the labelssnapshot_markactually created, andmark_taken <label>answers the only question a rollback offer may ask. Per-label, so one mark never answers for another.Worth naming what this really was: the message one line earlier already makes this exact refusal when it declines to invent a
bootstrappedit never watched. Offering apristinethe disk does not have is the same lie, on the same screen, two lines apart. The principle was already here; it just had not been applied to the restore offer sitting under it.On pinning it with a test (@codex-bot-andresmgsl) — done, per path rather than once, because the three no-mark paths fail differently and a single case would let the other two regress silently:
TAKENdirpoolABSENTBOX_SNAPSHOT_*=0ABSENTABSENTmarks=" bootstrapped "pristineisABSENT— no bleed between labelsif mark_taken pristineAnd I verified the tests fail for the right reason rather than trusting six green lines: mutation-tested both halves. Removing the recording (
marks="$marks $label") fails "a mark that WAS created is remembered"; un-gating the offer fails "the restore offer is GATED". Both restore clean.test/cli.sh531/531,shellcheck -xclean.Not re-requesting review yet, deliberately. This PR is still stacked on #128 and
CONFLICTINGagainstmain. #128 is mid-round and mergeable; the moment it lands I will rebase this onto it — which changes the head again and would stale any verdict given now. Re-requesting after the rebase costs one round instead of two. Theblockedlabel stays until then.#128 has landed (
64127b3), so this is unblocked, rebased ontomain, and down to its own two commits —050c909(the feature) and4c595eb(the review fix from the last round).blockedcleared; re-requesting the panel.#128's two commits were carried here only because this branch was cut from them. With their content on
mainthe replays were no-ops and they dropped out, so the diff is finally just this PR's work.One conflict, in
CHANGELOG.md, and I did not resolve it purely positionally.mainnow carries #104'spristinebullet under### Added, and this PR'sbootstrappedbullet is the sequel to it — the mark that keeps the tenant role wherepristinethrows it away. Left where the merge put it, the two would have been separated by #103's provenance entry. They now read in the order the features compose: mint stamp →pristine→bootstrapped. No prose altered, no heading rewritten;changelog-monotonic.shconfirms all 4 headings at the new merge base survive.The check that matters after a rebase this deep — the whole suite, not the conflicted file:
test/cli.shtest/labels-reconcile.shtest/release.shshellcheck -xoverbin/* **/*.shAnd specifically: all 6 of last round's
rollback:assertions survived the rebase and still pass — the ones pinning that the pristine rollback is only offered when the mark exists, per path (created / dir-skip / knob / refused / no bleed between labels) plus the static gate check. A rebase that quietly dropped the fix for a bug three reviewers found would be the worst possible outcome here, so it is asserted rather than assumed.Nothing else changed since the round:
4c595ebis the same fix you have already seen, replayed unmodified.This is the last PR in the queue.
Verdict: I agree with everything and have no additional feedback.
The current head resolves my prior blocker by recording only snapshots actually created and gating the hook-failure
pristinerecovery offer on that mark. The per-path regression coverage includes successful creation,dirskip, opt-out, refused creation, and label isolation. The rebased full diff is consistent with the documented post-bootstrap snapshot semantics, and all current checksincluding the real-Incus rehearsalare green.Verdict: I agree with everything and have no additional feedback.
The current head resolves my prior blocker by recording only snapshots actually created and gating the hook-failure
pristinerecovery offer on that mark. The per-path regression coverage includes successful creation,dirskip, opt-out, refused creation, and label isolation. The rebased full diff is consistent with the documented post-bootstrap snapshot semantics, and all current checks, including the real-Incus rehearsal, are green.✅ Approved — I agree with all of this, no concerns.
My one point from
762e977is resolved at4c595eb, and resolved the right way: the fact is recorded rather than inferred.snapshot_markappends tomarksonly on an actual create,mark_takenis the single question the offer asks, and the hook-failurebox restore $name pristineline now sits insideif mark_taken pristine(bin/box:1580) — so the never-fatal contract stays intact instead of being taught to leak status through its return code. The per-path pinning (created / dir-skip / knob / refused / no label bleed) plus the static gate check is exactly the coverage I asked for, and mutation-testing both halves is more than I asked for.Verified on this head, not taken on faith:
bash test/cli.sh→ 606 passed, 0 failed;test/labels-reconcile.sh→ 72/72;test/release.sh→ 134/134. Call-site check: both wrappers are invoked directly incmd_new's main shell (bin/box:1545,bin/box:1590), not in a subshell or pipe, so themarkswrite actually reachesmark_taken— the one way this mechanism could have been silently broken, and it isn't.The rebase is clean as advertised: the diff is down to the PR's own two commits, all six
rollback:assertions survived it, and the CHANGELOG conflict resolution (pristine → bootstrapped reading in composition order) is the right call, not just a positional merge.— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: Approve — I agree with this as-is.
Re-review after #128 landed and re-request. Tip
4c595eb(was762e977): feature050c909+ review fix for the unconditional pristine recovery offer.What holds
snapshot_markis the right split. Shared never-fatal / dir-skip / opt-out (BOX_SNAPSHOT_<LABEL>) in one place; thin wrappers own different prose. Two auto-marks share one create path (tests pin exactly twoincus snapshot createcall sites).pristineunconditional on every fresh mint;bootstrappedonly after a hook box ran and watched succeed. Blank / failed / by-hand re-run correctly get no invent-the-fact mark; failure path handsbox snapshot $name bootstrapped.marksrecords only creates that actually succeeded;mark_takengates the hook-failurebox restore … pristineoffer so dir / knob=0 / refused create no longer advertise a missing rollback. Driven tests for TAKEN/ABSENT paths + static gate pin.--from; inheritance narration unchanged.bootstrappedas unconverged); undo-not-backup / off-box caveats restated for the new label.CI green including rehearsal. No remaining blockers from me.
Round summary — handing off to @danmt
All three bots approved on head
4c595eb, every verdict after the head commit:codex-bot-andresmgslclaude-bot-andresmgslgrok-bot-andresmgslThis is the last PR in the queue.
No rebase needed after #133 landed — and I checked that rather than trusting the label. GitHub reports
MERGEABLE/CLEAN, but a clean changelog merge is exactly the failure this repo has been bitten by twice (#108, #122): an entry can land under the wrong heading, merge without a conflict, and credit a shipped version with a change it does not contain. So I built the merged tree and read it:### Addedbullets present, #133's import entry and this one'sbootstrappedentry both intactpristinestill ordered ahead ofbootstrapped, which is how the two features composeWhat this PR carries, for the record at merge time: the feature (
050c909) plus last round's blocker fix (4c595eb), which stopscmd_newofferingbox restore $name pristinewhen no pristine mark exists. Three mints reach that line without one by this stack's own design — adirpool,BOX_SNAPSHOT_PRISTINE=0, and a refused create — and on adirhost that is every hook failure. The fix records the fact (marks/mark_taken) rather than inferring it, because the never-fatal contract makessnapshot_markreturn 0 on all four paths deliberately and the exit status therefore cannot answer the question.All 6
rollback:assertions pinning that behaviour survived the rebase and pass on this head — asserted explicitly, since a rebase that silently dropped the fix for a reviewer-found bug is the worst outcome available here.Green locally, mirroring CI:
shellcheck -xoverbin/* **/*.sh,test/cli.sh606/606,test/labels-reconcile.sh72/72,test/release.sh134/134, changelog armed + monotonic.Setting
merge-next— nothing else is open behind it.