feat: /etc/rig/manifest — which rig converged this machine, and when #84
Labels
No labels
attention
blocked
blocker:ci-red
blocker:conflict
blocker:drill-pending
blocker:unrequested
bug
claimed
documentation
enhancement
epic
merge-next
needs-ruling
needs-triage
offsite
post-merge
ready
release
scope:bootstrap
scope:coolify
scope:db
scope:docs
scope:drill
scope:installer
scope:labels
scope:platform
scope:runner
scope:users
stale
state:addressing
state:bots-reviewing
state:building
state:needs-human
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: heavy-duty/rig#84
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/manifest-provenance"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Implements #61.
What it records, and why only immutable facts
/etc/rig/manifest,0644, beside the marker and the users ledger:Two pairs — birth (the rig that first converged this machine, pinned
forever) and latest (the newest rig to have converged it). Equal on a
fresh machine: mild redundancy, in exchange for an audit file no reader ever
has to infer a missing field from. Both versions are captured at run time from
the running tree's
VERSION, because a machine outlives the rig that built it—
rig --versionafter an upgrade answers a different question than either ofthese, and it was the only place
VERSIONwas read before this PR.key=value, one per line, never JSON or YAML. Not a style preference: thetree states the constraint three times already (
lib/users-config.sh:6-12,lib/runner-config.sh:6and:24) — a rig-bootstrapped box has no YAMLparser and no
jq, which is whyjson_field()is grep-and-sed. This is thefile that must be readable on the most broken machine in the fleet, so
readparses it for free.
The scope discipline is the whole trick and it is worth restating: facts
that are DECIDED go in the manifest; facts that are OBSERVED are computed at
run time and stored nowhere. Specs change without rig doing anything —
someone adds RAM, or the unattended-upgrades
bootstrap.sh:198-201itselfenables patches the kernel — so a stored spec is either stale or needs a
refresh on every run, and a refresh on every run is exactly what the next
section is about. Removing them removes the volatility problem instead of
managing it.
The crux: how convergence survives a timestamp
bootstrap.sh:3promises "Convergent: safe to re-run; a second run changesnothing", and the repo enforces it with a
cmp -sguard before every fileinstall — nine sites. A naive
converged_at=$(date)breaks that contract onevery single re-run: the file differs by a timestamp, the guard fires, and
rig reports a change it did not make. I suspect that is precisely why no
timestamp existed anywhere in this tree until now.
Two rules keep the clock out of the file:
Rule 1 —
bootstrapped_*is first-write-wins. An existingbootstrapped_atpins both birth fields verbatim. Regenerating birth as
now()each run wouldmake every re-run a diff.
Rule 2 —
converged_*moves only when the version actually differs.converged_atis the time the converging version last changed, not thetime of the last run. If it tracked every run it would be a clock. So the
running
VERSIONis compared against the recordedconverged_by; equal meansthe pair is already true and is copied through untouched.
Under those rules a re-run by the same rig renders byte-identical content
and the guard stays silent, while a re-converge by a different rig renders
a real diff — and the guard firing there is correct rather than spurious. It
was only ever the clock that was the fake change, never the version. No
special case in the cmp-guard; it works unmodified.
How this is pinned, and why the obvious test would not have pinned it.
manifest_render <path> <version> <now>is a pure text→text function (repoprecedent:
parse_users_file,assert_marker_human) — existing file plusrunning version plus a clock reading in, desired content out, no side effects.
That makes the property directly assertable: render the same fixture twice
with two clock readings a year apart and diff. Byte-identical output means
the clock cannot reach the file at all.
That is deliberately stronger than the test one reaches for first. I wrote the
writer-level version too (
stamp,stampagain, assert no byte changed) andit is in the suite — but it passes even against a deliberately broken
implementation, because two stamps in the same second produce the same string
by luck. I confirmed that directly: with Rule 2 removed, the writer-level
assertions stay green and only the render-level ones go red. The clock-apart
diff is the assertion that actually bites, and I would not want a reviewer to
read the writer-level pair as the guarantee.
The RED drill. Both rules were verified to fail loudly before being
trusted:
converged_*recomputed unconditionally) → 4 RED:byte-identical-across-a-year, reproduces-itself, foreign-keys-do-not-break-
convergence, and the-repair-settles.
plus both birth-stays-pinned assertions and both damaged-file repairs.
Reverted, 445 green.
The manifest vs. the marker
/etc/rig/roleis not touched, per the issue. It has six readers —warn_bootstrapped()(bin/rig:141-146), the tenant guard(
bootstrap-tenant.sh:109-123),assert_marker_human()(
lib/users-config.sh:120-147),install.sh:82-90and four advisory sites —and collapsing them is a separate, riskier change. The marker holds traits
(what this box is); the manifest holds provenance (what built it). Two
files, two jobs. A test asserts no code in the manifest lib reaches
/etc/rig/roleorRIG_ROLE_MARKER, and another asserts the marker's owncmp-guarded block is still there.
The manifest vs.
rig platform(#64 / #74)Complementary and non-overlapping by construction: decided vs observed.
The manifest stores provenance and nothing else;
rig platformcomputes specsand stores nothing. I have deliberately not touched
rig platform'ssurface — #74 is open and already reads a manifest through
RIG_MANIFESTinthe flat
key=valueshape this PR writes, so it needs no change when eitherlands. The two are independent in both directions.
rig manifest [<key>]is #61's own read-back surface and is not a secondplatform: it prints the raw file, or one value alone so shell callers do notre-parse. Whichever of the two PRs lands second will find the other already
compatible.
Judgment calls a reviewer should look at
1. Tenants get a manifest — the same one, unconditionally. This was an
open question in the issue.
bootstrap-tenant.shdeliberately declines toclobber a machine marker (
:370), and I did not copy that gate. The reasonthe marker needs it is that it holds competing traits: a guest's traits and
the traits it earns after an operator-run
rig bootstrap workloadjoin are twostatements about one box, so the marker has to pick the truer one. Provenance
has no such conflict — "which rig converged this guest, when" is a fact
whichever bootstrap ran, and the two-pair shape composes across them exactly as
designed: a staging guest later joined as a workload keeps the tenant
bootstrap as its birth (that genuinely is when this machine was first
converged) and the machine bootstrap moves
converged_*forward. Copying thegate would lose the birth stamp only that first run knows. One file, not a
tenant-shaped second one: the manifest answers a question about the machine,
and a guest is a machine; the marker already carries
tenant=yesfor anyonewho needs to know which kind.
2. Placement: immediately after the marker, not at the very end of the run.
The issue says "write last, and write nothing on a path that died", glossing
the marker's own discipline (
bootstrap.sh:420-424). I read "last" as afterthe tag verification and put the stamp directly beside the marker, so the two
land together and agree by construction. Trailing the box install and the
users phase would let a box exist with a marker and no manifest — and a box
whose people failed to converge was still converged by this rig at this
time. Stamping provenance is not a claim that everything after it succeeded;
the marker beside it makes exactly the same claim. A test asserts the stamp
follows
verify_effective_tag, with a companion assertion that both grepanchors were actually found (a silent-empty grep would make the ordering test
unfireable).
3. A birth stamp with no birth version records
unknown, not today's. Ona hand-edited or truncated manifest carrying
bootstrapped_atbut nobootstrapped_by, backfilling the running version would be a manifest thatlies about what built the box.
unknownis the honest answer, and it isstable across re-runs.
4. A downgrade is a change.
converged_byis "the rig that last convergedthis", not "the highest one ever seen" — rolling back with
rig useandre-converging is recorded, or the file would name a version that is no longer
what runs here. Tested.
5. The writer preserves keys it does not own. The schema promises readers
ignore unknown keys; that promise is worthless if the writer eats them. A
later command's
runner_installed_at, or a key from a newer rig's schema,survives a rewrite verbatim — and a file carrying foreign keys is still
byte-stable under a same-version re-render (both tested). This is what makes
the issue's "later commands may append their own provenance" future safe.
6.
rig manifest <key>distinguishes absent from empty.$(...)collapsesboth to
"", so absence is the exit code and only a present key ever prints.The key lookup is a string equality, never a pattern — operator input reaches
it, and a key of
.*must find nothing rather than match line one. Tested.What I deliberately left alone
/etc/rig/role— see above. Zero changes.bin/rig:9'sversion()— it could now source the lib and sharemanifest_running_version, butbin/rigsources no libs at all today andmaking it do so for one
catis a structural change for a cosmetic win.mkdir -p /etc/rigsites (bootstrap.sh,bootstrap-tenant.sh:374,users-apply.sh:301). The issue invites factoringthem; I did not. The manifest writer does its own
mkdir -p "$(dirname …)"inside the lib — which it needs anyway to honour
RIG_MANIFEST— andrewriting three unrelated marker/ledger writes to route through a new helper
is churn in files this PR otherwise does not touch, on paths that guard root
SSH policy. Happy to do it as a follow-up if a reviewer prefers it.
rig platform— #74's surface, untouched.runner_installed_at/coolify_installed_at/box_version.The issue sketches them; this PR ships only the bootstrap pair and makes the
file forward-compatible with them. The
never a credentialrule(
runner-install.sh:190's existing law for.rig-labels) is restated at thetop of the writer, where the next command that appends a line will read it —
and asserted by a test, since that comment is the only thing standing between
a 0644 audit file and a secret.
Checks
shellcheck -xover CI's exact file list (shopt -s globstar; bin/* **/*.sh)— clean, including the two new files.
bash test/cli.sh— 445 passed, 0 failed (400 on main, so 45 newassertions).
bash test/release.sh— 68 passed, 0 failed.CHANGELOG.mdentry under## Unreleased, per CONTRIBUTING step 8. No otherheading disturbed.
says the manifest lands beside the marker.
Closes #61
Verdict: Approve — I agree with this as-is.
Tip
66cf47fships a clean provenance file without breaking bootstrap's convergence contract.What holds
rig platform. key=value 0644 matches the rest of the tree's no-jq constraint.converged_*only on version change. Puremanifest_render+ year-apart clock pin is the assertion that actually bites (writer-level same-second would pass a broken impl)./etc/rig/roleuntouched.45 new assertions, CI green. No blockers.
Verdict: I have feedback.
Blocking:
manifest_value,manifest_has, andmanifest_foreignuse barewhile ... readloops, so they never process a populated final line when the file lacks a trailing newline. On the repair path this can reset owned provenance (for example, an unterminatedbootstrapped_at=is treated as absent), andmanifest_stampcan silently delete an unterminated unknown/future key despite the stated preservation contract. Please process a non-empty partial EOF record (the sameread || [ -n ... ]shape used elsewhere) and add coverage for owned and foreign final lines without a newline.🔧 Reviewed — I agree with most; feedback below.
The design is right where it was hardest: decided-vs-observed keeps the volatility problem from existing, Rule 1/Rule 2 keep the clock out of the file so the cmp-guard works unmodified, and the pure
manifest_renderwith the year-apart clock diff is the assertion that actually bites — the PR body's own demonstration that the writer-level pair passes a broken implementation is the honest version of that story. The unconditional tenant manifest and theunknown-not-backfill call are both correct.manifest_value,manifest_has, andmanifest_foreign(commands/lib/manifest.sh:53,:70,:84) all use barewhile … readloops, so a populated final line with no trailing newline is never processed. On the repair path that resets pinned birth provenance (an unterminatedbootstrapped_at=…reads as absent, so Rule 1 regenerates the field it exists to pin), andmanifest_stampsilently eats an unterminated foreign key — the exact preservation contract judgment-call 5 promises. The fix idiom is already in this repo, in the same directory:parse_users_fileatcommands/lib/users-config.sh:49useswhile IFS= read -r line || [ -n "$line" ](and heavy-duty/cast#122 just landed the same shape for the same reason). Three one-clause edits plus the two no-trailing-newline fixtures (owned key, foreign key) codex asked for. A nice side effect:manifest_renderthen also normalizes such a file — the rewritten copy gets its newline back — so the repair path repairs this too.— automated review by
claude-bot-andresmgsl· heavy-duty-review-bot🔧 Changes requested — I agree with most; feedback below.
Formalizing my earlier 🔧 review of this same revision as a request-changes verdict (the specific points are in that review, unchanged) — reviews from this bot now carry a formal verdict state instead of a comment.
— automated review by
claude-bot-andresmgsl· heavy-duty-review-botRound answered in
c266863. One blocker, raised independently by two reviewers, and it was real — fixed. Details per reviewer below, but the short version: the bug was worse than the report, because it reached the convergence property this PR is entirely about.The defect, as measured
manifest_value,manifest_hasandmanifest_foreigneach read with a barewhile read, which stops at EOF without ever handing over a populated partial line. So an unterminated final line read as absent — and absent is exactly the input both rules key off, which makes the file's last line the one least able to survive the miss.Three concrete failures against the unfixed reader, in descending order of cost:
1. Rule 1 loses the birth stamp. A file truncated mid-write ends at
bootstrapped_at, so the unreadable line is the birth stamp itself:That is the one field that can never be reconstructed, and no later run undoes it.
2. Rule 2 lets the clock back in — the crux property, broken by a missing byte. A complete five-key file whose last line has no newline reads
converged_atas empty, so the "one-time" repair re-fires on every run:The render stopped being a function of (existing file, running version). This is worth stating plainly because the PR body argues hard for exactly that purity: the property held for every file the writer itself produced, and failed for a hand-edited one. codex and claude both diagnosed it as a reset of pinned provenance; it is also a convergence break, which is the more expensive half.
3.
manifest_foreignate an unterminated foreign line outright, so the writer dropped a later command's provenance — the exact preservation contract judgment-call 5 promises.The fix
Three one-clause edits, using the idiom already in this directory —
lib/users-config.sh:49reads|| [ -n "$line" ]for the same reason (and, as claude noted, heavy-duty/cast#122 just landed the same shape):Per reviewer
codex-bot-andresmgsl — CHANGES_REQUESTED
while readdrops the final unterminated record in all three readers — fixed, as above. Your causal chain is right on both counts:bootstrapped_at=unterminated is treated as absent and Rule 1 regenerates it, andmanifest_stampsilently deletes an unterminated foreign key.claude-bot-andresmgsl — CHANGES_REQUESTED
Concurring blocker, independently verified (
manifest.sh:53,:70,:84) — fixed. Theparse_users_filepointer was the right one and is what the fix uses."a nice side effect:
manifest_renderthen also normalizes such a file — the rewritten copy gets its newline back" — confirmed, and pinned as a test. Worth flagging how that test is written, because the obvious form of it is worthless: a plain "does the output end in\n" check stays green on the unfixed reader, since a file with its tail eaten is newline-terminated too. So the assertion is the source, plus the newline it was missing, and nothing else:That version does bite (RED below). I mention it because it is the same failure mode as the writer-level-vs-renderer-level test in the PR body — a test that passes for a reason unrelated to what it claims.
grok-bot-andresmgsl — APPROVED
Nothing to answer; noting only that the approval's summary of the design is accurate and none of it changed. The fix is confined to how the three readers reach EOF — the rules, the purity of
manifest_render, the key set, placement, and the five judgment calls are all untouched.Evidence
Every new test observed RED against the unfixed reader (
git stashoncommands/lib/manifest.shalone, tests unchanged):(An earlier draft of the normalization pair passed under the broken reader; that is why it now reads
adds_only_the_newline. The seventh test, "the normalized file then settles", is a settle-check on the repaired file rather than a regression guard.)Restored:
CHANGELOG untouched: this fixes an unreleased feature whose entry the same PR adds, so there is no shipped behavior to record a change against.
✅ Approved — I agree with all of this, no concerns.
The blocker from
66cf47fis resolved atc266863: all three readers (manifest_value,manifest_has,manifest_foreign) now carry the|| [ -n … ]clause, matchingparse_users_file's idiom in the same directory. The test work is better than the fix required: the six assertions were observed RED against the unfixed reader, the fixtures hit both rules at their weakest point (the unterminated line is the birth stamp / isconverged_at), andadds_only_the_newlineis the honest form of the normalization pin — the plain does-it-end-in-newline check would have stayed green on a reader that ate the tail, and catching that before shipping it is the same discipline as the PR body's writer-level-vs-render-level story. Nothing outstanding; the rules, the pure renderer, and the five judgment calls are untouched by the fix, as claimed.— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: I agree with everything and have no additional feedback.\n\nThe head-current fix at
c266863resolves the prior blocker in all three readers with the repository’s partial-EOF idiom. The new tests cover unterminated owned and foreign tail records, preserve the birth stamp, keep the clock out of same-version renders, normalize only the missing newline, and confirm the repaired file settles. Current checks are green.Rebased onto
mainafter #74 merged, and re-requesting all three bots.On the
state:addressinglabel — it was right, not a glitch. @grok-bot-andresmgsl's approval was bound to66cf47f, the previous head; @claude-bot-andresmgsl and @codex-bot-andresmgsl approvedc266863. The reconciler counts only head-current verdicts and treats a stale one as "the author owes a re-request" — which is exactly whatstate:addressingmeans. Two head-current approvals out of three is not a passed round.This rebase had real code conflicts, not just the changelog cascade — worth reading rather than trusting:
commands/bootstrap.sh— #79 renamed the box-install section header (host-class only→host=yes only) in the same hunk this PR inserts the provenance block into. Kept the provenance block and main's header wording; asserted afterwards that nohost-class onlysurvives.commands/bootstrap-tenant.sh— #80 renamed theharden_sshdcomment to saystaging-box tenant, while this PR adds thelib/manifest.shsource line two lines below. Kept main's rename and added the source line; asserted both are present.CHANGELOG.md— this PR's/etc/rig/manifestentry (#61) is an Added entry, and #74 merged another one (rig platform, #64). Placed beside it under### Addedrather than appended after### Fixed, so the section order (Added → Changed → Fixed) holds.Neither code conflict was a semantic clash — the provenance stamp and the role/root-door rename are independent — but both landed in hunks the rename had touched, which is why they needed reading rather than a mechanical resolution.
Verdict: Approve — I agree with this as-is at
d2e03ce.Re-review after the explicit re-request + rebase onto main (post-#74). Head moved from my prior approve at
66cf47f→d2e03ce(partial-EOF fix + rebase).What holds
/etc/rig/manifest; specs stay withrig platform. key=value 0644 matches the tree's no-jq constraint.converged_*only on version change. Puremanifest_render+ year-apart clock pin is the assertion that actually bites.d2e03ce— all three readers (manifest_value,manifest_has,manifest_foreign) use|| [ -n … ], matchingparse_users_file. Tests pin the expensive failures: unterminated birth reborn, clock re-entering via unreadableconverged_at, foreign tail eaten, andadds_only_the_newline(not a weak ends-in-\ncheck)./etc/rig/roleuntouched.host=yes/staging-boxwording compose cleanly; CHANGELOG entry under### Addedbesiderig platform.CI green (check, db-integration, reconcile, scope). No blockers.
✅ Approved — I agree with all of this, no concerns.
Re-approving at
d2e03ceafter the rebase onto main (my prior approval was bound toc266863). The two commits are the same logical content I approved — the feature plus the partial-EOF reader fix, with all three readers (manifest_value,manifest_has,manifest_foreign) still carrying the|| [ -n … ]clause — and I verified each of the three flagged conflict resolutions in the current diff rather than trusting the summary:commands/bootstrap.sh— the box-install header readshost=yes only(main's #79 wording; nohost-class onlysurvives) with the provenance block intact directly after the marker.commands/bootstrap-tenant.sh— main'sharden_sshd (the staging-box tenant)comment and the newlib/manifest.shsource line are both present.CHANGELOG.md— the #61 entry sits under### Addedahead of### Changed, so the section order holds.CI is green on the new head (check, db-integration, reconcile, scope). Nothing in the rebase changes the rules, the pure renderer, or the judgment calls, and
rig platform(#74, now on main) is untouched by this diff as designed.— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: I agree with everything and have no additional feedback. The renewed review at
d2e03cepreserves the prior partial-EOF fix across all three manifest readers, and the rebase conflict resolutions retain the provenance paths alongside the current main-branch wording and changelog structure. All current checks pass.Post-mortem
Round passed — three head-current approvals at
d2e03ce, CI green,CLEAN. Handing to @danmt. This PR took an unusually long path to get here, and three things in it are worth writing down rather than forgetting.1. A stale approval made a failed round look passed
After the partial-EOF fix landed at
c266863, @claude-bot-andresmgsl and @codex-bot-andresmgsl re-approved — but @grok-bot-andresmgsl was never re-requested, so its approval stayed bound to66cf47f. Two head-current approvals out of three. The board readstate:addressingand that looked like a label bug; it was not. The reconciler counts only head-current verdicts and treats a stale one as the author owes a re-request, which is exactly what that label means.The failure mode is the dangerous direction: the PR appears reviewed while one reviewer has never seen the code at that head. GitHub keeps approvals alive across pushes, so nothing surfaces it except the label — and the label was disbelieved.
The rule this earns: re-request every reviewer after every push, not just the ones who did not approve. CONTRIBUTING says "re-request the bots that didn't approve", which is right within a round but reads as permission to leave an approver alone after a push. A push invalidates approvals just as thoroughly as a request-changes does.
2. The rebase conflicts were positional, not semantic — and that is exactly why they needed reading
Three conflicts, none a real disagreement:
commands/bootstrap.sh— #79 renamed the box-install header (host-class only→host=yes only) in the same hunk this PR inserts the provenance block into.commands/bootstrap-tenant.sh— #80 renamed theharden_sshdcomment two lines above where this PR adds itslib/manifest.shsource.CHANGELOG.md— this PR's#61entry is an Added entry, and #74 had merged another one.Provenance stamping and the role/root-door rename are completely independent features. They collided purely because they touched adjacent lines. A mechanical "take theirs" on the first would have silently reverted #79's rename; "take ours" would have dropped the provenance block. Both directions produce a green suite — which is what makes this class worth slowing down for.
3. The changelog cascade is a structural problem, now filed
Nearly every rebase in this batch hit
CHANGELOG.md, and none of those conflicts were about content. Filed as #86 (siblings heavy-duty/box#135, heavy-duty/cast#126): seed## Unreleasedwith all three headings and let the-devbump write them, so PRs only ever append under an existing heading. It fixes the disjoint-section collisions — including the ordering trap where a new### Addedhad to be hoisted to keepAdded → Changed → Fixed— and explicitly does not fix same-section ones.What a merger should know
The substance was already sound before any of the above; the delay was process, not defects. The one real code finding in the whole history was the partial-EOF reader bug, caught in round one and fixed across all three readers (
manifest_value,manifest_has,manifest_foreign) with the same|| [ -n … ]clauseparse_users_filealready uses./etc/rig/roleis untouched — provenance is a second file with a second job, and the marker keeps its six readers.