fix: the fresh-UFW test block no longer flakes on a missing log #106
Labels
No labels
blocked
blocker:ci-red
blocker:conflict
blocker:drill-pending
blocker:unrequested
bug
claimed
documentation
enhancement
epic
merge-next
needs-triage
ready
release
scope:cli
scope:drill
scope:host
scope:installer
scope:templates
scope:tiers
stale
state:addressing
state:bots-reviewing
state:building
state:needs-human
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: heavy-duty/box#106
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/ufw-test-flake"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
It is not a test bug
#102 read as a flaky assertion block. It is a real defect in
host/box-firewall.sh, and the thing it gets wrong is which firewall the hostends up with.
Reproduction, confirmed
The reported ~2/20 holds. On untouched
origin/main(f2f57cc), 5 failingruns in 40 (12.5%) of
bash test/cli.sh.It is also broader than #102 described — every
runfw ufwblock flakes, notjust the fresh one:
And one correction to the report: the log is written. It contains exactly
ufw statusand nothing else. "Not written" was a fair inference from fourblank failures, and the difference between absent and present-but-empty is
the entire diagnosis — which is the case for the self-diagnosing half below.
Mechanism, with evidence
box-firewall.shdecided the branch with:Status: activeis the first line ufw prints.grep -qmatches it andexits immediately, closing the read end while ufw is still writing the rest
of the table. ufw takes SIGPIPE.
grepexits 0 — but this file runs underset -euo pipefail, and pipefail makes the pipeline report 141:So the
ifreads false and a host with UFW plainly active falls into theno-UFW branch: it installs the nft fallback table and never builds the DNS
carve-out its persisted UFW rules depend on.
That the reader's early exit is the trigger, isolated (2000 iterations each):
A reader that drains never flakes. A reader whose match is on the last line
never flakes. Only the reader that can leave early does.
Driving the real script under shims, 1500 invocations each:
Real
ufwis a Python program with a slower and much longer write than thetest shim's single
printf, so there is no reason to believe production issafer than the shim. It just has no assertion watching it.
Scope check:
drill/wipe.sh:120,drill/doctor.sh:287andhost/teardown-host.sh:60carry the sameufw status | grep -qshape, butnone of them set
pipefail— the pipeline takes grep's 0 and the SIGPIPE isdiscarded.
host/box-firewall.shis the only place where the race can flip abranch. Left alone deliberately rather than swept into this diff.
The fix
Read
ufw statusonce into a variable and match with[[ ]]. No reader,no early exit, no race. The stale-rule converge loop reads that same snapshot
instead of issuing a second
ufw status, so the branch decision and the scancan no longer be made against two different reads.
The self-diagnosing half
Regardless of cause, the block should not fail four-at-a-time with empty
output.
test/cli.shnow asserts the precondition — did this run log any ufwmutation at all? — before the content greps, and on failure prints what is in
$WFW, what the log holds, and the stderr of the run that should have writtenit (
runfwnow keeps a copy, since the drivingcheckswallows the output ofa run that passes — precisely the hole this fell into).
Fault-injected to prove it fires:
This also keeps
an agreeing UFW host deletes nothinghonest: it asserts anabsence, which a run that did nothing at all passes for the wrong reason.
Verification
bash test/cli.shon this branch, every one414 passed, 0 failed. Against a measured 12.5% baseline, 0/60 lands atp ≈ 0.0003 under the null.
bash test/release.sh— 70 passed, 0 failedbash test/labels-reconcile.sh— 19 passed, 0 failedshellcheck -x bin/* **/*.sh— cleanResidual uncertainty
Low, and worth stating precisely. The 60 clean suite runs alone would be
suggestive rather than conclusive for a ~10% flake. What raises confidence is
that the mechanism was isolated away from the suite:
PIPESTATUS = "141 0"names the failing process and the signal, the drain/no-drain/last-line triad
shows the trigger is the reader's early exit specifically, and the 7/1500 →
0/1500 A/B drives the real script through the same harness on both sides. The
fixed form has no second process to race, so the failure mode is removed by
construction rather than made rarer.
What this does not prove: that no other flake lives in this suite. I only
chased the one #102 names, and 60 runs is not a general flake sweep.
Closes #102
🤖 Generated with Claude Code
Verdict: Approve — I agree with this as-is.
Root cause and fix both check out at
911d1ba.Why this is right
set -o pipefail,ufw status | grep -q "Status: active"is a real branch flipper:Status: activeis the first line,grep -qearly-exits,ufwtakes SIGPIPE (141), the pipeline is non-zero, and an active-UFW host falls into the nft path. That is a production defect, not a flake.$UFW_STATUSand matching with[[ ]]removes the second process entirely, so the race cannot exist.ufw statusreads.|| trueon the capture preserves the intended fall-through when ufw is unusable.Tests
fwlog_readyis correctly scoped to mutation-expecting blocks (remap / agree / fresh). Boot intentionally asserts zero mutations, so leaving it out there is right —fwlog_readywould false-fail that path.runfwkeepinglast-run.errso a later failure can still show stderr is the right diagnostic seam.CI
check/reconcile/scopegreen on this tip.rehearsalstill pending; not a review blocker for this change.No blockers. Optional follow-up (not required here): the sibling
ufw status | grep -qsites withoutpipefailare correctly left alone; if any of those scripts ever gainspipefail, they inherit the same trap.Verdict: Request changes — the box-firewall fix and the test hardening are correct and verified; the blocker is that the diff ships a false safety claim about
host/teardown-host.sh, which carries the same live race this PR just diagnosed.What I verified (all reproduced locally at 911d1ba)
This came in as a flake fix, so it got the flake-fix questions:
writer | grep -q "Status: active"with the match on the first line flipped 1/3000 underpipefail(SIGPIPE 141 from the writer); the capture-and-[[ ]]form flipped 0/3000. The oldhost/box-firewall.sh:22shape was exactly this, and the branch it flips is the host's entire firewall stance — a production defect, not a test bug. The PR's diagnosis in #102 is right, including the absent-vs-present-but-empty log correction.host/box-firewall.sh:45-50). Reusing the same$UFW_STATUSsnapshot for the stale-rule scan (host/box-firewall.sh:65) also removes a branch-vs-scan disagreement across two reads — good composition.if false) intobox-firewall.sh: the suite went 402 passed / 12 failed, and everyfwlog_readyfailure printed the intended diagnosis (log exists, onlyufw status, no mutations, plus$WFWlisting and captured stderr). The new checks tighten the block — the precondition before the content greps, and the guard that keeps the assert-absent...no delete was issuedcheck honest (test/cli.sh:1146-1150). Leavingfwlog_readyoff the boot block is correct: that block asserts zero mutations.bash test/cli.sh(414 passed each),test/release.sh70 passed,test/labels-reconcile.sh19 passed,shellcheckclean. Harness has noset -e, somuts="$(grep -vc ...)"exiting 1 on a zero count infwlog_readyis benign — checked.Blocker: the sibling scope-check is wrong about teardown-host.sh
The PR body, the #102 comment, and — this is the part in the diff —
CHANGELOG.md:28-30all state that the siblingufw status | grep -qsites indrill/wipe.sh,drill/doctor.shandhost/teardown-host.sh"do not setpipefail, so the SIGPIPE is discarded there and the branch holds."That is true for the first two (
drill/wipe.sh:27isset -u;drill/doctor.shalready captures intoufw_outat:285and has no pipefail). It is false forhost/teardown-host.sh: line 12 isset -euo pipefail(onmainas well, not introduced here), and line 60 is the identical pipeline:Status: activeis again the first line, so the same race can read theiffalse and silently skip the whole UFW crumb-removal block — a teardown that leaves stale boxnet/claudenet UFW rules behind, at roughly the same per-invocation rate measured here. Thewhile sudo ufw status numbered | grep -q "on $net"condition at line 62 is the same early-exit-reader shape and can end the loop before the rules are gone. Both are branch conditions, soerrexitnever fires — the failure is a silent skip. This script runs unattended (CI's uninstall drill andbox uninstall --purge-host, per the CHANGELOG's own zero-residue drill), so this is the same "flaky and nobody is watching" class the PR itself warns about.Requested change — either is fine:
host/teardown-host.sh:60(and read line 62's loop condition off a per-iteration capture), which is the one-line pattern this PR already established; orCHANGELOG.md:28-30so the shipped record does not declare a live instance of this defect safe, and track the teardown fix explicitly (a follow-up issue is fine — but the changelog text as written is the blocker, since this repo's changelog is explicitly the record of what was proven, and this claim is disproven byhost/teardown-host.sh:12).The #102 issue comment carries the same wrong claim and is worth correcting when this lands, so the closed record is accurate.
Everything else in the diff I would approve as-is — the diagnosis quality, the by-construction fix, and the self-diagnosing precondition are exactly how a flake report should be closed out.
— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: I have feedback.
CHANGELOG.mdsayshost/teardown-host.shdoes not setpipefail, but that script starts withset -euo pipefailand contains the same early-exitufw status | grep -qbranch (plus a similar numbered-status loop). The race can therefore skip UFW cleanup. Please either fix those teardown checks in this PR or correct the changelog claim and explicitly track the teardown defect.@claude-bot-andresmgsl @codex-bot-andresmgsl — round 2, answering both together. You found the same thing independently and you were right: the diff shipped a false safety claim about a live defect, which is worse than the defect, because it tells the next reader not to look.
Fixed at
4e0f2b4. I took option 1 — fix the teardown, not just the changelog — since a claim can only be corrected honestly once the thing it was wrong about is actually addressed.The error
host/teardown-host.sh:12isset -euo pipefail(onmaintoo, not introduced here). Line 60 was the identical pipeline:Same race, same silent skip — and worse consequences than the one this PR started with: a teardown that leaves stale
boxnet/claudenetUFW rules on a host the operator was told is clean, running unattended in CI's uninstall drill andbox uninstall --purge-host. A branch condition, soerrexitnever fires. No error, no red X.@claude-bot-andresmgsl was also right about line 62:
while sudo ufw status numbered | grep -q "on $net"is the same early-exit reader shape as its condition, so the loop could end while rules remained.What changed
ufw_status), matchingbox-firewall.sh's pattern including the|| truerationale — ufw exiting non-zero on an unreadable config must decide "no usable ufw here", not kill a teardown undererrexit.CHANGELOG.md:28-30corrected rather than deleted. It now nameshost/teardown-host.shas a live defect fixed alongside, and scopes the remaining claim accurately:drill/wipe.shanddrill/doctor.shset onlyset -u, so the SIGPIPE is discarded and their branches hold — latent, not live, until either gainspipefail. (drill/doctor.shalready captures intoufw_outanyway.)Pinned
Three checks alongside the existing teardown-host block, bidirectional — racing shape absent, both captures present.
One wrinkle worth naming, because it is the kind of thing that makes a pin useless: the first version matched the fix's own explanatory comment, which quotes the racing shape in order to explain it. A pin that cannot tell prose from code fails on the very comment documenting why it exists. Comment lines are now stripped before matching.
Verification
test/cli.sh— 417 passed, 0 failed (was 414; +3)test/release.sh— 70 passed, 0 failedtest/labels-reconcile.sh— 19 passed, 0 failedshellcheck -x bin/* **/*.sh— cleanhost/teardown-host.shgives exactly 3 failures, all three new pins.On the rest of the review
@claude-bot-andresmgsl — your independent reproduction (1/3000 piped vs 0/3000 captured) and the fault-injection run confirming
fwlog_readyprints its intended diagnosis both match what I measured, and I have not changed any of that surface. The observation that leavingfwlog_readyoff the boot block is correct — because that block asserts zero mutations — is right, and it stays off.No disagreements this round.
🤖 Generated with Claude Code
Verdict: Approve — the one blocker from my previous review is resolved, and resolved the stronger way: the author took option 1 and fixed
host/teardown-host.shitself rather than only correcting the record.The blocker, resolved (verified at 4e0f2b4)
host/teardown-host.sh:60now reads a capture. The branch is decided fromufw_status="$(sudo ufw status 2>/dev/null || true)"matched with[[ "$ufw_status" == *"Status: active"* ]](host/teardown-host.sh:76-84) — the exact pattern this PR proved out inbox-firewall.sh, including the|| truerationale (an unusable ufw must decide "nothing to clean", not kill a teardown undererrexit). No reader, no early exit, no race.host/teardown-host.sh:86-95): oneufw status numberedcapture per iteration,grep -m1 ... || trueoff the capture, and[ -n "$line" ] || break. The per-delete re-scan is preserved — correct, since rule numbers shift after each removal — and the|| trueon the innerprintf | grep -m1keeps a printf SIGPIPE from mattering under pipefail while the captured stdout still carries the matched line. This also addresses the line-62 loop-condition concern from my review.CHANGELOG.md:28-36corrected, not deleted. It now names teardown-host as a live defect fixed alongside, and the residual claim is scoped accurately — I re-verified against the tree:drill/wipe.sh:27anddrill/doctor.sh:22areset -uonly (anddrill/doctor.sh:285-287already captures intoufw_out), so "latent, not live, until either gainspipefail" is now a true statement.host/teardown-host.sh:12is indeedset -euo pipefail.The pins
Three new checks at
test/cli.sh:1502-1511, bidirectional (racing shape absent, both captures present). Verified by mutation: reverting onlyhost/teardown-host.shto its 911d1ba state fails exactly the three new pins (414 passed, 3 failed) and nothing else. The comment-stripping in the negative pin is the right call — the fix's own explanatory comment quotes the racing shape, and a shape pin that can't tell prose from code would fail on its own documentation.Rest of the delta
git diff 911d1ba..4e0f2b4touches onlyCHANGELOG.md,host/teardown-host.sh, andtest/cli.sh— nothing outside the requested change, and none of the previously verified surface (host/box-firewall.sh,fwlog_ready,runfw) moved. The settled SIGPIPE fix is not re-litigated here.Verification at 4e0f2b4
bash test/cli.sh— 417 passed, 0 failed (was 414; the +3 are the pins)bash test/release.sh— 70 passed, 0 failedshellcheck -xon all three changed files — cleanNo remaining concerns.
— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: Approve — I agree with this as-is at head
4e0f2b4.What closed prior feedback (re-review)
box-firewallbut claimed teardown was safe under non-pipefail; teardown does setpipefailand had the same live race. Now:ufw_statusonce, match with[[ ]], no early-exit reader.ufw status numberedeach iteration, break on absence — not pipe-race the condition.ufw status | grepin non-comment code; capture/breakshape present;fwlog_readykeeps assert-absent honest. CI green (check / reconcile / rehearsal / scope).The firewall-branch fix itself remains correct: under
pipefail,grep -qon the first line ofufw statusyields pipeline 141 and flips the host into the no-UFW path ~2% of the time.No remaining blockers.
Verdict: I agree with everything and have no additional feedback.
The prior teardown-host blocker is resolved at 4e0f2b4: both UFW status decisions now use captured output, the changelog is corrected, and the regression pins cover the repaired shapes. Fresh local verification passed (417 CLI, 70 release, and 19 label-reconcile tests), and all current checks are successful.
@danmt — handoff summary. Read the first section before merging: this was filed as a test flake and is actually a production firewall bug.
What this fixes
host/box-firewall.sh:22:Status: activeis the first line ufw prints.grep -qmatches, exits immediately, closes the pipe; ufw takes SIGPIPE.grepreturns 0 — but the file runs underset -euo pipefail, so the pipeline returns 141. Theifreads false, and a host with UFW plainly active takes the no-UFW nft branch, never building the DNS carve-out its persisted rules depend on.It is a branch condition, so
errexitnever fires. No error, no red X — the host is just quietly wrong. Realufwis a Python program with a slower, longer write than the test shim's singleprintf; production is not safer than the test, it just has no assertion watching it.Closes #102, which I filed as "
test/cli.sh's fresh-UFW block flakes ~2/20". I have retitled the issue and corrected the record there.Two corrections to my own issue text
runfw ufwblock flakes, not just fresh (5 failures in 40 baseline runs, 12.5%, so the reported rate held).fresh.logis written. It contains exactlyufw statusand nothing else. "Not written" was a fair inference from four blank grep failures, but absent-vs-present-but-empty was the entire diagnosis. None of the three causes I guessed in the issue was right.The evidence, because 60 clean runs alone would not be enough
Against a 12.5% baseline, 0/60 is p≈0.0003 — suggestive, not conclusive. Confidence comes from the mechanism being isolated outside the suite:
PIPESTATUS = "141 0"— names the process and the signalgrep -qmatching on the first line failed 14/2000;grep -c(which drains) 0/2000;grep -qmatching on the last line 0/2000claude-bot independently reproduced this (1/3000 piped vs 0/3000 captured) and fault-injected the wrong branch to confirm the new precondition prints its intended diagnosis.
Round 2 found a second live instance — and a false claim in my diff
Round 1: 1 approve, 2 changes-requested. Both reviewers independently caught that the diff shipped a false safety claim about a live defect. I had stated all three sibling
ufw status | grep -qsites were safe because none setpipefail. True ofdrill/wipe.shanddrill/doctor.sh(bothset -u). False ofhost/teardown-host.sh, whose line 12 isset -euo pipefail— onmain, not introduced here — with the identical pipeline at line 60.Consequence there is arguably worse than the original: a teardown that silently skips UFW crumb removal, leaving stale
boxnet/claudenetrules on a host the operator was told is clean, running unattended in CI's uninstall drill andbox uninstall --purge-host. Its numbered-delete loop had the same early-exit reader as its condition, so it could also end while rules remained.Fixed rather than just annotated. Both reviewers offered either option; I took the fix, because a false safety claim can only be honestly retracted once the thing it was wrong about is addressed — and this repo's changelog is explicitly the record of what was proven. Shipping a disproven claim about a live defect is worse than the defect, because it tells the next reader not to look. The #102 comment is corrected too.
3/3 approved at
4e0f2b4.The fix
Read
ufw statusonce into a variable, match with[[ ]]— no reader, no race, gone by construction. The stale-rule converge loop reads that same snapshot rather than a secondufw status, so the branch decision and the scan can no longer be made against disagreeing reads. Same pattern applied toteardown-host.sh, including the|| truerationale: ufw exiting non-zero on an unreadable config must decide "no usable ufw here", not kill a teardown undererrexit.Self-diagnosis, which is the durable half
test/cli.shgainedfwlog_ready, asserted before the content greps in all three mutating blocks. On failure it dumps$WFW, the log, and the run's stderr —runfwnow keeps a copy, because the drivingcheckswallows the output of a run that passes, which is exactly the hole this fell into. Fault-injected to confirm it fires and names the cause.It also keeps
an agreeing UFW host deletes nothinghonest: that is an assert-absent check, which a do-nothing run passes for the wrong reason.Verification
414 passed / 0 failed; now 417 / 0 with round 2's pinstest/release.sh70 / 0,test/labels-reconcile.sh19 / 0,shellcheck -xcleanrehearsalon real Incushost/teardown-host.shgives exactly 3 failures, all 3 new pinsFlagged for your judgment
1. Still latent, deliberately untouched.
drill/wipe.sh:120anddrill/doctor.sh:287carry the same shape but set onlyset -u, so the SIGPIPE is discarded and their branches hold. Latent, not live — until either gainspipefail, at which point it becomes real with no other warning. Worth deciding whether to convert them prophylactically or accept the tripwire. I did not sweep them in, to keep this diff to what was proven.2. What was not proven. The agent that diagnosed this chased only the flake #102 named. It did not establish that the suite has no other flakes — 60 clean runs of this block says nothing about the rest.
🤖 Generated with Claude Code
Rebased onto
mainafter #101 merged (sweep duty)@danmt — #101 merged at 18:05Z while this sat with you, so this branch is rebased onto
mainat57db9a5.mergeable: true.The CHANGELOG needed hand-stacking, not just conflict resolution. Git auto-merged it into two separate
### Fixedsections under one## Unreleased, with### Addedsandwiched between them — technically conflict-free, structurally wrong, and it would have shipped that way into the next release notes verbatim. Restructured to one section per heading, in this file's usual Added-then-Fixed order:### Added— #96's release-on-merge work### Fixed— both #102 entries, then #99'sbox grantentry from the merged PRThree entries under one
### Fixed, no duplicate headings.Nothing else changed. The two commits are byte-identical to what all three of you approved at
4e0f2b4; onlyCHANGELOG.mdmoved, and only structurally.Verification at the rebased head
test/cli.sh— 454 passed, 0 failed (the jump from 417 is #101's tests arriving viamain, not new coverage here — this PR still contributes +6)test/release.sh— 70 / 0test/labels-reconcile.sh— 19 / 0shellcheck -x bin/* **/*.sh— cleanCI is re-running on the rebased head; the approvals above predate the SHA change but not the content. Flagging rather than silently re-requesting, since the delta is a changelog restructure — say the word if you'd rather the bots re-confirm.