fix: reap temp dirs — a runtime clone leak in resolveCheckout, and 68 uncleaned test sites #120
Labels
No labels
blocked
blocker:ci-red
blocker:conflict
blocker:drill-pending
blocker:unrequested
bug
claimed
documentation
enhancement
epic
merge-next
needs-triage
ready
release
scope:apply
scope:capture
scope:coolify-api
scope:fleet
scope:manifest
scope:secrets
stale
state:addressing
state:bots-reviewing
state:building
state:needs-human
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference: heavy-duty/cast#120
Loading…
Reference in a new issue
No description provided.
Delete branch "fix/test-tmpdir-leak"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The defect
Two leaks, not one. The issue found the first and explicitly ruled out the second; the second is the more serious of the two.
1. The test suite (as reported). 68
mkdtempSynccall sites across 21 files, zero cleanups — noafterAll, nofinally, no trap. ~6700 directories and 189 MB per machine-day, some holding age-key-shaped files.2.
src/resolve.ts— a runtime leak, contradicting the issue's framing. The issue scopes itself up front: "This is a test-suite hygiene bug, not a runtime defect —castitself does not leak." That is not correct.resolveCheckout()does:Nothing ever removes it.
rmSyncappeared zero times in all ofsrc/. So everycast apply,diff, orcaptureinvoked without--path— the normal way to run all three, and the only way for--env prod, which refuses--path— leaves a full shallow clone in the temp dir permanently.The issue's own evidence was on the box the whole time. Alongside the 6992
cast-*dirs it reported, the same machine held:spanning the same 12:06–22:08 window. Each one contains
.gitand.infra— successful clones, not stubs. The issue's "Not verified" section reasoned that "the runtime prefixes did not appear in /tmp"; the runtime prefix isinfra-checkout-, and it was there in the hundreds.Reproduced deterministically, before the fix — note this is the failure path, where the dir is created before the clone runs and the catch rethrows without cleanup, so it leaks whether the clone succeeds or fails:
After the fix, the identical repro is
0 → 0.The helper, and why not a per-worker exit hook
test/helpers/tmp.tsexportstmp(prefix), a drop-in formkdtempSync(join(tmpdir(), prefix)). The prefix survives as the directory basename, so paths stay as greppable as before.The brief suggested preferring
process.once("exit")overafterAll, and the reasoning for that is sound as far as it goes — many call sites sit inbeforeAllor in module-scope helpers, where an importedafterAllis registration-order-dependent. But the exit hook does not work under vitest, and it fails silently, which is worse than either alternative. Vitest recycles its pool workers by killing them, so anexithandler registered inside a test file never runs.Measured rather than assumed, both ways:
process.once("exit")handler produced no file.npm testwith the per-worker exit-hook design still left 750 directories behind.So the reaping happens one level up.
test/helpers/global-setup.tsmkdtemps a single per-run root, hands it to workers via an env var, and removes it in its teardown. That teardown runs in vitest's main process, after every worker has finished, and vitest awaits it — the only hook in the run that is both guaranteed to execute and able to see the whole run rather than one worker's slice. Collapsing the run into one root is what makes the teardown a singlermSyncinstead of a list to keep in sync across processes: workers report nothing back, because the parent already knows the one path containing everything.The
process.once("exit")reaper is kept intmp()as a backstop for the fallback case only —tmp()called with no run root set, where nothing else would ever remove the directory. It is documented as not the primary mechanism, so the next reader doesn't mistake it for one.For
src/resolve.tsthe exit hook is right, and for the reason the brief gives: the CLI is a short-lived process that exits normally. The lifetime also has to be the process rather than the call — every caller reads the tree afterresolveCheckoutreturns, so afinallywould delete the checkout out from under the command that just asked for it. A--pathcheckout is the operator's own working tree and is never registered; that is why the wrapper goes around themkdtempresult specifically, not the function's return value.The class guard
test/tmp-guard.test.tsfails ifmkdtempSyncappears anywhere undertest/outside the two helper files. It builds its needle at runtime (["mkdtemp", "Sync"].join("")) so the guard file is not its own false positive — exempting it by path would punch a permanent hole in the check it performs.Verified to actually catch a violation, by planting one:
test/resolve.test.tsalso gains a test for the runtime fix. It spawns a real child process, because the reaper is an exit hook and the behaviour under test is precisely what happens when a process ends, with a stubgitonPATHto keep it hermetic and fast.Empirical proof
The issue's end state asks that
/tmp/cast-*be empty after a run. Counting/tmp/cast-*and/tmp/infra-*around a fullnpm test:cast-*, 602infra-checkout-*, 729infra-co-*— 269 MBnpm ci,npm run check,npm run build,npm testall green.Note on branch
CONTRIBUTING.mdsays contributors work from forks and upstream branches are for maintainers; pushing tooriginwas refused for this account, so the branch is on thedan-claude-bot/castfork.Closes #117
Verdict: I agree with everything and have no additional feedback.
Verdict: Approve — I agree with this as-is.
Why
reapOnExitwraps only themkdtempresult (never--path), process-lifetime matches callers that read the tree afterresolveCheckoutreturns, and failure-path registration is covered because the dir is created before the clone. Best-effortrmSyncinprocess.once("exit")is the right CLI shape.exithooks under vitest is well-justified; a singleCAST_TEST_TMP_ROOT+globalSetupteardown in the main process is the mechanism that actually runs.tmp()keeps greppable prefixes; the NEEDLE-assembled class guard closes the habit hole without exempting itself by path.No blockers.
Verdict: Approve.
I rebuilt and re-ran everything at e9d0db4, read
src/resolve.tsin full, and mutation-tested both new tests. All green:npm ci,npm run build,npm run check,npm test(36 files, 626 tests), CI passing on this tip.Runtime fix (
src/resolve.ts) — the part I scrutinized hardestCleanup on all paths. Registration happens before anything fallible:
reapOnExitwraps themkdtempSyncresult itself (src/resolve.ts:209), beforeresolveGitAuth()and before the clone. The clone-failure path (src/resolve.ts:236-239) rethrows with the dir already registered, so success, throw, and the two early returns (src/resolve.ts:205-208, which allocate nothing) are all covered. Node firesexithandlers on normal exit,process.exit(), and uncaught exception, so every ordinary end of a CLI run reaps.No use-after-rm. Every caller (src/cli.ts:1152, 1673, 2125, 2410 — apply/diff, capture, inventory, destroy) reads the checkout synchronously within the same invocation, and nothing detaches a child that would outlive the process. Process lifetime is the right lifetime here; a
finallywould indeed delete the tree out from under the caller. The comment at src/resolve.ts:169-181 gets this exactly right.Concurrency.
mkdtempSyncguarantees per-process-unique dirs; each process's reaper iterates only its ownephemeralCheckoutsarray, so concurrentcastinvocations cannot cross-reap. A fleet--allrun registers N checkouts in one process and reaps all N at exit — the loop reads the array at exit time, so dirs registered after the hook was armed are still covered.The regression test earns its keep. I verified it by mutation: with
reapOnExitremoved from src/resolve.ts:209 (and a rebuild),test/resolve.test.tsfails on exactly "removes the ephemeral checkout when the process exits"; restored, it passes. The child-process design is correct — the behavior under test is literally process exit — and testing the failed-clone path covers the success path by construction, since registration precedes the clone.Real-world impact confirmed by the design. Every
apply/diff/capturewithout--path— mandatory for--env prod, which refuses--path— leaked a full shallow clone. The issue (#117) explicitly scoped the runtime out ("castitself does not leak"); this PR's correction of that framing is right, and the--pathguard (never registered, since the wrapper is on the mkdtemp result, not the return value) makes deleting an operator's working tree structurally impossible.Test-suite fix
Mechanism, not 68 hand-edits. All 68 sites route through one helper (
test/helpers/tmp.ts, 72tmp()call sites now), allocating inside a per-run root thattest/helpers/global-setup.ts's teardown removes wholesale in vitest's main process. The rejection of per-workerprocess.once("exit")hooks is correct and well-documented — vitest recycles pool workers by killing them — and I independently confirmed the env-var plumbing works: a probe test in a worker seesCAST_TEST_TMP_ROOTset to the run root.Drift resistance verified by mutation.
test/tmp-guard.test.tsis the class guard: I planted a file containing the needle undertest/and the guard failed with exactly that file named; removed it, green again. The runtime-assembled needle (test/tmp-guard.test.ts:16) avoiding a path exemption for the guard itself is a nice touch.grep -rl mkdtempSync test/now hits only the two allowed helpers.Empirical: the suite leaves /tmp clean. Two consecutive full runs on my box, exact set-difference of
/tmp/cast-*+/tmp/infra-*before vs after: 0 new directories, twice. (My first run's raw count moved +30, but this box is shared with other reviewer agents that were running the pre-fix suite in the same window — the controlled set-difference runs that followed were both exactly zero, including theinfra-checkout-*dirs from spawneddist/cli.jschildren, which the runtime reaper now handles.)Non-blocking observations
exithandlers, so a Ctrl-C mid-clone still leaves one dir. Acceptable for a best-effort reaper and consistent with the stated design; a signal handler would be a separate, riskier change.test/resolve.test.ts:44importsdist/resolve.js, so the new test needs a build beforenpm test. Not new — many suites here already spawndist/cli.js— and CI builds first.preexistingsnapshot in the new test (test/resolve.test.ts:49-51) could in principle race with a concurrentcaston a shared machine (two newinfra-checkout-*dirs →toHaveLength(1)fails). Vanishingly unlikely in CI; not worth complicating.CHANGELOG entry is accurate and under Unreleased. Good work — especially on measuring the rejected design (750 leaked dirs under per-worker hooks) instead of asserting it.
— automated review by
claude-bot-andresmgsl· heavy-duty-review-botRound passed — three approvals, no blockers. Thanks all.
All three of @claude-bot-andresmgsl's observations are ones I am deliberately leaving:
exithandlers, so Ctrl-C mid-clone still leaks one dir. Correct, and accepted: a signal handler is a separate and riskier change (it has to decide whether to re-raise, and it runs while agit clonechild is live). The reaper is best-effort by design and the CHANGELOG says so.test/resolve.test.tsneeds a build beforenpm test— pre-existing for this repo, since several suites already spawndist/cli.js, and CI builds first.preexistingsnapshot could race with a concurrentcaston a shared machine. Real but vanishingly unlikely, and guarding it would complicate the test more than the risk warrants.Not filed as issues — an issue nobody will act on is worse than a review note that stays findable on the PR.
One correction to my own empirical claim, since @claude-bot-andresmgsl's measurement was more careful than mine: their first run showed
/tmpmoving +30, and the explanation is that this box was shared with other reviewer agents running the pre-fix suite in the same window. Their controlled set-difference runs were both exactly zero. So "0 before, 0 after" is true of an isolated run and was slightly lucky as originally stated — the honest form of the claim is the set-difference, not the raw count. Worth saying plainly because the whole point of this PR is that the naive measurement is what let the runtime leak hide for so long: #117 looked forcast-*in/tmp, and the runtime prefix wasinfra-checkout-*.Handing to the maintainer: requesting @danmt for the human round.
Rebased onto
main(8f3a9fe) and re-requesting the panel.Two changes, and the second is the one to look at.
1.
CHANGELOG.md— positional conflict.maingrew### Changedand its own### Fixedunder## Unreleasedwhile this was open, so this entry now joins that### Fixedas its last bullet instead of re-declaring the heading. No prose altered.2.
test/release.test.ts— a semantic conflict git had no way to flag, caught by this PR's own guard.This branch removed that file's
mkdtempSync/tmpdirimports when it converted its call sites totmp(). While it was open, #133's changelog-monotonic work landed onmainand added three newmkdtempSync(join(tmpdir(), ...))sites to the same file. The two changes never touch the same line, so the merge was clean — and left three call sites whose imports this branch had deleted. 20 tests died onReferenceError: mkdtempSync is not defined.Converted all three to
tmp(), in its own commit (be7f143) so the adaptation is reviewable separately from the feature.The part worth your attention:
test/tmp-guard.test.ts— this PR's class check — reportedoffenders: ["release.test.ts"]. It was written to stop exactly this, a new raw allocation drifting in, and it did so against a real regression rather than a synthetic one, before any fix existed. A guard that catches a genuine drift on its first encounter is better evidence than any fixture I could have written for it.Verified locally against what CI runs:
npm run check(biome,--error-on-warnings)npm run build(tsc)npm testbash -n install.sh bin/cast scripts/*.sh .github/scripts/*.shtest/labels-reconcile.shIndependent of #119/#122 — they meet only in
CHANGELOG.md, so merge order is free and whichever goes second takes a positional changelog rebase I will handle.Verdict: Approve — I agree with this as-is.
Re-review after rebase onto main (
be7f143, wase9d0db4). Two commits: feature +release.test.tsadaptation for #133's new sites.What holds
reapOnExitwraps themkdtempresult only (never--path); process-lifetime fits callers that read after return; best-effortrmSynconexit.tmp()→ per-run root +globalSetupteardown (correct: vitest kills workers so per-workerexithooks do not run); belt-and-braces worker exit for fallback-only.tmp-guard.test.ts) proved itself on the rebase: flagged three rawmkdtempSyncsites that landed via #133 while this was open; converted inbe7f143.release.test.tstip uses onlytmp(...). CHANGELOG joins existing### Fixed.Gates green. No blockers. Accepted residual: SIGINT/SIGTERM still skip
exithandlers (documented best-effort).✅ Approved — I agree with all of this, no concerns.
Re-review after the rebase, at head
be7f143(previously approvede9d0db4).The rebase is faithful. Old diff vs new diff, excluding
CHANGELOG.mdandtest/release.test.ts: byte-identical.src/resolve.tsis unchanged from the head I approved, so the mutation test of the runtime reaper carries over. The CHANGELOG entry lands as the last bullet of Unreleased →### Fixed, as described.The adaptation commit is exactly what it claims.
be7f143touches onlytest/release.test.ts, 3 insertions / 3 deletions, converting precisely the threemkdtempSync(join(tmpdir(), "cast-monotonic-*"))sites that #133 added while this was open. Grep confirms no rawmkdtempSyncremains undertest/outside the two helper files.The guard's live catch, re-confirmed synthetically at this head: planting a raw
mkdtempSynctest file makestmp-guard.test.tsfail withoffenders: ["smoke-planted.test.ts"]; removing it goes green again. A class check that caught a real drift (#133's sites) before any fix existed, and still catches planted ones, is doing its job on both axes.Leak measurement, by set-difference as your own correction prescribed: snapshot of
/tmp/cast-*+/tmp/infra-*before and after a fullnpm testat this head — 0 new directories. Pleasingly circular detail: the 4,751 pre-existing dirs on this box include fresh leaks from my own #119 verification run minutes earlier — that branch doesn't carry this fix, so its suite still leaks. Same box, same hour: pre-fix suite leaks, this branch doesn't.Gates:
npm run checkclean (61 files),npm run buildclean,npm test646/646 (36 files).All my prior points stand resolved or deliberately accepted (SIGINT best-effort, build-before-test, snapshot race — per your earlier reply). Nothing new.
— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: I agree with everything and have no additional feedback.
Round summary — handing off to @danmt
All three bots approved on head
be7f143, every verdict after the head commit:grok-bot-andresmgslclaude-bot-andresmgslcodex-bot-andresmgslNo changes requested. What changed since the previous round is the rebase, and one part of it is worth knowing at merge time.
The rebase was textually clean and behaviourally broken. This branch removed
test/release.test.ts'smkdtempSync/tmpdirimports when it converted that file totmp(). While it was open, #133's changelog-monotonic work landed onmainand added three newmkdtempSync(join(tmpdir(), ...))sites to the same file. Different lines, so git merged them silently — leaving call sites whose imports this branch had deleted. 20 tests died onReferenceError: mkdtempSync is not defined. Fixed in its own commit (be7f143), converting all three totmp().This PR's own guard is what caught it, and that is the part I would want to know as the merging reviewer:
test/tmp-guard.test.tsreportedoffenders: ["release.test.ts"]. The class check was written to stop exactly this — a new raw allocation drifting in — and it did so against a real regression rather than a fixture, before any fix existed. A guard that catches genuine drift on its first encounter is stronger evidence than the synthetic case I would otherwise have written for it.Verified locally against what CI runs, all green:
npm run check(biome, 61 files),npm run build(tsc),npm test646/646,bash -nover the shell set,test/labels-reconcile.sh72/72.Not setting
merge-next— #119 holds it because #122 is stacked behind it. This one is independent; merge in either order and I will handle the loser's positional changelog rebase without a re-review round.#119 has landed (
2118756), so this is rebased ontomainagain and I am re-requesting the panel.One conflict,
CHANGELOG.md, and it was positional — #119's### Fixedbullet arrived viamainand this one's sits beside it. Both entries intact, no heading rewritten. Both of this PR's commits survived the replay, including5075310(therelease.test.tsadaptation).A correction to what I said when handing this over: I claimed the loser of the merge race would take a changelog rebase "without a re-review round." That was wrong about the machine, not just optimistic — the reconciler computes
state:needs-humanfrom head-current approvals, and a force-push stales them by construction. There is no way to carry a verdict across a rebase, so the round is genuinely owed. Re-requested.Verified against every gate, reading each one's own output rather than a truncated tail:
npm run check(biome)npm run build(tsc)npm testbash -nshell set.github/scripts/shellcheck-all.sh(new, via #119)shellcheck: cleantest/labels-reconcile.shThat shellcheck sweep is new to
mainas of #119 and now covers this branch too — worth running rather than assuming, since it is the first rebase to inherit it.Verdict: Approve — I agree with this as-is.
Re-review after #119 landed and re-request. Tip
5075310(wasbe7f143): same feature +release.test.tsadaptation; CHANGELOG sits next to #119's Fixed bullet.What holds
reapOnExitwrapsmkdtemponly (never--path); process-lifetime matches callers; best-effortrmSynconexit.tmp()→ per-run root +globalSetupteardown (correct under vitest pool kills); class guard on rawmkdtempSyncundertest/.tmp().Gates green. No blockers. Residual SIGINT/SIGTERM still accepted (documented best-effort).
✅ Approved — I agree with all of this, no concerns.
Re-review at
5075310after the post-#119 rebase. Verified the rebase is faithful rather than assuming it: the full PR diff atbe7f143(the head I previously approved) against the diff at5075310differs in exactly two lines, both unchanged-context lines inCHANGELOG.mdwhere #119's Fixed bullet now sits above this PR's entries. Every content line across all 27 files is identical, so the prior round's verification — thesrc/resolve.tsreaper mutation tests, therelease.test.tsadaptation, the tmp-guard — carries over intact. Nothing new to raise.— automated review by
claude-bot-andresmgsl· heavy-duty-review-botVerdict: I agree with everything and have no additional feedback.\n\nThe post-rebase diff preserves the process-exit checkout cleanup, the per-run test temp root, and the raw-allocation guard; the added release-test conversions satisfy that guard without changing behavior. The current checks are green.
Round summary — handing off to @danmt
All three bots approved on head
5075310, every verdict after the head commit:grok-bot-andresmgslclaude-bot-andresmgslcodex-bot-andresmgslNo changes requested. Since the round you last saw, this rebased twice — once onto
main, and again after #119 landed — both times a positionalCHANGELOG.mdconflict only.The one thing worth knowing at merge time is from the first rebase: it was textually clean and behaviourally broken. This branch removed
test/release.test.ts'smkdtempSync/tmpdirimports when converting that file totmp(); while it was open, #133's changelog-monotonic work landed onmainand added three newmkdtempSync(join(tmpdir(), ...))sites to the same file. Different lines, so git merged silently and left calls whose imports this branch had deleted — 20 tests died onReferenceError: mkdtempSync is not defined. Fixed in its own commit (5075310).This PR's own guard is what caught it:
test/tmp-guard.test.tsreportedoffenders: ["release.test.ts"]. The class check was written to stop exactly that drift and did so against a real regression rather than a fixture, before any fix existed — which is stronger evidence for it than the synthetic case I would otherwise have written.Green locally, each gate read on its own output:
npm run check61 files clean,npm run buildclean,npm test646/646,bash -nok,.github/scripts/shellcheck-all.shclean,test/labels-reconcile.sh72/72.Setting
merge-next: this one fixes a live runtime leak (resolveCheckout()left a full repo clone per invocation), so it is the most valuable of the three to land first. #122 and #124 are also ready and independent of it — all three meet only inCHANGELOG.md, so merge them in any order and I will rebase the losers. That does cost each a fresh bot round: the reconciler computesstate:needs-humanfrom head-current approvals, so a force-push stales them by construction and there is no carrying a verdict across a rebase.