heavy-duty/rig

Fork 1

rig-installed Forgejo runners have their cache server disabled — ProtectHome=read-only vs $HOME/.cache #135

New issue

Closed

opened 2026-07-31 16:46:58 +00:00 by claude-bot-andresmgsl · 1 comment

claude-bot-andresmgsl commented

2026-07-31 16:46:58 +00:00

Member

Context

Every runner rig forgejo-runner install creates has its cache server disabled, silently. Measured on a live install on 2026-07-31:

level=error msg="Could not start the cache server, cache will be disabled:
                 mkdir /home/forgejo-runner/.cache: read-only file system"

The cause is rig's own unit hardening, commands/forgejo-runner-install.sh:

ProtectHome=read-only
ReadWritePaths=${RUNNER_DIR}          # /home/forgejo-runner/forgejo-runner

ProtectHome=read-only makes the whole home read-only, and only RUNNER_DIR is punched back through. forgejo-runner's cache server wants $HOME/.cache, which is not under RUNNER_DIR, so it cannot create it and disables itself.

Consequence: actions/cache does not work on any rig-installed Forgejo runner. Every workflow that caches silently re-downloads on every run. The only evidence is one level=error line in the journal — and rig forgejo-runner status reports the runner as healthy, so nothing surfaces it. (That is #133's shape, one layer down: the service is up, the work it is supposed to do is not happening.)

Spec

Create the cache directory at install time and punch it through the hardening. Both halves are required — measured, not assumed:

ReadWritePaths alone is not enough. With the path listed but absent, systemd refuses to start the unit at all: Failed to set up mount namespacing: /home/forgejo-runner/.cache: No such file or directory, three restart attempts, service down. That is strictly worse than the current bug.
With the directory created first (0755, owned by the runner user) and listed in ReadWritePaths, the daemon starts clean, the cache-server error is gone, and it binds its listeners:

LISTEN 0 4096 *:34885 users:(("forgejo-runner",pid=…))
LISTEN 0 4096 *:39625 users:(("forgejo-runner",pid=…))

So: install -d -m 0755 -o "$RUNNER_USER" -g "$RUNNER_GROUP" "$USER_HOME/.cache" beside the existing RUNNER_DIR creation, and ReadWritePaths=${RUNNER_DIR} ${USER_HOME}/.cache in the unit.

Keep ProtectHome=read-only. The point of the hardening is that the runner supervises job containers on this box's docker socket; widening it to ProtectHome=tmpfs or dropping it would give back more than the cache needs.

Tasks

Create $USER_HOME/.cache at install, owned by the runner user, beside the existing RUNNER_DIR install
Add it to the unit's ReadWritePaths
Converge on re-run like the rest of the installer (an existing directory is a no-op, and the unit is rewritten only when it differs)
test/cli.sh: the unit carries both paths; the install creates the directory
changelog.d/<n>.md

Acceptance criteria

A fresh install starts with no Could not start the cache server line
The unit lists both RUNNER_DIR and the cache directory in ReadWritePaths
ProtectHome=read-only is unchanged
Re-running install is a no-op on an already-correct box
The directory is owned by the runner user, not root — a root-owned .cache under a User= unit fails the same way
bash test/cli.sh passes; CI's shellcheck -x step is clean

Test plan

test/cli.sh asserts the unit text and the install -d call, in the file's existing grep-the-shipped-bytes idiom.
Must fail: list the path in ReadWritePaths without creating it → the unit does not start (Failed to set up mount namespacing). This is the trap; a test that only greps the unit would call the broken shape correct, so the ordering matters and belongs in a comment.
Live: install on a systemd box, assert no cache-server error and that the listeners appear.

Dependencies

Surfaced while probing for #129. Independent of it and of #133 — different file, different surface.
Sibling in spirit to #133: both are "the service is up, the work is not happening".

@andres — filed under your standing instruction to file findings. Verified live on this box before writing, including the failure mode of the obvious one-line version.

## Context Every runner `rig forgejo-runner install` creates has its cache server disabled, silently. Measured on a live install on 2026-07-31: ``` level=error msg="Could not start the cache server, cache will be disabled: mkdir /home/forgejo-runner/.cache: read-only file system" ``` The cause is rig's own unit hardening, `commands/forgejo-runner-install.sh`: ``` ProtectHome=read-only ReadWritePaths=${RUNNER_DIR} # /home/forgejo-runner/forgejo-runner ``` `ProtectHome=read-only` makes the whole home read-only, and only `RUNNER_DIR` is punched back through. forgejo-runner's cache server wants `$HOME/.cache`, which is not under `RUNNER_DIR`, so it cannot create it and disables itself. **Consequence:** `actions/cache` does not work on any rig-installed Forgejo runner. Every workflow that caches silently re-downloads on every run. The only evidence is one `level=error` line in the journal — and `rig forgejo-runner status` reports the runner as healthy, so nothing surfaces it. (That is #133's shape, one layer down: the service is up, the work it is supposed to do is not happening.) ## Spec Create the cache directory at install time and punch it through the hardening. Both halves are required — measured, not assumed: - **`ReadWritePaths` alone is not enough.** With the path listed but absent, systemd refuses to start the unit at all: `Failed to set up mount namespacing: /home/forgejo-runner/.cache: No such file or directory`, three restart attempts, service down. That is strictly worse than the current bug. - **With the directory created first** (`0755`, owned by the runner user) **and** listed in `ReadWritePaths`, the daemon starts clean, the cache-server error is gone, and it binds its listeners: ``` LISTEN 0 4096 *:34885 users:(("forgejo-runner",pid=…)) LISTEN 0 4096 *:39625 users:(("forgejo-runner",pid=…)) ``` So: `install -d -m 0755 -o "$RUNNER_USER" -g "$RUNNER_GROUP" "$USER_HOME/.cache"` beside the existing `RUNNER_DIR` creation, and `ReadWritePaths=${RUNNER_DIR} ${USER_HOME}/.cache` in the unit. Keep `ProtectHome=read-only`. The point of the hardening is that the runner supervises job containers on this box's docker socket; widening it to `ProtectHome=tmpfs` or dropping it would give back more than the cache needs. ## Tasks - [ ] Create `$USER_HOME/.cache` at install, owned by the runner user, beside the existing `RUNNER_DIR` install - [ ] Add it to the unit's `ReadWritePaths` - [ ] Converge on re-run like the rest of the installer (an existing directory is a no-op, and the unit is rewritten only when it differs) - [ ] `test/cli.sh`: the unit carries both paths; the install creates the directory - [ ] `changelog.d/<n>.md` ## Acceptance criteria - [ ] A fresh install starts with **no** `Could not start the cache server` line - [ ] The unit lists both `RUNNER_DIR` and the cache directory in `ReadWritePaths` - [ ] `ProtectHome=read-only` is unchanged - [ ] Re-running install is a no-op on an already-correct box - [ ] The directory is owned by the runner user, not root — a root-owned `.cache` under a `User=` unit fails the same way - [ ] `bash test/cli.sh` passes; CI's `shellcheck -x` step is clean ## Test plan - `test/cli.sh` asserts the unit text and the `install -d` call, in the file's existing grep-the-shipped-bytes idiom. - **Must fail:** list the path in `ReadWritePaths` without creating it → the unit does not start (`Failed to set up mount namespacing`). This is the trap; a test that only greps the unit would call the broken shape correct, so the ordering matters and belongs in a comment. - Live: install on a systemd box, assert no cache-server error and that the listeners appear. ## Dependencies - Surfaced while probing for #129. Independent of it and of #133 — different file, different surface. - Sibling in spirit to #133: both are "the service is up, the work is not happening". --- @andres — filed under your standing instruction to file findings. Verified live on this box before writing, including the failure mode of the obvious one-line version.

claude-bot-andresmgsl referenced this issue

2026-07-31 16:47:31 +00:00

test/cli.sh is not hermetic — 13 checks fail on any box that has a Forgejo runner installed #136

claude-bot-andresmgsl added the

bug

scope:runner

claimed

labels 2026-07-31 16:47:31 +00:00

claude-bot-andresmgsl self-assigned this 2026-07-31 16:47:32 +00:00

claude-bot-andresmgsl referenced this issue from a commit

2026-07-31 16:57:49 +00:00

fix(forgejo-runner): the cache server can start

claude-bot-andresmgsl referenced this issue from a commit

2026-07-31 16:59:10 +00:00

fix(forgejo-runner): the cache server can start

andres closed this issue

2026-07-31 20:35:34 +00:00

claude-bot-andresmgsl commented

2026-07-31 20:38:58 +00:00

Author

Member

Post-merge verification of `ce43c02` — clean

@andres — verified the merge rather than assuming it, since a merge is a new artifact:

git diff a45d848 ce43c02 is empty — the merged tree is byte-identical to the head the panel approved; nothing slipped in.
The fix is live on main: ReadWritePaths=${RUNNER_DIR} ${USER_HOME}/.cache at :472, and the install -d for the cache directory at :407.
CI's exact shellcheck -x step on merged main: 34 files, exit 0.
All three branches still open (!130, !134, !137) merge cleanly onto the new main — checked with git merge-tree, and Forgejo agrees.

test/cli.sh reads 776/13 on merged main, and all 13 are the environmental failures of #136 — this box has a real Forgejo runner installed, so ci-box's installer correctly early-exits and bootstrap --undo correctly refuses. Unmodified main scored the same 13 before this merge. !137 is the fix.

Every rig-installed Forgejo runner from this commit forward can start its cache server. Existing runners need one rig forgejo-runner install --instance <url> to converge — the unit is rewritten only when it differs, and the directory creation is idempotent.

## Post-merge verification of `ce43c02` — clean @andres — verified the merge rather than assuming it, since a merge is a new artifact: - `git diff a45d848 ce43c02` is **empty** — the merged tree is byte-identical to the head the panel approved; nothing slipped in. - The fix is live on `main`: `ReadWritePaths=${RUNNER_DIR} ${USER_HOME}/.cache` at `:472`, and the `install -d` for the cache directory at `:407`. - CI's exact `shellcheck -x` step on merged `main`: 34 files, **exit 0**. - All three branches still open (!130, !134, !137) merge cleanly onto the new `main` — checked with `git merge-tree`, and Forgejo agrees. `test/cli.sh` reads 776/13 on merged `main`, and all 13 are the environmental failures of #136 — this box has a real Forgejo runner installed, so `ci-box`'s installer correctly early-exits and `bootstrap --undo` correctly refuses. Unmodified `main` scored the same 13 before this merge. !137 is the fix. Every rig-installed Forgejo runner from this commit forward can start its cache server. Existing runners need one `rig forgejo-runner install --instance <url>` to converge — the unit is rewritten only when it differs, and the directory creation is idempotent.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/rig#135

No description provided.