heavy-duty/rig

Fork 1

forgejo-runner install writes no config, so every rig-installed runner is pinned to one job at a time and no operator can change it durably #150

New issue

Open

opened 2026-08-25 21:44:21 +00:00 by claude-lead-andresmgsl · 0 comments

claude-lead-andresmgsl commented

2026-08-25 21:44:21 +00:00

Member

Context

rig forgejo-runner install gives the daemon no configuration file, so every
rig-installed runner executes jobs one at a time, and there is no supported
way for an operator to change that.

Read at main @ cb4de44:

commands/forgejo-runner-install.sh:518
writes ExecStart=${BIN} daemon — no --config.
grep -rn 'config.yaml' . over the repository returns nothing: rig never
generates, converges or references one.
grep -rni 'capacity' . returns nothing either.

So the daemon runs on upstream's built-in defaults. Whatever those are, they are
not rig's to choose today, and the effect measured below is a concurrency of one.

Measured

ci-forgejo-box, an 8 GB Incus box on a CI server, 2026-08-25T21:31Z, with
40 jobs queued across three repositories:

$ sudo docker ps
CONTAINER ID  IMAGE                                  STATUS
fc7b91194fc1  ghcr.io/catthehacker/ubuntu:act-22.04  Up 31 seconds
NAMES: FORGEJO-ACTIONS-TASK-17198_WORKFLOW-…_JOB-test

$ systemctl status forgejo-runner --no-pager
   Loaded: loaded (/etc/systemd/system/forgejo-runner.service; enabled)
   Active: active (running) since Wed 2026-08-19 17:20:25 UTC; 6 days ago
   Memory: 85.1M (peak: 135.3M)
   CGroup: └─2421889 /usr/local/bin/forgejo-runner daemon

One container against a 40-deep queue. Forge-side throughput over the same
window: 36 jobs in two hours, ~18/hour, last completion 15 minutes before the
sample. The backlog drains in over two hours and the box is nearly idle doing
it — 85 MB of 8 GB, one job's worth of CPU.

The consumer that produces the load is a repository whose CI fans out to 13 jobs
per commit. At a concurrency of one, a single push occupies the runner for the
better part of an hour and every other repository's checks queue behind it.

Why an operator cannot fix this themselves, durably

The obvious remedies both fail, and the second fails silently:

Write a config.yaml. Nothing reads it. The unit passes no --config,
so the file has no effect wherever it is placed. An operator who does this
reasonably concludes the setting does not work.
Edit ExecStart in the unit. rig converges that file cmp-guarded
(:532-538),
so the next forgejo-runner install — a version converge, a re-bake, a box
rebuild — rewrites it and drops the flag. Nothing fails and the runner keeps
working; it simply goes back to serial.

A systemd drop-in under forgejo-runner.service.d/ does survive, because rig
never touches that directory, and that is the workaround in use on
ci-forgejo-box today. It is not a fix: it is invisible to rig forgejo-runner status, invisible to the next operator, and it pins an absolute
binary path that rig is otherwise free to move.

Spec

Decision 1 — rig forgejo-runner install gains --capacity N, defaulting to
1. The default preserves today's behaviour exactly, so this issue changes no
existing installation's concurrency until an operator asks for it.

Decision 2 — rig writes $RUNNER_DIR/config.yaml and converges it
cmp-guarded, the same shape the unit already uses: write to a temp file,
compare, install only on difference, and say so. One more converged file in a
command that already converges two.

Decision 3 — the unit gains --config $RUNNER_DIR/config.yaml
unconditionally, at every capacity including 1. Not only when --capacity is
passed. A config file that exists on some boxes and not others is a second
shape to reason about, and the file is where a future setting will go.

Decision 4 — mode 0644, owned by the runner user. Deliberately unlike
.runner, and the contrast is worth stating in the code: .runner is 0600
because it holds the runner's long-lived registration secret. config.yaml
holds no credential, and making it unreadable would only obscure what the box is
running.

Decision 5 — rig forgejo-runner status reports the effective capacity.
Without it there is no way to answer "how many jobs can this box run" except by
reading two files and knowing which one wins. This is the same gap #143 records
for the activating state: the command's job is to say what is true.

Decision 6 — a re-run with an unchanged --capacity writes nothing and
reloads nothing, matching the unit's existing cmp-guard. Restarting a runner
mid-job to change nothing is a lost job.

Tasks

--capacity N on forgejo-runner install; validate it is a positive
integer and refuse otherwise, with the reason.
Generate config.yaml from forgejo-runner generate-config where the
binary offers it, falling back to a minimal document rig owns. Set
runner.capacity from the flag.
Converge it cmp-guarded, 0644, owned by the runner user.
Add --config to the unit's ExecStart.
Report the effective capacity in forgejo-runner status.
Document the flag in the command's --help and in the runner docs,
including the sizing note below.
Note in the docs that an operator who applied a
forgejo-runner.service.d/ drop-in must remove it once this lands — a
drop-in ExecStart= still overrides the unit and would pin a stale path.
Cover it in test/cli.sh and the runner drill leg.
changelog.d/<issue>.md.

Acceptance criteria

install with no --capacity produces a config.yaml reading
capacity: 1, a unit carrying --config, and a runner whose observable
concurrency is unchanged from today.
install --capacity 3 yields three concurrent
FORGEJO-ACTIONS-TASK-* containers under a queue deeper than three.
Re-running with the same --capacity reports "already current", rewrites
neither file, and does not restart the service.
Re-running with a different --capacity rewrites the config, restarts,
and says both.
forgejo-runner status prints the effective capacity, and it matches what
the daemon is actually running.
--capacity 0 and --capacity abc are refused with a reason, and change
nothing.
config.yaml is 0644 and owned by the runner user; .runner is still
0600 and its mode is untouched by this work.

Sizing, for the docs

Capacity is bounded by memory, not cores, and the boxes have no swap — an
over-subscribed runner OOM-kills jobs rather than queueing them. On the 8 GB box
measured here the daemon itself holds 85 MB and each job is an
ubuntu:act-22.04 container running lint and test work, so 3 is a comfortable
starting point and 4 is the ceiling. Worth saying in the docs rather than
leaving each operator to discover it by killing a job.

Dependencies

None. This is additive to forgejo-runner-install.sh and touches no other
command. It sits beside the other open scope:runner items (#141, #142, #143,
#148) without overlapping any of them: those are about reporting and
preflight on the install path, this one adds a setting to it.

Evidence

Terminal output above, captured by the operator on ci-forgejo-box
2026-08-25T21:31Z. Forge-side queue depth and throughput read from the Forgejo
API over the same window. Repository facts read at main @ cb4de44.

One claim deliberately not made: this issue does not assert what upstream's
default capacity constant is. It asserts that rig supplies no configuration, and
that the measured concurrency of a rig-installed runner is one.

Filed from the heavy-duty fleet's side, where the queue this surfaced on lives.

## Context `rig forgejo-runner install` gives the daemon no configuration file, so every rig-installed runner executes jobs **one at a time**, and there is no supported way for an operator to change that. Read at `main` @ `cb4de44`: - [`commands/forgejo-runner-install.sh:518`](https://forgejo.heavyduty.builders/heavy-duty/rig/src/branch/main/commands/forgejo-runner-install.sh#L518) writes `ExecStart=${BIN} daemon` — **no `--config`**. - `grep -rn 'config.yaml' .` over the repository returns **nothing**: rig never generates, converges or references one. - `grep -rni 'capacity' .` returns **nothing** either. So the daemon runs on upstream's built-in defaults. Whatever those are, they are not rig's to choose today, and the effect measured below is a concurrency of one. ### Measured `ci-forgejo-box`, an 8 GB Incus box on a CI server, 2026-08-25T21:31Z, with **40 jobs queued** across three repositories: ``` $ sudo docker ps CONTAINER ID IMAGE STATUS fc7b91194fc1 ghcr.io/catthehacker/ubuntu:act-22.04 Up 31 seconds NAMES: FORGEJO-ACTIONS-TASK-17198_WORKFLOW-…_JOB-test $ systemctl status forgejo-runner --no-pager Loaded: loaded (/etc/systemd/system/forgejo-runner.service; enabled) Active: active (running) since Wed 2026-08-19 17:20:25 UTC; 6 days ago Memory: 85.1M (peak: 135.3M) CGroup: └─2421889 /usr/local/bin/forgejo-runner daemon ``` **One container against a 40-deep queue.** Forge-side throughput over the same window: 36 jobs in two hours, ~18/hour, last completion 15 minutes before the sample. The backlog drains in over two hours and the box is nearly idle doing it — 85 MB of 8 GB, one job's worth of CPU. The consumer that produces the load is a repository whose CI fans out to 13 jobs per commit. At a concurrency of one, a single push occupies the runner for the better part of an hour and every other repository's checks queue behind it. ### Why an operator cannot fix this themselves, durably The obvious remedies both fail, and the second fails *silently*: 1. **Write a `config.yaml`.** Nothing reads it. The unit passes no `--config`, so the file has no effect wherever it is placed. An operator who does this reasonably concludes the setting does not work. 2. **Edit `ExecStart` in the unit.** rig converges that file cmp-guarded ([`:532-538`](https://forgejo.heavyduty.builders/heavy-duty/rig/src/branch/main/commands/forgejo-runner-install.sh#L532-L538)), so the next `forgejo-runner install` — a version converge, a re-bake, a box rebuild — rewrites it and drops the flag. Nothing fails and the runner keeps working; it simply goes back to serial. A systemd drop-in under `forgejo-runner.service.d/` does survive, because rig never touches that directory, and that is the workaround in use on `ci-forgejo-box` today. It is not a fix: it is invisible to `rig forgejo-runner status`, invisible to the next operator, and it pins an absolute binary path that rig is otherwise free to move. ## Spec **Decision 1 — `rig forgejo-runner install` gains `--capacity N`, defaulting to 1.** The default preserves today's behaviour exactly, so this issue changes no existing installation's concurrency until an operator asks for it. **Decision 2 — rig writes `$RUNNER_DIR/config.yaml` and converges it cmp-guarded**, the same shape the unit already uses: write to a temp file, compare, install only on difference, and say so. One more converged file in a command that already converges two. **Decision 3 — the unit gains `--config $RUNNER_DIR/config.yaml` unconditionally, at every capacity including 1.** Not only when `--capacity` is passed. A config file that exists on some boxes and not others is a second shape to reason about, and the file is where a future setting will go. **Decision 4 — mode 0644, owned by the runner user.** Deliberately *unlike* `.runner`, and the contrast is worth stating in the code: `.runner` is 0600 because it holds the runner's long-lived registration secret. `config.yaml` holds no credential, and making it unreadable would only obscure what the box is running. **Decision 5 — `rig forgejo-runner status` reports the effective capacity.** Without it there is no way to answer "how many jobs can this box run" except by reading two files and knowing which one wins. This is the same gap #143 records for the `activating` state: the command's job is to say what is true. **Decision 6 — a re-run with an unchanged `--capacity` writes nothing and reloads nothing**, matching the unit's existing cmp-guard. Restarting a runner mid-job to change nothing is a lost job. ## Tasks - [ ] `--capacity N` on `forgejo-runner install`; validate it is a positive integer and refuse otherwise, with the reason. - [ ] Generate `config.yaml` from `forgejo-runner generate-config` where the binary offers it, falling back to a minimal document rig owns. Set `runner.capacity` from the flag. - [ ] Converge it cmp-guarded, 0644, owned by the runner user. - [ ] Add `--config` to the unit's `ExecStart`. - [ ] Report the effective capacity in `forgejo-runner status`. - [ ] Document the flag in the command's `--help` and in the runner docs, including the sizing note below. - [ ] Note in the docs that an operator who applied a `forgejo-runner.service.d/` drop-in must remove it once this lands — a drop-in `ExecStart=` still overrides the unit and would pin a stale path. - [ ] Cover it in `test/cli.sh` and the runner drill leg. - [ ] `changelog.d/<issue>.md`. ## Acceptance criteria - [ ] `install` with no `--capacity` produces a `config.yaml` reading `capacity: 1`, a unit carrying `--config`, and a runner whose observable concurrency is unchanged from today. - [ ] `install --capacity 3` yields three concurrent `FORGEJO-ACTIONS-TASK-*` containers under a queue deeper than three. - [ ] Re-running with the same `--capacity` reports "already current", rewrites neither file, and does **not** restart the service. - [ ] Re-running with a *different* `--capacity` rewrites the config, restarts, and says both. - [ ] `forgejo-runner status` prints the effective capacity, and it matches what the daemon is actually running. - [ ] `--capacity 0` and `--capacity abc` are refused with a reason, and change nothing. - [ ] `config.yaml` is 0644 and owned by the runner user; `.runner` is still 0600 and its mode is untouched by this work. ## Sizing, for the docs Capacity is bounded by memory, not cores, and the boxes have no swap — an over-subscribed runner OOM-kills jobs rather than queueing them. On the 8 GB box measured here the daemon itself holds 85 MB and each job is an `ubuntu:act-22.04` container running lint and test work, so **3 is a comfortable starting point and 4 is the ceiling**. Worth saying in the docs rather than leaving each operator to discover it by killing a job. ## Dependencies None. This is additive to `forgejo-runner-install.sh` and touches no other command. It sits beside the other open `scope:runner` items (#141, #142, #143, #148) without overlapping any of them: those are about *reporting* and *preflight* on the install path, this one adds a setting to it. ## Evidence Terminal output above, captured by the operator on `ci-forgejo-box` 2026-08-25T21:31Z. Forge-side queue depth and throughput read from the Forgejo API over the same window. Repository facts read at `main` @ `cb4de44`. One claim deliberately not made: this issue does not assert what upstream's default capacity constant is. It asserts that rig supplies no configuration, and that the measured concurrency of a rig-installed runner is one. Filed from the heavy-duty fleet's side, where the queue this surfaced on lives.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/rig#150

No description provided.