The dating expression took the newest stamp each run carries. That reads `completedAt` for a finished run and `startedAt` for a live one, so the comparison comes down to "when this one ended" against "when that one began" — which is not an ordering on runs at all. A run cancelled by the concurrency group does not stop the instant its replacement starts: the runner has to receive the signal and wind down. So `predecessor.completedAt > successor.startedAt` is the ordinary case, not a corner. On the box#137 tip that motivated the supersede rule the window was 13s wide — the superseding run started 15:19:38, the run it cancelled did not finish until 15:19:51 — and for that whole window the dying predecessor out-dated its own live replacement, so `last` discarded the replacement and judged the corpse. Both round-3 failure modes came back inside that window, narrowed rather than closed: a draining CANCELLED predecessor reported FAILURE and sent the agent to fix nothing, and a draining SUCCESS predecessor reported SUCCESS — mergeable, all bots approve, state:needs-human — over a tree whose merge button branch protection had already disabled. #136 again, one field over. Dated by `first` of the preference-ordered stamps rather than `max` of them: start time if the run recorded one, falling back only if it did not. The sentinel filtering is unchanged, and finished runs still date by completion when that is all they carry, so the supersede rule keeps the case it exists for. Found independently by claude-bot-andresmgsl and codex-bot-andresmgsl. No existing fixture could express it — `run_()` carries no startedAt, so every supersede fixture spaced the predecessor's completion safely before the successor's start, the same blind spot as round 3 one field over. New `drained_()` helper pins both directions; fixtures 48 -> 51 (with the reverse-direction in-flight fixture ported from cast#128). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
360 lines
18 KiB
Bash
360 lines
18 KiB
Bash
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
# labels-reconcile.sh — the automation LABELS.md promises: state labels are
|
|
# written by machinery, never by hand. Every run derives each open PR's
|
|
# state:* from GitHub's own facts (draft flag, requested reviewers, submitted
|
|
# reviews) and converges the labels to it, so a killed run or a hand-moved
|
|
# label heals on the next pass. Stale is judged from real activity — commits,
|
|
# comments, reviews — never from label churn, or the sweep would un-stale its
|
|
# own mark every tick.
|
|
#
|
|
# The verdict contract (CONTRIBUTING.md): reviews end in approve or
|
|
# request-changes. Some live bots are comment-only and post agreement as a
|
|
# COMMENTED review — a non-verdict this machine refuses to guess about (body
|
|
# parsing is a heuristic, and a wrong guess promotes an unapproved PR). The
|
|
# judgment call belongs to the PR AUTHOR, who reads the round and escalates
|
|
# by requesting the human's review — an explicit request is a fact, and it is
|
|
# the one this machine trusts (see decide_state's top precedence). The
|
|
# machine auto-requests the human only in the no-judgment-needed case: three
|
|
# formal head-current approvals. Any approval that counts must be bound to
|
|
# the CURRENT head SHA: GitHub keeps approvals alive across pushes, and a
|
|
# stale approval must never promote unreviewed code to the human.
|
|
#
|
|
# DRY_RUN=1 narrates every mutation instead of performing it (how this script
|
|
# is rehearsed against the live repo). A workflow_dispatch run also bootstraps
|
|
# the taxonomy (label create --force) — that heal is dispatch-only; the cron
|
|
# sweep tolerates a missing label rather than recreating it.
|
|
#
|
|
# The state machine below is pure (globals in, state out) and covered by
|
|
# fixture tests in test/labels-reconcile.sh.
|
|
|
|
HUMAN="${HUMAN_REVIEWER:-danmt}"
|
|
BOTS=(claude-bot-andresmgsl codex-bot-andresmgsl grok-bot-andresmgsl)
|
|
STATES=(state:building state:needs-rebase state:bots-reviewing state:addressing state:needs-human)
|
|
STALE_AFTER=$((48 * 3600))
|
|
|
|
log() { printf 'labels: %s\n' "$*"; }
|
|
|
|
run() { # every mutation goes through here — DRY_RUN=1 logs instead of doing
|
|
if [ -n "${DRY_RUN:-}" ]; then log "DRY_RUN: $*"; else "$@"; fi
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# The state machine. Pure functions over four globals, set per PR:
|
|
# DRAFT true|false
|
|
# HEAD_SHA the PR's current head commit
|
|
# REQUESTED newline-separated logins with a review currently requested
|
|
# REVIEWS_JSON JSON array of submitted (non-PENDING) reviews
|
|
# MERGEABLE MERGEABLE | CONFLICTING | UNKNOWN (GitHub's own verdict)
|
|
# CHECKS SUCCESS | FAILURE | PENDING | NONE (the check rollup)
|
|
# ---------------------------------------------------------------------------
|
|
|
|
requested() { grep -qxF "$1" <<<"$REQUESTED"; }
|
|
|
|
checks_state() { # rollup JSON on stdin → SUCCESS | FAILURE | PENDING | NONE
|
|
# The rollup mixes two node types with two different closed enums: CheckRun
|
|
# carries `conclusion` (CheckConclusionState), StatusContext carries `state`
|
|
# (StatusState). Rather than list the outcomes that block — the version that
|
|
# shipped in this PR's first round listed four, and ERROR, CANCELLED and
|
|
# STALE fell through its `else` into SUCCESS — this lists the outcomes that
|
|
# DON'T, and treats everything else as blocking.
|
|
#
|
|
# That direction is the point. An outcome we do not recognise is one we
|
|
# cannot certify as mergeable, and certifying the unrecognised as green is
|
|
# the exact shape of #136. The cost of being wrong is symmetric in form and
|
|
# not in consequence: a false FAILURE parks the PR on the agent, who looks;
|
|
# a false SUCCESS invites a human to merge a tree that will not merge.
|
|
jq -r '
|
|
# NEUTRAL and SKIPPED satisfy branch protection — a skipped required check
|
|
# is not a failed one, and path-filtered jobs skip constantly here.
|
|
["SUCCESS", "NEUTRAL", "SKIPPED"] as $passing
|
|
# "" covers a StatusContext still reported with no state at all.
|
|
| ["", "PENDING", "IN_PROGRESS", "QUEUED", "WAITING", "REQUESTED", "EXPECTED"] as $waiting
|
|
|
|
# A re-run does not evict the run it superseded — the rollup keeps both.
|
|
# This PR proved it: its own tip carried a CANCELLED `scope` (15:19:39)
|
|
# beside the SUCCESS `scope` (15:19:45) that replaced it, same workflow.
|
|
# Once CANCELLED blocks, judging every entry would strand this very PR in
|
|
# needs-rebase forever, so collapse each context to its newest entry first.
|
|
# Key on workflow + name because a bare job name is only unique within its
|
|
# workflow.
|
|
#
|
|
# Dating a run is the subtle part, and getting it wrong restores the bug.
|
|
# A run still in flight has no completion, but `gh` does not omit the
|
|
# field: its Go struct marshals the zero time as "0001-01-01T00:00:00Z",
|
|
# which is a string, so `//` will not fall through it. Ordering on
|
|
# completion therefore sorted the LIVE re-run to the bottom and let `last`
|
|
# pick the very run it superseded — reporting the old SUCCESS while a
|
|
# replacement was still running, which is #136 again.
|
|
#
|
|
# So: date a run by when it BEGAN, discarding both spellings of absent
|
|
# (null, and the zero sentinel), and fall back only if it never recorded a
|
|
# beginning. Deliberately `first` over the preference-ordered list and not
|
|
# `max` of it: max mixes "when it started" with "when it ended", which are
|
|
# different quantities, so it is not an ordering on runs at all. A run
|
|
# cancelled by the concurrency group does not stop the instant its
|
|
# replacement starts — the runner has to receive the signal and wind down —
|
|
# so predecessor.completedAt > successor.startedAt is the ordinary case
|
|
# (13s on the box#137 tip), and under max the dying predecessor
|
|
# out-dated its live replacement for the whole drain window.
|
|
#
|
|
# An entry that carries no usable timestamp at all sorts LAST rather than
|
|
# first — something we cannot date is most likely the thing just created,
|
|
# and treating it as newest keeps an undateable in-flight run from being
|
|
# discarded in favour of a stale success. Every ambiguity here resolves
|
|
# toward "not settled".
|
|
| [ (.statusCheckRollup // [])[]
|
|
| { ctx: [.workflowName // "", .name // .context // ""],
|
|
at: ([.startedAt, .createdAt, .completedAt]
|
|
| map(select(type == "string" and . != ""
|
|
and (startswith("0001-01-01") | not)))
|
|
| first // ""),
|
|
outcome: ((.conclusion // .state // "") | ascii_upcase) } ]
|
|
| group_by(.ctx)
|
|
| map(sort_by([(.at == ""), .at]) | last | .outcome) as $latest
|
|
|
|
| if ($latest | length) == 0 then "NONE"
|
|
elif (($latest - $passing - $waiting) | length) > 0 then "FAILURE"
|
|
elif (($latest - $passing) | length) > 0 then "PENDING"
|
|
else "SUCCESS" end'
|
|
}
|
|
|
|
bot_verdict() { # $1 = login → MISSING | BLOCK | APPROVE | STALE | FEEDBACK
|
|
local review state commit
|
|
review="$(jq -c --arg u "$1" \
|
|
'[.[] | select(.user.login == $u)] | sort_by(.submitted_at) | last // empty' \
|
|
<<<"$REVIEWS_JSON")"
|
|
if [ -z "$review" ]; then echo MISSING; return; fi
|
|
state="$(jq -r '.state' <<<"$review")"
|
|
commit="$(jq -r '.commit_id' <<<"$review")"
|
|
case "$state" in
|
|
CHANGES_REQUESTED)
|
|
# blocks at ANY head — GitHub's own semantic: only a newer review
|
|
# from the same reviewer clears it
|
|
echo BLOCK ;;
|
|
APPROVED)
|
|
if [ "$commit" = "$HEAD_SHA" ]; then echo APPROVE; else echo STALE; fi ;;
|
|
*)
|
|
# COMMENTED and anything else: a non-verdict. The machine does not
|
|
# read bodies — if the comment is really an agreement, the AUTHOR
|
|
# says so by requesting the human's review.
|
|
echo FEEDBACK ;;
|
|
esac
|
|
}
|
|
|
|
human_request_needed() { # 0 when needs-human requires a FRESH human request
|
|
# already requested → the handoff is live; head-current human approval →
|
|
# nothing left to ask. Anything else (never reviewed, an old comment, an
|
|
# approval of an older head) stalls the handoff unless we request —
|
|
# guarding on "has the human ever reviewed" wedged exactly that way.
|
|
if requested "$HUMAN"; then return 1; fi
|
|
if [ "$(bot_verdict "$HUMAN")" = APPROVE ]; then return 1; fi
|
|
return 0
|
|
}
|
|
|
|
decide_state() { # → the one state:* label this PR should carry
|
|
if [ "$DRAFT" = true ]; then echo state:building; return; fi
|
|
|
|
# state:needs-human means ONE thing: a human could merge this right now.
|
|
# Anything that makes that false outranks the request that put it there —
|
|
# otherwise the board invites a merge that cannot or must not happen, and
|
|
# nothing else on the page contradicts it (#136).
|
|
#
|
|
# A conflicted or red branch is the agent's to fix, not the human's to
|
|
# merge. UNKNOWN is deliberately NOT treated as unmergeable: GitHub reports
|
|
# it for a minute after every merge while it recomputes, and flapping every
|
|
# open PR through needs-rebase on each merge would be worse than the bug.
|
|
# An unknown mergeability simply does not trigger this arm; the next sweep
|
|
# sees the settled value.
|
|
# Both default to the "do not know" value: an unset global (older fixture,
|
|
# a failed fetch) must never invent a verdict it did not read.
|
|
case "${MERGEABLE:-UNKNOWN}" in CONFLICTING) echo state:needs-rebase; return ;; esac
|
|
case "${CHECKS:-NONE}" in FAILURE) echo state:needs-rebase; return ;; esac
|
|
|
|
local b verdicts=""
|
|
for b in "${BOTS[@]}"; do
|
|
if requested "$b"; then echo state:bots-reviewing; return; fi
|
|
done
|
|
# Collect the WHOLE round before applying any precedence. Deciding inside
|
|
# the loop let BOTS order pick the winner: a MISSING returned immediately,
|
|
# so a STALE belonging to a later bot was never even read, and the mixed
|
|
# round (one approval staled by a push, another bot yet to review) came out
|
|
# needs-human — the #136 headline shape, with zero reviews bound to the head.
|
|
for b in "${BOTS[@]}"; do
|
|
verdicts="$verdicts $(bot_verdict "$b")"
|
|
done
|
|
case "$verdicts" in
|
|
# STALE = a verdict for an older head. Unlike MISSING, this outranks the
|
|
# human request: every approval it covers was invalidated by a push, so
|
|
# NOBODY has reviewed this tree. Handing that to the human is the #136 case
|
|
# where everything reads green — mergeable, CI passing, "waiting on the
|
|
# human" — over code no reviewer has seen. The agent owes a re-request.
|
|
# Checked before MISSING because "unfinished" must not swallow "and also
|
|
# stale": a round that is both is a push that outran the re-requests, not
|
|
# a maintainer deliberately claiming the PR early.
|
|
*STALE*) echo state:addressing; return ;;
|
|
esac
|
|
case "$verdicts" in
|
|
# No verdict at all from some bot, and nothing staled. An explicit human
|
|
# request still outranks an unfinished round — a maintainer pulling a PR
|
|
# to themselves early is a deliberate act, and the original precedence.
|
|
*MISSING*)
|
|
if requested "$HUMAN"; then echo state:needs-human; return; fi
|
|
echo state:bots-reviewing; return ;;
|
|
esac
|
|
# an explicit human request outranks the remaining bot outcomes — it is the
|
|
# final gate, and a maintainer pulling a PR to themselves early counts too
|
|
if requested "$HUMAN"; then echo state:needs-human; return; fi
|
|
case "$verdicts" in
|
|
# FEEDBACK = a comment with no verdict → the agent owes the round-reply.
|
|
*BLOCK* | *FEEDBACK*) echo state:addressing; return ;;
|
|
esac
|
|
# the bots all approve — but if the human's standing word is
|
|
# changes-requested (and nobody re-requested them yet), the agent owes
|
|
# fixes, not the human a nag
|
|
if [ "$(bot_verdict "$HUMAN")" = BLOCK ]; then
|
|
echo state:addressing
|
|
else
|
|
echo state:needs-human
|
|
fi
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# The sweep: fetch facts, decide, converge. One PR's failure never aborts the
|
|
# others — each PR reconciles in a subshell and a failure just logs.
|
|
# ---------------------------------------------------------------------------
|
|
|
|
bootstrap_labels() { # dispatch-only: ~20 upserts is too chatty for every cron tick
|
|
while IFS='|' read -r name color desc; do
|
|
[ -n "$name" ] || continue
|
|
run gh label create "$name" -R "$REPO" --color "$color" --description "$desc" --force
|
|
done <<'EOF'
|
|
state:building|FBCA04|PR is a draft — the coding agent is still building
|
|
state:bots-reviewing|1D76DB|Waiting on the bot reviewers to finish the round
|
|
state:addressing|D93F0B|All bots reviewed — coding agent owes the single reply + fixes
|
|
state:needs-rebase|B60205|Does not merge — conflicts or failing checks; the agent owes a fix
|
|
state:needs-human|8250DF|Mergeable, green, all bots approve — waiting on the human reviewer
|
|
merge-next|0E8A16|Head of the merge queue — merge this one next (set by hand/agent, cleared here)
|
|
stale|B60205|No activity for 48h — needs a poke (sweep-managed)
|
|
blocked|6A737D|Waiting on another PR or issue to land first
|
|
release|0E8A16|Release flow and version/packaging work
|
|
scope:bootstrap|C5DEF5|bootstrap — hardening a pristine server into a node
|
|
scope:users|C5DEF5|users-* — class model, apply/status, close-root
|
|
scope:runner|C5DEF5|runner-* — GitHub runner lifecycle
|
|
scope:coolify|C5DEF5|coolify-* — Coolify and backup install
|
|
scope:db|C5DEF5|db.sh — dump/restore
|
|
scope:installer|C5DEF5|install.sh — how rig lands on a machine
|
|
EOF
|
|
}
|
|
|
|
has_label() { grep -qxF "$1" <<<"$LABELS"; }
|
|
|
|
reconcile_pr() { # $1 = PR number; relies on the globals set from its fetch
|
|
local n="$1" desired remove s args last_activity age
|
|
|
|
desired="$(decide_state)"
|
|
|
|
# encode the runbook's last step for the no-judgment case: three formal
|
|
# head-current approvals → the human is asked, once. The guard asks whether
|
|
# a FRESH human review is needed for THIS head — never "has the human ever
|
|
# reviewed", which wedged the handoff after any earlier human comment.
|
|
# Idempotent (a live request suppresses it); race-free via the shared
|
|
# concurrency group in labels.yml. With a comment-only bot on the panel
|
|
# this path stays cold and the AUTHOR requests the human.
|
|
if [ "$desired" = state:needs-human ] && human_request_needed; then
|
|
run gh api "repos/$REPO/pulls/$n/requested_reviewers" -f "reviewers[]=$HUMAN" --silent
|
|
log "#$n: requested $HUMAN (round passed)"
|
|
fi
|
|
|
|
# ---- converge the state:* labels ----
|
|
remove=""
|
|
for s in "${STATES[@]}"; do
|
|
if [ "$s" != "$desired" ] && has_label "$s"; then remove="$remove,$s"; fi
|
|
done
|
|
remove="${remove#,}"
|
|
if ! has_label "$desired" || [ -n "$remove" ]; then
|
|
args=(--add-label "$desired")
|
|
[ -n "$remove" ] && args+=(--remove-label "$remove")
|
|
if run gh issue edit "$n" -R "$REPO" "${args[@]}" >/dev/null; then
|
|
log "#$n: state -> $desired${remove:+ (cleared $remove)}"
|
|
else
|
|
# a deleted label must not wedge the sweep — dispatch heals the taxonomy
|
|
log "#$n: WARNING: label edit failed (missing label? run the workflow manually to bootstrap)"
|
|
fi
|
|
fi
|
|
|
|
# ---- merge-next: cleared, never set ----------------------------------
|
|
# Queue order is INTENT — which PR should land first is a judgement about
|
|
# conflicts and dependencies that GitHub knows nothing about, so the
|
|
# reconciler must not guess it (LABELS.md's rule for `blocked`/`release`).
|
|
# What it CAN do is stop the label going stale the way needs-human did:
|
|
# the moment the PR is no longer the thing a human should merge next, the
|
|
# claim is removed. Setting it stays with whoever owns the queue.
|
|
if has_label merge-next && [ "$desired" != state:needs-human ]; then
|
|
run gh issue edit "$n" -R "$REPO" --remove-label merge-next >/dev/null
|
|
log "#$n: cleared merge-next (state is $desired, not mergeable-by-a-human)"
|
|
fi
|
|
|
|
# ---- stale: real activity only, and blocked is legitimately quiet ----
|
|
last_activity="$(
|
|
{
|
|
jq -r '.created_at' <<<"$PR_JSON"
|
|
jq -r '.[].submitted_at' <<<"$REVIEWS_JSON"
|
|
gh api --paginate "repos/$REPO/issues/$n/comments" --jq '.[].created_at'
|
|
gh api --paginate "repos/$REPO/pulls/$n/comments" --jq '.[].created_at'
|
|
gh api --paginate "repos/$REPO/pulls/$n/commits" --jq '.[].commit.committer.date'
|
|
} | sort | tail -n1
|
|
)"
|
|
age=$((NOW - $(date -d "$last_activity" +%s)))
|
|
if has_label blocked || [ "$age" -le "$STALE_AFTER" ]; then
|
|
if has_label stale; then
|
|
run gh issue edit "$n" -R "$REPO" --remove-label stale >/dev/null
|
|
log "#$n: unstale"
|
|
fi
|
|
elif ! has_label stale; then
|
|
run gh issue edit "$n" -R "$REPO" --add-label stale >/dev/null
|
|
log "#$n: stale ($((age / 3600))h quiet)"
|
|
fi
|
|
}
|
|
|
|
main() {
|
|
REPO="${REPO:?set REPO to owner/name}"
|
|
NOW="$(date +%s)"
|
|
|
|
if [ "${GITHUB_EVENT_NAME:-}" = workflow_dispatch ]; then
|
|
log "workflow_dispatch: bootstrapping the taxonomy"
|
|
bootstrap_labels
|
|
fi
|
|
|
|
local n
|
|
for n in $(gh pr list -R "$REPO" --state open --limit 100 --json number --jq '.[].number'); do
|
|
(
|
|
PR_JSON="$(gh api "repos/$REPO/pulls/$n")"
|
|
DRAFT="$(jq -r '.draft' <<<"$PR_JSON")"
|
|
HEAD_SHA="$(jq -r '.head.sha' <<<"$PR_JSON")"
|
|
LABELS="$(jq -r '.labels[].name' <<<"$PR_JSON")"
|
|
REQUESTED="$(jq -r '.requested_reviewers[].login' <<<"$PR_JSON")"
|
|
# PENDING reviews are unsubmitted drafts in someone's browser — not a verdict
|
|
REVIEWS_JSON="$(gh api --paginate "repos/$REPO/pulls/$n/reviews" --jq '.[]' \
|
|
| jq -s '[.[] | select(.state != "PENDING")]')"
|
|
# mergeability + the check rollup, the two facts the state machine was
|
|
# blind to (#136). `gh pr view` rather than the REST PR object: the API's
|
|
# `mergeable` is a tri-state boolean that GitHub computes lazily, while
|
|
# this returns the same MERGEABLE/CONFLICTING/UNKNOWN string the UI shows.
|
|
# Failure to read them is NOT fatal and NOT treated as broken — an API
|
|
# hiccup must never flap every PR into needs-rebase, so both degrade to
|
|
# the "do not know" value that triggers nothing.
|
|
GH_VIEW="$(gh pr view "$n" -R "$REPO" --json mergeable,statusCheckRollup 2>/dev/null || echo '{}')"
|
|
MERGEABLE="$(jq -r '.mergeable // "UNKNOWN"' <<<"$GH_VIEW")"
|
|
CHECKS="$(checks_state <<<"$GH_VIEW")"
|
|
reconcile_pr "$n"
|
|
) || log "#$n: reconcile failed — continuing with the remaining PRs"
|
|
done
|
|
log "reconciled."
|
|
}
|
|
|
|
# sourced by test/labels-reconcile.sh for the fixture tests; executed in CI
|
|
if [ "${BASH_SOURCE[0]}" = "$0" ]; then
|
|
main "$@"
|
|
fi
|