bootstrap: infer the tailnet tag from the pre-auth key — verify the tag control granted, don't assert the one rig requested #16

New issue

Closed

opened 2026-07-17 14:18:20 +00:00 by dan-claude-bot · 0 comments

dan-claude-bot commented

2026-07-17 14:18:20 +00:00

(Migrated from github.com)

rig bootstrap takes --ts-tag and passes it to tailscale up --advertise-tags. But a pre-auth key already carries the tags it was minted with. So the tag is stated twice — once in the Tailscale admin console when the key is created, once on rig's command line — and rig has no idea whether the two agree. Two sources of truth for one fact, reconciled by the operator's memory.

They can be collapsed: the key is already the source of truth, and rig can just stop overriding it.

The tag is the sshd bug wearing a different hat

rig asserts the tag it requested, never the tag control actually granted. That is exactly the shape of the 99-rig.conf first-wins failure — rig asserted the file it wrote rather than what sshd resolved, and shipped green for a month while every Hetzner box served passwordauthentication yes. bootstrap now refuses to claim success unless sshd -T agrees.

The tag has had no such assert. And the fleet already has the scar: per #12, both M900s were carrying tag:server and had to be retagged by hand. Nothing in rig noticed, because nothing in rig ever looked.

What the research actually says

A tagged key applies its tags with no --advertise-tags at all. "if you use tags with an auth key, after a device logs in as the user who generated the auth key, the device assumes the identity of the auth key's tags." (auth-keys, tags) — so inference is not a trick, it is the documented default.
A mismatch is usually a loud failure, not a silent wrong tag — requested tags must "exactly match the full set of tags on the auth key, or each requested tag must be owned by one of the authenticating entity's tags as defined in tagOwners" (tags). So today's risk is mostly a failed bootstrap. Mostly — a tagOwners entry that lets the key's tag own the requested one turns it back into a silent override. rig should not depend on our ACL never growing that edge.
An UNTAGGED key is the real hazard, and it is silent. "The owner of a device authenticated by an auth key is either the user that created the auth key, or one or more tags that the user specified." With no tags the node joins owned by the key creator's user identity — it inherits that human's ACL grants, is subject to key expiry (tagged devices default to expiry disabled), and vanishes if the user is deleted. Today --ts-tag papers over this: pass an untagged key, and rig's --advertise-tags tags the node anyway. Dropping --advertise-tags removes that accidental safety net — which is why this issue is a verify story, not just a delete story.
Tags cannot converge post-join. tailscale set has no tag flag (verified against cmd/tailscale/cli/set.go; corroborated by the CLI KB). Re-tagging requires up --force-reauth — i.e. a fresh key — because "Re-tagging a device ... requires reauthentication." The --hostname trick (#12's tailscale set --hostname) has no analogue here. rig can detect a wrong tag; it cannot fix one. So it must fail loudly and print the repair, never claim convergence.
A key's tags cannot be read before use. GET /api/v2/tailnet/{tailnet}/keys/{keyId} needs an API token (tskey-api-…) or OAuth client; a tskey-auth-… is a device-registration credential, a different credential class. Pre-flight validation would mean rig holding an API token — which the "no credential, ever" contract forbids. Post-join verification is the only lever available.

Proposed

Stop requesting a tag. Verify the granted one.

tailscale up drops --advertise-tags entirely; the key's tags apply.
After join, poll tailscale status --json for .Self.Tags and assert on it. Tags arrive with the netmap, not synchronously from up — a single read right after up can legitimately come back empty, so poll until non-empty or .BackendState == "Running", then decide.
Untagged → hard refusal. The node joined user-owned; that is a fleet-shaped mistake, not a warning. rig should tailscale logout to back the node out rather than leave a half-joined user-owned device behind, then die naming the fix (mint a tagged key).
Role policy moves onto the effective tag. The runner refusal survives, strictly stronger than today: it stops being "you typed the wrong flag" and becomes "the key you actually used grants tag:server to repo-controlled code."
Verify on the already-joined path too, not just first join. This is the part that catches the M900s: a box bootstrapped before this change, or retagged behind rig's back, gets caught on the next ordinary re-run. Convergent boxes make this free — we re-run rig anyway.

`--ts-tag` is removed, not demoted

The flag goes away entirely. The key decides the tag; role policy guards the outcome. One source of truth, and the flag that could disagree with it no longer exists.

The alternative considered and rejected was demoting it to an --expect-tag assertion — never sent to tailscale up, only checked against .Self.Tags. It would catch "I grabbed the tag:ci key for a workload box", which removal only catches for runner. Rejected because it reintroduces the thing this issue exists to delete: a second place to state the tag, and therefore a second thing to keep in sync. The untagged-key refusal and the runner policy cover the failures that actually cost something.

Implementation notes

No jq on a rig box, and lib/runner-config.sh's json_field only reads string values — .Self.Tags is an array. Needs a small array-aware reader (same grep/sed spirit, same "installing jq to read one field is a poor trade" reasoning), or tailscale status --json piped through a targeted match. Whatever it is, it belongs next to json_field and gets its own test.
tailscale debug prefs is the wrong source and will lie — it prints AdvertiseTags, i.e. what was requested, not what control granted. That trap has a bug of its own (#2641). .Self.Tags reflects control's netmap and is ground truth. Using prefs here would re-commit the original sin in a new file.
--ts-tag should die with a message, not an "unknown flag" error. It is in runbooks and muscle memory; bootstrap.sh should keep matching it and exit 2 saying the tag now comes from the key. A bare unknown flag: --ts-tag reads like a rig bug and invites a re-run with the flag deleted but the untagged key still in hand — the one case this issue must not let through quietly.
test/cli.sh:39 must change. It asserts the runner refusal through the flag (bootstrap.sh runner --ts-tag tag:server → exit 2). With the flag gone, that test asserts nothing real: the refusal moves onto the effective tag, which needs a tailnet and cannot be reached from test/cli.sh. Replace it with the --ts-tag-is-removed exit-2 assertion, and let the rehearsal cover the actual policy — deleting the test outright would quietly drop the only tag:server coverage we have.
Not a concern for rig, but worth knowing: re-running an identical tagged-authkey tailscale up errors (#5597, closed as not planned). rig dodges it because it skips up on an already-joined box — a nice accident of the convergence design that this change must not undo.

Rehearsal

The Incus rehearsal is structurally blind to this (as it was to the sshd bug — no cloud-init drop-in to lose to). Cover in test/cli.sh what is testable without a tailnet (arg surface, refusal exits), and rehearse the real thing on a throwaway node with three keys: tagged-correct, tagged-wrong-for-role, and untagged — the last is the one that matters, and the one no test we have today would catch.

`rig bootstrap` takes `--ts-tag` and passes it to `tailscale up --advertise-tags`. But a pre-auth key **already carries the tags it was minted with**. So the tag is stated twice — once in the Tailscale admin console when the key is created, once on rig's command line — and rig has no idea whether the two agree. Two sources of truth for one fact, reconciled by the operator's memory. They can be collapsed: **the key is already the source of truth, and rig can just stop overriding it.** ## The tag is the sshd bug wearing a different hat rig asserts the tag it *requested*, never the tag control actually *granted*. That is exactly the shape of the `99-rig.conf` first-wins failure — rig asserted the *file it wrote* rather than what `sshd` resolved, and shipped green for a month while every Hetzner box served `passwordauthentication yes`. `bootstrap` now refuses to claim success unless `sshd -T` agrees. The tag has had no such assert. And the fleet already has the scar: per #12, **both M900s were carrying `tag:server` and had to be retagged by hand.** Nothing in rig noticed, because nothing in rig ever looked. ## What the research actually says 1. **A tagged key applies its tags with no `--advertise-tags` at all.** "if you use tags with an auth key, after a device logs in as the user who generated the auth key, the device assumes the identity of the auth key's tags." ([auth-keys](https://tailscale.com/kb/1085/auth-keys), [tags](https://tailscale.com/kb/1068/tags)) — so inference is not a trick, it is the documented default. 2. **A mismatch is usually a loud failure, not a silent wrong tag** — requested tags must "exactly match the full set of tags on the auth key, or each requested tag must be owned by one of the authenticating entity's tags as defined in `tagOwners`" ([tags](https://tailscale.com/kb/1068/tags)). So today's risk is mostly a failed bootstrap. *Mostly* — a `tagOwners` entry that lets the key's tag own the requested one turns it back into a silent override. rig should not depend on our ACL never growing that edge. 3. **An UNTAGGED key is the real hazard, and it is silent.** "The owner of a device authenticated by an auth key is either the user that created the auth key, or one or more tags that the user specified." With no tags the node joins **owned by the key creator's user identity** — it inherits that human's ACL grants, is subject to key expiry (tagged devices default to expiry disabled), and vanishes if the user is deleted. Today `--ts-tag` papers over this: pass an untagged key, and rig's `--advertise-tags` tags the node anyway. **Dropping `--advertise-tags` removes that accidental safety net** — which is why this issue is a verify story, not just a delete story. 4. **Tags cannot converge post-join.** `tailscale set` has **no tag flag** (verified against `cmd/tailscale/cli/set.go`; corroborated by the [CLI KB](https://tailscale.com/kb/1080/cli)). Re-tagging requires `up --force-reauth` — i.e. a fresh key — because "Re-tagging a device ... requires reauthentication." The `--hostname` trick (#12's `tailscale set --hostname`) has no analogue here. rig can *detect* a wrong tag; it cannot *fix* one. So it must fail loudly and print the repair, never claim convergence. 5. **A key's tags cannot be read before use.** `GET /api/v2/tailnet/{tailnet}/keys/{keyId}` needs an API token (`tskey-api-…`) or OAuth client; a `tskey-auth-…` is a device-registration credential, a [different credential class](https://tailscale.com/kb/1277/key-prefixes). Pre-flight validation would mean rig holding an API token — which the "no credential, ever" contract forbids. **Post-join verification is the only lever available.** ## Proposed Stop requesting a tag. Verify the granted one. - `tailscale up` drops `--advertise-tags` entirely; the key's tags apply. - **After join, poll `tailscale status --json` for `.Self.Tags`** and assert on it. Tags arrive with the netmap, not synchronously from `up` — a single read right after `up` can legitimately come back empty, so poll until non-empty or `.BackendState == "Running"`, then decide. - **Untagged → hard refusal.** The node joined user-owned; that is a fleet-shaped mistake, not a warning. rig should `tailscale logout` to back the node out rather than leave a half-joined user-owned device behind, then die naming the fix (mint a tagged key). - **Role policy moves onto the effective tag.** The `runner` refusal survives, strictly stronger than today: it stops being "you typed the wrong flag" and becomes "the key you actually used grants `tag:server` to repo-controlled code." - **Verify on the already-joined path too**, not just first join. This is the part that catches the M900s: a box bootstrapped before this change, or retagged behind rig's back, gets caught on the next ordinary re-run. Convergent boxes make this free — we re-run rig anyway. ## `--ts-tag` is removed, not demoted The flag goes away entirely. The key decides the tag; role policy guards the outcome. One source of truth, and the flag that could disagree with it no longer exists. The alternative considered and rejected was demoting it to an `--expect-tag` assertion — never sent to `tailscale up`, only checked against `.Self.Tags`. It would catch "I grabbed the `tag:ci` key for a workload box", which removal only catches for `runner`. Rejected because it reintroduces the thing this issue exists to delete: a second place to state the tag, and therefore a second thing to keep in sync. The untagged-key refusal and the `runner` policy cover the failures that actually cost something. ## Implementation notes - **No `jq` on a rig box**, and `lib/runner-config.sh`'s `json_field` only reads *string* values — `.Self.Tags` is an *array*. Needs a small array-aware reader (same grep/sed spirit, same "installing jq to read one field is a poor trade" reasoning), or `tailscale status --json` piped through a targeted match. Whatever it is, it belongs next to `json_field` and gets its own test. - **`tailscale debug prefs` is the wrong source and will lie** — it prints `AdvertiseTags`, i.e. what was *requested*, not what control *granted*. That trap has a bug of its own ([#2641](https://github.com/tailscale/tailscale/issues/2641)). `.Self.Tags` reflects control's netmap and is ground truth. Using prefs here would re-commit the original sin in a new file. - **`--ts-tag` should die with a message, not an "unknown flag" error.** It is in runbooks and muscle memory; `bootstrap.sh` should keep matching it and exit 2 saying the tag now comes from the key. A bare `unknown flag: --ts-tag` reads like a rig bug and invites a re-run with the flag deleted but the *untagged key* still in hand — the one case this issue must not let through quietly. - **`test/cli.sh:39` must change.** It asserts the `runner` refusal *through the flag* (`bootstrap.sh runner --ts-tag tag:server` → exit 2). With the flag gone, that test asserts nothing real: the refusal moves onto the effective tag, which needs a tailnet and cannot be reached from `test/cli.sh`. Replace it with the `--ts-tag`-is-removed exit-2 assertion, and let the rehearsal cover the actual policy — deleting the test outright would quietly drop the only `tag:server` coverage we have. - Not a concern for rig, but worth knowing: re-running an identical tagged-authkey `tailscale up` errors ([#5597](https://github.com/tailscale/tailscale/issues/5597), closed as not planned). rig dodges it because it skips `up` on an already-joined box — a nice accident of the convergence design that this change must not undo. ## Rehearsal The Incus rehearsal is structurally blind to this (as it was to the sshd bug — no cloud-init drop-in to lose to). Cover in `test/cli.sh` what is testable without a tailnet (arg surface, refusal exits), and rehearse the real thing on a throwaway node with **three** keys: tagged-correct, tagged-wrong-for-role, and **untagged** — the last is the one that matters, and the one no test we have today would catch.

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: heavy-duty/rig#16

No description provided.