Buildkite Hardening Guide

DevOps Last updated: 2026-08-20

CI/CD platform hardening for Buildkite including SAML SSO, team permissions, agent security, and pipeline controls

Code Packs: Terraform API Scripts CLI Scripts Config

View:

Buildkite — Requirements Cheat Sheet

All controls condensed. Click any requirement to jump to full implementation details.

Requirement	Description	Rationale

Overview

Buildkite is a CI/CD platform enabling organizations to run fast, secure builds on their own infrastructure. As a platform managing build pipelines and deployment workflows, Buildkite security configurations directly impact software supply chain security.

Intended Audience

Security engineers managing CI/CD platforms
Platform engineers configuring Buildkite
DevOps teams managing pipelines
GRC professionals assessing build security

How to Use This Guide

L1 (Crawl): Essential controls for all organizations
L2 (Walk): Enhanced controls for security-sensitive environments
L3 (Run): Strictest controls for regulated industries

Scope

This guide covers Buildkite security including SAML SSO, team permissions, agent security, and pipeline controls.

Authentication & SSO
Access Controls
Agent Security
Monitoring & Compliance
Compliance Quick Reference

1. Authentication & SSO

1.1 Configure SAML Single Sign-On

Profile Level: L1 (Crawl)

Framework	Control
CIS Controls	6.3, 12.5
NIST 800-53	IA-2, IA-8

Description

Configure SAML SSO to centralize authentication for Buildkite users.

Rationale

Why This Matters:

Centralizes Buildkite authentication in your corporate IdP, enforcing MFA, conditional access, and session policy on every login
Local Buildkite passwords bypass IdP controls and are prime targets for credential stuffing and phishing
Group-to-team mapping plus IdP deprovisioning removes departed employees’ access automatically, eliminating orphaned accounts
Buildkite pipelines hold deployment credentials and source-build access, so a single compromised login can poison the software supply chain

Attack Prevented: Credential theft, phishing, MFA bypass, orphaned-account access

Prerequisites

Buildkite organization admin access
Pro or Enterprise plan. SSO is not available below Pro, and there is no “Business” tier — if a runbook or vendor questionnaire references one, it is wrong.
SAML 2.0 compatible IdP

ClickOps Implementation

Step 1: Access SSO Settings

Navigate to: Organization Settings → Single Sign On
Select SAML provider type

Step 2: Configure SAML

Configure IdP settings:
- SSO URL
- Entity ID
- Certificate
Configure attribute mapping
Map groups to teams

Step 3: Test and Enforce

Test SSO authentication with a non-admin account before touching enforcement.
Set SSO to required rather than optional. Buildkite tracks this per user, so an organization is only actually covered once every member is on the required setting — a partially-required organization still has password login paths open.
Enforce SSO organization-wide by disabling 2FA authentication as a login method. This is the counterintuitive part: leaving password-plus-2FA available is what keeps a non-SSO path alive, so disabling it is how the IdP becomes the only front door.
New members are provisioned just-in-time on first login through the IdP, so group-to-team mapping needs to be correct before you enforce, not after.
Document the admin fallback path before enforcement, so an IdP outage does not lock the organization out.

Step 4: Tighten Session and Network Controls

Set the session timeout to the shortest value your teams will tolerate. The supported range runs from 6 hours to 1 year; a year-long session on a CI platform holding deployment credentials is functionally a permanent one.
IP address pinning (Enterprise): enable it to revoke a session the moment the source IP changes. This is a cheap and effective defense against a stolen session cookie being replayed from elsewhere.
SCIM deprovisioning (Enterprise): connect SCIM so that removing someone in the IdP removes their Buildkite access rather than leaving an orphaned account behind.

Source: Buildkite SSO

Time to Complete: ~1-2 hours

Code Implementation

Code Pack: API Script

hth-buildkite-1.01-configure-saml-sso.sh View source on GitHub ↗

# Read the organization's SSO providers, their state, and the two session
# controls. Run this BEFORE any mutation.
# sessionDurationInHours / pinSessionToIpAddress / testAuthorizationRequired are
# fields of the SSOProvider INTERFACE, so they are selected directly and resolve
# for SAML, Google Workspace and GitHub App providers alike (see TRAP 2).
read_sso() {
  jq -n --arg slug "${BUILDKITE_ORG_SLUG}" '{
    query: "query($slug:ID!){ organization(slug:$slug){ id name
              ssoProviders(first:10){ edges { node {
                id uuid type state note emailDomain
                sessionDurationInHours pinSessionToIpAddress
                testAuthorizationRequired
              } } } } }",
    variables: { slug: $slug }
  }' | gql
}

report_sso() {
  local raw
  raw="$(read_sso)" || return 1
  require_no_errors "${raw}" "read_sso" || return 1
  jq '{
    organization: .data.organization.name,
    providers: [ .data.organization.ssoProviders.edges[].node ],
    findings: [ .data.organization.ssoProviders.edges[].node
                | select(.state == "ENABLED")
                | select((.sessionDurationInHours == null) or (.pinSessionToIpAddress != true))
                | { id, type,
                    session_duration_hours: .sessionDurationInHours,
                    ip_pinned: .pinSessionToIpAddress,
                    finding: "enabled provider without a bounded session and IP-pinned session" } ]
  }' <<<"${raw}"
}

# Enable an existing, already-tested SSO provider.
# ORDER MATTERS: create -> members authorize -> enable. Enabling a provider whose
# members have not authorized it locks them out; there is no bypass except the
# disable mutation below, which needs an API token issued before the lockout.
#
# This is the most lockout-capable call in the pack, so it is the most gated: a
# literal CONFIRM, an acknowledgement variable, a fail-closed read-back of the
# target provider, and the recovery command printed before the write. Nothing
# here is advice — every one of them refuses.
enable_sso() {
  local provider_id="$1" confirm="${2:-}"

  [ "${confirm}" = "CONFIRM" ] || {
    echo "refusing: pass the literal argument CONFIRM after the provider id." >&2
    echo "usage: $0 enable <provider-id> CONFIRM" >&2
    return 2
  }
  [ "${HTH_SSO_LOCKOUT_ACK:-}" = "1" ] || {
    echo "refusing: enabling this provider makes the IdP the login path for every" >&2
    echo "member of ${BUILDKITE_ORG_SLUG}. Set HTH_SSO_LOCKOUT_ACK=1 to record that" >&2
    echo "you have read the LOCKOUT WARNING at the top of this file and that you" >&2
    echo "hold an API access token issued outside the SSO you are about to enable." >&2
    return 2
  }

  # Fail-closed read-back. The provider must be one this organization actually
  # has, in a state this script can reason about, before anything is written.
  local raw node state tested
  raw="$(read_sso)" || return 1
  require_no_errors "${raw}" "read_sso" || return 1
  node="$(jq -c --arg id "${provider_id}" '
    [ .data.organization.ssoProviders.edges[].node | select(.id == $id) ] | first // empty
  ' <<<"${raw}")"

  if [ -z "${node}" ]; then
    echo "refusing: no SSO provider with id '${provider_id}' exists in ${BUILDKITE_ORG_SLUG}." >&2
    echo "Run \`$0 verify\` and enable one of the provider ids it lists." >&2
    return 3
  fi

  state="$(jq -r '.state // "UNKNOWN"' <<<"${node}")"
  if [ "${state}" = "ENABLED" ]; then
    jq -n --arg id "${provider_id}" \
      '{changed: 0, id: $id, state: "ENABLED", note: "provider is already ENABLED"}'
    return 0
  fi
  case "${state}" in
    CREATED|DISABLED) ;;
    *) echo "refusing: provider ${provider_id} reports state '${state}', which is" >&2
       echo "neither CREATED nor DISABLED. Re-read it with \`$0 verify\`." >&2
       return 3 ;;
  esac

  # TRAP 6: only a definitive false proceeds. true and null are both "not proven
  # safe", because enabling a provider members have not authorized strands them.
  tested="$(jq -r 'if .testAuthorizationRequired == null
                   then "null" else (.testAuthorizationRequired | tostring) end' <<<"${node}")"
  if [ "${tested}" != "false" ] && [ "${HTH_SSO_UNTESTED_PROVIDER:-}" != "1" ]; then
    echo "refusing: provider ${provider_id} reports testAuthorizationRequired=${tested}," >&2
    echo "not a definitive false. Buildkite requires each member to authorize the" >&2
    echo "provider once before enforcement takes effect, so enabling now strands" >&2
    echo "everyone who has not. Authorize it and re-run \`$0 verify\`, or set" >&2
    echo "HTH_SSO_UNTESTED_PROVIDER=1 to record this as a deliberate exception." >&2
    return 3
  fi

  # The way back, on screen, before the lockout becomes possible.
  {
    echo "RECOVERY PATH — copy this line now, before the mutation is sent:"
    echo "  BUILDKITE_TOKEN=<off-session token> BUILDKITE_ORG_SLUG=${BUILDKITE_ORG_SLUG} \\"
    echo "    $0 disable ${provider_id}"
    echo "That token must already exist and must not depend on the SSO being enabled."
    echo "Enabling ${provider_id} (state ${state}, testAuthorizationRequired=${tested})..."
  } >&2

  local body
  body="$(jq -n --arg id "${provider_id}" '{
    query: "mutation($id:ID!){ ssoProviderEnable(input:{id:$id}){
              ssoProvider { id state } } }",
    variables: { id: $id }
  }' | gql_checked "ssoProviderEnable")" || return 1
  jq '.data.ssoProviderEnable' <<<"${body}"
}

# The documented way back in. Keep this reachable from a machine that is not
# behind the SSO you just enabled. Deliberately ungated — a recovery path with a
# confirmation gate is a recovery path you cannot use in an incident — but NOT
# unchecked: it routes through gql_checked so a revoked token or a wrong provider
# id fails loudly instead of printing null and exiting 0 (TRAP 5).
disable_sso() {
  local provider_id="$1"
  jq -n --arg id "${provider_id}" '{
    query: "mutation($id:ID!){ ssoProviderDisable(input:{id:$id}){
              ssoProvider { id state } } }",
    variables: { id: $id }
  }' | gql_checked "ssoProviderDisable"
}

# Bound the session and pin it to the address it was issued to.
# Duration caps how long a session outlives revocation at the IdP: disabling an
# account upstream does not terminate a Buildkite session already issued, so this
# number IS the worst-case window between offboarding and loss of access.
# IP pinning kills stolen-cookie replay from another network — Enterprise-gated
# (TRAP 4), so pass pin=false on other plans and set the duration alone.
harden_session() {
  local provider_id="$1" hours="$2" pin="${3:-true}"

  case "${hours}" in
    ''|*[!0-9]*) echo "session duration must be a positive integer number of hours" >&2; return 2 ;;
  esac
  # Local band, not a server rule (TRAP 3). 8760h is one year, Buildkite's
  # documented maximum; anything above 24h leaves a session usable for more than
  # a working day after an IdP revocation, so it must be stated deliberately.
  if [ "${hours}" -lt 1 ] || [ "${hours}" -gt 8760 ]; then
    echo "session duration ${hours}h is outside the supported 1-8760h range" >&2
    return 2
  fi
  if [ "${hours}" -gt 24 ] && [ "${HTH_ALLOW_LONG_SSO_SESSION:-}" != "1" ]; then
    echo "refusing ${hours}h: a session longer than 24h outlives same-day offboarding." >&2
    echo "Set HTH_ALLOW_LONG_SSO_SESSION=1 to record this as a deliberate exception." >&2
    return 3
  fi
  case "${pin}" in true|false) ;; *) echo "pin must be true or false" >&2; return 2 ;; esac

  jq -n --arg id "${provider_id}" --argjson hours "${hours}" --argjson pin "${pin}" '{
    query: "mutation($id:ID!,$hours:Int!,$pin:Boolean!){
              ssoProviderUpdate(input:{ id:$id,
                                        sessionDurationInHours:$hours,
                                        pinSessionToIpAddress:$pin }){
                ssoProvider { id state sessionDurationInHours pinSessionToIpAddress } } }",
    variables: { id: $id, hours: $hours, pin: $pin }
  }' | gql_checked "ssoProviderUpdate"
}

# Per-member SSO enforcement — the half of this control that has no console list
# view and no other detector (TRAP 1). `optional` is a server-side filtered
# count of the same connection, so the total and the finding count cannot drift
# apart between two round trips.
list_member_sso() {
  jq -n --arg slug "${BUILDKITE_ORG_SLUG}" --argjson n "${MEMBER_PAGE_SIZE}" '{
    query: "query($slug:ID!,$n:Int!){ organization(slug:$slug){
              members(first:$n){ count edges { node {
                id role sso { mode } user { name email }
              } } }
              optional: members(first:$n, sso:{mode:OPTIONAL}){ count } } }",
    variables: { slug: $slug, n: $n }
  }' | gql
}

# Exits 1 when any member can still authenticate without SSO, so this is usable
# as a pipeline gate and not only as a report.
members_report() {
  local raw total
  raw="$(list_member_sso)" || return 1
  require_no_errors "${raw}" "list_member_sso" || return 1

  total="$(jq -r '.data.organization.members.count' <<<"${raw}")"
  if [ "${total}" -gt "${MEMBER_PAGE_SIZE}" ]; then
    echo "refusing to report: ${total} members exceeds MEMBER_PAGE_SIZE=${MEMBER_PAGE_SIZE};" >&2
    echo "raise MEMBER_PAGE_SIZE so the roster is not silently truncated." >&2
    return 4
  fi

  jq '{
    members_total: .data.organization.members.count,
    sso_optional_total: .data.organization.optional.count,
    findings: [ .data.organization.members.edges[].node
                | select(.sso.mode != "REQUIRED")
                | { id, role, sso_mode: .sso.mode,
                    name: .user.name, email: .user.email,
                    finding: "member can authenticate without SSO" } ]
  }' <<<"${raw}"

  local optional
  optional="$(jq -r '.data.organization.optional.count' <<<"${raw}")"
  if [ "${optional}" -gt 0 ]; then
    echo "FAIL: ${optional} of ${total} members can still authenticate without SSO." >&2
    return 1
  fi
  return 0
}

# Flip one member to REQUIRED. The id is the OrganizationMember node id returned
# by list_member_sso — NOT the user id and NOT the uuid.
require_sso_for_member() {
  local member_id="$1"
  jq -n --arg id "${member_id}" '{
    query: "mutation($id:ID!){ organizationMemberUpdate(input:{ id:$id,
                                                                sso:{mode:REQUIRED} }){
              organizationMember { id role sso { mode } user { email } } } }",
    variables: { id: $id }
  }' | gql_checked "organizationMemberUpdate"
}

# Flip every non-REQUIRED member. Gated twice, because this is the one call in
# this pack that can lock out an entire organization in a single invocation:
#   1. a literal CONFIRM argument, so it cannot be reached by a typo; and
#   2. an ENABLED provider must exist — requiring SSO of members who have no
#      working provider to authorize against strands all of them at once.
require_sso_for_all() {
  [ "${1:-}" = "CONFIRM" ] || {
    echo "refusing: pass the literal argument CONFIRM to flip every member to REQUIRED" >&2
    return 2
  }

  local providers states
  providers="$(read_sso)" || return 1
  require_no_errors "${providers}" "read_sso" || return 1
  states="$(jq -r '[.data.organization.ssoProviders.edges[].node.state] | join(",")' <<<"${providers}")"
  case ",${states}," in
    *,ENABLED,*) ;;
    *) echo "refusing: no SSO provider is ENABLED (states: ${states:-none})." >&2
       echo "Requiring SSO with no enabled provider locks out every member." >&2
       return 3 ;;
  esac

  local raw ids
  raw="$(list_member_sso)" || return 1
  require_no_errors "${raw}" "list_member_sso" || return 1
  ids="$(jq -r '.data.organization.members.edges[].node
                | select(.sso.mode != "REQUIRED") | .id' <<<"${raw}")"

  if [ -z "${ids}" ]; then
    jq -n '{changed: 0, note: "every member is already REQUIRED"}'
    return 0
  fi

  # Per-member accounting. A member whose update is rejected must not be
  # rendered as `null` in a stream that otherwise reads as a completed flip
  # (TRAP 5) — every failure is counted, named, and makes the run exit non-zero.
  # The loop continues rather than aborting so one rejection does not hide the
  # rest, but the organization is then in a PARTIAL state and says so.
  local changed=0 failed=0 failed_ids="" body
  while IFS= read -r id; do
    [ -n "${id}" ] || continue
    if body="$(require_sso_for_member "${id}")"; then
      jq -c '.data.organizationMemberUpdate.organizationMember' <<<"${body}"
      changed=$((changed + 1))
    else
      failed=$((failed + 1))
      failed_ids="${failed_ids}${failed_ids:+ }${id}"
    fi
  done <<<"${ids}"

  if [ "${failed}" -gt 0 ]; then
    echo "FAIL: ${changed} member(s) flipped to REQUIRED, ${failed} REJECTED: ${failed_ids}" >&2
    echo "This organization is PARTIALLY enforced — the members above can still" >&2
    echo "authenticate without SSO. Fix the errors printed above and re-run;" >&2
    echo "this verb is idempotent and re-selects only non-REQUIRED members." >&2
    return 1
  fi
  jq -n --argjson changed "${changed}" '{changed: $changed, failed: 0}'
}

1.2 Enforce Two-Factor Authentication

Profile Level: L1 (Crawl)

Framework	Control
CIS Controls	6.5
NIST 800-53	IA-2(1)

Description

Require 2FA for all Buildkite users.

Rationale

Why This Matters:

A second factor blocks attackers who have already obtained a valid Buildkite password through phishing, reuse, or a breach
CI/CD accounts can trigger builds and deployments, so a single-factor takeover can reach production
Org-wide enforcement closes the gap left by users who would otherwise never enable 2FA voluntarily
Phishing-resistant factors for admins protect the highest-privilege accounts against real-time relay attacks

Attack Prevented: Credential stuffing, password reuse, phishing, account takeover

Prerequisites

ClickOps: none. The Security page toggle is available on every plan.
Terraform: a plan that includes the API IP allowlist feature. The buildkite_organization resource fails to create without it, even when your configuration sets only enforce_2fa and never mentions IP allowlisting — the provider touches that field regardless. Verified against a live organization: Unable to create Organization settings: input: The API IP allowlist feature is not available for your organization. Please upgrade your plan to access it. The same resource backs 4.1, so that control inherits the constraint. On plans without the feature, use the ClickOps path below; terraform validate and terraform plan both pass, so this surfaces only at apply.

ClickOps Implementation

Step 1: Enable 2FA Requirement

Navigate to: Organization Settings → Security
Enable Enforce Two-factor authentication
All users must configure 2FA

Step 2: Configure via IdP

Enable MFA in identity provider
Use phishing-resistant methods for admins
All SSO users subject to IdP MFA

Code Implementation

Code Pack: Terraform

hth-buildkite-1.02-enforce-two-factor-authentication.tf View source on GitHub ↗

resource "buildkite_organization" "hardened" {
  enforce_2fa = var.enforce_2fa
}

2. Access Controls

2.1 Configure Team Permissions

Profile Level: L1 (Crawl)

Framework	Control
CIS Controls	5.4
NIST 800-53	AC-6

Description

Implement least privilege using Buildkite teams.

Rationale

Why This Matters:

Scoping each team to only the pipelines it needs limits the blast radius if any one account is compromised
Granular permission levels (Read & Build versus Full Access) prevent over-broad rights that let any user modify pipeline configuration
Quarterly membership reviews catch privilege creep and remove access from people who changed roles
Function-based teams make access auditable and map cleanly to compliance least-privilege requirements

Attack Prevented: Privilege escalation, lateral movement, insider misuse, unauthorized pipeline changes

ClickOps Implementation

Step 1: Create Teams

Navigate to: Organization Settings → Teams
Create teams by function
Define team permissions

Step 2: Assign Pipeline Access

Assign pipelines to teams
Configure permission levels:
- Read & Build Access
- Full Access
Apply least privilege

Step 3: Regular Access Reviews

Review team membership quarterly
Update access as needed
Remove inactive members

Code Implementation

Code Pack: Terraform

hth-buildkite-2.01-configure-team-permissions.tf View source on GitHub ↗

# Create teams with least-privilege defaults
resource "buildkite_team" "teams" {
  for_each = var.teams

  name                = each.key
  description         = each.value.description
  privacy             = each.value.privacy
  default_team        = each.value.default_team
  default_member_role = each.value.default_member_role

  # All five members_can_* privileges are optional + COMPUTED. Declaring only one
  # of them, as this pack previously did, leaves the other four adopted from the
  # server on every refresh: a privilege granted in the console produces an empty
  # plan and is never reverted. Least privilege that Terraform cannot see is not
  # enforced, it is merely hoped for — so all five are declared here.
  members_can_create_pipelines  = each.value.members_can_create_pipelines
  members_can_create_registries = each.value.members_can_create_registries
  members_can_create_suites     = each.value.members_can_create_suites

  # DESTRUCTIVE PRIVILEGES — the two that were wholly unmanaged.
  # Package and registry deletion removes published artifacts and the registries
  # that hold them. Deletion is not an API-reversible operation, and a deleted
  # package takes its provenance and attestations with it, so this is the fastest
  # supply-chain-erasure path Buildkite exposes to a non-admin. Both default to
  # false in var.teams; granting either should be a reviewed, named exception.
  members_can_destroy_packages   = each.value.members_can_destroy_packages
  members_can_destroy_registries = each.value.members_can_destroy_registries
}

2.2 Configure Pipeline Permissions

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	5.4
NIST 800-53	AC-6

Description

Control access to specific pipelines.

Rationale

Why This Matters:

Per-pipeline visibility keeps sensitive production and deployment pipelines hidden from users who have no need to see them
Restricting who can trigger builds prevents unauthorized or accidental runs against production
Limiting manual builds on production reduces the chance of an attacker forcing a malicious deployment
Auditing build triggers creates accountability for every pipeline execution

Attack Prevented: Unauthorized deployment, pipeline tampering, supply chain injection, information disclosure

ClickOps Implementation

Step 1: Configure Pipeline Visibility

Set pipeline visibility per pipeline
Restrict sensitive pipelines
Use team-based access

Step 2: Configure Build Permissions

Control who can trigger builds
Restrict manual builds on production
Audit build triggers

Step 3: Set the Organization-Level Pipeline Toggles

Go to Settings → Security → Pipelines and review the organization-wide toggles: Create Pipelines, Delete Pipelines, Change Pipeline Visibility, Manage Notification Services, Manage Agent Registration Tokens, and Stop Agents.
Create Pipelines and the team-level members_can_create_pipelines setting in 2.1 are alternatives, not layers. Buildkite’s own permissions documentation says of Create Pipelines: “if the teams feature is enabled, then this permission is controlled at a team-level and therefore, this option will be unavailable on this page.” The org toggle is therefore absent exactly when teams exist, and there are no teams to carry members_can_create_pipelines when it is present. Decide which surface governs pipeline creation by deciding whether teams are on — no configuration applies both, so do not plan for one to backstop the other. The remaining five toggles have no team-level equivalent and always apply organization-wide.
Change Pipeline Visibility is the toggle that most directly undoes Step 1: leaving it open means any member can flip a private pipeline public regardless of how the pipeline itself is configured.

Automation: ClickOps only — Buildkite exposes no write interface for these toggles. The /organizations/{org.slug} REST family is not read-only, but none of its write verbs reaches them: PATCH .../pipeline-settings carries only organization defaults and timeouts (default_branch, default_cluster_id, default_timeout_in_minutes, maximum_timeout_in_minutes, scheduled_job_expiry_in_minutes), the sub-resource verbs cover hosted-agents-ssh, public-pipelines, advanced-queue-metrics and build-export, and PATCH .../api-settings belongs to 2.5 — there is no organization security or permissions resource in the documented set. None of the 85 GraphQL mutations sets them either, and buildkite_organization exposes only allowed_api_ip_addresses and enforce_2fa (pipeline settings API, organization security for pipelines, verified against the live schema 2026-08-18). Do not mistake the one writable neighbour for a substitute: public_pipeline_creation governs whether members may create a public pipeline, not whether they may make an existing private pipeline public, so it does not implement Change Pipeline Visibility. This step is Enterprise-gated and must be verified in the console.

Code Implementation

Code Pack: Terraform

hth-buildkite-2.02-configure-pipeline-permissions.tf View source on GitHub ↗

# Create pipelines with hardened defaults. Control 2.2 is an L2 control.
#
# NO PROFILE-LEVEL GATE ON for_each, DELIBERATELY. This line used to read
# `var.profile_level >= 2 ? var.pipelines : {}`. var.profile_level defaults to 1,
# so a single `terraform apply` that omitted `-var="profile_level=2"` emptied the
# map and Terraform DESTROYED every pipeline it had created, taking that
# pipeline's build history and its webhook URL with it. Profile level selects
# WHAT you declare; it must never decide whether declared resources survive.
# var.pipelines defaults to `{}`, so declaring nothing is already the "off"
# state. Pack 3.10 carries the same fix on buildkite_pipeline.templated.
resource "buildkite_pipeline" "pipelines" {
  for_each = var.pipelines

  name                       = each.key
  repository                 = each.value.repository
  description                = each.value.description
  default_branch             = each.value.default_branch
  branch_configuration       = each.value.branch_configuration
  skip_intermediate_builds   = each.value.skip_intermediate_builds
  cancel_intermediate_builds = each.value.cancel_intermediate_builds
  cluster_id                 = each.value.cluster_id
  default_timeout_in_minutes = each.value.default_timeout_in_minutes
  maximum_timeout_in_minutes = each.value.maximum_timeout_in_minutes
  allow_rebuilds             = each.value.allow_rebuilds

  # Guide Step 1 ("Configure Pipeline Visibility") implemented. PUBLIC exposes
  # build logs, job output, artifacts metadata and the pipeline definition to
  # anonymous internet users, so this defaults to PRIVATE in var.pipelines.
  #
  # WHY IT MUST BE DECLARED, NOT OMITTED: `visibility` is optional + COMPUTED in
  # buildkite/buildkite. An omitted computed attribute is not "the safe default" —
  # Terraform reads whatever the server currently holds into state and treats that
  # value as desired. A console flip to PUBLIC therefore produces an empty plan and
  # survives every subsequent apply, forever. Declaring it is what converts that
  # flip into drift the next `terraform plan` reverts.
  visibility = each.value.visibility

  # Restrict fork builds to prevent untrusted code execution.
  # provider_settings is an ATTRIBUTE in buildkite/buildkite ~> 1.0, not a block —
  # block syntax fails `terraform validate` with "Unsupported block type".
  provider_settings = {
    build_pull_request_forks              = false
    publish_commit_status                 = true
    publish_commit_status_per_step        = true
    skip_builds_for_existing_commits      = true
    cancel_deleted_branch_builds          = true
    prefix_pull_request_fork_branch_names = true
  }

  lifecycle {
    # A Buildkite pipeline carries its build history and its webhook URL; both
    # are destroyed with it and neither comes back. Dropping a key from
    # var.pipelines must be a deliberate act, not the side effect of a forgotten
    # -var or an edited tfvars file, so Terraform refuses the destroy at PLAN
    # time. Same guard as pack 3.11 puts on buildkite_registry and pack 3.10 on
    # buildkite_pipeline.templated. To retire a pipeline on purpose: delete this
    # line, apply, restore it.
    #
    # DELIBERATELY A LITERAL. OpenTofu 1.12 does accept an expression here
    # (measured, not assumed), which is precisely why this must not become
    # `var.profile_level >= 2`: the one forgotten -var would then empty the
    # for_each AND disarm the guard in the same plan, rebuilding the original
    # bug with an extra step.
    prevent_destroy = true
  }
}

# Assign team access to pipelines with explicit permission levels.
# No profile gate, same reasoning: an empty var.pipeline_team_access (the
# default) is the off state. A destroyed buildkite_pipeline_team revokes that
# team's access to the pipeline, which is recoverable by re-declaring it — but it
# should still not happen because someone forgot a -var.
resource "buildkite_pipeline_team" "access" {
  for_each = var.pipeline_team_access

  pipeline_id  = buildkite_pipeline.pipelines[each.value.pipeline_key].id
  team_id      = buildkite_team.teams[each.value.team_key].id
  access_level = each.value.access_level
}

2.3 Limit Admin Access

Profile Level: L1 (Crawl)

Framework	Control
CIS Controls	5.4
NIST 800-53	AC-6(1)

Description

Minimize and protect administrator accounts.

Rationale

Why This Matters:

Organization admins can change SSO, permissions, agent tokens, and billing, so a compromised admin account compromises everything
Keeping admins to a small set reduces the number of high-value targets an attacker can phish
Requiring SSO and 2FA on admins ensures the most powerful accounts get the strongest authentication
Monitoring admin activity surfaces anomalous configuration changes before they cause damage

Attack Prevented: Admin account takeover, privilege escalation, configuration tampering, persistence

ClickOps Implementation

Step 1: Inventory Admins

Review organization owners/admins
Document admin access
Identify unnecessary privileges

Step 2: Apply Restrictions

Limit admin to 2-3 users
Require 2FA/SSO for admins
Monitor admin activity

Code Implementation

Code Pack: Terraform

hth-buildkite-2.03-limit-admin-access.tf View source on GitHub ↗

# The full organization roster. Surfaces joiners and leavers as plan diffs.
data "buildkite_organization_members" "all" {}

# Assert the roster has not grown past the size you reviewed. A bare count is a
# weak control on its own, but it turns silent membership growth into a failed
# plan, which is the point.
check "membership_within_expected_size" {
  assert {
    condition = length(data.buildkite_organization_members.all.members) <= var.max_org_members
    error_message = format(
      "Organization has %d members, expected at most %d. Review the roster and either remove members or raise max_org_members deliberately.",
      length(data.buildkite_organization_members.all.members),
      var.max_org_members,
    )
  }
}

output "organization_roster" {
  description = "Every organization member (role is NOT exposed by this data source — see header)."
  value = [
    for m in data.buildkite_organization_members.all.members : {
      name  = m.name
      email = m.email
      uuid  = m.uuid
    }
  ]
}

# Team-scoped roles ARE manageable. Declaring maintainers explicitly means an
# out-of-band promotion in the console shows up as drift on the next plan.
resource "buildkite_team_member" "maintainers" {
  for_each = var.team_maintainers

  team_id = each.value.team_id
  user_id = each.value.user_id
  role    = "MAINTAINER"
}

resource "buildkite_team_member" "members" {
  for_each = var.team_members

  team_id = each.value.team_id
  user_id = each.value.user_id
  role    = "MEMBER"
}

Code Pack: API Script

hth-buildkite-2.03-limit-admin-access.sh View source on GitHub ↗

# Full-roster enumeration. Every member is fetched and classified locally; the
# server-side role filter is used only as an independent cross-check, because a
# filtered query that returns nothing looks identical to an org with no admins.
# shellcheck disable=SC2016  # GraphQL $variables must stay literal, not shell-expanded
ROSTER_QUERY='query($slug:ID!,$first:Int!,$after:String){
  organization(slug:$slug){
    id
    slug
    members(first:$first, after:$after, order:NAME){
      count
      pageInfo{ hasNextPage endCursor }
      edges{ node{
        id
        uuid
        role
        createdAt
        lastSeenAt
        security{ twoFactorEnabled passwordProtected }
        sso{ mode }
        user{ uuid name email bot machineUser }
      } }
    }
  }
}'

# Independent count of admins computed by the server. Must equal the count this
# script derives locally; if it does not, one of the two views is wrong and the
# census is not trustworthy.
# shellcheck disable=SC2016  # GraphQL $variables must stay literal, not shell-expanded
ADMIN_COUNT_QUERY='query($slug:ID!){
  organization(slug:$slug){ members(first:1, role:[ADMIN]){ count } }
}'

fetch_roster() {
  local after=null nodes='[]' declared='' page

  while :; do
    page="$(jq -n \
      --arg q "${ROSTER_QUERY}" \
      --arg slug "${BUILDKITE_ORG_SLUG}" \
      --argjson first "${PAGE_SIZE}" \
      --argjson after "${after}" \
      '{query:$q, variables:{slug:$slug, first:$first, after:$after}}' | gql_checked)"

    if [ "$(jq -r '.data.organization // "null"' <<<"${page}")" = "null" ]; then
      echo "organization '${BUILDKITE_ORG_SLUG}' not visible to this token" >&2
      return 3
    fi

    declared="$(jq -r '.data.organization.members.count' <<<"${page}")"
    nodes="$(jq -n --argjson acc "${nodes}" --argjson p "${page}" \
      '$acc + ($p.data.organization.members.edges | map(.node))')"

    [ "$(jq -r '.data.organization.members.pageInfo.hasNextPage' <<<"${page}")" = "true" ] || break
    after="$(jq -c '.data.organization.members.pageInfo.endCursor' <<<"${page}")"
  done

  # Truncation guard: a short read must fail loudly, never under-report admins.
  local collected
  collected="$(jq 'length' <<<"${nodes}")"
  if [ "${collected}" -ne "${declared}" ]; then
    echo "census incomplete: collected ${collected} of ${declared} members" >&2
    return 3
  fi

  printf '%s' "${nodes}"
}

census() {
  local nodes server_admins
  nodes="$(fetch_roster)" || return $?
  server_admins="$(jq -n --arg q "${ADMIN_COUNT_QUERY}" --arg slug "${BUILDKITE_ORG_SLUG}" \
    '{query:$q, variables:{slug:$slug}}' | gql_checked \
    | jq -r '.data.organization.members.count')"

  jq -n \
    --arg org "${BUILDKITE_ORG_SLUG}" \
    --argjson members "${nodes}" \
    --argjson server_admins "${server_admins}" \
    '
    ($members | map(select(.role == "ADMIN"))) as $admins
    | {
        organization: $org,
        total_members: ($members | length),
        admin_count: ($admins | length),
        server_reported_admin_count: $server_admins,
        counts_agree: (($admins | length) == $server_admins),
        admins: ($admins | map({
          member_id:  .id,
          member_uuid: .uuid,
          user_uuid: .user.uuid,
          name:  .user.name,
          email: .user.email,
          headless: (.user.bot or .user.machineUser),
          two_factor_enabled: (.security.twoFactorEnabled == true),
          sso_mode: .sso.mode,
          created_at: .createdAt,
          last_seen_at: .lastSeenAt
        })),
        headless_admins:      ($admins | map(select(.user.bot or .user.machineUser) | .user.email)),
        admins_without_2fa:   ($admins | map(select(.security.twoFactorEnabled != true) | .user.email)),
        admins_sso_optional:  ($admins | map(select(.sso.mode != "REQUIRED") | .user.email))
      }'
}

# CI gate. Non-zero exit on any violation, so this drops straight into a
# pipeline step. Every check is fail-closed: a census that cannot be trusted
# fails rather than reporting a comfortable zero.
assert_admins() {
  local report failures=0 admin_count agree

  report="$(census)" || return $?
  printf '%s\n' "${report}"

  admin_count="$(jq -r '.admin_count' <<<"${report}")"
  agree="$(jq -r '.counts_agree' <<<"${report}")"

  if [ "${agree}" != "true" ]; then
    echo "FAIL: local admin count $(jq -r '.admin_count' <<<"${report}") disagrees with server-reported $(jq -r '.server_reported_admin_count' <<<"${report}")" >&2
    failures=$((failures + 1))
  fi

  # Buildkite organizations always retain at least one admin, so zero means the
  # census failed rather than that the org is unusually well hardened.
  if [ "${admin_count}" -eq 0 ]; then
    echo "FAIL: census reported 0 administrators, which is not a valid organization state" >&2
    failures=$((failures + 1))
  fi

  if [ "${admin_count}" -gt "${MAX_ADMINS}" ]; then
    echo "FAIL: ${admin_count} organization administrators, limit is ${MAX_ADMINS}" >&2
    jq -r '.admins[] | "  admin: \(.email) headless=\(.headless) 2fa=\(.two_factor_enabled) last_seen=\(.last_seen_at)"' <<<"${report}" >&2
    failures=$((failures + 1))
  fi

  if [ "${REQUIRE_ADMIN_2FA}" = "true" ] && [ "$(jq '.admins_without_2fa | length' <<<"${report}")" -ne 0 ]; then
    echo "FAIL: administrators without two-factor authentication: $(jq -r '.admins_without_2fa | join(", ")' <<<"${report}")" >&2
    failures=$((failures + 1))
  fi

  if [ "${REQUIRE_ADMIN_SSO}" = "true" ] && [ "$(jq '.admins_sso_optional | length' <<<"${report}")" -ne 0 ]; then
    echo "FAIL: administrators whose SSO mode is not REQUIRED: $(jq -r '.admins_sso_optional | join(", ")' <<<"${report}")" >&2
    failures=$((failures + 1))
  fi

  # Advisory, not a gate: a headless admin may be legitimate automation, but it
  # should be a deliberate decision rather than something nobody noticed.
  if [ "$(jq '.headless_admins | length' <<<"${report}")" -ne 0 ]; then
    echo "WARN: bot or machine accounts hold organization admin: $(jq -r '.headless_admins | join(", ")' <<<"${report}")" >&2
  fi

  [ "${failures}" -eq 0 ] || return 1
}

# Write path. Guarded, because both mutations can strand an organization.
# Member ids come from the census above; a raw uuid or a REST id is rejected.
require_member_id() {
  local id="$1"
  case "${id}" in
    "${MEMBER_ID_PREFIX}"*) : ;;
    *)
      echo "refusing: '${id}' is not an OrganizationMember relay id." >&2
      echo "Use the member_id from '$0 census'. A bare uuid will not resolve, and" >&2
      echo "the id returned by REST /v2/organizations/{org}/members is the USER uuid." >&2
      return 2
      ;;
  esac
}

# Refuse to act on the calling identity or to empty the admin population.
preflight_target() {
  local id="$1" report caller_uuid target_user_uuid admin_count

  report="$(census)" || return $?

  # `viewer` resolves the identity behind BUILDKITE_TOKEN, which is how the
  # self-demotion guard below knows whose access it would be revoking.
  caller_uuid="$(jq -n --arg q 'query{viewer{user{uuid}}}' '{query:$q}' | gql_checked \
    | jq -r '.data.viewer.user.uuid')"

  target_user_uuid="$(jq -r --arg id "${id}" \
    'first(.admins[] | select(.member_id == $id) | .user_uuid) // ""' <<<"${report}")"
  admin_count="$(jq -r '.admin_count' <<<"${report}")"

  if [ -z "${target_user_uuid}" ]; then
    # TRAP 7(a): this is a scope statement, not a lookup failure. Both mutations
    # in this pack act on the admin population only.
    echo "refusing: ${id} is not currently an organization administrator" >&2
    echo "          this pack acts on administrators only — take member_id from" >&2
    echo "          'census', and remove an ordinary member with" >&2
    echo "          packs/buildkite/api/hth-buildkite-2.06-dormant-members.sh" >&2
    return 2
  fi

  if [ "${target_user_uuid}" = "${caller_uuid}" ]; then
    echo "refusing: ${id} is the calling identity (${caller_uuid}); self-demotion can lock you out" >&2
    return 2
  fi

  if [ "${admin_count}" -le 1 ]; then
    echo "refusing: ${id} is the last remaining administrator" >&2
    return 2
  fi

  echo "target member uuid ${target_user_uuid}, ${admin_count} admins before change" >&2
}

# Demote an organization administrator to MEMBER. Requires BUILDKITE_CONFIRM=demote.
demote_admin() {
  local id="$1"
  # Explicit `|| return` on every guard. Relying on `set -e` here would be
  # fail-open: errexit is suppressed for the whole call chain the moment anyone
  # writes `if demote_admin ...` or `demote_admin ... && ...`, and the mutation
  # would then run with its preflight quietly skipped.
  require_member_id "${id}" || return $?
  [ "${BUILDKITE_CONFIRM:-}" = "demote" ] || {
    echo "refusing: set BUILDKITE_CONFIRM=demote to authorise this mutation" >&2; return 2; }
  preflight_target "${id}" || return $?

  jq -n --arg id "${id}" '{
    query: "mutation($id:ID!){ organizationMemberUpdate(input:{id:$id, role: MEMBER}){
              organizationMember { id role user { email } } } }",
    variables: { id: $id }
  }' | gql_checked | jq '.data.organizationMemberUpdate.organizationMember'
}

# Remove an ADMIN's membership from the organization entirely (TRAP 7).
# Requires BUILDKITE_CONFIRM=remove AND HTH_SCIM_REVIEWED=1.
remove_member() {
  local id="$1"
  require_member_id "${id}" || return $?
  [ "${BUILDKITE_CONFIRM:-}" = "remove" ] || {
    echo "refusing: set BUILDKITE_CONFIRM=remove to authorise this mutation" >&2; return 2; }
  # TRAP 7(b). organizationMemberDelete is irreversible and no field exposes SCIM
  # management, so the operator asserts the check — the same gate the 2.6 pack
  # puts on the same mutation. Deletion is the one place this pack must not be a
  # guard weaker than its sibling.
  [ "${HTH_SCIM_REVIEWED:-}" = "1" ] || {
    echo "refusing: set HTH_SCIM_REVIEWED=1 to confirm you checked whether this" >&2
    echo "organization is SCIM-managed. If the IdP still asserts this user, this" >&2
    echo "deletion is reverted at the next sync and the access review evidences a" >&2
    echo "removal that did not hold. Deprovision in the IdP instead." >&2
    return 2; }
  preflight_target "${id}" || return $?

  jq -n --arg id "${id}" '{
    query: "mutation($id:ID!){ organizationMemberDelete(input:{id:$id}){
              deletedOrganizationMemberID user { email } } }",
    variables: { id: $id }
  }' | gql_checked | jq '.data.organizationMemberDelete'
}

2.4 Control Untrusted Input to Pipelines

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	16.1
NIST 800-53	SI-10, CM-7

Description

Constrain what a build is allowed to do when the code or configuration driving it comes from somewhere you do not fully control — forks, third-party plugins, or a pipeline.yml that an untrusted contributor can edit.

Rationale

Why This Matters:

A build triggered by a fork runs contributor-authored code on your agents with whatever credentials those agents hold, which turns an open-source contribution into a credential-harvesting opportunity
Plugins are third-party code executed inside your build; an unpinned plugin reference means whatever the plugin author publishes next runs in your pipeline without review
Command evaluation and interpolation let a pipeline.yml upload arbitrary steps mid-build, so a repository writer can escape the pipeline definition that was actually reviewed
Agent-side controls are the only ones a malicious pipeline.yml cannot switch off, which is why the defense has to live on the agent rather than in the pipeline file

Attack Prevented: Poisoned pipeline execution from fork builds, supply-chain compromise via mutable third-party plugins, secret exfiltration through injected build steps, agent takeover from untrusted contributions

ClickOps Implementation

Step 1: Close the Fork Path on Public Pipelines

For any pipeline with public visibility, disable builds triggered by forked repositories unless you have a specific reason to allow them and an agent pool with no production credentials to run them on.

Step 2: Constrain Plugins

Prefer plugins you host privately over public ones.
Pin every plugin reference to a specific version or commit — a floating reference is an unreviewed dependency.
On agents that should never load third-party code at all, run the agent with --no-plugins.

Step 3: Disable Command Evaluation Where It Is Not Needed

Configure agents to reject command evaluation and dynamic step uploads on pools that run untrusted code, so the reviewed pipeline definition is the only definition that executes.
Enable the agent’s reject-secrets guard so builds that try to surface secret-looking values are stopped rather than logged.

Step 4: Bound the Blast Radius

Set job time limits so a hijacked build cannot mine, scan, or exfiltrate indefinitely.
Put your enforcement in agent lifecycle hooks, which live on the agent host and cannot be overridden by pipeline.yml.

Code Implementation

Code Pack: Config

hth-buildkite-2.04-agent-untrusted-input.sh View source on GitHub ↗

# Emit or apply the hardened buildkite-agent.cfg for an agent pool that runs
# untrusted input, then prove the result actually enforces something.

die() { echo "FATAL: $*" >&2; exit 1; }

# T10: buildkite-agent.cfg is parsed by cliconfig/file.go, a godotenv derivative,
# not by a "split on the first =" reader. These helpers are a faithful port of
# parseLine (v3.137.0:86-148, byte-identical on main) so that this pack writes
# only configs the agent reads back unchanged, and audit only ever reports the
# value the agent will actually enforce.
_HTH_DQ='"'
_HTH_SQ="'"
_HTH_BS='\'

# Occurrences of a single character in a string, without spawning a process.
_hth_count() {
  local s="$1" c="$2" t
  t="${s//"${c}"/}"
  printf '%s' "$(( ${#s} - ${#t} ))"
}

# Given one raw config line, return the value the AGENT will hold for it.
# Empty output means "the agent gets nothing here", which is the fail-closed
# answer for every line parseLine would reject outright.
agent_parse_line() {
  local line="$1" seg rest more kept open out value nd ns
  # file.go:92-112 — strip comments, but keep a '#' inside a quoted segment.
  if [ "${line#*#}" != "${line}" ]; then
    rest="${line}"; kept=0; open=0; out=""
    while :; do
      if [ "${rest#*#}" != "${rest}" ]; then
        seg="${rest%%#*}"; rest="${rest#*#}"; more=1
      else
        seg="${rest}"; more=0
      fi
      nd="$(_hth_count "${seg}" "${_HTH_DQ}")"
      ns="$(_hth_count "${seg}" "${_HTH_SQ}")"
      if [ "${nd}" -eq 1 ] || [ "${ns}" -eq 1 ]; then
        if [ "${open}" -eq 1 ]; then
          open=0
          if [ "${kept}" -eq 0 ]; then out="${seg}"; else out="${out}#${seg}"; fi
          kept=$(( kept + 1 ))
        else
          open=1
        fi
      fi
      if [ "${kept}" -eq 0 ] || [ "${open}" -eq 1 ]; then
        if [ "${kept}" -eq 0 ]; then out="${seg}"; else out="${out}#${seg}"; fi
        kept=$(( kept + 1 ))
      fi
      [ "${more}" -eq 1 ] || break
    done
    line="${out}"
  fi
  # file.go:114-135 — '=' first; ':' only when the line has no '=' at all.
  case "${line}" in
    *=*) value="${line#*=}" ;;
    *:*) value="${line#*:}" ;;
    *)   return 0 ;;
  esac
  value="${value#"${value%%[![:space:]]*}"}"
  value="${value%"${value##*[![:space:]]}"}"
  # file.go:137-145 — exactly two quotes of one kind means the value is quoted:
  # strip every edge quote, then expand \" and \n.
  nd="$(_hth_count "${value}" "${_HTH_DQ}")"
  ns="$(_hth_count "${value}" "${_HTH_SQ}")"
  if [ "${nd}" -eq 2 ] || [ "${ns}" -eq 2 ]; then
    while :; do case "${value}" in [\"\']*) value="${value#?}" ;; *) break ;; esac; done
    while :; do case "${value}" in *[\"\']) value="${value%?}" ;; *) break ;; esac; done
    local _esc_dq="${_HTH_BS}${_HTH_DQ}" _esc_nl="${_HTH_BS}n" _nl
    _nl=$'\n'
    value="${value//"${_esc_dq}"/${_HTH_DQ}}"
    value="${value//"${_esc_nl}"/${_nl}}"
  fi
  printf '%s' "${value}"
}

# True when the pattern ends in a '$' that RE2 will read as an end-of-text
# anchor rather than as a literal dollar sign.
anchored_end() {
  local p="$1" body bs=0
  case "${p}" in *'$') ;; *) return 1 ;; esac
  body="${p%'$'}"
  while [ -n "${body}" ] && [ "${body%"${_HTH_BS}"}" != "${body}" ]; do
    body="${body%"${_HTH_BS}"}"
    bs=$(( bs + 1 ))
  done
  [ $(( bs % 2 )) -eq 0 ]
}

# T4: the agent does not anchor these for you, at either end. Refuse to ship a
# pattern that would match a substring or a prefix of a hostile URL or plugin
# source.
validate_regexes() {
  local label="$1" list="$2" pat
  local IFS=','
  for pat in ${list}; do
    [ -n "${pat}" ] || continue
    # T10: every value is written quoted, and the agent's parser cannot round-trip
    # a quoted value that itself contains a quote character.
    case "${pat}" in
      *'"'*|*"'"*) die "${label}: pattern '${pat}' contains a quote character. buildkite-agent.cfg values are written quoted (T10) and cliconfig/file.go cannot round-trip a nested quote — the agent would compile a different pattern than the one written. Remove it." ;;
    esac
    case "${pat}" in
      '^'*) ;;
      *) die "${label}: pattern '${pat}' is not anchored at the start. regexp.Compile + re.MatchString (run_job.go:267-279) is a substring search, so it would match anywhere in the value — '${pat}' would admit 'https://evil.example/${pat}'. Prefix it with '^'." ;;
    esac
    # T4: the end matters just as much. Without '$' the pattern matches a PREFIX
    # of the value, so '^git@github\.com:acme/app\.git' admits
    # 'git@github.com:acme/app.git.evil.example/x'. A trailing '$' preceded by an
    # ODD number of backslashes is an escaped literal dollar, not an anchor —
    # counting them is the difference between checking for an anchor and
    # checking for a character.
    anchored_end "${pat}" || die "${label}: pattern '${pat}' is not anchored at the end. MatchString matches a prefix, so it would admit '${pat}' followed by anything. Append an unescaped '\$' (and prefer a bounded character class over a trailing '.*' — see T4)."
    # grep -E exits 1 on "no match" (fine) and 2 on a malformed pattern. The
    # agent calls l.Fatalf on a pattern regexp.Compile rejects, so catching it
    # here is the difference between a config error and an agent restart loop.
    if printf '%s' "" | grep -Eq "${pat}" 2>/dev/null; then
      :
    else
      local grc=$?
      [ "${grc}" -le 1 ] || die "${label}: '${pat}' is not a valid regular expression."
    fi
  done
}

# T2: redacted-vars goes through path.Match, so regex metacharacters are matched
# literally and quietly protect nothing.
validate_globs() {
  local label="$1" list="$2" pat
  local IFS=','
  for pat in ${list}; do
    [ -n "${pat}" ] || continue
    case "${pat}" in
      *'"'*|*"'"*) die "${label}: '${pat}' contains a quote character, which cannot survive the quoted write this pack performs (T10)." ;;
    esac
    case "${pat}" in
      *'^'*|*'$'*|*'.*'*|*'\'*)
        die "${label}: '${pat}' looks like a regex. This field is matched with path.Match (globs); a regex here matches nothing and redacts nothing. Use '*_SUFFIX' form." ;;
    esac
  done
}

# The last line that sets ${key}, verbatim. `export ` is tolerated because
# parseLine strips that prefix (file.go:128); a config carrying it would
# otherwise be read by this pack and by the agent as two different files.
cfg_line() {
  [ -r "${AGENT_CFG}" ] || return 1
  sed -nE "/^[[:space:]]*(export[[:space:]]*)?$1[[:space:]]*=/p" "${AGENT_CFG}" | tail -1
}

# What the file APPEARS to say: the text after the first '=', with at most one
# layer of surrounding quotes removed. Only audit uses this, and only to compare
# it against read_cfg — a human reads this, the agent reads read_cfg (T10).
read_cfg_text() {
  local raw
  raw="$(cfg_line "$1")" || return 1
  case "${raw}" in *=*) raw="${raw#*=}" ;; *) raw="" ;; esac
  raw="${raw#"${raw%%[![:space:]]*}"}"
  raw="${raw%"${raw##*[![:space:]]}"}"
  raw="${raw%\"}"; raw="${raw#\"}"
  printf '%s' "${raw}"
}

# What the AGENT will hold. Every decision in this pack is made on this value and
# never on the file text, because an unquoted '#' makes the two differ (T10).
read_cfg() {
  local raw
  raw="$(cfg_line "$1")" || return 1
  agent_parse_line "${raw}"
}

# Delete-then-append rather than sed-substitute: these values are regexes and
# routinely contain the alternation pipe, which would terminate any sed
# replacement delimiter you pick. Only the key is ever interpolated into a
# pattern here; the value is only ever written by printf.
# ATOMIC CONFIG REPLACEMENT. buildkite-agent.cfg carries this host's registration
# token; a truncate-then-write (`cat "${tmp}" >"${AGENT_CFG}"`) interrupted by a
# signal, a full disk, or a set -e abort leaves a config the agent cannot parse,
# the agent then fails to start, and the host silently leaves the fleet. apply
# calls set_cfg once per emitted key, so that window was opened repeatedly in one
# run. Every mutation below stages a sibling file and rename(2)s it into place
# instead, so a reader sees the old config or the new one, never a partial one.
#   * the temp lives in the target's OWN directory — rename(2) is EXDEV across
#     filesystems and mv would fall back to a non-atomic copy
#   * `cp -p` seeds it from the incumbent so mode, ownership and times ride onto
#     the replacement inode; a bare mktemp would hand the agent mktemp's 0600
#   * AGENT_CFG is resolved through symlinks first, because this path is commonly
#     a link into a config-management tree and renaming over the link would
#     replace it with a regular file
#   * no .bak is written: a persistent second copy of the agent token on disk is
#     a worse trade than the failure mode the rename already removes
# The trap is what keeps that staged full copy of the config — token included —
# out of the directory when a run dies mid-mutation.
HTH_CFG_TMP=""
HTH_CFG_DST=""
hth_cfg_cleanup() {
  if [ -n "${HTH_CFG_TMP}" ]; then rm -f "${HTH_CFG_TMP}"; fi
  HTH_CFG_TMP=""
}
trap hth_cfg_cleanup EXIT
trap 'hth_cfg_cleanup; exit 130' INT
trap 'hth_cfg_cleanup; exit 143' TERM

# Real path of the config, following symlinks where the platform's readlink can.
hth_cfg_path() {
  if [ -L "${AGENT_CFG}" ]; then
    readlink -f "${AGENT_CFG}" 2>/dev/null || printf '%s\n' "${AGENT_CFG}"
  else
    printf '%s\n' "${AGENT_CFG}"
  fi
}

# Stage a writable copy beside the target: HTH_CFG_DST is the real path to read
# from and commit to, HTH_CFG_TMP is the staged file to write into.
# This assigns globals rather than printing a value on purpose. Written as
# `dst="$(hth_cfg_stage)"` the function would run in a SUBSHELL, the parent's
# HTH_CFG_TMP would stay empty, and the cleanup trap would never see — or remove
# — the full copy of the token-bearing config that mktemp just created.
hth_cfg_stage() {
  HTH_CFG_DST="$(hth_cfg_path)"
  HTH_CFG_TMP="$(mktemp "$(dirname "${HTH_CFG_DST}")/.hth-bk-cfg.XXXXXX")"
  cp -p "${HTH_CFG_DST}" "${HTH_CFG_TMP}"
}

# Atomic swap. After this returns there is no temp left for the trap to clean up.
hth_cfg_commit() {
  mv -f "${HTH_CFG_TMP}" "${HTH_CFG_DST}"
  HTH_CFG_TMP=""
}

# T10: the value is written QUOTED, and the exact line about to be written is
# fed back through the port of the agent's own parser first. A line the agent
# would read differently from what was intended is never written at all — a
# silently truncated allowlist regex is worse than a refusal, because the file,
# audit and the operator all then agree on a control the agent is not enforcing.
# The staged-and-renamed write above is preserved; only the line changes.
set_cfg() {
  local key="$1" value="$2" tmp dst line effective
  case "${value}" in
    *'"'*|*"'"*) die "${key}: value contains a quote character; cliconfig/file.go cannot round-trip it (T10)." ;;
  esac
  line="$(printf '%s="%s"' "${key}" "${value}")"
  effective="$(agent_parse_line "${line}")"
  [ "${effective}" = "${value}" ] || die "${key}: refusing to write. The agent would read '${effective}' from that line, not '${value}' (T10, cliconfig/file.go:86-148)."
  hth_cfg_stage; dst="${HTH_CFG_DST}"; tmp="${HTH_CFG_TMP}"
  grep -vE "^[[:space:]]*(export[[:space:]]*)?${key}[[:space:]]*=" "${dst}" >"${tmp}" || true
  printf '%s\n' "${line}" >>"${tmp}"
  hth_cfg_commit
}

# T3: union, never replace. Starts from whatever is already on disk if that is
# itself a superset of the built-ins, otherwise from the built-in nine.
merge_redacted_vars() {
  local current extra merged pat seen
  current="$(read_cfg redacted-vars || true)"
  [ -n "${current}" ] || current="${AGENT_DEFAULT_REDACTED_VARS}"
  merged="${AGENT_DEFAULT_REDACTED_VARS}"
  extra="${current},${HTH_EXTRA_REDACTED_VARS}"
  local IFS=','
  for pat in ${extra}; do
    [ -n "${pat}" ] || continue
    seen=0
    case ",${merged}," in *",${pat},"*) seen=1 ;; esac
    [ "${seen}" -eq 1 ] || merged="${merged},${pat}"
  done
  printf '%s' "${merged}"
}

# The config body. Two plugin postures, because T1 makes "allowlist" a different
# file from "none" rather than one extra line.
render_cfg() {
  local redacted plugin_lines env_lines
  redacted="$(merge_redacted_vars)"

  if [ "${HTH_ALLOWED_PLUGINS}" = "NONE" ]; then
    # Written explicitly even though no-command-eval would force it: an implicit
    # value cannot be audited, and a later reviewer must not have to know T1.
    plugin_lines='no-plugins="true"'
  else
    validate_regexes "allowed-plugins" "${HTH_ALLOWED_PLUGINS}"
    plugin_lines="no-plugins=\"false\"
allowed-plugins=\"${HTH_ALLOWED_PLUGINS}\""
  fi

  if [ "${HTH_ALLOWED_ENVIRONMENT_VARIABLES}" = "NONE" ]; then
    # T8/T9: the switch alone is valid and is the tightest posture available —
    # and it strips every pipeline-supplied variable on this host, so reaching it
    # requires writing NONE rather than leaving a variable unset. The list alone
    # is a startup Fatalf, which is why it is never emitted without the switch.
    env_lines='enable-environment-variable-allowlist="true"'
  else
    validate_regexes "allowed-environment-variables" "${HTH_ALLOWED_ENVIRONMENT_VARIABLES}"
    env_lines="enable-environment-variable-allowlist=\"true\"
allowed-environment-variables=\"${HTH_ALLOWED_ENVIRONMENT_VARIABLES}\""
  fi

  validate_regexes "allowed-repositories" "${HTH_ALLOWED_REPOSITORIES}"
  validate_globs "redacted-vars" "${redacted}"

  cat <<CFGEOF
# --- HTH control 2.4: untrusted input ----------------------------------------
# Every value is QUOTED. buildkite-agent.cfg is parsed by a godotenv derivative
# that discards everything after an unquoted '#', which silently truncates any
# pinned plugin pattern ("source#version") and un-anchors it (T10). Keep the
# quotes if you hand-edit this block.
# Only the reviewed definition executes: the pipeline's own command string is
# refused and checkout-override-mode is forced to 'strict'.
no-command-eval="true"
# Repository hooks are attacker-authored on an untrusted branch. See T6.
no-local-hooks="true"
${plugin_lines}
# Anchored at BOTH ends. MatchString is a substring search, so a pattern without
# '^' matches inside a hostile URL and one without '$' matches a prefix of it (T4).
allowed-repositories="${HTH_ALLOWED_REPOSITORIES}"
${env_lines}
# Built-in nine plus local additions; globs, not regexes; 6-byte floor (T2, T3).
redacted-vars="${redacted}"
# --- end HTH control 2.4 -----------------------------------------------------
CFGEOF
}

# Print the block for baking into an image or a config-management template.
emit_cfg() { render_cfg; }

# Idempotent in-place application against an existing config file.
apply_cfg() {
  [ -w "${AGENT_CFG}" ] || die "cannot write ${AGENT_CFG} (run as root)."
  local line key value rendered
  # Render FIRST, into a variable, and only then loop.
  # `done <<<"$(render_cfg)"` is fail-OPEN: die's `exit 1` inside the command
  # substitution kills only that subshell, `set -e` does not propagate a failed
  # substitution used as a redirection word, so the loop reads an empty string,
  # the function runs to completion and apply exits 0 having written NOTHING.
  # An operator provisioning a host then gets "Applied." and exit 0 on an agent
  # with no control on it — the exact "believe 2.4 is implemented when it is
  # not" failure T1 exists to prevent. A plain assignment DOES propagate, and
  # the explicit `|| die` makes it independent of how this function is called.
  rendered="$(render_cfg)" || die "config rendering failed; NOTHING was written to ${AGENT_CFG}. Fix the policy inputs above and re-run."
  [ -n "${rendered}" ] || die "config rendering produced no output; NOTHING was written to ${AGENT_CFG}."
  while IFS= read -r line; do
    case "${line}" in ''|'#'*) continue ;; esac
    key="${line%%=*}"
    # The rendered value is quoted (T10); recover the intended value with the
    # same parser the agent uses, then let set_cfg re-quote and re-verify it.
    value="$(agent_parse_line "${line}")"
    set_cfg "${key}" "${value}"
  done <<<"${rendered}"
  # A key we deliberately never write: reject-secrets. It is a pipeline-upload
  # flag and has no meaning in this file (T7). If a previous attempt put it
  # here, remove it rather than leaving a line that implies a control.
  if grep -qE '^[[:space:]]*reject-secrets[[:space:]]*=' "${AGENT_CFG}"; then
    local tmp dst
    hth_cfg_stage; dst="${HTH_CFG_DST}"; tmp="${HTH_CFG_TMP}"
    grep -vE '^[[:space:]]*reject-secrets[[:space:]]*=' "${dst}" >"${tmp}"
    hth_cfg_commit
    echo "removed inert 'reject-secrets' key from ${AGENT_CFG} (see T7)."
  fi
  echo "Applied. Restart buildkite-agent, then run: $0 audit"
}

# Fail closed on every combination that looks configured but enforces nothing.
audit_cfg() {
  local rc=0 eval_off hooks_off noplugins allowplugins repos envswitch envlist redacted pat
  [ -r "${AGENT_CFG}" ] || die "cannot read ${AGENT_CFG}"

  eval_off="$(read_cfg no-command-eval || true)"
  hooks_off="$(read_cfg no-local-hooks || true)"
  noplugins="$(read_cfg no-plugins || true)"
  allowplugins="$(read_cfg allowed-plugins || true)"
  repos="$(read_cfg allowed-repositories || true)"
  envswitch="$(read_cfg enable-environment-variable-allowlist || true)"
  envlist="$(read_cfg allowed-environment-variables || true)"
  redacted="$(read_cfg redacted-vars || true)"

  # Every value below is the AGENT's view of the file, not the file's text (T10).
  echo "config file                           : ${AGENT_CFG}"
  echo "no-command-eval                       : ${eval_off:-<unset> (default false)}"
  echo "no-local-hooks                        : ${hooks_off:-<unset> (default false)}"
  echo "no-plugins                            : ${noplugins:-<unset>}"
  echo "allowed-plugins                       : ${allowplugins:-<unset>}"
  echo "allowed-repositories                  : ${repos:-<unset>}"
  echo "enable-environment-variable-allowlist : ${envswitch:-<unset> (default false)}"
  echo "redacted-vars                         : ${redacted:-<unset> (built-in nine)}"

  # T10, and it must run BEFORE every other check, because every other check
  # reads read_cfg — the agent's view — and would otherwise report a healthy
  # config without ever mentioning that the file says something else. A sed
  # reader cannot see this class of defect: it prints the pinned, anchored
  # pattern the operator wrote while the agent compiled the truncated one.
  local ktext keff
  for pat in no-command-eval no-local-hooks no-plugins allowed-plugins \
             allowed-repositories enable-environment-variable-allowlist \
             allowed-environment-variables redacted-vars; do
    ktext="$(read_cfg_text "${pat}" || true)"
    keff="$(read_cfg "${pat}" || true)"
    [ -n "${ktext}" ] || continue
    [ "${ktext}" != "${keff}" ] || continue
    echo "FAIL: ${pat} is written as '${ktext}' but the agent will read '${keff}' (T10)." >&2
    echo "      cliconfig/file.go parseLine drops everything after an unquoted '#', so a pinned" >&2
    echo "      plugin pattern loses its version AND its trailing '\$' anchor and degrades to a" >&2
    echo "      prefix match. Quote the value, or re-run '$0 apply' — this pack now quotes on write." >&2
    rc=1
  done

  [ "${eval_off}" = "true" ] || { echo "FAIL: no-command-eval is not true. A pipeline.yml can run arbitrary commands here." >&2; rc=1; }
  [ "${hooks_off}" = "true" ] || { echo "FAIL: no-local-hooks is not true. .buildkite/hooks/* from the checkout executes, and can undo every hook-based control (T6)." >&2; rc=1; }

  # T1, the headline check.
  if [ -n "${allowplugins}" ] && [ -z "${noplugins}" ] \
     && { [ "${eval_off}" = "true" ] || [ "${hooks_off}" = "true" ]; }; then
    echo "FAIL: allowed-plugins is set but the 'no-plugins' key is ABSENT while no-command-eval/no-local-hooks is on." >&2
    echo "      agent_start.go forces no-plugins=true in exactly this case: plugins are OFF and your allowlist is dead configuration." >&2
    echo "      Write 'no-plugins=false' explicitly to enable the allowlist, or drop allowed-plugins and write 'no-plugins=true'." >&2
    rc=1
  fi
  if [ "${noplugins}" = "false" ] && [ -z "${allowplugins}" ]; then
    echo "FAIL: no-plugins=false with no allowed-plugins — every third-party plugin on the internet may execute here." >&2
    rc=1
  fi

  if [ -z "${repos}" ]; then
    echo "FAIL: allowed-repositories unset. The agent will clone any repository a job names." >&2
    rc=1
  else
    local IFS=','
    for pat in ${repos} ${allowplugins} ${envlist}; do
      [ -n "${pat}" ] || continue
      case "${pat}" in '^'*) ;; *) echo "FAIL: allowlist pattern '${pat}' is unanchored (T4)." >&2; rc=1 ;; esac
    done
    unset IFS
  fi

  # T8: this ordering is a startup crash, so catch it before the restart.
  if [ -n "${envlist}" ] && [ "${envswitch}" != "true" ]; then
    echo "FAIL: allowed-environment-variables is set without enable-environment-variable-allowlist." >&2
    echo "      The agent calls l.Fatalf on this and will not start." >&2
    rc=1
  fi

  # T3: prove nothing was dropped from the built-in nine.
  if [ -n "${redacted}" ]; then
    local IFS=','
    for pat in ${AGENT_DEFAULT_REDACTED_VARS}; do
      case ",${redacted}," in
        *",${pat},"*) ;;
        *) echo "FAIL: redacted-vars overrides the built-in list and DROPPED '${pat}'." >&2; rc=1 ;;
      esac
    done
    unset IFS
    validate_globs "redacted-vars" "${redacted}"
  fi

  # T7: this key can only ever be cargo cult here.
  if grep -qE '^[[:space:]]*reject-secrets[[:space:]]*=' "${AGENT_CFG}"; then
    echo "FAIL: 'reject-secrets' in buildkite-agent.cfg is inert — it is a 'pipeline upload' flag, not an 'agent start' one (T7)." >&2
    rc=1
  fi

  for h in pre-bootstrap pre-command; do
    if [ ! -x "${HOOKS_PATH}/${h}" ]; then
      echo "FAIL: ${HOOKS_PATH}/${h} missing or not executable. Run: $0 install-hooks" >&2
      rc=1
    fi
  done

  [ "${rc}" -eq 0 ] && echo "PASS: agent enforces control 2.4."
  return "${rc}"
}

# Job admission BEFORE checkout. exit 0 permits the job; ANY non-zero rejects it
# and no untrusted code has touched the disk yet (job_runner.go:940-950).
# This is the only place in Buildkite where a job can be refused on its own
# inputs, and the only place where BUILDKITE_REPO is still the server-supplied
# value — the docs mark it modifiable by environment/pre-checkout hooks, both of
# which run later.

# printf %q gives us shell-safe embedding of policy values into the generated
# hook without a second config file to protect.
shq() { printf '%q' "$1"; }

write_pre_bootstrap() {
  local dest="${HOOKS_PATH}/pre-bootstrap" tmp
  [ -d "${HOOKS_PATH}" ] || die "hooks path ${HOOKS_PATH} does not exist."
  tmp="${dest}.hth.tmp"
  {
    printf '#!/usr/bin/env bash\n'
    printf '# Generated by HTH buildkite 2.4 pack. Edit the pack, not this file.\n'
    printf 'HTH_ALLOWED_REPOSITORIES=%s\n' "$(shq "${HTH_ALLOWED_REPOSITORIES}")"
    printf 'HTH_ALLOWED_PLUGINS=%s\n'      "$(shq "${HTH_ALLOWED_PLUGINS}")"
    printf 'HTH_ALLOW_FORK_BUILDS=%s\n'    "$(shq "${HTH_ALLOW_FORK_BUILDS}")"
    printf 'HTH_REQUIRE_PLUGIN_SHA=%s\n'   "$(shq "${HTH_REQUIRE_PLUGIN_SHA}")"
    cat <<'PREBOOTSTRAP'

# No `set -e`: every exit path here is deliberate, and an unexpected non-zero
# from a helper must not be mistaken for a considered rejection.
set -uo pipefail

reject() { echo "pre-bootstrap: REJECTED job ${BUILDKITE_JOB_ID:-?}: $*" >&2; exit 1; }
permit() { echo "pre-bootstrap: admitted job ${BUILDKITE_JOB_ID:-?}: $*"; exit 0; }

# Fail closed on a broken agent host. A missing jq must not mean "allow".
command -v jq >/dev/null 2>&1 || reject "jq is not installed on this agent host"

# T5: the job's variables are NOT in this hook's environment. They are in a file.
[ -n "${BUILDKITE_ENV_JSON_FILE:-}" ] || reject "BUILDKITE_ENV_JSON_FILE is unset"
[ -r "${BUILDKITE_ENV_JSON_FILE}" ]   || reject "cannot read ${BUILDKITE_ENV_JSON_FILE}"

# READ, never source. The sibling shell-format file is Go %q-quoted, not shell
# escaped, and its values are attacker-supplied (job_runner.go:565-573).
jget() { jq -r --arg k "$1" '.[$k] // ""' "${BUILDKITE_ENV_JSON_FILE}"; }

repo="$(jget BUILDKITE_REPO)"
pr_repo="$(jget BUILDKITE_PULL_REQUEST_REPO)"
pr_num="$(jget BUILDKITE_PULL_REQUEST)"
plugins_json="$(jget BUILDKITE_PLUGINS)"
pipeline="$(jget BUILDKITE_PIPELINE_SLUG)"

# grep -E is POSIX ERE while the agent uses Go RE2. Keep policy patterns in the
# common subset (anchors, character classes, ., *, +, ?, alternation) so the
# admission decision here and the agent's own allowlist cannot disagree.
#
# The one place they WOULD disagree is a newline: RE2's `^` and `$` anchor the
# whole text, grep's anchor each LINE, so a value spliced together as
# "git@github.com:evil/x\ngit@github.com:acme/ok.git" satisfies grep on its
# second line while the agent's pattern matches none of it. Refuse such a value
# outright rather than let this hook be the more permissive of the two.
HTH_NL='
'
matches_any() {
  local value="$1" list="$2" pat
  case "${value}" in
    *"${HTH_NL}"*) return 1 ;;
  esac
  local IFS=','
  for pat in ${list}; do
    [ -n "${pat}" ] || continue
    if printf '%s' "${value}" | grep -Eq "${pat}"; then return 0; fi
  done
  return 1
}

[ -n "${repo}" ] || reject "BUILDKITE_REPO is empty; cannot evaluate repository policy"
matches_any "${repo}" "${HTH_ALLOWED_REPOSITORIES}" \
  || reject "repository ${repo} is not in the allowlist"

# T11: the two repository variables arrive in DIFFERENT URL FORMS for the SAME
# repository, so `[ "$pr_repo" != "$repo" ]` calls every internal pull request a
# fork. Buildkite's own examples are BUILDKITE_REPO
# "git@github.com:acme-inc/my-project.git" against BUILDKITE_PULL_REQUEST_REPO
# "git://github.com/acme-inc/my-project.git". Compare identity, not text: the
# scheme, any userinfo, a port and a trailing ".git" carry none of it.
norm_repo() {
  local u="${1}" host rest sep port had_scheme=0
  u="${u%/}"
  # Whether the ORIGINAL, untouched input carried an explicit scheme matters
  # downstream: a scheme is the only thing that can legitimately put a port
  # after a colon. Record it before this same case stripping manufactures a
  # colon-shaped remainder that looks identical to genuine SCP syntax.
  case "${u}" in *://*) had_scheme=1; u="${u#*://}" ;; esac  # git:// https:// ssh:// http://
  case "${u%%[/:]*}" in *@*) u="${u#*@}" ;; esac     # git@ / user@ / token@, authority only
  u="${u%.git}"
  host="${u%%[:/]*}"
  rest="${u#"${host}"}"
  sep="${rest:0:1}"
  rest="${rest:1}"
  # A colon here is a real port ONLY when the input had an explicit scheme
  # (e.g. "ssh://host:2222/org/repo" -> after stripping "ssh://", "host:2222/org/repo").
  # Git's scp-like syntax ("user@host:path" or "host:path", no scheme) has NO
  # port field at all — per git-clone(1), that form "should not be used with a
  # port number, as that will be interpreted as part of the path" — so a
  # digits-only leading segment after an SCP colon is ALWAYS part of the
  # repository's path, never a port, and must never be stripped. Applying the
  # port heuristic to a genuine SCP colon is exactly what let a repo whose
  # namespace happens to start with digits normalize to two different
  # identities depending on which form Buildkite happened to hand us
  # (git@host:1234/x.git kept "1234" from https://host/1234/x.git, or vice
  # versa) — the same misclassify-an-internal-PR-as-a-fork failure T11 exists
  # to close, just reached through the URL-form side instead of the compare.
  if [ "${sep}" = ":" ] && [ "${had_scheme}" = "1" ]; then
    port="${rest%%/*}"
    case "${port}" in
      ""|*[!0-9]*) : ;;
      *) [ "${port}" = "${rest}" ] || rest="${rest#*/}" ;;
    esac
  fi
  rest="${rest#/}"
  printf '%s/%s' "$(printf '%s' "${host}" | tr 'A-Z' 'a-z')" "${rest}"
}

# Fork detection. BUILDKITE_PULL_REQUEST_REPO is "" when the build is not a pull
# request, and holds the SOURCE repository's URL when it is. A value that is not
# the same repository as BUILDKITE_REPO means the code is coming from a
# repository you do not own.
if [ -n "${pr_repo}" ] && [ "$(norm_repo "${pr_repo}")" != "$(norm_repo "${repo}")" ]; then
  [ "${HTH_ALLOW_FORK_BUILDS}" = "true" ] \
    || reject "fork build (PR #${pr_num} from ${pr_repo}) — forks run contributor code on this agent's credentials"
  # The allowlist is matched against the RAW value, because that is the form the
  # agent's own allowed-repositories sees. Buildkite hands fork URLs as
  # "git://host/org/repo.git", so an allowlist written only as "^git@host:org/…"
  # rejects every fork even with fork builds enabled — add a "^git://…"
  # alternate when HTH_ALLOW_FORK_BUILDS is true.
  matches_any "${pr_repo}" "${HTH_ALLOWED_REPOSITORIES}" \
    || reject "fork ${pr_repo} is not in the allowlist (fork URLs arrive in git:// form; HTH_ALLOWED_REPOSITORIES needs a '^git://' alternate)"
fi

# Plugins. BUILDKITE_PLUGINS is a JSON array of single-key objects whose key is
# the plugin reference ("source#ref"); the agent parses it the same way
# (job_runner.go:888-897, []map[string]json.RawMessage).
if [ -n "${plugins_json}" ] && [ "${plugins_json}" != "null" ]; then
  if [ "${HTH_ALLOWED_PLUGINS}" = "NONE" ]; then
    reject "plugins are forbidden on this agent but the step declares some"
  fi
  plugin_refs="$(printf '%s' "${plugins_json}" | jq -r '
      def refs: if type == "string" then . elif type == "object" then keys_unsorted[] else empty end;
      if type == "array" then (.[] | refs) elif type == "object" then keys_unsorted[] else empty end
    ' 2>/dev/null)" || reject "BUILDKITE_PLUGINS is not parseable JSON"
  [ -n "${plugin_refs}" ] || reject "BUILDKITE_PLUGINS is set but no plugin reference could be extracted"

  while IFS= read -r ref; do
    [ -n "${ref}" ] || continue
    matches_any "${ref}" "${HTH_ALLOWED_PLUGINS}" \
      || reject "plugin ${ref} is not in the allowlist"

    # Guide step 2.2: pin every plugin reference. A floating ref is an
    # unreviewed dependency that the plugin author can change under you.
    case "${ref}" in
      *'#'*) pin="${ref##*#}" ;;
      *)     pin="" ;;
    esac
    [ -n "${pin}" ] || reject "plugin ${ref} is unpinned (no #version or #commit)"
    case "${pin}" in
      main|master|HEAD|latest|stable)
        reject "plugin ${ref} is pinned to the moving ref '${pin}'" ;;
    esac
    if [ "${HTH_REQUIRE_PLUGIN_SHA}" = "true" ]; then
      printf '%s' "${pin}" | grep -Eq '^[0-9a-f]{40}$' \
        || reject "plugin ${ref} is not pinned to a 40-hex commit (tags are mutable)"
    fi
  done <<EOF
${plugin_refs}
EOF
fi

permit "pipeline=${pipeline} repo=${repo}"
PREBOOTSTRAP
  } >"${tmp}"
  chmod 0755 "${tmp}"
  mv "${tmp}" "${dest}"
  echo "wrote ${dest}"
}

# Pin the pipeline-upload secret guard at the last global hook before the
# command runs (T6), correctly on both agent majors (T7).

write_pre_command() {
  local dest="${HOOKS_PATH}/pre-command" tmp
  [ -d "${HOOKS_PATH}" ] || die "hooks path ${HOOKS_PATH} does not exist."
  tmp="${dest}.hth.tmp"
  cat >"${tmp}" <<'PRECOMMAND'
#!/usr/bin/env bash
# Generated by HTH buildkite 2.4 pack. Edit the pack, not this file.
set -uo pipefail

# T7: v3 reads REJECT_SECRETS (default off, must be turned ON).
#     v4 removed it and reads ALLOW_SECRETS (default off, must stay UNSET).
# Each major ignores the other's variable, so doing both is correct everywhere
# and needs no version detection to be safe.
export BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true
unset BUILDKITE_AGENT_PIPELINE_UPLOAD_ALLOW_SECRETS

# T6: runtime ratchet. The executor re-reads this before each local hook and
# only ever uses it to DISABLE, so a pipeline cannot flip it back. This is a
# belt: local post-checkout hooks already ran, which is why no-local-hooks=true
# in buildkite-agent.cfg remains the actual control.
export BUILDKITE_NO_LOCAL_HOOKS=true

# Informational only — the exports above are already correct on both majors.
if command -v buildkite-agent >/dev/null 2>&1; then
  agent_version="$(buildkite-agent --version 2>/dev/null | grep -Eo '[0-9]+\.[0-9]+\.[0-9]+[^ ,]*' | head -1)"
  case "${agent_version%%.*}" in
    3) echo "pre-command: agent v${agent_version}: pipeline uploads containing interpolated secrets will be rejected (--reject-secrets)." ;;
    4|5|6|7|8|9) echo "pre-command: agent v${agent_version}: secret rejection is the default; --allow-secrets is cleared." ;;
    *) echo "pre-command: could not determine agent major version; both secret-guard variables set defensively." >&2 ;;
  esac
fi
PRECOMMAND
  chmod 0755 "${tmp}"
  mv "${tmp}" "${dest}"
  echo "wrote ${dest}"
}

install_hooks() {
  validate_regexes "allowed-repositories" "${HTH_ALLOWED_REPOSITORIES}"
  [ "${HTH_ALLOWED_PLUGINS}" = "NONE" ] \
    || validate_regexes "allowed-plugins" "${HTH_ALLOWED_PLUGINS}"
  # T11: fork URLs arrive as git://host/org/repo.git. An allowlist that only
  # spells the git@ or https:// form admits no fork at all, so enabling fork
  # builds without a git:// alternate produces a control that looks configured
  # and rejects every pull request from outside the org.
  if [ "${HTH_ALLOW_FORK_BUILDS}" = "true" ]; then
    case ",${HTH_ALLOWED_REPOSITORIES}," in
      *',^git://'*) : ;;
      *) echo "WARNING: HTH_ALLOW_FORK_BUILDS=true but no '^git://' pattern is in HTH_ALLOWED_REPOSITORIES. Buildkite reports BUILDKITE_PULL_REQUEST_REPO in git:// form (T11), so every fork build will be rejected by the allowlist." >&2 ;;
    esac
  fi
  write_pre_bootstrap
  write_pre_command
  echo "Hooks installed under ${HOOKS_PATH}. Restart buildkite-agent so the"
  echo "pre-bootstrap hook is picked up, then run: $0 audit"
}

Source: Buildkite security controls

2.5 Manage API Access Token Hygiene

Profile Level: L1 (Crawl)

Framework	Control
CIS Controls	5.4
NIST 800-53	IA-5, AC-6

Description

Govern Buildkite API access tokens — the user-scoped REST and GraphQL credentials — as a separate credential class from the agent tokens covered in 3.1.

Rationale

Why This Matters:

API access tokens and agent tokens are routinely conflated, and rotating one while forgetting the other leaves half the credential surface untouched
An API token carries the permissions of the user who created it, so a broad token from an admin is an admin credential sitting in a script
Tokens without expiry accumulate silently in CI systems, laptops, and automation until nobody knows which ones are still live
GraphQL is expressive enough that a token with unnecessary scope can read far more of the organization than the integration it was created for ever needed
Every member can mint tokens by default, so the inventory an administrator is asked to govern is one that anyone in the organization can add to at any time without review

Attack Prevented: Standing API access from leaked tokens, privilege inheritance from over-scoped admin tokens, undetected token reuse, data harvesting via over-broad GraphQL access, unreviewed token sprawl from non-administrator accounts

ClickOps Implementation

Step 1: Restrict Who Can Mint Tokens

Set restrict_user_api_token_creation to true so that, in Buildkite’s words, “only organization administrators can create API access tokens.” Everything below governs an inventory; this is the only setting that governs how that inventory grows, and without it every member can add to it without review.
Unlike the IP allowlist and the inactive-token revocation clock, this field is not plan-gated — the features map returned by the API settings resource lists only api_ip_allow_list and inactive_api_token_revocation, so restrict_user_api_token_creation is available on every plan. It is the one organization-wide token control a non-Enterprise organization can actually turn on.
Restricting creation does not revoke what already exists. Run the inventory in Step 5 afterwards, because tokens minted by members before the restriction stay valid.

Step 2: Scope and Time-Bound Every Token

Grant each token the narrowest scope set that its integration actually uses — start from nothing and add scopes until the integration works, rather than trimming from full access.
Give tokens an expiry and automate their rotation, so expiry is a scheduled event rather than an outage.

Step 3: Restrict Where Tokens Can Be Used

Apply an IP range restriction to each token so a stolen token is unusable from outside your network egress.

Step 4: Narrow GraphQL Exposure

Where an integration only needs a fixed set of queries, use Portals to expose those specific operations instead of handing out a general GraphQL token.
A portal is not automatically a downgrade in privilege. Portal tokens carry administrator-level permissions within the operations the portal exposes, and they are long-lived. A portal whose query is written loosely is an admin credential with a friendlier name — scope the query itself, and set the portal’s IP allowlist.

Step 5: Monitor Token Use

Review token activity in the audit log (see 4.1) and revoke tokens that stop appearing — an unused token is pure standing risk.

Step 6: Automate Revocation of Inactive Tokens

Set the organization’s inactive token revocation period so Buildkite revokes tokens that go unused for a chosen window (30, 60, 90, 180 or 365 days). This turns Step 5 from a recurring human task into a platform guarantee, and it is the single highest-leverage organization-wide setting on this control.
The setting is Enterprise-gated. The current value is readable on any plan, so you can at least detect that it is unset.

Code Implementation

Code Pack: Terraform

hth-buildkite-2.05-portals.tf View source on GitHub ↗

# One portal per machine-to-machine operation. Each query should be the smallest
# document that satisfies exactly one integration — never a general-purpose
# document that several callers share, because the token is admin-privileged.
resource "buildkite_portal" "scoped" {
  for_each = var.portals

  name        = each.key
  slug        = each.value.slug
  description = each.value.description

  # The stored GraphQL document. This IS the permission boundary — nothing the
  # document does not name can be executed with this portal's token.
  query = each.value.query

  # The attribute is a space-delimited STRING of CIDRs, not a list. An empty
  # string leaves the portal callable from anywhere, which for an admin-level
  # token means anywhere on the internet with the bearer value.
  allowed_ip_addresses = join(" ", each.value.allowed_cidrs)

  # Leave false unless members genuinely need to invoke this portal as
  # themselves. See TRAP 2 — false is the smaller surface, not the mitigation.
  user_invokable = each.value.user_invokable

  lifecycle {
    # TRAP 5. Destroying a portal is unrecoverable, and the two ways to trigger
    # it do not look destructive in a diff:
    #   * renaming a key in var.portals changes the RESOURCE ADDRESS, which is a
    #     destroy-then-create no matter what the provider marks force-new — and
    #     because name = each.key, renaming the portal and destroying it are the
    #     same edit; and
    #   * dropping an entry from the map destroys it outright.
    # TRAP 3 is the consequence: `token` is returned once, never read back, so
    # the replacement portal issues a NEW token and every caller holding the old
    # one is broken until it is redistributed. This is a long-lived,
    # admin-privileged credential — the same reason 3.11 pins its registries.
    # To retire a portal deliberately, delete this line in the same commit that
    # removes the map entry, so the destroy is a reviewed decision rather than a
    # side effect of a tfvars rename.
    prevent_destroy = true

    # A portal that can WRITE and can be called from any address is the worst
    # combination this resource can express: an unauthenticated-by-network,
    # non-expiring, admin-level mutation endpoint. Refuse to plan it.
    precondition {
      condition = (
        length(regexall("(?im)(^|[};])[[:space:]]*mutation[[:space:]{(]", each.value.query)) == 0
        || length(each.value.allowed_cidrs) > 0
      )
      error_message = format(
        "Portal '%s' executes a mutation but declares no allowed_cidrs. Its token is long-lived and carries administrator-level permissions, so an unrestricted mutation portal is a standing admin credential reachable from any address. Add the egress CIDRs of the caller, or convert the portal to a read-only query.",
        each.key,
      )
    }
  }
}

output "portal_endpoints" {
  description = "Invocation URLs for the declared portals. Safe to share; the token is not."
  value = {
    for k, p in buildkite_portal.scoped :
    k => "https://portal.buildkite.com/organizations/${var.buildkite_organization}/portals/${p.slug}"
  }
}

output "portal_tokens" {
  description = "Long-lived, admin-privileged service tokens — returned on creation only. Move these straight into a secret store; they cannot be read back from Buildkite."
  sensitive   = true
  value       = { for k, p in buildkite_portal.scoped : k => p.token }
}

# Every portal in the organization, including ones created by hand in the
# console. This is the drift surface that matters for 2.5: a portal nobody
# declared is an admin-privileged credential nobody is rotating or reviewing.
data "buildkite_portals" "all" {}

locals {
  # TRAP 5, found by running this pack rather than reading the schema: on an
  # organization with zero portals the data source returns portals = NULL, not
  # an empty list, and every `for` expression over it aborts the plan with
  # "Iteration over null value". Normalise once, here, and iterate the local.
  discovered_portals = data.buildkite_portals.all.portals == null ? [] : data.buildkite_portals.all.portals

  declared_portal_slugs = toset([for k, v in var.portals : v.slug])

  undeclared_portal_slugs = setsubtract(
    toset([for p in local.discovered_portals : p.slug]),
    local.declared_portal_slugs,
  )

  unrestricted_portal_slugs = [
    for p in local.discovered_portals : p.slug
    if trimspace(coalesce(p.allowed_ip_addresses, "")) == ""
  ]
}

check "no_undeclared_portals" {
  assert {
    condition = length(local.undeclared_portal_slugs) == 0
    error_message = format(
      "Portals exist in this organization that are not declared in var.portals: %s. Each one holds a long-lived, administrator-level service token. Import it into this pack or delete it in Organization Settings > Integrations > Portals.",
      join(", ", local.undeclared_portal_slugs),
    )
  }
}

# Unrestricted portals, declared or not. Reported rather than blocked, because a
# read-only portal behind a caller you cannot pin to fixed egress is a judgement
# call — but it should never be a silent one.
check "portals_are_ip_restricted" {
  assert {
    condition = length(local.unrestricted_portal_slugs) == 0
    error_message = format(
      "Portals callable from any IP address: %s. Buildkite allows all addresses when the allowlist is unset, and portal tokens carry administrator-level permissions.",
      join(", ", local.unrestricted_portal_slugs),
    )
  }
}

output "portal_inventory" {
  description = "Audit view of every portal in the organization — including console-created ones this pack does not manage."
  value = [
    for p in local.discovered_portals : {
      slug              = p.slug
      name              = p.name
      declared_here     = contains(local.declared_portal_slugs, p.slug)
      ip_restricted     = trimspace(coalesce(p.allowed_ip_addresses, "")) != ""
      user_invokable    = p.user_invokable
      performs_mutation = length(regexall("(?im)(^|[};])[[:space:]]*mutation[[:space:]{(]", coalesce(p.query, ""))) > 0
      created_by        = try(p.created_by.email, null)
      created_at        = p.created_at
    }
  ]
}

Code Pack: API Script

hth-buildkite-2.05-token-hygiene.sh View source on GitHub ↗

# Guide Step 1. `restrict_user_api_token_creation` = "only organization
# administrators can create API access tokens" (vendor wording). It is the only
# organization-wide token setting here that is NOT plan-gated: the `features`
# map this resource returns enumerates the gated ones — api_ip_allow_list and
# inactive_api_token_revocation — and this field is not among them. So a
# non-Enterprise organization that cannot arm the inactivity clock can still
# close the tap.
rest_api_settings_get() {
  curl -sS --fail-with-body \
    -H "Authorization: Bearer ${BUILDKITE_TOKEN}" \
    "${REST}/organizations/${BUILDKITE_ORG_SLUG}/api-settings"
}

# Read-only. Reports the setting plus the plan-gate map, so an operator can tell
# "configured off" apart from "not available on this plan" — the distinction the
# `features` map exists to make.
restrict_token_creation_status() {
  rest_api_settings_get | jq '{
    restrict_user_api_token_creation,
    compliant: (.restrict_user_api_token_creation == true),
    ip_allowlist_configured: (.allowed_ip_addresses != null),
    revoke_inactive_tokens_after_days,
    plan_gated_features: .features,
    note: "restrict_user_api_token_creation is absent from .features, so it is not plan-gated"
  }'
}

# Write. TRAP 7: single-key PATCH body, built here and never derived from a read.
# Nothing in this function can emit allowed_ip_addresses, so it cannot re-assert
# (or clear) an IP allowlist as a side effect of toggling token creation.
set_restrict_token_creation() {
  local value="$1"
  case "${value}" in
    true|false) ;;
    *) echo "invalid value '${value}'; expected true or false" >&2; exit 2 ;;
  esac

  # Turning this OFF re-opens token creation to every member. That is a
  # loosening, so say so rather than performing it silently.
  if [ "${value}" = "false" ]; then
    echo "NOTE: setting restrict_user_api_token_creation=false lets every" >&2
    echo "organization member mint API access tokens again. This widens the" >&2
    echo "credential surface the rest of this pack is written to police." >&2
  fi

  jq -n --argjson v "${value}" '{restrict_user_api_token_creation: $v}' \
  | curl -sS --fail-with-body \
      -H "Authorization: Bearer ${BUILDKITE_TOKEN}" \
      -H "Content-Type: application/json" \
      -X PATCH "${REST}/organizations/${BUILDKITE_ORG_SLUG}/api-settings" \
      --data @- \
  | jq '{restrict_user_api_token_creation, plan_gated_features: .features}'
}

# Full inventory of the organization's API access tokens, paginated.
# There is no `count` on this connection — pull every edge and count locally.
fetch_token_page() {
  local after="$1"
  jq -n --arg slug "${BUILDKITE_ORG_SLUG}" --arg after "${after}" '{
    query: "query($slug:ID!,$after:String){ organization(slug:$slug){
              revokeInactiveTokensAfter
              apiAccessTokens(first:100, after:$after){
                edges { node {
                  id uuid description scopes
                  createdAt expiresAt lastAccessedAt ipAddress
                  owner { name email }
                } }
                pageInfo { hasNextPage endCursor }
              } } }",
    variables: { slug: $slug, after: (if $after == "" then null else $after end) }
  }' | gql
}

HTH_TOKENS_TMP=""
hth_tokens_cleanup() {
  if [ -n "${HTH_TOKENS_TMP}" ]; then rm -f "${HTH_TOKENS_TMP}"; fi
  HTH_TOKENS_TMP=""
}

collect_tokens() {
  local after="" page
  # mktemp, not `: >/tmp/hth-bk-tokens.$$.jsonl`. A PID-derived name in a shared
  # world-writable directory is guessable, and `>` follows a symlink planted at
  # that path. What accumulates here is the whole token inventory —
  # descriptions, uuids, owner emails, last-used IPs. No secret VALUES are in it
  # (GraphQL never returns them), which is why this is hygiene and not
  # disclosure, but it is still a reconnaissance map of every credential in the
  # organization.
  HTH_TOKENS_TMP="$(mktemp "${TMPDIR:-/tmp}/hth-bk-tokens.XXXXXX")"
  # RETURN covers the normal path. EXIT is the one that matters: die_on_gql_errors
  # calls exit mid-loop on any GraphQL error, so a plain `rm -f` after the loop is
  # unreachable on exactly the runs most likely to strand the file.
  trap hth_tokens_cleanup RETURN EXIT
  while :; do
    page=$(fetch_token_page "${after}")
    die_on_gql_errors "${page}"
    jq -c '.data.organization.apiAccessTokens.edges[].node' <<<"${page}" >>"${HTH_TOKENS_TMP}"
    if [ "$(jq -r '.data.organization.apiAccessTokens.pageInfo.hasNextPage' <<<"${page}")" != "true" ]; then
      break
    fi
    after=$(jq -r '.data.organization.apiAccessTokens.pageInfo.endCursor' <<<"${page}")
  done
  jq -s '.' "${HTH_TOKENS_TMP}"
}

# The uuid of the token running this script. Revoking it is unrecoverable.
self_token_uuid() {
  curl -sS --fail-with-body -H "Authorization: Bearer ${BUILDKITE_TOKEN}" \
    "${REST}/access-token" | jq -r '.uuid'
}

# Risk-annotated inventory. ORG_ADMIN_SCOPES are the REST scopes that let a
# token change who can do what — a token holding any of them is an admin
# credential no matter what it was created for.
inventory() {
  local self
  self=$(self_token_uuid)
  collect_tokens | jq --arg self "${self}" '
    def age_days: if . == null then null
                  else ((now - (sub("\\.[0-9]+";"") | fromdateiso8601)) / 86400 | floor) end;
    def admin_scopes: ["WRITE_ORGANIZATIONS","WRITE_ORGANIZATION_SETTINGS",
                       "WRITE_ORGANIZATION_INVITATIONS","WRITE_TEAMS","WRITE_CLUSTERS"];
    {
      total: length,
      never_expiring: [ .[] | select(.expiresAt == null) ] | length,
      never_used:     [ .[] | select(.lastAccessedAt == null) ] | length,
      admin_capable:  [ .[] | select((.scopes - admin_scopes) != .scopes) ] | length,
      tokens: [ .[] | {
        uuid, description,
        owner: .owner.name,
        is_self: (.uuid == $self),
        scope_count: (.scopes | length),
        admin_capable: ((.scopes - admin_scopes) != .scopes),
        expires_at: .expiresAt,
        never_expires: (.expiresAt == null),
        last_used_days_ago: (.lastAccessedAt | age_days),
        last_used_from_ip: .ipAddress
      } ] | sort_by(.last_used_days_ago == null, -(.last_used_days_ago // 0))
    }'
}

# Tokens unused for longer than the threshold. Never-used tokens are reported
# in a separate bucket because Buildkite provisions system tokens that read null.
stale() {
  local days="${1:-90}" self
  self=$(self_token_uuid)
  collect_tokens | jq --argjson days "${days}" --arg self "${self}" '
    def age_days: if . == null then null
                  else ((now - (sub("\\.[0-9]+";"") | fromdateiso8601)) / 86400 | floor) end;
    {
      threshold_days: $days,
      stale: [ .[] | . + {age: (.lastAccessedAt | age_days)}
               | select(.age != null and .age >= $days)
               | {uuid, description, owner: .owner.name, age_days: .age,
                  last_used_from_ip: .ipAddress, is_self: (.uuid == $self)} ],
      never_used_review_manually: [ .[] | select(.lastAccessedAt == null)
               | {uuid, description, owner: .owner.name, created_at: .createdAt} ]
    }'
}

# The org-wide inactivity clock. This is the single setting that turns Step 4's
# manual review into an automatic control: Buildkite revokes any API access token
# that has not been used within the period, without anyone running a script.
#
# ENTERPRISE-GATED (the mutation, not the read). Organization
# .revokeInactiveTokensAfter reads on every plan and returns null when unset —
# so the CHECK half below runs anywhere. The update mutation requires Enterprise
# and returns a plan error otherwise.
#
# Valid RevokeInactiveTokenPeriod values:
#   DAYS_30  DAYS_60  DAYS_90  DAYS_180  DAYS_365  NEVER
# NEVER is the insecure setting; it is also what null means in practice.
read_auto_revoke() {
  local body
  body=$(jq -n --arg slug "${BUILDKITE_ORG_SLUG}" '{
    query: "query($slug:ID!){ organization(slug:$slug){ id revokeInactiveTokensAfter } }",
    variables: { slug: $slug }
  }' | gql)
  die_on_gql_errors "${body}"
  jq '{
    organization_id: .data.organization.id,
    revoke_inactive_tokens_after: .data.organization.revokeInactiveTokensAfter,
    compliant: (.data.organization.revokeInactiveTokensAfter
                | . != null and . != "NEVER")
  }' <<<"${body}"
}

# RevokeInactiveTokenPeriod -> days, so the pre-flight below can ask `stale` the
# same question the clock will ask. NEVER has no day count; it never reaches here.
period_days() {
  case "$1" in
    DAYS_30) echo 30 ;; DAYS_60) echo 60 ;; DAYS_90) echo 90 ;;
    DAYS_180) echo 180 ;; DAYS_365) echo 365 ;;
    *) echo 90 ;;
  esac
}

# TRAP 6 IS THE WHOLE REASON THIS VERB IS GATED.
# `stale` deliberately buckets null lastAccessedAt separately because Buildkite
# provisions system tokens that legitimately read null, and "revoke everything
# never seen" breaks hosted agents. set-auto-revoke hands that same sweep to
# Buildkite to run automatically and permanently, org-wide, against every token
# including the ones nobody in this organization created. Dormant-BY-DESIGN
# credentials are the casualties: break-glass tokens, DR tokens, the token behind
# a quarterly job, and the vendor's own system tokens. None of them announce
# themselves, and a token revoked by the clock cannot be un-revoked.
#
# So this is the one mutation here that refuses to run blind. The operator must
# have looked at what the clock will eat — the never-used bucket is printed on
# every invocation — and must assert it with HTH_AUTO_REVOKE_REVIEWED=1, the same
# shape as the SCIM gate in the 2.6 pack. NEVER is exempt: it only ever loosens.
set_auto_revoke() {
  local period="$1" org_id body never_used
  case "${period}" in
    DAYS_30|DAYS_60|DAYS_90|DAYS_180|DAYS_365|NEVER) ;;
    *) echo "invalid period '${period}'; expected one of DAYS_30 DAYS_60 DAYS_90 DAYS_180 DAYS_365 NEVER" >&2
       exit 2 ;;
  esac

  if [ "${period}" != "NEVER" ]; then
    # Pre-flight, not advice: show the tokens the clock is most likely to take
    # first. Anything below with no legitimate reason to be used inside the
    # period is a token this setting will revoke without asking again.
    never_used=$(stale "$(period_days "${period}")" | jq '.never_used_review_manually')
    echo "Tokens with NO recorded use (TRAP 6 — includes Buildkite's own system" >&2
    echo "tokens, break-glass and DR credentials). Every one of these is revoked" >&2
    echo "by ${period} unless it is exercised inside the period:" >&2
    jq -r 'if length == 0 then "  (none)"
           else .[] | "  \(.uuid)  \(.description // "<no description>")  owner=\(.owner // "-")  created=\(.created_at)"
           end' <<<"${never_used}" >&2
    echo >&2

    if [ "${HTH_AUTO_REVOKE_REVIEWED:-}" != "1" ]; then
      echo "REFUSING: set HTH_AUTO_REVOKE_REVIEWED=1 to confirm you have read the" >&2
      echo "list above and the '${period}' bucket in 'stale', and that no token" >&2
      echo "that is dormant BY DESIGN will be destroyed by this clock. This arms" >&2
      echo "an automatic, permanent, org-wide revocation of API access tokens." >&2
      echo "Run '$0 inventory' and '$0 stale $(period_days "${period}")' first." >&2
      exit 5
    fi
  fi

  org_id=$(read_auto_revoke | jq -r '.organization_id')

  # NOTE the exact input type name: OrganizationRevokeInactiveTokensAfterUpdate
  # MutationInput. It carries the "Mutation" infix; the shorter ...UpdateInput
  # spelling does not exist and fails schema validation before it ever executes.
  body=$(jq -n --arg org "${org_id}" --arg period "${period}" '{
    query: "mutation($org:ID!,$period:RevokeInactiveTokenPeriod!){
              organizationRevokeInactiveTokensAfterUpdate(input:{
                organizationId:$org, revokeInactiveTokensAfter:$period
              }){ organization { id revokeInactiveTokensAfter } } }",
    variables: { org: $org, period: $period }
  }' | gql)
  die_on_gql_errors "${body}"
  jq '.data.organizationRevokeInactiveTokensAfterUpdate.organization' <<<"${body}"
}

# Revoke one token by uuid. Resolves uuid -> GraphQL id, and refuses to revoke
# the token this script is authenticating with.
revoke_token() {
  local target_uuid="$1" self node org_id body
  self=$(self_token_uuid)

  if [ "${target_uuid}" = "${self}" ]; then
    echo "REFUSING: ${target_uuid} is the token authenticating this script." >&2
    echo "Revoking it ends your API access with no way back. Mint a replacement," >&2
    echo "re-export BUILDKITE_TOKEN, then revoke this one." >&2
    exit 3
  fi

  node=$(collect_tokens | jq -c --arg u "${target_uuid}" 'map(select(.uuid == $u)) | first')
  if [ -z "${node}" ] || [ "${node}" = "null" ]; then
    echo "no API access token in ${BUILDKITE_ORG_SLUG} with uuid ${target_uuid}" >&2
    echo "run 'inventory' to list the uuids this organization actually has." >&2
    exit 4
  fi

  org_id=$(read_auto_revoke | jq -r '.organization_id')

  # Exact input type: OrganizationAPIAccessTokenRevokeMutationInput — capital
  # API, "Mutation" infix. Fields: organizationId, apiAccessTokenId.
  body=$(jq -n --arg org "${org_id}" --arg id "$(jq -r '.id' <<<"${node}")" '{
    query: "mutation($org:ID!,$id:ID!){
              organizationApiAccessTokenRevoke(input:{
                organizationId:$org, apiAccessTokenId:$id
              }){ revokedApiAccessTokenId } }",
    variables: { org: $org, id: $id }
  }' | gql)
  die_on_gql_errors "${body}"

  jq -n --argjson node "${node}" --argjson res "${body}" '{
    revoked_description: $node.description,
    revoked_uuid: $node.uuid,
    revoked_api_access_token_id: $res.data.organizationApiAccessTokenRevoke.revokedApiAccessTokenId
  }'
}

Sources: Buildkite security controls · Organization API settings

2.6 Remove Dormant Organization Members

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	5.3
NIST 800-53	AC-2(3)

Description

Find organization members who have stopped using Buildkite and remove them, as a recurring review rather than a one-off cleanup.

Rationale

Why This Matters:

SCIM deprovisioning only catches people who left the company; it does nothing about a current employee who moved to another team two years ago and still holds pipeline access
A dormant account is the ideal account to compromise precisely because nobody is watching it — there is no legitimate activity for malicious activity to stand out against
Buildkite membership survives changes that feel like they should have removed it, including team deletion and repository access changes, so access accumulates by default
Nobody notices an account that does nothing, which means the window between compromise and detection on a dormant account is bounded only by your audit-log retention

Attack Prevented: Persistent access via abandoned accounts, privilege accumulation across internal role changes, undetected credential reuse against accounts with no baseline activity

ClickOps Implementation

Step 1: Establish a Dormancy Threshold

Pick an inactivity window that matches your access-review cadence — 90 days is a common starting point, and it should be shorter than your audit-log retention so you can still investigate what a dormant account did before you remove it.
Write the threshold down as policy, because a threshold that lives only in someone’s head produces inconsistent reviews.

Step 2: Review Members Against It

On Enterprise: go to Settings → Audit → Inactive User List and select a time period. Buildkite offers 30 days, 90 days (the default), and 120 days only, so pick the listed period closest to — and not longer than — the threshold you set in Step 1. Each entry shows the member’s name, email address, and the date they were last active.
On every other plan there is no console inactivity view. Read each member’s lastSeenAt through the GraphQL API and filter client-side; the Code Pack does this for you.
Treat a last-active date of 30 July 2020 as “has never logged in,” not as genuine 2020 activity — Buildkite uses that date as the placeholder for members with no recorded activity.
For each member past the threshold, confirm with their manager whether the access is still needed rather than assuming dormancy means departure.

Step 3: Remove and Record

Export the filtered list to CSV before acting on it, so the population you reviewed is evidenced rather than reconstructed afterwards.
Remove members whose access is no longer needed — from the Inactive User List, select the checkbox beside each user and choose Remove selected users. Removing a user from the organization does not delete their Buildkite account, and their builds are retained.
Record the review itself — an access review you cannot evidence is an access review you cannot claim in an audit.

Note on availability: the Inactive User List and the inactiveSince filter are Enterprise features that also require Audit Logging. On lower plans, read lastSeenAt per member and filter client-side; the Code Pack does this automatically when the filter is unavailable.

Code Implementation

Code Pack: API Script

hth-buildkite-2.06-dormant-members.sh View source on GitHub ↗

# Full membership roster, paginated. Selects only introspection-confirmed fields.
# `role` is fetched and filtered client-side rather than passed as an argument
# (TRAP 8). `sso { mode }` matters because an SSO-OPTIONAL dormant member can
# still authenticate with a password that no IdP policy governs.
fetch_member_page() {
  local after="$1" since="$2"
  if [ -n "${since}" ]; then
    jq -n --arg slug "${BUILDKITE_ORG_SLUG}" --arg after "${after}" --arg since "${since}" '{
      query: "query($slug:ID!,$after:String,$since:DateTime){ organization(slug:$slug){
                members(first:100, after:$after, inactiveSince:$since){
                  edges { node {
                    id uuid role createdAt lastSeenAt
                    sso { mode }
                    user { name email }
                  } }
                  pageInfo { hasNextPage endCursor }
                } } }",
      variables: { slug: $slug, since: $since,
                   after: (if $after == "" then null else $after end) }
    }' | gql
  else
    jq -n --arg slug "${BUILDKITE_ORG_SLUG}" --arg after "${after}" '{
      query: "query($slug:ID!,$after:String){ organization(slug:$slug){
                members(first:100, after:$after){
                  edges { node {
                    id uuid role createdAt lastSeenAt
                    sso { mode }
                    user { name email }
                  } }
                  pageInfo { hasNextPage endCursor }
                } } }",
      variables: { slug: $slug, after: (if $after == "" then null else $after end) }
    }' | gql
  fi
}

collect_members() {
  local since="${1:-}" after="" page tmp
  tmp="$(mktemp "${TMPDIR:-/tmp}/hth-bk-members.XXXXXX")"
  while :; do
    page=$(fetch_member_page "${after}" "${since}")
    die_on_gql_errors "${page}"
    jq -c '.data.organization.members.edges[].node' <<<"${page}" >>"${tmp}"
    if [ "$(jq -r '.data.organization.members.pageInfo.hasNextPage' <<<"${page}")" != "true" ]; then
      break
    fi
    after=$(jq -r '.data.organization.members.pageInfo.endCursor' <<<"${page}")
  done
  jq -s '.' "${tmp}"
  rm -f "${tmp}"
}

# TRAP 1 / Enterprise gate. Probe the server-side filter with a one-row request
# rather than assuming a plan tier. Any error at all -> client-side path.
filter_supported() {
  local body
  body=$(jq -n --arg slug "${BUILDKITE_ORG_SLUG}" --arg since "$(iso_days_ago 3650)" '{
    query: "query($slug:ID!,$since:DateTime){ organization(slug:$slug){
              members(first:1, inactiveSince:$since){ edges { node { uuid } } } } }",
    variables: { slug: $slug, since: $since }
  }' | gql) || return 1
  ! jq -e '.errors' >/dev/null 2>&1 <<<"${body}"
}

# TRAP 1b. The audit log is the half that is actually plan-gated, and it fails
# with a TOP-LEVEL "type":"permission_error" rather than a plain errors envelope.
# Probed separately so a review never assumes it has corroboration it does not.
audit_log_available() {
  local body
  body=$(jq -n --arg slug "${BUILDKITE_ORG_SLUG}" '{
    query: "query($slug:ID!){ organization(slug:$slug){ auditEvents(first:1){ count } } }",
    variables: { slug: $slug }
  }' | gql) || return 1
  ! jq -e '.errors' >/dev/null 2>&1 <<<"${body}"
}

filter_support_status() {
  local filter audit
  if filter_supported; then filter=available; else filter=unavailable; fi
  if audit_log_available; then audit=available; else audit=unavailable; fi

  jq -n --arg filter "${filter}" --arg audit "${audit}" '{
    inactiveSince_filter: $filter,
    filter_note: (if $filter == "available"
      then "Server-side dormancy filtering works here. `report` still pulls unfiltered on purpose (TRAP 2), so this is provenance, not a switch."
      else "The argument was rejected. `report` was already filtering client-side on lastSeenAt, so detection is unaffected; only the request count changes." end),
    audit_log: $audit,
    audit_note: (if $audit == "available"
      then "auditEvents readable — a dormant account can be corroborated with what it last did."
      else "auditEvents is plan-gated on this organization. Dormancy is still fully DETECTABLE; what the account did before going quiet is not recoverable, so removal decisions rest on activity timestamps alone." end),
    detection_capability: "unconditional — never depends on either flag above"
  }'
}

# TRAP 3. Members who own an API access token that is actually being used are
# not dormant, whatever lastSeenAt says. Verified handles, exercised live by
# pack 2.5: apiAccessTokens has no `count`, so paginate and join on owner.email.
collect_token_owners() {
  local after="" page tmp
  tmp="$(mktemp "${TMPDIR:-/tmp}/hth-bk-owners.XXXXXX")"
  while :; do
    page=$(jq -n --arg slug "${BUILDKITE_ORG_SLUG}" --arg after "${after}" '{
      query: "query($slug:ID!,$after:String){ organization(slug:$slug){
                apiAccessTokens(first:100, after:$after){
                  edges { node { description lastAccessedAt owner { email } } }
                  pageInfo { hasNextPage endCursor }
                } } }",
      variables: { slug: $slug, after: (if $after == "" then null else $after end) }
    }' | gql)
    die_on_gql_errors "${page}"
    jq -c '.data.organization.apiAccessTokens.edges[].node' <<<"${page}" >>"${tmp}"
    if [ "$(jq -r '.data.organization.apiAccessTokens.pageInfo.hasNextPage' <<<"${page}")" != "true" ]; then
      break
    fi
    after=$(jq -r '.data.organization.apiAccessTokens.pageInfo.endCursor' <<<"${page}")
  done
  jq -s '.' "${tmp}"
  rm -f "${tmp}"
}

# The dormancy review. Three buckets, deliberately not one ranking:
#   dormant             — signed in once, not since the threshold
#   never_signed_in     — accepted the invite and never authenticated (TRAP 2)
#   dormant_but_api_active — lastSeenAt says gone, a live token says otherwise (TRAP 3)
report() {
  local days="${1:-90}" cutoff members owners mode
  case "${days}" in
    ''|*[!0-9]*) echo "threshold must be a whole number of days, got '${days}'" >&2; exit 2 ;;
  esac
  cutoff="$(iso_days_ago "${days}")"

  # The roster is ALWAYS pulled unfiltered, on every plan. The server-side
  # filter would drop the never_signed_in bucket if it treats null lastSeenAt as
  # "not inactive" (TRAP 2), and that bucket is the point of the control. The
  # probe is therefore reported as provenance, not used as a shortcut.
  if filter_supported; then mode="server-filter-available-unused"; else mode="client-side-only"; fi
  members=$(collect_members "")
  owners=$(collect_token_owners)

  jq -n --argjson members "${members}" --argjson owners "${owners}" \
        --argjson days "${days}" --arg cutoff "${cutoff}" --arg mode "${mode}" '
    def age_days: if . == null then null
                  else ((now - (sub("\\.[0-9]+";"") | fromdateiso8601)) / 86400 | floor) end;

    # email -> most recent token use, for the API-activity join.
    ($owners | map(select(.owner.email != null))
             | group_by(.owner.email)
             | map({ key: .[0].owner.email,
                     value: { tokens: length,
                              last_used_days_ago: ([ .[] | .lastAccessedAt | age_days ]
                                                   | map(select(. != null))
                                                   | if length == 0 then null else min end),
                              descriptions: [ .[] | .description ] } })
             | from_entries) as $tok

    | ($members | map(. + {
         idle_days:  (.lastSeenAt | age_days),
         email:      .user.email,
         name:       .user.name,
         sso_mode:   .sso.mode,
         token:      ($tok[.user.email // ""] // null)
       })) as $m

    | {
        threshold_days: $days,
        cutoff: $cutoff,
        dormancy_source: $mode,
        total_members: ($m | length),
        admins: ($m | map(select(.role == "ADMIN")) | length),

        # Signed in at least once, but not inside the window. Ordered by an
        # ADMIN-first, then longest-idle ranking: an idle admin is the finding.
        dormant: [ $m[]
          | select(.idle_days != null and .idle_days >= $days)
          | select(.token == null or .token.last_used_days_ago == null
                                  or .token.last_used_days_ago >= $days)
          | { uuid, id, name, email, role, sso_mode,
              idle_days, last_seen_at: .lastSeenAt, member_since: .createdAt } ]
          | sort_by(.role != "ADMIN", -.idle_days),

        # TRAP 2. Null never satisfies the comparison above, so it is collected
        # explicitly and aged on createdAt. These are the highest-risk accounts.
        never_signed_in: [ $m[]
          | select(.lastSeenAt == null)
          | { uuid, id, name, email, role, sso_mode,
              member_since: .createdAt,
              days_since_invite: (.createdAt | age_days) } ]
          | sort_by(.role != "ADMIN", -(.days_since_invite // 0)),

        # TRAP 3. Console-dormant, API-live. Do NOT remove these on this report.
        dormant_but_api_active: [ $m[]
          | select(.idle_days != null and .idle_days >= $days)
          | select(.token != null and .token.last_used_days_ago != null
                                   and .token.last_used_days_ago < $days)
          | { uuid, name, email, role,
              console_idle_days: .idle_days,
              token_last_used_days_ago: .token.last_used_days_ago,
              tokens: .token.descriptions,
              verdict: "service identity — migrate the automation off a human membership before removing" } ],

        # An SSO-OPTIONAL dormant member can still sign in with a password no
        # IdP policy governs, so SSO mode changes the urgency of the removal.
        dormant_outside_sso: [ $m[]
          | select((.idle_days != null and .idle_days >= $days) or .lastSeenAt == null)
          | select(.sso_mode != "REQUIRED")
          | { uuid, name, email, role, sso_mode } ]
      }'
}

# Remove one membership by uuid, behind every guard the API does not provide.
# organizationMemberDelete takes OrganizationMemberDeleteInput { clientMutationId, id }.
remove_member() {
  local target_uuid="$1" members node self_email admin_count body

  # TRAP 7. No field exposes SCIM management, so the operator asserts the check.
  if [ "${HTH_SCIM_REVIEWED:-}" != "1" ]; then
    echo "REFUSING: set HTH_SCIM_REVIEWED=1 to confirm you checked whether this" >&2
    echo "organization is SCIM-managed. If the IdP still asserts this user, this" >&2
    echo "deletion is reverted at the next sync and the access review evidences a" >&2
    echo "removal that did not hold. Deprovision in the IdP instead." >&2
    exit 5
  fi

  # TRAP 5. Fail closed: no resolvable caller identity means no self-guard.
  self_email=$(curl -sS --fail-with-body -H "Authorization: Bearer ${BUILDKITE_TOKEN}" \
                 "${REST}/user" | jq -r '.email // empty')
  if [ -z "${self_email}" ]; then
    echo "REFUSING: GET /v2/user returned no email, so the self-removal guard" >&2
    echo "cannot be evaluated. Aborting rather than deleting unguarded." >&2
    exit 6
  fi

  members=$(collect_members "")
  node=$(jq -c --arg u "${target_uuid}" 'map(select(.uuid == $u)) | first' <<<"${members}")
  if [ -z "${node}" ] || [ "${node}" = "null" ]; then
    echo "no organization member in ${BUILDKITE_ORG_SLUG} with uuid ${target_uuid}" >&2
    echo "run 'report' to list the uuids this organization actually has." >&2
    exit 4
  fi

  if [ "$(jq -r '.user.email // ""' <<<"${node}")" = "${self_email}" ]; then
    echo "REFUSING: ${target_uuid} is the membership of ${self_email}, the identity" >&2
    echo "authenticating this script. Removing it ends your own organization access." >&2
    exit 3
  fi

  # TRAP 6. Refuse to delete the last ADMIN.
  if [ "$(jq -r '.role' <<<"${node}")" = "ADMIN" ]; then
    admin_count=$(jq '[ .[] | select(.role == "ADMIN") ] | length' <<<"${members}")
    if [ "${admin_count}" -le 1 ]; then
      echo "REFUSING: ${target_uuid} is the only ADMIN in ${BUILDKITE_ORG_SLUG}." >&2
      echo "Deleting it leaves nobody able to manage the organization. Promote a" >&2
      echo "replacement admin first, then re-run." >&2
      exit 7
    fi
  fi

  # TRAP 4. The mutation takes the membership node `id`, not the uuid shown in
  # the console and not the user's id. Resolve it here so it cannot be confused.
  body=$(jq -n --arg id "$(jq -r '.id' <<<"${node}")" '{
    query: "mutation($id:ID!){
              organizationMemberDelete(input:{ id:$id }){
                clientMutationId
                organization { name }
                user { name email } } }",
    variables: { id: $id }
  }' | gql)
  die_on_gql_errors "${body}"

  jq -n --argjson node "${node}" --argjson res "${body}" '{
    removed_uuid: $node.uuid,
    removed_member_id: $node.id,
    removed_name: $node.user.name,
    removed_email: $node.user.email,
    removed_role: $node.role,
    last_seen_at: $node.lastSeenAt,
    member_since: $node.createdAt,
    confirmed_by_server: $res.data.organizationMemberDelete.user
  }'
}

Sources: Inactive user list · User and team permissions

2.7 Govern Cross-Pipeline Access with Rules

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	6.8
NIST 800-53	AC-3, AC-4

Description

Use Buildkite Rules to declare explicitly which pipelines may trigger which other pipelines and read each other’s artifacts, instead of relying on the implicit permissions that team membership grants.

Rationale

Why This Matters:

Rules override the usual trigger-step permission checks, so a rule is not a tightening of existing permissions — it is a separate grant that can widen them, and an unreviewed rule is a standing authorization nobody audited
Artifact-read rules cross cluster boundaries, which means a rule can quietly undo the isolation that 3.2 was configured to enforce
A public pipeline can be granted the ability to trigger a private one, turning a low-trust entry point into a path to high-trust infrastructure
Cross-pipeline triggering is the most direct lateral-movement primitive Buildkite offers, and without an inventory of rules there is no way to answer “what can this pipeline reach”

Attack Prevented: Lateral movement from a compromised low-trust pipeline into production pipelines, cross-cluster artifact exfiltration, privilege escalation through unreviewed trigger grants

ClickOps Implementation

Step 1: Inventory What Exists

Go to Settings → Rules and list every rule currently defined.
For each, identify the source pipeline, the target pipeline, and who added it — a rule with no owner is a rule to remove.

Step 2: Apply Deny-by-Default

Remove rules that are not tied to a documented workflow. Cross-pipeline access should be the exception you justified, not the default you inherited.
Where a rule is needed, scope it to the narrowest source and target pair that makes the workflow function.

Step 3: Review Rules on the Same Cadence as Permissions

Add rules to your access-review cycle. They grant access, so they age the same way permissions do.

Note: Rules are in public preview and available on all plans. They are exposed through Terraform, GraphQL (ruleCreate/ruleUpdate/ruleDelete) and REST (/v2/organizations/{org}/rules).

Code Implementation

Code Pack: Terraform

hth-buildkite-2.07-organization-rules.tf View source on GitHub ↗

locals {
  # The action Buildkite assigns for each rule type. Introspected from the live
  # RuleAction enum: TRIGGER_BUILD, ARTIFACTS_READ. Used to prove the rule that
  # was created does what the declared type claims.
  hth_rule_expected_action = {
    "pipeline.trigger_build.pipeline"  = "TRIGGER_BUILD"
    "pipeline.artifacts_read.pipeline" = "ARTIFACTS_READ"
  }

  hth_uuid_pattern = "^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$"
}

# One resource per approved cross-boundary grant. Each entry is a documented
# exception to cluster and visibility isolation, reviewed as code.
resource "buildkite_organization_rule" "cross_pipeline" {
  for_each = var.cross_pipeline_rules

  type = each.value.type

  description = coalesce(
    each.value.description,
    format("HTH 2.7 - %s: %s -> %s", each.value.type, each.value.source_pipeline, each.value.target_pipeline),
  )

  # `conditions` is omitted from the document entirely when empty, because
  # Buildkite treats an absent conditions array and an empty one the same way:
  # unconditional. The precondition below is what actually stops that happening.
  value = jsonencode(merge(
    {
      source_pipeline = each.value.source_pipeline
      target_pipeline = each.value.target_pipeline
    },
    length(each.value.conditions) > 0 ? { conditions = each.value.conditions } : {},
  ))

  lifecycle {
    # TRAP 2. A rule with no conditions is a standing, unscoped grant. Creating
    # one has to be a named decision, not a default.
    precondition {
      condition = length(each.value.conditions) > 0 || contains(var.unconditional_rule_exceptions, each.key)
      error_message = format(
        "Rule '%s' declares no conditions, which grants %s from '%s' to '%s' for every build on every branch, permanently. Add conditions (for example \"source.build.branch == 'main'\") or add '%s' to var.unconditional_rule_exceptions to record the decision.",
        each.key,
        each.value.type,
        each.value.source_pipeline,
        each.value.target_pipeline,
        each.key,
      )
    }

    # TRAP 3. Re-read the action Buildkite assigned and compare it to the action
    # the declared type implies. Catches a type/direction copy-paste that would
    # otherwise apply successfully with the hole facing the wrong way.
    postcondition {
      condition = upper(self.action) == local.hth_rule_expected_action[each.value.type]
      error_message = format(
        "Rule '%s' declared type '%s' (expected action %s) but Buildkite created action '%s'. Re-read the source/target direction before keeping this rule.",
        each.key,
        each.value.type,
        local.hth_rule_expected_action[each.value.type],
        self.action,
      )
    }

    # TRAP 4. When the operator supplied UUIDs, prove Buildkite resolved to those
    # exact pipelines. Slug-declared rules skip this check by design - which is
    # itself the argument for declaring UUIDs.
    postcondition {
      condition = (
        !can(regex(local.hth_uuid_pattern, each.value.source_pipeline))
        || lower(self.source_uuid) == lower(each.value.source_pipeline)
      )
      error_message = format(
        "Rule '%s' declared source pipeline UUID '%s' but resolved to '%s'.",
        each.key, each.value.source_pipeline, self.source_uuid,
      )
    }

    postcondition {
      condition = (
        !can(regex(local.hth_uuid_pattern, each.value.target_pipeline))
        || lower(self.target_uuid) == lower(each.value.target_pipeline)
      )
      error_message = format(
        "Rule '%s' declared target pipeline UUID '%s' but resolved to '%s'.",
        each.key, each.value.target_pipeline, self.target_uuid,
      )
    }
  }
}

output "cross_pipeline_rule_grants" {
  description = "Every standing cross-pipeline grant, as Buildkite resolved it. Review this list, not the input variables - these are the UUIDs the platform actually enforces against."
  value = {
    for k, r in buildkite_organization_rule.cross_pipeline : k => {
      uuid        = r.uuid
      effect      = r.effect
      action      = r.action
      source_uuid = r.source_uuid
      target_uuid = r.target_uuid
      conditional = length(var.cross_pipeline_rules[k].conditions) > 0
    }
  }
}

# Rules can also be created in the Buildkite console, and the provider offers no
# resource or data source that enumerates every rule in the organization - only a
# lookup by known UUID. So the Terraform-side audit is: pin the rules you
# reviewed and prove they still point where you approved. Use the REST endpoint
# GET /v2/organizations/{org}/rules (verified HTTP 200) to discover UUIDs that
# appeared out of band, then pin them here or delete them.
data "buildkite_organization_rule" "reviewed" {
  for_each = var.reviewed_rules

  uuid = each.value.uuid
}

# A `check` block is continuous validation: it reports on every plan and apply
# and does NOT block the apply. It is the right shape here because re-pointed
# rules are a review finding, not a reason to fail an unrelated deploy. If a
# pinned rule is deleted in the console the data source read errors instead,
# which is the loud failure you want for a vanished approval record.
check "reviewed_rules_still_point_where_approved" {
  assert {
    condition = alltrue([
      for k, r in data.buildkite_organization_rule.reviewed :
      lower(r.source_uuid) == lower(var.reviewed_rules[k].source_uuid)
      && lower(r.target_uuid) == lower(var.reviewed_rules[k].target_uuid)
    ])
    error_message = format(
      "A reviewed Buildkite rule now grants access between different pipelines than the ones approved. Approved: %s. Live: %s.",
      jsonencode({ for k, v in var.reviewed_rules : k => "${v.source_uuid} -> ${v.target_uuid}" }),
      jsonencode({ for k, r in data.buildkite_organization_rule.reviewed : k => "${r.source_uuid} -> ${r.target_uuid}" }),
    )
  }
}

output "reviewed_rule_audit" {
  description = "Live state of every out-of-band rule pinned for review. `effect` is always ALLOW - the enum has no other value - so the security question is whether the grant should exist at all."
  value = {
    for k, r in data.buildkite_organization_rule.reviewed : k => {
      effect      = r.effect
      action      = r.action
      source_uuid = r.source_uuid
      target_uuid = r.target_uuid
      description = r.description
    }
  }
}

Sources: Rules overview · Manage rules

3. Agent Security

3.1 Configure Agent Tokens

Profile Level: L1 (Crawl)

Framework	Control
CIS Controls	3.11
NIST 800-53	SC-12

Description

Securely manage agent registration tokens.

Rationale

Why This Matters:

Agent tokens let any holder register a build agent and execute pipeline jobs, so a leaked token is effectively code execution in your CI
Scoping tokens per environment confines a leaked token to a single cluster rather than the whole organization
Regular rotation shrinks the window an exposed token remains usable
Revoking unused tokens removes standing credentials that attackers could discover in code, logs, or images

Attack Prevented: Agent impersonation, unauthorized job execution, credential leakage, supply chain compromise

ClickOps Implementation

Step 1: Create Tokens Inside a Cluster

Navigate to the cluster’s Agent tokens page — agent tokens are cluster-scoped. A token belongs to exactly one cluster and cannot be used to register an agent into a different cluster or organization, which means the scoping work is largely done for you once your clusters are right (see 3.2).
Create a separate token per cluster rather than reusing one across environments.

Step 2: Bound the Token in Time

Set an expiration timestamp when creating the token. Two constraints matter operationally:
- Expirations can only be set through the API — tokens created in the web UI have no expiry at all and must be rotated by hand.
- The expiry must be at least 10 minutes in the future, and once set it is immutable. Rotation means creating a replacement token, not extending the existing one, so build the replacement step into your rotation runbook.

Step 3: Bound the Token in Space

Set Allowed IP Addresses on the token — a CIDR allowlist of the networks your agents register from. A token that leaks outside those ranges is inert.
Lockout warning — this allowlist strands agents. The list is enforced at registration, so an agent whose egress address falls outside it cannot register at all. A wrong CIDR silently takes every agent presenting that token offline on its next restart, and an empty list means unrestricted rather than blocked. Derive the ranges from your runners’ observed egress addresses, not from the VPC block you assume they use, and leave the allowlist unset rather than guessing at it. The undo path here is not self-sealing — unlike the API IP allowlist in 4.1, your API and console access are unaffected, so you can widen or clear the list to recover the fleet.

Step 4: Handle Tokens Safely

Store tokens in a secrets manager, never in an agent AMI, container image, or repository.
Rotate on a schedule and revoke tokens that no agent is presenting.

Sources: Buildkite agent tokens · Manage clusters and queues

Code Implementation

Code Pack: Terraform

hth-buildkite-3.01-configure-agent-tokens.tf View source on GitHub ↗

# Bind tokens to clusters that ALREADY EXIST rather than creating them here.
# A cluster is a trust boundary owned by control 3.3 (or by the console); token
# policy is a separate concern with a different change cadence, and coupling the
# two means every token rotation re-plans the cluster.
#
# `name` is the only selector this data source accepts, and it is case- and
# space-sensitive: Buildkite's auto-provisioned cluster is literally named
# "Default cluster". A typo surfaces as a plan-time "cluster not found", which is
# the desired failure — it is never silently created.
data "buildkite_cluster" "token_target" {
  for_each = toset([
    for token in var.agent_tokens :
    coalesce(token.cluster, var.agent_token_cluster_name)
  ])

  name = each.value
}

# Buildkite agent tokens have no server-side expiry that Terraform can set, so
# rotation on this surface is "mint a second token, move the fleet, delete the
# first". There is deliberately NO keeper resource and NO replace_triggered_by
# here: a same-apply replacement destroys the incumbent before the operator can
# possibly have distributed the replacement secret, which strands every host that
# has not yet re-registered (TRAP 5). Rotation is expressed by ADDING a map entry
# and, one apply later, REMOVING the old one — two applies, with the host roll in
# between, so the dangerous half is an explicit destroy the operator has read.
resource "buildkite_cluster_agent_token" "scoped_lifecycle" {
  for_each = var.agent_tokens

  cluster_id = data.buildkite_cluster.token_target[
    coalesce(each.value.cluster, var.agent_token_cluster_name)
  ].id

  # The description is the only human-readable handle on a token in the console
  # and in `GET /v2/organizations/{org}/clusters/{uuid}/tokens`. Stamping the
  # generation into it makes "which token are the agents actually presenting?"
  # answerable during an incident, and makes a two-phase rotation legible in the
  # console: during phase 2 you will see gen 1 and gen 2 side by side, which is
  # the state you are supposed to be in until every host has rolled.
  description = "${each.value.description} [hth-3.1 gen ${each.value.rotation_id}]"

  # CIDR allowlist. Empty list = unrestricted; the precondition below refuses to
  # plan that by default. A wrong CIDR is equally dangerous in the other
  # direction — agents outside it cannot register at all — so derive this from
  # your runners' observed egress addresses, not from the VPC block you assume
  # they use.
  allowed_ip_addresses = each.value.allowed_ip_addresses

  lifecycle {
    # Retained for any replacement the PROVIDER forces (changing cluster_id moves
    # the token to a different cluster, which cannot be done in place): the new
    # token is minted before the old one is destroyed. It is NOT a rotation
    # safety net — TRAP 5 — because the ordering it guarantees is within one
    # apply, and the gap that actually matters is the human one between reading
    # the new secret and every agent host presenting it.
    create_before_destroy = true

    precondition {
      condition = !var.agent_token_require_ip_allowlist || length(each.value.allowed_ip_addresses) > 0
      error_message = join(" ", [
        "Agent token '${each.key}' declares no allowed_ip_addresses.",
        "Buildkite treats an empty allowlist as UNRESTRICTED: the token would register an agent from any source address.",
        "Set allowed_ip_addresses to your runners' egress CIDRs, or set agent_token_require_ip_allowlist = false to accept the risk deliberately."
      ])
    }
  }
}

# Phase-2 reminder, not a guard — the destructive half of a rotation is the
# operator deleting a map entry, and nothing here should try to stop that. What
# this DOES catch is the half that fails silently: a rotation started and never
# finished, leaving the superseded token live and registerable indefinitely.
# Two entries sharing a cluster and a description are two generations of one
# token, which is the correct state during a roll and a finding afterwards.
check "agent_token_rotations_are_completed" {
  assert {
    condition = length(distinct([
      for k, t in var.agent_tokens :
      format("%s|%s", coalesce(t.cluster, var.agent_token_cluster_name), t.description)
      if length([
        for k2, t2 in var.agent_tokens : k2
        if coalesce(t2.cluster, var.agent_token_cluster_name) == coalesce(t.cluster, var.agent_token_cluster_name)
        && t2.description == t.description
      ]) > 1
    ])) == 0
    error_message = format(
      "Rotation in flight: %s. Two or more agent_tokens entries share a cluster and a description, so more than one generation of the same token is registerable. That is the CORRECT state during phase 2 of a rotation (both tokens valid while the agent hosts roll onto the new secret). It is a finding once the roll is done: confirm every host has re-registered, then delete the superseded entry and apply again. Revoking a token does not disconnect agents already connected with it, so a forgotten old generation is a live registration credential nobody is watching.",
      jsonencode({
        for k, t in var.agent_tokens :
        k => format("%s [gen %s] in %s", t.description, t.rotation_id, coalesce(t.cluster, var.agent_token_cluster_name))
        if length([
          for k2, t2 in var.agent_tokens : k2
          if coalesce(t2.cluster, var.agent_token_cluster_name) == coalesce(t.cluster, var.agent_token_cluster_name)
          && t2.description == t.description
        ]) > 1
      })
    )
  }
}

Code Pack: API Script

hth-buildkite-3.01-agent-token-lifecycle.sh View source on GitHub ↗

# Read every cluster agent token in the organization and judge it against the
# two conditions the control actually requires. Run this first, and run it again
# after any change — it is the only surface that reports expiry at all.
#
#   BK-3.01a  expiresAt must not be null   (UI-created tokens are always null)
#   BK-3.01b  allowedIpAddresses must not be empty (empty == any source address)
audit_tokens() {
  local body
  body="$(jq -n --arg slug "${BUILDKITE_ORG_SLUG}" '{
    query: "query($slug:ID!){ organization(slug:$slug){ name
              clusters(first:100){ edges { node {
                id uuid name
                agentTokens(first:100){ count edges { node {
                  id uuid description expiresAt allowedIpAddresses
                } } }
              } } } } }",
    variables: { slug: $slug }
  }' | gql)"
  assert_no_errors "${body}" >/dev/null

  printf '%s' "${body}" | jq '
    [ .data.organization.clusters.edges[].node
      | .name as $cluster
      | .id   as $cluster_graphql_id
      | .uuid as $cluster_uuid
      | .agentTokens.edges[]?.node
      | {
          cluster: $cluster,
          cluster_graphql_id: $cluster_graphql_id,   # GraphQL mutations use this
          cluster_uuid: $cluster_uuid,               # REST paths use this
          token_id: .id,
          description: .description,
          expires_at: .expiresAt,
          allowed_ip_addresses: .allowedIpAddresses,
          "BK-3.01a_has_expiry": (.expiresAt != null),
          "BK-3.01b_ip_restricted":
            ((.allowedIpAddresses // "") | gsub("\\s";"") | length > 0)
        }
    ]
    | { total: length,
        failing_expiry:      [ .[] | select(."BK-3.01a_has_expiry"      | not) | .token_id ],
        failing_ip_allowlist:[ .[] | select(."BK-3.01b_ip_restricted"   | not) | .token_id ],
        tokens: . }'
}

# Same audit over REST, for tokens that only carry REST scopes. The cluster UUID
# — not the GraphQL ID — belongs in this path.
rest_audit_tokens() {
  local cluster_uuid="$1"
  curl -sS -H "Authorization: Bearer ${BUILDKITE_TOKEN}" \
    "${REST}/organizations/${BUILDKITE_ORG_SLUG}/clusters/${cluster_uuid}/tokens" \
  | jq '[ .[] | { id, description, expires_at, allowed_ip_addresses } ]'
}

# Portable RFC3339 timestamp N days out. GNU date and BSD/macOS date disagree on
# every relevant flag, so try both rather than shipping a Linux-only pack.
rfc3339_in_days() {
  local days="$1"
  date -u -d "+${days} days" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
    || date -u -v"+${days}d" +%Y-%m-%dT%H:%M:%SZ
}

to_epoch() {
  date -u -d "$1" +%s 2>/dev/null \
    || date -u -j -f "%Y-%m-%dT%H:%M:%SZ" "$1" +%s
}

# Enforce Buildkite's own floor locally, so a rejected mutation is not the first
# time you learn the timestamp was too close. The API refuses anything under ten
# minutes out, and refuses to change it afterwards.
assert_expiry_valid() {
  local expires_at="$1" target now
  target="$(to_epoch "${expires_at}")"
  now="$(date -u +%s)"
  if [ $(( target - now )) -lt 600 ]; then
    echo "expiresAt ${expires_at} is less than 10 minutes out; Buildkite rejects this and the value is immutable once accepted" >&2
    exit 1
  fi
}

# Create a token that is bounded in BOTH dimensions: time (expiresAt) and space
# (allowedIpAddresses). allowedIpAddresses is a single space-separated string of
# CIDRs here — "203.0.113.0/24 198.51.100.7/32" — unlike the Terraform list.
# An empty string means unrestricted, so this function refuses it.
create_token() {
  local cluster_graphql_id="$1" description="$2" allowed_ips="$3" expires_at="$4"

  [ -n "${allowed_ips// /}" ] || {
    echo "refusing to create an unrestricted token: pass a space-separated CIDR list" >&2
    exit 1
  }
  assert_expiry_valid "${expires_at}"

  local body
  body="$(jq -n \
    --arg org "$(org_id)" \
    --arg cluster "${cluster_graphql_id}" \
    --arg desc "${description}" \
    --arg ips "${allowed_ips}" \
    --arg exp "${expires_at}" '{
      query: "mutation($org:ID!,$cluster:ID!,$desc:String!,$ips:String,$exp:DateTime){
                clusterAgentTokenCreate(input:{
                  organizationId:$org, clusterId:$cluster, description:$desc,
                  allowedIpAddresses:$ips, expiresAt:$exp
                }){ tokenValue clusterAgentToken { id uuid description expiresAt allowedIpAddresses } } }",
      variables: { org:$org, cluster:$cluster, desc:$desc, ips:$ips, exp:$exp }
    }' | gql)"
  assert_no_errors "${body}" >/dev/null

  # tokenValue is returned here and nowhere else, ever. Route it to a secret
  # manager on this line rather than letting it land in a shell history file.
  printf '%s' "${body}" | jq '.data.clusterAgentTokenCreate'
}

# REST equivalent. Path takes the cluster UUID; expires_at is the same ISO8601
# value with the same 10-minute floor and the same immutability.
rest_create_token() {
  local cluster_uuid="$1" description="$2" allowed_ips="$3" expires_at="$4"
  assert_expiry_valid "${expires_at}"
  jq -n --arg d "${description}" --arg ips "${allowed_ips}" --arg exp "${expires_at}" \
      '{description:$d, allowed_ip_addresses:$ips, expires_at:$exp}' \
  | curl -sS -X POST \
      -H "Authorization: Bearer ${BUILDKITE_TOKEN}" \
      -H "Content-Type: application/json" \
      --data @- \
      "${REST}/organizations/${BUILDKITE_ORG_SLUG}/clusters/${cluster_uuid}/tokens"
}

# Revoke a token. ClusterAgentTokenRevokeInput requires organizationId as well as
# the token id — passing id alone is rejected.
revoke_token() {
  local token_id="$1" body
  body="$(jq -n --arg org "$(org_id)" --arg id "${token_id}" '{
    query: "mutation($org:ID!,$id:ID!){ clusterAgentTokenRevoke(input:{organizationId:$org,id:$id}){ deletedClusterAgentTokenId } }",
    variables: { org:$org, id:$id }
  }' | gql)"
  assert_no_errors "${body}" >/dev/null
  printf '%s' "${body}" | jq '.data.clusterAgentTokenRevoke'
}

# REST equivalent — returns 204 with an empty body, so read the status code and
# do not try to parse output that will not exist.
rest_revoke_token() {
  local cluster_uuid="$1" token_id="$2" code
  code="$(curl -sS -o /dev/null -w '%{http_code}' -X DELETE \
    -H "Authorization: Bearer ${BUILDKITE_TOKEN}" \
    "${REST}/organizations/${BUILDKITE_ORG_SLUG}/clusters/${cluster_uuid}/tokens/${token_id}")"
  [ "${code}" = "204" ] || { echo "revoke failed: HTTP ${code}" >&2; exit 1; }
  echo "revoked ${token_id} (HTTP 204)"
}

# CONTAINMENT. This is the step the guide's Step 4 has no mechanism for.
# Revoking a leaked token closes the door for new registrations; it leaves every
# agent that already walked through it running jobs. Stop those agents too.
# graceful=true lets in-flight jobs finish; pass "false" when you believe the
# agent itself is hostile and you want it gone mid-job.
#
# TWO WAYS THIS USED TO REPORT A CONTAINMENT IT HAD NOT PERFORMED, both closed:
#   1. `agents(first:500)` was unpaginated with no truncation guard, so a fleet
#      larger than one page was silently under-contained. AgentConnection exposes
#      `count` (NON_NULL Int) and `pageInfo`, so the page loop below walks
#      hasNextPage AND reconciles what it collected against the server's own
#      count — a mismatch aborts rather than proceeding on a partial list.
#   2. Each agentStop piped straight into `jq '.data.agentStop.agent // .errors'`.
#      Buildkite returns HTTP 200 with an `errors` array on application errors, so
#      a permission-scoped token or a per-agent rejection printed the error and
#      the loop continued, the function returned 0, and `contain` reported
#      success. Every stop is now checked and counted; the function returns
#      non-zero naming the agents it failed to stop.
fetch_cluster_agent_page() {
  local cluster_graphql_id="$1" after="$2"
  jq -n --arg slug "${BUILDKITE_ORG_SLUG}" --arg c "${cluster_graphql_id}" --arg after "${after}" '{
    query: "query($slug:ID!,$c:ID,$after:String){ organization(slug:$slug){ agents(first:100, cluster:$c, after:$after){ count edges { node { id name connectionState } } pageInfo { hasNextPage endCursor } } } }",
    variables: { slug:$slug, c:$c, after: (if $after == "" then null else $after end) }
  }' | gql
}

stop_cluster_agents() {
  local cluster_graphql_id="$1" graceful="${2:-true}"
  local after="" page agents="[]" reported_count collected id name stop_body
  local -a failed=()

  # NOTE: agents(cluster:) takes the base64 cluster ID. The UUID is rejected with
  # "An invalid ID was supplied" — verified against a live organization.
  while :; do
    page="$(fetch_cluster_agent_page "${cluster_graphql_id}" "${after}")"
    assert_no_errors "${page}" >/dev/null
    reported_count="$(printf '%s' "${page}" | jq -r '.data.organization.agents.count')"
    agents="$(jq -s 'add' \
      <(printf '%s' "${agents}") \
      <(printf '%s' "${page}" | jq '[.data.organization.agents.edges[].node]'))"
    [ "$(printf '%s' "${page}" | jq -r '.data.organization.agents.pageInfo.hasNextPage')" = "true" ] || break
    after="$(printf '%s' "${page}" | jq -r '.data.organization.agents.pageInfo.endCursor')"
  done

  # Truncation guard. Under-containment must never be silent: if the server says
  # there are more agents than the pages handed back, stop rather than contain a
  # subset and report success.
  collected="$(printf '%s' "${agents}" | jq 'length')"
  case "${reported_count}" in
    ''|*[!0-9]*)
      echo "FATAL: the API did not report an agent count for cluster ${cluster_graphql_id} (got '${reported_count}'). Without it there is no way to prove the fleet was fully enumerated, and an under-contained fleet must never be reported as contained." >&2
      return 1 ;;
  esac
  if [ "${collected}" -ne "${reported_count}" ]; then
    echo "FATAL: collected ${collected} agent(s) but the API reports ${reported_count} in cluster ${cluster_graphql_id}. Refusing to report a containment over a partial fleet — re-run, and stop the remainder by id." >&2
    return 1
  fi

  if [ "${collected}" -eq 0 ]; then
    echo "no agents connected to ${cluster_graphql_id}; revocation alone is sufficient"
    return 0
  fi

  while IFS=$'\t' read -r id name; do
    [ -n "${id}" ] || continue
    stop_body="$(jq -n --arg id "${id}" --argjson g "${graceful}" '{
      query: "mutation($id:ID!,$g:Boolean){ agentStop(input:{id:$id, graceful:$g}){ agent { id name connectionState } } }",
      variables: { id:$id, g:$g }
    }' | gql)"
    # A GraphQL error here is a FAILED STOP, not a datum to print and move past.
    if printf '%s' "${stop_body}" | jq -e '.errors // empty' >/dev/null 2>&1; then
      printf 'FAILED to stop %s (%s): %s\n' "${name}" "${id}" \
        "$(printf '%s' "${stop_body}" | jq -c '[.errors[].message]')" >&2
      failed+=( "${name}:${id}" )
      continue
    fi
    # A 200 with no error but no agent in the payload is also not a stop.
    if [ "$(printf '%s' "${stop_body}" | jq -r '.data.agentStop.agent // "null"')" = "null" ]; then
      printf 'FAILED to stop %s (%s): agentStop returned no agent\n' "${name}" "${id}" >&2
      failed+=( "${name}:${id}" )
      continue
    fi
    printf '%s' "${stop_body}" | jq -c '.data.agentStop.agent'
  done < <(printf '%s' "${agents}" | jq -r '.[] | [.id, .name] | @tsv')

  if [ "${#failed[@]}" -gt 0 ]; then
    echo "CONTAINMENT INCOMPLETE: ${#failed[@]} of ${collected} agent(s) were NOT stopped: ${failed[*]}. The token may be revoked, but these agents are still connected and still running jobs — revocation does not disconnect them (TRAP 1)." >&2
    return 4
  fi
  echo "stopped ${collected}/${collected} agent(s) in cluster ${cluster_graphql_id}"
}

# Full containment for a leaked token, in the order that actually contains it.
# Propagates the agent-stop verdict: a revoke that succeeded while agents kept
# running is NOT a containment and must not exit 0.
contain_leaked_token() {
  local cluster_graphql_id="$1" token_id="$2" graceful="${3:-true}" rc=0
  revoke_token "${token_id}"
  stop_cluster_agents "${cluster_graphql_id}" "${graceful}" || rc=$?
  return "${rc}"
}

3.2 Configure Agent Clusters

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	13.5
NIST 800-53	AC-17

Description

Isolate agents by environment or sensitivity using clusters, which are now the standard organizing unit for agents rather than an optional enhancement.

Deprecation: Unclustered Agents

Unclustered agents and unclustered agent tokens are deprecated, and organizations created after 2024-02-26 cannot use them at all — for those organizations every agent lives in a cluster by construction. Older organizations that still run unclustered agents should follow Buildkite’s documented migration path onto clusters rather than treating clustering as optional hardening. (Manage clusters)

Rationale

Why This Matters:

Separating production, development, and sensitive builds prevents a compromised low-trust agent from reaching production secrets
Targeting pipelines to specific clusters enforces a hard boundary that a malicious build cannot cross
Restricting production cluster access limits which jobs can touch deployment credentials and live systems
Cluster-level isolation contains the blast radius of any single compromised build host

Attack Prevented: Lateral movement, secret exfiltration, cross-environment contamination, privilege escalation

ClickOps Implementation

Step 1: Create Agent Clusters

Create separate clusters for:
- Production deployments
- Development builds
- Security-sensitive builds
Tag agents appropriately

Step 2: Configure Pipeline Targets

Target pipelines to specific clusters
Restrict production access
Audit cluster assignments

Code Implementation

Code Pack: Terraform

hth-buildkite-3.02-configure-agent-clusters.tf View source on GitHub ↗

# Create isolated agent clusters per environment. Control 3.2 is an L2 control.
#
# NO PROFILE-LEVEL GATE ON for_each, DELIBERATELY. This line used to read
# `var.profile_level >= 2 ? var.clusters : {}`. var.profile_level defaults to 1,
# so a single `terraform apply` that omitted `-var="profile_level=2"` emptied the
# map and Terraform DESTROYED every cluster — and a cluster destroy takes its
# queues and its cluster-scoped agent tokens with it, which is the whole fleet's
# registration surface. Those token secrets are returned by the create call
# exactly once, so they are not recoverable from state. Profile level selects
# WHAT you declare; it must never decide whether declared resources survive.
# var.clusters defaults to `{}`, so declaring nothing is already the "off" state
# and is the only one.
resource "buildkite_cluster" "clusters" {
  for_each = var.clusters

  name        = each.key
  description = each.value.description
  color       = each.value.color
  emoji       = each.value.emoji

  lifecycle {
    # Dropping a key here, or renaming a cluster (the map key IS the name), is a
    # fleet-wide event: the queues and the cluster agent tokens go with it.
    # Refuse it at plan time rather than discovering it in an applied diff. To
    # retire a cluster on purpose: delete this line, apply, restore it.
    prevent_destroy = true
  }
}

# Create cluster queues for workload routing.
#
# Same reasoning as above: no profile gate. An empty var.cluster_queues — the
# default — is the off state. No prevent_destroy here, because a queue is
# recreatable from its key; note only that destroying one strands every agent
# configured to target that queue until it comes back.
resource "buildkite_cluster_queue" "queues" {
  for_each = var.cluster_queues

  cluster_id  = buildkite_cluster.clusters[each.value.cluster_key].id
  key         = each.value.key
  description = each.value.description
}

3.3 Secure Agent Infrastructure

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	4.1
NIST 800-53	CM-6

Description

Secure agent host infrastructure.

Rationale

Why This Matters:

Agents execute arbitrary pipeline code, so a hardened host is the primary defense against builds being used to attack your network
Ephemeral agents destroy any attacker foothold after each job, preventing persistence and cross-build contamination
Minimizing installed software and applying OS hardening reduces the exploitable attack surface on every build host
Restricting agent network access prevents a compromised build from pivoting to internal systems or exfiltrating data

Attack Prevented: Build host compromise, persistence, lateral movement, data exfiltration

ClickOps Implementation

Step 1: Harden Agent Hosts

Use ephemeral agents where possible
Minimize installed software
Apply OS hardening

Step 2: Network Security

Restrict agent network access
Use private networks
Monitor agent traffic

Code Implementation

Code Pack: Terraform

hth-buildkite-3.03-secure-agent-infrastructure.tf View source on GitHub ↗

# One cluster per trust boundary. Pipelines that handle production credentials do
# not share a cluster with pipelines that build untrusted contributor branches.
resource "buildkite_cluster" "isolated" {
  for_each = var.agent_clusters

  name        = each.key
  description = each.value.description
  emoji       = each.value.emoji
  color       = each.value.color
}

# Queues partition work WITHIN a cluster. dispatch_paused lets you stop a queue
# taking new jobs during an incident without deleting it and losing its config.
resource "buildkite_cluster_queue" "isolated" {
  for_each = var.agent_queues

  cluster_id      = buildkite_cluster.isolated[each.value.cluster].id
  key             = each.value.key
  description     = each.value.description
  dispatch_paused = each.value.paused
}

# Cluster-scoped registration tokens. A token leaked from one cluster cannot
# enroll an agent into another.
resource "buildkite_cluster_agent_token" "scoped" {
  for_each = var.agent_clusters

  cluster_id  = buildkite_cluster.isolated[each.key].id
  description = "HTH 3.3 — scoped registration token for cluster ${each.key}"

  # Empty list = no IP restriction. Set var.agent_allowed_cidrs only when you
  # know every runner's egress address; a wrong entry strands the whole cluster.
  allowed_ip_addresses = var.agent_allowed_cidrs
}

output "agent_cluster_tokens" {
  description = "Cluster registration tokens. Treat as credentials — write straight to your secret store, never to a log."
  sensitive   = true
  value = {
    for k, t in buildkite_cluster_agent_token.scoped : k => t.token
  }
}

3.4 Enable Pipeline Signing and Verification

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	2.7
NIST 800-53	SI-7

Description

Cryptographically sign pipeline steps so agents will only execute a pipeline whose commands are provably the ones Buildkite was given, rejecting anything altered in transit or injected along the way.

Rationale

Why This Matters:

Without signing, an agent executes whatever step definition reaches it, and the agent has no way to distinguish a legitimate command from a substituted one
Signing moves the trust boundary to a key you control rather than to the integrity of every system between the pipeline definition and the agent
Verification happens on the agent, which is the last point before execution and therefore the only place the check cannot be bypassed by an earlier compromise
This is not on by default. An organization that has never configured signing keys is running unverified pipelines, which is easy to miss because nothing about the build output looks different

Attack Prevented: Command injection into pipeline steps, tampering with step definitions in transit, execution of unauthorized build commands, supply-chain compromise between pipeline definition and agent

ClickOps Implementation

Step 1: Choose a Key Backend

JWKS file: generate a key set with buildkite-agent tool keygen, then configure the agent with signing-jwks-file and signing-jwks-key-id on uploading agents, and verification-jwks-file on executing agents. Note: --jwks-file and --jwks-key-id are flags on the buildkite-agent tool sign command, not agent configuration keys. Setting them where an agent config key belongs does nothing.
AWS KMS: keep the private key in a managed KMS so it is never on the agent host at all, via signing-aws-kms-key. On GCP KMS: the agent configuration reference lists a signing-gcp-kms-key option, but Buildkite’s signed-pipelines documentation describes AWS KMS only. Confirm current support with Buildkite before designing around GCP KMS.

Step 2: Enable Signing and Verification

Configure the signing key on the agents responsible for uploading pipelines, and the verification key on every agent that executes steps.
Configuring verification-jwks-file is what turns verification on. With no verification key the agent has nothing to check an unsigned job against, so an agent with verification-failure-behavior=block and no key configured executes unsigned jobs anyway. Setting the behavior without the key is security theater. A keyless agent is not inert, though: a job that arrives already signed is rejected under the default block, because the agent treats a signature it cannot verify as a verification failure rather than as an absent one. Verification is gated on the key only for unsigned work — which is precisely the work you were trying to catch.
verification-failure-behavior defaults to block, so a partial rollout fails closed in both directions and neither direction is the safe one. An agent that has a verification key but whose uploaders are not yet signing will fail every build. The mirror case fails the same way: once uploaders start signing, any executing agent that has not yet been given verification-jwks-file rejects those signed jobs outright. The rollout therefore runs in the opposite direction to the usual pattern — set warn on every agent first, distribute keys to uploaders and executors, then remove the override to return to block once both halves are complete.

Step 3: Keep Signing Diagnostics Out of Production

Do not leave --debug-signing enabled. It writes signing diagnostics into build logs and can expose secret values there. Use it to troubleshoot a rollout and turn it off immediately afterward.

Code Implementation

Code Pack: Config

hth-buildkite-3.04-verification.sh View source on GitHub ↗

# Install the public keyset and pin the agent to the secure default.
# Run on every agent that EXECUTES jobs, not just the ones that upload pipelines.

# T2: reject private key material. 'd' is the private parameter for Ed25519,
# EC and RSA keys alike, so this one test covers EdDSA, ES512 and PS512.
assert_public_only() {
  local path="$1"
  [ -r "${path}" ] || { echo "FATAL: keyset '${path}' not readable." >&2; exit 4; }
  jq -e 'if (.keys | type) != "array" then false
         else ([.keys[] | has("d")] | any | not) end' "${path}" >/dev/null || {
    echo "FATAL: '${path}' contains PRIVATE key material (JWK parameter 'd')." >&2
    echo "       Install the --public-jwks-file output here, never the private one." >&2
    exit 4
  }
}

# ATOMIC CONFIG REPLACEMENT. buildkite-agent.cfg carries this host's registration
# token; a truncate-then-write (`cat "${tmp}" >"${AGENT_CFG}"`) interrupted by a
# signal, a full disk, or a set -e abort leaves a config the agent cannot parse,
# the agent then fails to start, and the host silently leaves the fleet. Every
# mutation below stages a sibling file and rename(2)s it into place instead, so a
# reader sees the old config or the new one, never a partial one.
#   * the temp lives in the target's OWN directory — rename(2) is EXDEV across
#     filesystems and mv would fall back to a non-atomic copy
#   * `cp -p` seeds it from the incumbent so mode, ownership and times ride onto
#     the replacement inode; a bare mktemp would hand the agent mktemp's 0600
#   * AGENT_CFG is resolved through symlinks first, because this path is commonly
#     a link into a config-management tree and renaming over the link would
#     replace it with a regular file
#   * no .bak is written: a persistent second copy of the agent token on disk is
#     a worse trade than the failure mode the rename already removes
# The trap is what keeps that staged full copy of the config — token included —
# out of the directory when a run dies mid-mutation.
HTH_CFG_TMP=""
HTH_CFG_DST=""
hth_cfg_cleanup() {
  if [ -n "${HTH_CFG_TMP}" ]; then rm -f "${HTH_CFG_TMP}"; fi
  HTH_CFG_TMP=""
}
trap hth_cfg_cleanup EXIT
trap 'hth_cfg_cleanup; exit 130' INT
trap 'hth_cfg_cleanup; exit 143' TERM

# Real path of the config, following symlinks where the platform's readlink can.
hth_cfg_path() {
  if [ -L "${AGENT_CFG}" ]; then
    readlink -f "${AGENT_CFG}" 2>/dev/null || printf '%s\n' "${AGENT_CFG}"
  else
    printf '%s\n' "${AGENT_CFG}"
  fi
}

# Stage a writable copy beside the target: HTH_CFG_DST is the real path to read
# from and commit to, HTH_CFG_TMP is the staged file to write into.
# This assigns globals rather than printing a value on purpose. Written as
# `dst="$(hth_cfg_stage)"` the function would run in a SUBSHELL, the parent's
# HTH_CFG_TMP would stay empty, and the cleanup trap would never see — or remove
# — the full copy of the token-bearing config that mktemp just created.
hth_cfg_stage() {
  HTH_CFG_DST="$(hth_cfg_path)"
  HTH_CFG_TMP="$(mktemp "$(dirname "${HTH_CFG_DST}")/.hth-bk-cfg.XXXXXX")"
  cp -p "${HTH_CFG_DST}" "${HTH_CFG_TMP}"
}

# Atomic swap. After this returns there is no temp left for the trap to clean up.
hth_cfg_commit() {
  mv -f "${HTH_CFG_TMP}" "${HTH_CFG_DST}"
  HTH_CFG_TMP=""
}

# Idempotent upsert of a single `key=value` line in buildkite-agent.cfg.
set_cfg() {
  local key="$1" value="$2" tmp dst
  hth_cfg_stage; dst="${HTH_CFG_DST}"; tmp="${HTH_CFG_TMP}"
  if grep -qE "^[[:space:]]*${key}[[:space:]]*=" "${dst}"; then
    sed -E "s|^[[:space:]]*${key}[[:space:]]*=.*$|${key}=${value}|" "${dst}" >"${tmp}"
  else
    cat "${dst}" >"${tmp}"
    printf '%s=%s\n' "${key}" "${value}" >>"${tmp}"
  fi
  hth_cfg_commit
}

unset_cfg() {
  local key="$1" tmp dst
  hth_cfg_stage; dst="${HTH_CFG_DST}"; tmp="${HTH_CFG_TMP}"
  grep -vE "^[[:space:]]*${key}[[:space:]]*=" "${dst}" >"${tmp}" || true
  hth_cfg_commit
}

# Step 1 + 4: the end state. verification-failure-behavior=block is written
# explicitly so the posture is auditable on disk rather than inherited silently.
apply_block() {
  [ -w "${AGENT_CFG}" ] || { echo "FATAL: cannot write ${AGENT_CFG} (run as root)." >&2; exit 5; }
  assert_public_only "${VERIFICATION_JWKS}"
  chmod 0644 "${VERIFICATION_JWKS}"

  set_cfg "verification-jwks-file" "${VERIFICATION_JWKS}"
  set_cfg "verification-failure-behavior" "block"

  echo "Applied. Restart the agent, then run: $0 audit"
}

# Step 2: the DELIBERATE LOOSENING for a staged rollout. This is strictly less
# secure than the default. The keyset is still installed first — without it the
# warn posture reports nothing at all on unsigned jobs.
apply_warn() {
  [ -w "${AGENT_CFG}" ] || { echo "FATAL: cannot write ${AGENT_CFG} (run as root)." >&2; exit 5; }
  assert_public_only "${VERIFICATION_JWKS}"

  set_cfg "verification-jwks-file" "${VERIFICATION_JWKS}"
  set_cfg "verification-failure-behavior" "warn"

  echo "WARNING: this agent now EXECUTES jobs that fail signature verification."
  echo "         This is a temporary rollout state. Run '$0 apply-block' to restore"
  echo "         the vendor default once every uploader is signing."
}

# Container / env-var equivalent. T1: note the two env var names do not follow
# the same pattern as each other, and neither matches its config key exactly.
print_env_equivalent() {
  cat <<ENVEOF
# buildkite-agent v3 — verification via environment (containers, systemd units)
BUILDKITE_AGENT_VERIFICATION_JWKS_FILE=${VERIFICATION_JWKS}
BUILDKITE_AGENT_JOB_VERIFICATION_NO_SIGNATURE_BEHAVIOR=block
# NOTE: BUILDKITE_AGENT_VERIFICATION_FAILURE_BEHAVIOR is NOT a real variable.
ENVEOF
}

# Prove the control is real. Fails closed on the security-theater combination:
# a verification-failure-behavior with no verification key to enforce it with.
audit_verification() {
  local jwks behavior rc=0
  jwks="$(sed -nE 's|^[[:space:]]*verification-jwks-file[[:space:]]*=[[:space:]]*(.*)$|\1|p' "${AGENT_CFG}" | tail -1)"
  behavior="$(sed -nE 's|^[[:space:]]*verification-failure-behavior[[:space:]]*=[[:space:]]*(.*)$|\1|p' "${AGENT_CFG}" | tail -1)"

  # The agent default is block, so an absent key is NOT an absent posture.
  [ -n "${behavior}" ] || behavior="block (agent default)"

  echo "config file                   : ${AGENT_CFG}"
  echo "verification-jwks-file        : ${jwks:-<unset>}"
  echo "verification-failure-behavior : ${behavior}"

  if [ -z "${jwks}" ]; then
    echo "FAIL: no verification key configured. run_job.go only verifies when" >&2
    echo "      r.conf.JWKS != nil — an UNSIGNED job runs here regardless of" >&2
    echo "      verification-failure-behavior. This agent enforces nothing." >&2
    rc=1
  else
    if [ ! -r "${jwks}" ]; then
      echo "FAIL: verification-jwks-file '${jwks}' is missing or unreadable." >&2
      rc=1
    else
      assert_public_only "${jwks}"
      echo "PASS: verification key present and contains no private material."
    fi
  fi

  # The behavior verdict is only meaningful when a key exists. Reporting
  # "unverifiable jobs are rejected" on a keyless agent would restate the very
  # false assurance this pack exists to remove.
  case "${behavior}" in
    block*)
      if [ -n "${jwks}" ]; then
        echo "PASS: unverifiable jobs are rejected."
      else
        echo "FAIL: 'block' is INERT here — with no key, unsigned jobs bypass" >&2
        echo "      both branches of run_job.go and execute." >&2
      fi ;;
    warn)   echo "WARN: unverifiable jobs EXECUTE. Rollout state only — restore 'block'." ;;
    *)      echo "FAIL: unrecognised behavior '${behavior}' (expected warn|block)." >&2; rc=1 ;;
  esac

  # T3: KMS-backed agents verify with the KMS key, not a JWKS file.
  if grep -qE '^[[:space:]]*signing-(aws|gcp)-kms-key[[:space:]]*=' "${AGENT_CFG}" && [ -n "${jwks}" ]; then
    echo "FAIL: both a KMS key and verification-jwks-file are configured. Pick one" >&2
    echo "      backend — this is a misconfiguration, not defence in depth." >&2
    rc=1
  fi

  return "${rc}"
}

Code Pack: CLI Script

hth-buildkite-3.04-pipeline-signing.sh View source on GitHub ↗

# Generate the signing key pair. EdDSA is the agent default; PS512 and ES512 are
# the only other accepted algorithms (jwkutil.ValidSigningAlgorithms).
require_agent() {
  command -v "${BK_AGENT}" >/dev/null 2>&1 || {
    echo "FATAL: '${BK_AGENT}' not on PATH. Install buildkite-agent v3." >&2
    exit 127
  }
  command -v jq >/dev/null 2>&1 || { echo "FATAL: jq required." >&2; exit 127; }
}

# T8: never write private key material into a path git will track.
assert_ignored() {
  local path="$1"
  git rev-parse --is-inside-work-tree >/dev/null 2>&1 || return 0
  git check-ignore -q "${path}" && return 0
  echo "REFUSING: '${path}' is not git-ignored. Add '${KEY_DIR}/' to" >&2
  echo "          .gitignore before generating signing keys." >&2
  exit 3
}

# T7: a JWK carrying the private parameter 'd' is private material for every
# algorithm the agent accepts (Ed25519, EC, RSA). A verification keyset must
# never contain one.
assert_public_only() {
  local path="$1"
  jq -e 'if (.keys | type) != "array" then false
         else ([.keys[] | has("d")] | any | not) end' "${path}" >/dev/null || {
    echo "FATAL: '${path}' contains private key material (JWK parameter 'd')." >&2
    echo "       Never distribute this file as verification-jwks-file." >&2
    exit 4
  }
}

keygen() {
  require_agent
  case "${KEY_ALG}" in
    EdDSA|PS512|ES512) ;;
    *) echo "FATAL: --alg must be EdDSA, PS512 or ES512 (got '${KEY_ALG}')." >&2; exit 2 ;;
  esac

  mkdir -p "${KEY_DIR}"
  chmod 700 "${KEY_DIR}"
  assert_ignored "${PRIVATE_JWKS}"

  "${BK_AGENT}" tool keygen \
    --alg "${KEY_ALG}" \
    --key-id "${KEY_ID}" \
    --private-jwks-file "${PRIVATE_JWKS}" \
    --public-jwks-file "${PUBLIC_JWKS}"

  chmod 600 "${PRIVATE_JWKS}"
  chmod 644 "${PUBLIC_JWKS}"
  assert_public_only "${PUBLIC_JWKS}"

  echo "private (signers only, never distribute): ${PRIVATE_JWKS}"
  echo "public  (-> agent verification-jwks-file): ${PUBLIC_JWKS}"
}

# Sign pipeline steps with the self-managed JWKS private key.
#
# Two distinct modes, and the flags are NOT interchangeable:
#   offline  - signs the local YAML file. Requires --repo (T2). Rejects any
#              pipeline containing $ interpolations (T3). Prints the signed
#              pipeline to stdout; nothing is uploaded.
#   publish  - supplies the GraphQL token (via the environment, never argv — T9),
#              so the tool downloads the pipeline and repo URL from Buildkite and
#              IGNORES the local file entirely (T1), then writes the signed
#              definition back with --update.
sign_offline() {
  local pipeline_file="${1:?path to pipeline YAML required}"
  require_agent
  : "${BUILDKITE_REPO:?set BUILDKITE_REPO to the pipeline repository URL (bound into the signature)}"
  [ -r "${PRIVATE_JWKS}" ] || { echo "FATAL: no signing key at ${PRIVATE_JWKS}; run keygen." >&2; exit 5; }

  # `env -u` is load-bearing, not tidiness. tool_sign.go:110-112 binds
  # --graphql-token to BUILDKITE_GRAPHQL_TOKEN, and :218 selects signWithGraphQL
  # on cfg.GraphQLToken being non-empty REGARDLESS of which source filled it. So
  # a token merely present in this shell's environment silently converts an
  # offline signing run into a GraphQL one — T1, reached without ever typing the
  # flag. Strip it from the child's environment so `offline` means offline.
  env -u BUILDKITE_GRAPHQL_TOKEN "${BK_AGENT}" tool sign \
    --jwks-file "${PRIVATE_JWKS}" \
    --jwks-key-id "${KEY_ID}" \
    --repo "${BUILDKITE_REPO}" \
    "${pipeline_file}"
}

sign_and_publish() {
  require_agent
  : "${BUILDKITE_GRAPHQL_TOKEN:?set BUILDKITE_GRAPHQL_TOKEN (GraphQL token with write_pipelines)}"
  : "${BUILDKITE_ORGANIZATION_SLUG:?set BUILDKITE_ORGANIZATION_SLUG}"
  : "${BUILDKITE_PIPELINE_SLUG:?set BUILDKITE_PIPELINE_SLUG}"
  [ -r "${PRIVATE_JWKS}" ] || { echo "FATAL: no signing key at ${PRIVATE_JWKS}; run keygen." >&2; exit 5; }

  # T9: the write-scoped token is NEVER passed as a flag. Anything in argv is
  # readable via `ps` by every user on the host and lands in process accounting
  # and audit logs. tool_sign.go:110-112 binds --graphql-token to
  # BUILDKITE_GRAPHQL_TOKEN, and :218 picks signWithGraphQL on cfg.GraphQLToken
  # being non-empty whichever source filled it — so the env-only form is the
  # identical code path with the secret off the process table. The export is what
  # makes the child inherit it: the :? guard above proves the variable is SET,
  # not that it is EXPORTED.
  export BUILDKITE_GRAPHQL_TOKEN

  local -a args=(
    tool sign
    --jwks-file "${PRIVATE_JWKS}"
    --jwks-key-id "${KEY_ID}"
    --organization-slug "${BUILDKITE_ORGANIZATION_SLUG}"
    --pipeline-slug "${BUILDKITE_PIPELINE_SLUG}"
    --update
  )
  # T4: --update prompts on a TTY. In CI there is no TTY to answer it.
  [ -t 0 ] || args+=( --no-confirm )
  # T6: opt-in only; this flag prints every step in full and can leak secrets.
  [ "${HTH_ALLOW_DEBUG_SIGNING:-0}" = "1" ] && args+=( --debug-signing )

  "${BK_AGENT}" "${args[@]}"
}

# AWS KMS backend: the private key never lands on the signing host.
# T5: setting --signing-aws-kms-key makes --jwks-file dead weight — the agent
# selects AWS KMS first and never reads the file. Pass one backend, not both.
# Agents verifying these signatures configure signing-aws-kms-key (the same key
# serves both directions for the KMS backend), not verification-jwks-file.
sign_with_aws_kms() {
  require_agent
  : "${BUILDKITE_SIGNING_AWS_KMS_KEY:?set BUILDKITE_SIGNING_AWS_KMS_KEY (KMS key id or alias)}"
  : "${BUILDKITE_GRAPHQL_TOKEN:?set BUILDKITE_GRAPHQL_TOKEN (GraphQL token with write_pipelines)}"
  : "${BUILDKITE_ORGANIZATION_SLUG:?set BUILDKITE_ORGANIZATION_SLUG}"
  : "${BUILDKITE_PIPELINE_SLUG:?set BUILDKITE_PIPELINE_SLUG}"

  # T9 again: token via the environment binding, never argv. See sign_and_publish.
  export BUILDKITE_GRAPHQL_TOKEN

  local -a args=(
    tool sign
    --signing-aws-kms-key "${BUILDKITE_SIGNING_AWS_KMS_KEY}"
    --organization-slug "${BUILDKITE_ORGANIZATION_SLUG}"
    --pipeline-slug "${BUILDKITE_PIPELINE_SLUG}"
    --update
  )
  [ -t 0 ] || args+=( --no-confirm )

  # Standard AWS SDK credential resolution applies; prefer an instance/OIDC role
  # over static keys so the signing identity is short-lived.
  "${BK_AGENT}" "${args[@]}"
}

Sources: Signed pipelines · buildkite-agent tool

3.5 Manage Build Secrets

Profile Level: L1 (Crawl)

Framework	Control
CIS Controls	3.11
NIST 800-53	SC-12, SC-28

Description

Deliver secrets to builds through a managed secret store rather than environment variables baked into agent hosts, pipeline settings, or repositories.

Rationale

Why This Matters:

Secrets set as plain pipeline or agent environment variables are visible to every step in the build, including third-party plugins, and frequently end up echoed into logs
A secret stored on the agent host outlives the job, so one compromised build can harvest credentials belonging to every other pipeline that agent serves
Cluster-scoped storage means a secret is reachable only by agents in that cluster, which turns the cluster boundary established in 3.2 into a secrets boundary as well
Automatic log redaction removes the most common exposure path — an accidental echo — without relying on every pipeline author to be careful

Attack Prevented: Secret leakage through build logs, credential theft from agent hosts, cross-pipeline secret exposure, hardcoded credentials in repositories

Prerequisites

Buildkite agent v3.106.0 or later for Buildkite-managed secrets.

ClickOps Implementation

Step 1: Prefer an External Secret Service Where You Have One

Buildkite’s own first recommendation is to use an external secrets service — HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, or equivalent — retrieved at job time. If you already run one, that remains the primary path, and Buildkite secrets are the option for teams that do not.

Step 2: Use Cluster-Scoped Buildkite Secrets

Create secrets on the cluster that needs them. Each cluster has its own encryption key, and secrets are encrypted in transit and at rest.
Values are automatically redacted from build logs.
Work within the documented limits: keys up to 255 characters and keys are immutable — changing a key name means creating a new secret and deleting the old one. On value size: Buildkite’s secrets documentation states 32 KB while the Terraform provider documentation states 8 KB. The two have not been reconciled; design against the lower figure until Buildkite confirms which applies to your path.

Step 3: Never Put Secrets in Pipeline Settings

Do not place secrets in a pipeline’s env block, step configuration, or pipeline settings. Buildkite states plainly that pipeline settings are often returned in REST and GraphQL API payloads — so a secret there is exposed to every token that can read the pipeline, not only to the build.
This is a separate exposure path from log echo, and log redaction does not close it.
Where a literal $ must survive into a command, escape it as $$. An unescaped $ is interpolated at upload time, which both breaks the value and can print it.
Uploaded pipelines are visible in the build’s timeline, so a secret interpolated into an uploaded step is exposed there too.

Step 4: Scope and Review

Keep production secrets in the production cluster only, so a development agent has no path to them.
Review secrets alongside the cluster’s agent tokens, since the two together define what a compromised agent can reach.

Code Implementation

Code Pack: Terraform

hth-buildkite-3.05-cluster-secrets.tf View source on GitHub ↗

locals {
  # Claims Buildkite generates itself. A rule anchored on one of these cannot be
  # satisfied by naming a branch or reusing a slug (TRAP 5).
  hth_first_party_claims = ["pipeline_id", "build_source", "cluster_queue_id"]

  # Claims whose values originate with users or third-party tools.
  hth_third_party_claims = [
    "pipeline_slug", "build_branch", "build_creator",
    "build_creator_team", "cluster_queue_key",
  ]

  hth_known_claims = concat(local.hth_first_party_claims, local.hth_third_party_claims)

  # Every cluster that holds a declared secret. Resolved by NAME so the uuid this
  # resource requires is never hand-copied from a queue's GraphQL id (TRAP 3).
  hth_secret_clusters = toset([for s in var.cluster_secrets : s.cluster])

  # Per-secret policy analysis, computed once and reused by the resource
  # preconditions and by the audit checks below.
  hth_secret_policy = {
    for k, s in var.cluster_secrets : k => {
      # Every claim key used anywhere in the policy, for typo detection.
      claims_used = distinct(flatten([for rule in s.policy_rules : keys(rule)]))

      # Rules that name no first-party claim at all. Each is a boundary that a
      # branch push or a recreated pipeline slug can walk through.
      third_party_only_rules = [
        for i, rule in s.policy_rules : i
        if length(setintersection(toset(keys(rule)), toset(local.hth_first_party_claims))) == 0
      ]

      # A rule with no claims would match every build. Terraform's type system
      # permits the empty map, so it is rejected here.
      empty_rules = [for i, rule in s.policy_rules : i if length(keys(rule)) == 0]

      # TRAP 7b: how many declarations target this same (cluster, key) pair.
      # The map label makes the Terraform address unique; nothing makes the
      # Buildkite key unique, and two entries claiming one key in one cluster is
      # two applies overwriting each other's credential.
      key_collisions = [
        for k2, s2 in var.cluster_secrets : k2
        if s2.cluster == s.cluster && s2.key == s.key
      ]
    }
  }
}

data "buildkite_cluster" "secret_scoped" {
  for_each = local.hth_secret_clusters

  name = each.value
}

# Cluster-scoped secrets. The value never enters Terraform state or a plan file:
# it arrives through the ephemeral var.cluster_secret_values map and leaves
# through the write-only value_wo argument.
resource "buildkite_cluster_secret" "managed" {
  # TRAP 1: this map is deliberately NOT sensitive. It carries no secret values —
  # only the cluster, version marker and policy — so it is legal to iterate.
  for_each = var.cluster_secrets

  cluster_id = data.buildkite_cluster.secret_scoped[each.value.cluster].uuid

  # TRAP 7b: the DECLARED key, never the map label. The label is a Terraform
  # address; this is the name Buildkite creates the secret under and the name a
  # pipeline references. It is immutable — changing it is a create-plus-delete.
  key = each.value.key

  description = each.value.description

  # TRAP 1 + the reason this pack exists. Indexed inside the resource body, never
  # in for_each. Ephemeral input -> write-only argument: no plan file, no state.
  value_wo = var.cluster_secret_values[each.key]

  # TRAP 2: string, and the ONLY signal Terraform has that the value changed.
  value_wo_version = each.value.value_wo_version

  # TRAP 6: a YAML list of rules, OR'd. Built from a typed structure so a
  # mistyped claim fails the plan instead of becoming an inert line of YAML.
  policy = yamlencode(each.value.policy_rules)

  lifecycle {
    # TRAP 4: refuse to create a secret with no policy.
    precondition {
      condition     = length(each.value.policy_rules) > 0
      error_message = format("Secret '%s' (Buildkite key '%s') declares no policy_rules. A cluster secret with no access policy is not a documented state — Buildkite does not specify whether it is reachable by every build in cluster '%s'. Declare the pipelines that may read it.", each.key, each.value.key, each.value.cluster)
    }

    # TRAP 6, degenerate case: a claimless rule constrains nothing, and because
    # rules are OR'd it makes every other rule in the policy irrelevant.
    precondition {
      condition     = length(local.hth_secret_policy[each.key].empty_rules) == 0
      error_message = format("Secret '%s' (Buildkite key '%s') has policy rule(s) at index %s with no claims. Rules are OR'd, so a claimless rule matches every build and supersedes every other rule in the policy.", each.key, each.value.key, join(", ", [for i in local.hth_secret_policy[each.key].empty_rules : tostring(i)]))
    }

    # A misspelled claim is the worst failure mode available here: it reads like
    # a restriction and enforces nothing.
    precondition {
      condition     = length(setsubtract(toset(local.hth_secret_policy[each.key].claims_used), toset(local.hth_known_claims))) == 0
      error_message = format("Secret '%s' (Buildkite key '%s') uses unrecognised policy claim(s): %s. Buildkite implements exactly these: first-party %s; third-party %s.", each.key, each.value.key, join(", ", tolist(setsubtract(toset(local.hth_secret_policy[each.key].claims_used), toset(local.hth_known_claims)))), join(", ", local.hth_first_party_claims), join(", ", local.hth_third_party_claims))
    }

    # TRAP 5, enforced per secret when the operator opts in.
    precondition {
      condition     = !var.require_first_party_claim || length(local.hth_secret_policy[each.key].third_party_only_rules) == 0
      error_message = format("Secret '%s' (Buildkite key '%s') has policy rule(s) at index %s built only from third-party claims, whose values are supplied by users or third-party tools. Anchor each rule on pipeline_id, build_source or cluster_queue_id, or set require_first_party_claim = false to accept the softer boundary deliberately.", each.key, each.value.key, join(", ", [for i in local.hth_secret_policy[each.key].third_party_only_rules : tostring(i)]))
    }

    # TRAP 7 + 7b: the key is immutable, so a typo costs a create-and-delete
    # cycle — and the string validated here is the DECLARED key that Buildkite
    # will actually receive, not the map label. Validating the label would pass a
    # well-formed address while creating a malformed credential.
    precondition {
      condition     = can(regex("^[A-Za-z][A-Za-z0-9_]{0,254}$", each.value.key))
      error_message = format("Secret '%s' declares an invalid Buildkite key '%s'. Keys must start with a letter, contain only letters, numbers and underscores, and be at most 255 characters. The key cannot be changed after creation.", each.key, each.value.key)
    }

    precondition {
      condition     = !startswith(lower(each.value.key), "buildkite") && !startswith(lower(each.value.key), "bk")
      error_message = format("Secret '%s' declares the Buildkite key '%s', which uses a reserved prefix. Keys must not begin with 'buildkite' or 'bk' in any casing.", each.key, each.value.key)
    }

    # TRAP 7b: the map label guarantees a unique Terraform address; nothing
    # guarantees a unique Buildkite key. Two declarations naming one key in one
    # cluster are two resources writing the same credential.
    precondition {
      condition     = length(local.hth_secret_policy[each.key].key_collisions) == 1
      error_message = format("Buildkite key '%s' in cluster '%s' is declared by %d entries of var.cluster_secrets: %s. The map label is a Terraform address and does not make the key unique — these declarations would overwrite one another's value and policy. Give each secret its own key.", each.value.key, each.value.cluster, length(local.hth_secret_policy[each.key].key_collisions), join(", ", local.hth_secret_policy[each.key].key_collisions))
    }

    # TRAP 2, the other half: the provider requires a non-empty version marker.
    precondition {
      condition     = trimspace(each.value.value_wo_version) != ""
      error_message = format("Secret '%s' (Buildkite key '%s') has an empty value_wo_version. It is required whenever value_wo is set, and it is the only change Terraform can detect when the secret value rotates.", each.key, each.value.key)
    }
  }
}

# The preconditions above govern secrets this configuration creates. These checks
# report on the shape of the estate every plan and apply without blocking an
# unrelated deploy — the standing question at review time is not "did it apply"
# but "how wide is the widest rule".

# TRAP 5 as a reporting control, so the finding is visible even when
# require_first_party_claim is deliberately false.
check "secret_policies_anchored_on_first_party_claims" {
  assert {
    condition = length([
      for k, p in local.hth_secret_policy : k if length(p.third_party_only_rules) > 0
    ]) == 0
    error_message = format(
      "%d secret(s) have at least one policy rule built only from third-party claims (values supplied by users or third-party tools, per Buildkite): %s. A pipeline_slug is reusable by a recreated pipeline and a build_branch is created by anyone who can push, so these rules do not bind the way they read.",
      length([for k, p in local.hth_secret_policy : k if length(p.third_party_only_rules) > 0]),
      jsonencode({ for k, p in local.hth_secret_policy : k => p.third_party_only_rules if length(p.third_party_only_rules) > 0 }),
    )
  }
}

# A secret every pipeline can read is the state control 3.2's cluster boundary
# was built to prevent — it re-broadens the blast radius inside the cluster.
check "secret_policies_are_scoped_to_pipelines" {
  assert {
    condition = length([
      for k, s in var.cluster_secrets : k
      if !contains(local.hth_secret_policy[k].claims_used, "pipeline_id")
      && !contains(local.hth_secret_policy[k].claims_used, "pipeline_slug")
    ]) == 0
    error_message = format(
      "%d secret(s) carry a policy that never names a pipeline: %s. Every build in the cluster that satisfies the remaining claims can read them. Prefer pipeline_id — it survives a pipeline being deleted and recreated under the same slug.",
      length([for k, s in var.cluster_secrets : k if !contains(local.hth_secret_policy[k].claims_used, "pipeline_id") && !contains(local.hth_secret_policy[k].claims_used, "pipeline_slug")]),
      jsonencode([for k, s in var.cluster_secrets : k if !contains(local.hth_secret_policy[k].claims_used, "pipeline_id") && !contains(local.hth_secret_policy[k].claims_used, "pipeline_slug")]),
    )
  }
}

# Cross-check against control 3.5 Step 4: production secrets belong in the
# production cluster only. Any secret whose declared cluster is not on the
# approved list for its sensitivity is a placement finding.
check "secrets_placed_in_declared_clusters" {
  assert {
    condition = length([
      for k, s in var.cluster_secrets : k
      if length(var.production_secret_clusters) > 0
      && s.production
      && !contains(var.production_secret_clusters, s.cluster)
    ]) == 0
    error_message = format(
      "%d secret(s) marked production are stored in a cluster not listed in production_secret_clusters: %s. A production credential in a development cluster is reachable by every agent registered to that cluster, which undoes the isolation established in control 3.2.",
      length([for k, s in var.cluster_secrets : k if length(var.production_secret_clusters) > 0 && s.production && !contains(var.production_secret_clusters, s.cluster)]),
      jsonencode([for k, s in var.cluster_secrets : k if length(var.production_secret_clusters) > 0 && s.production && !contains(var.production_secret_clusters, s.cluster)]),
    )
  }
}

# Metadata only. The value is write-only in the Buildkite API and unreadable by
# design, so there is deliberately nothing here that could carry it.
output "cluster_secrets_managed" {
  description = "Declared cluster secrets and the shape of their access policies, keyed by the var.cluster_secrets DECLARATION LABEL. `key` is the separate, immutable Buildkite key the secret was created under and the name a pipeline must reference (TRAP 7b) — compare the two when a pipeline reports an empty secret. Contains no secret material — values are write-only and cannot be read back from Buildkite (TRAP 8: this means Terraform can never detect a console overwrite of a value)."
  value = {
    for k, s in buildkite_cluster_secret.managed : k => {
      id                     = s.id
      key                    = s.key
      cluster                = var.cluster_secrets[k].cluster
      cluster_uuid           = s.cluster_id
      value_wo_version       = s.value_wo_version
      created_at             = s.created_at
      updated_at             = s.updated_at
      policy_rule_count      = length(var.cluster_secrets[k].policy_rules)
      policy_claims_used     = local.hth_secret_policy[k].claims_used
      third_party_only_rules = local.hth_secret_policy[k].third_party_only_rules
    }
  }
}

Code Pack: Config

hth-buildkite-3.05-redaction.sh View source on GitHub ↗

# Read the value the agent would actually use. An ABSENT setting means the agent
# applies its built-in defaults; an empty or present setting means the file wins
# (TRAP 1). Those two states are reported differently on purpose.
read_cfg_redacted_vars() {
  [ -r "${AGENT_CFG}" ] || return 1
  sed -nE 's|^[[:space:]]*redacted-vars[[:space:]]*=[[:space:]]*(.*)$|\1|p' "${AGENT_CFG}" \
    | tail -1 \
    | sed -E 's/^"(.*)"$/\1/; s/^'"'"'(.*)'"'"'$/\1/'
}

# Split on commas, trim, drop blanks, de-duplicate while preserving first-seen
# order so a diff of the config file stays readable across runs.
normalise_patterns() {
  tr ',' '\n' \
    | sed -E 's/^[[:space:]]+//; s/[[:space:]]+$//' \
    | grep -v '^$' \
    | awk '!seen[$0]++'
}

# The union that makes this an appender: whatever is configured now, plus the
# documented defaults, plus the operator's additions. Running it twice changes
# nothing; running it on a truncated list repairs the list.
merged_patterns() {
  local current=""
  current="$(read_cfg_redacted_vars || true)"
  printf '%s\n%s\n%s\n' "${current}" "${HTH_REDACTED_DEFAULTS}" "${HTH_REDACTED_EXTRA}" \
    | normalise_patterns
}

show() {
  local current
  current="$(read_cfg_redacted_vars || true)"
  echo "config file      : ${AGENT_CFG}"
  if [ -z "${current}" ]; then
    echo "redacted-vars    : <unset> - the agent applies its built-in defaults"
  else
    echo "redacted-vars    : ${current}"
  fi
  echo
  echo "effective patterns after 'apply' would be:"
  merged_patterns | sed 's/^/  /'
}

# ATOMIC CONFIG REPLACEMENT. buildkite-agent.cfg carries this host's registration
# token; a truncate-then-write (`cat "${tmp}" >"${AGENT_CFG}"`) interrupted by a
# signal, a full disk, or a set -e abort leaves a config the agent cannot parse,
# the agent then fails to start, and the host silently leaves the fleet. Every
# mutation below stages a sibling file and rename(2)s it into place instead, so a
# reader sees the old config or the new one, never a partial one.
#   * the temp lives in the target's OWN directory — rename(2) is EXDEV across
#     filesystems and mv would fall back to a non-atomic copy
#   * `cp -p` seeds it from the incumbent so mode, ownership and times ride onto
#     the replacement inode; a bare mktemp would hand the agent mktemp's 0600
#   * AGENT_CFG is resolved through symlinks first, because this path is commonly
#     a link into a config-management tree and renaming over the link would
#     replace it with a regular file
#   * no .bak is written: a persistent second copy of the agent token on disk is
#     a worse trade than the failure mode the rename already removes
# The trap is what keeps that staged full copy of the config — token included —
# out of the directory when a run dies mid-mutation.
HTH_CFG_TMP=""
HTH_CFG_DST=""
hth_cfg_cleanup() {
  if [ -n "${HTH_CFG_TMP}" ]; then rm -f "${HTH_CFG_TMP}"; fi
  HTH_CFG_TMP=""
}
trap hth_cfg_cleanup EXIT
trap 'hth_cfg_cleanup; exit 130' INT
trap 'hth_cfg_cleanup; exit 143' TERM

# Real path of the config, following symlinks where the platform's readlink can.
hth_cfg_path() {
  if [ -L "${AGENT_CFG}" ]; then
    readlink -f "${AGENT_CFG}" 2>/dev/null || printf '%s\n' "${AGENT_CFG}"
  else
    printf '%s\n' "${AGENT_CFG}"
  fi
}

# Stage a writable copy beside the target: HTH_CFG_DST is the real path to read
# from and commit to, HTH_CFG_TMP is the staged file to write into.
# This assigns globals rather than printing a value on purpose. Written as
# `dst="$(hth_cfg_stage)"` the function would run in a SUBSHELL, the parent's
# HTH_CFG_TMP would stay empty, and the cleanup trap would never see — or remove
# — the full copy of the token-bearing config that mktemp just created.
hth_cfg_stage() {
  HTH_CFG_DST="$(hth_cfg_path)"
  HTH_CFG_TMP="$(mktemp "$(dirname "${HTH_CFG_DST}")/.hth-bk-cfg.XXXXXX")"
  cp -p "${HTH_CFG_DST}" "${HTH_CFG_TMP}"
}

# Atomic swap. After this returns there is no temp left for the trap to clean up.
hth_cfg_commit() {
  mv -f "${HTH_CFG_TMP}" "${HTH_CFG_DST}"
  HTH_CFG_TMP=""
}

# Idempotent single-key upsert, matching the style used by the 3.4 config pack.
set_cfg() {
  local key="$1" value="$2" tmp dst
  hth_cfg_stage; dst="${HTH_CFG_DST}"; tmp="${HTH_CFG_TMP}"
  if grep -qE "^[[:space:]]*${key}[[:space:]]*=" "${dst}"; then
    # Replace via awk rather than sed: the value contains '*' and ',' and must
    # not be reinterpreted as part of a replacement expression.
    awk -v k="${key}" -v v="${value}" '
      $0 ~ "^[[:space:]]*" k "[[:space:]]*=" { print k "=\"" v "\""; next }
      { print }' "${dst}" >"${tmp}"
  else
    cat "${dst}" >"${tmp}"
    printf '%s="%s"\n' "${key}" "${value}" >>"${tmp}"
  fi
  hth_cfg_commit
}

apply() {
  [ -w "${AGENT_CFG}" ] || { echo "FATAL: cannot write ${AGENT_CFG} (run as root)." >&2; exit 5; }
  local merged
  merged="$(merged_patterns | paste -sd, -)"
  set_cfg "redacted-vars" "${merged}"
  echo "redacted-vars=\"${merged}\""
  echo
  echo "Restart buildkite-agent for this to take effect (TRAP 7), then: $0 audit"
}

# Container / systemd equivalent. TRAP 1 applies identically here: this variable
# REPLACES the built-in list, so it must always carry the full merged set.
print_env_equivalent() {
  printf '# buildkite-agent v3 - redaction via environment (containers, systemd units)\n'
  printf 'BUILDKITE_REDACTED_VARS="%s"\n' "$(merged_patterns | paste -sd, -)"
  printf '# Equivalent CLI flag: buildkite-agent start --redacted-vars "..."\n'
  printf '# This value REPLACES the built-in defaults - never set a partial list.\n'
}

# Fails closed on the regression TRAP 1 and TRAP 2 describe together: a
# redacted-vars that is SET and is missing patterns the agent would have applied
# had it been left alone. That configuration is strictly worse than no
# configuration, and it looks deliberate.
audit_redaction() {
  local current rc=0 missing=""
  current="$(read_cfg_redacted_vars || true)"

  echo "config file   : ${AGENT_CFG}"
  echo "redacted-vars : ${current:-<unset>}"
  echo "length floor  : 6 bytes (internal/redact/redact.go LengthMin) - shorter values are never redacted"
  echo

  if [ -z "${current}" ]; then
    echo "PASS: unset, so the agent applies all nine built-in patterns."
    echo "NOTE: that baseline covers no name outside the default globs. If secrets"
    echo "      arrive from Vault or a cloud secret manager under other names, run"
    echo "      '$0 apply' with HTH_REDACTED_EXTRA set (TRAP 3)."
  else
    local p
    while IFS= read -r p; do
      [ -n "${p}" ] || continue
      printf '%s\n' "${current}" | normalise_patterns | grep -qxF "${p}" || missing="${missing} ${p}"
    done < <(printf '%s' "${HTH_REDACTED_DEFAULTS}" | normalise_patterns)

    if [ -n "${missing}" ]; then
      echo "FAIL: redacted-vars is set and is MISSING documented default pattern(s):${missing}" >&2
      echo "      Setting this key replaces the built-in list rather than extending it," >&2
      echo "      so these names are no longer redacted from build logs on this agent." >&2
      echo "      Repair with: $0 apply" >&2
      rc=1
    else
      echo "PASS: every documented default pattern is present."
    fi
  fi

  # Not a failure, but the question an auditor asks next.
  if ! grep -qE '^[[:space:]]*redacted-vars[[:space:]]*=' "${AGENT_CFG}" 2>/dev/null; then
    echo
    echo "REMINDER: redaction is a log filter, not a boundary (TRAP 6). It does not"
    echo "          cover artifacts, pipeline settings, or values the job transformed"
    echo "          before printing."
  fi

  return "${rc}"
}

# TRAP 3 and TRAP 5, solved properly. `redacted-vars` matches variable NAMES that
# exist in the job environment. A secret fetched inside a step - the Vault, AWS
# Secrets Manager and GCP Secret Manager path control 3.5 Step 1 prefers - has
# neither property: it may be named anything, and it appeared after the
# environment was taken. `buildkite-agent redactor add` registers the VALUE
# itself, so every later line of the log is filtered regardless of the name.
#
# Emit these as agent hooks (hooks/environment, or a step's pre-command). They
# are printed rather than installed because the fetch command is site-specific.
print_vault_hook() {
  cat <<'HOOKEOF'
#!/usr/bin/env bash
# buildkite-agent v3 - hooks/environment
# Register externally-fetched secrets with the redactor BEFORE anything can echo
# them. Values are piped on stdin so they never appear in argv, in `ps`, or in
# the shell history of a debugging session.
set -euo pipefail

# --- HashiCorp Vault ---------------------------------------------------------
# --format json makes the redactor register every VALUE in the object and ignore
# the keys, which is what you want for a whole secret bundle.
vault kv get -format=json secret/data/ci/deploy \
  | jq -c '.data.data' \
  | buildkite-agent redactor add --format json

# Export afterwards. Registration first means an accidental `set -x` between the
# two lines still prints a redacted value.
DEPLOY_TOKEN="$(vault kv get -field=token secret/data/ci/deploy)"
export DEPLOY_TOKEN

# --- AWS Secrets Manager -----------------------------------------------------
aws secretsmanager get-secret-value --secret-id ci/deploy --query SecretString --output text \
  | buildkite-agent redactor add --format json

# --- GCP Secret Manager ------------------------------------------------------
# A single opaque value: no --format, so the whole input is registered verbatim.
gcloud secrets versions access latest --secret=ci-deploy \
  | buildkite-agent redactor add

# --- A file of key material --------------------------------------------------
# Registers the file's contents; the redactor also accepts a path argument.
# NOT /tmp. That directory is world-writable, shared with every process on the
# host and with every other job this agent runs, and a fixed filename in it is a
# pre-plantable symlink target as well as a co-tenant's read. Private key
# material belongs in the agent's own directory, owned by the agent user, mode
# 0600 — this is the one line in the pack a reader will copy verbatim, and a pack
# about keeping key material out of reach must not model leaving it in the open.
#   install -o buildkite-agent -g buildkite-agent -m 0600 \
#     id_ed25519 /etc/buildkite-agent/keys/id_ed25519
buildkite-agent redactor add /etc/buildkite-agent/keys/id_ed25519
HOOKEOF
}

# The name-filtered variant. --apply-vars-filter makes `redactor add` honour the
# same rules as the environment redactor: only entries whose NAME matches the
# redacted-vars patterns, and only values of at least 6 bytes (TRAP 4). Use it
# when piping a bundle that legitimately mixes secrets with non-secrets, and
# accept that anything misnamed is skipped.
print_filtered_hook() {
  cat <<'HOOKEOF'
#!/usr/bin/env bash
# buildkite-agent v3 - hooks/environment (name-filtered variant)
set -euo pipefail

# Only object entries whose key matches a redacted-vars pattern are registered,
# and only if the value is at least 6 bytes. Everything else is passed over in
# silence - which is the trade: fewer false redactions, and a misnamed secret is
# NOT protected. Prefer the unfiltered form for a bundle you know is all secret.
vault kv get -format=json secret/data/ci/app \
  | jq -c '.data.data' \
  | buildkite-agent redactor add --format json --apply-vars-filter
HOOKEOF
}

Code Pack: API Script

hth-buildkite-3.05-pipeline-env-scan.sh View source on GitHub ↗

# TRAP 1: the only thing this script is ever permitted to say about a value.
# Input is the value's base64 form, which keeps the digest portable and keeps it
# from doubling as a lookup key for the plaintext.
fingerprint_b64() {
  if command -v sha256sum >/dev/null 2>&1; then
    printf '%s' "$1" | sha256sum | cut -c1-12
  elif command -v shasum >/dev/null 2>&1; then
    printf '%s' "$1" | shasum -a 256 | cut -c1-12
  else
    echo "FATAL: need sha256sum or shasum to fingerprint findings without printing them." >&2
    exit 5
  fi
}

# Enumerate every pipeline, archived ones included (TRAP 8), following the
# documented Link-header pagination rather than guessing at page counts.
collect_pipelines() {
  local url="${REST}/organizations/${BUILDKITE_ORG_SLUG}/pipelines?per_page=${PER_PAGE}"
  local hdr body next jsonl
  hdr="$(mktemp)"; jsonl="$(mktemp)"
  trap 'rm -f "${hdr}" "${jsonl}"' RETURN

  while [ -n "${url}" ]; do
    body="$(curl -sS --fail-with-body -D "${hdr}" \
      -H "Authorization: Bearer ${BUILDKITE_TOKEN}" \
      -H "Accept: application/json" "${url}")"

    jq -e 'type == "array"' >/dev/null <<<"${body}" || {
      echo "unexpected response from ${url}:" >&2
      jq -r '.message // "non-array payload"' <<<"${body}" >&2
      exit 4
    }
    jq -c '.[]' <<<"${body}" >>"${jsonl}"

    # rel="next" is authoritative; its absence ends the collection.
    next="$(tr -d '\r' <"${hdr}" \
      | sed -n 's/^[Ll]ink:.*//p; s/.*<\([^>]*\)>; *rel="next".*/\1/p' | tail -1)"
    [ "${next}" = "${url}" ] && next=""
    url="${next}"
  done

  jq -s '.' "${jsonl}"
}

# Digests are computed in the shell because jq has no hash builtin. Values travel
# as base64 so a value containing tabs or newlines cannot corrupt the stream, and
# so no plaintext is ever passed through argv (TRAP 1).
build_digest_table() {
  jq -r '
      .[] as $p
      | ( ($p.env // {}) | to_entries[]
          | ["pipeline.env", $p.slug, .key, ((.value|tostring)|@base64)] )
      , ( ($p.steps // [])[] | (.env // {}) | to_entries[]
          | ["step.env", $p.slug, .key, ((.value|tostring)|@base64)] )
      , ( ($p.provider.settings // {}) | to_entries[]
          | ["provider.settings", $p.slug, .key, ((.value|tostring)|@base64)] )
      | @tsv' \
  | while IFS=$'\t' read -r loc slug name b64; do
      [ -n "${slug:-}" ] || continue
      jq -cn --arg k "${loc}::${slug}::${name}" \
             --arg d "$(fingerprint_b64 "${b64}")" '{($k): $d}'
    done \
  | jq -s 'add // {}'
}

# The classifier. Held in a single-quoted heredoc so the jq source needs no shell
# escaping — an escaped-jq-inside-double-quotes program is where these scripts
# silently break.
read -r -d '' HTH_SCAN_JQ <<'JQEOF' || true
# Shannon entropy over the raw bytes, in bits per character. Credential material
# drawn from a random alphabet sits above 4.2; prose, paths, versions and
# identifiers sit below it.
def shannon:
  (explode) as $b | ($b | length) as $n
  | if $n == 0 then 0
    else ($b | group_by(.) | map(length / $n) | map(. * (log / (2 | log))) | add | -.)
    end;

# TRAP 4: a pointer INTO a secret store is the pattern this control wants people
# to adopt. Includes the Buildkite interpolation form, which is the shape the
# guide's own examples use.
def is_reference:
  test("^(https?://|s3://|gs://|vault:|arn:|/|\\./|secret/|projects/[^/]+/secrets/)")
  or test("^\\$\\$?\\{?[A-Za-z_][A-Za-z0-9_]*\\}?$");

# Suffix-anchored, mirroring the agent's redacted-vars globs so this pack and the
# 3.5 config pack agree on "secret-shaped". Consequence: MYSECRET matches
# nothing, only *_SECRET does.
def name_is_secret_shaped:
  ascii_upcase
  | test("(_PASSWORD|_SECRET|_TOKEN|_PRIVATE_KEY|_ACCESS_KEY|_SECRET_KEY|_CONNECTION_STRING|_SSH_KEY|_API_KEY|_CREDENTIALS?|_PASSWD|_APIKEY|_AUTH)$")
    or test("^(PASSWORD|SECRET|TOKEN|API_KEY|AWS_SECRET_ACCESS_KEY|NPM_TOKEN|GITHUB_TOKEN|GH_TOKEN|DOCKER_PASSWORD|SLACK_TOKEN)$");

# Issuer-documented credential formats. These describe SHAPES, never values, so
# they are safe to publish and stable across rotations. The PEM test matches the
# armour envelope generically, catching RSA, EC and OPENSSH key blocks without
# this file having to carry a key header verbatim.
def issuer_format_kind:
  if   test("^-----BEGIN [A-Z0-9 ]*KEY-----")                                          then "pem_key_block"
  elif test("(^|[^A-Z0-9])(AKIA|ASIA|AGPA|AIDA|AROA|AIPA|ANPA|ABIA|ACCA)[A-Z0-9]{16}") then "aws_access_key_id"
  elif test("^gh[pousr]_[A-Za-z0-9]{36,}$")                                            then "github_token"
  elif test("^github_pat_[A-Za-z0-9_]{20,}$")                                          then "github_fine_grained_pat"
  elif test("^xox[abposr]-[A-Za-z0-9-]{10,}$")                                          then "slack_token"
  elif test("^sk-[A-Za-z0-9_-]{20,}$")                                                  then "openai_style_key"
  elif test("^AIza[0-9A-Za-z_-]{35}$")                                                  then "google_api_key"
  elif test("^eyJ[A-Za-z0-9_-]{10,}\\.[A-Za-z0-9_-]{10,}\\.")                            then "jwt"
  else null end;

# Entropy heuristics, applied only to individual values and to single
# configuration lines — never to a whole YAML document (TRAP 2). The mixed-class
# requirement is what separates a random credential from a path, a semver, a
# hostname or an identifier.
def opaque_material_kind:
  if   test("^[A-Za-z0-9+/]{32,}={0,2}$") and (shannon > 4.0)                       then "high_entropy_base64"
  elif test("^[0-9a-fA-F]{40,}$")                                                   then "long_hex_digest_or_key"
  elif test("^[!-~]{24,256}$") and test("[a-z]") and test("[A-Z]") and test("[0-9]")
       and (shannon > 4.2) and (is_reference | not)                                 then "high_entropy_opaque"
  else null end;

def value_material_kind: issuer_format_kind // opaque_material_kind;

# TRAP 1: name, length, classification, digest. Never a value.
def finding($digests; $pipeline; $location; $name; $value):
  ($value | value_material_kind) as $kind
  | ($name  | name_is_secret_shaped) as $named
  | if ($kind == null and $named == false) then empty
    else {
      pipeline: $pipeline,
      location: $location,
      variable: $name,
      name_is_secret_shaped: $named,
      value_material_kind: $kind,
      value_length: ($value | length),
      # TRAP 10: under 6 bytes the agent's redactor would not have masked it
      # either, so such a value is exposed on both paths at once.
      below_redaction_floor: (($value | length) > 0 and ($value | length) < 6),
      value_sha256_prefix: ($digests[$location + "::" + $pipeline + "::" + $name] // "n/a"),
      severity: (if $kind != null then "CRITICAL"
                 elif ($value | is_reference) then "OK_REFERENCE"
                 else "REVIEW" end),
      finding: (if $kind != null then
                  "Holds text matching a credential format. Every API token that can read this pipeline can read this value."
                elif ($value | is_reference) then
                  "Named like a secret but holds a pointer into a secret store. That is the pattern control 3.5 steers toward - no action."
                else
                  "Named like a secret. Confirm by hand whether the value is material or a harmless reference."
                end)
    } end;

# TRAP 2: the configuration YAML arrives as one string, so it is split and judged
# a line at a time. Each line is tested twice — as a `key: value` pair, which lets
# the name rules and the anchored issuer formats apply, and as raw text, which
# catches material pasted straight into a command with no variable at all.
def configuration_findings($p):
  [ ($p.configuration // "") | split("\n") | to_entries[]
    | .key as $ln | (.value) as $line
    | ( ( $line
          | capture("^\\s*(?<k>[A-Za-z_][A-Za-z0-9_]*)\\s*:\\s*(?<v>\\S.*?)\\s*$") // empty
          | { k: .k, v: (.v | sub("^[\"']"; "") | sub("[\"'],?$"; "")) }
          | select((.k | name_is_secret_shaped) or ((.v | value_material_kind) != null))
          | { pipeline: $p.slug,
              location: ("pipeline.configuration:line " + (($ln + 1) | tostring)),
              variable: .k,
              name_is_secret_shaped: (.k | name_is_secret_shaped),
              value_material_kind: (.v | value_material_kind),
              value_length: (.v | length),
              below_redaction_floor: ((.v | length) > 0 and (.v | length) < 6),
              value_sha256_prefix: "n/a",
              severity: (if (.v | value_material_kind) != null then "CRITICAL"
                         elif (.v | is_reference) then "OK_REFERENCE"
                         else "REVIEW" end),
              finding: "Declared inside the stored pipeline YAML, which is returned in full by the REST and GraphQL pipeline payloads." } )
      , ( $line
          | select((capture("^\\s*[A-Za-z_][A-Za-z0-9_]*\\s*:\\s*\\S") // null) == null)
          | select(issuer_format_kind != null)
          | { pipeline: $p.slug,
              location: ("pipeline.configuration:line " + (($ln + 1) | tostring)),
              variable: "<literal in step configuration>",
              name_is_secret_shaped: false,
              value_material_kind: issuer_format_kind,
              value_length: length,
              below_redaction_floor: false,
              value_sha256_prefix: "n/a",
              severity: "CRITICAL",
              finding: "The stored pipeline YAML contains text matching a credential format - material pasted into a command rather than referenced through a variable." } ) )
  ];

def classify($digests):
  map(. as $p | {
    pipeline: $p.slug,
    visibility: ($p.visibility // "unknown"),
    # TRAP 8: archived stops builds, not disclosure.
    archived: (($p.archived_at // null) != null),
    # TRAP 3: on an upload-driven pipeline a clean verdict describes the
    # settings, not the steps that actually run.
    upload_driven: ((($p.configuration // "")
                     + (($p.steps // []) | map(.command // "") | join("\n")))
                    | test("buildkite-agent[[:space:]]+pipeline[[:space:]]+upload")),
    # Surfaced so a REST shape change shows up in the report instead of
    # silently narrowing what was scanned.
    fields_seen: ($p | keys | map(select(. == "env" or . == "steps" or . == "configuration" or . == "provider"))),
    findings: (
        [ ($p.env // {}) | to_entries[]
          | finding($digests; $p.slug; "pipeline.env"; .key; (.value | tostring)) ]
      + [ ($p.steps // [])[] | (.env // {}) | to_entries[]
          | finding($digests; $p.slug; "step.env"; .key; (.value | tostring)) ]
      # TRAP 5: provider settings ride in the same payload and are where webhook
      # and integration tokens end up.
      + [ ($p.provider.settings // {}) | to_entries[]
          | finding($digests; $p.slug; "provider.settings"; .key; (.value | tostring)) ]
      + configuration_findings($p)
    )
  })
  | map(select((.findings | length) > 0 or .upload_driven));
JQEOF

scan() {
  local pipelines digests
  pipelines=$(collect_pipelines)
  digests=$(build_digest_table <<<"${pipelines}")
  jq --argjson digests "${digests}" "${HTH_SCAN_JQ}"'
    classify($digests)' <<<"${pipelines}"
}

# Exits non-zero when credential material is being served, so this drops into a
# scheduled pipeline as a gate rather than a report nobody opens.
summary() {
  local report
  report=$(scan)
  jq '{
    pipelines_with_findings: (map(select((.findings | length) > 0)) | length),
    critical:     ([.[].findings[] | select(.severity == "CRITICAL")]     | length),
    review:       ([.[].findings[] | select(.severity == "REVIEW")]       | length),
    ok_reference: ([.[].findings[] | select(.severity == "OK_REFERENCE")] | length),
    # TRAP 10: served in the payload AND unredactable in logs.
    below_redaction_floor: ([.[].findings[] | select(.below_redaction_floor)] | length),
    # TRAP 8: least likely to be rotated, still fully readable.
    archived_pipelines_with_findings: (map(select(.archived and ((.findings | length) > 0)) | .pipeline)),
    # TRAP 3, made unmissable: a clean scan across upload-driven pipelines is not
    # a clean estate, it is an unscanned one.
    upload_driven_not_meaningfully_scanned: (map(select(.upload_driven) | .pipeline)),
    verdict: (if ([.[].findings[] | select(.severity == "CRITICAL")] | length) > 0
              then "FAIL - credential material is being served in pipeline settings"
              else "no credential material found in pipeline settings" end)
  }' <<<"${report}"
  [ "$(jq '[.[].findings[] | select(.severity == "CRITICAL")] | length' <<<"${report}")" -eq 0 ]
}

# TRAP 6. `$VAR` is substituted when the pipeline is UPLOADED and the result is
# visible in the build timeline; `$$VAR` reaches the shell and is expanded inside
# the job. Against a secret-shaped name the difference is whether the credential
# gets printed. This reports single-`$` references and deliberately ignores
# correctly escaped `$$` and `\$`.
interpolation() {
  collect_pipelines | jq '
    def secret_shaped: ascii_upcase
      | test("(_PASSWORD|_SECRET|_TOKEN|_PRIVATE_KEY|_ACCESS_KEY|_SECRET_KEY|_CONNECTION_STRING|_SSH_KEY|_API_KEY|_CREDENTIALS?)$");

    map(. as $p
      | (($p.configuration // "") + "\n"
         + (($p.steps // []) | map((.command // "") + "\n" + (.label // "")) | join("\n"))) as $text
      | {
          pipeline: $p.slug,
          # A `$` preceded by neither another `$` nor a backslash, followed by a
          # secret-shaped identifier, bare or braced.
          interpolated_at_upload_time: (
            [ $text
              | scan("(?:^|[^$\\\\])\\$\\{?([A-Za-z_][A-Za-z0-9_]*)\\}?")
              | .[0] | select(secret_shaped) ] | unique
          )
        }
      | select((.interpolated_at_upload_time | length) > 0)
      | . + { finding: "These secret-shaped names are interpolated at UPLOAD time, so their values are substituted into the uploaded pipeline and shown in the build timeline. Escape them as $$NAME so the shell expands them inside the job instead." }
    )'
}

# TRAP 7. Removal is not remediation. Anything `scan` finds has already been
# served in REST and GraphQL payloads to every token that can read the pipeline,
# and if it was interpolated it sits in past build timelines. Rotation at the
# issuer is the remediation; this only stops the bleeding afterwards.
rest_get() {
  curl -sS --fail-with-body \
    -H "Authorization: Bearer ${BUILDKITE_TOKEN}" \
    -H "Accept: application/json" \
    "${REST}$1"
}

# The PATCH sends the COMPLETE remaining env object, read immediately before the
# write. That is correct whether the API merges the field or replaces it — and
# under replace semantics a partial env would silently delete every other
# variable on the pipeline.
strip_env() {
  local slug="$1" var="$2" ack="${3:-}" current remaining response after

  if [ "${ack}" != "--rotated" ]; then
    cat >&2 <<'ROTATE'
REFUSING: pass --rotated as the third argument.

Deleting the variable does not un-expose the credential. It has already been
returned in REST and GraphQL payloads to every token that can read this
pipeline, and if it was interpolated into an uploaded pipeline it is in the
build timeline of every build since it was added.

Correct order:
  1. Rotate the credential at its issuer, so the exposed value is worthless.
  2. Move the new value into a cluster secret with an access policy
     (packs/buildkite/terraform/hth-buildkite-3.05-cluster-secrets.tf)
     or into your external secret store.
  3. Re-run this command with --rotated.
ROTATE
    exit 3
  fi

  current=$(rest_get "/organizations/${BUILDKITE_ORG_SLUG}/pipelines/${slug}" | jq '.env // {}')

  if [ "$(jq --arg v "${var}" 'has($v)' <<<"${current}")" != "true" ]; then
    echo "pipeline '${slug}' has no env variable named '${var}'; nothing to remove." >&2
    exit 4
  fi

  remaining=$(jq --arg v "${var}" 'del(.[$v])' <<<"${current}")

  if ! response=$(curl -sS --fail-with-body -X PATCH \
    -H "Authorization: Bearer ${BUILDKITE_TOKEN}" \
    -H "Content-Type: application/json" \
    --data "$(jq -n --argjson env "${remaining}" '{env: $env}')" \
    "${REST}/organizations/${BUILDKITE_ORG_SLUG}/pipelines/${slug}"); then
    echo "FAILED: the API rejected the PATCH for pipeline '${slug}':" >&2
    printf '%s\n' "${response}" >&2
    echo "'${var}' is still set and the credential remains exposed." >&2
    exit 5
  fi

  # VERIFY THE REMOVAL — do not assert it. The comment above establishes that the
  # request is correct under EITHER merge or replace semantics; it does not
  # establish that the key is gone. Under merge semantics the variable survives
  # the PATCH, and printing removed_variable from the write's own reply reports a
  # removal that did not happen. This runs during a live credential exposure,
  # immediately after the operator confirmed rotation, so a false "removed" is
  # exactly the output that ends an incident response one step too early.
  # Re-read from the server rather than reading the write's echo: the PATCH
  # response is not guaranteed to carry .env, and an omitted field is
  # indistinguishable from a deleted key when you only look at the reply.
  after=$(rest_get "/organizations/${BUILDKITE_ORG_SLUG}/pipelines/${slug}" | jq '.env // {}')

  if [ "$(jq --arg v "${var}" 'has($v)' <<<"${after}")" = "true" ]; then
    echo "FAILED: '${var}' is STILL present on pipeline '${slug}' after the PATCH." >&2
    echo "This API merged the env object rather than replacing it, so a payload" >&2
    echo "that omits a key cannot delete it. Remove the variable in the console" >&2
    echo "(Pipeline Settings -> Environment Variables) and re-run this command to" >&2
    echo "confirm. Until it reports verified_absent, treat the value as exposed." >&2
    exit 6
  fi

  jq -n --arg s "${slug}" --arg v "${var}" --argjson after "${after}" '{
      pipeline: $s,
      removed_variable: $v,
      verified_absent: true,
      remaining_env_keys: ($after | keys),
      reminder: "Rotation at the issuer is what actually remediated this. Confirm the old credential is dead."
    }'
}

Sources: Buildkite secrets · Managing pipeline secrets · Secrets risk considerations

3.6 Use OIDC Instead of Static Cloud Credentials

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	5.4
NIST 800-53	IA-5, IA-9

Description

Let builds obtain short-lived cloud credentials by exchanging a Buildkite OIDC token, rather than storing long-lived cloud access keys anywhere in the CI system.

Rationale

Why This Matters:

A static cloud access key stored for CI never expires on its own, so its exposure window is bounded only by how quickly someone notices
OIDC-issued credentials are minted per job and expire on their own, which makes a leaked build log far less valuable to an attacker
The trust relationship is pinned to a specific pipeline and organization in the cloud provider’s policy, so a token from elsewhere in your CI cannot assume the role
Removing static keys removes the single highest-value target in most CI environments — the credential that grants production cloud access

Attack Prevented: Cloud account takeover through leaked CI credentials, long-lived standing access to cloud accounts, credential reuse outside the build that earned it

ClickOps Implementation

Step 1: Establish the Trust Relationship

In your cloud provider, configure Buildkite as an OIDC identity provider and constrain the trust policy to your organization and the specific pipelines that need the role.

Step 2: Adopt the Exchange Plugins

AWS: use the aws-assume-role-with-web-identity plugin to trade the Buildkite OIDC token for temporary STS credentials.
GCP: use the gcp-workload-identity-federation plugin.
HashiCorp Vault: use the vault-secrets plugin to authenticate and fetch secrets without a static Vault token in the pipeline.

Step 3: Remove What You Replaced

Delete the static access keys the pipelines previously used, and confirm in the cloud provider that they are no longer being exercised. A migration that leaves the old key live has added a path rather than closed one.

Code Implementation

Code Pack: Config

hth-buildkite-3.06-oidc-secrets.yml View source on GitHub ↗

# .buildkite/pipeline.yml — exchange a Buildkite OIDC token for short-lived cloud
# credentials. No static cloud key is stored in Buildkite, in the repository, or
# on the agent host.
steps:
  # ---- AWS: STS AssumeRoleWithWebIdentity -----------------------------------
  # Pinned by tag for readability here; the scoping region below shows the SHA
  # pin you should actually ship (T5).
  - key: "build-artifacts"
    label: ":aws: build and publish artifacts"
    agents:
      queue: "build"
    plugins:
      - aws-assume-role-with-web-identity#v1.7.0:
          role-arn: "arn:aws:iam::111122223333:role/buildkite-build-artifacts"
          # Static: no `$` interpolation, <= 64 chars, charset [\w+=,.@-] (T6).
          role-session-name: "bk-build-artifacts"
          # Both clocks set explicitly so neither is inferred (T3). 900s is the
          # AWS minimum for --duration-seconds; keep the token no longer-lived
          # than the credential it buys.
          role-session-duration: 900
          oidc-token-lifetime: 900
          region: "us-east-1"
          # Emitted under the nested "https://aws.amazon.com/tags" claim as
          # principal_tags, usable in the role's trust policy via
          # sts:RequestTag/<name> and aws:PrincipalTag/<name>. Requires the trust
          # policy to allow sts:TagSession as well as
          # sts:AssumeRoleWithWebIdentity, or the exchange is denied.
          session-tags:
            - "organization_slug"
            - "pipeline_id"
            - "build_branch"
            - "build_source"
    command: ".buildkite/steps/build-artifacts.sh"

  # ---- GCP: Workload Identity Federation -------------------------------------
  - key: "deploy-gcp"
    label: ":googlecloud: deploy"
    depends_on: "build-artifacts"
    agents:
      queue: "deploy"
    plugins:
      - gcp-workload-identity-federation#v1.6.0:
          # This exact string is echoed into credentials.json as the STS audience,
          # so it must match the provider resource name byte for byte.
          audience: "//iam.googleapis.com/projects/123456789012/locations/global/workloadIdentityPools/buildkite/providers/buildkite-oidc"
          service-account: "buildkite-deploy@my-project.iam.gserviceaccount.com"
          lifetime: 900
          # Passed through as repeated `--claim` flags (lib/plugin.bash v1.6.0:
          # `args+=("--claim" "$line")`, under the comment "the list of OPTIONAL
          # claims"). Only names from the docs' "### Optional claims" table may go
          # here — organization_id, pipeline_id, build_id, cluster_id,
          # cluster_name, queue_id, queue_key, agent_tag:NAME. build_source,
          # build_branch and step_key are DEFAULT claims: already in the token,
          # not addable, and asking for them here is at best noise and at worst a
          # rejected token request. Constrain the WIF provider attribute mapping
          # on the immutable UUIDs below and on the default claims the token
          # already carries — never on the mutable slugs.
          claims:
            - "organization_id"
            - "pipeline_id"
            - "cluster_id"
    command: ".buildkite/steps/deploy-gcp.sh"

  # ---- HashiCorp Vault: JWT auth ---------------------------------------------
  # The Vault plugin does NOT mint a token. It reads one out of the environment
  # (T1 applies doubly here). BUILDKITE_OIDC_VAULT_JWT is populated by the agent
  # `environment` hook in the paired .sh pack, which runs first.
  - key: "db-migrate"
    label: ":vault: run migrations"
    depends_on: "deploy-gcp"
    agents:
      queue: "deploy"
    plugins:
      - vault-secrets#v2.4.2:
          server: "https://vault.internal.example.com:8200"
          path: "secret/buildkite/db-migrate"
          auth:
            # Never leave this empty or misspelled — it fails open (T9).
            method: "jwt"
            # Name of the env var holding the JWT. If omitted the plugin falls
            # back to $VAULT_JWT and hard-fails when that is unset.
            jwt-env: "BUILDKITE_OIDC_VAULT_JWT"
            # Omitting this silently defaults to the Vault role literally named
            # "buildkite" — almost certainly broader than this step needs.
            jwt-role: "buildkite-db-migrate"
    command: ".buildkite/steps/db-migrate.sh"

# Narrow the blast radius on the Buildkite side, before the cloud policy is ever
# consulted. A step that is never scheduled cannot mint a token, which is strictly
# stronger than minting one and relying on the cloud provider to reject it.
steps:
  - key: "deploy-prod"
    label: ":rocket: deploy production"
    # `pipeline.slug` and `build.source` are Buildkite conditional variables.
    # Gating on build.source blocks API- and trigger-initiated builds, which can
    # nominate an arbitrary branch and commit, from reaching the privileged role.
    if: >-
      pipeline.slug == "payments-service" &&
      build.branch == "main" &&
      (build.source == "webhook" || build.source == "ui")
    agents:
      # Pair with control 3.2: only agents in the production cluster/queue ever
      # see this step, so cluster_id in the token is a meaningful assertion.
      queue: "prod-deploy"
    plugins:
      # SHA pin of tag v1.7.0 (T5). Re-resolve with:
      #   git ls-remote https://github.com/buildkite-plugins/aws-assume-role-with-web-identity-buildkite-plugin refs/tags/v1.7.0
      - aws-assume-role-with-web-identity#ed5cff4f02493c0f0edff5a899aa03817efe0b24:
          role-arn: "arn:aws:iam::111122223333:role/buildkite-deploy-prod"
          role-session-name: "bk-deploy-prod"
          role-session-duration: 900
          oidc-token-lifetime: 900
          region: "us-east-1"
          # Keeps credentials out of the environment for checkout and for the
          # repository's own post-checkout hook, which is editable by anyone who
          # can open a pull request. Within the pre-command phase the executor
          # runs global -> local -> plugin (internal/job/hooks.go,
          # executeHooksForward; called for "pre-command" at
          # internal/job/executor.go:1144), so the plugin hook runs LAST — after
          # the repository's own .buildkite/hooks/pre-command, which therefore
          # never sees the credentials either. Only the command itself does, which
          # is the point. Pack 2.4 T6 states the same order. Exact-string switch —
          # see T4.
          hook: "pre-command"
          session-tags:
            - "pipeline_id"
            - "build_branch"
            - "build_source"
    command: ".buildkite/steps/deploy-prod.sh"

  # The unprivileged majority of the pipeline carries no plugin and therefore no
  # path to a cloud credential. This is the shape to aim for: OIDC on the two
  # steps that deploy, not on the pipeline.
  - key: "unit-tests"
    label: ":test_tube: unit tests"
    command: ".buildkite/steps/test.sh"

Code Pack: CLI Script

hth-buildkite-3.06-oidc-subject.sh View source on GitHub ↗

# Mint a token with an explicit, immutable subject. Run inside a Buildkite job:
# --job defaults from $BUILDKITE_JOB_ID and the command requires it.

require_agent() {
  command -v "${BK_AGENT}" >/dev/null 2>&1 || {
    echo "FATAL: '${BK_AGENT}' not on PATH. Requires buildkite-agent v3.121.0+." >&2
    exit 127
  }
  command -v jq >/dev/null 2>&1 || { echo "FATAL: jq required." >&2; exit 127; }
}

# T1: reject a UUID passed where a claim NAME belongs, and refuse to silently
# broaden the trust boundary. This is the guard the guide's one-line prose lacks.
assert_subject_claim() {
  local claim="$1" ok=0 c

  if printf '%s' "${claim}" | grep -qiE '^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$'; then
    echo "FATAL: '--subject-claim ${claim}' passes a UUID where a claim NAME belongs." >&2
    echo "       Pass the key (e.g. pipeline_id); Buildkite supplies the value." >&2
    exit 3
  fi

  for c in ${SUBJECT_CLAIMS_ALLOWED}; do [ "${c}" = "${claim}" ] && ok=1; done
  if [ "${ok}" -ne 1 ]; then
    echo "FATAL: '${claim}' is not an accepted subject claim." >&2
    echo "       Accepted: ${SUBJECT_CLAIMS_ALLOWED}" >&2
    echo "       Slugs and branch names are refused by Buildkite by design —" >&2
    echo "       renaming them would silently break trust relationships." >&2
    exit 3
  fi

  for c in ${SUBJECT_CLAIMS_PER_BUILD}; do
    [ "${c}" = "${claim}" ] && {
      echo "WARNING: '${claim}' changes on every build. No static cloud trust" >&2
      echo "         policy can match it on 'sub'." >&2
    }
  done

  for c in ${SUBJECT_CLAIMS_BROADENING}; do
    if [ "${c}" = "${claim}" ] && [ "${HTH_ALLOW_BROAD_SUBJECT:-0}" != "1" ]; then
      echo "FATAL: '${claim}' is WIDER than the default compound subject — every" >&2
      echo "       pipeline in that ${claim%%_id} presents an identical 'sub'." >&2
      echo "       It buys immutability by discarding pipeline, ref, commit and" >&2
      echo "       step. If the relying party genuinely requires it, re-run with" >&2
      echo "       HTH_ALLOW_BROAD_SUBJECT=1 and carry the real scoping in claim" >&2
      echo "       conditions (build_branch, step_key, pipeline_id)." >&2
      exit 3
    fi
  done
}

# T5b: refuse a default claim passed where an OPTIONAL claim name belongs. Reading
# `--claim "build_branch"` as a scoping control is the specific mistake this
# catches; the value is already in the token and the flag cannot add it.
assert_optional_claims() {
  local csv="$1" claim ok d list

  # Commas to spaces rather than `local IFS=,`: the allow-lists below are
  # space-separated, and re-scoping IFS would stop THEM splitting.
  list="$(printf '%s' "${csv}" | tr ',' ' ')"

  for claim in ${list}; do
    [ -n "${claim}" ] || continue

    # agent_tag:NAME — the suffix is an operator-chosen agent tag, not a fixed
    # name, so only the prefix can be checked.
    case "${claim}" in agent_tag:?*) continue ;; esac

    for d in ${DEFAULT_CLAIMS}; do
      if [ "${d}" = "${claim}" ]; then
        echo "FATAL: '${claim}' is a DEFAULT claim — every token already carries it." >&2
        echo "       --claim adds from the optional table only: ${OPTIONAL_CLAIMS_ALLOWED}" >&2
        echo "       Condition on '${claim}' in the relying party's policy instead;" >&2
        echo "       for AWS, carry it with --aws-session-tag, which DOES accept" >&2
        echo "       default claims." >&2
        exit 3
      fi
    done

    ok=0
    for d in ${OPTIONAL_CLAIMS_ALLOWED}; do [ "${d}" = "${claim}" ] && ok=1; done
    if [ "${ok}" -ne 1 ]; then
      echo "FATAL: '${claim}' is not an addable claim." >&2
      echo "       Accepted: ${OPTIONAL_CLAIMS_ALLOWED} agent_tag:<NAME>" >&2
      exit 3
    fi
  done
}

# T7 audience, T6 lifetime, T4 no redundant --claim for the subject itself.
# T5: extra claims go in ONE comma-separated --claim, never --claims.
# T5b: and only OPTIONAL claims may go in it. The default adds organization_id —
# the immutable counterpart to the mutable organization_slug the token already
# carries — so a relying party can pin the tenant to a UUID that survives a
# rename. Branch and step are deliberately absent: they are default claims,
# present already, and --claim cannot add them. Pass "" to send no --claim at all.
# If you ever set --subject-claim organization_id (which needs
# HTH_ALLOW_BROAD_SUBJECT=1 and is refused otherwise), drop it from extra-claims:
# T4 — the subject claim is auto-included and repeating it is redundant.
request_token() {
  local audience="${1:?audience required, e.g. sts.amazonaws.com}"
  local subject_claim="${2:?subject claim name required, e.g. pipeline_id}"
  local lifetime="${3:-900}"
  local extra_claims="${4-organization_id}"

  require_agent
  assert_subject_claim "${subject_claim}"
  assert_optional_claims "${extra_claims}"

  [ "${lifetime}" -ge 1 ] 2>/dev/null || {
    echo "FATAL: lifetime must be a positive integer of seconds. 0 means 'API" >&2
    echo "       default' (5 minutes), not 'unlimited'." >&2
    exit 3
  }

  if [ -z "${extra_claims}" ]; then
    "${BK_AGENT}" oidc request-token \
      --audience "${audience}" \
      --subject-claim "${subject_claim}" \
      --lifetime "${lifetime}"
  else
    "${BK_AGENT}" oidc request-token \
      --audience "${audience}" \
      --subject-claim "${subject_claim}" \
      --lifetime "${lifetime}" \
      --claim "${extra_claims}"
  fi
}

# AWS variant. The audience is fixed by STS, and the claims that a trust policy
# will test go in as SESSION TAGS (nested under "https://aws.amazon.com/tags")
# rather than plain claims, because that is the only form
# sts:AssumeRoleWithWebIdentity can condition on. The role's trust policy must
# also permit sts:TagSession.
#
# The default tag set below deliberately mixes DEFAULT claims (organization_slug,
# build_branch, build_source) with an OPTIONAL one (pipeline_id), which would be
# rejected by assert_optional_claims one function up. That is not an
# inconsistency: --aws-session-tag takes "any of the supported claims" (docs, "AWS
# session tags") and the vendor's own example tags organization_slug, whereas
# --claim adds from the optional table only (T5b). Session tags are where branch
# and step conditions belong on AWS.
request_token_aws() {
  local subject_claim="${1:-pipeline_id}"
  local lifetime="${2:-900}"
  local session_tags="${3:-organization_slug,pipeline_id,build_branch,build_source}"

  require_agent
  assert_subject_claim "${subject_claim}"

  "${BK_AGENT}" oidc request-token \
    --audience "sts.amazonaws.com" \
    --subject-claim "${subject_claim}" \
    --lifetime "${lifetime}" \
    --aws-session-tag "${session_tags}"
}

# The agent-side scoping the pipeline cannot override (T3).
#
# Ordering is what makes this work: the agent `environment` hook runs BEFORE any
# non-vendored plugin's `environment` hook (docs/agent/v3/hooks — vendored plugin
# environment hooks are singled out as the exception, running after checkout).
# So this file sets BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM before
# aws-assume-role-with-web-identity or gcp-workload-identity-federation mints
# anything, and mints the Vault JWT before vault-secrets looks for it.
#
# It lives on the agent host under hooks-path. It is not in the build's
# repository, so a pull request cannot change the subject of its own token.
#
# ── ⚠️ THE HOOK FILE IS SHARED. DO NOT WRITE IT WHOLE. ──────────────────────
# An agent has exactly ONE ${hooks-path}/environment, and control 3.9
# (packs/buildkite/config/hth-buildkite-3.09-agent-env.sh) needs the same file:
# it is the only place BUILDKITE_CLEAN_CHECKOUT is read, and the only place that
# can win the GIT_SSH_COMMAND first-wins race. A whole-file `cat >` by either
# pack silently deletes the other control — and neither `audit` verb would see
# it, because each greps for its own settings elsewhere. Adopt 3.6 then 3.9 that
# way round and every OIDC token quietly reverts to the default compound subject
# while three PASSes print.
#
# So both packs write a DELIMITED BLOCK and rewrite only their own:
#   # >>> HTH-BLOCK <id>
#   ...
#   # <<< HTH-BLOCK <id>
# hth_write_hook_block() below is byte-identical in 3.6 and 3.9 apart from the
# block id. It preserves every other line in the file — the other pack's block,
# and any hook the operator wrote themselves — and takes a timestamped backup
# before touching anything. Running either pack twice is idempotent.

# The one line that differs between the two copies of this protocol.
HTH_HOOK_BLOCK_ID="hth-3.6-oidc-subject"

# $1 = hook path, $2 = block id, block body on stdin.
hth_write_hook_block() {
  local hook="$1" id="$2"
  local dir tmp begin end
  dir="$(dirname "${hook}")"
  begin="# >>> HTH-BLOCK ${id}"
  end="# <<< HTH-BLOCK ${id}"

  [ -d "${dir}" ] || { echo "FATAL: hooks-path '${dir}' does not exist." >&2; exit 5; }
  [ -w "${dir}" ] || { echo "FATAL: cannot write '${dir}' (run as root)." >&2; exit 5; }

  tmp="$(mktemp "${dir}/.hth-environment.XXXXXX")"

  if [ -e "${hook}" ]; then
    # Backup first, always — including when the result will be identical. A hook
    # is arbitrary code that runs as the agent user on every job; there is no
    # such thing as an edit here that is not worth being able to undo. The
    # counter matters: the stamp has one-second resolution, and installing 3.6
    # then 3.9 back to back lands in the same second, so a bare stamp would let
    # the second install overwrite the backup of the operator's ORIGINAL file.
    # A backup is never overwritten. (`environment.hth-bak.*` is inert — the
    # agent looks up hooks by exact filename, not by glob.)
    local stamp bak n=0
    stamp="$(date -u +%Y%m%dT%H%M%SZ)"
    bak="${hook}.hth-bak.${stamp}"
    while [ -e "${bak}" ]; do n=$((n + 1)); bak="${hook}.hth-bak.${stamp}-${n}"; done
    cp -p "${hook}" "${bak}"
    echo "Backed up ${hook} -> ${bak}"

    # Carry over everything except a previous copy of THIS block. awk with
    # index() rather than a regex: the markers contain > and < and the id
    # contains dots, none of which should be read as metacharacters.
    awk -v b="${begin}" -v e="${end}" '
      index($0, b) == 1 { skip = 1; next }
      index($0, e) == 1 { skip = 0; next }
      !skip { print }
    ' "${hook}" >"${tmp}"

    # An `exit` in code that already ran means this block never will. Cheap to
    # detect, invisible at runtime (the hook "succeeds" and does nothing).
    # Only unmanaged lines are scanned: the sibling pack's block legitimately
    # exits to refuse a job, and warning about that every run would be noise.
    if awk '
         /^# >>> HTH-BLOCK / { skip = 1; next }
         /^# <<< HTH-BLOCK / { skip = 0; next }
         !skip { print }
       ' "${tmp}" | grep -qE '^[[:space:]]*exit([[:space:]]|$)'; then
      echo "WARN: ${hook} contains an 'exit' outside any HTH-BLOCK. If it runs" >&2
      echo "      before the block below, the block never executes." >&2
    fi
  else
    cat >"${tmp}" <<'HEADEOF'
#!/usr/bin/env bash
# Buildkite agent `environment` hook.
# Runs once per job, before checkout and before every plugin environment hook.
# Sections delimited by "HTH-BLOCK <id>" markers are managed by How to Harden
# packs and are rewritten in place; edit outside them.
set -euo pipefail
HEADEOF
  fi

  printf '\n%s\n' "${begin}" >>"${tmp}"
  cat >>"${tmp}"
  printf '%s\n' "${end}" >>"${tmp}"

  chmod 0755 "${tmp}"
  chown root:root "${tmp}" 2>/dev/null || true
  mv "${tmp}" "${hook}"
}

install_environment_hook() {
  local hook="${AGENT_HOOKS_PATH}/environment"

  hth_write_hook_block "${hook}" "${HTH_HOOK_BLOCK_ID}" <<'HOOKEOF'
# OIDC subject scoping. Managed by HTH control 3.6.

# Unconditional assignment, never a ${VAR:-default}: a pipeline.yml `env:` block
# is applied to the job environment, and deferring to an inherited value would
# let the repository choose its own token subject.
export BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM="pipeline_id"

# Per-pipeline / per-step widening or narrowing, decided here rather than in the
# repository. BUILDKITE_PIPELINE_SLUG and BUILDKITE_STEP_KEY are protected
# read-only job variables, so a pipeline cannot forge its way into another
# branch of this case. BUILDKITE_STEP_KEY is EMPTY for a step with no `key:`.
case "${BUILDKITE_PIPELINE_SLUG:-}" in
  payments-service)
    case "${BUILDKITE_STEP_KEY:-}" in
      deploy-prod)
        # Cluster-wide subject: accepted here only because this queue is a
        # dedicated production cluster (control 3.2) and the relying party
        # demands an exact-match subject. Strictly WIDER than the default —
        # the branch and step conditions live in the cloud trust policy.
        export BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM="cluster_id"
        ;;
      "")
        echo "hth-3.6: refusing to mint an OIDC token for a step with no 'key:'" >&2
        echo "         in pipeline '${BUILDKITE_PIPELINE_SLUG}'. Add a step key." >&2
        exit 1
        ;;
    esac
    ;;
esac

# Vault's plugin does not mint a token; it reads one from the environment. Mint
# it here so it exists before the vault-secrets environment hook runs. The
# audience must match the `bound_audiences` on the Vault JWT role.
#
# No --claim here. The Vault JWT role's bound_claims are written against
# build_branch / step_key / pipeline_slug, and all three are DEFAULT claims that
# every token already carries — --claim adds from the optional table only and
# cannot re-add them. organization_id is the one worth adding: it pins the role to
# a tenant UUID that survives an organization rename, unlike organization_slug.
if [ "${BUILDKITE_PIPELINE_SLUG:-}" = "payments-service" ] &&
   [ "${BUILDKITE_STEP_KEY:-}" = "db-migrate" ]; then
  BUILDKITE_OIDC_VAULT_JWT="$(buildkite-agent oidc request-token \
    --audience "https://vault.internal.example.com" \
    --subject-claim "${BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM}" \
    --lifetime 300 \
    --claim "organization_id")"
  export BUILDKITE_OIDC_VAULT_JWT
fi
HOOKEOF

  echo "Wrote block '${HTH_HOOK_BLOCK_ID}' in ${hook}."
  echo "Restart buildkite-agent, then run: $0 audit ${AGENT_HOOKS_PATH}"
}

# Prove what the token actually says, rather than what the pipeline intended.
# Decodes the JWT payload locally — no network, no verification of the signature
# (that is the relying party's job; this is a configuration check).

decode_jwt_payload() {
  local jwt="${1:?jwt required}" payload
  payload="$(printf '%s' "${jwt}" | cut -d. -f2)"
  # base64url -> base64, then pad to a multiple of 4.
  payload="$(printf '%s' "${payload}" | tr '_-' '/+')"
  case $(( ${#payload} % 4 )) in
    2) payload="${payload}==" ;;
    3) payload="${payload}=" ;;
  esac
  # GNU coreutils uses -d; BSD/macOS uses -D.
  printf '%s' "${payload}" | base64 -d 2>/dev/null ||
    printf '%s' "${payload}" | base64 -D
}

# Fails closed when the minted subject is not the one that was asked for — the
# exact symptom of T2 (a plugin minting a default compound subject while the
# cloud policy expects a bare UUID).
verify_subject() {
  local jwt="${1:?jwt required}" expect_claim="${2:?expected subject claim name required}"
  local payload sub expect_value rc=0

  command -v jq >/dev/null 2>&1 || { echo "FATAL: jq required." >&2; exit 127; }
  payload="$(decode_jwt_payload "${jwt}")"

  sub="$(jq -r '.sub // empty' <<<"${payload}")"
  expect_value="$(jq -r --arg c "${expect_claim}" '.[$c] // empty' <<<"${payload}")"

  echo "iss  : $(jq -r '.iss // "<absent>"' <<<"${payload}")"
  echo "aud  : $(jq -r '.aud // "<absent>"' <<<"${payload}")"
  echo "sub  : ${sub:-<absent>}"
  echo "exp-iat (s): $(jq -r 'if .exp and .iat then (.exp - .iat) else "<unknown>" end' <<<"${payload}")"

  local is_default=0
  case "${sub}" in organization:*:pipeline:*) is_default=1 ;; esac

  if [ "${is_default}" -eq 1 ]; then
    echo "SUBJECT: default compound (org/pipeline/ref/commit/step)."
    echo "FAIL: expected the subject to be the '${expect_claim}' value, but the" >&2
    echo "      token carries the DEFAULT compound subject. The token was minted" >&2
    echo "      without --subject-claim and without BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM" >&2
    echo "      in scope — neither cloud plugin passes that flag itself." >&2
    rc=1
  elif [ -z "${expect_value}" ]; then
    echo "FAIL: claim '${expect_claim}' is absent from the token, so the subject" >&2
    echo "      cannot have come from it." >&2
    rc=1
  elif [ "${sub}" != "${expect_value}" ]; then
    echo "FAIL: sub does not equal the '${expect_claim}' value ('${expect_value}')." >&2
    rc=1
  else
    echo "PASS: sub is the ${expect_claim} value."
  fi

  # T9/T6: a long-lived bearer token is the residual risk of the whole control.
  local life
  life="$(jq -r 'if .exp and .iat then (.exp - .iat) else -1 end' <<<"${payload}")"
  if [ "${life}" -gt 3600 ] 2>/dev/null; then
    echo "WARN: token lifetime ${life}s exceeds one hour. It is a bearer credential" >&2
    echo "      for that entire window; nothing binds it to the process that minted it." >&2
  fi

  return "${rc}"
}

# Host-side review. Catches the configurations that turn this control off: a
# repository choosing its own subject, redaction being disabled, and — the one
# this pack used to be blind to — the agent-side block being gone. Nothing here
# greps the hook file until now, so a whole-file overwrite by another pack (or by
# a config-management run) removed the entire control while every check passed.
audit_oidc_usage() {
  local root="${1:-.}" rc=0 hits
  local hook="${AGENT_HOOKS_PATH}/environment"

  echo "scanning: ${root}"

  # 0. Is the control still installed on this host? Only meaningful where the
  # hooks-path exists, i.e. when auditing an agent rather than a repository.
  if [ -d "${AGENT_HOOKS_PATH}" ]; then
    if [ ! -e "${hook}" ]; then
      echo "FAIL: ${hook} does not exist. Nothing sets" >&2
      echo "      BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM off-repo, so every token" >&2
      echo "      carries the default compound subject. Run: $0 install-hook" >&2
      rc=1
    elif ! grep -q "HTH-BLOCK ${HTH_HOOK_BLOCK_ID}" "${hook}"; then
      echo "FAIL: ${hook} exists but carries no '${HTH_HOOK_BLOCK_ID}' block." >&2
      echo "      Either it was never installed, or something rewrote the file" >&2
      echo "      whole — control 3.9 writes the same hook. Check for a" >&2
      echo "      ${hook}.hth-bak.* sibling, then re-run: $0 install-hook" >&2
      rc=1
    elif ! grep -q 'BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM' "${hook}"; then
      echo "FAIL: the '${HTH_HOOK_BLOCK_ID}' block no longer sets" >&2
      echo "      BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM." >&2
      rc=1
    else
      echo "PASS: ${hook} carries the ${HTH_HOOK_BLOCK_ID} block and sets the subject claim."
    fi

    # A hook anyone but root can rewrite is arbitrary code on every job, and
    # would let the pipeline's own owners restore a subject of their choosing.
    if [ -e "${hook}" ] && [ -n "$(find "${hook}" -perm -g+w -o -perm -o+w 2>/dev/null)" ]; then
      echo "FAIL: ${hook} is group- or world-writable. The off-repo guarantee this" >&2
      echo "      control depends on is only as strong as the file's permissions." >&2
      rc=1
    fi
  fi

  hits="$(grep -rIn --include='*.yml' --include='*.yaml' \
          'BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM' "${root}" 2>/dev/null || true)"
  if [ -n "${hits}" ]; then
    echo "FAIL: pipeline YAML sets the OIDC subject claim. Anyone who can merge to" >&2
    echo "      this repository can then widen their own token's subject. Move it" >&2
    echo "      to the agent 'environment' hook." >&2
    printf '%s\n' "${hits}" >&2
    rc=1
  else
    echo "PASS: no pipeline YAML sets BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM."
  fi

  hits="$(grep -rIn -e '--skip-redaction' \
          -e 'BUILDKITE_AGENT_OIDC_REQUEST_TOKEN_SKIP_TOKEN_REDACTION' \
          "${root}" 2>/dev/null || true)"
  if [ -n "${hits}" ]; then
    echo "FAIL: OIDC token redaction is disabled somewhere in this tree. The raw" >&2
    echo "      JWT will be printed to the build log." >&2
    printf '%s\n' "${hits}" >&2
    rc=1
  else
    echo "PASS: OIDC log redaction is not disabled."
  fi

  # --claims is not a flag; the request fails outright. Cheap to catch statically.
  hits="$(grep -rIn -e '--claims' "${root}" 2>/dev/null || true)"
  if [ -n "${hits}" ]; then
    echo "FAIL: '--claims' is not a buildkite-agent flag. The flag is '--claim'," >&2
    echo "      taking one comma-separated value." >&2
    printf '%s\n' "${hits}" >&2
    rc=1
  fi

  # T5b. A default claim passed to --claim reads as scoping and is not. Matched on
  # the same line as the flag so an --aws-session-tag carrying the same name — which
  # is correct — is not flagged.
  hits="$(grep -rIn -E -e '--claim[ =]"?[^"]*\b(organization_slug|pipeline_slug|build_number|build_branch|build_tag|build_commit|step_key|runner_environment|build_source)\b' \
          "${root}" 2>/dev/null || true)"
  if [ -n "${hits}" ]; then
    echo "WARN: '--claim' is passed a DEFAULT claim, which every token already" >&2
    echo "      carries. --claim adds from the optional table only" >&2
    echo "      (${OPTIONAL_CLAIMS_ALLOWED} agent_tag:<NAME>)." >&2
    echo "      Condition on it in the relying party's policy, or on AWS carry it" >&2
    echo "      with --aws-session-tag, which does accept default claims." >&2
    printf '%s\n' "${hits}" >&2
  fi

  return "${rc}"
}

Source: Buildkite security controls

3.7 Delegate Cluster Administration

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	5.4
NIST 800-53	AC-5, AC-6(1)

Description

Name explicit cluster maintainers so that managing a cluster’s agent tokens, queues, and secrets does not require granting organization-wide administrator access.

Rationale

Why This Matters:

Without maintainers, the only way to let someone manage a cluster is to make them an organization admin — so the isolation boundary 3.2 establishes is routinely undone by the access needed to operate it
Cluster maintainers hold authority over agent tokens, queues and cluster secrets, which makes an unmanaged maintainer list equivalent to an unmanaged list of people who can read production credentials
Maintainer grants are invisible from the organization member list, so an access review that only reads roles will miss them entirely
The set of people who need to operate a cluster changes faster than the set of people who should administer the organization, and conflating the two guarantees the wider grant persists

Attack Prevented: Privilege escalation through unnecessary org-admin grants, unaudited access to cluster secrets, persistence via maintainer grants that survive an org-role review

ClickOps Implementation

Step 1: Inventory Current Maintainers

Go to Agents → Clusters, open each cluster, and review its Maintainers list.
Record whether each entry is a user or a team — team-scoped maintainers age better, because team membership is already reviewed.

Step 2: Replace Org-Admin Grants

For each person who holds organization admin solely to operate a cluster, add them as a maintainer of that cluster and remove the org-admin grant per 2.3.

Step 3: Prefer Teams Over Individuals

Assign maintainership to a team rather than named individuals wherever the workflow allows, so joiner/leaver handling flows through the existing team process instead of a second list nobody remembers.

Code Implementation

Code Pack: Terraform

hth-buildkite-3.07-cluster-maintainers.tf View source on GitHub ↗

locals {
  # Every cluster this pack touches: the ones explicitly audited plus every
  # cluster named by a declared maintainer. Resolving by name keeps one source of
  # truth and yields the plain `uuid` the maintainer resource requires (TRAP 4).
  hth_governed_clusters = toset(concat(
    var.audited_clusters,
    [for m in var.cluster_maintainers : m.cluster],
  ))
}

data "buildkite_cluster" "governed" {
  for_each = local.hth_governed_clusters

  name = each.value
}

# Scoped cluster administration. Each entry replaces an organization-admin grant
# that would otherwise have been issued just to manage one cluster's tokens,
# queues and secrets.
resource "buildkite_cluster_maintainer" "delegated" {
  for_each = var.cluster_maintainers

  cluster_uuid = data.buildkite_cluster.governed[each.value.cluster].uuid

  # Exactly one of these is non-null. Teams are the default shape; see TRAP 3.
  team_uuid = each.value.team_uuid
  user_uuid = each.value.user_uuid

  lifecycle {
    # TRAP 1. Surface the exactly-one-of constraint with the entry's key in it.
    precondition {
      condition     = (each.value.team_uuid == null) != (each.value.user_uuid == null)
      error_message = format("Maintainer '%s' must set exactly one of team_uuid or user_uuid. A team is preferred: team membership changes revoke cluster authority automatically, a direct user grant does not.", each.key)
    }
  }
}

# TRAP 3, enforced. Direct user maintainers are the grants that outlive
# offboarding, so the count is bounded and the bound is zero by default.
check "direct_user_maintainers_within_bound" {
  assert {
    condition = length([for m in var.cluster_maintainers : m if m.user_uuid != null]) <= var.max_direct_user_maintainers
    error_message = format(
      "%d cluster maintainers are individual users; at most %d are permitted. Grant cluster administration to a team_uuid instead, or raise var.max_direct_user_maintainers deliberately to record a break-glass individual.",
      length([for m in var.cluster_maintainers : m if m.user_uuid != null]),
      var.max_direct_user_maintainers,
    )
  }
}

output "cluster_maintainer_grants" {
  description = "Every declared cluster administration grant as Buildkite resolved it. actor_type is computed - it reflects which UUID was supplied, not a declaration."
  value = {
    for k, m in buildkite_cluster_maintainer.delegated : k => {
      cluster      = var.cluster_maintainers[k].cluster
      cluster_uuid = m.cluster_uuid
      actor_type   = m.actor_type
      actor_name   = m.actor_name
      actor_slug   = m.actor_slug
      actor_uuid   = m.actor_uuid
    }
  }
}

# The write half above only governs the grants Terraform made. The boundary risk
# is the grant somebody added in the console. data.buildkite_cluster returns the
# live maintainer list for a cluster, so drift is detectable without a resource.
locals {
  # Actors approved to administer a cluster: everyone this configuration grants,
  # plus any UUID explicitly accepted as pre-existing.
  hth_approved_maintainer_uuids = toset(concat(
    [for m in var.cluster_maintainers : coalesce(m.team_uuid, m.user_uuid)],
    var.additional_approved_maintainer_uuids,
  ))

  # Live maintainers across every governed cluster, flattened for comparison.
  hth_live_maintainers = flatten([
    for name, c in data.buildkite_cluster.governed : [
      for m in c.maintainers : {
        cluster    = name
        actor_type = m.actor_type
        actor_uuid = m.actor_uuid
        actor_name = m.actor_name
      }
    ]
  ])

  hth_unapproved_maintainers = [
    for m in local.hth_live_maintainers : m
    if !contains(local.hth_approved_maintainer_uuids, m.actor_uuid)
  ]
}

# Continuous validation: reports on every plan and apply, does not block. An
# unapproved maintainer is a review finding about who can mint agent tokens and
# read cluster secrets - surface it every run rather than failing an unrelated
# deploy at 2am.
check "no_unapproved_cluster_maintainers" {
  assert {
    condition = length(local.hth_unapproved_maintainers) == 0
    error_message = format(
      "%d cluster maintainer(s) are not declared in Terraform and hold agent-token, queue and secret authority over their cluster: %s. Either declare them in var.cluster_maintainers, accept them in var.additional_approved_maintainer_uuids, or remove them in the Buildkite console.",
      length(local.hth_unapproved_maintainers),
      jsonencode(local.hth_unapproved_maintainers),
    )
  }
}

output "cluster_maintainer_audit" {
  description = "Live maintainer roster for every governed cluster, and the subset holding authority without a Terraform declaration."
  value = {
    live       = local.hth_live_maintainers
    unapproved = local.hth_unapproved_maintainers
  }
}

Source: Manage clusters and queues

3.8 Attest Build Artifacts

Profile Level: L3 (Run)

Framework	Control
CIS Controls	16.1
NIST 800-53	SI-7, SR-4

Description

Generate SLSA build provenance for the artifacts a pipeline produces, so a consumer can verify where an artifact came from rather than only that the pipeline definition was signed.

Rationale

Why This Matters:

Pipeline signing (3.4) proves the instructions were not tampered with; it says nothing about the artifact those instructions produced, and the two are separate integrity claims
Without provenance, an artifact in a registry is an anonymous blob — nothing binds it to the commit, pipeline, and build that created it, so a substituted artifact is indistinguishable from a legitimate one
Downstream consumers cannot enforce a policy such as “only deploy artifacts built by our pipeline from our main branch” unless the artifact carries a verifiable statement of that fact
Provenance is what makes a compromise investigable after the fact: without it, establishing which builds produced which shipped artifacts is archaeology rather than a query

Attack Prevented: Artifact substitution between build and deploy, laundering a malicious artifact through a trusted registry, unverifiable software supply chains during incident response

ClickOps Implementation

Step 1: Decide What Must Carry Provenance

Identify the artifacts that cross a trust boundary — anything published to a registry, shipped to customers, or deployed to production.
Artifacts that never leave the build are lower priority; start where the blast radius is largest.

Step 2: Enable Attestation Generation

Add the provenance plugin to the pipelines producing those artifacts. It runs as a post-artifact hook and emits an in-toto Statement wrapped in a DSSE v1.0 Envelope, meeting SLSA Build Level 1.

Step 3: Verify Downstream

Make the deploy step check the attestation rather than merely accepting that one exists. An attestation nobody verifies provides no security property.
Check the statement’s contents, not its signature. Buildkite’s plugin documents that the in-toto Envelope “is currently signed using a hard-coded private key for demonstration purposes” — which is exactly why this flow reaches SLSA Build Level 1 (provenance exists) and not Level 2 (provenance is authentic). Verify that the pipeline, repository, and commit the statement names are the ones you expected to build, and treat the envelope signature as carrying no origin guarantee until Buildkite supports a signing key you control.

Note on plan gating: Buildkite gates the whole SLSA provenance feature, not just the publishing half — its documentation states “The SLSA provenance feature is only available to Buildkite customers on Enterprise plans,” and places that gate ahead of the generation step, not between generation and publishing. Treat both halves as Enterprise-only unless Buildkite confirms otherwise for your organization. On lower plans, obtain the same property outside Buildkite — generate an in-toto statement in the build itself and sign and store it with a tool such as cosign — rather than assuming the Buildkite plugin path is available to you.

Code Implementation

Code Pack: Config

hth-buildkite-3.08-provenance.yml View source on GitHub ↗

# pipeline.yml -- add under `steps:` in the pipeline that BUILDS the artifact.
# Two steps: the plugin GENERATES provenance (SLSA Build L1, unauthenticated),
# then a second step gives that provenance a signature worth trusting.

- label: ":package: build"
  key: "build"
  command: "./scripts/build.sh"
  # post-artifact fires after THIS upload, so the globs must be declared here.
  artifact_paths:
    - "dist/*.tar.gz"
  plugins:
    # Pin to a tag or commit SHA. This reference becomes the `builder.id` the
    # attestation asserts, and it is the code that computes the digests.
    - generate-provenance-attestation#v1.1.0:
        # Single glob STRING (plugin.yml: type: string). Both keys are
        # required; additionalProperties is false, so nothing else is accepted.
        artifacts: "dist/*.tar.gz"
        attestation_name: "provenance.json"

- label: ":lock: sign provenance (sigstore keyless)"
  key: "sign-provenance"
  depends_on: "build"
  command: |
    set -euo pipefail

    buildkite-agent artifact download provenance.json .

    # Refuse to build on the plugin's own signature. v1.1.0 signs with a key
    # published in its repository; treating it as authentic is worse than
    # having no signature, because it looks like one.
    KEYID="$$(jq -r '.signatures[0].keyid // ""' provenance.json)"
    if [ "$$KEYID" = "generate_provenance_attestation_plugin_example_key" ]; then
      echo "note: plugin DSSE signature is the published example key - discarding it" >&2
    fi

    # The Statement is the payload. That is what gets a real signature.
    jq -r '.payload' provenance.json | base64 -d > statement.json
    jq -e '._type == "https://in-toto.io/Statement/v1"
           and .predicateType == "https://slsa.dev/provenance/v1"
           and (.subject | length) > 0' statement.json > /dev/null

    # Job-scoped OIDC token. Fulcio trusts issuer https://agent.buildkite.com
    # with audience `sigstore`; the certificate SAN it mints is
    # https://buildkite.com/<org-slug>/<pipeline-slug>.
    OIDC_TOKEN="$$(buildkite-agent oidc request-token --audience sigstore --lifetime 300)"

    cosign sign-blob \
      --yes \
      --identity-token "$$OIDC_TOKEN" \
      --bundle statement.cosign.bundle \
      statement.json

    buildkite-agent artifact upload statement.json
    buildkite-agent artifact upload statement.cosign.bundle

# pipeline.yml -- ENTERPRISE ONLY (Buildkite Package Registries).
# DRIFT-CHECKED-ONLY: authored from publish-to-packages v2.2.0's pinned
# plugin.yml; not executed. `artifacts` and `registry` are both required,
# `attestations` accepts a string or an array, additionalProperties is false.
# On any other plan, delete this step: the generation and signing above are
# ungated and the cosign bundle publishes to your own registry unchanged.

- label: ":outbox_tray: publish with attestation"
  key: "publish"
  depends_on: "sign-provenance"
  plugins:
    - publish-to-packages#v2.2.0:
        artifacts: "dist/*.tar.gz"
        registry: "my-org/my-registry"
        # Ship the attestation WITH the package. An attestation that stays in
        # the build is provenance nobody downstream can reach.
        attestations:
          - "provenance.json"

# pipeline.yml -- run this in the CONSUMING pipeline before deploy.
# Guide Step 3 is the load-bearing one: an attestation nobody verifies is a
# file, not a control. Every check below exits non-zero on failure, so the
# deploy step gated on it cannot run against an artifact whose provenance is
# missing, unsigned, forged, or points at the wrong commit.

- label: ":mag: verify provenance before deploy"
  key: "verify-provenance"
  depends_on: "sign-provenance"
  command: |
    set -euo pipefail

    # Policy. Deploy is allowed ONLY for artifacts this pipeline built from
    # this repository on this branch. Anything looser is not a control.
    EXPECT_IDENTITY="https://buildkite.com/my-org/my-build-pipeline"
    EXPECT_ISSUER="https://agent.buildkite.com"
    EXPECT_REPO="https://github.com/my-org/my-repo.git"
    EXPECT_BRANCH="main"
    ARTIFACT="dist/app.tar.gz"

    # `artifact download` defaults --build to $BUILDKITE_BUILD_ID (the current
    # build). When the consumer is a SEPARATE pipeline, append
    # `--build <upstream-build-uuid>` to each of these three commands.
    buildkite-agent artifact download "$$ARTIFACT" .
    buildkite-agent artifact download statement.json .
    buildkite-agent artifact download statement.cosign.bundle .

    # 1. AUTHENTICITY. Pin both the identity and the issuer: --certificate-identity
    #    alone is satisfiable by any issuer Fulcio trusts.
    cosign verify-blob \
      --bundle statement.cosign.bundle \
      --certificate-identity "$$EXPECT_IDENTITY" \
      --certificate-oidc-issuer "$$EXPECT_ISSUER" \
      statement.json

    # 2. INTEGRITY. Bind the signed statement to the bytes actually downloaded.
    #    Without this the signature only proves a statement existed, not that
    #    it describes this file. sha256sum is GNU-only; macOS agents have shasum.
    if command -v sha256sum > /dev/null 2>&1; then
      ACTUAL_SHA="$$(sha256sum "$$ARTIFACT" | cut -d' ' -f1)"
    else
      ACTUAL_SHA="$$(shasum -a 256 "$$ARTIFACT" | cut -d' ' -f1)"
    fi
    jq -e --arg sha "$$ACTUAL_SHA" \
      '[.subject[] | select(.digest.sha256 == $$sha)] | length == 1' \
      statement.json > /dev/null \
      || { echo "FAIL: $$ARTIFACT (sha256 $$ACTUAL_SHA) is not a subject of the signed statement" >&2; exit 1; }

    # 3. PROVENANCE POLICY. The statement is authentic and describes this file;
    #    now decide whether that origin is one you deploy.
    jq -e --arg repo "$$EXPECT_REPO" --arg branch "$$EXPECT_BRANCH" \
      '.predicate.buildDefinition.externalParameters.repository == $$repo
       and .predicate.buildDefinition.externalParameters.build.branch == $$branch
       and (.predicate.buildDefinition.externalParameters.build.commit | length) == 40' \
      statement.json > /dev/null \
      || { echo "FAIL: provenance does not match the deploy policy (repo/branch/commit)" >&2; exit 1; }

    # 4. Never accept the plugin's own envelope as the authenticity evidence.
    #    If a consumer ever hands you provenance.json instead of the signed
    #    statement + bundle, this is the check that stops it.
    if [ -f provenance.json ]; then
      BAD="$$(jq -r '.signatures[0].keyid // ""' provenance.json)"
      if [ "$$BAD" = "generate_provenance_attestation_plugin_example_key" ]; then
        echo "FAIL: DSSE envelope is signed with the plugin's published example key." >&2
        echo "      That key is in the plugin repo; anyone can forge this envelope." >&2
        echo "      Verify statement.json against statement.cosign.bundle instead." >&2
        exit 1
      fi
    fi

    echo "PASS: provenance authentic, bound to $$ARTIFACT, and within deploy policy"

Sources: Generate and store SLSA provenance · Generate Provenance Attestation plugin

3.9 Harden the Agent Execution Environment

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	4.1
NIST 800-53	CM-6, CM-7

Description

Configure the agent’s own execution behavior — host-key handling, workspace hygiene between jobs, and the bootstrap path — as distinct from the input controls in 2.4.

Rationale

Why This Matters:

no-ssh-keyscan defaults to false, which means the agent blindly accepts whatever host key your git server presents on first checkout — a textbook trust-on-first-use window against the source of everything it builds
On a persistent agent, the workspace survives between jobs, so one poisoned build can leave artifacts, credentials, or modified tooling behind for the next unrelated build to pick up
Steps 1 and 3 are genuinely agent-authoritative — no-ssh-keyscan and bootstrap-script are agent configuration options with no job-level surface, which makes them controls a malicious pipeline definition cannot switch off
Step 2 is the exception, and knowing that is the point of reading this control. BUILDKITE_CLEAN_CHECKOUT is not an agent configuration option at all and is outside the scope checkout-override-mode governs, so nothing locks it against a pipeline — assuming otherwise is how a clean-workspace control gets recorded as implemented while remaining switchable from the pipeline it is supposed to constrain
The controls are individually small and collectively decisive, which is exactly the profile of settings that get skipped because no single one looks urgent

Attack Prevented: Man-in-the-middle against the source host during checkout, build-to-build contamination on persistent agents, tampering with the bootstrap path

ClickOps Implementation

Step 1: Disable Automatic Host-Key Acceptance

Set no-ssh-keyscan=true in buildkite-agent.cfg and pre-populate known_hosts on the agent host with the verified key for your git server. Provisioning the key deliberately is the point; disabling keyscan without it just breaks checkout.

Step 2: Force a Clean Workspace

Set BUILDKITE_CLEAN_CHECKOUT=true. This is not a buildkite-agent.cfg key — it does not appear in the agent configuration reference at all; it is a bootstrap/job environment variable, and writing it into the config file is silently ignored. Set it through a global environment hook on the agent host, which runs after the pipeline’s own step environment and therefore overrides a pipeline.yml that set it to false.
The hook wins over a pipeline, not over a plugin. Unlike no-ssh-keyscan, this variable is not agent-authoritative: checkout-override-mode governs the agent’s checkout settings — flags, timeout, submodules, skip-checkout, skip-fetch, sparse-checkout, mirror and submodule-clone config — and BUILDKITE_CLEAN_CHECKOUT is not among them, so even strict does not lock it. Plugin environment hooks run after the global one, so a plugin can set it back to false. The plugin allowlist in 2.4 is load-bearing for this step in a way it is not for Step 1.
The form that cannot be argued with is an ephemeral agent. Run agents with disconnect-after-job so the workspace is destroyed with the agent rather than cleaned in place. It is an agent-daemon option, so no pipeline, secret, hook, or plugin has a surface to turn it off — which is the property Step 2 is otherwise missing.

Step 3: Consider a Bootstrap Handler

For maximum control, point bootstrap-script at a wrapper that applies your own admission logic before calling buildkite-agent bootstrap. Keep the wrapper thin — Buildkite documents no handler contract beyond invoking bootstrap, so elaborate logic here is building on unspecified behavior.

Code Implementation

Code Pack: Config

hth-buildkite-3.09-agent-env.sh View source on GitHub ↗

# Pin the git host key. Provision known_hosts FIRST, then turn off TOFU —
# reversing that order just breaks every checkout on this agent.

# Capture the server's key, then require a human to confirm the fingerprint out
# of band. This deliberately does not auto-accept: fetching a key over the same
# path an attacker would sit on and calling it verified is TOFU with extra steps.
propose_known_hosts() {
  local host="${1:?usage: propose_known_hosts <git-host> [port]}"
  local port="${2:-22}"
  local staged
  staged="$(mktemp)"

  # ssh-keyscan emits a "# host:port SSH-2.0-..." banner alongside the keys; drop
  # it so an emptiness check cannot pass on a banner alone.
  ssh-keyscan -p "${port}" -t rsa,ecdsa,ed25519 "${host}" 2>/dev/null \
    | grep -vE '^[[:space:]]*(#.*)?$' > "${staged}" || true
  [ -s "${staged}" ] || { echo "FATAL: no host keys returned for ${host}:${port}" >&2; rm -f "${staged}"; exit 3; }

  echo "Fingerprints for ${host}:${port} — CONFIRM THESE against the vendor's"
  echo "published SSH key fingerprints before running accept_known_hosts:"
  ssh-keygen -lf "${staged}"
  echo
  echo "staged file: ${staged}"
  echo "then: $0 accept-known-hosts ${staged}"
}

accept_known_hosts() {
  local staged="${1:?usage: accept_known_hosts <staged-file-from-propose>}"
  [ -s "${staged}" ] || { echo "FATAL: '${staged}' is empty or missing." >&2; exit 3; }

  # Append-and-dedupe so multiple git hosts can be pinned over several runs.
  touch "${KNOWN_HOSTS}"
  sort -u "${KNOWN_HOSTS}" "${staged}" | grep -vE '^[[:space:]]*(#.*)?$' > "${KNOWN_HOSTS}.new"
  mv "${KNOWN_HOSTS}.new" "${KNOWN_HOSTS}"
  chmod 0644 "${KNOWN_HOSTS}"
  echo "known_hosts now pins $(grep -c . "${KNOWN_HOSTS}") key(s) at ${KNOWN_HOSTS}"
}

# Turn off automatic host-key acceptance. Fails closed if known_hosts is not
# populated, because StrictHostKeyChecking=yes with an empty known_hosts is an
# outage, not a control.
apply_host_key_pinning() {
  [ -w "${AGENT_CFG}" ] || { echo "FATAL: cannot write ${AGENT_CFG} (run as root)." >&2; exit 5; }
  if [ ! -s "${KNOWN_HOSTS}" ]; then
    echo "FATAL: ${KNOWN_HOSTS} is empty or missing." >&2
    echo "       Run '$0 propose-known-hosts <git-host>' and verify the fingerprints" >&2
    echo "       first. Setting no-ssh-keyscan=true without this breaks checkout on" >&2
    echo "       every job." >&2
    exit 3
  fi
  set_cfg "no-ssh-keyscan" "true"
  echo "Applied no-ssh-keyscan=true. Restart the agent, then run: $0 audit"
}

# The agent-level `environment` hook. This is the ONLY supported place to set
# BUILDKITE_CLEAN_CHECKOUT, and the only place that can win the GIT_SSH_COMMAND
# race, because it runs after the agent configured SSH and before checkout.
#
# ── ⚠️ THE HOOK FILE IS SHARED. DO NOT WRITE IT WHOLE. ──────────────────────
# An agent has exactly ONE ${hooks-path}/environment, and control 3.6
# (packs/buildkite/cli/hth-buildkite-3.06-oidc-subject.sh) needs the same file:
# it is the only off-repo place to set BUILDKITE_OIDC_TOKEN_SUBJECT_CLAIM, and a
# pipeline that can set that variable can widen its own cloud trust. This pack
# previously did `cat > "${hook}"` with no existence check and no backup, so
# adopting 3.6 and then 3.9 deleted 3.6's control outright — silently, because
# neither audit read the other's settings.
#
# So both packs write a DELIMITED BLOCK and rewrite only their own:
#   # >>> HTH-BLOCK <id>
#   ...
#   # <<< HTH-BLOCK <id>
# hth_write_hook_block() below is byte-identical to the copy in pack 3.6 apart
# from the block id. It preserves every other line in the file — the other pack's
# block, and any hook the operator wrote themselves — and takes a timestamped
# backup before touching anything. Running either pack twice is idempotent.

# The one line that differs between the two copies of this protocol.
HTH_HOOK_BLOCK_ID="hth-3.9-agent-env"

# $1 = hook path, $2 = block id, block body on stdin.
hth_write_hook_block() {
  local hook="$1" id="$2"
  local dir tmp begin end
  dir="$(dirname "${hook}")"
  begin="# >>> HTH-BLOCK ${id}"
  end="# <<< HTH-BLOCK ${id}"

  [ -d "${dir}" ] || { echo "FATAL: hooks-path '${dir}' does not exist." >&2; exit 5; }
  [ -w "${dir}" ] || { echo "FATAL: cannot write '${dir}' (run as root)." >&2; exit 5; }

  tmp="$(mktemp "${dir}/.hth-environment.XXXXXX")"

  if [ -e "${hook}" ]; then
    # Backup first, always — including when the result will be identical. A hook
    # is arbitrary code that runs as the agent user on every job; there is no
    # such thing as an edit here that is not worth being able to undo. The
    # counter matters: the stamp has one-second resolution, and installing 3.6
    # then 3.9 back to back lands in the same second, so a bare stamp would let
    # the second install overwrite the backup of the operator's ORIGINAL file.
    # A backup is never overwritten. (`environment.hth-bak.*` is inert — the
    # agent looks up hooks by exact filename, not by glob.)
    local stamp bak n=0
    stamp="$(date -u +%Y%m%dT%H%M%SZ)"
    bak="${hook}.hth-bak.${stamp}"
    while [ -e "${bak}" ]; do n=$((n + 1)); bak="${hook}.hth-bak.${stamp}-${n}"; done
    cp -p "${hook}" "${bak}"
    echo "Backed up ${hook} -> ${bak}"

    # Carry over everything except a previous copy of THIS block. awk with
    # index() rather than a regex: the markers contain > and < and the id
    # contains dots, none of which should be read as metacharacters.
    awk -v b="${begin}" -v e="${end}" '
      index($0, b) == 1 { skip = 1; next }
      index($0, e) == 1 { skip = 0; next }
      !skip { print }
    ' "${hook}" >"${tmp}"

    # An `exit` in code that already ran means this block never will. Cheap to
    # detect, invisible at runtime (the hook "succeeds" and does nothing).
    # Only unmanaged lines are scanned: the sibling pack's block legitimately
    # exits to refuse a job, and warning about that every run would be noise.
    if awk '
         /^# >>> HTH-BLOCK / { skip = 1; next }
         /^# <<< HTH-BLOCK / { skip = 0; next }
         !skip { print }
       ' "${tmp}" | grep -qE '^[[:space:]]*exit([[:space:]]|$)'; then
      echo "WARN: ${hook} contains an 'exit' outside any HTH-BLOCK. If it runs" >&2
      echo "      before the block below, the block never executes." >&2
    fi
  else
    cat >"${tmp}" <<'HEADEOF'
#!/usr/bin/env bash
# Buildkite agent `environment` hook.
# Runs once per job, before checkout and before every plugin environment hook.
# Sections delimited by "HTH-BLOCK <id>" markers are managed by How to Harden
# packs and are rewritten in place; edit outside them.
set -euo pipefail
HEADEOF
  fi

  printf '\n%s\n' "${begin}" >>"${tmp}"
  cat >>"${tmp}"
  printf '%s\n' "${end}" >>"${tmp}"

  chmod 0755 "${tmp}"
  chown root:root "${tmp}" 2>/dev/null || true
  mv "${tmp}" "${hook}"
}

write_environment_hook() {
  local hooks hook
  hooks="$(hooks_dir)"
  hook="${hooks}/environment"
  mkdir -p "${hooks}"

  # Quoted heredoc so nothing is expanded at write time, piped through sed for
  # the one placeholder. The block body carries no shebang and no `set` — those
  # belong to the file, which hth_write_hook_block owns.
  sed "s|__KNOWN_HOSTS__|${KNOWN_HOSTS}|g" <<'HOOKEOF' | hth_write_hook_block "${hook}" "${HTH_HOOK_BLOCK_ID}"
# git host-key trust + workspace hygiene. Managed by HTH control 3.9.

# --- git host-key trust -----------------------------------------------------
# REBUILT, not appended. ssh resolves duplicate -o options first-wins, so a
# pipeline-supplied GIT_SSH_COMMAND containing StrictHostKeyChecking=no would
# beat the option the agent appends. Discard whatever arrived and state the
# policy from scratch. GIT_SSH is unset because ssh_host_key_checking.go skips
# all configuration when it is present, and git prefers GIT_SSH_COMMAND anyway.
unset GIT_SSH
export GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=yes -o UserKnownHostsFile=__KNOWN_HOSTS__"

# --- workspace hygiene ------------------------------------------------------
# NOT a buildkite-agent.cfg key. Exported here so applyEnvironmentChanges ->
# ReadFromEnvironment refreshes the executor's CleanCheckout before the checkout
# phase reads it. Drop this block on agents that already run one job and exit
# (disconnect-after-job): re-cloning a workspace nothing else will ever touch is
# pure build time.
export BUILDKITE_CLEAN_CHECKOUT=true

# Do not print the environment from this hook. BUILDKITE_AGENT_ACCESS_TOKEN is
# in it.
HOOKEOF

  echo "Wrote block '${HTH_HOOK_BLOCK_ID}' in ${hook}."
}

# The maximum-control option, kept deliberately thin.
write_bootstrap_wrapper() {
  [ -x "${AGENT_BIN}" ] || { echo "FATAL: agent binary '${AGENT_BIN}' not found or not executable." >&2; exit 3; }

  cat > "${BOOTSTRAP_WRAPPER}" <<HOOKEOF
#!/usr/bin/env bash
set -euo pipefail

# Buildkite documents no handler contract beyond invoking bootstrap, and passes
# no job data in argv — it is all in the environment. Add admission checks here
# ONLY if they read documented BUILDKITE_* variables; never dump the
# environment, which carries BUILDKITE_AGENT_ACCESS_TOKEN.

exec "${AGENT_BIN}" bootstrap "\$@"
HOOKEOF

  chown root:root "${BOOTSTRAP_WRAPPER}" 2>/dev/null || true
  chmod 0755 "${BOOTSTRAP_WRAPPER}"
  set_cfg "bootstrap-script" "${BOOTSTRAP_WRAPPER}"
  echo "Wrote ${BOOTSTRAP_WRAPPER} and pointed bootstrap-script at it."
}

# Ephemeral agents: one job per agent process, then disconnect. Structurally
# removes build-to-build contamination instead of cleaning up after it.
apply_ephemeral() {
  [ -w "${AGENT_CFG}" ] || { echo "FATAL: cannot write ${AGENT_CFG} (run as root)." >&2; exit 5; }
  set_cfg "disconnect-after-job" "true"
  echo "Applied disconnect-after-job=true. Your supervisor must restart the agent"
  echo "after each job, or capacity drops to zero once every worker has run once."
}

# Prove the control is real. Fails closed on the two combinations that look
# configured and enforce nothing: keyscan disabled with no pinned host keys, and
# BUILDKITE_CLEAN_CHECKOUT written into a file that never reads it.
audit_agent_env() {
  local rc=0 keyscan disconnect bootstrap hooks hook
  keyscan="$(read_cfg no-ssh-keyscan)"
  disconnect="$(read_cfg disconnect-after-job)"
  bootstrap="$(read_cfg bootstrap-script)"
  hooks="$(hooks_dir)"
  hook="${hooks}/environment"

  echo "config file          : ${AGENT_CFG}"
  echo "no-ssh-keyscan       : ${keyscan:-<unset> (agent default: false)}"
  echo "disconnect-after-job : ${disconnect:-<unset> (agent default: false)}"
  echo "bootstrap-script     : ${bootstrap:-<unset> (agent default: buildkite-agent bootstrap)}"
  echo "environment hook     : ${hook}"

  # 1. The silent-failure check. Nothing reads these out of the config file.
  if grep -qiE '^[[:space:]]*(BUILDKITE_)?CLEAN[-_]CHECKOUT[[:space:]]*=' "${AGENT_CFG}"; then
    echo "FAIL: a clean-checkout key is set in ${AGENT_CFG}. It is NOT an agent flag" >&2
    echo "      (absent from clicommand/agent_start.go) and is silently ignored." >&2
    echo "      Move it to the environment hook: $0 write-environment-hook" >&2
    rc=1
  fi

  # 2. Host-key trust.
  case "${keyscan}" in
    true)
      if [ -s "${KNOWN_HOSTS}" ]; then
        echo "PASS: automatic host-key acceptance is off and ${KNOWN_HOSTS} pins $(grep -c . "${KNOWN_HOSTS}") key(s)."
      else
        echo "FAIL: no-ssh-keyscan=true but ${KNOWN_HOSTS} is empty/missing." >&2
        echo "      StrictHostKeyChecking=yes with no pinned keys fails every checkout." >&2
        rc=1
      fi ;;
    *)
      echo "FAIL: no-ssh-keyscan is not true. The agent accepts whatever host key" >&2
      echo "      answers on first checkout (StrictHostKeyChecking=accept-new), or" >&2
      echo "      disables host-key checking outright on OpenSSH < 7.6." >&2
      rc=1 ;;
  esac

  # 3. The environment hook, and whether it actually says anything.
  if [ ! -f "${hook}" ]; then
    if [ "${disconnect}" = "true" ]; then
      echo "WARN: no environment hook, but disconnect-after-job=true — no second job"
      echo "      exists to contaminate. GIT_SSH_COMMAND is still overridable by a"
      echo "      pipeline env block; write the hook to close that."
    else
      echo "FAIL: no environment hook and the agent is persistent. Workspaces survive" >&2
      echo "      between jobs and BUILDKITE_CLEAN_CHECKOUT is set nowhere that reads it." >&2
      rc=1
    fi
  else
    # Own-block presence. Without it this pack's settings may still be in the
    # file by hand, but nothing can be rewritten safely and a re-run would
    # append a second copy of the policy rather than replace the first.
    if grep -q "HTH-BLOCK ${HTH_HOOK_BLOCK_ID}" "${hook}"; then
      echo "PASS: ${hook} carries the ${HTH_HOOK_BLOCK_ID} block."
    else
      echo "WARN: ${hook} exists but carries no '${HTH_HOOK_BLOCK_ID}' block, so"
      echo "      this pack does not own its contents. Run '$0 write-environment-hook'"
      echo "      to bring it under management (your current file is preserved and"
      echo "      backed up; the block is appended)."
    fi

    if grep -q 'BUILDKITE_CLEAN_CHECKOUT=true' "${hook}"; then
      echo "PASS: environment hook forces a clean checkout."
    elif [ "${disconnect}" = "true" ]; then
      echo "WARN: hook does not force a clean checkout; relying on disconnect-after-job."
    else
      echo "FAIL: environment hook does not export BUILDKITE_CLEAN_CHECKOUT=true, and" >&2
      echo "      this agent is persistent. One job's leftovers reach the next." >&2
      rc=1
    fi

    if grep -q 'StrictHostKeyChecking=yes' "${hook}" && grep -q '^unset GIT_SSH$' "${hook}"; then
      echo "PASS: environment hook rebuilds GIT_SSH_COMMAND and unsets GIT_SSH."
    else
      echo "FAIL: environment hook does not pin GIT_SSH_COMMAND. Because ssh resolves" >&2
      echo "      duplicate -o options first-wins and the agent APPENDS its own, a" >&2
      echo "      pipeline env block setting GIT_SSH_COMMAND or GIT_SSH defeats" >&2
      echo "      no-ssh-keyscan entirely." >&2
      rc=1
    fi

    # A hook anyone but root can rewrite is arbitrary code on every job.
    if [ -n "$(find "${hook}" -perm -g+w -o -perm -o+w 2>/dev/null)" ]; then
      echo "FAIL: ${hook} is group- or world-writable. Anyone who can edit it runs code" >&2
      echo "      on every job before the checkout, as the agent user." >&2
      rc=1
    fi
  fi

  # 4. Bootstrap wrapper integrity, when one is configured.
  if [ -n "${bootstrap}" ]; then
    local path="${bootstrap%% *}"
    if [ ! -x "${path}" ]; then
      echo "FAIL: bootstrap-script '${path}' is missing or not executable — no job can start." >&2
      rc=1
    elif [ -n "$(find "${path}" -perm -g+w -o -perm -o+w 2>/dev/null)" ]; then
      echo "FAIL: bootstrap-script '${path}' is group- or world-writable. It runs as the" >&2
      echo "      agent user on every job, before any of your other controls do." >&2
      rc=1
    else
      echo "PASS: bootstrap-script present and not group/world-writable."
    fi
  fi

  return "${rc}"
}

Source: Buildkite agent configuration

3.10 Standardize Pipeline Steps with Templates

Profile Level: L3 (Run)

Framework	Control
CIS Controls	4.1
NIST 800-53	CM-2, CM-6

Description

Define pipeline templates centrally and assign them to pipelines, which makes the assigned pipeline’s step configuration read-only to pipeline editors.

Rationale

Why This Matters:

Without templates, anyone who can edit a pipeline can delete the security scan step from it, and the pipeline keeps passing — the control disappears and the signal it produced disappears with it
A mandated step that lives in each pipeline’s own configuration is mandated only by convention; a template makes it structural
Templates give you one place to fix a step across every pipeline, rather than a migration every time a scanning tool changes its invocation
The difference matters most for the pipelines nobody looks at, which are also the pipelines where a removed control goes unnoticed longest

Attack Prevented: Silent removal of security gates by an insider or a compromised maintainer account, configuration drift between pipelines that were meant to enforce the same controls

ClickOps Implementation

Step 1: Author the Template

Go to Settings → Pipeline Templates and define the step sequence every pipeline of that class must run.

Step 2: Assign It

Assign the template to each pipeline. Once assigned, the pipeline’s step configuration becomes read-only in the UI and via the API.

Step 3: Decide Whether Templates Are Mandatory

Organization-wide “require templates” strictness is set in the console. There is no Terraform resource for it — do not assume infrastructure-as-code covers this setting; verify it in the UI.

Automation: Terraform manages template definition and assignment. The org-level strictness toggle is ClickOps only — Buildkite exposes no Terraform resource and no documented mutation for it (pipeline templates, 2026-08-18).

Note on plan gating: pipeline templates are an Enterprise feature.

Code Implementation

Code Pack: Terraform

hth-buildkite-3.10-pipeline-templates.tf View source on GitHub ↗

locals {
  # TRAP 4. Templates that hand step definition back to the repository without an
  # explicit acceptance on file.
  hth_delegating_templates = [
    for name, t in var.pipeline_templates : name
    if length(regexall("buildkite-agent\\s+pipeline\\s+upload", t.configuration)) > 0
    && !contains(var.templates_allowed_to_delegate_upload, name)
  ]

  # Every (template, mandated pattern) pair the template's configuration fails.
  # A template is only a control while it still contains the step.
  hth_templates_missing_required_steps = flatten([
    for name, t in var.pipeline_templates : [
      for pattern in var.required_step_patterns : {
        template = name
        pattern  = pattern
      } if length(regexall(pattern, t.configuration)) == 0
    ]
  ])
}

# The step sequence every pipeline of this class must run. THIS resource is the
# control; the assignment below only points a pipeline at it.
resource "buildkite_pipeline_template" "governed" {
  # TRAP 7. No profile-level gate. The declaration map is the switch; an empty
  # var.pipeline_templates is the off state.
  for_each = var.pipeline_templates

  name          = each.key
  configuration = each.value.configuration
  description   = each.value.description

  # TRAP 1. Always explicit — omitting this adopts server state.
  available = each.value.available

  lifecycle {
    # No prevent_destroy here, deliberately. A template's entire content lives in
    # this configuration, so a destroyed template is recreatable byte-for-byte
    # from `configuration`. A destroyed PIPELINE is not - see the guard on
    # buildkite_pipeline.templated below. Removing a template that a declared
    # pipeline still names is refused anyway, by that pipeline's first
    # precondition.
    precondition {
      condition     = trimspace(each.value.configuration) != ""
      error_message = format("Pipeline template '%s' has an empty configuration. Assigning it would make every adopting pipeline's step configuration read-only AND empty, which deletes their steps instead of standardizing them.", each.key)
    }

    # The configuration is a whole step document, not a fragment: it carries its
    # own `steps:` key. A fragment produces a template that fails at build time on
    # every pipeline assigned to it, all at once.
    precondition {
      condition     = length(regexall("(?m)^steps:", each.value.configuration)) > 0
      error_message = format("Pipeline template '%s' has no top-level `steps:` key. buildkite_pipeline_template.configuration is the complete YAML step configuration, not a list fragment.", each.key)
    }
  }
}

locals {
  # TRAP 2, resolved once. `.id` is the GraphQL ID this attribute wants; `.uuid`
  # is the wrong handle and fails at apply.
  hth_template_ids = {
    for name, t in buildkite_pipeline_template.governed : name => t.id
  }
}

# Pipelines whose step configuration is surrendered to a template. Note what is
# absent: `steps` is never set. It is optional+computed, so leaving it unset lets
# Terraform adopt the template-rendered value instead of putting two sources of
# truth on one attribute.
resource "buildkite_pipeline" "templated" {
  # TRAP 7. No profile-level gate. The declaration map is the switch; an empty
  # var.templated_pipelines is the off state.
  for_each = var.templated_pipelines

  name                       = each.key
  repository                 = each.value.repository
  description                = each.value.description
  default_branch             = each.value.default_branch
  branch_configuration       = each.value.branch_configuration
  cluster_id                 = each.value.cluster_id
  default_team_id            = each.value.default_team_id
  default_timeout_in_minutes = each.value.default_timeout_in_minutes
  maximum_timeout_in_minutes = each.value.maximum_timeout_in_minutes

  # TRAP 2/TRAP 3. lookup() rather than a bare index so the precondition below
  # names the offending pipeline instead of Terraform raising a raw index error.
  # The null branch is unreachable: the precondition refuses an undeclared
  # template, and asserting null here is exactly the detachment TRAP 3 describes.
  pipeline_template_id = lookup(local.hth_template_ids, each.value.template, null)

  # Fork-build restriction carried over from controls 2.2/2.4 so a pipeline
  # created by this pack is never less hardened than one created by that one.
  provider_settings = {
    build_pull_request_forks              = false
    publish_commit_status                 = true
    publish_commit_status_per_step        = true
    skip_builds_for_existing_commits      = true
    cancel_deleted_branch_builds          = true
    prefix_pull_request_fork_branch_names = true
  }

  lifecycle {
    # TRAP 7, enforced rather than narrated. A Buildkite pipeline carries its
    # build history and its webhook URL; both are destroyed with it and neither
    # comes back. Dropping a key from var.templated_pipelines must therefore be a
    # deliberate act, not the side effect of a forgotten -var or an edited tfvars
    # file - so Terraform refuses the destroy at PLAN time. This is the same
    # guard pack 3.11 puts on buildkite_registry. To retire a pipeline on
    # purpose: delete this line, apply, restore it.
    #
    # DELIBERATELY A LITERAL. OpenTofu 1.12 does accept an expression here
    # (measured, not assumed), which is precisely why this must not become
    # `var.profile_level >= 3`: the one forgotten -var would then empty the
    # for_each AND disarm the guard in the same plan, rebuilding the original
    # bug with an extra step.
    prevent_destroy = true

    precondition {
      condition     = contains(keys(var.pipeline_templates), each.value.template)
      error_message = format("Templated pipeline '%s' names template '%s', which is not declared in var.pipeline_templates. This pack refuses to point a pipeline at a console-authored template: an undeclared template's contents can be edited in the UI without drift showing, so the read-only guarantee would be enforcing an ungoverned step list. Bring the template under var.pipeline_templates, or audit it through var.audited_templates and accept that it is not enforced from code.", each.key, each.value.template)
    }

    precondition {
      condition     = each.value.default_team_id != null
      error_message = format("Templated pipeline '%s' sets no default_team_id. The provider documents this attribute as required by the Buildkite API when creating a new pipeline, so the apply would fail at the API with a vaguer message. Supply a team GraphQL ID — pack 2.1 emits them as the `team_ids` output.", each.key)
    }
  }
}

# A template nobody asserts anything about is a template somebody can hollow out
# while every pipeline on it still reports as governed.
check "managed_templates_contain_their_required_steps" {
  assert {
    condition = length(local.hth_templates_missing_required_steps) == 0
    error_message = format(
      "%d template/step-pattern pair(s) are unsatisfied: %s. Pipelines on these templates are held read-only around a step list that no longer contains the mandated step — the failure this control exists to prevent, reached without a single pipeline edit.",
      length(local.hth_templates_missing_required_steps),
      jsonencode(local.hth_templates_missing_required_steps),
    )
  }
}

check "required_step_patterns_are_declared" {
  assert {
    condition = length(var.pipeline_templates) == 0 || length(var.required_step_patterns) > 0
    error_message = format(
      "%d pipeline template(s) are managed but var.required_step_patterns is empty, so nothing asserts what those templates contain. Making a step list read-only guarantees the steps are FIXED, not that they are the RIGHT ones.",
      length(var.pipeline_templates),
    )
  }
}

# TRAP 4, enforced.
check "templates_do_not_silently_delegate_steps_to_the_repository" {
  assert {
    condition = length(local.hth_delegating_templates) == 0
    error_message = format(
      "Template(s) %s run `buildkite-agent pipeline upload`, so their real steps come from the repository's pipeline file rather than from the template. The read-only guarantee stops at the upload boundary and enforcement moves to branch protection and code review on that repository. Name them in var.templates_allowed_to_delegate_upload to record that decision, or inline the mandated step in the template.",
      jsonencode(local.hth_delegating_templates),
    )
  }
}

# WARNING ONLY, and labelled as such. This is a `check` block: Terraform check
# blocks annotate a run, they cannot fail a plan or halt an apply. It is here to
# keep the RECORDED profile level honest, not to guard anything - per TRAP 7
# nothing in this file is gated on profile_level any more, so there is nothing
# left for a guard to stop.
check "template_declarations_record_an_l3_control" {
  assert {
    condition = var.profile_level >= 3 || (length(var.pipeline_templates) == 0 && length(var.templated_pipelines) == 0)
    error_message = format(
      "profile_level is %d, but %d template(s) and %d templated pipeline(s) are declared. Control 3.10 is an L3 control. Those resources ARE created at the level you set - profile level selects WHAT you declare, never whether declared resources survive - so the only thing wrong here is the number: this configuration enforces an L3 control while reporting level %d. Raise profile_level to 3 so the recorded level matches what is applied, or clear the declarations. This message is a warning and does not stop the apply.",
      var.profile_level,
      length(var.pipeline_templates),
      length(var.templated_pipelines),
      var.profile_level,
    )
  }
}

output "pipeline_template_governance" {
  description = "Templates this configuration owns and the pipelines bound to them. `available` is read back off the resource rather than the variable, so a console flip surfaces here."
  value = {
    templates = {
      for name, t in buildkite_pipeline_template.governed : name => {
        uuid                 = t.uuid
        graphql_id           = t.id
        available_to_members = t.available
        configuration_sha256 = sha256(t.configuration)
        delegates_upload     = length(regexall("buildkite-agent\\s+pipeline\\s+upload", t.configuration)) > 0
      }
    }
    pipelines = {
      for name, p in buildkite_pipeline.templated : name => {
        slug                 = p.slug
        template             = var.templated_pipelines[name].template
        pipeline_template_id = p.pipeline_template_id
      }
    }
  }
}

# The region above governs what Terraform owns. The residual risk is the template
# authored in the console and the pipeline nobody put under one. Losing a
# mandated step from a template is the same failure as losing it from a pipeline,
# applied to every pipeline on that template at once — and per TRAP 6 Terraform
# can enumerate neither templates nor a pipeline's assignment. What it CAN do is
# re-read a named template and compare it to what was approved.
data "buildkite_pipeline_template" "audited" {
  for_each = var.audited_templates

  name = each.value.name
}

locals {
  # Content drift on a template reviewed once and then left in the console.
  # sha256 over the live configuration is the whole comparison.
  hth_audited_template_drift = [
    for key, t in data.buildkite_pipeline_template.audited : {
      template = t.name
      uuid     = t.uuid
      approved = var.audited_templates[key].approved_configuration_sha256
      observed = sha256(t.configuration)
    }
    if var.audited_templates[key].approved_configuration_sha256 != null
    && sha256(t.configuration) != var.audited_templates[key].approved_configuration_sha256
  ]

  # The same mandated-step assertion applied to templates this configuration does
  # not own. An audited template that never carried the step is as much a finding
  # as one that lost it.
  hth_audited_missing_required_steps = flatten([
    for key, t in data.buildkite_pipeline_template.audited : [
      for pattern in var.required_step_patterns : {
        template = t.name
        pattern  = pattern
      } if length(regexall(pattern, t.configuration)) == 0
    ]
  ])

  # TRAP 1 on the audit side.
  hth_audited_self_assignable = [
    for key, t in data.buildkite_pipeline_template.audited : {
      template = t.name
      uuid     = t.uuid
    } if t.available && !var.audited_templates[key].allow_non_admin_assignment
  ]

  # TRAP 3. Pipelines managed by pack 2.2, which sets no pipeline_template_id and
  # therefore keeps them un-templated on every apply.
  hth_untemplated_pipelines = [
    for name in keys(var.pipelines) : name
    if !contains(var.untemplated_pipelines_allowed, name)
    && !contains(keys(var.templated_pipelines), name)
  ]

  # The same pipeline name claimed by two resource addresses.
  hth_pipeline_address_conflicts = [
    for name in keys(var.templated_pipelines) : name
    if contains(keys(var.pipelines), name)
  ]
}

check "audited_template_configuration_unchanged" {
  assert {
    condition = length(local.hth_audited_template_drift) == 0
    error_message = format(
      "%d audited template(s) no longer hash to their approved configuration: %s. Every pipeline assigned to these templates had its read-only step list rewritten in one edit. Review the change, then update approved_configuration_sha256 in var.audited_templates to re-baseline.",
      length(local.hth_audited_template_drift),
      jsonencode(local.hth_audited_template_drift),
    )
  }
}

check "audited_templates_contain_their_required_steps" {
  assert {
    condition = length(local.hth_audited_missing_required_steps) == 0
    error_message = format(
      "%d audited template/step-pattern pair(s) are unsatisfied: %s. These templates are not owned by this configuration, so nothing stops the next console edit either — bring them under var.pipeline_templates to make their contents enforceable rather than merely observable.",
      length(local.hth_audited_missing_required_steps),
      jsonencode(local.hth_audited_missing_required_steps),
    )
  }
}

check "audited_templates_are_not_self_assignable" {
  assert {
    condition = length(local.hth_audited_self_assignable) == 0
    error_message = format(
      "%d audited template(s) are marked available to non-admin users: %s. Any member who can edit a pipeline can move it onto one of these, which is an escape route out of a stricter template. Set allow_non_admin_assignment on the entry to accept it, or clear the flag in the Buildkite console.",
      length(local.hth_audited_self_assignable),
      jsonencode(local.hth_audited_self_assignable),
    )
  }
}

check "pipeline_not_managed_by_two_resources" {
  assert {
    condition = length(local.hth_pipeline_address_conflicts) == 0
    error_message = format(
      "Pipeline(s) %s appear in BOTH var.pipelines (pack 2.2, buildkite_pipeline.pipelines) and var.templated_pipelines (buildkite_pipeline.templated). Two resource addresses fighting over one pipeline is bad enough; because pipeline_template_id is optional-NOT-computed, the 2.2 address wins by detaching the template. Declare each pipeline in exactly one of the two maps.",
      jsonencode(local.hth_pipeline_address_conflicts),
    )
  }
}

check "pipelines_outside_template_governance" {
  assert {
    condition = length(local.hth_untemplated_pipelines) == 0
    error_message = format(
      "%d pipeline(s) are managed with no template and will have any console-assigned template stripped on apply: %s. Their step configuration stays editable, so a mandated scan step can be removed from them without leaving a Terraform trace. Move them to var.templated_pipelines, or list them in var.untemplated_pipelines_allowed to record the exception.",
      length(local.hth_untemplated_pipelines),
      jsonencode(local.hth_untemplated_pipelines),
    )
  }
}

output "pipeline_template_audit" {
  description = "Live state of every audited template plus the pipelines this configuration leaves outside template governance. A FULL template inventory is not obtainable from Terraform — there is no plural data source — so use the GraphQL/REST API for that."
  value = {
    audited_templates = {
      for key, t in data.buildkite_pipeline_template.audited : key => {
        name                 = t.name
        uuid                 = t.uuid
        graphql_id           = t.id
        available_to_members = t.available
        configuration_sha256 = sha256(t.configuration)
      }
    }
    configuration_drift    = local.hth_audited_template_drift
    missing_required_steps = local.hth_audited_missing_required_steps
    self_assignable        = local.hth_audited_self_assignable
    untemplated_pipelines  = local.hth_untemplated_pipelines
    address_conflicts      = local.hth_pipeline_address_conflicts
  }
}

Source: Buildkite pipeline templates

3.11 Govern Inbound OIDC Trust

Profile Level: L3 (Run)

Framework	Control
CIS Controls	5.5
NIST 800-53	AC-3, IA-9

Description

Constrain which pipelines may authenticate into Buildkite-hosted services — package registries and test suites — using OIDC policies, the inbound counterpart to the outbound cloud federation in 3.6.

Rationale

Why This Matters:

3.6 governs Buildkite authenticating outward to your cloud; this governs pipelines authenticating inward to your registries, and they are different trust directions with different blast radii
Publish authority to a registry is a supply-chain control: whoever can publish decides what your consumers install, which makes “which pipeline may publish here” a first-class security question
The alternative is a long-lived static registry token, and a test suite’s api_token is exactly the standing credential an OIDC policy exists to displace
Without a policy, any pipeline holding the token can publish — the registry cannot distinguish your release pipeline from a pull-request build that happened to read the same secret

Attack Prevented: Malicious package publication from a non-release pipeline, supply-chain compromise via a leaked static registry token, branch-based trust bypass during publication

ClickOps Implementation

Step 1: Identify Publication Points

List every registry and test suite that a pipeline writes to, and the single pipeline that legitimately writes to each.

Step 2: Write the Policy

Define an OIDC policy scoping the issuer, scopes, and claims — organization_slug, pipeline_slug, build_branch, repository, actor. Constrain build_branch where publication should only happen from your release branch.

Step 3: Remove the Static Credential

Once the policy works, stop distributing the static token. Leaving it live means you added a path rather than closing one.

Code Implementation

Code Pack: Terraform

hth-buildkite-3.11-inbound-oidc.tf View source on GitHub ↗

locals {
  # TRAP 8. The complete supported issuer set. CircleCI's is per-organization
  # (https://oidc.circleci.com/org/$ORG), so it is matched by prefix.
  hth_oidc_exact_issuers   = ["https://agent.buildkite.com", "https://token.actions.githubusercontent.com"]
  hth_oidc_circleci_prefix = "https://oidc.circleci.com/org/"

  # TRAP 4. Per-product scope vocabularies. Disjoint on purpose.
  hth_registry_scopes = ["read_packages", "write_packages", "delete_packages"]
  hth_suite_scopes    = ["read_suites", "write_uploads", "read_test_plan", "write_test_plan"]

  # Scopes that mutate. A statement holding one of these and not naming the
  # caller is a standing publish grant for the whole organization.
  hth_registry_write_scopes = ["write_packages", "delete_packages"]
  hth_suite_write_scopes    = ["write_uploads", "write_test_plan"]

  # TRAP 6. Claims that actually identify WHO is calling. organization_slug is
  # deliberately absent: every pipeline in the org asserts it.
  hth_identifying_claims = ["pipeline_slug", "repository"]

  # TRAP 13 / TRAP 7. One encoder for both products. Emits the documented
  # statement list - { iss, scopes, claims } - as simple YAML, translating the
  # variable's any_of / none_of back to the wire names `in` / `not_in` and
  # dropping every matcher the caller left unset.
  hth_registry_policy_yaml = {
    for rk, r in var.registry_oidc_policies : rk => length(r.statements) == 0 ? "" : yamlencode([
      for st in r.statements : {
        "iss"    = st.issuer
        "scopes" = st.scopes
        "claims" = {
          for cn, rule in st.claims : cn => {
            for matcher, value in {
              "equals"     = rule.equals
              "not_equals" = rule.not_equals
              "in"         = rule.any_of
              "not_in"     = rule.none_of
              "matches"    = rule.matches
            } : matcher => value if value != null
          }
        }
      }
    ])
  }

  hth_suite_policy_yaml = {
    for sk, s in var.test_suite_oidc_policies : sk => length(s.statements) == 0 ? "" : yamlencode([
      for st in s.statements : {
        "iss"    = st.issuer
        "scopes" = st.scopes
        "claims" = {
          for cn, rule in st.claims : cn => {
            for matcher, value in {
              "equals"     = rule.equals
              "not_equals" = rule.not_equals
              "in"         = rule.any_of
              "not_in"     = rule.none_of
              "matches"    = rule.matches
            } : matcher => value if value != null
          }
        }
      }
    ])
  }
}

# Publish authority expressed as policy rather than as possession of a token.
resource "buildkite_registry" "inbound_oidc" {
  for_each = var.registry_oidc_policies

  name      = each.key
  ecosystem = each.value.ecosystem

  # TRAP 2. UUIDs here - buildkite_team.<name>.uuid, never .id.
  team_ids = each.value.team_uuids

  description = each.value.description
  emoji       = each.value.emoji
  color       = each.value.color

  # TRAP 3. Always explicit; "" is the documented way to assert no trust.
  oidc_policy = local.hth_registry_policy_yaml[each.key]

  lifecycle {
    # TRAP 9. ecosystem and team_ids are force-new; a tfvars edit would destroy
    # the registry and every package published to it.
    prevent_destroy = true

    # TRAP 8. A typo'd issuer plans green and rejects every token at runtime.
    precondition {
      condition = alltrue([
        for st in each.value.statements :
        contains(local.hth_oidc_exact_issuers, st.issuer) || startswith(st.issuer, local.hth_oidc_circleci_prefix)
      ])
      error_message = format("Registry '%s': issuer must be https://agent.buildkite.com, https://token.actions.githubusercontent.com, or https://oidc.circleci.com/org/<ORG> - the only issuers Buildkite supports. Any other value is accepted by Terraform and rejects every token at build time.", each.key)
    }

    # TRAP 4, restated at the resource so the message names the registry.
    precondition {
      condition = alltrue([
        for st in each.value.statements :
        length(setsubtract(toset(st.scopes), toset(local.hth_registry_scopes))) == 0
      ])
      error_message = format("Registry '%s': registry OIDC policies support only read_packages, write_packages and delete_packages. Test-suite scopes are valid YAML here and authorise nothing - this is what a policy copied from a test suite looks like.", each.key)
    }

    # TRAP 6. A write grant that does not name the caller is org-wide publish.
    precondition {
      condition = alltrue([
        for st in each.value.statements :
        length(setintersection(toset(st.scopes), toset(local.hth_registry_write_scopes))) == 0
        || length(setintersection(toset(keys(st.claims)), toset(local.hth_identifying_claims))) > 0
      ])
      error_message = format("Registry '%s': a statement grants write_packages or delete_packages without constraining pipeline_slug or repository. organization_slug does not narrow anything - every pipeline in the organization asserts it - so this grants publish authority over your artifacts to any build in the org.", each.key)
    }

    # TRAP 5. Order is evaluation order: a statement matching everything from its
    # issuer makes every later statement unreachable, including a write grant.
    precondition {
      condition = alltrue([
        for idx, st in each.value.statements :
        idx == length(each.value.statements) - 1 ||
        length(setintersection(toset(keys(st.claims)), toset(local.hth_identifying_claims))) > 0
      ])
      error_message = format("Registry '%s': a statement with no pipeline_slug or repository claim appears before another statement. Buildkite stops at the first match and grants only that statement's scopes, so everything after it is dead code. Move the broad statement last.", each.key)
    }

    # TRAP 7. equals and not_equals on the same value can never both hold.
    precondition {
      condition = alltrue(flatten([
        for st in each.value.statements : [
          for cn, rule in st.claims :
          rule.equals == null || rule.not_equals == null || rule.equals != rule.not_equals
        ]
      ]))
      error_message = format("Registry '%s': a claim sets equals and not_equals to the same value. All matchers in a rule are ANDed, so the rule can never match and the statement is permanently dead.", each.key)
    }
  }
}

# TRAP 11. Audience and agent command assembled from the computed slug.
output "registry_oidc_audiences" {
  description = "Per-registry OIDC audience and the buildkite-agent command that mints a token for it. Registry tokens are capped at 300s - exp minus iat cannot exceed 5 minutes."
  value = {
    for k, r in buildkite_registry.inbound_oidc : k => {
      registry_slug  = r.slug
      audience       = "https://packages.buildkite.com/${var.buildkite_organization}/${r.slug}"
      max_lifetime_s = 300
      agent_command  = "buildkite-agent oidc request-token --audience \"https://packages.buildkite.com/${var.buildkite_organization}/${r.slug}\" --lifetime 300"
      docker_login   = "docker login packages.buildkite.com/${var.buildkite_organization}/${r.slug} --username buildkite --password-stdin"
      statements     = length(var.registry_oidc_policies[k].statements)
      trust_declared = length(var.registry_oidc_policies[k].statements) > 0
    }
  }
}

# Same control, different scope vocabulary (TRAP 4), different id namespace
# (TRAP 2), and a standing credential OIDC displaces but does not remove (TRAP 10).
resource "buildkite_test_suite" "inbound_oidc" {
  for_each = var.test_suite_oidc_policies

  name           = each.key
  default_branch = each.value.default_branch

  # TRAP 2. GraphQL ID here - the opposite of buildkite_registry.team_ids.
  team_owner_id = each.value.team_owner_id

  application_name = each.value.application_name
  emoji            = each.value.emoji
  color            = each.value.color

  # TRAP 3. Optional + COMPUTED on this resource: omit it and a console-authored
  # policy survives silently with no drift reported. Always assert a value.
  oidc_policy = local.hth_suite_policy_yaml[each.key]

  lifecycle {
    precondition {
      condition = alltrue([
        for st in each.value.statements :
        contains(local.hth_oidc_exact_issuers, st.issuer) || startswith(st.issuer, local.hth_oidc_circleci_prefix)
      ])
      error_message = format("Test suite '%s': issuer must be one of Buildkite's three supported issuers. Any other value plans cleanly and rejects every token at build time.", each.key)
    }

    # TRAP 4, in the other direction: registry scopes are dead here.
    precondition {
      condition = alltrue([
        for st in each.value.statements :
        length(setsubtract(toset(st.scopes), toset(local.hth_suite_scopes))) == 0
      ])
      error_message = format("Test suite '%s': suite OIDC policies support only read_suites, write_uploads, read_test_plan and write_test_plan. Registry scopes are a disjoint vocabulary and authorise nothing here.", each.key)
    }

    precondition {
      condition = alltrue([
        for st in each.value.statements :
        length(setintersection(toset(st.scopes), toset(local.hth_suite_write_scopes))) == 0
        || length(setintersection(toset(keys(st.claims)), toset(local.hth_identifying_claims))) > 0
      ])
      error_message = format("Test suite '%s': a statement grants write_uploads or write_test_plan without constraining pipeline_slug or repository, so any pipeline in the organization can write results into this suite and the suite stops being evidence of anything.", each.key)
    }

    precondition {
      condition = alltrue([
        for idx, st in each.value.statements :
        idx == length(each.value.statements) - 1 ||
        length(setintersection(toset(keys(st.claims)), toset(local.hth_identifying_claims))) > 0
      ])
      error_message = format("Test suite '%s': a statement matching every token from its issuer precedes another statement, making it unreachable - first match wins.", each.key)
    }

    precondition {
      condition = alltrue(flatten([
        for st in each.value.statements : [
          for cn, rule in st.claims :
          rule.equals == null || rule.not_equals == null || rule.equals != rule.not_equals
        ]
      ]))
      error_message = format("Test suite '%s': a claim sets equals and not_equals to the same value, so the rule can never match.", each.key)
    }
  }
}

# TRAP 10, surfaced rather than assumed. Reports every plan; does not block.
check "suite_static_tokens_still_outstanding" {
  assert {
    condition = length([
      for k, v in var.test_suite_oidc_policies : k
      if length(v.statements) > 0 && !contains(var.suite_api_tokens_rotated, k)
    ]) == 0
    error_message = format(
      "Test suite(s) %s now accept OIDC but their static api_token has not been recorded as rotated. OIDC ADDS an authentication path; it does not close the old one, and the provider exposes no attribute to revoke or rotate api_token. Rotate it in the console, remove it from the pipeline that read it, then list the suite in var.suite_api_tokens_rotated.",
      jsonencode([for k, v in var.test_suite_oidc_policies : k if length(v.statements) > 0 && !contains(var.suite_api_tokens_rotated, k)]),
    )
  }
}

output "test_suite_oidc_audiences" {
  description = "Per-suite OIDC audience and token command. The audience shape differs from a registry's - buildkite.com/organizations/... not packages.buildkite.com/... - and the collector reads the token from BUILDKITE_ANALYTICS_TOKEN."
  value = {
    for k, s in buildkite_test_suite.inbound_oidc : k => {
      suite_slug       = s.slug
      audience         = "https://buildkite.com/organizations/${var.buildkite_organization}/analytics/suites/${s.slug}"
      agent_command    = "BUILDKITE_ANALYTICS_TOKEN=$(buildkite-agent oidc request-token --audience \"https://buildkite.com/organizations/${var.buildkite_organization}/analytics/suites/${s.slug}\" --lifetime 300)"
      statements       = length(var.test_suite_oidc_policies[k].statements)
      static_api_token = contains(var.suite_api_tokens_rotated, k) ? "recorded as rotated" : "STILL OUTSTANDING - api_token remains a valid credential for this suite"
    }
  }
}

# TRAP 12. The resources above govern only what Terraform declares. A registry or
# suite created in the console is where an ungoverned publish path lives, and the
# provider offers no plural data source to find one - so the audit runs over an
# explicitly maintained slug list, and the output says so rather than implying
# completeness.
data "buildkite_registry" "audited" {
  for_each = toset(var.audited_registry_slugs)

  slug = each.value
}

data "buildkite_test_suite" "audited" {
  for_each = toset(var.audited_test_suite_slugs)

  slug = each.value
}

locals {
  # An empty live policy means the resource accepts no OIDC token at all - so
  # anything publishing to it is doing so with a static credential.
  hth_registries_without_policy = [
    for slug, r in data.buildkite_registry.audited : slug
    if trimspace(coalesce(r.oidc_policy, "")) == ""
  ]

  hth_suites_without_policy = [
    for slug, s in data.buildkite_test_suite.audited : slug
    if trimspace(coalesce(s.oidc_policy, "")) == ""
  ]

  # Live posture next to whether this configuration authored it. Policy SIZE is
  # reported rather than policy text: the console round-trips YAML, so a byte
  # delta is a review signal without printing trust rules into plan output.
  hth_inbound_oidc_live = merge(
    {
      for slug, r in data.buildkite_registry.audited : "registry/${slug}" => {
        terraform_managed = contains(keys(var.registry_oidc_policies), r.name)
        policy_present    = trimspace(coalesce(r.oidc_policy, "")) != ""
        policy_bytes      = length(coalesce(r.oidc_policy, ""))
      }
    },
    {
      for slug, s in data.buildkite_test_suite.audited : "test_suite/${slug}" => {
        terraform_managed = contains(keys(var.test_suite_oidc_policies), s.name)
        policy_present    = trimspace(coalesce(s.oidc_policy, "")) != ""
        policy_bytes      = length(coalesce(s.oidc_policy, ""))
      }
    },
  )
}

# Reports every plan and apply; does not block. An ungoverned publish path is a
# review finding about who can put artifacts in front of your consumers, not a
# reason to fail an unrelated deploy at 2am.
check "no_registry_publishes_without_an_oidc_policy" {
  assert {
    condition = length(local.hth_registries_without_policy) == 0
    error_message = format(
      "Registry/registries %s have no OIDC policy, so every publish to them is authenticated by a long-lived static credential and the registry cannot tell a release pipeline from a pull-request build. Declare them in var.registry_oidc_policies, or record why a static credential is acceptable.",
      jsonencode(local.hth_registries_without_policy),
    )
  }
}

check "no_test_suite_uploads_without_an_oidc_policy" {
  assert {
    condition = length(local.hth_suites_without_policy) == 0
    error_message = format(
      "Test suite(s) %s have no OIDC policy and accept uploads on the strength of the suite api_token alone, so any pipeline holding that token can write results into them.",
      jsonencode(local.hth_suites_without_policy),
    )
  }
}

output "inbound_oidc_audit" {
  description = "Live inbound-OIDC posture for every audited slug, plus the honest caveat about what the audit cannot see."
  value = {
    registries_without_policy = local.hth_registries_without_policy
    suites_without_policy     = local.hth_suites_without_policy
    live_state                = local.hth_inbound_oidc_live
    coverage_caveat           = "data.buildkite_registry and data.buildkite_test_suite are keyed on a required slug and the provider ships no plural data source for either. Anything absent from var.audited_registry_slugs / var.audited_test_suite_slugs is invisible here; reconciling those lists against the console is a manual control."
  }
}

Source: Buildkite package registries OIDC

4. Monitoring & Compliance

4.1 Configure Audit Logging

Profile Level: L1 (Crawl)

Framework	Control
CIS Controls	8.2
NIST 800-53	AU-2

Description

Enable and monitor audit logs.

Rationale

Why This Matters:

Audit logs of authentication, pipeline changes, and permission edits are the primary evidence for detecting and investigating compromise
Monitoring agent token usage surfaces stolen tokens being used from unexpected sources
Appropriate retention ensures records survive long enough to support incident response and compliance audits
Without logging, attacker actions such as permission changes and malicious pipeline edits go unnoticed

Attack Prevented: Undetected intrusion, repudiation, delayed incident response, audit gaps

Prerequisites

Buildkite Enterprise. The audit log is an Enterprise-plan feature; organizations below Enterprise have no audit log to review, and compliance commitments that assume one need to account for that gap.
Terraform (L3 IP restriction only): a plan including the API IP allowlist feature, and the same buildkite_organization caveat described in 1.2 — the two controls share one singleton resource, so adopt one Terraform path or merge them, never both.

Lockout warning — API IP allowlist. allowed_api_ip_addresses is a hard allowlist on REST and GraphQL access. A CIDR list that omits the network you automate from severs your own API access the moment it applies, including the access required to reverse it. The undo path (organizationApiIpAllowlistUpdate) is itself an API call, so a wrong list is self-sealing. Confirm your egress address is covered before applying, and keep a route in from an allowlisted network. The agent-token IP allowlist in 3.1 is lockout-capable in its own way — a wrong CIDR strands every agent presenting that token — but it differs in one respect that matters here: it does not seal its own undo path, because it restricts agent registration only and leaves your API and console access intact.

ClickOps Implementation

Step 1: Access the Audit Log

Navigate to: Organization Settings → Audit → Audit Log
There is no retention setting to configure. Buildkite stores audit events indefinitely; what varies is where you can reach them:
- The web UI browses the most recent 12 months
- Older events are retrieved through the GraphQL API
Understand the search limits before relying on the UI for an investigation: search covers the last 90 days and accepts at most 3 terms totalling 250 characters. Anything wider than that is a query against the API, not a search in the console.

Step 2: Monitor Events

User authentication
Pipeline changes
Permission modifications
Agent token usage
API access token activity (see 2.5)

Step 3: Export Rather Than Browse

Amazon EventBridge: stream audit events continuously into your own pipeline, which is the practical way to alert on them rather than discover them later.
REST and GraphQL APIs: retrieve events programmatically — the GraphQL API is also the only route to events older than the 12-month UI window.

Source: Buildkite audit log

Code Implementation

Code Pack: Terraform

hth-buildkite-4.01-configure-audit-logging.tf View source on GitHub ↗

# Restrict API access to known IP addresses (L3)
# This limits which networks can query audit logs and other API endpoints
resource "buildkite_organization" "api_restrictions" {
  count = var.profile_level >= 3 && length(var.allowed_api_ip_addresses) > 0 ? 1 : 0

  allowed_api_ip_addresses = var.allowed_api_ip_addresses
}

# Audit log monitoring is performed via the Buildkite API.
# Key events to monitor:
#   - User authentication (login/logout)
#   - Pipeline changes (create/update/delete)
#   - Permission modifications (team/member changes)
#   - Agent token usage (create/revoke)
#   - Organization setting changes
#
# Query audit events via GraphQL:
#
# query {
#   organization(slug: "your-org") {
#     auditEvents(first: 50) {
#       edges {
#         node {
#           type
#           occurredAt
#           actor {
#             name
#           }
#           subject {
#             name
#             type
#           }
#         }
#       }
#     }
#   }
# }

4.2 Contain a Compromised Build Fleet

Profile Level: L2 (Walk)

Framework	Control
CIS Controls	17.4
NIST 800-53	IR-4

Description

Establish and rehearse the mechanisms for stopping work on a compromised cluster — pausing queue dispatch, pausing or stopping agents, and revoking agent tokens — before an incident requires them.

Rationale

Why This Matters:

Revoking an agent token does not disconnect agents that are already connected. A responder who revokes tokens and believes the fleet is contained has stopped new agents from registering while the compromised ones keep running jobs
Containment has an order — pause dispatch, then stop agents, then revoke tokens — and working it out during an incident costs the time the incident is consuming
Pausing queue dispatch stops new work without deleting the queue and losing its configuration, which is the difference between containment and an outage you then have to rebuild from
There is no buildkite_agent Terraform resource, so the agent half of containment is CLI and API only; a team whose entire operational muscle is terraform apply has no path to the controls that matter here

Attack Prevented: Continued execution of malicious jobs during an incident, credential exfiltration from agents that survived an incomplete containment, destruction of evidence by builds that keep running

ClickOps Implementation

Step 1: Pause Dispatch First

Go to Agents → Clusters → Queues and pause dispatch on the affected queue. New jobs stop being handed out; running jobs are unaffected, which is why this is first rather than sufficient.

Step 2: Stop the Agents

Stop the affected agents. Use a forced stop only when you accept losing the running job’s output — sometimes that output is the evidence.

Step 3: Revoke Tokens

Revoke the cluster’s agent tokens so nothing re-registers. Do this after stopping agents, not instead of it.

Step 4: Rehearse It

Run the sequence against a non-production cluster at least once. A containment procedure nobody has executed is a hypothesis.

Step 5: Restore Deliberately

Resume dispatch only after issuing fresh tokens and confirming the agent hosts were rebuilt rather than restarted.

Code Implementation

Code Pack: Terraform

hth-buildkite-4.02-containment.tf View source on GitHub ↗

locals {
  # Clusters holding a queue under containment management, resolved by name so the
  # GraphQL id (TRAP 2) is never typed by hand.
  hth_containment_clusters = toset([for q in var.containment_queues : q.cluster])

  # Effective dispatch_paused per queue. `null` is the important value: it leaves
  # the attribute unset, so Terraform adopts server state and can never revert an
  # out-of-band incident pause (see the header).
  hth_containment_dispatch = {
    for k, q in var.containment_queues : k => (
      var.break_glass_pause_all ? true :
      q.break_glass_pause == true ? true :
      (q.break_glass_pause == false && contains(var.acknowledged_resume_queues, k)) ? false :
      null
    )
  }

  # Entries asking to resume without the acknowledgement. Coerced to null above,
  # reported below — a silently-ignored resume is worse than a refused one.
  hth_containment_unacknowledged_resumes = [
    for k, q in var.containment_queues : k
    if q.break_glass_pause == false && !contains(var.acknowledged_resume_queues, k)
  ]
}

data "buildkite_cluster" "containment" {
  for_each = local.hth_containment_clusters

  name = each.value
}

# Queues whose dispatch state is part of the incident-response surface. Pausing
# dispatch stops new jobs being handed out WITHOUT deleting the queue, so restore
# is a resume rather than a rebuild of queue configuration.
resource "buildkite_cluster_queue" "containment" {
  for_each = var.containment_queues

  cluster_id  = data.buildkite_cluster.containment[each.value.cluster].id
  key         = each.value.key
  description = each.value.description

  # null in the ordinary case. Never a bare `false`.
  dispatch_paused = local.hth_containment_dispatch[each.key]

  lifecycle {
    # A fleet-wide break-glass pause and a per-queue resume are contradictory
    # instructions. Mid-incident that is a typo, not an intent — fail the plan
    # rather than let the reader guess which one won.
    precondition {
      condition = !(var.break_glass_pause_all && each.value.break_glass_pause == false)
      error_message = format(
        "Queue '%s' sets break_glass_pause = false while var.break_glass_pause_all is true. Resolve the contradiction: drop the entry's false, or clear the fleet-wide break glass.",
        each.key,
      )
    }
  }
}

output "containment_queue_targets" {
  description = "Queues under containment management and the dispatch_paused value Terraform will send. `null` means the attribute is left unset, so an out-of-band pause survives the next apply."
  value = {
    for k, q in var.containment_queues : k => {
      cluster         = q.cluster
      key             = q.key
      cluster_id      = data.buildkite_cluster.containment[q.cluster].id
      cluster_uuid    = data.buildkite_cluster.containment[q.cluster].uuid
      dispatch_paused = local.hth_containment_dispatch[k]
      managed         = local.hth_containment_dispatch[k] != null
    }
  }
}

# The write half above governs intent. This half reads what is actually true.
# There is no cluster_queue data source (TRAP 4), so live state is only legible
# through each managed resource's own computed dispatch_paused after refresh —
# which is exactly enough to answer the two questions that matter after an
# incident: is anything still paused, and did anyone ask for a resume that this
# configuration silently declined to make.
locals {
  hth_containment_paused_now = [
    for k, q in buildkite_cluster_queue.containment : k if q.dispatch_paused
  ]
}

# Continuous validation: reports on every plan and apply, does not block. A queue
# left paused after the incident closed is a silent outage — the builds simply
# never start — so surface it every run instead of waiting for someone to ask why
# the deploy pipeline has been quiet.
check "no_queue_left_paused" {
  assert {
    condition = length(local.hth_containment_paused_now) == 0
    error_message = format(
      "%d cluster queue(s) currently have dispatch PAUSED and are handing out no jobs: %s. If the incident is closed, resume them deliberately (set break_glass_pause = false AND list the key in var.acknowledged_resume_queues, or resume via the 4.2 CLI pack). If it is open, this is the expected state.",
      length(local.hth_containment_paused_now),
      jsonencode(local.hth_containment_paused_now),
    )
  }
}

# A resume that did not happen must never look like a resume that did.
check "resume_requests_are_acknowledged" {
  assert {
    condition = length(local.hth_containment_unacknowledged_resumes) == 0
    error_message = format(
      "%d queue(s) request break_glass_pause = false without acknowledgement and were COERCED TO UNSET, so dispatch was NOT resumed: %s. Add each key to var.acknowledged_resume_queues to make the resume take effect. The coercion is deliberate: an unreviewed `false` in a queue pack is how a routine apply un-contains a fleet.",
      length(local.hth_containment_unacknowledged_resumes),
      jsonencode(local.hth_containment_unacknowledged_resumes),
    )
  }
}

output "containment_dispatch_state" {
  description = "Live dispatch state for every managed containment queue, plus resume requests this configuration declined to act on. Terraform cannot see queues it does not manage — use the 4.2 CLI pack's `status` verb for the unmanaged sweep."
  value = {
    live = {
      for k, q in buildkite_cluster_queue.containment : k => {
        queue_id        = q.id
        queue_uuid      = q.uuid
        cluster_uuid    = q.cluster_uuid
        dispatch_paused = q.dispatch_paused
      }
    }
    paused_now             = local.hth_containment_paused_now
    unacknowledged_resumes = local.hth_containment_unacknowledged_resumes
  }
}

Code Pack: CLI Script

hth-buildkite-4.02-agent-containment.sh View source on GitHub ↗

# Everything containment needs, in one pass: which queues are already paused, and
# every agent in the cluster with BOTH identifiers (T3) plus whether it is
# currently executing a job (which decides pause-vs-force-stop, T2/T8).
#
# Paginated properly rather than through `bk agent list`, whose --limit defaults
# to 100 and whose filters run client-side (T6), and which cannot scope to a
# cluster at all (T7). Cluster scoping is done here on
# Agent.clusterQueue.cluster.uuid — a field, not a filter argument, so it is
# read and compared rather than guessed at.
fetch_queue_page() {
  local cluster_uuid="$1" after="$2"
  jq -n --arg slug "${BUILDKITE_ORG_SLUG}" --arg cu "${cluster_uuid}" --arg after "${after}" '{
    query: "query($slug:ID!,$cu:ID!,$after:String){ organization(slug:$slug){
              cluster(id:$cu){ uuid name
                queues(first:100, after:$after){
                  edges { node {
                    id uuid key
                    dispatchPaused dispatchPausedAt dispatchPausedNote
                    dispatchPausedBy { name email }
                  } }
                  pageInfo { hasNextPage endCursor }
                } } } }",
    variables: { slug: $slug, cu: $cu, after: (if $after == "" then null else $after end) }
  }' | gql
}

list_queues() {
  local cluster_uuid="${1:?usage: queues <cluster-uuid>}"
  local after="" page out="[]"
  while :; do
    page=$(fetch_queue_page "${cluster_uuid}" "${after}")
    die_on_gql_errors "${page}"
    out=$(jq -s 'add' \
      <(printf '%s' "${out}") \
      <(jq '[.data.organization.cluster.queues.edges[].node]' <<<"${page}"))
    [ "$(jq -r '.data.organization.cluster.queues.pageInfo.hasNextPage' <<<"${page}")" = "true" ] || break
    after=$(jq -r '.data.organization.cluster.queues.pageInfo.endCursor' <<<"${page}")
  done
  printf '%s\n' "${out}"
}

fetch_agent_page() {
  local after="$1"
  jq -n --arg slug "${BUILDKITE_ORG_SLUG}" --arg after "${after}" '{
    query: "query($slug:ID!,$after:String){ organization(slug:$slug){
              agents(first:100, after:$after){
                edges { node {
                  id uuid name hostname ipAddress version
                  connectionState isRunningJob
                  paused pausedAt pausedNote pausedTimeoutInMinutes
                  stopForcedAt stoppedGracefullyAt
                  clusterQueue { id key cluster { uuid name } }
                } }
                pageInfo { hasNextPage endCursor }
              } } }",
    variables: { slug: $slug, after: (if $after == "" then null else $after end) }
  }' | gql
}

# Every agent in the ORGANIZATION, unfiltered. `id` here is the GraphQL node id
# (GraphQL mutations); `uuid` is what `bk` and REST want (T3). Both are emitted
# so no caller has to know which is which.
list_all_agents() {
  local after="" page out="[]"
  while :; do
    page=$(fetch_agent_page "${after}")
    die_on_gql_errors "${page}"
    out=$(jq -s 'add' \
      <(printf '%s' "${out}") \
      <(jq '[.data.organization.agents.edges[].node]' <<<"${page}"))
    [ "$(jq -r '.data.organization.agents.pageInfo.hasNextPage' <<<"${page}")" = "true" ] || break
    after=$(jq -r '.data.organization.agents.pageInfo.endCursor' <<<"${page}")
  done
  printf '%s\n' "${out}"
}

# Agents whose cluster uuid cannot be resolved AT ALL — clusterQueue is null, or
# it is present but its cluster is (both are nullable, T7b). These are the agents
# cluster-scoped containment structurally cannot reach. Kept as its own function
# so the omission has a name and can be printed, rather than being an invisible
# consequence of a `select`.
select_unassociated_agents() {
  jq '[.[] | select((.clusterQueue.cluster.uuid // null) == null)]'
}

select_cluster_agents() {
  jq --arg cu "${1}" '[.[] | select(.clusterQueue.cluster.uuid == $cu)]'
}

# Cluster-scoped view. WARNS on stderr about anything the scope cannot reach
# (T7b) so a caller that only reads stdout still cannot be misled about totals.
list_agents() {
  local cluster_uuid="${1:-}"
  local all unassoc n_unassoc
  all=$(list_all_agents)
  [ -n "${cluster_uuid}" ] || { printf '%s\n' "${all}"; return 0; }

  unassoc=$(select_unassociated_agents <<<"${all}")
  n_unassoc=$(jq 'length' <<<"${unassoc}")
  [ "${n_unassoc}" -eq 0 ] || echo "WARNING: ${n_unassoc} agent(s) have no resolvable cluster and are NOT in this list; cluster-scoped containment does not reach them (T7b). See: status ${cluster_uuid} | jq .agents_unassociated" >&2
  select_cluster_agents "${cluster_uuid}" <<<"${all}"
}

# Read-only situation report. Safe to run at any time and the first thing to run
# during a drill: it answers "is anything still contained" for queues Terraform
# does not manage, which the 4.2 Terraform pack structurally cannot see.
#
# The counts are deliberately THREE numbers, not one. `agents_total` used to be
# reported after the cluster filter, so an agent the filter had dropped (T7b) was
# missing from the list AND from the total that was supposed to reveal the gap.
# Reconciliation is now explicit: agents_in_cluster + agents_unassociated +
# (agents in other clusters) = agents_org_total.
status() {
  local cluster_uuid="${1:?usage: status <cluster-uuid>}"
  local queues all
  queues=$(list_queues "${cluster_uuid}")
  all=$(list_all_agents)
  jq -n --arg cu "${cluster_uuid}" --argjson q "${queues}" --argjson all "${all}" '
    ($all | map(select(.clusterQueue.cluster.uuid == $cu)))          as $a |
    ($all | map(select((.clusterQueue.cluster.uuid // null) == null))) as $u |
    {
      queues_paused:      [ $q[] | select(.dispatchPaused)       | {key, id, dispatchPausedAt, dispatchPausedNote} ],
      queues_dispatching: [ $q[] | select(.dispatchPaused | not) | .key ],
      agents_in_cluster:  ($a   | length),
      agents_org_total:   ($all | length),
      agents_running_job: [ $a[] | select(.isRunningJob) | {name, uuid, graphql_id: .id, queue: .clusterQueue.key} ],
      agents_paused:      [ $a[] | select(.paused) | {name, uuid, graphql_id: .id, pausedNote, pausedTimeoutInMinutes} ],
      agents_connected:   [ $a[] | select(.connectionState == "connected") | {name, uuid, graphql_id: .id} ],
      agents_unassociated_count: ($u | length),
      agents_unassociated: [ $u[] | {name, uuid, graphql_id: .id, connectionState, isRunningJob} ],
      agents_unassociated_note:
        "Agents with no resolvable cluster (Agent.clusterQueue or ClusterQueue.cluster is null — legacy unclustered agents). Cluster-scoped containment does not reach them: `contain <cluster-uuid>` will NOT pause or stop these. Contain each directly with stop-agent <graphql_id> graceful|force."
    }'
}

# STEP 1 — pause dispatch. Stops new jobs being handed out. Does not recall work
# already dispatched (T9), and does not touch a running job (T2).
pause_queue() {
  local queue_id="${1:?usage: pause-queue <queue-graphql-id> [note]}"
  local note="${2:-HTH 4.2 containment}"
  local body
  body=$(jq -n --arg id "${queue_id}" --arg note "${note}" '{
    query: "mutation($id:ID!,$note:String){ clusterQueuePauseDispatch(input:{id:$id, note:$note}){
              queue { id key dispatchPaused dispatchPausedAt dispatchPausedNote
                      dispatchPausedBy { name email } } } }",
    variables: { id: $id, note: $note }
  }' | gql)
  die_on_gql_errors "${body}"
  jq -r '.data.clusterQueuePauseDispatch.queue
         | "queue \(.key) dispatchPaused=\(.dispatchPaused) at \(.dispatchPausedAt // "-") by \(.dispatchPausedBy.name // "-")"' <<<"${body}"
}

# STEP 1b — pause agents. `timeoutInMinutes` is OMITTED, which `bk agent pause`
# cannot do (its --timeout-in-minutes defaults to 5 and is always sent, T1).
# The INTENT is a pause with no auto-resume. That intent is not verified (T1b),
# so it is checked rather than announced: Agent.pausedTimeoutInMinutes is NON_NULL
# in the schema, so the server always tells us what it actually applied, and a
# non-zero answer means containment has a clock on it and the responder must be
# told before they walk away. Pause is not containment on its own regardless —
# a paused agent finishes its current job (T2); `stop` is the step that ends it.
#
# Returns 4 when the pause carries an auto-resume, so a caller can count the
# agents that need re-pausing or stopping instead of trusting a green run.
assert_pause_has_no_auto_resume() {
  local body="$1" name gid timeout
  name=$(jq -r '.data.agentPause.agent.name // "<unknown>"' <<<"${body}")
  gid=$(jq -r '.data.agentPause.agent.id // "<agent-graphql-id>"' <<<"${body}")
  timeout=$(jq -r '.data.agentPause.agent.pausedTimeoutInMinutes // "unreported"' <<<"${body}")
  [ "${timeout}" = "0" ] && return 0
  cat >&2 <<EOF
WARNING: agent ${name} is paused WITH AN AUTO-RESUME of ${timeout} minute(s).
  This pack omits timeoutInMinutes intending an indefinite pause, but the server
  reported a non-zero timeout — so this agent WILL start accepting jobs again on
  its own, without anyone deciding that it should.
  Do not treat this agent as contained. Either re-pause it before the clock
  expires, or (correctly) stop it:
      stop-agent ${gid} graceful|force
EOF
  return 4
}

pause_agent() {
  local agent_gql_id="${1:?usage: pause-agent <agent-graphql-id> [note]}"
  local note="${2:-HTH 4.2 containment — no auto-resume intended, verify timeout below}"
  local body
  body=$(jq -n --arg id "${agent_gql_id}" --arg note "${note}" '{
    query: "mutation($id:ID!,$note:String){ agentPause(input:{id:$id, note:$note}){
              agent { id uuid name paused pausedAt pausedNote
                      pausedTimeoutInMinutes isRunningJob } } }",
    variables: { id: $id, note: $note }
  }' | gql)
  die_on_gql_errors "${body}"
  # No `// "indefinite"` fallback here: pausedTimeoutInMinutes is NON_NULL, so a
  # jq alternative operator on it is dead code that would print a reassurance the
  # server never sent. Print the number the server actually returned.
  jq -r '.data.agentPause.agent
         | "agent \(.name) paused=\(.paused) pausedTimeoutInMinutes=\(.pausedTimeoutInMinutes) stillRunningJob=\(.isRunningJob)"' <<<"${body}"
  assert_pause_has_no_auto_resume "${body}"
}

# STEP 2 — stop agents. This is the step that ends execution.
#   graceful=true  -> the running job finishes. Evidence preserved, containment slower.
#   graceful=false -> equivalent to `bk agent stop --force`. Job terminated (T8).
stop_agent() {
  local agent_gql_id="${1:?usage: stop-agent <agent-graphql-id> [graceful|force]}"
  local mode="${2:-graceful}"
  local graceful
  case "${mode}" in
    graceful) graceful=true ;;
    force)    graceful=false ;;
    *) echo "FATAL: mode must be 'graceful' or 'force' (got '${mode}')." >&2; exit 2 ;;
  esac
  local body
  body=$(jq -n --arg id "${agent_gql_id}" --argjson g "${graceful}" '{
    query: "mutation($id:ID!,$g:Boolean){ agentStop(input:{id:$id, graceful:$g}){
              agent { id uuid name connectionState
                      stopForcedAt stoppedGracefullyAt } } }",
    variables: { id: $id, g: $g }
  }' | gql)
  die_on_gql_errors "${body}"
  jq -r '.data.agentStop.agent
         | "agent \(.name) state=\(.connectionState) forcedAt=\(.stopForcedAt // "-") gracefulAt=\(.stoppedGracefullyAt // "-")"' <<<"${body}"
}

# Bulk stop. `bk agent stop` parallelises (--limit, default 5 workers) so it is
# preferred when present; it takes the UUID, never the GraphQL id (T3), is always
# organization-qualified (T5), and is always given </dev/null so a redirected
# stdin cannot silently replace the target list (T4). Without `bk`, the same work
# runs serially over GraphQL — slower, identical outcome.
stop_agents_bulk() {
  local cluster_uuid="${1:?usage: stop-agents <cluster-uuid> [graceful|force]}"
  local mode="${2:-graceful}"
  local agents count
  agents=$(list_agents "${cluster_uuid}")
  count=$(jq 'length' <<<"${agents}")
  [ "${count}" -gt 0 ] || { echo "no agents in cluster ${cluster_uuid}; nothing to stop"; return 0; }
  echo "stopping ${count} agent(s) in cluster ${cluster_uuid} (mode=${mode})"

  if command -v "${BK}" >/dev/null 2>&1; then
    # Organization-qualified UUIDs (T5, T3), passed as POSITIONAL arguments with
    # stdin nailed to /dev/null. Piping them into xargs would not work: `bk` reads
    # stdin in preference to its arguments (T4), so redirecting xargs' stdin
    # discards the very list being piped. Build the argv instead.
    local -a targets=()
    while read -r qualified; do targets+=( "${qualified}" ); done \
      < <(jq -r --arg org "${BUILDKITE_ORG_SLUG}" '.[] | "\($org)/\(.uuid)"' <<<"${agents}")

    local -a bk_args=( agent stop --limit 5 )
    [ "${mode}" = "force" ] && bk_args+=( --force )
    "${BK}" "${bk_args[@]}" "${targets[@]}" < /dev/null
  else
    # No `bk`: same outcome serially over GraphQL, using the node id, not the uuid.
    while read -r gid; do stop_agent "${gid}" "${mode}"; done \
      < <(jq -r '.[].id' <<<"${agents}")
  fi
}

# The whole runbook, in the order that works. Deliberately stops one step short:
# token revocation is controls 3.1 / 2.5 and is printed as a handoff rather than
# performed, so nobody can mistake "this script finished" for "the fleet cannot
# re-register".
contain() {
  local cluster_uuid="${1:?usage: contain <cluster-uuid> [graceful|force]}"
  local mode="${2:-graceful}"
  local all unassoc n_unassoc aid
  local -a auto_resumed=()

  # STEP 0 — say what this run structurally cannot reach BEFORE doing anything,
  # so it is at the top of the responder's scrollback rather than buried (T7b).
  all=$(list_all_agents)
  unassoc=$(select_unassociated_agents <<<"${all}")
  n_unassoc=$(jq 'length' <<<"${unassoc}")
  echo "── step 0: scope — $(jq 'length' <<<"${all}") agent(s) in the organization, $(select_cluster_agents "${cluster_uuid}" <<<"${all}" | jq 'length') in this cluster"
  if [ "${n_unassoc}" -gt 0 ]; then
    {
      echo "!! ${n_unassoc} AGENT(S) ARE OUT OF SCOPE FOR THIS CONTAINMENT."
      echo "   They have no resolvable cluster (Agent.clusterQueue / ClusterQueue.cluster"
      echo "   is null — legacy unclustered agents look like this). Nothing below pauses or"
      echo "   stops them; they keep taking jobs while this runbook reports success."
      jq -r '.[] | "   OUT OF SCOPE: \(.name) graphql_id=\(.id) uuid=\(.uuid) state=\(.connectionState) runningJob=\(.isRunningJob)"' <<<"${unassoc}"
      echo "   Contain each directly:  stop-agent <graphql_id> ${mode}"
    } >&2
  fi

  echo "── step 1: pause dispatch on every queue in the cluster"
  while read -r qid; do pause_queue "${qid}" "HTH 4.2 containment"; done \
    < <(list_queues "${cluster_uuid}" | jq -r '.[] | select(.dispatchPaused | not) | .id')

  echo "── step 1b: pause agents (timeoutInMinutes omitted; the reported timeout is asserted)"
  # A pause that came back with an auto-resume clock must NOT abort the run:
  # step 2 is the step that actually contains, and aborting before it would leave
  # the fleet running. Collect the failures and surface them at the end instead.
  while read -r aid; do
    pause_agent "${aid}" || auto_resumed+=( "${aid}" )
  done < <(select_cluster_agents "${cluster_uuid}" <<<"${all}" | jq -r '.[] | select(.paused | not) | .id')

  echo "── step 2: stop agents (mode=${mode})"
  stop_agents_bulk "${cluster_uuid}" "${mode}"

  if [ "${#auto_resumed[@]}" -gt 0 ]; then
    {
      echo "!! ${#auto_resumed[@]} agent(s) were paused WITH AN AUTO-RESUME CLOCK (see warnings above):"
      printf '   %s\n' "${auto_resumed[@]}"
      echo "   The indefinite-pause assumption did not hold on this tenant. Verify each was"
      echo "   stopped by step 2, and correct T1b in this pack's header with what you saw."
    } >&2
  fi

  echo "── step 3: REVOKE TOKENS — NOT DONE BY THIS SCRIPT, AND NOT OPTIONAL"
  cat >&2 <<'HANDOFF'
Agents are stopped; nothing has been revoked. Revocation does not disconnect
connected agents, which is why it comes last — but skipping it means the cluster
registration token is still valid and a rebuilt-from-the-same-image host will
re-register straight back into the incident.

  cluster agent tokens : GraphQL clusterAgentTokenRevoke{ id: ID!, organizationId: ID! },
                         or packs/buildkite/api/hth-buildkite-3.01-agent-token-lifecycle.sh
                         (`revoke <token_id>`, or `contain <cluster_graphql_id> <token_id>`
                         which revokes AND stops the agents the token registered).
                         Terraform: DELETE the token's entry from var.agent_tokens in
                         hth-buildkite-3.01-configure-agent-tokens.tf and apply — the plan
                         reads "1 to destroy". Do NOT reach for a rotation here: rotation in
                         that pack is deliberately a two-apply add-then-remove (its TRAP 5),
                         because the new secret is returned exactly once and revoking the
                         incumbent in the same apply strands every host that has not yet
                         re-registered. During an incident you WANT the revoke; issue the
                         replacement token as a separate, later act.
  org API tokens       : packs/buildkite/api/hth-buildkite-2.05-token-hygiene.sh revoke

Do not resume dispatch until fresh tokens are issued and the agent hosts are
REBUILT rather than restarted.
HANDOFF

  # A run that could not reach every agent, or that paused agents onto an
  # auto-resume clock, must not exit 0 into a runbook that treats 0 as "the
  # fleet is contained".
  if [ "${n_unassoc}" -gt 0 ] || [ "${#auto_resumed[@]}" -gt 0 ]; then
    echo "CONTAINMENT INCOMPLETE: ${n_unassoc} agent(s) out of cluster scope, ${#auto_resumed[@]} agent(s) paused with an auto-resume clock. Exit 5." >&2
    return 5
  fi
}

# Restore is the step that hands work back to a fleet that was contained for a
# reason, so it refuses to run on an assumption. HTH_HOSTS_REBUILT=1 is the
# operator asserting the agent hosts were rebuilt from a known-good image and
# fresh registration tokens were issued — the two things that make resuming
# something other than restarting the incident.
assert_restore_authorised() {
  [ "${HTH_HOSTS_REBUILT:-0}" = "1" ] || {
    cat >&2 <<'REFUSE'
REFUSING to resume dispatch.

Set HTH_HOSTS_REBUILT=1 only once BOTH are true:
  * the agent hosts were REBUILT from a known-good image, not restarted — a
    restarted host still carries whatever the compromised job wrote to it;
  * the cluster's agent registration tokens were rotated, so the old token
    cannot bring the old fleet back.

Resuming without these returns the compromised fleet to production with a
green checkmark next to it.
REFUSE
    exit 3
  }
}

resume_queue() {
  assert_restore_authorised
  local queue_id="${1:?usage: resume-queue <queue-graphql-id>}"
  local body
  body=$(jq -n --arg id "${queue_id}" '{
    query: "mutation($id:ID!){ clusterQueueResumeDispatch(input:{id:$id}){
              queue { id key dispatchPaused } } }",
    variables: { id: $id }
  }' | gql)
  die_on_gql_errors "${body}"
  jq -r '.data.clusterQueueResumeDispatch.queue
         | "queue \(.key) dispatchPaused=\(.dispatchPaused)"' <<<"${body}"
}

resume_agent() {
  assert_restore_authorised
  local agent_gql_id="${1:?usage: resume-agent <agent-graphql-id>}"
  local body
  body=$(jq -n --arg id "${agent_gql_id}" '{
    query: "mutation($id:ID!){ agentResume(input:{id:$id}){
              agent { id uuid name paused connectionState } } }",
    variables: { id: $id }
  }' | gql)
  die_on_gql_errors "${body}"
  jq -r '.data.agentResume.agent
         | "agent \(.name) paused=\(.paused) state=\(.connectionState)"' <<<"${body}"
}

restore() {
  assert_restore_authorised
  local cluster_uuid="${1:?usage: restore <cluster-uuid>}"
  echo "── resuming paused agents"
  while read -r aid; do resume_agent "${aid}"; done \
    < <(list_agents "${cluster_uuid}" | jq -r '.[] | select(.paused) | .id')
  echo "── resuming queue dispatch"
  while read -r qid; do resume_queue "${qid}"; done \
    < <(list_queues "${cluster_uuid}" | jq -r '.[] | select(.dispatchPaused) | .id')
  echo "── post-restore state"
  status "${cluster_uuid}"
}

Sources: Buildkite agent tokens · Managing queues · buildkite-agent stop

5. Compliance Quick Reference

SOC 2 Trust Services Criteria Mapping

Control ID	Buildkite Control	Guide Section
CC6.1	SSO/2FA	1.1
CC6.1	Inbound OIDC trust for registries	3.11
CC6.2	Team permissions	2.1
CC6.2	Dormant member removal	2.6
CC6.3	Cross-pipeline access rules	2.7
CC6.3	Cluster maintainer delegation	3.7
CC6.6	API access token restrictions	2.5
CC6.7	Agent tokens	3.1
CC6.7	Build secrets management	3.5
CC6.8	Agent execution environment	3.9
CC7.1	Untrusted input controls	2.4
CC7.2	Audit logging	4.1
CC7.4	Build-fleet incident containment	4.2
CC8.1	Pipeline signing	3.4
CC8.1	Build artifact attestation	3.8
CC8.1	Pipeline templates	3.10

NIST 800-53 Rev 5 Mapping

Control	Buildkite Control	Guide Section
IA-2	SSO	1.1
IA-2(1)	2FA	1.2
AC-6	Team permissions	2.1
SI-10	Untrusted input controls	2.4
IA-5	API access token hygiene	2.5
AC-2(3)	Dormant member removal	2.6
AC-4	Cross-pipeline access rules	2.7
SC-12	Agent tokens	3.1
SI-7	Pipeline signing	3.4
SC-28	Build secrets management	3.5
IA-9	OIDC federation for cloud access	3.6
AC-5	Cluster maintainer delegation	3.7
SR-4	Build artifact attestation	3.8
CM-6	Agent execution environment	3.9
CM-2	Pipeline templates	3.10
IA-9	Inbound OIDC trust for registries	3.11
AU-2	Audit logging	4.1
IR-4	Build-fleet incident containment	4.2

On benchmark coverage: there is no CIS Benchmark, DISA STIG, or CISA SCuBA baseline for Buildkite — the CIS Benchmark index was checked and returned no Buildkite entry. The mappings above are to the general control catalogs only, and no product-specific benchmark IDs exist to cite.

Appendix A: References

Official Buildkite Documentation:

API Documentation:

Compliance Frameworks:

SOC 2 Type II — Buildkite undergoes an annual audit covering Pipelines, Package Registries, and Test Engine; request the current report through your Buildkite account team

Security Incidents:

No major public security breaches identified. Buildkite maintains annual third-party penetration testing and a private HackerOne bug bounty program.

Changelog

Date	Version	Maturity	Changes	Author
2026-08-20	0.3.1	ai-drafted · ai-validated	Added ai-validated to this guide’s status set, which now reads ai-drafted + ai-validated — the first changelog entry to record that this guide was exercised against a live Buildkite organization, which is what makes the new per-requirement badges legible. An AI agent did the exercising; no human practitioner has reviewed or applied this guide, so it claims no ni- status. What was exercised (2026-08-16): an authenticated console session confirmed 1.1’s nav item reads Single Sign On at `/sso` — the previous doc-derived “SSO Settings” label was wrong, and trusting the vendor doc over the UI had made the guide more wrong, not less — plus 1.2’s Organization Settings → Security page and its Enforce Two-factor authentication checkbox verbatim, 2.1’s `/teams` page, and 2.5’s `/api-access-audit` page with its Owner/Used/Created/Scopes/Pipelines columns and CSV export. Terraform was then applied against a real organization: 3 teams (2.1) and 2 agent tokens (3.1) were created, confirmed in the console, and destroyed with the organization returned to baseline; control 1.2’s `buildkite_organization` resource failed at apply on an API-IP-allowlist plan gate that `validate` and `plan` both reported clean, which is why 1.2 now carries a Prerequisites block. What was NOT exercised: only 5 of 22 controls carry a badge. Most mutations were deliberately never run, and no `api/`, `cli/`, or `config/` pack was executed against the tenant. The eight controls added after the live pass — 2.6, 2.7, 3.7, 3.8, 3.9, 3.10, 3.11 and 4.2 — carry no live evidence whatsoever. 4.1’s audit log is Enterprise-only and returns 404 on the trial organization walked, so it is plan-gated and unbadged. No control text, control number, or heading changed.	Claude Code (Opus 5)
2026-08-18	0.3.0	ai-drafted	Close the Buildkite reconciliation: end the Terraform monoculture and make every leveled control carry a real automation verdict. New controls: 2.6 dormant organization members, 2.7 cross-pipeline access via Buildkite Rules, 3.7 delegated cluster administration, 3.8 build-artifact attestation (SLSA provenance), 3.9 agent execution environment, 3.10 pipeline templates, 3.11 inbound OIDC trust, 4.2 build-fleet incident containment. New Code Packs: 21 new files across four surfaces — `api/` (5 new, 6 total), `cli/` (3 new, 3 total), `config/` (6 new, 6 total) and `terraform/` (7 new, 15 total); before this version the corpus was 8 Terraform control packs plus the single GraphQL `api/` pack added in 0.2.1. Non-comment HCL across the vendor’s control `.tf` files rises from 121 lines over 8 files to more than 1,100 over 15, and the REST, CLI and agent-config surfaces go from 0% coverage to shipped code. Corrections: 2.2 documents the organization-level pipeline toggles as ClickOps-only, replacing an incorrect “REST `/organizations` is GET-only” premise with the real evidence (the resource family is writable — `PATCH`/`PUT`/`DELETE` on `pipeline-settings` and its sub-resources, `PATCH` on `api-settings` — but carries no permissions payload, and no organization security resource exists), and corrects the Create Pipelines claim: the vendor’s own permissions page states the org toggle “will be unavailable on this page” when teams are enabled, so it and the team-level `members_can_create_pipelines` are mutually exclusive alternatives rather than layers, making the previously described state unreachable in both configurations; 2.5 adds inactive-token auto-revocation, warns that Portal tokens are admin-privileged and long-lived, and adds the previously missing `restrict_user_api_token_creation` (“only organization administrators can create API access tokens”) as Step 1 — the one organization-wide token control that is not plan-gated — with `restrict-token-creation-status` / `set-restrict-token-creation` verbs in the `api/` pack that send a single-key `PATCH` so toggling it can never re-assert the IP allowlist sharing that resource; 3.4 fixes `--jwks-file`/`--jwks-key-id` being presented as agent config keys when they are `tool sign` flags, records that `verification-failure-behavior` already defaults to `block` (so the rollout runs the other way), corrects the over-broad claim that verification is gated entirely on a configured JWKS — unsigned jobs do execute without one, but a signed job reaching a keyless agent is still rejected under `block`, so a partial rollout fails closed in both directions — and flags the unverified GCP KMS claim; 3.9 corrects a blanket assertion that all three of its settings are beyond a `pipeline.yml`’s reach: `no-ssh-keyscan` and `bootstrap-script` are agent configuration options, but `BUILDKITE_CLEAN_CHECKOUT` is not an agent config option at all and sits outside the scope `checkout-override-mode` governs, so even `strict` does not lock it — the step now names the plugin allowlist and `disconnect-after-job` as what actually closes the gap; 3.5 adds the API-payload exposure path, `$$` escaping, and the unreconciled 32 KB / 8 KB value-size discrepancy. §5 and Appendix A: add SOC 2 and NIST 800-53 rows for all eight new controls (the tables had been left at the 0.2.0 control set) and add the newly cited Buildkite documentation. Compliance mapping fixes: 3.4’s CIS safeguard was 16.9 “Train Developers in Application Security Concepts and Secure Coding”, a training safeguard with nothing to do with signing — replaced with 2.7 “Allowlist Authorized Scripts”, whose catalog text prescribes “digital signatures … to ensure that only authorized scripts … are allowed to execute” and “Block unauthorized scripts from executing”; 3.11’s was 5.4 “Restrict Administrator Privileges to Dedicated Administrator Accounts”, which governs human admin accounts rather than the pipeline machine identity the control constrains — replaced with 5.5 “Establish and Maintain an Inventory of Service Accounts”.	Claude Code (Opus 5)
2026-08-17	0.2.1	ai-drafted	Replace three prose-only Code Packs with real, schema-verified code. 1.1: the Terraform provider exposes no SSO resource (21 resources, 16 data sources, none for SSO), so the empty `.tf` is replaced by a GraphQL `api/` pack using the live-introspected `ssoProvider` mutation family, with the disable path documented as the way back from an SSO lockout. 2.3:* now a real verification pack over the `buildkite_organization_members` data source and `buildkite_team_member` roles, stating honestly that org-level role is not exposed to Terraform and lives in GraphQL. 3.3: now real cluster isolation — `buildkite_cluster`, `buildkite_cluster_queue`, and cluster-scoped `buildkite_cluster_agent_token` with the lockout-capable IP allowlist.	Claude Code (Opus 5)
2026-08-08	0.2.0	ai-drafted	Currency pass. 1.1: correct the plan gate to Pro or Enterprise (no “Business” tier exists) and expand enforcement — SSO required/optional is per user, organization-wide enforcement works by disabling 2FA authentication as a login method, session timeout ranges from 6 hours to 1 year, IP address pinning revokes a session on IP change (Enterprise), SCIM deprovisioning (Enterprise), and members are provisioned just-in-time on first login. 3.1: correct agent tokens to cluster-scoped, and add expiration timestamps (API-only, at least 10 minutes out, immutable once set; web-UI tokens have no expiry) and the Allowed IP Addresses CIDR allowlist. 3.2: add the unclustered agents and tokens deprecation, unavailable to organizations created after 2024-02-26. 4.1: correct the non-existent retention setting — the audit log is Enterprise-only at Organization Settings → Audit → Audit Log, events are stored indefinitely, the UI browses 12 months with older events via GraphQL, and search covers 90 days with 3 terms and 250 characters; add EventBridge streaming and REST/GraphQL retrieval. New controls: 2.4 untrusted-input pipeline controls, 2.5 API access token hygiene, 3.4 pipeline signing and verification, 3.5 build secrets management, 3.6 OIDC instead of static cloud credentials. §5: add mappings for the new controls and record that no CIS, DISA, or SCuBA baseline exists for Buildkite. Appendix A: remove the Trust Center and marketing security-page rows and add the newly cited documentation. Not surveyed this pass: Tier 3/4 research	Claude Code (Opus 5)
2026-06-29	0.1.1	ai-drafted	Add cheat-sheet Description and Rationale for all controls	Claude Code (Opus 4.8)
2025-02-05	0.1.0	ai-drafted	Initial guide with SSO, teams, and agent security	Claude Code (Opus 4.5)

Contributing

Found an issue or want to improve this guide?

Report outdated information: Open an issue with tag content-outdated
Propose new controls: Open an issue with tag new-control
Submit improvements: See Contributing Guide

Buildkite Hardening Guide

Overview

Intended Audience

How to Use This Guide

Scope

Table of Contents

1. Authentication & SSO

1.1 Configure SAML Single Sign-On

Description

Rationale

Prerequisites

ClickOps Implementation

Code Implementation

1.2 Enforce Two-Factor Authentication

Description

Rationale

Prerequisites

ClickOps Implementation

Code Implementation

2. Access Controls

2.1 Configure Team Permissions

Description

Rationale

ClickOps Implementation

Code Implementation

2.2 Configure Pipeline Permissions

Description

Rationale

ClickOps Implementation

Code Implementation

2.3 Limit Admin Access

Description

Rationale

ClickOps Implementation

Code Implementation

2.4 Control Untrusted Input to Pipelines

Description

Rationale

ClickOps Implementation

Code Implementation

2.5 Manage API Access Token Hygiene

Description

Rationale

ClickOps Implementation

Code Implementation

2.6 Remove Dormant Organization Members

Description

Rationale

ClickOps Implementation

Code Implementation

2.7 Govern Cross-Pipeline Access with Rules

Description

Rationale

ClickOps Implementation

Code Implementation

3. Agent Security

3.1 Configure Agent Tokens

Description

Rationale

ClickOps Implementation

Code Implementation

3.2 Configure Agent Clusters

Description

Deprecation: Unclustered Agents

Rationale

ClickOps Implementation

Code Implementation

3.3 Secure Agent Infrastructure

Description

Rationale

ClickOps Implementation

Code Implementation

3.4 Enable Pipeline Signing and Verification

Description

Rationale

ClickOps Implementation

Code Implementation

3.5 Manage Build Secrets

Description

Rationale