Grafana Breach Caused by Missed Token Rotation After TanStack Attack

Overview

Grafana Labs has published details on the root cause of its recent source code breach, revealing that the incident was caused by a single GitHub workflow token that was missed during the post-incident token rotation process following the TanStack npm supply chain attack.

The disclosure is a cautionary tale about the complexity of post-incident remediation: even when an organization responds quickly and competently to a supply chain compromise, incomplete credential rotation can leave a residual exposure window that a sophisticated attacker can exploit.

What Happened

The sequence of events that led to Grafana's breach unfolded in two phases:

Phase 1 — TanStack npm Compromise (Early May 2026)

Grafana engineers, like many JavaScript developers, had TanStack packages in their development environments. When TeamPCP's infostealer was deployed through compromised TanStack npm packages, GitHub tokens were harvested from affected developer machines. Grafana identified the exposure and initiated a broad token rotation process.

Phase 2 — The Missed Token (May 2026)

During the post-TanStack remediation effort, Grafana rotated tokens across its GitHub environment. However, one GitHub Actions workflow token used in an internal CI/CD workflow was not included in the rotation sweep. This token remained valid and retained access to Grafana's private repositories.

TeamPCP, having collected the token from the earlier TanStack compromise, discovered it had not been invalidated and used it to access and exfiltrate Grafana's private source code.

Attack Timeline:
 
  TanStack attack
      ↓
  Grafana token harvested from developer machine
      ↓
  Grafana initiates token rotation — MOST tokens rotated
      ↓
  ONE GitHub workflow token missed in rotation sweep
      ↓
  TeamPCP uses the surviving token to access Grafana repos
      ↓
  Private Grafana source code exfiltrated
      ↓
  Breach confirmed

Why Token Rotation Is Hard

This incident illustrates why complete credential rotation is operationally difficult, especially in large engineering organizations:

Technical Analysis: GitHub Actions Workflow Tokens

GitHub Actions workflows can use several types of tokens, each with different characteristics:

# Automatic GITHUB_TOKEN — ephemeral, expires after workflow run
# This type is NOT vulnerable to this attack pattern
 
# Repository-scoped PAT stored as a secret — VULNERABLE
env:
  GH_TOKEN: ${{ secrets.GITHUB_TOKEN_CUSTOM }}
 
# Fine-grained PAT with cross-repo access — VULNERABLE if not rotated
- uses: actions/checkout@v4
  with:
    token: ${{ secrets.CROSS_REPO_TOKEN }}
 
# GitHub App installation token — VULNERABLE if app credentials not rotated

The token that Grafana missed was likely a PAT or GitHub App token stored as a GitHub Actions secret and used to access multiple internal repositories during CI/CD workflows — a common pattern for monorepo-adjacent tooling or release automation.

Lessons for Post-Incident Token Rotation

This breach provides a clear playbook for what needs to happen after a supply chain compromise that may have exposed GitHub credentials:

Step 1: Full Token Inventory Before Rotation

# List ALL fine-grained PATs at the organization level
gh api /orgs/{org}/personal-access-token-requests --paginate | \
  jq '.[] | {id, owner: .owner.login, token_last_eight: .token_last_eight}'
 
# List all GitHub Apps installed on the organization
gh api /orgs/{org}/installations --paginate | \
  jq '.installations[] | {id, app_slug: .app.slug, created_at}'
 
# Audit GitHub Actions secrets (these may hold PATs)
for repo in $(gh repo list {org} --json name -q '.[].name'); do
  echo "=== $repo ==="
  gh secret list --repo {org}/$repo
done
 
# Check organization-level secrets
gh secret list --org {org}

Step 2: Systematic Rotation by Token Type

# 1. Revoke and regenerate all classic PATs
#    GitHub.com > Settings > Developer settings > Personal access tokens (classic)
#    Click "Delete" on ALL tokens — regenerate only what's immediately needed
 
# 2. Revoke all fine-grained PATs
#    Settings > Developer settings > Personal access tokens (fine-grained)
 
# 3. Rotate GitHub Apps private keys
#    GitHub.com > Settings > Developer settings > GitHub Apps > {App} > Private keys
 
# 4. Update ALL GitHub Actions secrets that contain tokens
gh secret set NEW_TOKEN --repo {org}/{repo} < new_token.txt
 
# 5. Verify rotation: attempt to use old tokens (should return 401)
curl -H "Authorization: token {old_token}" https://api.github.com/user

Step 3: Validate Rotation Completeness

# Search GitHub Actions workflow files for token usage patterns
# to ensure every reference has been covered in rotation
find .github/workflows -name "*.yml" -o -name "*.yaml" | \
  xargs grep -l "secrets\." | \
  xargs grep -E "(TOKEN|PAT|KEY|SECRET)" | \
  sort -u
 
# Check for tokens in environment variable references
find .github/workflows -name "*.yml" | \
  xargs grep -E "\$\{\{.*secrets\." | \
  awk -F'secrets\.' '{print $2}' | tr -d '}}' | sort -u

Grafana's Response

Grafana Labs stated that upon discovering the breach:

• The surviving token was immediately revoked
• A full audit of all GitHub tokens and secrets was conducted
• The compromised source code repositories were identified and scoped
• GitHub was notified and participated in the joint investigation
• No customer data was exposed — the breach was limited to Grafana source code

Grafana indicated it is implementing improved token inventory tooling to prevent a similar gap in future rotation exercises.

Broader Context: The TanStack Attack Aftermath

The TanStack npm attack rippled through the developer ecosystem in ways that are still being fully assessed. Confirmed downstream breaches include:

Each organization had a different failure mode — demonstrating that supply chain attacks create complex, multi-organization fallout that takes weeks to fully trace and remediate.

Key Takeaway

Post-incident token rotation must be treated as a comprehensive, audited process — not a best-effort sweep. The Grafana breach shows that a single missed token can turn a successfully contained initial incident into a full source code breach. Organizations should build token inventory and rotation verification into their incident response runbooks, not as an afterthought, but as a first-class step.

Sources

• BleepingComputer — Grafana Breach Caused by Missed Token Rotation After TanStack Attack

Related Reading

• GitHub Breached — Employee Device Hack Led to Exfiltration of 3,800+ Internal Repos
• GitHub Confirms Breach, 4K Internal Repos Stolen
• Grafana GitHub Breach Exposes Source Code via TanStack npm Attack
• TanStack Supply Chain Attack Hits Two OpenAI Employee Devices