Skip to content

Cloud · Google

Google Cloud global API outage from Service Control crash loop

Started
Jun 12, 2025, 17:51 UTC
Resolved
Jun 13, 2025, 01:18 UTC
Duration
7h 27m (from provider timestamps)
Affected
Google and Google Cloud APIs (Service Control / API management), Google Compute Engine, Google Cloud Storage, Google BigQuery, Cloud Run, Identity and Access Management, Google Cloud Console, Google Workspace products (per mini report), and other products listed on the incident page
Cause
Final reviewA Service Control code path added on May 29, 2025 had no error handling and no feature flag. On June 12 a policy change with unintended blank fields replicated globally within seconds, hit a null pointer and put Service Control into a crash loop in every regional deployment. Restarting tasks in us-central1 overloaded the Spanner table behind it because Service Control lacked randomized exponential backoff.
Was it an attack?
No, per the provider or investigators
State
resolved

Anatomy

  1. 1. Trigger

    Policy or data push

  2. 2. Already broken

    Latent bug, No feature flag, Missing validation

  3. 3. What made it spread

    Global configuration propagation, No backoff, Control-plane dependency

  4. 4. Blast radius

    global: Global, us-central1 (prolonged recovery)

  5. 5. Cause category

    Change or deployment

Labels follow the root-cause taxonomy. A person's change is recorded as the trigger and the missing safeguard as what was already broken; “human error” is never a cause here.

How the explanation changed

  1. June 12, 2025 · Preliminary cause

    “From our initial analysis, the issue occurred due to an invalid automated quota update to our API management system which was distributed globally, causing external API requests to be rejected.”

    Google Cloud Service Health incident ow5i3PPK96RduMcb1SsW (mini report 2025-06-12 23:34 PDT; incident report 2025-06-13 16:45 PDT)

  2. June 13, 2025 · Final review

    “The issue with this change was that it did not have appropriate error handling nor was it feature flag protected. Without the appropriate error handling, the null pointer caused the binary to crash.”

    Google Cloud Service Health incident ow5i3PPK96RduMcb1SsW (mini report 2025-06-12 23:34 PDT; incident report 2025-06-13 16:45 PDT)

  3. June 13, 2025 · Final review

    “Service Control did not have the appropriate randomized exponential backoff implemented to avoid this.”

    Google Cloud Service Health incident ow5i3PPK96RduMcb1SsW (mini report 2025-06-12 23:34 PDT; incident report 2025-06-13 16:45 PDT)

What the provider committed to change

  • Freeze changes to the Service Control stack and manual policy pushes until remediated
  • Modularize Service Control so checks fail open
  • Audit systems consuming globally replicated data; propagate replication incrementally with time to validate
  • Require feature flag protection, disabled by default, for changes to critical binaries
  • Improve static analysis and testing for error handling
  • Audit systems for randomized exponential backoff
  • Improve automated and human external communications
  • Keep monitoring and communication infrastructure up when Google Cloud is down

Post-incident reviews

Sources

  1. Google Cloud Service Health incident ow5i3PPK96RduMcb1SsW (mini report 2025-06-12 23:34 PDT; incident report 2025-06-13 16:45 PDT) (provider, read November 17, 2026)

Last verified November 17, 2026. Spot an error? The methodology page says how corrections work.