Cloud · Google
Google Cloud global API outage from Service Control crash loop
- Started
- Jun 12, 2025, 17:51 UTC
- Resolved
- Jun 13, 2025, 01:18 UTC
- Duration
- 7h 27m (from provider timestamps)
- Affected
- Google and Google Cloud APIs (Service Control / API management), Google Compute Engine, Google Cloud Storage, Google BigQuery, Cloud Run, Identity and Access Management, Google Cloud Console, Google Workspace products (per mini report), and other products listed on the incident page
- Cause
- Final reviewA Service Control code path added on May 29, 2025 had no error handling and no feature flag. On June 12 a policy change with unintended blank fields replicated globally within seconds, hit a null pointer and put Service Control into a crash loop in every regional deployment. Restarting tasks in us-central1 overloaded the Spanner table behind it because Service Control lacked randomized exponential backoff.
- Was it an attack?
- No, per the provider or investigators
- State
- resolved
Anatomy
1. Trigger
Policy or data push
2. Already broken
Latent bug, No feature flag, Missing validation
3. What made it spread
Global configuration propagation, No backoff, Control-plane dependency
4. Blast radius
global: Global, us-central1 (prolonged recovery)
5. Cause category
Change or deployment
Labels follow the root-cause taxonomy. A person's change is recorded as the trigger and the missing safeguard as what was already broken; “human error” is never a cause here.
How the explanation changed
June 12, 2025 · Preliminary cause
“From our initial analysis, the issue occurred due to an invalid automated quota update to our API management system which was distributed globally, causing external API requests to be rejected.”
June 13, 2025 · Final review
“The issue with this change was that it did not have appropriate error handling nor was it feature flag protected. Without the appropriate error handling, the null pointer caused the binary to crash.”
June 13, 2025 · Final review
“Service Control did not have the appropriate randomized exponential backoff implemented to avoid this.”
What the provider committed to change
- Freeze changes to the Service Control stack and manual policy pushes until remediated
- Modularize Service Control so checks fail open
- Audit systems consuming globally replicated data; propagate replication incrementally with time to validate
- Require feature flag protection, disabled by default, for changes to critical binaries
- Improve static analysis and testing for error handling
- Audit systems for randomized exponential backoff
- Improve automated and human external communications
- Keep monitoring and communication infrastructure up when Google Cloud is down
Post-incident reviews
- Preliminary review, June 12, 2025
- Final review, June 13, 2025
Other Google incidents
Sources
- Google Cloud Service Health incident ow5i3PPK96RduMcb1SsW (mini report 2025-06-12 23:34 PDT; incident report 2025-06-13 16:45 PDT) (provider, read November 17, 2026)
Last verified November 17, 2026. Spot an error? The methodology page says how corrections work.