Skip to content

Cloud · Amazon Web Services

AWS us-east-1 outage from DynamoDB DNS automation race condition

Started
Oct 20, 2025, 06:48 UTC
Resolved
Oct 20, 2025, 21:20 UTC
Duration
14h 32m (from provider timestamps)
Affected
Amazon DynamoDB, Amazon EC2, Network Load Balancer, AWS Lambda, Amazon ECS, Amazon EKS, AWS Fargate, Amazon Connect, AWS Support Center, other services dependent on DynamoDB
Cause
Final reviewA latent race condition between two DNS Enactors in DynamoDB's automated DNS management system left an incorrect empty DNS record for the regional DynamoDB endpoint. Systems that depended on DynamoDB, including EC2's droplet lease manager, failed. After DNS was restored, EC2 lease recovery collapsed under its backlog, network state propagation lagged, and NLB health check failures caused further connection errors.
Was it an attack?
No, per the provider or investigators
State
resolved

Anatomy

  1. 1. Trigger

    Maintenance automation

  2. 2. Already broken

    Race condition, Latent bug

  3. 3. What made it spread

    DNS dependency, Control-plane dependency, Health-check cascade, Recovery backlog

  4. 4. Blast radius

    single region: us-east-1 (N. Virginia)

  5. 5. Cause category

    Software or automation

Labels follow the root-cause taxonomy. A person's change is recorded as the trigger and the missing safeguard as what was already broken; “human error” is never a cause here.

How the explanation changed

  1. Date not shown on the source · Final review

    “The root cause of this issue was a latent race condition in the DynamoDB DNS management system that resulted in an incorrect empty DNS record for the service’s regional endpoint (dynamodb.us-east-1.amazonaws.com) that the automation failed to repair.”

    AWS: Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region

What the provider committed to change

  • DynamoDB DNS Planner and DNS Enactor automation disabled worldwide
  • Fix the race condition and add protections against applying incorrect DNS plans before re-enabling
  • NLB velocity control limiting capacity a single NLB can remove on AZ failover
  • Additional EC2 test suite exercising the DWFM recovery workflow
  • EC2 data propagation throttling based on queue size

Post-incident reviews

Sources

  1. AWS: Summary of the Amazon DynamoDB Service Disruption in the Northern Virginia (US-EAST-1) Region (provider, read November 17, 2026)

Last verified November 17, 2026. Spot an error? The methodology page says how corrections work.