  |
Aug 23, 4:18 AM PDT Beginning at 8:36 PM PDT a small percentage of EC2 servers in a single Availability Zone in the AP-NORTHEAST-1 Region shutdown due to overheating.
This resulted in impaired EC2 instances and degraded EBS volume performance for resources in the affected area of the Availability Zone.
The overheating was caused by a control system failure that caused multiple, redundant cooling systems to fail in parts of the affected Availability Zone.
The chillers were restored at 11:21 PM PDT and temperatures in the affected areas began to return to normal.
As temperatures returned to normal, power was restored to the affected instances.
By 2:30 AM PDT, the vast majority of instances and volumes had recovered.
We have been working to recover the remaining instances and volumes.
A small number of remaining instances and volumes are hosted on hardware which was adversely affected by the loss of power.
We continue to work to recover all affected instances and volumes.
For immediate recovery, we recommend replacing any remaining affected instances or volumes if possible.
Some of the affected instances may require action from customers and we will be reaching out to those customers with next steps.
29
