AWS Outage (June 29th) – Weathering the Storm
On the evening of Friday, June 29th, Amazon Web Services (AWS) experienced a major outage at its North Virginia location due to a loss of power. This outage, the second in June, affected numerous AWS customers who use PagerDuty. As PagerDuty is also an AWS customer, we are pleased to inform you that we were able to “weather the storm” and did not suffer from any downtime.
As we strive to keep our level of customer support and transparency at its highest level, we would like to briefly recap the evenings events, what we did to resolve, and what we are continuing to do to ensure that your alerting services through PagerDuty are uninterrupted to the best of our ability.
What PagerDuty Saw
At 8:06pm (Pacific time) on Friday, June 29th, our system noticed an unusual spike of alerts occurring across the PagerDuty platform. According to Amazon’s own analysis “Approximately 7% of the EC2 instances in the US-EAST-1 Region were in the impacted Availability Zone and impacted by the power loss.”
Notable alert spikes which made us aware of the AWS event are below:
~20x increase in traffic due to AWS outage. (The second spike coincides with the addition of a leap second into UTC and is likely unrelated).
As mentioned in our June 18th blog post, after the first AWS outage in June of 2012, we have done, and are doing, the following to ensure the availability of PagerDuty to our customers:
Migrated data centers off of AWS US-East and into US-West:
This data center migration was performed on June 19th, 2012. The purpose of this migration was to de-correlate our failures from those of our customers. In other words, it’s obviously not a good idea for us to run on the same infrastructure as a large percentage (over 20%) of our customers. Thus, we moved off of US-East.
We at PagerDuty take all matters related to the availability of our services very seriously. If you have any questions or concerns, please do not hesitate to contact us at support@pagerduty.com.