Service Disruption Timeline for October 21st, 2016
On Friday October 21, 2016, PagerDuty had an outage followed by a period of service degradation. We take outages very seriously, especially when it impacts our customers.
In this post, we summarize a timeline of events and steps our engineers took to mitigate the problem. In a follow-up post tomorrow, we will be outlining what steps we will be taking to reduce the likelihood of similar outages in the future.
DNS Usage at PagerDuty
PagerDuty utilizes DNS for two purposes. Our main use of DNS is for giving our customers routing information in order to access PagerDuty services. In addition, we also use DNS for internal communication between our servers. At the time of the outage, we were leveraging our primary provider for powering both our external and internal routing information.
What happened
Timeline of Events (All times in UTC)
16:00 – PagerDuty Engineers are alerted to primary DNS provider lookup failures
16:04 – PagerDuty Engineers confirm intermittent lookup failures for pagerduty.com
16:06 – Primary DNS provider discloses service degradation
16:40 – PagerDuty Website, APIs, notifications, mobile experiences, and webhooks have service degradation for a subset of our customer base
17:00 – Secondary DNS provider nameservers published for pagerduty.com for 8 total nameservers
17:16 – 2 of 4 Primary DNS provider nameservers removed from pagerduty.com
17:17 – PagerDuty internal servers are unable to communicate with each other due to primary DNS service being unavailable
17:18 – PagerDuty Website, APIs, notifications, mobile experiences, and webhooks are not available for all customers
17:38 – PagerDuty internal servers are able to communicate with each other by overriding local /etc/hosts files across all PagerDuty servers
17:40 – PagerDuty Website, APIs, notifications, mobile experience, and webhooks start to recover for most customers
17:57 – 3 of 4 Primary DNS provider nameservers removed from pagerduty.com
19:00 – All primary DNS provider nameservers removed from pagerduty.com and external DNS traffic shifted over to secondary DNS provider
19:04 – PagerDuty Website, APIs, notifications, mobile experiences, and webhooks start to recover for all customers
20:22 – PagerDuty Notification Acknowledgements/Resolves are restored to full functionality
Mitigation steps taken during outage
During the outage, PagerDuty engineers moved from our primary DNS provider to our secondary provider. This provided proper routing information to our customers. Issues still impacting internal servers were resolved by updating files across all of our servers to bypass DNS.
Immediate post-outage actions taken
After the outage and initial recovery, PagerDuty engineers took actions in order to clear through our backlog of events and notifications. During the outage, some redundant notifications were sent. To address this, our engineers removed redundant notifications. For approximately two hours after we were able to restore service, we were focused on dispatching notifications to our customers.
Follow-up
Tomorrow, we will be publishing actions that we will be taking over the coming days and weeks to harden our DNS infrastructure. While this was a unique event, we have high expectations for the level of service that we provide, and are working to enhance the reliability and availability of our systems.
In the meantime, if you have additional questions, please do not hesitate to reach out to our support team.