Turn any signal into insight and action. See how PagerDuty Digital Operations Management Platform integrates machine data and human intelligence to improve visibility and agility across organizations.
Connect insights to real-time action by aligning teams through the shared language of business impact.
Check out the latest products we’ve been working on—including event intelligence, machine learning, response automation, on-call, analytics, operations health management, integrations, and more.
Digital Operations Management arms organizations with the insights needed to turn data into opportunity across every operational use case, from DevOps, ITOps, Security, Support, and beyond.
Over 300 Integrations
Discover DevOps best practices with our library of webinars, whitepapers, reports, and much more.
Learn best practices and get support help with resources from our award-winning support team.
See how PagerDuty works with our live product demo — twice a week, every week.
Join live and on-demand webinars for product deep dives, industry trends, configuration training, and use case-specific best practices.
Interactive, simple-to-use API and technical documentation enables users to easily try updates and extend PagerDuty.
Engage with users and PagerDuty experts from our global community of 200k+ users. Become a member, connect, and share insights for success.
Get all your PagerDuty-related questions answered by exploring our in-depth support documentation and community forums.
“I need to be notified if there’s a significant event ongoing with SignalFx.” This is what I tell my team. However, despite being the CTO...
PagerDuty helps organizations transform their digital operations. Learn more about PagerDuty's mission and what we do.
Meet our experienced and passionate executive team.
We are risk-taking innovators dedicated to delivering amazing products and delighting customers. Join us and do the best work of your career.
With the PagerDuty Foundation, we are committed to doing our part in giving back to the community.
On Dec 11th, PagerDuty suffered an outage which affected a subset of customers and blocked access to all pagerduty.com addresses. First off, we deeply apologize for this. Any outage, no matter how many customers were affected, is unacceptable. The root cause of the outage can be traced to a problem with our DNS infrastructure, specifically, DNSSEC. This post details what happened and improvements we will make to prevent this from happening in the future.
Background on DNSSEC at PagerDuty
Back in June, we enabled Domain Name System Security Extensions (DNSSEC) for all PagerDuty domains. We did this in order to give customers the ability to validate DNS records received from PagerDuty in a secure manner. Part of the signing process for DNSSEC involves using Zone Signing Keys (ZSK) to sign Resource Records (RR), and the resulting signature (RRSIG) is then automatically deployed by our DNS provider. New RRSIG’s are generated every three months and are regularly rotated and deployed. In order to make sure that there is always a valid RRSIG, both the new and the existing RRSIG’s are deployed along side each other to make sure there is some overlap. DNSSEC has been in use since 2005, with most major DNS providers have implemented in the past five years. Earlier this year, both Comcast and Google DNS started enforcing DNSSEC. This means that if a DNS record has DNSSEC enabled for it, but the RRSIG cannot be validated, the DNS request will return as a failure in order to protect the requestor from potentially manipulated DNS data. That is what occurred on the evening of Dec 11, 2013.
Timeline of Events (all times are in PST):
During the outage, any customer that was using a DNS provider which enforces DNSSEC could not access any pagerduty.com address. This means a small subset of our customers were not able to use the website, our API or our mobile apps. We advised affected customers to use another DNS provider to get around this problem. Since we were not receiving traffic from affected customers, it is difficult to calculate what the impact was. Our estimate, based on looking at our median load balancer traffic levels, is that less than 2% of our total request volume was affected.
What We Learned
When we originally enabled DNSSEC, we set up monitoring around our DNS records. However, the monitoring that was set up had a networking bias instead of a security bias. So even though our primary external monitoring provider detected that Google DNS was returning invalid results, we didn’t receive appropriate alarms for it. If we had been looking at the expiration times on the RRSIG’s, we could have caught this problem much earlier. Even though a secondary DNS provider would not have solved this specific problem, we need to have additional redundancy to protect ourselves from DNS outages. Lastly, the reason that this outage was not more visible was because not all DNS providers are enforcing DNSSEC. Both Google and Comcast have taken steps to help improve the security around DNS and we hope to see more DNS providers do the same.
What We Are Doing About This
While the root cause of this outage was not something that we had direct control over, there are improvements that we will be making in order to prevent this from happening again and catch the problem sooner.
We are continuously improving our infrastructure to provide our customers with the most reliable service. If you have any questions or comments, please let us know.
“Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in...
We’re excited to share that we’re open-sourcing the tool we use to gather and transform the metrics from our managed DNS providers. We use DNSmetrics...
600 Townsend St., #200
San Francisco, CA 94103
905 King Street West, Suite 600
Toronto, ON, M6K 3G9, Canada
1416 NW 46th St., St. 301
Seattle, WA 98107
5 Martin Place
1 Fore St,
London EC2Y 9DT
© 2009 - 2018