Turn any signal into insight and action. See how PagerDuty Digital Operations Management Platform integrates machine data and human intelligence to improve visibility and agility across organizations.
Connect insights to real-time action by aligning teams through the shared language of business impact.
Check out the latest products we’ve been working on—including event intelligence, machine learning, response automation, on-call, analytics, operations health management, integrations, and more.
Digital Operations Management arms organizations with the insights needed to turn data into opportunity across every operational use case, from DevOps, ITOps, Security, Support, and beyond.
Over 300 Integrations
Discover DevOps best practices with our library of webinars, whitepapers, reports, and much more.
Learn best practices and get support help with resources from our award-winning support team.
See how PagerDuty works with our live product demo — twice a week, every week.
Join live and on-demand webinars for product deep dives, industry trends, configuration training, and use case-specific best practices.
Interactive, simple-to-use API and technical documentation enables users to easily try updates and extend PagerDuty.
Engage with users and PagerDuty experts from our global community of 200k+ users. Become a member, connect, and share insights for success.
Get all your PagerDuty-related questions answered by exploring our in-depth support documentation and community forums.
We just held our annual conference, PagerDuty Summit 2018, where we shared new product announcements and demoed new capabilities. But while we always have big...
PagerDuty helps organizations transform their digital operations. Learn more about PagerDuty's mission and what we do.
Meet our experienced and passionate executive team.
We are risk-taking innovators dedicated to delivering amazing products and delighting customers. Join us and do the best work of your career.
With the PagerDuty Foundation, we are committed to doing our part in giving back to the community.
As you may already know, PagerDuty suffered an outage of 30 minutes yesterday, followed by a period of increased alert delivery times. We’re taking the downtime very seriously, especially considering that it overlapped with downtime many of our customers were facing.
Please understand that we aren’t trying to shift the blame to any other parties, but part of our processes involves understanding any serious downtime and coping with it openly.
PagerDuty is presently hosted on Amazon Web Services (AWS) Elastic Computing Cluster (EC2). One of AWS’s most attractive features are “Availability Zones”. These are “distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region”.
Like many high availability applications, PagerDuty uses multiple Availability Zones to protect our application from data center level failures. AWS’s high speed inter-AZ networks allow us to synchronously replicate every event, notification, and incident we process to at least two physically separate locations. Under normal circumstances, in the event of an AZ (i.e. data center) wide failure, we are able to redirect all traffic to one of the surviving AZs within 60 seconds with absolutely no loss of incoming events.
Unfortunately, yesterday the system did not perform as designed. While we’re looking forward to reading AWS’s official post mortem, our own investigation indicates that at least three nominally independent AZs in US-East-1 all simultaneously dropped from the Internet for 30 minutes. This left us with no hardware to accept incoming events, nor to dispatch notifications for events we’d already received.
The region wide failure of EC2 impacted a large fraction of our customers. Once connectivity was restored, we received an extremely high load of incoming events and emails, and our (only semi-recovered) infrastructure was not able to process the backlog quickly enough. The load also exposed some performance-related issues within our notification dispatch system. In the future, our load testing framework will test a scenario where we are hit with a similar level and distribution of traffic.
We strive to ensure that PagerDuty delivers every alert within 3 minutes of its scheduled delivery date. A system-wide outage of 30 minutes is obviously completely unacceptable. We’ve already taken the following steps to ensure that a similar region-wide event won’t cause an extended outage:
Over the coming months, we are planning on making a number of additional improvements to our infrastructure. These changes will further decrease the chance of a system-wide outage.
Another problem uncovered by yesterday’s outage is that we had no effective way to alert our customers that there was a gap in PagerDuty’s coverage. While we obviously intend to prevent such a gap from ever occurring again, we believe it’s important to plan for all eventualities.
To that end, we’ve created a Twitter account where we will only announce PagerDuty downtime. By subscribing your cell phone to this Twitter feed, you’ll be alerted any time there is a gap in your PagerDuty coverage. To learn more about how to set this up, please see our previous blog post.
In addition, we intend to create a custom facility where users can subscribe to receive phone alerts if PagerDuty experiences another system-wide outage. Naturally, it is our intention to never need to use this system. However, we want to ensure that we have a way to rapidly notify interested customers of any gap in their PagerDuty coverage. Obviously, we’ll ensure that this emergency system shares no dependencies with our main notification dispatch service.
Needless to say, we’re sorry for letting you all down. We’ve already taken several steps to ensure this won’t happen again, and we will be taking several more in the upcoming weeks.
Please don’t hesitate to contact us if you have any questions or concerns. We look forward to earning back your trust.
“Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in...
We’re excited to share that we’re open-sourcing the tool we use to gather and transform the metrics from our managed DNS providers. We use DNSmetrics...
600 Townsend St., #200
San Francisco, CA 94103
905 King Street West, Suite 600
Toronto, ON, M6K 3G9, Canada
1416 NW 46th St., St. 301
Seattle, WA 98107
5 Martin Place
1 Fore St,
London EC2Y 9DT
© 2009 - 2018