Turn any signal into insight and action. See how PagerDuty Digital Operations Management Platform integrates machine data and human intelligence to improve visibility and agility across organizations.
Connect insights to real-time action by aligning teams through the shared language of business impact.
Check out the latest products we’ve been working on—including event intelligence, machine learning, response automation, on-call, analytics, operations health management, integrations, and more.
Digital Operations Management arms organizations with the insights needed to turn data into opportunity across every operational use case, from DevOps, ITOps, Security, Support, and beyond.
Over 300 Integrations
Discover DevOps best practices with our library of webinars, whitepapers, reports, and much more.
Learn best practices and get support help with resources from our award-winning support team.
See how PagerDuty works with our live product demo — twice a week, every week.
We've created a maturity model to assist on the journey to digital operations excellence. Take our short assessment to find out where your team falls!
Interactive, simple-to-use API and technical documentation enables users to easily try updates and extend PagerDuty.
Engage with users and PagerDuty experts from our global community of 200k+ users. Become a member, connect, and share insights for success.
Get all your PagerDuty-related questions answered by exploring our in-depth support documentation and community forums.
I love writing software, but I hate dealing with bugs. They take you away from what you want to be doing and often lead you...
PagerDuty helps organizations transform their digital operations. Learn more about PagerDuty's mission and what we do.
Meet our experienced and passionate executive team.
We are risk-taking innovators dedicated to delivering amazing products and delighting customers. Join us and do the best work of your career.
With the PagerDuty Foundation, we are committed to doing our part in giving back to the community.
On Thursday, June 14, starting at 8:44pm Pacific time, PagerDuty suffered a serious outage. The application experienced 30 minutes of downtime, followed by a period of high load. We take the reliability of our systems very seriously; it’s our number one priority. We’ve written this post-mortem to give you full disclosure on what happened, what went wrong, what we did right and what we’re doing to ensure this never happens again.
We also want to apologize for the outage. It shouldn’t have lasted as long as it did. Really, it shouldn’t have happened at all.
The PagerDuty infrastructure is hosted on multiple providers. At the time of this writing, our main provider is Amazon AWS and our primary system is hosted on AWS in the US-East region.
We have designed our systems from the outset to be fault tolerant across multiple Availability Zones (AZs). AZs are basically separate data centers within an AWS region. So, in the event of a single AZ failure, PagerDuty can recover quickly by flipping to a backup AZ.
AWS can fail in multiple different ways. The failure of a single AZ is the simple case, which we’ve designed for by distributing our primary stack (including both database and web servers) across multiple AZs.
Another, much more serious, AWS failure scenario is the failure of an entire region at once. Historically, this has happened a few times. PagerDuty is designed to be fault tolerant in this case as well. We’ve set up a hot backup of our full stack on a separate hosting provider (non-Amazon). As part of this, we’ve developed an emergency flip process which allows us to quickly switch from AWS to our backup provider.
Note: All times referenced below are Pacific time.
We are not happy with the fact that it took us 30 minutes to do the emergency flip to our backup stack. Also, we are not happy with how we dealt with the high load after the flip was completed. Here is the full list of issues:
All in all, we did an emergency data center flip within 30 minutes of an outage and the flip process worked. Here’s the list of things that went well:
We are migrating data centers. We are moving off of AWS US-East and into US-West. This data center migration was scheduled (well before the outage) to happen on June 19. The reason for this migration is that many of our customers (over 20%) run their applications in US-East. When the region is experiencing problems, we lose capacity and we are under heavy load from our customers. Essentially, our failures are correlated with those of our customers. So, in the short term (ie. tomorrow) we are moving to US-West. As a quick aside, the timing of this outage couldn’t have been worse: we were just 5 days away from our scheduled flip to US-West when the outage happened.
Long term, we’ve realized two main things:
As a result, we will move to a 3 hosting provider setup where the PagerDuty system is distributed across 3 data centers (3 DCs to start, 5 later on). We are also moving to a multi-node distributed fault-tolerant data store (Cassandra) which doesn’t require flips when a data center fails. This project has been started last December and we already have the notification sending component of PagerDuty running on the new stack.
Voices wield power. Staying silent is not an option. We must speak up and honor those who do. October is National Domestic Violence Awareness Month,...
“Chaos Engineering is the discipline of experimenting on a distributed system in order to build confidence in the system’s capability to withstand turbulent conditions in...
600 Townsend St., #200
San Francisco, CA 94103
905 King Street West, Suite 600
Toronto, ON, M6K 3G9, Canada
1416 NW 46th St., St. 301
Seattle, WA 98107
5 Martin Place
1 Fore St,
London EC2Y 9DT
© 2009 - 2018