How Does PagerDuty Use PagerDuty?
We are frequently asked by our customers if PagerDuty uses PagerDuty. The answer to that is simple, Yes. While we could end the blog post here, we thought we would dive a bit deeper giving you an inside look to how we utilize our own service to stay available.
PagerDuty Using PagerDuty, It’s Pretty Meta
PagerDuty gives us the ability to keep our service reliable and available. A central person is scheduled to handle incidents so everyone else can relax and enjoy their time off. Our product allows us to evenly spread on-call duties across our DevOps teams to minimize alert fatigue, while ensuring that all incidents are handled properly. Just like our customers we need to be alerted when an outage occurs in our service or a threshold is breached in our infrastructure.
PagerDuty will notify us when there is a critical subcomponent change, such as disk space limits, backend services being slow, high error rates, etc. Even if these do not result in an outage we’ll use this data to analyze trends in our alerting systems to reduce false positives and make PagerDuty a better product for our customers and ourselves.
In our DevOps team, we use a one-week rotational on-call schedule to make sure there is always someone available to fix incidents as they are reported. We utilize a secondary on-call person as a back-up and third in line is our Ops team manager, just in case the other two are not available.
What If PagerDuty is Down, How Do You Get Alerts?
PagerDuty is a highly reliable service, so we don’t run into this often. However, we have redundancies in place with third party services to notify us of any incidents that may cause a PagerDuty outage.
We use a combination of custom internal monitoring tools, DataDog, New Relic, Sumologic and others which are integrated with PagerDuty to send us alerts. However, in case of a system outage, we use Wormly and Monitis to send us notifications independently of our system.
How Do Your On-Call Engineers Benefit by Using PagerDuty?
Using rotations, going on-call becomes a shared responsibility for the entire team. If an alert is missed, PagerDuty will automatically grab the attention of someone else on the team. Using the mobile app our team often acknowledges, resolves and reassigns incidents on-the-go.
Our on-call engineers also benefit from PagerDuty’s API, as we often treat out alerting scheme as code. So even when we are knee deep buried in code we can make adjustments to PagerDuty without leaving our terminals.
The flexibility of the product lets our on-call engineers interact with PagerDuty in a way that works best for them, but also keeps our team connected. This lets everyone focus when they need to be on, while lessening the weight on our shoulders when we aren’t on-call.
How Do You Set Up Your Alert Layers?
The beauty of PagerDuty is that no one has to conform to a specific alerting process. Back in the day when pagers were handed out the on-call engineer only had one notification method. With PagerDuty, everyone has their own habits and with several alerting options that adapt to their lifestyle.
However, phone call alerts are the most popular alerting method for PagerDuty on-call engineers. Phone call alerts wake us up. The team is divided in how they decide to layer their alerts. Half the team receives phone calls first to wake up and then follow-up with an SMS or Push Notification the other half of the team receives SMS or Push Notifications; then follows up with the loudest alerting method available, phone calls.
Our team is a huge fan of receiving alerts via our HipChat integration. We utilize HipChat as our internal communication tool to send notes and funny GIF images to each other during the day. Not only is HipChat great for finding out when lunch is ready, but integrating their service with PagerDuty makes it easy for the entire team to stay in the loop to know if something is goes wrong during business hours or even when they are chatting late at night.