Turn any signal into insight and action. See how PagerDuty Digital Operations Management Platform integrates machine data and human intelligence to improve visibility and agility across organizations.
Connect insights to real-time action by aligning teams through the shared language of business impact.
Check out the latest products we’ve been working on—including event intelligence, machine learning, response automation, on-call, analytics, operations health management, integrations, and more.
Digital Operations Management arms organizations with the insights needed to turn data into opportunity across every operational use case, from DevOps, ITOps, Security, Support, and beyond.
Over 300 Integrations
Discover DevOps best practices with our library of webinars, whitepapers, reports, and much more.
Learn best practices and get support help with resources from our award-winning support team.
See how PagerDuty works with our live product demo — twice a week, every week.
We've created a maturity model to assist on the journey to digital operations excellence. Take our short assessment to find out where your team falls!
Interactive, simple-to-use API and technical documentation enables users to easily try updates and extend PagerDuty.
Engage with users and PagerDuty experts from our global community of 200k+ users. Become a member, connect, and share insights for success.
Get all your PagerDuty-related questions answered by exploring our in-depth support documentation and community forums.
In a world where everything comes down to moments of truth, teams must respond to issues and opportunities in seconds. Rising customer expectations demand real-time...
PagerDuty helps organizations transform their digital operations. Learn more about PagerDuty's mission and what we do.
Meet our experienced and passionate executive team.
We are risk-taking innovators dedicated to delivering amazing products and delighting customers. Join us and do the best work of your career.
With the PagerDuty Foundation, we are committed to doing our part in giving back to the community.
Having one person on-call isn’t enough. What happens if your on-call engineer sleeps through their alert? What happens if their phone’s battery dies without them knowing, or if they get an alert at a really inconvenient time, like when stuck on a bus or in traffic? It will happen.
You need a backup! One or more people, waiting in the wings, ready to spring into action if your primary on-call is criminally negligent unable to perform his or her duties to the best of their abilities at any given time.
These backups don’t need to be AS “on-call” as your primary engineer, but what they lose in readiness they make up in numbers. It often makes sense to have multiple backups.
In PagerDuty we group the currently-on-call engineer and all of his or her backups into an “escalation policy”, which sets the order in which we alert (email/phone/SMS) people, and the delays in between alerts. There are a lot of ways to organize these escalation policies, but I’ll go over some patterns used by many of our customers (both big and small).
You of course already have some sort of “primary” on-call engineer, and this glorious position is hopefully determined by a calendar that rotates between the people in your ops team in a fair way.
Many of our customers will supplement this rotation with an (unimaginatively-named) “secondary” rotation. This secondary rotation is usually setup to shadow the primary rotation by being configured identically to the primary rotation, but offset by a certain amount of time so it’s impossible to be both primary on-call and secondary on-call at the same time. For example, if your primary rotation contains “Alex, Bob, and Charlie”, then your secondary rotation can contain “Bob, Charlie, and Alex”, in that order.
Over 25% of the (many) on-call calendars we manage here at PagerDuty have either the word “primary” or “secondary” in them, so this is a very commonly-used pattern throughout our customer base.
If your company is big enough or your operations tasks plentiful enough, you may also have a separate first-tier support team that handles all the basic operations tasks before even your “primary” on-call engineer. This front-line team is usually trained in handling all of the repetitive and annoying small problems that crop up often and have clear resolution procedures. (You know, that stuff that you should seriously just fix already, but hey, you’re busy.) These first-tier support teams are often shared amongst multiple engineering teams, and are placed first in an escalation policy.
So who is placed last in the escalation policy? Who is the last-tier support team? Management, of course!
Your escalation policy shouldn’t just end with your primary or secondary ops teams, but should escalate up to their manager’s phone, and then maybe even their manager’s manager’s phone, assuming nobody acknowledges the alert in time.
This has two purposes: (1) management is ultimately responsible for these important systems and is logically who should be informed if any major problems fall through the cracks, and (2) your ops team will be less likely to “miss” an automated PagerDuty phone call if they know it just means that their boss will phoning them up personally in a few minutes and asking some very pointed questions.
So, putting this all together, a (very complete) Escalation Policy example for a hypothetical “Database Ops” team might look something like this:
With timeouts of maybe 10-30 minutes in between escalations, depending on your needs. Note that any individuals (managers) put directly in the escalation policy are essentially on-call all the time, as last lines of defense, so should try to keep their cell phones handy and charged at all times.
 They don’t need to carry around a mobile broadband device, for instance, and can feel less guilty about doing things like binge drinking while on duty occasionally partaking in activities that might slightly decrease their on-call abilities.
 The primary engineer doesn’t have to be determined by a rotation, and you can instead have some poor soul be primary on-call every minute of every day for his whole tenure at your company, but this tenureship may be short due to burnout.
This post originally appeared on the PagerDuty blog on March 8, 2012
This is a guest post by Ilan Rabinovitch, Director of Product Management at Datadog. The convergence of rapid feature development, automation, continuous delivery, and the shifting...
Dynamic Notifications are now out in the wild! With our launch today, we give PagerDuty users the power to dynamically adjust how they are notified...
600 Townsend St., #200
San Francisco, CA 94103
905 King Street West, Suite 600
Toronto, ON, M6K 3G9, Canada
1416 NW 46th St., St. 301
Seattle, WA 98107
5 Martin Place
1 Fore St,
London EC2Y 9DT
© 2009 - 2018