Turn any signal into insight and action. See how PagerDuty Digital Operations Management Platform integrates machine data and human intelligence to improve visibility and agility across organizations.
Connect insights to real-time action by aligning teams through the shared language of business impact.
Check out the latest products we’ve been working on—including event intelligence, machine learning, response automation, on-call, analytics, operations health management, integrations, and more.
Digital Operations Management arms organizations with the insights needed to turn data into opportunity across every operational use case, from DevOps, ITOps, Security, Support, and beyond.
Over 300 Integrations
Discover DevOps best practices with our library of webinars, whitepapers, reports, and much more.
Learn best practices and get support help with resources from our award-winning support team.
See how PagerDuty works with our live product demo — twice a week, every week.
We've created a maturity model to assist on the journey to digital operations excellence. Take our short assessment to find out where your team falls!
Interactive, simple-to-use API and technical documentation enables users to easily try updates and extend PagerDuty.
Engage with users and PagerDuty experts from our global community of 200k+ users. Become a member, connect, and share insights for success.
Get all your PagerDuty-related questions answered by exploring our in-depth support documentation and community forums.
In a world where everything comes down to moments of truth, teams must respond to issues and opportunities in seconds. Rising customer expectations demand real-time...
PagerDuty helps organizations transform their digital operations. Learn more about PagerDuty's mission and what we do.
Meet our experienced and passionate executive team.
We are risk-taking innovators dedicated to delivering amazing products and delighting customers. Join us and do the best work of your career.
With the PagerDuty Foundation, we are committed to doing our part in giving back to the community.
Organizations need many incident commanders to provide a high level of service to their customers while avoiding on-call load. Many shy away from becoming an incident commander because they assume only senior technical leads can be one. However, soft skills are actually more important, and with a well-defined process like the one outlined in this post, your team can train multiple people to be successful in leading and driving coordination during customer-impacting outages.
I’m a ScrumMaster at PagerDuty, not an engineer, and I recently became an Incident Commander. I learned first hand that you don’t need to be a senior engineer to serve as Incident Commander.
Effective incident response requires an Incident Commander to serve as the decision maker and to provide clear coordination. It’s a demanding role that requires a unique set of skills. As incidents can happen at any time, organizations need a sufficient number of Incident Commanders to reduce on-call load and avoid burnout. It’s therefore important to develop an Incident Commander training process that is both welcoming and effective.
While high level knowledge of how your organization’s services interact with each other is important for an Incident Commander to have, you by no means need to be highly technical to lead an incident response. The Incident Commander should remain focused on coordinating the response with key soft skills, not performing any technical remediation or investigation tasks themselves. The Incident Commander needs to actively listen to identified symptoms and proposed actions to decide on the best course of action and should not let any technical knowledge bias your chosen approach.
Anyone can be an Incident Commander, regardless of rank or technical expertise! PagerDuty’s open-sourced incident response documentation is a great starting point to formalize your own processes, but written documentation, as clear as it may be, isn’t enough to fully train a new Incident Commander. Proper training requires hands-on practice. At PagerDuty, we have developed a supportive training program that can get even the most junior team members comfortable with leading a major incident response.
Here are some of its key tenets:
Let’s contextualize these and discuss exactly how we put process, directive communication, time management, and listening into practice.
Our Incident Commanders host regular office hours open to all employees that are interested in incident response. Here, prospective Incident Commanders who have decided to begin their training can ask questions and learn about the process. This is an opportunity to explain the need for more Incident Commanders, and help everyone give it a try and learn.
If you’re interested in seeing how we train our staff to get ready for incident response, be sure to register for our free webinars:
After we kickstart the process with office hours, we then get the trainees on the Incident Commander shadow schedule. Shadowing the Incident Commander helps the trainee get a feel for what it’s like to be on-call. They also get woken up or interrupted, whenever the Incident Commander gets paged. While shadowing, the trainee joins the incident call to listen in. It is important for the trainee to remain a silent observer, holding any questions until the end, to avoid distracting from the response. Incident Commanders then spend time answering trainees’ questions to help them learn, increase their comfort, and make them feel supported by the community.
After the trainee listens to a few calls, encourage them to jump in as scribe, documenting the timeline of an incident as it progresses. We’ve learned that for longer incidents, frequent handoffs make a huge difference for maintaining an effective response and avoiding burnout. Serving as scribe is a great way for someone to start helping before they are fully ready to be an Incident Commander. Starting to get involved with the incident response in this way also helps the trainee get even more familiar with the process, increasing their confidence.
After listening to and scribing for a few incident calls, encourage the trainee to take the plunge and get on the main schedule. They can reverse shadow, leading the call with the support of a backup Incident Commander. Assure them the backup will be there to help throughout the response. It is important to let the new Incident Commander take point on the call, so they can earn credibility and further build their confidence. Privately message them with any tips and reminders, as well as with encouragement.
After a new Incident Commander leads their first incident response, celebrate them! Leading an effective Incident Response directly impacts the success of your business and the happiness of your customer’s, and is also key in maintaining the morale of the team. By creating an Incident Response community that supports each other, you’ll be able to welcome more Incident Commanders and reduce on-call load for all.
Incident Commander Training: Leading the Response During Major Incidents
Watch OnDemand »
Incident Responder Training: Best Practices for Success During Major Incidents
This blog was co-authored by myself and Simon Darken. Once a year, PagerDuty’s SREs get together for a three-day, in-person offsite. With the team spread...
In the United States, it’s almost that time of year again where we count our blessings and give thanks. For retail workers, it’s also that...
600 Townsend St., #200
San Francisco, CA 94103
905 King Street West, Suite 600
Toronto, ON, M6K 3G9, Canada
1416 NW 46th St., St. 301
Seattle, WA 98107
5 Martin Place
1 Fore St,
London EC2Y 9DT
© 2009 - 2018