Turn any signal into insight and action. See how PagerDuty Digital Operations Management Platform integrates machine data and human intelligence to improve visibility and agility across organizations.
Connect insights to real-time action by aligning teams through the shared language of business impact.
Check out the latest products we’ve been working on—including event intelligence, machine learning, response automation, on-call, analytics, operations health management, integrations, and more.
Digital Operations Management arms organizations with the insights needed to turn data into opportunity across every operational use case, from DevOps, ITOps, Security, Support, and beyond.
Over 300 Integrations
Discover DevOps best practices with our library of webinars, whitepapers, reports, and much more.
Learn best practices and get support help with resources from our award-winning support team.
See how PagerDuty works with our live product demo — twice a week, every week.
We've created a maturity model to assist on the journey to digital operations excellence. Take our short assessment to find out where your team falls!
Interactive, simple-to-use API and technical documentation enables users to easily try updates and extend PagerDuty.
Engage with users and PagerDuty experts from our global community of 200k+ users. Become a member, connect, and share insights for success.
Get all your PagerDuty-related questions answered by exploring our in-depth support documentation and community forums.
In a world where everything comes down to moments of truth, teams must respond to issues and opportunities in seconds. Rising customer expectations demand real-time...
PagerDuty helps organizations transform their digital operations. Learn more about PagerDuty's mission and what we do.
Meet our experienced and passionate executive team.
We are risk-taking innovators dedicated to delivering amazing products and delighting customers. Join us and do the best work of your career.
With the PagerDuty Foundation, we are committed to doing our part in giving back to the community.
Guest blog post by Ron Vidal, Rob Schnepp, and Chris Hawley of Blackrock 3 Partners LLC. Blackrock 3 Partners are experts in Incident Management, combining decades of experience in the fire service, law enforcement and anti-terrorism managing large-scale public safety emergencies with decades of experience in managing web operations, critical infrastructure, capital markets and M&A activities for international broadband network operators and high performance computing companies.
It’s a quiet, sunny day, like so many other quiet, sunny Peacetime days…
Then, somebody sees flames and smoke pouring out of a building… Wartime…
Call 911… Help is on the way!
In seconds, the 911 Communications Center dispatches fire engines, ambulances, rescue companies and Incident Commanders to the location of the emergency. In about 4 minutes, the first 25 trained firefighters arrive on scene with their specialized vehicles, tools and skills. The Incident Commander establishes command, sizes-up the situation, sets tactical objectives, begins operations, evaluates resources needed and organizes the effective resolution of the emergency.
Does this sound like how your DevOps team responds to high severity incidents?
If you have Operations, then you will also have Emergency Operations. Like an alarm sounding at a fire station, immediately launching firefighters into action, DevOps teams must respond with the same level of urgency to resolve their emergency. In both cases, the clock is ticking. The problem is unlikely to get better until the right resources are dispatched and respond at the right time, working under a leader making the right decisions, all within an organizational framework.
The Shift from Peacetime to Wartime
Peacetime is the mode of operation that occurs during the normal day-to-day activities of any IT organization. Developers write code. Operations keep the infrastructure running. Business as usual. In other words, a perfect Peacetime day.
Here’s what a typical Peacetime organization chart looks like:
Wartime is the mode of operation that occurs when systems are NOT normal. Operations has declared a SEV level event and initiated an Incident conference bridge. On-call subject matter expert engineers are querying alerts and looking at performance data. Customers are out of service. Business is NOT ususal. We have a serious problem and and it needs to be fixed, right now. In other words, a Wartime Incident.
In the Fire Department, the shift from Peacetime to Wartime occurs when an emergency is reported. Wartime is different. People behave differently. Their language and method of communications is different. Conversations are typically much shorter, more direct and aimed at problem solving on a compressed timeframe. To the uninitiated it sounds abrupt. It sounds choppy. It sounds sterile. And it should.
As 60 Minutes reported in its March 17, 2013 interview with Jack Dorsey, “Young Jack was intrigued by the messages he heard coming out of the St. Louis emergency dispatch center. At home he listened to it all on a police scanner. And he was struck by the fact that everyone talked in short bursts of sound – a system of communication that later inspired him to invent Twitter.”
In Wartime, the Incident Commander is thinking faster than the emergency is unfolding. To do that, communications must be direct, crisp and clean, like “Voice Twitter”.
Understanding Wartime Communication
The Wartime organizational chart will look different than the Peacetime organizational chart. In fact, the CEO (Peacetime leader) is exactly the wrong person to lead the emergency (Wartime) response, because someone still has the run the business and the unaffected parts of the organization. Roles and responsibilities, chain of command, and the assignment of tasks in Wartime will be very different than they are in Peacetime.
Here’s what a Wartime organization looks like:
Here’s a comparison of how a Fire Department and DevOps respond to an emergency:
In short, solving Wartime problems requires a Wartime mentality, and a defined process for incident management. Fire Departments have developed a system that has been in use for over 40 years and has managed tens of millions of Wartime incidents. Without a doubt, DevOps teams are emergency response organizations much like Fire Departments.
When an emergency is reported, it is a clear signal to all that the organization has shifted from Peacetime to Wartime. All responders must understand and accept the sense of urgency and accountability that comes with the shift to Wartime and perform their assigned tasks accordingly.
Every emergency (Wartime) response absolutely depends on robust communications for the entire incident, from alert notifications to dispatch to dedicated tactical channels to escalation to resource accountability. Peacetime has a clutter of multiple communications systems that distract Wartime responders. Wartime demands clear communications for every incident and every emergency responder, and centralized incident management system will help cut through the noise. Each element of the emergency (Wartime) response depends on effective, reliable, communications. Without rock solid communications, Incident Management fails.
So, next time you see a fire engine speeding to an emergency Code 3 (lights and siren), just remember that they are in Wartime and using the same tools and systems that you can use to manage your DevOps fires.
For more information on Blackrock 3 Partners LLC, please visit www.blackrock3.com.
This blog was co-authored by myself and Simon Darken. Once a year, PagerDuty’s SREs get together for a three-day, in-person offsite. With the team spread...
At the latest PagerDuty Connect event in Toronto, DevOps expert Arthur Maltson shared a recent story about chaperoning his daughter’s school field trip to a...
600 Townsend St., #200
San Francisco, CA 94103
905 King Street West, Suite 600
Toronto, ON, M6K 3G9, Canada
1416 NW 46th St., St. 301
Seattle, WA 98107
5 Martin Place
1 Fore St,
London EC2Y 9DT
© 2009 - 2018