Postmortems Part 3: Getting the Most out of Your Postmortem Meetings
When we announced the launch of our Postmortem Guide, I wrote about the value of performing blameless postmortems and how to establish a culture of continuous learning. In this final installment of our blog series on postmortems, I share how to have effective postmortem meetings.
In the last postmortem meeting I attended, we spoke calmly about what happened with our systems, how we responded, what improvements we planned to make to prevent the problem from happening again, and how to improve the way we respond to incidents in the future. Because we have a strong culture of blamelessness at PagerDuty, no fingers were pointed and no feelings were hurt. But with continuous improvement in mind, I still wondered if we were using this meeting time most effectively.
Much writing has been done on how to best analyze incidents. Talk about “doing postmortems” tends to focus on the process of analysis and the format of the resulting document. Often, the process will conclude with a postmortem meeting to discuss the results of the analysis. We recently launched our comprehensive Postmortem Guide to offer our community a detailed resource on why postmortems are important and how to adopt these practices. When conducting research for this guide, however, I noticed little writing had been done on the postmortem meeting itself.
Performing the analysis and completing the postmortem document are essential first steps to learning from an incident. Teams can get even more value out of their analysis by following up with a meeting to discuss the results. The goal of the postmortem meeting is to deepen understanding of incident causes and get buy-in for action items so that they actually get done.
Get Together & Be Cool
Getting in a room together (physical or virtual) to have a live conversation about what happened takes the analysis in new directions and deepens learning points for everyone. Most importantly, discussing the postmortem analysis helps get buy-in from team leadership to prioritize the action items needed to prevent the same problem from recurring.
This is all easier said than done. Teams fail to get the most value out of these meetings if all they do is read through the written postmortem together. These meetings can be particularly challenging when emotions run high. Even when teams make a best effort to conduct a blameless analysis, people may still get defensive when discussing a failure they contributed to.
As a former scrum master, I‘ve noticed similarities between postmortem meetings and retrospectives. In a retrospective meeting, teams consider how their last iteration went and how they want to improve. Similarly, postmortems are used to consider a past incident and how to learn from it. When a team is struggling, retrospective meetings can uncover personal friction and emotional responses. Postmortems can also involve challenging emotions such as fear and anger.
Enter the Facilitator
There is one key difference between how we run postmortem meetings and retrospectives at PagerDuty: Retrospectives are facilitated by the team’s agile coach, whereas postmortem meetings are typically led by the Incident Commander or postmortem owner.
Considering the similarities between these two meetings, I realized postmortem meetings would also benefit from the participation of a skilled facilitator. The facilitator’s role in the meeting is different from the other participants. They do not voice their own ideas, but keep the discussion on track and encourage the group to speak up. We learned that when a responder is facilitating the postmortem meeting, it is difficult for them to share their experience and ideas while keeping the entire group engaged.
A designated facilitator helps the group stay focused on achieving the two main goals of the meeting: 1) deepen understanding of what led to the incident, and 2) get buy-in for the postmortem action items. While the team is contributing their perspectives to this discussion, the facilitator stays tuned in to group dynamics to make sure everyone feels comfortable sharing and no one individual dominates the meeting. Let’s get into how exactly the facilitator does this.
Discussing Incident Causes
There are two main challenges to a productive discussion of incident causes that the facilitator helps mitigate: 1) the natural tendency to blame individuals for system failure, and 2) not digging deep enough in the analysis.
The facilitator can help the team avoid blame by redirecting the conversation if and when they observe blame. The goal of the postmortem is to understand what systemic factors led to the incident and identify actions that can prevent this kind of failure from recurring. A blameless postmortem stays focused on how a mistake was made instead of who made the mistake.
The facilitator looks out for and avoids “who” or “why” questions, which limit analysis and imply blame. Instead, ask “what” and “how” questions, such as “What did you think was happening?” and “How did you get the help you needed?”
Avoid blame when inquiring about a human action by abstracting to an non-specific responder. Remind the team anyone could have made the same mistake. Ask “What could have led any responder to take that action?” This opens it up for anyone in the group to contribute more suggestions as to what could have contributed to the system failure, taking the attention off a single individual who may feel blamed.
Even when you have a strong culture of blamelessness, it can be challenging to deeply understand the numerous conditions that led to the incident. The facilitator’s role is to ask the right questions to get the group thinking. Always ask open-ended questions instead of ones that can be answered with simply “yes” or “no.” See our analysis questions in the PagerDuty Postmortem Guide for conversation starters.
Getting Buy-In
The most important outcome of the postmortem meeting is buy-in for the action plan. This is an opportunity to discuss proposed action items, brainstorm other options, and gain consensus among team leadership. You can get a group to commit to a goal by including them in the discussion. It is the facilitator’s role to get everyone to contribute.
To encourage team leaders that prioritize work—such as product managers or engineering managers—to attend the postmortem meeting, explain that it will give them a better view of threats to the customer experience and needs for technical investment. The postmortem meeting is a time to discuss the difficult decisions on work that will and will not be done, as well as the expected implications of those choices.
Awkward team dynamics can prevent everyone in the room from contributing. The facilitator looks out for these patterns to keep the group on track. If one person is dominating the meeting, reiterate what the person is saying (e.g., “I’m hearing you say…”) so they feel heard, but then say, “I’d like to hear if there are any other perspectives,” to open it up for others to speak.
If you notice people are being interrupted by someone who is dominating the discussion, you can say “I wasn’t able to hear what the first person was saying,” or “Hold that thought—I want to make sure Mari has a chance to finish.”
The flip side of this problem is when you have people who are not speaking up. You can encourage them to contribute by asking, “What else might we need to consider?”
With these conversation tricks, the facilitator can engage everyone in the room and, most importantly, drive the conversation toward consensus on the action plan. Even if there is some disagreement about the best course of action, you can get the group to buy in to the plan if everyone feels heard.
Facilitate for Success
Get the most out of your postmortem analysis by following up with a meeting to discuss the results as a team. Having a live conversation deepens everyone’s learning and can lead to new insights. However, simply scheduling that meeting and reading the document together isn’t enough. For a successful postmortem meeting, enlist the help of a skilled facilitator.
For more facilitation tips and details on how to perform effective postmortems, check out our Postmortem Guide. We’d love to hear about how you get the most value out of your postmortem meetings on the forums.