This blog was co-authored by myself and Simon Darken. Once a year, PagerDuty’s SREs get together for a three-day, in-person offsite. With the team spread...by Dave Bresci
December 5, 2018
The on-call engineer has a critical role to play in incident management. Since on-call engineers are the first responders, they can mean the difference between a critical incident or one that is resolved quickly.
Smaller companies have few choices regarding who should be on call, but as the organization grows and incident management becomes complex and critical, it’s important to have a structured process for the on-call engineer.
Whether you’re a small business or an enterprise, you can benefit from having a clear process for selecting and equipping your on-call engineer. Here are a few guidelines.
In the first few minutes of the incident occurring, the on-call engineer needs to know the severity and extent of the incident. Based on that, they must gauge who is needed to resolve the incident and how to onboard them quickly. This requires having a working knowledge of how the system functions, so that when something breaks, they are able to identify what is normal versus what is broken.
In the case of small to mid-sized teams, the role of the on-call engineer would be rotated. This way, the load is shared and everyone is aware of how to handle incidents and don’t lose their touch. In the case of a larger team, they have the luxury of having dedicated incident managers who can initiate the first response. In either case, the primary goal of the on-call engineer is not to resolve the incident, but to sound the alarm and get the necessary resources looped in to resolve the incident.
It’s crucial to have a secondary on-call engineer ready for escalation. This means that there needs to be a schedule for rotation of roles within the team. It’s easy to set up automated rules so PagerDuty escalates to the backup engineer if there’s no response from the primary engineer.
Since there’s a lot at stake when an incident occurs, your on-call engineer needs to be a developer who can follow protocol and think on the go. They need to be aware of different strategies for point-of-care marketing and customer support. It is also useful to hand the on-call engineer a checklist or flowchart to follow when incidents occur.
Here are the steps an on-call engineer needs to take during an incident:
Choosing an on-call engineer must not be ignored or put aside. Having one with sufficient backups and a well-thought-out plan can mean efficiency when things go south. If your on-call engineer follows these basic steps, your team can spend more time creating and less time fixing.