Practitioners Share How They Remove the Fear of On-Call
Being on-call isn’t likely to be the most enjoyable aspect of a job. In fact, there might be a certain level of stress and fear around engineering teams about going on call: maybe the page will be missed, or maybe a page will come in at 2am and require troubleshooting a production issue for hours.
Even when the team follows guides to best practices (check out PagerDuty’s Full Service Ownership and Best Practices for On Call Teams), each professional can create their own mechanisms to deal with the unknowns of incident management. We asked members of the PagerDuty Community what they do to remove the fear of being on-call and also asked them to share a piece of advice for those starting out on the on-call rotation and here are some of their insightful tips!
What have you done to remove fear of going on-call at your organization?
“We push the practice of building playbooks for each team. A good playbook doesn’t need to have all the answers in it. However, a good playbook does need to contain, at least, the following: What are the services called that the team supports? Where are the services found (git/deploy zone/links)? Who on the team is the subject expert for each service? How to tell if it’s running/not running (links to monitors)? Finally, any error that was seen in the past and how it was fixed.” – John Miller, Software Engineer
“Time and experience solve this fear to some very extent. Know the responsibilities and other on calls on other departments.” – Ehsan Ebrahimian, NOC Engineer
“Empowerment to resolve and prevent the incident in the future.” – David Heinz, SRE Manager
“It is time to fix it, not run from it.” – Akhil Agarwal, Software Engineer
“We encourage taking responsibility and owning incidents for the on-call person. Of course they know the team is there to support if needed.” – Kamal Chhajed, DevOps Engineer
What piece of advice would you give to someone going on-call for the first time?
“You aren’t alone, you aren’t responsible for knowing everything, and you can do this. You’re the point of contact when contact is needed. You have the team’s knowledge (hopefully!) in the playbooks. If the answers aren’t there for you, make the call to page a subject expert in.” – John Miller, Software Engineer
“All who are on-call, once had a first time on-call 🙂 If you are working in a team, you will not be alone in the air, and somebody will shadow you.” – Ehsan Ebrahimian, NOC Engineer
“Document document document! Documentation is key. If those going on-call know what to do with every alert they receive and how to get help when they need it, that will alleviate a lot of stress.” – Nathaniel Swanson NOC Engineer
“Working with people to fine tune the monitoring/observability thresholds relevant to them. When they’ve had an active hand in determining the points at which something they care about will reach out to them, they seem to get less defensive when discussing on-call coverage.” – J.S. Observability Admin
Looking for a deep dive on real-time operations resources? PagerDuty’s Ops Guides is the next stop. These best practices are compiled from practical use, research and experience. Check out Best Practices for On Call Teams and learn what it takes to start implementing a healthier on-call culture. PagerDuty’s Dev Advocacy team updates them frequently and invites input. Real-time operations is the practice of quickly and effectively responding to digital events that require a coordinated human response.
PagerDuty University courses and certificates are also your source of thought leadership and continuous learning. Here we suggest the On-Call Readiness Reports course. See the full range of options here: https://university.pagerduty.com/.
Starting your on-call duty? Head to the PagerDuty Community Youtube playlist and get inspired by Developer Advocates’ bright talk sessions:
Big shout out to our amazing PagerDuty Community members who shared their unique experiences with us: John, Ehsan, David, Akhil, Kamal, Nathaniel, and J.S. We want to hear from you too! Join us in the Community Forums!