Blueprint For An On-Call Scheduling System
The On-Call Scheduling Best Practices Series is back! In the first on-call best practices series, we covered what equipment is needed and how people want to be alerted while on-call. In the second part of the series, we covered who should be part of an on-call escalation policy.
The best way to deal with any outage scenario is to plan ahead. From maintaining predictable schedules to analyzing incident trends, follow these 5 best practices when building an on-call scheduling system to be ready for any IT fire.
1. Keep it consistent. Above all, your on-call schedule should be predictable, regular, and fair. Your team should never be surprised when they are on-call. They also need to be armed with the right tools, such as mobile phones to ensure they never miss an alert when away from their desk or home.
2. Appoint a Point Man (or Woman). Only one person should be in charge of setting up the on-call schedule—with input from the team, of course. Usually, that person should be the IT manager. Ideally, the schedule should be automated so that once it’s set up, there is no need to constantly maintain it. If changes are needed, such as times when someone is out sick, make use of overrides instead of reconfiguring the entire schedule.
3. Add More Hands on Deck. An alert policy must deal with escalations for multiple scenarios. If the primary on-call engineer doesn’t resolve the issue in X minutes, someone else needs to be notified ASAP. If the incident is a level-1 severity, more people must be alerted to resolve the problem faster. Never be afraid to over-engage.
4. Know How You’re Performing. A great team has insight into their incident management metrics and can answer the question “How many times did our team get paged in the last week?” within 30 seconds. Review data regularly to make sure that the right processes are in place to handle all system issues to decrease the overall number of incidents and their mean time to repair (MTTR). Analyze trends and tag alerts within the monitoring system, or set better alerting thresholds to reduce alert fatigue.
5. Think About Your Customers. Create a culture in which everyone understands the importance of making sure that your software is working properly for your customers. Having a well-oiled on-call process is merely a by-product of that. You need to hire the right people that care about providing a great product for your customers at all times.
Setting up an on-call system is easier said than done. Cultural change takes times, and the processes will evolve. It is important to not set this up in a black box. Get feedback from your team around what they feel is a fair on-call rotation, how they would like to be alerted, and other details. They will be more than happy to give their opinions and it will be easier to get buy-in.