This is a guest post by Ilan Rabinovitch, Director of Product Management at Datadog. The convergence of rapid feature development, automation, continuous delivery, and the shifting...by Ilan Rabinovitch
August 24, 2017
Okay, breathe. Everything is going to be okay!
In a recent survey we conducted, we learned that over 80% of you have admitted you have forgotten that you were on call. It makes sense, you aren’t in the office and out living your life. Then pow! You receive an alert that desperately requires your attention!
Here are a few real-life scenarios and what you should do!
This is the best scenario to be in when you forgot you were on-call. Sure, you forgot, but nothing bad has happened.
We recommend you keep an on-call emergency backpack ready for moments just like these. Throw your laptop in the bag that includes a MiFi, a couple energy drinks or anything else you would typically need when you are on-call. Then go out for a beer, just keep it light with a Belgian White over an IPA in case you end up needing some of those brain cells.
Oh no! You should have stayed home streamed a movie featuring a Strong Female Lead on Netflix, but you had to see to Sandra Bullock float through space in IMAX. Do you listen to the singing popcorn that tells you to shut off your phone or risk annoying everyone in the theater?
This isn’t the most ideal situation to be in. But crossing your fingers that nothing will go wrong probably isn’t the best course of action. But it’s not the end of the world. We recommend giving a call to the secondary person in your escalation policy or another teammate. Let them know that you won’t be available and ask them to schedule an override. This way incidents can be handled immediately instead of waiting until the alert moved through your escalation chain to your secondary on-call teammate.
Just apologize and offer to cover one of their shifts in the future.
Ah, an unexpected alert while I’m at home, playing videos and getting lost in the Wikipedia rabbit hole. What’s there to think about? Press pause, stop playing Six Degrees of Kevin Bacon and go fix the issue.
Okay, worst case scenario. You’re rocking out at a Deadmau5 show and a 45 minute drive from home, but you just received a critical alert.
In this case, we recommend acting fast. Don’t ignore the alert! (If you even heard or felt it) Instead, quickly escalate the incident to the secondary on-call in your escalation policy. This way they won’t need to wait for the notification rules to reach them.
Nothing you can do, you might as well keep rocking out and apologize tomorrow.