What did the parent say to the on-call engineer at 3am? “At least yours doesn’t smell.” Believe it or not, lack of sleep isn’t the only similarity between managing critical infrastructure and the wonderful adventure of new parenthood. One of our own, Dave Cliffe, recently delivered a talk at Velocity Santa Clara on this very topic, after experiencing the many joys and stresses of both on-call duty and parenting. Here are a few expert tips that he wanted to pass on.
No, having twins or triplets doesn’t mean “redundancy” (they’re not replaceable)! Instead, think in terms of resources. You and your partner will serve as crutches for each other – which means that maybe, when you’re lucky, one of you will be able to snatch a few hours of precious sleep. (And if you’re a single parent, just know that we are in absolute AWE of you!)
“Share the load!”
Redundancy is important in infrastructure design, too. At PagerDuty, not only do we have an active-active architecture for our application and infrastructure, but we also use multiple contact providers for redundancy to ensure you always receive the alert you need in the method of your choice. We’re relentless about reliability. We’re rigorous about redundancy. We’re cuckoo for Cocoa Puffs! *Ahem* … sorry, too many kid’s commercials.
Which of yoursystems can you not afford to lose? Think in terms of highest priority and build redundancy that way.
Seek Out a Mentor
Dave named a couple of baby books that were recommended to him by his mentor that helped him navigate the first few months of parenthood (they’re Brain Rules for Baby and Raising Resilient Children in case you wanted to buy them). Mentors are also helpful in the engineering world.
Most developers who joined Sumo Logic were never on-call before. To help these new developers experience on-call life without feeling like they are thrown into the deep end, these junior engineers shared on-call responsibility with a veteran to help them get up to speed. This not only helps them to understand their systems more holistically but also acclimate to their on-call culture.
Practice shadowing to learn from senior colleagues how the critical systems in your organization work, and establish crisis-mode roles before things go belly-up. But don’t shadow other parents: that’s creepy.
Be Smart About Alerts
A screaming baby or an outage notification demands immediate attention; you know not to ignore those types of alerts.
But do you need to take action on each and every alert? If you receive an alert that your network is lagging, but you’re in charge of datastores, your services may not be required. Similarly, paying attention to how your baby cries will help you determine what’s going on – is s/he hungry, in need of a diaper change, teething, or do they just want attention?
We haven’t yet developed a BabyDuty platform for deciphering cries, but PagerDuty does let you customize the kind of alerts you receive, as well as determine how you like to be alerted. As children get older, they develop the ability to communicate what they want or need – yet with PagerDuty, this functionality is available right now. You can skip right over the Terrible Twos!
Dave concluded his talk with two suggestions: apply your on-call learning to parenting, and be sure to step back and take moments to admire what you’ve created.
“Enjoy your kids. They’re wonderful, and so is on-call duty.”