This blog was co-authored by myself and Simon Darken. Once a year, PagerDuty’s SREs get together for a three-day, in-person offsite. With the team spread...by Dave Bresci
December 5, 2018
Here are some words you hear a lot in DevOps: If you’re not failing at least some of the time, you’re not moving fast enough. Teams that don’t fear failure are better provisioned to take bigger risks and achieve what’s never been done.
Many companies have mission statements that say they encourage experimentation and embrace failure. It’s part of the culture behind the movement towards blameless post-mortems. However, there’s a great gulf between accepting the risk of failure rationally and dealing with it emotionally.
Here’s a bit of reasoned advice on how to have a better working relationship with productive failures in DevOps.
Make it clear to your team that failure is not always a bad thing, and they won’t get fired if a project fails. In fact, failure is integral to innovation: Nearly every startup in Silicon Valley has a sign on the wall encouraging risk. The key is to separate failure from blame and learn not to take it personally. Taking risks helps your company stay innovative and failures are par for the course.
Fear of failing is hard-wired in most of us. You can’t just ignore it, because it’s wrapped up inside many other personal and cultural issues. Take a day or two to deal with the emotions, but don’t let it continue past 48 hours. In the meantime, compartmentalize the failure and don’t let it damage your performance in other areas. Smart companies budget for error and incorporate failure recovery into their normal feedback loops.
It may help to remember that everybody fails. Redirect your energy and turn your attention to the tasks in front of you. People tend to lash out at teammates and look for others to blame. Prepare for that reaction in yourself. Recognize it and shut it down before you compromise team cohesion. Most of all, accept that failure is a part of growth and that growth is always painful.
The pro-failure, pro-risk DevOps environment is all about understanding, accepting and growing from mistakes as well as projects that succeed but don’t push the envelope. The key is to prioritize improving, learning and moving on. Success is waiting on the far side of failure.