Why We Use On-Call Shadowing On-call shadowing is an essential practice at PagerDuty. For a new engineer, a shadowing period serves as a kinder, smoother...by Max Timchenko
March 26, 2019
Since joining PagerDuty as an Agile Coach, I’ve come to appreciate the engineering culture that has been fostered here. To share what has worked for us (and how we plan to nurture it as we continue to grow), here are three ways to create an awesome engineering culture.
There’s a popular quote from 3M’s former president and chairman, William McKnight: “Hire good people, and leave them alone.” PagerDuty’s engineering culture is exemplary of this philosophy. Teams here are self-organizing and largely in control of their own destiny. Each team has end-to-end ownership of a piece of the PagerDuty ecosystem. Leadership provides guidance and aligns the team on a problem to solve. The team is entrusted to figure out how to solve it.
Our engineering teams follow a true DevOps “you code it, you own it” approach. This starts with a strategic blueprint that relates company goals to the product roadmap via implementation, deployment, and maintenance of their code and infrastructure in production. We didn’t always work this way, but after growing 3x in the last 3 years, we’ve learned a lot.
While all teams follow Agile principles, some choose Scrum, others Kanban, to manage their day-to-day work. As a team evolves and their work changes, they may choose a different way to get things done. Each team is supported by a Product Owner and Agile Coach. The Product Owner collaborates with the team to develop a roadmap and product backlog. The Agile Coach guides the team on Agile principles and provides support if there are impediments to the team’s ability to self-organize.
We haven’t entirely figured out team empowerment at PagerDuty, though. We are still seeking the right balance between guidance from leadership and team empowerment. Sometimes the pendulum swings too far in one direction. But we are at least aware of it when this happens and work to get the organization back into the right balance.
Each month, software engineers have a chance to build and demo something they think PagerDuty needs (or at least something they are excited about and can bring back to PagerDuty from the experience). They have complete freedom over what they work on, who they work with, and how they do it. Afterward, winning prototypes can get selected to be resourced as full-on projects, taking them from a concept to the hands of customers. Hack Days are one of many ways we encourage bottom-up innovation.
Successful projects that started as hack days include our mobile app, an internal employee directory and locator called Dutonium, and the Custom Event Transformer feature to name a few. There are also plenty of strangely useful innovations: pajamas that vibrate and light up when you are paged, a turret that fires a foam dart at the person on-call whenever there is a PagerDuty alert, or helping you ‘catch em all in Pokémon Go.
Collaboration is supported by two types of communities within engineering: Tribes and Guilds (both were first popularized by Spotify’s team model). Some teams at PagerDuty have shared customer needs and a shared purpose. These teams have grouped together and formed Tribes to enable synergy between teams. Tribes here combine recurring meetings where it makes sense. For example, a tribe of teams will hold one Tribal Review instead of three separate Team Reviews. We have seen Tribes add significant value by encouraging knowledge sharing, code/operational work sharing, and aligning teams on a set of priorities and goals rather than having each team operate in a silo.
Other communities within engineering, called Guilds, are formed by groups of like-minded people with shared interests. Guilds have proven to be extremely useful in integrating and sharing valuable knowledge, especially in areas that have been a pain point for teams. For example, The Ancient Guardians of the API is a Guild formed, “to protect the conceptual integrity of the PagerDuty APIs and spread API best practices to the Engineering organization.” We’ve also seen Guilds be helpful in adding structure to our existing recurring engineering events, such as the Chaos Guild’s support of our Failure Friday event and other chaos engineering initiatives.
Tribes and Guilds have created a more collaborative space at PagerDuty. But as teams and their work evolve, we have learned that a Tribe or Guild that once provided value may have diminishing returns. For example, if teams are no longer having the same technical pain point that the Guild was created to help solve, it may make sense to spin it down. The same rule applies for a Tribe in which a product or technical space changes make it so that the individual teams no longer have a shared goal. We find it beneficial to continually re-evaluate the value that each Tribe or Guild is providing.
To sustain this culture within engineering, there needs to be a continued focus on hiring software engineers that are the right fit (at PagerDuty, we have a strict “no ***hole” hiring practice). A healthy balance should continually be sought between team autonomy and alignment from leadership. And most importantly, things need to stay fun. These three ingredients create a recipe for an engineering culture that will continue to be awesome for years to come.