This blog was co-authored by myself and Simon Darken. Once a year, PagerDuty’s SREs get together for a three-day, in-person offsite. With the team spread...by Dave Bresci
December 5, 2018
The point of continuous integration is to automate builds and tests, and bring efficiency and quality to the pipeline. However, things do sometimes go wrong with the faster pace of development and more frequent updates that accompany a continuous integration process.
When major incidents or something goes wrong occur, there’s a lot of panic. That’s where incident management comes into the picture. But does it always have to be after something goes wrong? Integrating incident management from the beginning and into your continuous integration process will help take accountability, visibility, and transparency to a whole new level.
In this post, we’ll discuss how incident management brings deeper monitoring to DevOps, and how it can transform your application development.
The goal of DevOps is to facilitate collaboration between Development and Operations teams so they understand each other’s needs and don’t just point at each other when things go wrong. Uptime does not always have to be the Ops team’s burden to bear. With DevOps, even a new developer should feel responsible for uptime and should be able to chip in during downtime.
One of the big advantages of implementing continuous integration is that Dev and QA teams are also accountable for shipping quality code. Each time a new build is committed, it is automatically verified by a series of automated unit tests. If incident management is implemented at this level, when something does break, your teams are ready with the right data at hand to resolve the issue effectively. This way, they can quickly troubleshoot without panic and without having to blame anyone. Incident management automatically enforces a culture of quality and makes Dev and QA teams accountable for availability.
Like real-life emergency teams, it’s also good to have a first-response engineer, or on-call engineer, who acts first during an incident before someone with higher responsibility can arrive on scene. To enable this culture of accountability, you need monitoring and on-call management systems that respectively make monitoring data visible across teams, and divide unplanned work based on equitable shifts.
A good overview of what the entire team is working on and the progress made helps everyone focus their efforts. Many businesses let the Ops team in on any new code implementations only when things go wrong or when an incident occurs. As a result, Ops teams are sometimes blamed for holding back on changes due to mistrust, which results in slower updates.
If the Dev team is transparent with Ops about new changes even at the planning phase, they can be more open to changes and understand how changes benefit the entire business. Letting the Ops team know of new ideas, upcoming features, and possible risks even at the development phase will do wonders for the awareness of the entire team. The Ops team can rest assured that even if something breaks, the entire team is always ready and prepared.
Implementing incident management in the earlier phases helps everyone understand the health of the application and what they ought to do when issues arise. Everyone is aware of the big picture and can troubleshoot more quickly.
The more the entire team is aware of each other’s responsibilities during a crisis, the more effectively they can work and the quicker things can get back to normal.
Too often, Dev and Ops use a completely different set of metrics and monitoring tools without unifying the data into one centralized hub and trying to understand patterns, anomalies, and dependencies. A car cannot be driven without a windshield; in the same way, it’s crucial to centralize all your monitoring data to proactively and holistically give everyone a good view of what’s going on.
Collecting, correlating, and analyzing data from multiple sources gives Dev and Ops continuous insight. But that data is only valuable if it’s made actionable. With an incident management solution, you can provide an overview of the churning gears to the right people, and even empower them to zero in on the things that might eventually break your app.
Finally, make sure your incident management tools are actually helping by providing real-time notifications when an issue is simmering or hits. It’s crucial to define a process around how issues of different severities should route; while you don’t want to throw away data, you don’t want to get notified on vanity metrics that do not contribute to solving the issue at hand.
For a successful DevOps transformation, continuous integration and incident management must go hand-in-hand. This will provide huge relief across the entire team and much quicker responses to downtime. Incident management makes the DevOps engine function smoothly, without breakdowns.