We just held our annual conference, PagerDuty Summit 2018, where we shared new product announcements and demoed new capabilities. But while we always have big...by Rachel Obstler
October 18, 2018
Incident management is a key facet of supporting applications. When working on an application, we spend the vast majority of time on its release to production. This includes roadmap conversations, identifying needs and requests, and building our stories and features. Then many cycles are spent developing, testing, and on QA. The Engineering team works alongside, preparing the environment. And then the app launches, and we move on to the next app, and it is the responsibility of the Operations team to run the app in production. If this is the end of the interaction with the app, the Dev team leaves a lot of valuable feedback on improvements unaddressed and undiscovered. This is where an incident management process can be key to improving applications, and ultimately providing a better experience for your customer.
With a well-defined, and well-used incident management process, application support becomes a natural part of your organizational culture. Incidents get resolved faster, more consistently, and in a way that reflects a best practice. Poorly documented or irregular incident management can lead to multiple attempts at resolutions and constant firefighting.
Following the “I would rather someone else get up in the middle of the night and fix it” principle, an incident management process encourages cross-training both within the Dev team, and between teams. This has the side benefit of encouraging operational documentation and configuration management to be kept up-to-date, while emphasizing the importance of readability of code, and commenting.
Everyone on the Dev team should be in the escalation rotation, both as backup and primary. This drives a vested interest in communication and team camaraderie. Also, by encouraging transparency, time to resolution will be reduced because the on-call developer should already have a general sense of the application. This is further enhanced if the teams are following a microservices paradigm and containing one service to each app.
We often forget to look back on where we came from in our rush to look forward. Teams also benefit from greater diversity of thought and opinion. An incident management process can encourage this by exposing every level of the escalation path to the application. Resolving incidents helps inform more junior members of the team. They gain valuable knowledge on specific incident resolution but are also exposed to the overarching design of the application topology. Recruiting and retaining talent is important for an organization. Providing a visible path from first-tier incident response through to the Dev and Engineering teams is a valuable hiring tool.
Paired with continuous integration and continuous delivery techniques, more deployments occur more rapidly than with previous monthly or quarterly deployments. This drives incidents to reduce in volume and frequency. An artifact of this is that a bug can be fixed on a much shorter time frame, significantly curtailing the need for repetitive temporary fixes. This also drives less technical debt accumulation for the Engineering and Operations teams, creating a virtuous pathway of programmatic fixes.
Each incident tracked is the encapsulation of many things. It includes the time of several individuals for repair, the documentation created noting the resolution, and possibly the filing of a bug report. It should also inform an assessment of pain points in operating the application. This can inform the application roadmap and also spur conversations on high-value, low-effort enhancements that can be implemented.
Once a team reaches a certain size, differentiation of duties will take place. This is only a natural progression of an organization, and a way to continue to scale. Tools to operate the application that were previously niceties now become imperative for the organization to sustain its growth. An incident management process can bring to light not only this need, but also where to start when creating these tools.
Incident management for applications is often a relegated component of customer support and success, but the customer sees only their portion of the application. Their experience is a narrow path through the layers of the application. The more perceptively resilient the application is, and the more quickly an incident is resolved, the sooner the customer can go back to using your amazing application.