Monitoring Business Metrics and Refining Outage Response
The case for monitoring business metrics
Whether your server’s CPU is pegged at 100% or someone is chopping down your rainforest, PagerDuty has no opinions on how you use our platform to trigger a response from your on-call team. But here’s one area where we do have a strong opinion: alerting on business metrics. You should do it.
Now, what do we mean by “business metrics”? As monitoring tools have made it easier and easier to collect operational metrics like disk utilization, request latency, and so forth, it’s become even simpler to configure alerting to PagerDuty when those metrics start to go awry.
Hitting the Big Red Button
Fundamentally, we see those metrics as indicators of a larger, business-impacting outage. Understanding what your metrics are telling you before a larger outage occurs is absolutely crucial, and it is often complex and difficult to express. In fact, it’s something that many seasoned NOC personnel just develop as a sort of ‘sixth sense’. (Interestingly, it’s also something that can’t be effectively trained. Kathy Sierra’s book Badass: Making Users Awesome talks about the concept of ‘perceptual knowledge’ and how the brain is able to learn much better based on practicing pattern recognition than by attempting to verbalize training, using examples of chick sexing and flight training. This appears to apply in the same way given the combinations of operational metrics that indicate an outage.) Regardless of whether you use a NOC or a distributed on-call team, there is an investigation and triage decision made (presumably) by a human that results in an urgent, coordinated response. At PagerDuty, we call this “hitting the Big Red Button.” It works. But it always requires human intervention in order to confirm a potentially widespread issue.
Get ahead of the outage by monitoring business metrics in real-time
What’s simpler? Start monitoring your business metrics in real-time. Your CFO, Business Analysts, and even Product Managers are already looking at this data on a regular basis, maybe even daily. The key is to operationalize that data. Maybe you’re an e-commerce company that relies heavily on a shopping cart that typically sees thousands of items in it across your entire customer base during the work day. What happens if the shopping cart suddenly shows zeros across the board? Hint: something is wrong, and you need to get everyone working on it ASAP. It’s what the most effective companies do. Amazon sounds the alarm when there’s a discernible drop in orders per second. Netflix monitors stream starts per second. Unexpected changes in these important metrics triggers a widespread investigation and emergency response.
How we monitor business metrics in house
At PagerDuty, we live by a code of reliability: we need to be up more than our providers, more than the data centers we’re hosted on, and more than you. Our SLA is sacred to us, and at the heart of that is our event ingestion and alerting pipeline. We’ve engineered our system such that any slowdown in our pipeline alerts ten people simultaneously and immediately triggers an urgent, critical response. No human triage step needed. We know we need the emergency response team right away because our business metrics have indicated that something is off.
Engineering means business
As engineers, we should always understand how we’re bringing value to the business. You are more than just “keeping the lights on” for your organization, especially as it grows, scales, and finds new ways to delight customers. It’s more than just “server uptime.” Change your perspective to a business-minded, customer-first approach to monitoring. To adopt it yourselves, determine the metrics that reflect YOUR business, monitor them in real-time, learn how to detect anomalies, and trigger an appropriate response when something is out of whack.
Remember: 100% CPU could be a terrible thing (a precursor to an outage) or a great thing (maximum resource usage). You won’t know unless you understand how your customers and your business are impacted.
