Eliminate Alert Fatigue with PagerDuty and Event Enrichment
Too many alerts got you down? There’s a better way
This is a guest post written by Ophir Ronen, founder of Event Enrichment.
We, as IT professionals, have ever-expanding access to more accurate Ops telemetry. With this data, we have an incredible amount of visibility into what’s going on. However, more information isn’t always a good thing when it comes to alerting. You can definitely have too many alerts, and alert fatigue is a growing problem among Operations teams. More detailed telemetry isn’t bad; it’s just that much of this information is generally better suited for forensics rather than alerting.
Enter The Event Enrichment Platform (EEP). We’re teaming up with PagerDuty to help you better manage your alerts. Using PagerDuty and the EEP together, you can ensure that you are notified only for actionable alerts, and that you have all of the required information to resolve them quickly. With the EEP, you can classify alerts as actionable or non-actionable, and suppress the non-actionable ones. You can also add “enrichments”, which are specific resolution steps, so that anyone responding to the incident has the information on hand. With our new PagerDuty integration, you can ensure your critical alerts are noticed every time.
Classifications Suppress Noisy Alerts
Classifications determine if an alert is actionable, or should be suppressed as noise. The EEP receives the full flow of alerts from all of your Operations Management Systems, like Nagios, Pingdom, Zenoss, etc. and converts them into our common base event format. Those alerts, which now have a common structure, are then evaluated against any existing classifications and enrichments. The screenshot below illustrates examples of EEP noise suppression classifications.
One of our customers, supporting just over 300 nodes (a heterogeneous mix of Windows, Linux, load balancers, firewalls, and networking equipment), was able to cut their alerts by 68%. They configured 37 suppression classifications for non-actionable alerts and 17 enrichment classifications for alerts that require remediation. With that level of suppression, they enjoy a dramatically reduced event flow, enriched with context relevant event remediation information.
Enrichments Simplify Remediation
Enrichments are the specific steps needed to remediate or troubleshoot the problem. With the EEP, they are embedded in the alert for immediate access by your incident responders. For example, let’s assume that we are receiving a Windows “not enough storage is available to process this command” alert. The on-call or NOC engineer would need to know what can be deleted in order to free up storage. It may be that the information is already in the Operations Wiki and readily at hand, or it may not.
The following is an example of an enriched EEP event:
How the PagerDuty Integration Works
With our new PagerDuty integration, you can ensure all of your actionable alerts are noticed. You can route enriched alerts to EEP notifiers to send to specific PagerDuty services. The enrichment steps will show in the incident, and a link back to the incident in the EEP is also included. Clicking “acknowledge” or “resolve” for an alert in EEP automatically performs that action in PagerDuty. You can find step-by-step instructions for configuring the integration in our Integration Guide, and the screenshot below shows an example of an enriched EEP incident in PagerDuty.
Use a Weekly Process to Keep Alerts Clean
PagerDuty has blogged before about the 7 steps you can take to relieve alert fatigue, and one of these is reviewing alerts regularly. This weekly clean-up is even easier with Event Enrichment. From the EEP, you can download a list of all of your recent incidents. Then, in a meeting with the owners of Ops related domain knowledge (e.g. DBAs, Neteng, Syseng, Dev), review the alerts and assign them to one of two categories: suppress or actionable. In the future, new alerts that match suppressed alerts won’t wake you up. After the first month, you would probably only need to tack on an extra 10 minutes to your regularly scheduled Ops meetings in order to review any new unclassified alerts.
By using the EEP, our customers have been able to dramatically cut the number of alerts they receive and streamline their response processes. Many customers report that within a couple of months of using the EEP, they’ve been able to cut their alerts to just a few per day, and on some days to none at all. We built the EEP after years of working in IT Operations and feeling the pain of overwhelming noise combined with lack of remediation information. We’re excited for you to benefit from EEP as well, so start your 30-day free trial today.
