Please Stop My Monitoring Alert Noise

by Vivian Au March 6, 2014 | 3 min read

We get it. You hate getting alerts. As Jason Floyd, Senior DevOps Manager at Real Networks put it, “I love you and I hate you. PagerDuty makes my job easier and wakes me up at 4 AM.” You are (kind of) ok with getting woken up at 2 AM if your server is really on fire. But if it’s for a minor issue that doesn’t affect your end user experience, you want your sleep. You want your alerts to be smarter and to only bother you when you really need it. Below are 4 monitoring alerting pains we commonly hear and how PagerDuty helps.

I am woken up for unimportant issues

Alert_noise_wolfYour monitoring tools’ alerts are the digital version of the boy who cried wolf. After getting bombarded with low severity alerts, when the big one hits, you treat it as another false notification. Noisy alerts make you feel sluggish and not your usual nimble self. With PagerDuty, you can filter alerts, similar to how your inbox filters, based upon subject, body and from address. Only get alerted when you need it.

I am getting multiple alerts for the same issue

Getting alerted at 2 AM is hard enough without getting alerted throughout the night for the same issue you already acknowledged. When multiple alerts for the same issue comes in around the same time, PagerDuty automatically de-duplicates the alert so only one incident is created. As long as the incident is open, any alerts for the same incident will be appended to the original incident and no new alerts will be fired off. You will have a log of all the alerts for reporting purposes, but you won’t have to suffer from alert fatigue.

I am getting alerted for issues I’m not responsible for

alert_routing_issuesGetting mail from tenants who used to live in your unit is similar to getting alerts for something you’re not responsible for. You just don’t want them. You feel guilty for ignoring them but you’re not really sure what to do with it, so you can either let it sit or try to send it back. If you’re the owner of the urgent mail, you want to get it immediately. To decrease confusion and delays, PagerDuty routes alerts directly to the person responsible for it and you can re-assign issues with a click. No more awkward exchanges.

I am getting many alerts at the same time and can’t keep track

During a large outage, many alerts are sent from multiple tools. Your phone is screaming at you and you’re ready throw it against the wall. Avoid breaking your phone with PagerDuty’s bundling of alerts. PagerDuty groups alerts occurring at the same time into one alert. But each unique alert will be logged individually so you can manage each incident at a time. Additionally, instead of logging into multiple tools to view incidents, get a complete view and track all your incidents in a single window with PagerDuty. Split screens causes split minds, and you need to be sharp when fighting your IT fires.