Triage
How to reduce noise
Good vs better vs best practices for reducing noise in PagerDuty.
When Does This Matter
When you are getting inundated with too many non-urgent alerts, it becomes hard to find what to do first, or to highlight the most urgent issue.
Why You Should Care
Reducing alert noise is crucial because it prevents alert fatigue, where teams become desensitized to notifications and may miss critical issues among the flood of false positives. By filtering out unnecessary alerts, organizations can ensure their teams focus on genuine problems, leading to faster response times and more reliable system operations.
PagerDuty Practices
PagerDuty offers teams multiple options for managing their alert volume so that they can prioritize incidents without experiencing alert fatigue.
Description of Practices
Good
Use alert keys (aka "dedup_key") to automatically deduplicate repeat alerts triggered for the same issue.
Better
Group similar alerts within or across services to consolidate them under a single high or low urgent incident.
Best
Pause alerts (either automatically or with defined thresholds) to determine when an incident should be created. Suppress alerts that are pure noise and non-actionable.