This is a guest post by Ilan Rabinovitch, Director of Product Management at Datadog. The convergence of rapid feature development, automation, continuous delivery, and the shifting...by Ilan Rabinovitch
August 24, 2017
Let’s say a staging environment is set up to closely mirror production environment in terms of configuration, data, etc. It’s a very important part of the deployment pipeline and effectively business-critical during the work day for all engineers who need it. It’s constantly monitored to ensure that it’s working properly, mirroring production as best it can, and available for deploys.
Then something goes wrong, and you start seeing “CRITICAL” or “ERROR” all over the place. Oh… I forgot to mention that it’s 3am where you live. Is it really “critical” in that moment? Well, technically it is. The environment is still busted. But do you want to fix it now? Is it urgent?
PagerDuty’s incident response model was originally based on the assumption that there is an issue and you need someone NOW: we’ll call the primary on-call repeatedly, we’ll escalate if they are unavailable or indisposed, but come hell or high water: we will get someone immediately and reliably – guaranteed. That’s how we give you the peace-of-mind to sleep at night. If it’s critical, you can count on us: we’re gonna getcha getcha getcha.
But what does “critical” mean? Who defines it? Some of our customers use the ITIL Framework to define incident priority. (In fact, it used to be called “incident severity” until ITIL v3 came around.) But the majority of our customers look to their monitoring tools to define a severity/priority: what threshold has been breached? How many times has it failed in the past 60 seconds? Is it mostly dead or all dead?
It came down to asking one main question: do I need to do something about this right now? Or paraphrasing: is it urgent?
Having a range of alert responses is incredibly beneficial, because it often means you have advanced notice about something starting to trend “badly” before it actually gets “bad.” Just because something isn’t urgent doesn’t mean it isn’t important, and having all of your incidents, big and small, in PagerDuty gives you valuable analytics about trouble that could be brewing.
Unfortunately, some of our customers have found that turning every Warning into a PagerDuty incident can be a recipe for sleepless nights, miserable on-calls, and burnout. So now, with Incident Urgencies, you can avoid being woken up for things you know don’t need urgent attention, and you can figure out what’s important now and what can wait.
If you’re still getting woken up for stuff that isn’t urgent, use the following runbook:
And as always, send us some feedback at email@example.com! Any day, any time. Just don’t be upset if it doesn’t wake us up!