Turn on Maintenance. Go Exploring. Break Stuff.
Sometimes you just have to tinker. Experimentation, trial and error are all part and parcel of the learning experience, and the gateway to bigger and better things. Reed Richards didn’t get into the Negative Zone overnight, and he may have inadvertently released one or two would-be world conquering monstrosities along the way. But that’s OK. You invent the ray-gun that you need to solve the problem, at just the right time.
You fix it and move on.
While improving your technical situation, you may arrive at times where you just know you’re going to break something (sometimes on purpose). In the old days, this may have resulted in paging people at random times during deployments, during periods of transition, database migrations or DNS flips. You may have sent out announcements to your team saying things like, “Ignore the pages you’re about to receive for the next 30 minutes or so. I’m re-tooling some of our Chef infrastructure and I expect some false alarms.”
You don’t need to do that anymore.
You can now enable temporary, scheduled maintenance on any number of PagerDuty services. Many of you have already discovered the feature in your daily meanderings through the PagerDuty website. I salute you stalwart adventurers. I’ve seen people schedule maintenance with simple descriptions like “Deploy”, “Quarterly Maintenance”, “Kicking over RDS”, and “fdsaf” (very informative 😉 ).
I’ve also seen messages of desperation: “I’m getting paged to death and as far as I can tell nothing is wrong.”
This is all OK. This is how this feature was intended to be used. Sometimes you just don’t know what’s wrong. Sometimes you just need things to shut up so you can have some time to think. Put your whacked out service into an hour of quiet time with “1-Click Immediate Maintenance.” Figure it out. Fix it. The service will re-enable automatically.
Maintenance windows are also intended to be 100% scriptable. You can script your deploy processes to disable a set of services for the next five minutes after your deploy, while symlinks are flipping, load average is smoothing out, and anything else that just seems to be a bit too noisy for your current monitoring level to handle is doing what it does. Nobody wants to be paged for something that’s not actionable (or actionable long term, but you don’t have the budget to work on it right now).
So maintenance windows are here. Do some exploring. Make things better. Break stuff. We’ll be waiting for you when you’re done.