It Came From Below
Kelsey Shannahan is a Senior Developer at a leading healthcare IT company in Columbus, Ohio. She is passionate about Elixir, her three cats, and Eldritch Horror.
I’m going to assume most people who read this blog are familiar with PagerDuty. But just in case anyone isn’t, PagerDuty is a tool we use in IT to notify us if some predefined check has failed. Maybe a key process has died or maybe we’re not seeing our expected traffic volume or maybe our server has stopped responding to ping. Whatever it is, PagerDuty will relentlessly, remorselessly, and loudly notify whoever is on call that something needs attention.
For a while, my phone would serenade me (at full volume) with a barbershop quartet singing about how the server was on fire. However, after I got over 100 alerts in less than 15 minutes when our entire stack went down, I was a little tired of that and now I get a classic “red alert” siren. There’s nothing like sitting in an office filled with a cacophony of barbershop quartets, sirens, and sad trombones to make you regret your choice in career.
It’s even worse when it goes off when you’re sleeping. It’s like a shot of adrenaline to the heart at three in the morning. I suspect it takes a year off my life every time I’m startled awake—or startled by it at any time, really.
About a month ago, it woke me close to midnight. I hadn’t been in bed very long; just long enough to settle into a deep sleep that left me confused when I woke. I fumbled for my bedside light by instinct, squinted hard at my phone to find the “Acknowledge” button, and then put my glasses on. I sat there a moment, dazed, trying to remember where I’d left my laptop in the house.
Then I remembered: I’m not on call this week. I’m not even the backup on-call person. I shouldn’t be getting paged at all.
Maybe someone escalated to me because they specifically needed my help, I reasoned. I tried to think of what recent changes I’d deployed to production that could have caused an outage so late at night. Nothing came to mind. Regardless, I stumbled out of my bedroom and down the dark hallway to my office. The floor creaked under me—and then my phone went off again with that damn siren. I nearly dropped it before cursing and fumbling to acknowledge the new page.
I remember standing there after the phone was silenced and thinking, Huh, that’s odd. The floor continued to creak even though I was no longer moving. I reached over and turned the lights on and looked back behind me.
Nothing. The house was silent.
I continued to my office, trying to ignore the sense of unease. First thing I checked on my laptop were my messages on Slack. Nothing from my coworkers. There were still a couple people online (I work with a lot of night owls) but there wasn’t any chatter in our maintenance or outage rooms. Then I checked the PagerDuty website to review what exactly the pages were that were causing all these alerts.
I didn’t recognize either of them. The first one read: Build failed: compilation error in watchman.rb.
First off, a file ending in .rb is a Ruby file. Ruby isn’t a compiled language. It can’t throw a compilation error. Second,: we don’t alert anyone for failed CI builds. That’s just being mean.
The other error was nymaster1n2: Process STOPped. Which also makes no sense because even though we’re based in New York, we don’t have servers here. And what process is it referring to, anyway?
All our alerts are configured to link either to the monitor that’s alerting or to instructions on how to resolve the page for more complicated scenarios. I tried clicking on the link that should take me to the monitor so I could check the status, but it just directed me to a Google Maps page set to the location of my house.
At that point, I assumed this was a hoax. You can manually create an incident in PagerDuty after all, so maybe someone got a little creative and was playing a prank on me. They know I live alone. I’d be an easy target, especially since I don’t have a spouse that would get pissed off.
I resolved the fake incidents, shot a message off in Slack to my team that this wasn’t funny late at night (it is clever, but not while I’m trying to sleep), and went back to bed. A couple hours later, my phone went off again. I repeated the process of turning on the light, acknowledging the alert, and putting on my glasses. I’d also brought my laptop into the bedroom with me earlier so I could just pick it up without leaving my bed. At this point, it was 2 a.m. and none of my coworkers were online. I checked the error.
Unrecognized entity from whoami
I guess that could be legitimate, but I’m not ops. I’m a developer, and I should only be getting alerts for when our apps misbehave, not for server-level stuff. And unless I’m mistaken, whoami is a server thing.
I escalated it to our ops on-call person, slammed my laptop shut, threw myself back down on the bed, and turned the lights off. I hadn’t even fallen asleep yet when my phone went off again. But it wasn’t my chosen siren ringtone.
It was the “Engineer Laughing” one, which, let me tell you, sounds really damn creepy when you’re in a house by yourself at two in the morning.
The notification read raise StandardTerror and included a link to what looked like a log file, but it was just Google Maps again—Street View this time, zoomed in on one of my basement half-windows.
Okay, hahah, I see what you did there, nameless coworker. Standard Error. Standard Terror. Real funny. This time, I dropped the screenshot—and my ire—into our developer Slack channel.
I turned the lights off, but did not lay back down. I waited, suspicious as to whether I would be allowed to sleep. In the silence, I heard something like scratching; distant, perhaps some small animal outside, near the house.
My phone shrilled again.
Heartbeat check failed: ‘nymasterbedroom1n2’
This was getting out of hand. I considered just turning my phone off, but I’ll be honest: I actually wanted to see where this was going. Sure, maybe it’d keep me up all night, but I’d have a hell of a story to tell and I could always sleep in late tomorrow and blame PagerDuty for the late start.
In the end, I kept my lights off to keep the glare off my phone screen. Then I waited, listening to the faint, steady scratching noises and wondering what it was. A racoon, digging at the foundation below my bedroom window, perhaps? I was on the second floor.
Another alert.
Static exceeding established threshold. Terminating supervision of child processes.
That first part? None of that makes sense. That’s not a real error. That’s nonsense. Since none of my coworkers were fessing up to the prank (they were still all offline), I opened my laptop, pulled up the PagerDuty website, and filed a support ticket. I included the text of the alerts and explained that these errors did not exist in our ecosystem. If it wasn’t some mischievous coworker responsible, then I wanted the root cause resolved as quickly as possible. I wondered if I was getting errors from someone else’s system. That’s the thing about being paged late at night—it makes you highly motivated to make sure it never happens again.
Then I sat there and waited in my dark bedroom, eyes fixed on the glowing laptop screen. The scratching from outside my house seemed to be getting… longer. More a scraping sound now. I wondered if perhaps I should go shine a flashlight out the window and scare off whatever it was. I wasn’t necessarily afraid yet. I don’t scare easily and I have an alarm system, which meant my house was a less attractive target than the neighbors’.
Another alert with another ringtone. The one with a small child singing, “Something’s broken, something’s broken, it’s your fault!”
“It damn well isn’t my fault,” I muttered as I refreshed the screen.
“keeplightson.sh: Syntax error: “a̗͌b̠̔n̩͕̲͂̑̈̕͟o̡̧̘̾͋́r̢͙̣͍̗͌̀̐̿͝m̧̨̛̼̘̗̓͌̅̚ã̺͎͕̑̿̈́͢ḻ̥̇̈́i͈͎͚̮͗͐̅̚ẗ͉̮̬̗́́̏͡y̢̲̆̑” unexpected”
It was at that point that I took the nuclear route.
I @channel’d the General room on Slack, saying that I was seriously freaked out, this prank was going way too far, and I’d really appreciate it if someone would fess up and stop sending me alerts.
Did that push a notification to literally every person in my 1,000+ employee company? Yes. Did I care? No.
No one responded. The little icons of all my coworkers’ availability status remained stubbornly blank and meanwhile, my phone was blowing up with alerts, interrupting its own siren sound before it had finished with the next incoming klaxon.
The same notification, over and over and over.
Heartbeat check failed: ‘nymasterbedroom1n2’. Run keeplightson.sh
Then. A lull. Just a minute or so. But in that silence, I realized I no longer heard the scratching sound; instead, I heard something else: a creaking. The same creaking I’d heard in the hallway when I went to retrieve my laptop. It came from somewhere inside the house.
I froze, straining to listen to the sounds around me. Then I heard it: a rhythmic thud that repeated over and over, heavy and deliberate. Something was hitting the wall—no, a door. I heard the rattle of the hinges.
The basement door.
A PagerDuty notification lit up my screen and the speaker began blasting the harmonized tones of the barbershop quartet. It was a song I’d never heard before, one that wasn’t included in the app’s options.
“It is cooomiiiiiiiiing,” they crooned.
I reached over and, with shaking hands, jerked the cord on the bedside lamp, flooding my room with a warm glow. Then I acknowledged the alert to shut up the barbershop quartet and dialed 911. I told the operator there was something in my house.
A shuddering crash, the sound of a door being thrown open and slamming into the wall. I tumbled out of bed, trying to think of an escape route as the operator on the other end calmly instructed me to find a way out of the house.
“There is none,” I gibbered. “I’m on the second floor.”
She told me to shut the bedroom door and drag something heavy in front of it. Police were on the way, she said. I did as she instructed, shoving my wardrobe over to barricade myself inside the room before curling up in the far corner, next to the bedside table and the glow of the lamp. And waited.
From the hallway, a noise like nails on a chalkboard grew steadily closer. My mouth was dry and my eyes were fixed on the doorway. The 911 operator stayed on the phone with me, telling me to stay calm, that the police were almost there.
“I don’t think they’re going to make it in time,” I whispered.
Something slammed against the bedroom door. My wall shook with the impact. Something large, something viciously strong. The wardrobe moved an inch. Another impact. Another inch. And again. I couldn’t breathe. I could barely hear the operator on the phone telling me to stay calm, that the police would get there, telling me to hide if I could. I fumbled in the drawer of the nightstand and found a heavy flashlight that I kept nearby in case of a power outage in the night. I recognized the futility of using it for a weapon against something so massive, but I felt I had to try something.
The doorknob rattled and the door swung open, bumping up against the wardrobe and going no further than a handful of inches. The hallway was pitch dark beyond the doorframe and as I watched, a hand squeezed through the gap. Black, an empty darkness like the void, with irregular edges that fizzled in erratic lines like static. A hand as large as my head with fingers like spikes. It slithered inside my room, along the doorframe, along the drywall, trying to reach the panel of three switches that controlled all the outlets in my room.
Heartbeat check failed in the New York master bedroom. I was going to die.
PagerDuty had said to keep the lights on.
I flicked the flashlight on and shone it at the hand. The fingers vanished where the light hit it, I moved the beam to the left and the hand vanished as well, like erasing a shadow.
I heard police sirens in the distance, quickly growing louder. My breath came in wild gasps and I wanted to break down and cry, but I kept the flashlight fixed on the doorway. Whatever was out there did not try to enter the room again. I only relaxed when I heard the police pounding on the front door, before breaking it down, and I yelled at them to turn the lights on. Turn the damned lights on.
Only then did I see, over on my laptop monitor, that all of my active incidents got marked as auto-resolved in PagerDuty.
The Aftermath
The police searched my house and found that something had certainly been inside, but whatever it was, was now gone. There were three long gouges in the hallway, cutting clear through the drywall, at about the height of my shoulders. One of the officers told me it was probably a wild animal that got in through the basement and suggested I come see. I went with him to look at the basement.
The walls were covered in gouges, half an inch deep, in sets of three. Long, trailing marks, each line spaced about two inches apart.
“What kind of animal can claw up cement?” I asked, incredulous, running my finger along the grooves.
“The kind that you really don’t want in the house,” he replied.
He advised me to take a careful survey of the exterior in the morning. Maybe call someone out to inspect the house. See if I could find where it had got in—and out—of. Despite his attempt at calm professionalism, I could tell he was unnerved. He didn’t want to say what I was thinking. He and the other officers had scoured the house and found nothing that would explain how something big enough to leave these kind of claw marks behind would have gotten in.
That thing had started out in my house and was still here.
As they left, I told them to leave the lights on. I wanted to take some photos for insurance purposes, I claimed. I left the lights on all night and had never been so happy to see the dawn.
The next day I made an apology to my coworkers in the Slack General room. “Weird stuff happened,” I typed. “I got some bizarre pages and there was a wild animal in my basement making creepy noises and I got a bit freaked out.” People were understanding. I got trolled a bit. My coworkers sent me memes. They all had a good laugh. I pretended to laugh along as I did some more digging.
I found out our ops on-call person never got my escalated page. When I checked PagerDuty, all my resolved pages were gone like they had never existed. A few days later, PagerDuty support resolved my ticket with a stock “working as expected” message.
I haven’t dismissed what happened, though. Those claw marks are real. This wasn’t a prank or my imagination. Whatever it was, it could come back.
I’ve set up my own PagerDuty account. I’ve got a series of hooks so that whenever it alerts, it sends a message to my smart lightbulbs to turn on every light in the house. There are floodlights in the basement because I’m not screwing around with whatever is down there. Sure, it was expensive, but considering PagerDuty is now alerting at least once a week … I think it’s worth the investment.
I’m not sure who all they have working over there, but kudos to the PagerDuty team. It’s nice to see a company take the well-being of its client base so seriously. I’ll be honest—if my current gig doesn’t work out long term, I think I’m going to apply to work there as well. They don’t list any open positions for a secret department that handles supernatural events… but I bet they’re hiring.
I gotta say though: They really need more descriptive error messages.