This is a guest post by Ilan Rabinovitch, Director of Product Management at Datadog. The convergence of rapid feature development, automation, continuous delivery, and the shifting...by Ilan Rabinovitch
August 24, 2017
Hackday is my favorite day of the month. Yet, it’s becoming increasingly difficult to win as PagerDuty hires more employees. It’s no longer a guaranteed winning strategy to present something technical or useful, so my approach has shifted. I’ve been building integrations for PagerDuty that are technically challenging, but also incredibly silly.
PagerDuty integrates with loads of tools, but what functionality is it missing? Bombardment. No one, to my knowledge, has wired PagerDuty up to a turret that will fire foam darts at the person on call whenever there is a PagerDuty alert.
To add a layer of shame to it, the turret also records a video of the assault with its on-board webcam.
To make things easy, we purchased a USB turret from Dream Cheeky. The problem was that this device had no API exposed and was only operated from an OS X application. After some research and experimentation, the device could be controlled by sending the appropriate payloads with libusb. We plugged it in and ran it on a Raspberry Pi sitting in the real-time engineering room of our San Francisco office.
We put an application in Heroku to receive PagerDuty alerts, with targets stored in a database of each engineer sitting near the turret and the relative coordinates of their desk. Webhooks from PagerDuty would mark the on-call as a target. The Raspberry Pi queries this data and fires a dart at the target on a regular interval.
The Heroku application made it easy for other integrations to be added. Slack was included so that the turret could be operated, annoyingly, by anyone at PagerDuty, even the Toronto office. The turret was intimidating, I decided to bring it to life with a little branding. The Slack integration was updated with a photo of my manager, Roman. His photo was also put on the front of the turret. The project became known as “The Roman Pillar of Justice”.
There was a brief moment while building this where Slack sent a few duplicate webhooks, and the results were awesome. Ashwin, an engineer on the real-time team, was shot by the turret multiple times for about a minute. The Slack output from this can be seen below:
Want to build your own Pillar of Justice? Check out the code on Github.