A (Lobster) Tale of Two Systems aka the ServiceNow Chronicles
Hi Yangsters, I hope everyone is staying safe during these unpredictable times. As a fellow professional in technology, I’m guessing your workload has either remained the same or you’re now extra busy since more people are moving to doing everything online. I’ve been so busy that I haven’t been able to provide a proper update on the ongoings of Bikini Bottom—until now.
But don’t worry! We’ll get through this together!
ARE YOU READY, KIDS? I CAN’T HEAR YOU! OHHHH!
The update: The local burger shop, Krusty Krab, is still open. They’re adhering to social distancing, keeping six feet apart, and offering take-out and delivery options only. Apparently, even under the sea, you can’t be too careful. Especially when you’re surrounded by sardines.
Plankton, Mr. Krab’s archnemesis, is trying to step up his restaurant’s (the Chum Bucket) technical operations to stay competitive with the Krusty Krab. He’s offering online ordering so customers can minimize crustacean-to-crustacean contact.
Plankton, with the help of his robo-wife, Karen, has to scale up his incident management tool stack to organize and manage the requests for chum that are coming in. Karen definitely understands the impact of wires being crossed, especially being a robot underwater. Plankton went with ServiceNow as his center of truth for request management. The good news is that there’s a robust integration available between ServiceNow and PagerDuty to notify the right folks when the requests turn into incidents.
By setting up the ServiceNow + PagerDuty integration, the entire incident lifecycle is logged between the two systems. Users can then access PagerDuty products and features, such as Event Intelligence, Modern Incident Response, Postmortems, incident creation, stakeholder notification, and incident resolution, to name just a few. Our integration guide does a wonderful job of explaining “how” to set up the integration, but Plankton would like to know “why” a certain configuration will catch more chum than others. After all, he’s already invested his hard-earned money in both ServiceNow and PagerDuty.
The ServiceNow PagerDuty integration allows you to sync incidents two different ways: Assignment Group Only or Configuration Items + Assignment Group (CI + AG).
Assignment Group Only Mapping
Assignment Group Only mapping is the easiest way to get the two systems talking to each other and enables your teams to start actioning on critical ServiceNow incidents in real-time. Once an Assignment Group is provisioned to PagerDuty, the integration will create a Technical Service and an Escalation Policy in PagerDuty. Any ServiceNow incident assigned to this mapped Assignment Group will sync and create a PagerDuty incident on that service, notifying that Assignment Group’s escalation policy. Immediately, you’ll be able to start tracking metrics such as Mean Time to Acknowledge (MTTA) and Mean Time to Resolve (MTTR).
This is a good option for companies that do not use Configuration Items in ServiceNow.
In the example above, the Order Intake team (comprised of Karen and Plankton) is set up as an Assignment Group. Once provisioned, an “Order Intake” technical service will be created and linked to the “Order Intake” Escalation Policy, which contains the “Order Intake” Schedule, with Karen and Plankton in the schedule.
Any ServiceNow incidents assigned to the “Order Intake” Assignment Group will create a corresponding incident on the PagerDuty technical service “Order Intake” and notify the linked Escalation Policy. Any PagerDuty incident created on the “Order Intake” technical service will be synced to ServiceNow, creating a ServiceNow incident, assigned to the “Order Intake” Assignment Group.
Since each Assignment Group becomes a Technical Service in PagerDuty, it becomes very hard to relate the Technical Services back to a Business Service in PagerDuty. The Assignment Group services become a bit of a monitoring sink. Again, if Plankton didn’t use Configuration Items in his ServiceNow Implementation, this option will get the two systems up and running in no time.
Configuration Items + Assignment Group Mapping
The Configuration Items + Assignment Group (CI + AG) mapping model allows for more robust notification between the two systems. When a CI is provisioned, it will create a PagerDuty Technical Service for the CI, based off of the Assignment Group for that CI, and that group’s escalation policy will be used for that service.
In the example above, you’ll see that we’ve provisioned the “Chum Bucket Ordering Service” CI, which is linked to the “Order Intake” Assignment Group in PagerDuty, which created the “Chum Bucket Ordering Service” Technical Service, linking to the “Order Intake” Escalation Policy.
When an incident is created in ServiceNow, the integration looks at both the CI and Assignment Group fields to determine which technical service to create the incident on and which escalation policy to notify. If there’s an issue with the “Chum Bucket Ordering Service,” and this is an issue for the “Network” team (instead of the “Order Intake” team), the integration will respect that assignment and create the incident on the “Chum Bucket Ordering Service” Technical Service and notify the “Network” Escalation Policy. Unfortunately, for our least favorite duo in Bikini Bottom, it’s still Plankton or Karen that will be paged, since they have no employees (or friends).
If you reach into the back of your brain and pull out Lisa’s Best Practices for PagerDuty Service Configuration, you’ll know that creating Services based off of a team isn’t a great approach. Event Intelligence won’t be able to group events appropriately, Technical Services would fall outside of Business Service designations, and you won’t have Visibility into which business services to invest in.
If Assignment Group mapping is too broad, then Plankton should provision every Configuration Item into PagerDuty, right? But then he would have replicated ServiceNow in PagerDuty, and now have two sets of CIs to maintain. In the ephermal world of the cloud, Plankton would want to provision CIs to PagerDuty that do not change very often.
So what’s the sweet spot?
Technical Service or Microservice CIs.
If Plankton’s ServiceNow CMDB is not set up with the Technical Services/Microservices level of visibility, there are a few ways around this. First, I would advise Plankton to create a new PagerDuty class of CIs. This table is isolated from CMDB discovery tools, therefore happily coexisting with other CIs records, but fills the gap between Business Services and Infrastructure CIs.
Alternatively, we could provision Plankton’s Business Service CIs to PagerDuty. If you’re not able to create Technical or Microservices CIs, or create a PagerDuty class, then we will leverage the existing Business Service CIs. This is a bit less of a monitoring sink than Assignment Group Only configuration, and it gets the teams to start thinking about the underlying services they are (or not) responsible for.
To keep synchronization between ServiceNow and PagerDuty, the following items must be mapped:
- Users Record must have correct PagerDuty User ID
- Assignment Group records must have the correct PagerDuty Team ID for Team sync
- Assignment Group records must have the correct PagerDuty Escalation Policy ID for the right group to be paged
Assignment Group Only Mapping:
- Assignment Group records must have the correct PagerDuty Service ID for the incident to be created successfully.
- Assignment Group records must have the correct PagerDuty Webhook ID for the ServiceNow incident link to appear in PagerDuty
Configuration Items + Assignment Group Mapping:
- CI records must have the correct PagerDuty Service ID for the incident to be successfully created.
- CI records must have the correct PagerDuty Webhook ID for the ServiceNow incident link to appear in PagerDuty
As you can see, the PagerDuty ServiceNow integration is incredibly powerful. Plankton has many options to choose from when it comes to how he wants to set up and configure the integration to fit the Chum Bucket’s needs. He now has all the information to make the best configuration choices for his Incident Management workflow. He just has to befriend more fishes that are willing to work for him.
