This integration requires use of our v2 REST API.
This is designed to help administrators integrate ScienceLogic SL1 system with PagerDuty’s Incident Management SaaS service. This guide will help prepare you for the integration tasks. If you are having trouble completing the installation, please contact us.
To integrate PagerDuty with ScienceLogic the following steps will be required:
It is recommended that you familiarize yourself with the ScienceLogic Run Book Automation (RBA) functionality before activating the PagerDuty Power-Pack. The default PagerDuty RBA policies are very broad and will create new PagerDuty incidents for every event in ScienceLogic that is of severity minor, major, and critical.
The ScienceLogic PagerDuty integration Power-Pack offers several key functions:
The PagerDuty integration relies on Run Book Automation policies to “push” both events and related event actions to PagerDuty. Activities emanating from PagerDuty, for instance, acknowledging an incident, are synchronized through a Dynamic Application. The following diagram shows the dataflow for how ScienceLogic events and PagerDuty incidents are synchronized.
Incidents resolved in PagerDuty will not automatically resolve events in ScienceLogic. This is because most events in ScienceLogic will automatically resolve themselves if they are no longer active or detected. For instance, if a monitored device is detected as being unavailable, ScienceLogic will create an event, and then create an incident in PagerDuty. If the incident in PagerDuty is resolved but the device is still detected as being unavailable, ScienceLogic will automatically create another event and PagerDuty incident. However if the incident is resolved in PagerDuty, but the event still remains in ScienceLogic, duplicate events will be suppressed. Once the event is no longer valid, ScienceLogic will automatically resolve the event and update the incident in PagerDuty.
We will create a PagerDuty API integration in the PagerDuty web portal for the ScienceLogic Run Book Automation integration. We will also add an API access key for ScienceLogic Dynamic Application performance and synchronization. Both steps will require you record the “key” to add to the respective ScienceLogic credential. You will need administrative access to your PagerDuty account.
Note: You may create multiple integrations for use with different PagerDuty policies. The ScienceLogic PagerDuty Power-Pack can be aligned to any number of different PagerDuty accounts and services.
In this section we will install the PagerDuty Power-Pack and then configure both the Run Book Automation credential and the Dynamic Application credential.
Obtain the latest copy of the “PagerDuty Integration” Power-Pack. In this document we will be using version 2.1 of the PowerPack.
Once the Power-Pack is installed, proceed to the next section to configure the credential.
With the Power-Pack installed, we can now configure our Run Book Automation credential.
The Dynamic Application credential is needed if you wish to synchronize incident changes from PagerDuty to ScienceLogic.
In this section we will configure the PagerDuty Run Book Automation Policy, aligning the credential, and begin sending events to PagerDuty. The Run Book Automation policies provided by ScienceLogic will create outbound incidents in PagerDuty.
Navigate to the Run Book Actions page by clicking on Registry tab, then Run Book, and then Actions. You will notice three PagerDuty actions:
Each of these actions performs a different function and allows you to align different Automation policies based on your business needs. To configure these actions, we must manually edit each and align the proper PagerDuty credential that contains the PagerDuty Integration key.
Note: PagerDuty Actions must run on the ScienceLogic Database, double check that the Action Run Context is set to Database. Once complete, let’s double-check the PagerDuty Automation Policies.
Like the PagerDuty Run Book Actions, there are three Automation Policies. Each Automation Policy performs a different task based on criteria established in the Policy. By default the PagerDuty Automation Policies are very broad, allowing every ScienceLogic event that has a severity higher than or equal to “minor” to trigger a PagerDuty incident. Although this may be good to begin testing your PagerDuty integration, it is advised to adjust each PagerDuty Automation policy to meet the needs of your business.
To prevent your changes to an Automation Policy from being overwritten when the PagerDuty PowerPack is updated, you must remove it from the PowerPack.
Note: It is not recommended that you remove the Automation Actions from the PowerPack even though you edited the Automation Action to change the aligned credential. If you remove an Automation Action from the PowerPack, any functional changes made to the Automation Action code will not be installed on your system when the PowerPack is updated.
In this section we will create a PagerDuty device, and manually align the PagerDuty Synchronization and Performance Dynamic Application. The Synchronization and Performance Dynamic Application provided by ScienceLogic will provide near-real time performance data regarding your PagerDuty service, as well as synchronize changes emanating from PagerDuty.
Although the PagerDuty Dynamic Application may be aligned to any ScienceLogic device, in this section we will walkthrough creating a dedicated PagerDuty device.
To align the PagerDuty Synchronization and Performance Dynamic Application:
Once aligned, the Dynamic Application will run every 15 minutes looking for updates to existing ScienceLogic events and updating performance data.
To increase the frequency of execution, select the checkbox for the PagerDuty Synchronization and Performance Dynamic Application, select a new value in the Poll Frequency section of the Select Action field, and then click the Go button.
To view the performance metrics collected by the Dynamic Application, click the graph icon, go to the Performance tab, and expand PagerDuty Synchronization and Performance in the left NavBar:
Every ScienceLogic event that matches the PagerDuty Run Book Automation policy will create a new PagerDuty incident. Once an incident is created, notification and escalation policies on the Pager Duty system will go into effect.
ScienceLogic’s Run Book Automation integration is a forward synchronization process, meaning that events and activities emanate from the ScienceLogic system to the PagerDuty service system. Just as new ScienceLogic events will create new PagerDuty incidents, acknowledging or clearing events from within ScienceLogic’s event monitor will perform the same function via the PagerDuty API.
Acknowledging incidents from the PagerDuty service portal will only update events in ScienceLogic if the PagerDuty Synchronization and Performance Dynamic Application is configured. If events are auto cleared by ScienceLogic, because either the event has timed-out or the system no-longer detects there’s still a problem, events will also be automatically resolved in PagerDuty. The below example shows a ScienceLogic Event Console with several different active events. The same events are synchronized in PagerDuty as triggered incidents.
Note: For this example all events are creating incidents in PagerDuty, which is a function of the ScienceLogic Run Book Automation policy and can be adjusted to meet the needs of your business.
Since PagerDuty requires a unique incident ID to de-duplicate events, ScienceLogic uses the device ID, called the DID, to help eliminate duplicate event storms for a single device. If a device has multiple events, the parent event (usually the highest severity event) will be used for the PagerDuty incident. If subsequent events appear after the initial event correlation process by ScienceLogic (usually time based), the new event will update the PagerDuty incident with the new description.
When events are acknowledged in ScienceLogic, the acknowledged status will be synchronized to PagerDuty. This process can take up to 60 seconds.
Once synchronized the status of the PagerDuty incident is updated. Resolving an event in the ScienceLogic event monitor also updates the status of the Incident in PagerDuty.
Incidents that are acknowledged in the PagerDuty portal or Smart Phone applications will synchronize back to ScienceLogic if the PagerDuty Dynamic Application has been installed.
By default, synchronization can take up to 15 minutes, however users can change the frequency be editing the Dynamic Application properties. In order to maintain continuity of user assignment, ScienceLogic matches the PagerDuty assigned username to the ScienceLogic username. If there is a match ScienceLogic events will be updated to matching PagerDuty incidents. If no username can be found, no updates will be made.
For instance, if the username in ScienceLogic is “jdoe”, the same username must exist in PagerDuty for the reverse synchronization process to update events in ScienceLogic. The primary reason for this is because of ScienceLogic uses advanced auditing and change control process that must know which user account is acknowledging events.
In PagerDuty, any ScienceLogic created incident will have additional notes and details about the event. The details include information about the device, including the last occurrence, severity, and IP address. Users can also navigate from PagerDuty to ScienceLogic by clicking the Client URL link.
If the “PagerDuty Synchronization & Performance” Dynamic Application is installed, users can see several different performance metrics, including:
In addition to the above performance metrics, the “Percentage of Acknowledged” metric also has an alarm threshold that can be adjusted to meet the needs of your environment. The threshold value can be set on the Device Properties > Thresholds tab.
In addition to performance metrics and alerts, the ScienceLogic PagerDuty solution provides an interactive performance dashboard. In case you have multiple PagerDuty accounts, the dashboard will support multi-tenancy allowing a consolidated view of all PagerDuty performance metrics.
For distributed ScienceLogic implementations, special setting must be made in order for the PagerDuty Synchronization & Performance Dynamic Application to work. Edit the Dynamic Application from the System > Applications page. Click the yellow wrench next to the “PagerDuty: Synchronization & Performance” application. After the browser window opens, click on the “Snippet” tab. Click on the yellow wrench next to the Snippet in the Snippet Registry.
The following variables must be changed to reflect your environment.
Change the MASTER_DATABASE_HOST to the IP address of the ScienceLogic central database server. If the username or password is different than the default, change those as well.The collector must be able to communicate with the central database server. As a result, port 7706 must be open. This can be validated by testing the MySQL connection from the collector’s command line.
mysql --host=192.168.2.87 --port=7706 –u root -p
If you get “ERROR 1130: Host is not allowed to connect to this MySQL server”, you will need to allow a specific client IP address (for example: 192.168.1.4) to access the MySQL database.Logon the Central Database CLI or use the DB Tool in the UI.
mysql> use mysql; mysql> GRANT ALL ON *.* to root@'192.168.1.4' IDENTIFIED BY 'your-root-password'; mysql> FLUSH PRIVILEGES;
Lastly, update firewall rules to make sure TCP port 7706 is open on the Central Database. In our testing of ScienceLogic 7.3.0, the port 7706 was found to be open.
When an event is acknowledged or resolved in ScienceLogic (event monitor or auto-clear), it runs the matching RBA policy and tells the PagerDuty API to acknowledge/resolve the matching incident. The PagerDuty API does not support any fields to indicate who acknowledged the incident; as a result API acknowledged incidents show up as “Through the API”.Although this is normal behavior, ScienceLogic also provides audit logging of who on the ScienceLogic system acknowledges or resolves an incident. This is available by navigating to the Incident Log of any incident.
Please contact us if you require further assistance in getting set up.