Splunk Integration Guide

Splunk + PagerDuty Benefits

  • Send richly formatted event data from Splunk to PagerDuty, allowing you to engage the right people, accelerate resolution and improve learning.
  • Create high and low urgency incidents based on the severity of the event from the Splunk event payload.

How it Works

  • This integration uses Splunk’s native webhooks to send events to PagerDuty. 
  • Events from Splunk will trigger a new incident on the corresponding PagerDuty service, or group as alerts into an existing incident.

Requirements

  • In Splunk: This integration supports Splunk Enterprise and Cloud.
  • In PagerDuty: This integration requires a Manager base role or higher to configure. If you’re not sure what role you have, or if you need your permissions adjusted, visit our sections on Checking Your User Role or Changing User Roles.

Integration Walkthrough

In PagerDuty

There are two ways to integrate with PagerDuty: via global event routing or directly through an integration on a PagerDuty service. Integrating with global event routing may be beneficial if you want to build different routing rules based on the events coming from the integrated tool. Integrating with a PagerDuty service directly can be beneficial if you don’t need to route alerts from the integrated tool to different responders based on the event payload. 

Integrating with Global Event Routing

1. From the Configuration menu, select Event Rules.

2. On the Event Rules screen, copy your Integration Key.


3. Once you have your Integration Key, the Integration URL will be:

https://events.pagerduty.com/x-ere/[YOUR_INTEGRATION_KEY_HERE]

You can now proceed to the In Splunk section below. 

Integrating With a PagerDuty Service

1. From the Configuration menu, select Services.

2. If you are creating a new service for your integration, please follow the steps outlined in the Create a New Service section, selecting Splunk as the Integration Type in step 4. Continue with step 4 (below) once you have finished these steps.

If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then click the Integrations tab and click the +New Integration button.

3. Select Splunk from the Integration Type menu and enter an Integration Name in the format monitoring-tool-service-name (e.g. “Splunk-Checkout-Server”). Click the Add Integration button to save your new integration.

4. You will be redirected to the Integrations page for your service. Click the name of the integration and copy the Integration URL for your new integration.

In Splunk

1. Click the + in the left hand menu to download the PagerDuty Incidents app from the Splunkbase.

2. Search for the PagerDuty App for Splunk and then click Install.

3. Once the PagerDuty app has been installed, navigate to the Settings menu and choose Alert actions.

4. Ensure that the PagerDuty app’s Status is Enabled and then click Setup PagerDuty.

5. Paste the Integration URL (generated in the In PagerDuty section, above) for your PagerDuty service into the supplied field.

6. Run the search in Splunk that you would like to create an alert for. We recommend testing by searching Splunk’s internal logs for failed login attempts: index=_internal component=UiAuth action=login status=failure

7. Click the Save As dropdown and select Alert.

8. Add a Title for the new alert, specify the conditions under which you’d like it to trigger an alert, and add a new Trigger Action. Select PagerDuty as the trigger action type. By default, this will notify the Integration URL you established when you set up the PagerDuty app (step 5, above). Click Save when complete.

Testing your integration

We recommend testing your integration with a simple search that is easy to manipulate, like the above-referenced:

index=_internal component=UiAuth action=login status=failure

This will produce results any time there is a failed login attempt and is an easy situation to reproduce. Set your search either to run on a schedule or to be run in real-time and set your threshold low (Number of results is greater than 0, for example). Open Splunk in another browser or in an incognito window and make some failed login attempts. Not long after, you should see a new PagerDuty incident.

Clicking to the incident details will provide a breakdown of the alert and will provide you with a link to view the search in Splunk.

FAQ

Can Splunk send alerts to more than one PagerDuty service?

Yes. One option is integrating using global event rules and routing to different services based on event rules. Another option is pasting a different service’s Integration URL in the optional Integration URL field under Trigger Actions when creating a new alert. This will override the global PagerDuty Integration URL established when you set up the PagerDuty Incidents app.

Will PagerDuty incidents resolve once the Splunk search is no longer producing results?

No. At this time, the PagerDuty incident must be resolved from PagerDuty.

How does PagerDuty group incoming alerts from Splunk?

If you select Edit Service from the main service view, you’ll be able to choose to group incidents by search name, component, host, source or to attach all incoming alerts to an open incident.