Splunk Integration Guide

Splunk collects and indexes data from just about any source imaginable – network traffic, Web servers, custom applications, application servers, hypervisors, GPS systems, stock market feeds, social media, and preexisting structured databases.

This integration sends alerts to PagerDuty using Splunk’s native webhooks. If you would prefer to use our older, Python-based integration, you can find that here.

 

 

In PagerDuty

  1. From the Configuration menu, select Services.
    config_service
  2. Click Add New Service.addservice
  3. Enter a Name for your new service, and select Splunk Alerts from the Integration Type menu. Choose an Escalation Policy to use when the service receives an alert from Splunk. Click Add Service when you are finished.pdsplunkalerts-splunkalerts-03
  4. Copy the Integration URL for your new service:pdsplunkalerts-splunkalerts-04

In Splunk

  1. First you’ll need to install the PagerDuty Incidents app from the Splunkbase. You can find the app by first clicking on the box with the + on it on the left side of the page.
    splunkalerts-01
  2. Search for the PagerDuty App for Splunk and then click Install.
    spluk-app
  3. Once the app has been installed, head to the Settings menu and choose Alert actions.
    splunkalerts-05
  4. Make sure that the PagerDuty app is Enabled and then click Setup PagerDuty Incidents.
    splunkalerts-06
  5. Paste the Integration URL for your PagerDuty service into the supplied field.
    splunkalerts-05
  6. Run the search in Splunk for which you’d like to create an alert. We recommend testing by searching Splunk’s internal logs for failed login attempts: "index=_internal component=UiAuth action=login status=failure"
  7. Click Save As and select Alert.
  8. Add a Title for the new alert, specify the conditions under which you’d like it to trigger an alert and add a new Trigger Action. Select PagerDuty as the trigger action type. By default, this will notify the integration URL you established when you set up the PagerDuty app.splunkalerts-09

 

Testing your integration

We recommend testing your integration with a simple search that is easy to manipulate, like the above-referenced:
"index=_internal component=UiAuth action=login status=failure"

This will produce results any time there is a failed login attempt and is an easy situation to reproduce.

Set your search either to run on a schedule or to be run in real-time and set your threshold low (Number of results is greater than 0, for example). Open Splunk in another browser or in an incognito window and make some failed login attempts.

Not long after, you should see a PagerDuty alert.

pdsplunkalerts-splunkalerts-example

Clicking to the incident details will provide a breakdown of the alert and will provide you with a link to view the search in Splunk.

pdsplunkalerts-splunkalerts-exampledetails

Frequently Asked Questions

Can Splunk send alerts to more than one PagerDuty service?

Yes. If you are interested in notifying a different Splunk service in PagerDuty, you can paste that service’s integration URL in the optional Integration URL field under Trigger Actions when creating a new alert. This will override the global PagerDuty Integration URL established when you set up the PagerDuty Incidents app.

splunkalerts-optional

Will PagerDuty incidents resolve once the Splunk search is no longer producing results?

No. At this time, the PagerDuty incident must be resolved from PagerDuty.

How does PagerDuty group incoming alerts from Splunk?

If you select Edit Service from the main service view, you’ll be able to choose to group incidents by search name, component, host, source or to attach all incoming alerts to an open incident.

pdsplunkalerts-splunkalerts-config