How to Integrate Splunk with PagerDuty

Introduction

Splunk collects and indexes data from just about any source imaginable, such as network traffic, Web servers, custom applications, application servers, hypervisors, GPS systems, stock market feeds, social media, and preexisting structured databases.

Splunk can be configured to pass all alerts to PagerDuty. Using PagerDuty, you can receive your Splunk alerts via phone call, SMS, or email; configure automatic escalation of alerts; escalate alerts right from your mobile phone; and set up on-call duty scheduling.

What you’ll need to get started

First set up Splunk. You’ll also need a PagerDuty account (either a paid account or a free trial account will work).

Setting up Email integration

Splunk can integrate with PagerDuty either through the email API or through a Splunk Integration available from the Splunk Community

  1. Go to Services and click on Add New Service.
    SL-AddNewServiceButton
  2. Enter a Service Name, choose an Escalation Policy.
  3. Start typing “Splunk” under “Integration Type” to filter your choices. Then, click the Add Service button.
    Splunk
  4. Create an alert in Splunk
  5. Enter the PagerDuty email

Using API integration (beta)

Another alternative is to use the Splunk integration

  1. You can either build from the source on github or download pagerduty.spl
  2. Create an email service
  3. Note the API key
  4. In Splunk, from the app menu, select “Manage Apps” and then press “Install app from file”
  5. Upload pagerduty.spl to Splunk
  6. When you install the app, you’ll be given a chance to enter your API Key:

  7. Enter “pagerduty.py” in the “Run a script” field

Note, you can use both methods simultaneously by creating 2 services in PagerDuty and filling in both the “Run a script” and “Send email” fields.