How To Integrate Nagios with PagerDuty

Nagios is one of the leading providers of open source and enterprise-grade IT infrastructure monitoring tools. Used by hundreds of thousands of users worldwide, Nagios allows it’s users to monitor their entire IT infrastructure, spot problems before they occur, detect security breaches and plan/budget for IT upgrades.

PagerDuty allows Nagios users to add PagerDuty’s on-call scheduling, alerts and incident tracking to their existing monitoring solution through the PagerDuty API.

By integrating PagerDuty into your existing Nagios monitoring solution, you can have Nagios alerts go directly to the the right person who can solve the issue.

The guide below describes how to integrate your Nagios 2, 3 or 4 installation with PagerDuty using a simple Perl-based plugin. Note that you must be logged in as root to complete the installation. You might need to slightly alter these instructions depending on your exact Linux distribution and your Nagios configuration. If you are having trouble completing the installation, please contact us.

Getting Started

If you don’t already have a PagerDuty “Nagios” service, you should create one:

  1. In your account, under the Services tab, click “Add New Service”.
    SL-AddNewServiceButton
  2. Enter a Service Name, choose an Escalation Policy. Start typing “Nagios” under “Integration Type” to filter your choices.
    Nagios
  3. Click the “Add Service” button.
  4. Once the service is created, you’ll be taken to the service page. On this page, you’ll see the “Service key”, which will be needed when you configure your Nagios server to send events to PagerDuty.

Setup for Debian, Ubuntu, and other Debian-derived systems:

  1. Install the necessary Perl dependencies:
    aptitude install libwww-perl libcrypt-ssleay-perl
  2. Download pagerduty_nagios.cfg from github (for all versions of Nagios):
     wget https://raw.github.com/PagerDuty/pagerduty-nagios-pl/master/pagerduty_nagios.cfg

    If you’re using Nagios 4, modify the commands in the pagerduty_nagios.cfg file to:

    define command {
          command_name    notify-service-by-pagerduty
          command_line   /usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=service -f CONTACTPAGER="$CONTACTPAGER$" -f NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" -f HOSTNAME="$HOSTNAME$" -f SERVICEDESC="$SERVICEDESC$" -f SERVICESTATE="$SERVICESTATE$"
    }
    define command {
          command_name    notify-host-by-pagerduty
          command_line    /usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=host -f CONTACTPAGER="$CONTACTPAGER$" -f NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" -f HOSTNAME="$HOSTNAME$" -f HOSTSTATE="$HOSTSTATE$"
    }
    
  3. Open the file in your favorite editor.
  4. Enter the service key corresponding to your Nagios service into the pager field. The service key is a 32 character string that can be found on the service’s detail page.
  5. Copy the Nagios configuration file into place:
    cp pagerduty_nagios.cfg /etc/nagios3/conf.d
  6. Add the contact “pagerduty” to your Nagios configuration’s main contact group. If you’re using the default configuration, open /etc/nagios3/conf.d/contacts_nagios2.cfg and look for the “admins” contact group. Then, simply add the “pagerduty” contact.
    define contactgroup{
         contactgroup_name admins 
         alias Nagios
         Administrators 
         members root,pagerduty ; Add pagerduty here
    }
  7. Download pagerduty_nagios.pl from github and copy it to /usr/local/bin.
    wget https://raw.github.com/PagerDuty/pagerduty-nagios-pl/master/pagerduty_nagios.pl 
    cp pagerduty_nagios.pl /usr/local/bin
  8. Make sure the file is executable by Nagios
    chmod 755 /usr/local/bin/pagerduty_nagios.pl
  9. Enable environment variable macros in /etc/nagios3/nagios.cfg (if not enabled already)
    enable_environment_macros=1
  10. Edit the nagios user’s crontab
    crontab -u nagios -e
  11. Add the following line to the crontab
    * * * * * /usr/local/bin/pagerduty_nagios.pl flush
  12. Restart Nagios
    /etc/init.d/nagios3 restart

Setup for RHEL, Fedora, CentOS, and other Redhat-derived systems:

  1. Install the necessary Perl dependencies:
    yum install perl-libwww-perl perl-Crypt-SSLeay
  2. Download pagerduty_nagios.cfg from github (for all versions of Nagios):
     wget https://raw.github.com/PagerDuty/pagerduty-nagios-pl/master/pagerduty_nagios.cfg

    If you’re using Nagios 4, modify the commands in the pagerduty_nagios.cfg file to:

    define command {
          command_name    notify-service-by-pagerduty
          command_line   /usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=service -f CONTACTPAGER="$CONTACTPAGER$" -f NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" -f HOSTNAME="$HOSTNAME$" -f SERVICEDESC="$SERVICEDESC$" -f SERVICESTATE="$SERVICESTATE$"
    }
    define command {
          command_name    notify-host-by-pagerduty
          command_line    /usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=host -f CONTACTPAGER="$CONTACTPAGER$" -f NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" -f HOSTNAME="$HOSTNAME$" -f HOSTSTATE="$HOSTSTATE$"
    }
    
  3. Open the file in your favorite editor.
  4. Enter the service key corresponding to your Nagios service into the pager field. The service key is a 32 character string that can be found on the service’s detail page.
  5. Copy the Nagios configuration file into place:
    cp pagerduty_nagios.cfg /etc/nagios
  6. Edit the Nagios config to load the PagerDuty config. To do this, open /etc/nagios/nagios.cfg and add this line to the file:
    cfg_file=/etc/nagios/pagerduty_nagios.cfg
  7. Add the contact “pagerduty” to your Nagios configuration’s main contact group. If you’re using the default configuration, open /etc/nagios/localhost.cfg and look for the “admins” contact group. Then, simply add the “pagerduty” contact.
    define contactgroup{ 
         contactgroup_name admins 
         alias Nagios Administrators 
         members root,pagerduty ; Add pagerduty here
    }
  8. Download pagerduty_nagios.pl from github and copy it to /usr/local/bin.
    wget https://raw.github.com/PagerDuty/pagerduty-nagios-pl/master/pagerduty_nagios.pl 
    cp pagerduty_nagios.pl /usr/local/bin
  9. Make sure the file is executable by Nagios
    chmod 755 /usr/local/bin/pagerduty_nagios.pl
  10. Enable environment variable macros in /etc/nagios/nagios.cfg (if not enabled already)
    enable_environment_macros=1
  11. Edit the nagios user’s crontab
    crontab -u nagios -e
  12. Add the following line to the crontab
    * * * * * /usr/local/bin/pagerduty_nagios.pl flush
  13. Restart Nagios
    /etc/init.d/nagios3 restart

FAQ

How do I setup Nagios to work with multiple PagerDuty services?

This is easy to do with the current integration, as a Nagios Service in PagerDuty is directly mapped to a “contact” in Nagios. By default, this contact is named “pagerduty” and defined in the pagerduty_nagios.cfg file.

In order to setup multiple services, just duplicate the existing contact definition and rename it (i.e. pagerduty_database, pagerduty_network, etc.). Then copy and paste the corresponding API Key from PagerDuty into the “pager” field. Don’t forget to restart your Nagios for the changes to take effect.

What if a Nagios event happens while my network is down?

If a PagerDuty server can’t be reached for any reason, events will be stored to an on-disk queue. The installed cron job will attempt to re-send the events at one minute interval.

Since Nagios needs my external Internet connection to send failure reports to PagerDuty, how will I receive notification if our site loses external connectivity?

You should configure an external ping check service such as StatusCake or NodePing to monitor your site’s external connectivity. Of course, you can use PagerDuty to forward alerts from these services.

It doesn’t seem to be working. What’s going on?

Check the syslog for messages from “pagerduty_nagios”. On most systems:

grep pagerduty_nagios /var/log/syslog

Please contact us if you’re unable to sort out the difficulty.

What sort of Nagios messages does PagerDuty understand?

PagerDuty can process PROBLEM, ACKNOWLEDGEMENT, and RECOVERY messages. All other messages, including FLAPPINGSTART and FLAPPINGSTOP, are ignored. If you’d like PagerDuty to process additional Nagios messages, please let us know!

How can I customize my Nagios alerts?

If you would like to customize your Nagios alerts, follow our guide here.