How To Integrate Zenoss with PagerDuty

Introduction

Zenoss is a very popular open source network, server and application monitoring system used by thousands of IT organizations. It provides one of the best event management systems available in any open source monitoring system. The plug-in architecture of Zenoss allows it to be extensible by anyone.

PagerDuty extends Zenoss’s functionality by providing on-call scheduling, alerts and incidents tracking through the PagerDuty API. You are notified of the most critical Zenoss events by PagerDuty so you are able to take immediate action.

This guide describes how to integrate your Zenoss installation with PagerDuty using cURL. You might need to slightly alter these instructions depending on your exact Linux distribution and your Zenoss configuration.  If you are having trouble completing the installation, please contact us.

Setting up the PagerDuty / Zenoss integration

In PagerDuty

Create a “Generic API system” service:

  1. In your account, under the Services tab, click “Add New Service”.
  2. Enter a name for the service and select an escalation policy. Then, select “Generic API system” for the Service Type.
  3. Click the “Add Service” button.
  4. Once the service is created, you’ll be taken to the service page. On this page, you’ll see the “Service key”, which will be needed when you configure your Zenoss to send events to PagerDuty.

In Zenoss

  1. Install cURLand its dependencies on the machine running Zenoss (if not installed already):For Debian, Ubuntu, and other Debian-derived systems: 
    sudo aptitude install curl

    RHEL, Fedora, CentOS, and other Redhat-derived systems:

    sudo yum install curl
  2. Log into your Zenoss Console
  3. Click on the Events tab
  4. Click on the Event Manager sub-tab
  5. Click on the Commands link on the left panel
  6. Add a new command by entering the name “PagerDuty” and then click the Add button
  7. Click on the “PagerDuty” command to view the command details
  8. Enable the command by selecting “True” for the label titled “Enabled”
  9. Copy and past the following into the field labeled “Command”: 
    curl -H "Content-Type: application/json" -X POST -d '{"service_key": "YOUR-SERVICE-KEY", "incident_key": "${evt/evid}", "event_type": "trigger", "description": "${evt/device}: '${evt/summary}'", "details": { "device": "${evt/device}", "ipAddress": "${evt/ipAddress}", "severity": "${evt/severity}", "summary": ${evt/summary}, "message": ${evt/message}, "evid": "${evt/evid}"} }' https://events.pagerduty.com/generic/2010-04-15/create_event.json
  10. Replace YOUR-SERVICE-KEY with the key from PagerDuty
  11. Copy and past the following into the field labeled “Clear Command”: 
    curl -H "Content-Type: application/json" -X POST -d '{"service_key": "YOUR-SERVICE-KEY", "incident_key": "${evt/evid}", "event_type": "resolve"}' https://events.pagerduty.com/generic/2010-04-15/create_event.json
  12. Replace YOUR-SERVICE-KEY with the key from PagerDuty
  13. Click on the “Save” button at the bottom to save the changes

FAQ

What if a Zenoss event happens while my network is down?

At the moment, if the event command fails to submit notifications to PagerDuty, it will be registered as another event in Zenoss. The repeat time can be set in command settings such that notification would be retried. Note: PagerDuty will deduplicate events based on the event id, thus only one PagerDuty incident would be created even if Zenoss submits events multiple times.

You should also configure an external ping check service, such as Pingdom to monitor your site’s external connectivity. Of course, you can use PagerDuty to forward alerts from these services as well.

How do I setup Zenoss to work with multiple PagerDuty services?

This is easy to do with the current integration, as a Zenoss Service in PagerDuty is directly mapped to an “Event Command” in Zenoss.

In order to setup multiple services, just duplicate the existing event command rename it (i.e. pagerduty_database, pagerduty_network, etc.) and update the command with the appropriate service key. Don’t forget to set filters on the command such that only the desired events are sent to PagerDuty.