Agent Install Guide

The PagerDuty Agent is a helper program that you install on your monitoring system to integrate your monitoring tools with PagerDuty.

The Agent also includes command line tools to accept events from your monitoring system and a daemon process that handles the sending of these events to PagerDuty. The daemon process takes care of buffering, retrying & throttling of events to PagerDuty. The supported events are those listed in the PagerDuty Events API documentation.

System Requirements

The PagerDuty Agent is currently available in deb and rpm package formats and supported on Ubuntu 12.04, 14.04, and 16.04, as well as CentOS 6.

The Agent does not run on CentOS 5 or lower, as it requires a newer version of Python than the version included with CentOS 5. Please use the Perl-based integration for Nagios or older Python-based integration for Zabbix on older CentOS systems.

Creating a Service in PagerDuty

The Agent supports sending events to “Generic API” services if you’re building a custom integration using the PagerDuty Agent. If you do not have a Generic API service set up, you can create one as follows:

  1. Go to the Configuration menu and select Services, then click Add New Service.

  2. Enter a Name for your new service, select Use our API directly for the Integration Type, choose an Escalation Policy to use when the service receives an alert from the Agent, and select the desired Notification Urgency for incidents (available only with Basic or higher plans). Click Add Service when you are finished.

  3. Once the service is created, you’ll be taken to the service page. On the Integrations tab, you’ll see the Integration Key, which you will need when you send events to PagerDuty via the Agent’s command line tools.

    Note: Not all integration types and integration keys are compatible with the Events API. You must select Use our API directly (or Generic API from the drop down list) as the Integration Type to guarantee compatibility with our API when testing your integration or when using the pd-send command.

Agent Source, Installation and Updates

The PagerDuty Agent is open source (as are its integrations) and can be found on GitHub, but most users will simply want to install the Agent from our apt or yum repositories as described below.

Once you have the Agent installed, it’s important to remember to update the Agent packages regularly to ensure that you have the latest fixes. You can update the Agent packages with your regular package updates, or individually as shown:

Ubuntu

  1. To add the PagerDuty repository to your system:

    wget -O - https://packages.pagerduty.com/GPG-KEY-pagerduty | sudo apt-key add -
    sudo sh -c 'echo "deb https://packages.pagerduty.com/pdagent deb/" >/etc/apt/sources.list.d/pdagent.list'
  2. To install (or update) the PagerDuty Agent packages:

    sudo apt-get update
    sudo apt-get install pdagent pdagent-integrations

CentOS

Note: The PagerDuty Agent is not compatible with CentOS 5 or lower, so the installation will not succeed on CentOS 5.

  1. To install, add the PagerDuty repository to your system:

    sudo sh -c 'cat >/etc/yum.repos.d/pdagent.repo <<EOF
    [pdagent]
    name=PDAgent
    baseurl=https://packages.pagerduty.com/pdagent/rpm
    enabled=1
    gpgcheck=1
    gpgkey=https://packages.pagerduty.com/GPG-KEY-RPM-pagerduty
    EOF'
  2. Install the PagerDuty Agent packages:

    sudo yum install pdagent pdagent-integrations
  3. To update just the Agent the packages:

    sudo yum update pdagent pdagent-integrations

Verify the Agent is Running

If the installation is successful, the Agent daemon process should start automatically. You can use the systemctl (for systemd-based systems, like Ubuntu 16.04) or service (for SysV init-based systems, like Ubuntu 14.04) command to start, stop, and check the status of the Agent.

systemd example:

sudo systemctl status pdagent

SysV init example:

sudo service pdagent status

If the daemon is not running and will not start, check /var/log/pdagent/pdagentd.log for errors.

Sending an Event to PagerDuty

To send an event to PagerDuty, you can use the pd-send command. This command enqueues the event on the local disk. The Agent daemon will take care of actually sending the event to the PagerDuty Integration API. If the pdagentd service is stopped, events will not be sent to PagerDuty until the service is started again.

Using the -h or --help option will return usage information:

$ pd-send --help
usage: pd-send [-h] -k SERVICE_KEY -t {trigger,acknowledge,resolve}
 [-d DESCRIPTION] [-i INCIDENT_KEY] [-f FIELDS]

Queue up a trigger, acknowledge or resolve event to PagerDuty.
...

The SERVICE_KEY above is the Integration Key that is shown on the Service page on the PagerDuty website.

Here’s an example command that would trigger an incident in PagerDuty if your service’s integration key was 65d9cd0e14c04dae8ef86867277d138c:

$ export PD_SERVICE_KEY=65d9cd0e14c04dae8ef86867277d138c
$ pd-send -k $PD_SERVICE_KEY -t trigger -d "Server is on fire" -i server.fire
Event processed. Incident Key: server.fire

If copying this example, remember to substitute the correct Integration API Key for your PagerDuty Service!

Note: The user that runs pd-send and other pd-* integration scripts should preferably have a umask that allows world-readability of files that the user creates (i.e. 022). If the umask is more restrictive, the pd-* script changes the umask internally (only for itself) so enqueue-event files are created with the correct permissions.

Managing the Queue

To check the queue of outgoing alerts you can use the pd-queue command. This command can be used to get a quick idea of the queue status, and to retry events for an integration key. Note that the command needs to be run as root so as to gain access to directories owned by the pdagent user (i.e. sudo pd-queue status).

Below are sample uses and output of the command:

pd-queue -h

$ pd-queue -h
usage: pd-queue [-h] {status,retry} ...

Access local queue of PagerDuty Agent.

optional arguments:
  -h, --help      show this help message and exit

sub-commands:
  {status,retry}
    retry         set up 'dead' PagerDuty events for retry.
    status        print out status of local event queue.

pd-queue status

Displays the pending, successful and in-error events in the Agent queue currently, as a table keyed by service key.

$ pd-queue status -h
usage: pd-queue status [-h] [-k SERVICE_KEY]

Print out status of local event queue.

optional arguments:
  -h, --help            show this help message and exit
  -k SERVICE_KEY, --service-key SERVICE_KEY
                        print status of events in given Service API Key

$ pd-queue status
Service Key                           Pending   Success  In Error
=================================================================
key1                                        1         0         0
key2                                        1         0         1

$ pd-queue status -k key2
Service Key                           Pending   Success  In Error
=================================================================
key2                                        1         0         1

pd-queue retry

Moves events in error-state to pending state en mass for a specified integration key. Hopefully, the user has corrected any user-induced problems that caused the errors in the first place, so trying the events again will not result in errors. For example, if the event was too long, the user has edited the event to reduce its length. If an event results in an error again, it will be moved to an error state again.

$ pd-queue retry -h
usage: pd-queue retry [-h] [-k SERVICE_KEY] [-a]

Set up 'dead' pagerduty events for retry.

optional arguments:
  -h, --help            show this help message and exit
  -k SERVICE_KEY, --service-key SERVICE_KEY
                        retry events in given Service API Key (not to be used
                        with -a)
  -a, --all-keys        retry events in all Service API Keys (not to be used
                        with -k)

$ pd-queue retry -k key2
1 events set up for retry.

$ pd-queue retry -a
0 event(s) set up for retry.

Integrating with Monitoring Tools

The PagerDuty Agent includes scripts to help you integrate with some common monitoring tools. Please see the appropriate guide for more details:

Using a Proxy

If your environment requires a proxy to be set for outbound HTTP/HTTPS connections, you can specify proxy configuration by editing sudo nano /lib/systemd/system/pdagent.service (on systemd-based systems) or /etc/init.d/pdagent (on SysV init-based systems) as the root user.

systemd-based Systems

sudo nano /lib/systemd/system/pdagent.service

Simply add in two lines, replacing the URL with your proxy information, after the ExecStartPre=/bin/chown -R pdagent:pdagent /var/run/pdagent line which is approximately line #8. It should look like the following once completed.

ExecStartPre=/bin/chown -R pdagent:pdagent /var/run/pdagent
Environment="http_proxy=http://proxy:3128/"
Environment="https_proxy=http://proxy:3128/"

Note: Be sure to restart the PagerDuty Agent with the command systemctl restart pdagent.

SysV init-based Systems

sudo nano /etc/init.d/pdagent

Simply add in two lines, replacing the URL with your proxy information, after the PID_FILE=$PID_DIR/pdagentd.pid line which is approximately line #48. It should look like the following once completed.

PID_FILE=$PID_DIR/pdagentd.pid

export http_proxy=http://proxy:3128/
export https_proxy=http://proxy:3128/

Note: Be sure to restart the PagerDuty Agent with the command service pdagent restart.

Troubleshooting

If you have trouble sending events to PagerDuty, check out our PagerDuty Agent Troubleshooting guide.