Check_MK Integration Guide – Agentless

Check_MK is built on top of Nagios, which is one of the leading providers of open source and enterprise-grade IT infrastructure monitoring tools. Used by hundreds of thousands of users worldwide, Nagios allows its users to monitor their entire IT infrastructure, spot problems before they occur, detect security breaches and plan/budget for IT upgrades. By integrating PagerDuty into your existing Check_MK monitoring solution, you can have alerts go directly to the the person on-call in your PagerDuty schedule.

This guide describes how to integrate Check_MK 1.2.x, by itself or as part of the Open Monitoring Distribution (OMD), with PagerDuty using a simple Perl-based plugin. Note that you must be logged in as root to complete the installation. You might need to slightly alter these instructions depending on your exact Linux distribution as well as your Check_MK configuration and version. Please contact our support team if you have any trouble completing the integration.

Note: This guide has been replaced by the Check_MK Integration Guide that uses the PagerDuty Agent. Please try following the steps in the new integration guide first, and use this one only if you have a system which the PagerDuty Agent can not be installed on.

In PagerDuty

    1. From the Configuration menu, select Services. 
    2. On your Services page:

      If you are creating a new service for your integration, click +Add New Service.

      If you are adding your integration to an existing service, click the name of the service you want to add the integration to. then click the Integrations tab and click the +New Integration button.

    3. RS-Add-New-Service
      RS-Add-Integration-Existing-Service

    4. Select your app from the Integration Type menu and enter an Integration Name.

      If you are creating a new service for your integration, in General Settings, enter a Name for your new service. Then, in Incident Settings, specify the Escalation Policy, Notification Urgency, and Incident Behavior for your new service.

    5. Click the Add Service or Add Integration button to save your new integration. You will be redirected to the Integrations page for your service.
      RS-Integration-Settings
    6. Copy the Integration Key for your new integration: RS_API_pd_3

    On Your Check_MK Server

    This guide includes steps for Debian-based (i.e. Ubuntu) and RHEL-based (i.e. CentOS, Fedora) Linux distributions, and the standalone version of Check_MK as well as the OMD version. You do not need to execute all commands in this guide, only the ones for your type of system, and you will need to adjust the paths used depending on the version of Check_MK you’re using. Note that all commands provided are intended to be run as the root user.

    1. Install the necessary Perl dependencies.

      For Debian-based systems:

      apt-get install libwww-perl libcrypt-ssleay-perl

      For RHEL-based systems:

      yum install perl-libwww-perl perl-Crypt-SSLeay
    2. Download pagerduty_nagios.pl from GitHub, copy it to /usr/local/bin, and make sure the file is executable by Nagios.

      wget https://raw.github.com/PagerDuty/pagerduty-nagios-pl/master/pagerduty_nagios.pl 
      cp pagerduty_nagios.pl /usr/local/bin
      chmod 755 /usr/local/bin/pagerduty_nagios.pl
    3. If you’re using the standalone version of Check_MK, edit the nagios user’s cron table:

      crontab -u nagios -e

      If you’re using the OMD version of Check_MK, use su to become the user for your OMD site:

      su - {site-name-here}
    4. If you’re using the standalone version of Check_MK, add the following line to the file:

      * * * * * /usr/local/bin/pagerduty_nagios.pl flush

      If you’re using the OMD version of Check_MK, create a file under your site’s cron.d directory (usually /omd/sites/{site-name-here}/etc/cron.d) and add the following line to the file:

      * * * * * /usr/local/bin/pagerduty_nagios.pl flush

      Save the file as pagerduty and run the following for the change to take effect, then return to root:

      omd reload crontab
      exit
    5. Download pagerduty-perl from GitHub and make it executable:

      wget https://gist.githubusercontent.com/jcurreee/b8f4472659d7c2e402a6/raw/5f7fff04b45a62538eda458e287b2b3e0c61156e/pagerduty-perl
      chmod +x pagerduty-perl
          

      pagerduty-perl is a notification script for Check_MK and is not the Perl script or a replacement for it. You must still keep the Perl script (pagerduty_nagios.pl) on your system as well.

    6. Move the notification script into place.

      For the standalone version of Check_MK this is usually /usr/share/check_mk/notifications:

      mv pagerduty-perl /usr/share/check_mk/notifications

      For the OMD version of Check_MK this is usually /omd/sites/{site-name-here}/local/share/check_mk/notifications:

      mv pagerduty-perl /omd/sites/{site-name-here}/local/share/check_mk/notifications
    7. Log in to the Check_MK web interface, go to Users (located in the WATO · Configuration box) and click New User.

    8. Enter a Username and, optionally, a Full name for the PagerDuty user. You may find it beneficial to set the full name to match the name of the PagerDuty service you created if you will want to configure Check_MK hosts and services to alert multiple PagerDuty services in the future.

      Do not enter a password for this user; instead you will want to check disable the login to this account, as this account exists solely to send notifications to the PagerDuty Perl script.

      Set the user’s role to Normal monitoring user, or any custom role you’ve created with permissions to send notifications, and add the user to the Contact Groups which the hosts/services you want to receive alerts for are part of. Click Save when you are done.

    9. Click the Notifications icon (broadcast tower) for the user you created. If you are using Check_MK 1.2.4 or earlier, click the Edit icon (pencil) instead.

    10. Click New Rule. If you are using Check_MK 1.2.4 or earlier, scroll down to the Notifications box instead.

    11. Enter a Description for the new notification method, then set Notification Method to PagerDuty (Perl). Paste the Integration Key you copied from PagerDuty earlier in the text box that appears once you select PagerDuty (Perl), and select any desired conditions to limit the alerts that get sent to PagerDuty. Click Save when you are done.

      If you are using Check_MK 1.2.4 or earlier, check enable notifications and set the Notification Method to Flexible Custom Notifications. Click Add notification and set the Notification Plugin to PagerDuty (Perl). Paste the Integration Key you copied from PagerDuty earlier in the first Plugin Arguments text box that appears once you select PagerDuty (Perl), then uncheck the boxes Start or end of flapping state and Start or end of scheduled downtime under Host Events and Service Events for the Notification Method (not the Notification Options). Click Save when you are done.

    12. Go back to the Users list and click # Changes, then click Activate Changes.

    13. Congratulations! When you see Configuration successfully activated you are done! Check_MK will now be able to trigger, acknowledge and resolve incidents in PagerDuty, and the cron job will re-try sending events in case they aren’t successfully sent in the first attempt (i.e. due to connectivity issues).

    Next Steps

    You can test the integration to make sure everything works as expected by going to a host or service in the Check_MK interface and clicking the Execute icon (hammer). In the Fake check results box, click Critical (if on a service) or Down (if on a host), then click Yes! to confirm you want to send the fake alert. You should see a new incident created in PagerDuty momentarily, however keep in mind that the test incident may be resolved quickly, as the fake check results are replaced by real check results on the next scheduled check.

    FAQ

    How do I configure Check_MK to work with multiple PagerDuty services?

    This is easy to do with the current integration, as a Check_MK service in PagerDuty is directly mapped to a user in Check_MK. In order to configure multiple services, just create multiple users in Check_MK with different names (i.e. pagerduty_database, pagerduty_network, etc). Then copy and paste the corresponding Integration Key from PagerDuty into the Notification Method parameters/Plugin Arguments field. Don’t forget to activate your changes for the configuration to take effect.

    What if a Check_MK event happens while my network is down?

    If a PagerDuty server can’t be reached for any reason, events will be stored to an on-disk queue. The cron job will attempt to re-send the events once every minute.

    Since Check_MK needs my external Internet connection to send failure reports to PagerDuty, how will I receive notification if our site loses external connectivity?

    You should configure an external ping check service such as StatusCake or NodePing to monitor your site’s external connectivity. Of course, you can use PagerDuty to receive alerts from these services as well.

    The integration doesn’t seem to be working. What’s going on?

    First, check syslog for messages containing pagerduty_nagios.

    For Debian-based systems:

    grep pagerduty_nagios /var/log/syslog

    For RHEL-based systems:

    grep pagerduty_nagios /var/log/messages

    What sort of Nagios messages does PagerDuty understand with the Check_MK integration?

    PagerDuty can process PROBLEMACKNOWLEDGEMENT, and RECOVERY messages. All other messages, including FLAPPINGSTART and FLAPPINGSTOP, or custom messages, are ignored.