SolarWinds Integration Guide

SolarWinds Network Performance Monitor (NPM) and Server & Application Monitor (SAM), formerly SolarWinds APM, provide network, performance, availability and server monitoring for multi-vendor networks, Linux and Windows Server systems, and detailed application performance monitoring metrics for over 200 applications such as Apache HTTP Server, Microsoft Exchange, and Oracle Database. Both NPM and SAM allow you to trigger and reset alerts based on custom thresholds that can trigger and resolve incidents in PagerDuty.

This integration works by logging SolarWinds alerts to a file containing JSON formatted for our Integration API. A VBScript then iterates through every file in the queue directory where alerts are written to and sends the JSON to PagerDuty, deleting the alert file after it has been accepted by the server.

Compatibility

This guide and VBScript were written and tested on Windows Server 2012 R2 with SolarWinds 2015.1.2. Please see our Legacy SolarWinds NPM Integration Guide if you have an older version of NPM. Earlier versions of SolarWinds SAM/APM can be configured to send alerts to PagerDuty using an email integration instead.

In PagerDuty

  1. Create a new Service by going to the Configuration menu and selecting Services, then click Add New Service.

  2. Enter a Name for your new service, select SolarWinds from the Integration Type menu, choose an Escalation Policy to use when the service receives an alert from SolarWinds, and select the desired Notification Urgency for incidents (available only with Basic or higher plans). Click Add Service when you are finished.

  3. Copy the Integration Key for your new service:

On Your SolarWinds Server

  1. Download the following VBScript and SolarWinds alert definitions, then extract the file to C:\PagerDuty.

    PagerDuty for SolarWinds.zip
    30 KB

  2. Log in to SolarWinds as an admin, then click Settings on the top right corner of the page. Click Manage Custom Properties under the Node & Group Management section.

  3. Click Add Custom Property.

  4. Select Nodes and click Next.

  5. Set the Property Name to PDIntegrationKey and, optionally, set a Description, such as “PagerDuty Service Integration Key,” then click Next.

  6. Click Select Nodes and check all the nodes in the Available Nodes column that you want to receive alerts for in PagerDuty, click Add to move the nodes to the Selected Nodes column, then click Select Nodes.

  7. Paste the Integration Key you copied earlier in the box under Assign value for “PDIntegrationKey” to selected Nodes, then click Submit.

  8. Go back to the Main Settings & Administration page by clicking Settings on the top right corner of the page. Click Manage Alerts under the Alerts & Reports section.

  9. Click Export/Import to import some of the most popular alerts we’ve created templates for. Click Import Alert and select the alert definitions you would like to import to SolarWinds from C:\PagerDuty\Alerts.

    Common alerts that can be used for both NPM and SAM are in the Alerts folder, while SAM-specific alerts are in Alerts\SAM. Note that only one alert can be imported at a time.

  10. Open Windows Task Scheduler and click Create Task… on the Actions sidebar. Give the task a Name, like PagerDuty, and select Run whether user is logged in or not.

  11. Switch to the Triggers tab and click New…, then click the drop down list for Begin the task and select At startup. Now check the box labeled Repeat task every: and type in 1 minute (this option isn’t in the list, you must manually type it in), then change the duration to Indefinitely and click OK. Repeat these steps, only selecting At task creation/modification the second time around.

  12. Select the Actions tab and click New…. Set the Program/script to C:\Windows\System32\cscript.exe and Add arguments to //B "C:\PagerDuty\PagerDuty.vbs", then click OK.

  13. Click OK.

  14. Congratulations! SolarWinds will now be able to trigger and resolve incidents in PagerDuty, and the scheduled task will re-try sending events in case they aren’t successfully sent in the first attempt (i.e. due to connectivity issues).

Next Steps

You can send additional alerts to PagerDuty by selecting the alert you want to use and going to Trigger Actions, then simply add an action using the Log the Alert to a File option. The file should be logged to C:\PagerDuty\Queue, and the content should be JSON that is formatted for our Integration API. Check out the alert definitions we provide for examples of how to write your own alerts to be sent to PagerDuty.

Special Characters: If you have an alert which uses characters that will affect the JSON output, such as a quotation marks, then you will need to escape those special characters. For example, if you add quotation marks to an alert yourself, add a backslash (\) before them so they look like \". Other special characters can be escaped if needed by modifying the VBScript and using the Replace method, which is what we use in the script to escape backslashes, as backslashes have a special purpose in JSON as the escape character, but are also used in alerts containing Windows paths.

Lastly, you’ll want to add an action to Execute an External VB Script and call "C:\PagerDuty\PagerDuty.vbs" using the CScript.exe interpreter. This allows SolarWinds to send the alert to PagerDuty immediately, rather than having to wait for the scheduled task to run before it gets sent to us.

FAQ

Can I trigger incidents on multiple PagerDuty services?

Yes! Simply go to the details page for the node you want to trigger incidents on a different PagerDuty service, scroll down to the Custom Properties section, then click Edit Custom Property Values to set a different value for PDIntegrationKey on that node. No need to change your alerts (unless you’ve hard-coded the integration key for your custom alerts), as the templates we provide use a SQL query to pull the integration key from this custom property on each node.

Why isn’t an alert file created or the VBScript executed when I click the Simulate button?

This is a known issue with the SolarWinds interface and is not unique to this integration. On simulation dialogs where you see both Simulate and Execute buttons, you’ll need to click Execute in order for the action to actually be executed.

I’m not receiving alerts in PagerDuty. How can I check for errors?

First, make sure you’re executing the VBScript using the CScript.exe interpreter, rather than WScript.exe, and that the script filename is in quotation marks in both your scheduled task and SolarWinds triggers.

The script will log errors to the Windows Application Event Log to help resolve problems with alerts not being sent to PagerDuty. You can also manually execute the VBScript via Command Prompt using CScript.exe "C:\PagerDuty\PagerDuty.vbs" to see what happens to alert files in the queue step-by-step.

Can I put the VBScript and write alert files somewhere other than C:\PagerDuty?

If you would like the VBScript and alert queue directory to reside somewhere other than C:\PagerDuty, you will need to perform a find and replace on the XML alert definition files before importing them to change the path to your desired location before proceeding. Unfortunately this path must be hard coded in the alert definition, as SolarWinds does not currently support the execution of VBScript or writing alert files to directories using Windows environment variables such as %SystemDrive%.

You will also need to modify the QueuePath variable in the VBScript from C:\PagerDuty\Queue to point to the location where you would like the alert queue to reside.

Can I write alerts to or run the VBScript from a shared folder or mapped network drive (i.e. to use with multiple SolarWinds servers)?

Although SolarWinds can execute VBScript from a shared folder or mapped network drive, we recommend using a local copy and alert queue on each system in case the host with the script becomes unavailable. This is also recommended in the SolarWinds knowledge base: Tips for executing external scripts and batch files with alerts.

For example, if you configure SolarWinds to write alerts to a queue on a file server and execute the VBScript hosted on that server, and that server goes down for any reason, SolarWinds will not be able to write alert files or execute the VBScript to trigger an incident in PagerDuty for any SolarWinds alerts until the file server is available again.

What if a SolarWinds alert is generated while my network is down?

If PagerDuty can’t be reached for any reason, events will be stored in C:\PagerDuty\Queue. The scheduled task to execute the VBScript will attempt to re-send the events at one minute intervals, so your SolarWinds server should begin sending incidents to PagerDuty within a minute of being able to access the internet again.