SolarWinds Orion platform products, such as Network Performance Monitor (NPM) and Server & Application Monitor (SAM), provide network, performance, availability and server monitoring for multi-vendor networks, Linux and Windows Server systems, and detailed application performance monitoring metrics for over 200 applications such as Apache HTTP Server, Microsoft Exchange, and Oracle Database. Both NPM and SAM allow you to trigger and reset alerts based on custom thresholds that can trigger and resolve incidents in PagerDuty.
This integration works by logging SolarWinds Orion alerts to a file containing JSON formatted for our Events API. A VBScript then iterates through every file in the queue directory where alerts are written to and sends the JSON to PagerDuty, deleting the alert file after it has been accepted by the server.
This guide and VBScript were written and tested on Windows Server 2012 R2 with SolarWinds Orion platform 2015.1 and 2016.1. Please see our Legacy SolarWinds NPM Integration Guide if you have an older version of NPM. Earlier versions of SAM/APM and SolarWinds products not using the Orion platform can be configured to send alerts to PagerDuty using an email integration instead.
Go to the Configuration menu and select Services.
On the Services page:
If you are creating a new service for your integration, click Add New Service.
If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then click the Integrations tab and click the New Integration button.
Select your app from the Integration Type menu and enter an Integration Name.
If you are creating a new service for your integration, in General Settings, enter a Name for your new service. Then, in Incident Settings, specify the Escalation Policy, Notification Urgency, and Incident Behavior for your new service.
Click the Add Service or Add Integration button to save your new integration. You will be redirected to the Integrations page for your service.
Copy the Integration Key for your new integration.
Create the following directories on your SolarWinds server’s
Download the following VBScript and sample alerts, saving the VBScript to
C:\PagerDuty on your SolarWinds server, and the sample alerts anywhere you like on your workstation or SolarWinds server.
Sample Alerts.zip on your workstation or SolarWinds server (i.e. on your desktop,
Downloads folder, or
Log in to SolarWinds as an admin, then go to the Settings menu and select All Settings (or click Settings in the top right corner of the page if you have an older version of SolarWinds). Click Manage Custom Properties under the Node & Group Management section.
Click Add Custom Property.
Select Nodes and click Next.
Set the Property Name to
PDIntegrationKey and, optionally, set a Description, such as “PagerDuty Service Integration Key,” then click Next.
Click Select Nodes and check all the nodes in the Available Nodes column that you want to receive alerts for in PagerDuty, click Add to move the nodes to the Selected Nodes column, then click Select Nodes.
Paste the Integration Key you copied earlier in the box under Assign value for “PDIntegrationKey” to selected Nodes, then click Submit.
Go back to the Main Settings & Administration page by going to the Settings menu and selecting All Settings (or click Settings in the top right corner of the page if you have an older version of SolarWinds). Click Manage Alerts under the Alerts & Reports section.
In SolarWinds, click Export/Import to import some of the most popular alerts we’ve created templates for. Click Import Alert and select the alert definitions you would like to import to SolarWinds from the location you extracted the
Sample Alerts.zip file to earlier (i.e. on your desktop,
Downloads folder, or
Common alerts that can be used for both NPM and SAM are in the main
Sample Alerts folder, while SAM-specific alerts are in
Sample Alerts\SAM. Note that only one alert can be imported at a time.
Open Windows Task Scheduler and click Create Task… on the Actions sidebar. Give the task a Name, like PagerDuty, and select Run whether user is logged in or not.
Switch to the Triggers tab and click New…, then click the drop down list for Begin the task and select At startup. Now check the box labeled Repeat task every: and type in 1 minute (this option isn’t in the list, you must manually type it in), then change the duration to Indefinitely and click OK. Repeat these steps, only selecting At task creation/modification the second time around.
Select the Actions tab and click New…. Set the Program/script to
C:\Windows\System32\cscript.exe and Add arguments to
//B "C:\PagerDuty\PagerDuty.vbs", then click OK.
Congratulations! SolarWinds will now be able to trigger and resolve incidents in PagerDuty, and the scheduled task will re-try sending events in case they aren’t successfully sent in the first attempt (i.e. due to connectivity issues).
You can send additional alerts to PagerDuty by selecting the alert you want to use and going to Trigger Actions, then simply add an action using the Log the Alert to a File option. The file should be logged to
C:\PagerDuty\Queue with a
.txt file extension, and the content should be JSON that is formatted for our Events API. Check out the alert definitions we provide for examples of how to write your own alerts to be sent to PagerDuty.
Special Characters: you must avoid including quotation mark characters in the alert template. These will cause string termination in the output, resulting in invalid JSON syntax, which will result in the events failing to submit to the Events API. If you need to include quotation marks inside of the template, then it is highly recommended that you use curly quotes, i.e. ” and “, rather than straight quotes (").
Lastly, you’ll want to add an action to Execute an External VB Script and call
"C:\PagerDuty\PagerDuty.vbs" using the
CScript.exe interpreter. This allows SolarWinds to send the alert to PagerDuty immediately, rather than having to wait for the scheduled task to run before it gets sent to us.
Yes! Simply go to the details page for the node you want to trigger incidents on a different PagerDuty service, scroll down to the Custom Properties section, then click Edit Custom Property Values to set a different value for
PDIntegrationKey on that node. No need to change your alerts (unless you’ve hard-coded the integration key for your custom alerts), as the templates we provide use a SQL query to pull the integration key from this custom property on each node.
First, make sure you’re executing the VBScript using the
CScript.exe interpreter, rather than
WScript.exe, and that the script filename is in quotation marks in both your scheduled task and SolarWinds triggers.
The script will log errors to the Windows Application Event Log to help resolve problems with alerts not being sent to PagerDuty. You can also manually execute the VBScript via Command Prompt using
CScript.exe "C:\PagerDuty\PagerDuty.vbs" to see what happens to alert files in the queue step-by-step.
If you see any status 400 (Invalid Request) errors, examine the contents of the file in question. If you see the following, check to see if you have the PDIntegrationKey custom field configured for the node or host in question on which the alert was raised:
If you would like the VBScript and alert queue directory to reside somewhere other than
C:\PagerDuty, you will need to perform a find and replace on the XML alert definition files before importing them to change the path to your desired location before proceeding. Unfortunately this path must be hard coded in the alert definition, as SolarWinds does not currently support the execution of VBScript or writing alert files to directories using Windows environment variables such as
You will also need to modify the
QueuePath variable in the VBScript from
C:\PagerDuty\Queue to point to the location where you would like the alert queue to reside.
Although SolarWinds can execute VBScript from a shared folder or mapped network drive, we recommend using a local copy and alert queue on each system in case the host with the script becomes unavailable. This is also recommended in the SolarWinds knowledge base: Tips for executing external scripts and batch files with alerts.
For example, if you configure SolarWinds to write alerts to a queue on a file server and execute the VBScript hosted on that server, and that server goes down for any reason, SolarWinds will not be able to write alert files or execute the VBScript to trigger an incident in PagerDuty for any SolarWinds alerts until the file server is available again.
If PagerDuty can’t be reached for any reason, events will be stored in
C:\PagerDuty\Queue. The scheduled task to execute the VBScript will attempt to re-send the events at one minute intervals, so your SolarWinds server should begin sending incidents to PagerDuty within a minute of being able to access the internet again.
If this is happening, check the permission on the files. It may be that SolarWinds is creating the files with permissions such that the scheduled task cannot remove them, but can read them.