How To Integrate Librato with PagerDuty
Librato is a powerful service that allows you to collect and visualize metrics for your critical IT systems. With PagerDuty integration, you can now use Librato’s alerts to automatically trigger PagerDuty incidents so that your team can be promptly notified about potential issues with your systems. The PagerDuty integration is quick and easy to setup, just follow this guide to get your Librato and PagerDuty accounts connected.
If you are having trouble completing the installation, please contact us.
If you don’t already have a PagerDuty Generic API service, you should create one:
- In your account, under the Services tab, click “Add New Service”.
- Enter a name for the service and select an escalation policy.
- Start typing “Librato” under “Integration Type” to filter your choices.
- Click the “Add Service” button.
- Once the service is created, you’ll be taken to the service page. On this page, you’ll see the “Service API key”, which will be needed when you configure your Librato account to send events to PagerDuty.
- Go to your Account Settings page (link at the bottom of the page).
- Click on the Services icon (bullhorn).
- Click on PagerDuty.
- Click the (+) next to Configured Services to add a service and the “New Pagerduty Service Configuration” panel will appear.
- Give the service a name, for example “Pagerduty Hook”.
Enter the PagerDuty configuration details:
- Service Key: The Service API Key you copied from the PagerDuty services page (REQUIRED).
- Event Type: Set this to “trigger” (REQUIRED).
- Description: A short description of the problem that led to this trigger. This field (or a truncated version) will be used when generating phone calls, SMS messages and alert emails. It will also appear on the incidents tables in the PagerDuty UI (REQUIRED).
- Incident Key: Identifies the incident to which this trigger event should be applied. If there’s no open (i.e. unresolved) incident with this key, a new one will be created. If there’s already an open incident with a matching key, this event will be appended to that incident’s log. The event key provides an easy way to “de-dupe” problem reports. If this field isn’t provided, PagerDuty will automatically open a new incident with a unique key (OPTIONAL).
- After creating the PagerDuty service you can set up alerts. Click on the Metrics tab and select the metric you want to set the alert on.
- Click on the Alerts icon (bell) on the left of the page.
- In the pull down menu you will find the service hook you just created. Select that service and enter a value for the Minimum and / or Maximum Threshold.
- As soon as any source in the graph hits a threshold it will trigger the alert and send the payload to PagerDuty.
- You can then view the details for the incident from the Details pane by selecting view message.
- With Librato’s latest feature, incidents that are triggered from a Librato service will auto resolve when they receive a clear from the Librato to UI.
How do I setup Librato to work with multiple PagerDuty services?
This is easy to do with the current integration. Each PagerDuty service within Librato can be tied to a different service API key. You would need to make multiple PagerDuty services within PagerDuty and Librato.
Can Librato support automatic clearing of triggered alerts?
Librato now supports automatic clearing of triggered incidents. What this means for PagerDuty integrations, is that open incidents for a Librato service will be resolved when they receive a clear from the Librato UI. Webhook integrations will contain a “clear” attribute in the payload.
On Librato’s end an alert goes into a triggered state when a metric/source violates an alert’s conditions.When all of the faulted signals return to normal values an alert will clear, and a clear notification will be sent.
For threshold conditions, a normal state means that it no longer violates the threshold. For absent conditions, a normal state means that the signals started reporting again after an absence. While in a triggered state, a notification will be sent every re-notify interval until the alert clears. Disabling an alert will clear any triggered state. A new “resolve” button on the alert page can be used to manually clear alerts in a triggered state. When an alert transitions from triggered to cleared, the re-notify timer is reset. Note that alert clearing is only supported for new standalone alerts launched earlier this year.
It doesn’t seem to be working. What’s going on?
Please contact us if you’re unable to sort out the difficulty.