Customers / Invoca
  • Size
  • 131 employees

  • Industry
  • Computer Software

  • Location
  • Santa Barbara, CA

  • Customer Since
  • 2012

Challenges

  • Configuring the right escalation policies for incident notifications to ensure the right expert(s) take action before customers are impacted
  • Separating mission-critical events that require immediate attention from those that are less urgent

Business Impact

  • Since Invoca’s service became available to customers, the company has experienced an ever decreasing number of customer impacting service disruptions with the help of PagerDuty

Benefits With PagerDuty

  • Notifications are automatically escalated through the appropriate chain until they are addressed
  • Engineers are able to distinguish easily between incidents requiring immediate attention and those that can wait until normal business hours
  • Scheduling on-call engineers is now simple, flexible, and automated

Invoca Expands Use of PagerDuty to Manage Incident Urgency and Escalation

Invoca’s call intelligence platform provides attribution, analytics and optimization tools to help marketers drive more revenue and a better customer experience. A vital part of the company’s product offering is the guaranteed quality of phone calls, which makes resolving technical incidents before they negatively impact call quality absolutely essential for Invoca. While PagerDuty has been a part of Invoca’s software solutions stack since the company’s beginning, James Brown, Senior Manager for Cloud Operations at Invoca, and his team now use PagerDuty to manage incident urgencies and notification behavior in the most effective and programmatic way.

Challenges: Incident Urgency Levels and Escalations

Guaranteeing the highest quality of service is an absolute must for Invoca. “We’re all about phone calls, so the one thing we absolutely cannot have is degradation of phones calls,” said Brown. To help prevent call degradations and disruptions, Invoca uses a modified version of Sensu to monitor its production infrastructure, along with a custom IT alerting system that analyzes monitoring data to detect problems. Yet on its own, detecting problems is not enough. In order for Brown’s team to address problems in a timely fashion, two additional steps are required. First, notifications have to reach someone who is ready and able to respond to an incident. In the event that the on-call support engineer is not available, the notification must be escalated to someone else who can respond before service is disrupted. Second, Brown and his team must be able to distinguish critical alerts from less serious ones. The ability to categorize alerts based on urgency, group related alerts into incidents, and suppress anything that’s non-actionable is essential for making sure that the most severe incidents are addressed immediately, no matter when they occur.

Using PagerDuty to Escalate Notifications, Set Urgencies and Reduce Alert Noise

PagerDuty plays an instrumental role in helping Invoca to address both of these challenges. By being able to ensure that notifications for critical incidents are escalated appropriately until an engineer responds, the company has experienced a dramatic decrease in customer-impacting service disruptions, “we have never had a call outage because the monitoring and alerting allows us to catch any problems well before they impact customers,” Brown said. He added that PagerDuty is particularly helpful with escalations because it allows his team to configure different escalation policies for different types of incidents. “We wanted the ability to have different escalation paths for different kinds of problems, and PagerDuty made that possible,” said Brown.

In addition, Invoca benefits from the ability to configure varying incident urgency settings in PagerDuty in order to differentiate issues that can wait to be addressed during business hours from those that require immediate attention. The company also uses PagerDuty to allow its partners to page a senior engineer in the event that they experience a major disruption.

PagerDuty enables Brown’s team to sort false positives out of its alerts in order to ensure that engineers are not distracted and can minimize unplanned work. “One thing that PagerDuty has really helped us with is reducing noise and keeping the false positives way, way down,” said Brown.

Making the Most of PagerDuty’s Advanced Features

Brown explained that, shortly after Invoca’s founding, the company adopted PagerDuty because it was the only hosted alerting system that could meet its needs. While Invoca’s engineers at first used PagerDuty only for basic tasks such as routing phone calls and sending email notifications, the value they’ve gained from PagerDuty has exponentially grown as they have expanded their use of the platform to include advanced, end-to-end incident lifecycle capabilities. From event intelligence, to on-call management, to incident prioritization, to response orchestration at scale, and more, PagerDuty’s advanced capabilities now help Invoca’s IT Ops team completely avoid critical service disruptions. Brown’s team even uses PagerDuty data to perform post mortem analysis of its incident response, system health, and operational performance trends by tracking metrics like mean-time-to-respond (MTTR) and number of incidents on critical services.

In short, “when we started leveraging PagerDuty’s advanced features, we really saw the benefits. Reducing false positives and identifying problems in order to address them before they become fires is really useful,” stated Brown.

“When we started leveraging PagerDuty's advanced features, we really saw the benefits. Reducing false positives and identifying problems in order to address them before they become fires is really useful.”

AAEAAQAAAAAAAAYwAAAAJDNlYTcyMWVhLTEyMTQtNDc4NS05ZmMwLTEwNmM5MjliY2Y4ZA
James Brown, Senior Manager for Cloud Operations at Invoca