SignalFx provides real-time monitoring and operational intelligence for today’s dynamic environments. Customers count on SignalFx to discover and collect metrics and monitor the performance and stability of applications deployed in the cloud. To guarantee the reliability and uptime of its own monitoring services, SignalFx relies on PagerDuty ‘s integration with SignalFx’s Intelligent Alerting feature. PagerDuty provides engineers at SignalFx with the incident notification, IT alert management, and response orchestration capabilities required to identify and respond to problems before customer experience is impacted.
Challenges: Automating Incident Management and Team Collaboration
SignalFx leveraged PagerDuty from the very beginning to help provide the end-to-end incident lifecycle capabilities it needed to deliver highly available monitoring services on a massive scale. “Even before we had any customers, we started using PagerDuty,” Jothikumar explained.
Jothikumar said that SignalFx adopted PagerDuty because the solution was recognized as the industry leader among incident response platforms. His team looks forward to continuing to grow with PagerDuty because of the platform’s scalability as well as extensive automation features.
Streamlining Communication Through Automation
SignalFx fully embraces the DevOps approach to software delivery and management, which means that every engineer at SignalFx is responsible for monitoring the company’s services and keeping its website available. In order to ensure that all of the engineers at SignalFx are able to communicate effectively and respond quickly and accurately to incidents, while they are geographically dispersed, automation is crucial.
With PagerDuty, Jothikumar and his colleagues can configure schedules and escalation policies, as well as centralize dynamic rules for event notification and behavior at scale, then let PagerDuty do the rest. “Once the schedules and the policies are set in place, everything is automatic,” Jothikumar said. “If an alert fires, it goes to engineer A, or on-call policy A. And if that on-call policy person doesn’t respond within a certain time, it goes to the next person.”
He added that PagerDuty delivers, “fully automated, well-oiled wheels” to drive SignalFx’s operations in fixing issues and delivering highly resilient services.
Scheduling DevOps Engineers
SignalFx uses PagerDuty to manage scheduling for all of its engineering teams, too. Organizing the on-call schedules of DevOps engineers is a complicated and tedious process if performed manually, and mis-managed scheduling information has significant consequences to the company. Luckily, PagerDuty automates all their on-call management needs, and is flexible enough to support custom rotations, Jothikumar said.
“One of the challenges that we have is in the scheduling area—Who do you schedule where, and how do you schedule people?” Jothikumar explained. Thanks to PagerDuty, SignalFx is able to schedule engineers to be on-call, shift on-call duties when scheduling priorities change, and move engineers from one incident management role to another in an automated fashion.
Maintaining Industry-Leading Response Times with PagerDuty
As a company that delivers monitoring services itself, SignalFx has to guarantee an extremely high level of uptime to its customers or risk losing them. PagerDuty plays a key role in allowing the company to maintain their customers’ trust.
With PagerDuty, SignalFx can proactively detect and respond to incidents and mitigate customer impact — while maintaining industry-leading response times. In doing so, it meets and often exceeds its contractual obligations under the service-level agreements (SLAs) that it upholds with customers.
Jothikumar says PagerDuty integrated with SignalFx’s real-time monitoring system which allows his team to, “take action before the issue becomes critical, all because PagerDuty alerts the right people at the right time. If we didn’t have PagerDuty integrations with SignalFx’s Intelligent Real-Time Alerting, we wouldn’t be able to reach our SLA goals.”
“With PagerDuty, once the schedules and the policies are set in place, everything is automatic. If an alert fires, it goes to engineer A, or on-call policy A. And if that on-call policy person doesn’t respond within a certain time, it goes to the next person. PagerDuty provides fully automated, well-oiled wheels for incident alerting and response”