Fortune 500 FinServ Company Automates With PagerDuty

The Infrastructure Engineering team at this Fortune 500 financial services company manages administering the dev, test, and production environments to 20,000 Linux servers distributed across many global data centers. 

The team has to comply with strict industry regulations and to stay in compliance they had to follow a tedious change management process. They sent all change management actions to the due diligence team for approval before their IT services partner could execute. Changes were made manually so the Infrastructure Engineering team had to validate the results of the service partner’s work. Further, alert noise from their monitoring tool, Sensu, was a constant distraction for engineers. It was clear, to continue scaling operations, the Infrastructure Engineering team needed to become more efficient. 

PagerDuty Process Automation was deployed to automate workflows across systems and infrastructure. Now, the service provider can call to gather system info and execute system changes—simplifying the change management process and eliminating most of the associated labor intensive work. Using Process Automation, the Infrastructure Engineering team can run automated jobs to validate that maintenance operations are completed correctly, giving back an hour each day to the engineers. “We found an opportunity to use PagerDuty Process Automation to improve the efficiency of working with our outsourced IT service partner,” said the Infrastructure Engineer Manager. 

Additionally, standardizing and automating workflows has improved the company’s security posture. Access to the workflows in Process Automation is controlled by user rolls, so login credentials and SSH keys don’t need to be shared. 

Process Automation helped reduce alert noise by automating remediation playbooks in Sensu for  common alerts. For known cases, a workflow is executed automatically to remediate the issue without escalating to a human operator. The Infrastructure Engineer Manager explained, “That’s fewer interruptions so they can focus on longer term projects and be available in the event a more serious problem arises.”

The Infrastructure Engineering team plans to expand their use of Process Automation. The next project is to enable better auditing user activity. “It’s complicated to track down executions by specific users, especially when we need to do this in a rapid timeframe. PagerDuty Process Automation will be able to simplify this for us as well,” the Infrastructure Engineer Manager shared. 

To learn how PagerDuty can help your team make things simple and transform operations in a digital-first world, contact your account manager or try a 14-day free trial today.