Inspired by Shopify’s boring update, we decided to dedicate a focused sprint this Fall to working on customer-driven improvements outside of our regular roadmap across the product development organization. This post goes behind the scenes, describes our journey, and shares some highlights from the release, as well as our learnings.

The sprint was all about customer love. It’s easy to get hyper-focused on innovations, shipping new stuff, and expanding the power of the platform (see AI agents!), but customers also have everyday pain points and friction with the core product. PagerDuty’s platform is incredibly powerful, but there are definitely areas where the product could be easier to use, address some long-standing pain points and just be more polished and delightful.

In the spirit of making many small steps that add up to something big, the timing and theme of Steptember felt right, so we called it the Step-tember customer love sprint:

Step-tember principles

Some of the principles behind the sprint:

Every team participates

We had all teams participating, including our internal platform teams. Making our developer platforms and systems easier to use, faster, and more productive helps our product teams, which in turn allows us to move faster, with higher confidence and improve the experience for customers.

Feel the friction, walk a mile in the customer’s shoes

We had some simple heuristics for teams to consider when prioritizing improvements.

No change is too small

Little pain points add up to a sub-optimal customer experience. That could be as simple as a doc link returning a 404 to a confusing error message.

Other than this simple guidance, there were really no rules; teams could get creative and fix things that bothered them, scratch itches or make enhancements they wanted to see in the product themselves.

One of the things that’s cool about working at PagerDuty is that we are also a customer! In addition to collecting and action-ing on customer feedback, we also have our own feedback, perspectives and expertise as practitioners and users of the product every day.

Gamifying customer love

In the spirit of having fun and driving some healthy competitive spirit across our teams, we decided to make a leaderboard to track and celebrate progress. Teams competed for the most improvements shipped.

We started with a challenge – can we ship 100 improvements to customers in 2 weeks?

We created a Slack channel to rally folks, share progress, and showcase demos as things got going. When we started, we had around 30 improvements planned, but things picked up quickly from there.

As the sprint progressed, the momentum started to accelerate as folks got more and more excited:

We kept beating the targets I was setting, so I kept upping the ante 😃

When we hit those too – it was time for one final throwdown:

And on the final day, we hit our goal of 150! 👏 (we actually kept shipping some more after this too 🙂)

It was incredibly energizing and motivating to watch and everyone was pleasantly surprised at the pace and progress!

Highlights

Ok, so we shipped a lot of stuff! Let’s talk about some highlights. 

I want to share just a few of the improvements that we are really proud of. There are so many to list (it was tough to pick!) but as we cataloged the 150+ improvements and updates, a few themes emerged.

Performance

Let’s start with every engineer’s favourite topic: performance 🏎️😀. The fact is, performance is a feature. This release packs a number of impressive performance improvements across the product – some improving all pages and others to specific areas of the product.

First off, you might notice all pages are a little snappier now. This is due to a series of performance improvements in base front-end logic that loads as part of all pages.

As part of another project, one of our senior engineers noticed some table scans on several core database tables as part of preparing common metadata passed to our front-end apps. He was able to optimize them away by isolating a few expensive calls to a single specific flow related to upgrading/downgrading plans during checkout. This, coupled with some other cleanup, resulted in some impressive performance gains.

Improving PagerDuty Service performance

Another significant performance improvement we made was related to loading PagerDuty services (loading paginated lists of services and service-adjacent objects, service pick lists, etc) across the product, especially for accounts with large service topologies.

This one is fun because it came down to a pure algorithm improvement, which resulted in a change in time complexity of how services are filtered based on associated permissions. Through some code optimizations, the Services and Operations Console team took the time complexity from O(n*m) to O(n+m), where n = # of services and m = # of teams. 

The team also noticed that some larger cache entries weren’t getting cached due to the configuration around max_item_size in memcached. Tweaking memcached configuration resulted in a significant improvement in performance for larger accounts. Taken together with the algorithm improvement, this resulted in >90% performance improvements in some cases! The graph speaks for itself: 🤩

Big kudos to the Services and Operations Console team for shipping these improvements! 👏

Long-standing customer requests

We’ve been focused on delivering a number of long-standing customer asks over the last year, but we wanted to ship even more during this sprint. One such feature, which has been available to a limited number of customers since 2020, but not generally available and widely requested, is the ability to reopen incidents:

One thing I frequently champion with my teams is having strong documentation and functional specifications, especially for complex areas of our product. It just so happens that the Incidents team put together some truly excellent documentation describing the original reopen implementation earlier this year – its gaps, edge cases, and caveats required to enable it for more customers.

From here, it was relatively straightforward to make a plan to address these gaps and edge cases with high confidence, culminating in a broad release during the sprint!

Big thanks to the Incidents team on shipping this long-standing feature request 🙏

Quality of Life / User Delights

Finally, I want to highlight some nice quality of life improvements that would have come in handy during our recent major incident from late August.

Status pages improvements 

One frequently requested feature from customers (including us!) was the ability to retroactively set a post time for status page posts. As we discussed in our incident review, it would have been extremely helpful to have had this handy then, as we posted later than we would have liked.

The team also made the page auto-refresh too!

A big kudos to the Status Page team for shipping these and other improvements. Incidentally, they also took the top spot for the team that shipped the most improvements across the organization! 🎉🥇

Reducing noise in Slack

The chat experience team made some nice quality-of-life improvements to the Slack experience as well.

Subsequent updates to incidents can now be threaded to reduce noise and improve clarity in both notification and dedicated incident channels.

The team also improved the verbosity of messages by adding embedded responder links, rather than fully rendering out links.

As an incident commander, I LOVE these improvements ♥️The more streamlined and organized our chats can be during incidents, the easier it makes my job!

Kudos to the Chat Experience team. Also stay tuned, there are a lot more improvements coming to chat real soon 😀

AI that works alongside you 🧠

What post about improvements would be complete without talking about AI? 😂 

As part of our H2 product launch, we also released a set of AI Agents that can work alongside you, directly reducing toil before, during and after an incident.

So far our customers have had a very positive experience using these agents, though our partners in Japan quickly pointed out that it would be even better if they could interact with the PagerDuty Agents in their native language.

Leveraging multilingual LLMs and building on previous work done to set up our evaluation framework, our Applied AI and Japan Professional Services teams partnered to make this happen. Adding support for an additional language can be time-consuming because, to do so safely, we need to ensure we have a robust way to test agent performance on the new language and that all supported languages provide an equally positive user experience.

Our evaluation framework reduced that complexity significantly, streamlining these iterative tests and allowing our AI teams and language experts to quickly validate that our agents’ performance in Japanese met our requirements and get things shipped within our sprint.

The result: users are now able to interact with Shift and Insights Agents in Japanese, with other agents to follow soon. This work also paves the way for future language additions.

Conclusion

We exceeded our expectations for this sprint. Everyone really leaned in and was super excited to make improvements! We had so much fun and one thing everyone agrees on is that we will definitely do this again, along with bringing this spirit and energy back into our day-to-day work. 

A HUGE thank you to all our teams across the whole company who helped make this release so successful and special. 

It’s easy to keep your head down shipping the next innovation or new feature, but the core flows of the product are where users spend the most time. Making small improvements in these areas adds up and makes a really big difference.

Making people’s lives easier has always been at the core of what we do at PagerDuty. We’re energized by this work and excited to see all these improvements get into the hands of our customers. 

Finally, please give us feedback! We introduced a user feedback box on every page of our product earlier this year.

If you have feedback on the product, please share it. It doesn’t matter how small it is – you might be delighted to see that improvement happen! We are listening 😁