What is a Runbook?

Life in operations is a mix of planned and unplanned work. Teams often face incidents or tickets without a known fix. Finding one could mean a quick Google search, digging through the company wiki or docs, hunting for shared scripts, asking a coworker, or escalating the issue to another department. That can take hours and still produce a solution that may not reflect the company’s best practice.

Runbooks solve this problem. A runbook gives the operator standardized, detailed instructions for resolving common issues and tasks quickly and consistently.

What is a runbook?

A runbook is a detailed “how-to” guide for completing a commonly repeated task or procedure within a company’s IT operations process, such as provisioning, software updates and deployments, configuration changes, and opening ports. Runbooks give everyone on the team, new or experienced, the knowledge and steps to resolve a given issue quickly and accurately. For example, a runbook may outline routine tasks such as patching a server or renewing a website’s SSL certificate.

Think of a runbook as a recipe. It provides detailed instructions for completing a specific task quickly, based on how the team has resolved it before. Runbooks let experienced team members share their knowledge so newer members can resolve common issues on their own. They also let everyone refresh their memory and follow detailed steps without memorizing countless procedures.

When should runbooks be used?

Runbooks are especially useful for incident response. Runbooks for specific incidents turn knowledge that would otherwise stay in experts’ heads into a shared resource. With detailed, up-to-date runbooks, teams escalate less often and companies can often run smaller on-call IT teams.

Runbooks also support day-to-day IT operations like regular maintenance of IT systems and applications. For example, a runbook can outline common tasks such as creating database backups or updating access permissions.

Runbooks fall into three types based on how much operator involvement they require:

Runbook type

Operator involvement

Best for (per page)

Manual

Step-by-step instructions followed by the operator

 

Semi-automated

A combination of operator-followed steps with automated steps

 

Fully automated

All steps are automated and require no operator

Running a website audit

What is the difference between a runbook and a playbook?

Runbooks and playbooks are often confused, but they serve different purposes. A playbook covers the overall response to larger issues and events, and it can include multiple runbooks and team members as part of the complete workflow.

Going back to the earlier analogy, if a runbook is a recipe, a playbook is the guide to hosting the whole event. The recipe helps you cook the meal, but the food is just one part of the event.

The playbook covers the big picture, while runbooks outline the individual tasks.

Creating a runbook template for your company

Building a runbook template comes down to three steps: planning it, writing it, and testing it over time.

Step 1: Planning a new runbook

When planning a new runbook, consider two questions:

  1. What are the most common incidents or tasks your team faces?
  2. What solutions have worked best for handling them in the past?

Detailed incident reports and postmortems show where a runbook would help most. Your ticketing system can also reveal common, recurring tasks assigned to your team. Adding runbooks for these tasks and issues speeds up operations and improves accuracy.

For example, if your team regularly renews a website’s SSL certificate, a runbook for that task gives the operator detailed instructions for completing it correctly and quickly.

Once you’ve identified a task that needs a runbook, find and document the best solution. Review the same incident reports and postmortems to see how the task was resolved in the past and which approach was most efficient and accurate. Experts who have handled the issue before can also share useful insights, so document what they consider best practice. The runbook should present the agreed-upon solution clearly for the operator.

Step 2: Write your runbook

Once you’ve determined the procedure, start documenting it. Keep these guidelines in mind:

  • Keep it clear and simple, and leave out unnecessary details
  • Use language that is easy to understand and follow
  • Make it specific and unique to your processes
  • Make it flexible enough to adapt to changes in your systems and applications

Keep your runbooks consistent across all applications. Structure each one the same way, with consistent naming and headers, and give the operator all the details they need.

Once the runbook is complete, field test the documented process and update it as needed.

Tom Limoncelli, an author and former Google sysadmin, recommends seven sections for every runbook:

#

Runbook section

Description (per page)

1

Service overview

What the service is, why it exists, primary contacts, and how to report bugs

2

Service build information

How to build the software that runs the service

3

Instructions for deploying the software

How to deploy the software and build a server from scratch

4

Instructions for common tasks

Step-by-step instructions for tasks like provisioning, plus common problems and their fixes

5

“Pager playbook”

An outline of every possible monitoring system alert and step-by-step instructions for when they are triggered

6

Disaster recovery plans

Procedures for failing over to a spare if a service machine dies

7

Service level agreement

The contract with customers, including uptime goals, recovery point objective (RPO), and recovery time objective (RTO)

You can read more about these sections in Limoncelli’s Ops Report Card.

Step 3: Test, update, and improve your runbooks

A runbook isn’t something you set and forget. Test and update it regularly so it keeps working as your systems and applications change.

PagerDuty Runbook Automation turns documented runbooks into automated workflows you can run on demand, trigger from events, or delegate as self-service tasks. Start a free Runbook Automation trial to automate your team’s most common procedures.