Runbook automation: when to script it and when to keep the commands

You've written a runbook, your team has used it a few times, and someone suggests turning it into a script. Fair enough - copying the same service name into six commands every time you get paged gets old pretty quickly.

Though if the script needs half an hour of setup before anyone can use it, you haven't helped the next person getting paged.

In this article, we're going to work through which steps to automate, and what your script needs to tell the person running it.

Table of Contents

• What is runbook automation
• Start with a runbook your team uses
• Automate the checks first
• Explain what the script is doing
• Tell people whether they can run it again
• Keep the instructions
• Have someone else try it

What is runbook automation

Runbook automation means turning the steps in an operational procedure into a script or workflow. That might be collecting logs, changing a service's configuration, or checking whether it recovered after a change.

You can still have a person start the script. You don't need to connect it to your alerts and let it change production automatically.

If you haven't documented the procedure yet, start by writing your first runbook. You'll have a much easier time automating something once your team can explain how it works.

Start with a runbook your team uses

Look at the last few incidents your team handled. Which steps did people repeat? Where did they get stuck? Did they have to ask the author what a command was supposed to do?

For example, you might have a procedure where someone copies a service name into several commands, gathers the output, and pastes it into the incident channel. A script could take the service name once, check that it exists in the chosen environment, and gather those results.

On the other hand, if your runbook contains one command using a tool everyone already has installed, it might be fine as it is.

The number of commands doesn't tell you much about whether you should automate them. A script that needs its own runtime, dependencies, and credentials can take longer to set up than the procedure it replaces.

Don't throw away a working runbook while waiting for someone to finish the automation. Track the remaining work and keep the instructions usable in the meantime.

Automate the checks first

Say your team gets paged because jobs are taking too long to process. Your runbook asks the on-call engineer to check:

1. How many jobs are waiting, and how old the oldest job is.
2. How many workers are running.
3. Whether the workers are reporting errors.
4. Whether someone recently deployed a change.

These are useful steps to automate first. You can collect the information without having the script decide how to fix the problem. Keep those queries small enough that they don't add significant load to a struggling service.

Make sure a failed check is shown as a failure. If the script couldn't retrieve the worker count, printing an empty result leaves the reader guessing whether there are no workers or the check didn't work.

Once your team understands the problem and the usual fix, you can automate that too. For example, a script could show the current worker count, ask for the new count, and confirm the environment before making the change.

The runbook still needs to explain when adding workers would help. If they're all waiting for an overloaded database, adding more could make the incident worse.

Explain what the script is doing

Now imagine you're running the script at 2am. It hasn't printed anything for five minutes.

Is it stuck? Is it still working? Has it changed anything yet?

Give the person running it enough information to answer those questions. For example:

Environment: production
Queue: invoice-processing
Changing worker count from 4 to 6.
Configuration updated.
Waiting for workers: 5 of 6 ready. Elapsed: 45 seconds.

If it times out while waiting, tell the reader that the configuration was already updated. Otherwise, they might assume nothing happened and try the whole procedure again.

Include useful errors, too. "Couldn't read the worker count: permission denied" gives someone a place to start. "Something went wrong" means they now have to debug your script as well as the incident. Keep credentials and sensitive data out of that output.

Tell people whether they can run it again

If a script stops halfway through, the next question is usually "can I just run it again?"

Where possible, write it so repeating the command doesn't repeat the change. For example, setting the worker count to six is easier to retry than adding two workers each time.

When a procedure can't safely be repeated, explain how to check which steps completed before continuing.

The same goes for stopping a script. Pressing Ctrl+C might stop it waiting for a remote operation, while that operation continues running. It won't necessarily undo anything. Put that information in the runbook, along with how to check the result and undo the change if needed.

Keep the instructions

Your runbook should still explain when to use the script, how to run it, and what to do if it fails.

Link to the script and include the command to run, with the inputs clearly marked. Explain what successful output looks like. If there's a manual fallback, say when to use it and how to check for changes the script already made.

You don't need to copy the script's entire implementation into the document. You'll just end up maintaining two versions of the procedure.

You should also make sure folks have the required tools and permissions before their first on-call shift. Finding out that you need someone to approve access is frustrating enough during the workday, let alone when they're asleep.

Have someone else try it

As with writing better runbooks, someone who didn't write the script will notice assumptions you've missed.

Have them follow the instructions in a test environment. Try it with missing permissions, an unavailable dependency, and a timeout after a change has already been made. Check whether they can work out what happened from the output.

Record when it was last tested and which version was used. Test it again after changing the procedure or its dependencies. For scripts you rarely use, schedule a rehearsal rather than waiting for an incident to find out they've stopped working.

Finally, check whether the original problem went away. In the queue example, six healthy workers aren't enough: jobs should be processing again and the backlog should be shrinking.

That's the result your runbook should tell people to look for before they close the incident and go back to bed.