← Production Support L3: investigate and recover
08 / 12 · 20 MIN

Runbooks that support decisions

Design steps with preconditions, limits, and recovery evidence.

Understand the concept

A useful runbook explains when to apply the procedure, required access, and how to recognize success or stop. For every action with effects, define its precondition, expected result, and next decision. Include functional validation after the action. “The command returned zero” proves only that command’s defined outcome; it may not mean customers recovered. Automation needs scope and concurrency limits, logging, and failure handling.

Apply and decide

Design exercise: a tool can restart a group of instances. Before using it, identify service and environment, check minimum capacity, limit the batch, and confirm how to stop. After each batch, observe technical signals and a representative operation before continuing. A dry run can show targets and intended changes but does not demonstrate real behavior with concurrency, load, and dependencies. A recovery rehearsal should reconcile outcomes as well as confirm running infrastructure.

Workplace application

Design the runbook for success, failure, and partial state. A batched operation needs a limit, validation after each batch, and a stop condition. If the tool exits zero but service fails, the procedure should continue functional investigation. Retain targets, version, parameters, and results in the authorized record so later analysis does not depend on operator memory.

IN PRACTICE

A file-recovery procedure declares success as soon as the importer process starts. Add fictional exercise criteria: expected file received, record count reconciled, exceptions analyzed, and process owner confirmation. Real criteria depend on the service.

Common pitfalls

Ignoring preconditions; automatically retrying after partial effects.

Related topics: Triage and organize the response · Investigate hypotheses and dependencies

Take this idea with you

The procedure should support stopping and demonstrate service recovery beyond command execution.

Create account

Reference: Evolution of automation · DR Production Support L3 2026.4; Linux, JDK 25 HotSpot, OpenSSL 3.5 and Kubernetes examples require installed-version checks