LEARN. PRACTICE. GROW.Does the procedure apply to this target
A runbook needs to match the observed service, environment, version and state. In the original model, the procedure expects positions 2.4 in prod-eu but finds version 2.5. Authorization is confirmed, but the required batch state is unknown. The model returns two gaps: version and precondition. It executes no command and decides no real authorization. In practice, L2 should give the owner the concrete difference rather than edit the record to appear compatible. Earlier experience helps investigation but does not automatically confirm a changed context.
Translate scope into parameters
If the decision covers one worker, a wildcard selecting every worker is a material difference. Review the effective target list, parameter origin and action bounds before executing. A preview mode may help, but confirm that it truly changes no state and uses the same selection as execution. The aim is not an extra generic approval: it is matching the concrete operation to already authorized scope. If the procedure cannot demonstrate that match, present the gap and request the necessary correction.
Calculate the complete window
The exercise reserves nine minutes to execute, seven to validate and six for rollback. That is 22 minutes in a sequential fixed-duration model; a 20-minute window does not satisfy that plan. The calculation predicts neither variability, dependencies nor human decision time. The team must assess those factors in the real environment. Comparing only the nine execution minutes with the window omits stages needed to accept or recover the service. L2 communicates options and limits; changing the window or scope belongs to the authorized decision owner.
Reverting code does not reconcile data
An earlier image may start without being able to interpret records produced by the new version. Recovery planning must consider persistent state, compatibility and external effects. In the fictional case, the format changed and operations were already written. Do not delete those records merely to turn a dashboard green. Give the responsible team the states and compatibility problem while preserving identities. The solution may involve compatible migration, controlled recovery or another agreed option. This course executes no migration and does not assume every rollback undoes all changes.
Probes and actions that may amplify an incident
A liveness probe must be interpreted according to its configuration and the problem it aims to detect. If it depends on a slow external service, restarts may increase instability without fixing that dependency. Collect restart reasons, probe results and external-service symptoms before proposing repetition. Readiness and liveness have different purposes; passing readiness also does not prove delivery of a closing file. Kubernetes documentation supports the mechanism distinction. This exercise uses only a fictional case, changing no probes, restarting no Pods and validating no real cluster.
Expiry and continuity of a mitigation
A temporary mitigation has an observed state, an owner and a limit or removal criterion. In the model, current time and expiry are both 09:00 UTC, no extension exists and the next shift has not accepted ownership. The result flags expiry and missing ownership without automatically removing or extending anything. That distinction matters: a clock does not prove a reversal was executed. At handover, distinguish what remains active, what was authorized and the outstanding urgent decision. Handover completes only with appropriate acceptance of responsibility.
The model gate identifies version 2.5 instead of 2.4 and unknown batch state. Another calculation shows 9 + 7 + 6 minutes exceeding a 20-minute window by two. No model executes changes.
Common pitfalls
Executing on a different version by habit; accepting a wildcard beyond scope; treating unknown as confirmed; forgetting rollback in the window; assuming expiry removes or extends mitigation.
Related topics: Operational mitigation and validation · Handover and improvement · Timeline, recovery, and L2 handover
A usable runbook links observable conditions to bounded actions and recovery criteria. Automation must preserve that link and evidence of what it did.
Reference: The evolution of automation at Google · Operational support; PostgreSQL 18, OpenSSL 3.5 and BIND 9.20.29 examples; reviewed 2026-09-30