LEARN. PRACTICE. GROW.Measure what the user needs
Latency, traffic, errors, and saturation are useful starting points but need context. Business success may require a batch to finish before cutoff or a file to be reconciled. Define each indicator’s numerator, denominator, window, and exclusions. Distinguish SLI, the measure, from SLO, the objective, and SLA, an agreement with specific conditions and consequences.
Aggregates can hide failures
Average latency can hide very slow requests. High overall success can conceal complete failure for a small customer or route. Examine relevant percentiles and segments without uncontrolled cardinality. Fewer errors may mean absent traffic rather than recovery. Compare volume and outcomes together.
Recovery and observation
Define evidence of service return: functional operation, dependencies, shrinking queues, and reconciliation as applicable. Maintain an observation window appropriate to the service cycle. A nightly batch may need validation of the next cycle to confirm correction effectiveness. Record recovery time separately from closure of root-cause investigation.
Workplace application
After restoring workers, measure arrivals and completions over the same window. Queue-draining capacity is the difference between the rates rather than gross throughput. Compare predicted duration with the deadline and include uncertainty. If a critical route remains unavailable, a favorable global average does not justify closing that service’s impact.
Errors fell from 500 per minute to zero, but traffic also fell to zero because a route was blocked. The green dashboard is misleading. Recovery should demonstrate representative requests arriving and completing with expected outcomes.
Common pitfalls
Ignoring demand and backlog; using averages that hide critical failure.
Related topics: Turn incidents into verifiable improvement · Shift handover and escalation
Absence of alerts is insufficient: recovery requires served demand and demonstrated functional outcomes.
Reference: Implementing SLOs · DR Production Support L3 2026.4; Linux, JDK 25 HotSpot, OpenSSL 3.5 and Kubernetes examples require installed-version checks