HTTP lab and uncertain outcomes | Incident Management: coordinate and recover

Prepare one observation per layer

The script creates a loopback HTTP server and a temporary SQLite database containing only synthetic instructions. The first request validates the normal functional path. Next, a controlled flag represents an unavailable dependency: /health still returns 200 while /ready and the operation return 503. That attempt’s record states no commit occurred, and a query finds zero effects. Before running, predict these results in a table containing endpoint, stage, status and functional criterion. The difference between signals is deliberate and supports discussing dashboards that observe only the process. Endpoint names are not universal standards; another application needs to document what each check actually covers.

Lose the response after commit

In the next phase, the operation is stored and the server deliberately closes the connection before sending acknowledgement. The client gets RemoteDisconnected, but the database query finds one effect. This is not an external-network timeout or a process crash; the script controls the failure point. The unprotected route applies the same intent again when the client retries. The count rises from one to two. Ask the learner to explain the result without claiming the first call never happened. In a fictional APS incident, this distinction determines whether retrying is appropriate, whether the consumer must be queried or whether state should remain explicitly uncertain.

Compare the retry contract

The protected route records key and effect in the same local transaction. Retrying identical key and content retrieves the same identifier and returns replayed=true. Eight concurrent calls produce one application and seven replays in this execution. The key alone does not explain everything: the script checks the existing record under BEGIN IMMEDIATE and keeps the change within the same database. Changed content makes the route return 409; a missing key returns 400. These codes belong to the contract created for this exercise. Do not change a key merely to bypass conflict without deciding whether new intent exists. Analysis should retain the connection between the original request, retries and observed effects.

Recognize the evidence boundary

At the end, the database contains six effects for five intents because the unprotected route left a duplicate. integrity_check returns ok, showing that structural checking does not apply an undefined business-uniqueness rule. The lab observes actual HTTP requests and actual transactions, but within one local process and database. It implements no authentication, payments, cross-host consensus or transaction encompassing an external system. Execution stops the server, releases its listener and removes the temporary database. Retain the report before discussing results with colleagues. Use observations to formulate more representative acceptance criteria without presenting this training server as a production-ready solution.

# Local training only; binds 127.0.0.1 on an ephemeral port.
python3 content/labs/incident-runtime/run.py --output /tmp/dr-incident-observation.json
# Actual recorded runtime: Python 3.13.1, SQLite 3.51.2, Darwin arm64.
# Own temporary database is removed; no real credentials or business effects.

IN PRACTICE

The process answers /health while the operation fails; in another phase, the operation is stored and the client loses the response.

Common pitfalls

Treating one 200 as restored service, a closed connection as cancellation or a key alone as a distributed guarantee.

Related topics: Diagnosis and mitigation · Communication and RUN handover

Take this idea with you

Separate received response from produced effect. A retry decision needs the contract and observed state.

Create account

Reference: Effective Troubleshooting · Incident management practices 2026-09; scoped Google SRE, PagerDuty and Atlassian examples