LEARN. PRACTICE. GROW.Authority can become stale
A fictional worker checks that it may run a job and pauses before writing. Meanwhile, another worker receives new authority. When the first returns, its memory still contains a decision that no longer matches current state. Shortening the interval between checking and writing does not remove the problem. The destination accepting the effect must distinguish the old request. The Chubby paper describes generation information in protected requests; this lesson uses its own simplified contract without implementing Chubby, distributed issuance, or a fencing product. Keep the conceptual reference separate from the behavior actually executed here.
Explicit contract per resource
The lab creates two synthetic resources, fund-A and fund-B, each with an installed generation and integer value. A function representing trusted authority installs a higher generation. The worker supplies resource, generation, request identity, and delta. The destination requires equality with the installed generation: a larger number invented by the worker is also rejected. This choice is specific to the exercise rather than a universal fencing-token rule. Trusted generation origin is an assumption, not an implemented feature. Real integration needs demonstrated protection of installation access and the association between authority and caller.
Apply the decision in the transaction
Two SQLite connections access the same temporary file. The apply function starts BEGIN IMMEDIATE before reading generation, checking the request, and changing the integer. Installing another generation and writing therefore cannot pass between validation and effect within that local transaction. The exercise shows a second connection rejected by contention while the first holds its transaction; after release, it progresses again. Lock contention and generation rejection are different causes. The harness uses explicit SQL transactions and zero timeout to make this observation reproducible without presenting that value as recommended production configuration.
Authority does not remove duplicates
A valid generation can submit the same intent twice. The model therefore keeps a separate record keyed by resource and request identity. The first call applies the delta and stores its result in the same transaction. Repeating the same intent returns the stored result; changing the delta under the same identifier is rejected. If r1 returned 7 and r2 took the value to 12, replaying r1 returns 7 without reducing current state. The response belongs to the original request. To learn present state, perform an appropriate read instead of reinterpreting a replay response.
Counterexamples bound the protection
The lab injects an exception between updating the integer and recording the request: ROLLBACK preserves the earlier value and leaves no record of the aborted request. This covers only effects in the same local database. An external API call would not be undone by that rollback. The exercise then writes directly through SQL, deliberately bypassing the protected function. It also simulates bad recovery that resets the old generation; a request from that generation becomes accepted again. These counterexamples show why every write path, restored state, and authority origin are part of the guarantee. Reopening the file is not a crash or power-loss test.
Workshop: review an APS design
Use forty-five minutes in pairs. During the first fifteen, draw a fictional scheduler with two workers, the generation issuer, and the destination. Mark every path capable of producing effects. During the next fifteen, introduce a long pause, a lost response, and direct SQL access. Explain in English which control addresses each failure and which remains uncovered. During the final fifteen, present a recommendation containing evidence, limitation, and owner to the manager. If the destination cannot enforce the contract, identify the supported isolation mechanism that must be exercised. Do not claim physical isolation merely because the local model passed.
python3 content/labs/ha-fencing/run.py
# Python 3.13; standard library only.
# Creates and deletes its own temporary SQLite database.
# No network, existing database or physical fencing operation.A reads generation 1; the destination installs 2; apply(A,1) is rejected. A direct SQL write bypasses the contract and demonstrates the need to cover every path.
Common pitfalls
Trusting an earlier query; accepting invented numbers; using generation as deduplication; allowing direct paths; restoring old authority; assuming rollback of an external API.
Related topics: Failure domains and residual capacity · Quorum and writer isolation · Replacement, identity, and acceptance
The guarantee depends on who issues authority, where the effect is admitted, and how state and request identity survive recovery.
Reference: The Chubby lock service for loosely-coupled distributed systems · BigSavant HA 2026-09; Pacemaker 3.0, etcd 3.6, PostgreSQL 18 and selected Kubernetes/AWS behavior