LEARN. PRACTICE. GROW.Design an exercise with bounded conclusions
The lab creates three etcd 3.6.15 processes with disposable directories, a dedicated cluster token, and loopback-only ports. The official binary was checked by SHA256. The 3.6 branch follows existing examples; this does not claim it is the project newest branch. All three processes share the laptop, so they do not represent independent zones. The exercise uses unauthenticated HTTP only within this local scope and provides no deployment configuration. The code accepts no external endpoints and terminates the processes it created. Before using results in a committee, state what was observed and which environment conditions remained outside the exercise.
Bind the decision to the operation in a transaction
Two clients attempt to create /dr/owner. Each transaction checks version equal to zero and writes only if the key does not yet exist. The first receives succeeded=true; the second receives false and finds worker-a as owner. The second call responded normally but did not execute the acquisition branch. This distinction prevents treating every response without a transport error as authorization. The exercise puts a condition and write in the same etcd transaction, removing the gap between a separate read and local decision. The result protects this key under that contract. It does not automatically grant exclusivity over a database or file outside the operation.
Detect a change that returned to the same value
The fixture retains the initial owner mod_revision, changes the value from A to B, and returns it to A. An old client may see the same text again and conclude that nothing changed. The guard compares the observed revision and therefore rejects the /dr/guarded-resource write. The lab confirms that this key was not created. This distinction matters when a task resumes after a pause: identical text identity does not establish continuity of prior authorization. A revision represents a historical change, not wall-clock seconds or lease duration. Keep scope explicit: here, the check and protected effect belong to the same etcd transaction.
Choose the read guarantee for the decision
After two members are stopped, the remaining process still accepts a TCP connection and responds with local status. A linearizable read does not complete, but a read using --consistency=s returns after-leader-stop. This value helps observe state retained by the member, but the response establishes neither quorum nor current authority. In a fictional batch-coordination system, a local read may serve an informational screen if its limitation is accepted. It should not silently replace the read required to authorize a writer. Define the decision first and choose an operation whose guarantee supports it; receiving a response differs from obtaining the required evidence.
A lease does not terminate the process
The exercise attaches a key to a two-second lease and starts a subprocess waiting for a local instruction. When the key disappears, the subprocess remains alive. On receiving resume, it writes late-write to a temporary file outside etcd. There is no banking storage or remote call: the file is a controlled counterexample to the idea that lease expiry kills a worker. Coordination stopped recognizing the key, but the destination enforced no write control. To protect a real resource, use an appropriate mechanism that enforces destination exclusivity or fences the previous writer, with evidence of the effect.
Deadlines and events do not replace state observation
The lab waits for verifiable conditions with a bounded deadline instead of declaring success after sleeping for a fixed number of seconds. A lease does not guarantee every client observes deletion at exactly the same wall-clock instant. Failures, election, and event delivery need consideration. Documentation distinguishes operation guarantees from watch observation. In a fictional case, a missing event does not prove renewal; observation-path delay may exist. Record the operation executed, guarantee requested, and observed result. A local timer may decide to stop an attempt, but does not by itself create proof of absent effects or current authority.
python3 content/labs/ha-quorum/run.py --bin-dir /path/to/etcd-3.6.15
# Creates its own loopback cluster and temporary data.
# No existing endpoints or data directories are accepted.With one live member, TCP and status respond and a serializable read returns data; the linearizable read fails. Reachability does not establish the contract needed to authorize writes.
Common pitfalls
Treating succeeded=false as acquisition; comparing only the value after a pause; replacing a linearizable read with a local read; confusing a lease with fencing.
Related topics: Election, rejoining, and maintenance · Residual capacity and failure domains · Fencing and service recovery
Authority requires a guarantee appropriate to the decision and protection on the path that actually accepts the effect.
Reference: etcd API · BigSavant HA 2026-09; Pacemaker 3.0, etcd 3.6, PostgreSQL 18 and selected Kubernetes/AWS behavior