LEARN. PRACTICE. GROW.Registration does not mean readiness
Imagine a fictional APS host renewal: three voters are healthy and the team prepares a replacement. The first exercise milestone registers a learner without starting its process. Inventory now has four identities, but configured votes remain three. A synthetic write continues to be acknowledged by the original members. Reporting should state exactly that. Do not label the candidate operational merely because it appears in the list. Preparation includes startup, communication, data catch-up, and confirmation of the intended role. Each milestone needs evidence and an owner before it can support a dependent maintenance decision.
Readiness to change membership
In the first local execution, endpoint health passed before an addition attempt returned unhealthy cluster. This is useful evidence: a health check does not automatically represent every condition for a change. The lab retains reconfiguration safeguards. It checks membership before attempting registration and retries only the expected explicit rejection within a bounded deadline. An unexpected result stops the exercise. In a real system, a timeout can leave the outcome uncertain; reconcile identity and peer URLs before deciding another change. Distinguish controlled waiting from repeatedly issuing commands without a state-based decision.
Startup and observed data
The new process starts with updated membership and existing state. In the exercise, it receives a key written before startup. A serializable read observes that local value and status identifies the learner role. These observations answer different questions: role describes participation, while the read establishes only the data observed at that moment. Do not turn one recovered key into proof of permanent synchronization, performance, or every consumer. Transfer can consume network, disk, and leader work even without adding a vote. Plan data catch-up with headroom for the existing service and observe its impact during execution.
Not every operation is accepted
With etcd 3.6.15, the lab read the key from the learner using explicit serializable mode. A linearizable read and a write submitted to the same endpoint were rejected. The attempted write’s key was then queried on a voter and was absent. Record each operation’s contract and result instead of summarizing everything as endpoint available. A design page may contain historical descriptions, exceptions, and proposed features; do not use an isolated sentence as a guarantee for another version. In this course, the operational conclusion is limited to the executed version and the requests recorded in evidence.
Promotion changes the arithmetic
Promotion before startup was rejected. After data catch-up, promotion was accepted and membership showed four voters. The new member accepted the exercised linearizable read and synthetic write. The majority changed from two to three; tolerance remained one unavailable voter, assuming communication among survivors. This transitional state requires maintenance coordination. Two tickets stopping different members can remove the majority when executed together. Before each action, draw effective membership and count the votes that remain communicating during the action. A milestone should include the resulting headroom, not merely a successful command exit code.
English decision workshop
Work in pairs and use fifteen minutes to prepare a five-line briefing: initial state, candidate role, catch-up evidence, next change, and stop condition. One participant represents APS and the other the change manager. Then introduce an inject: another team wants to stop a voter during promotion. Within ten minutes, recalculate membership and decide whether the windows can overlap. Finish with ten minutes of questions: which result is local, which requirement was not exercised, and who confirms the consumer? Review reasoning and clarity; the lab does not replace practice facilitated by a specialist.
python3 content/labs/ha-membership/run.py --bin-dir /path/to/etcd-3.6.15
# Own temporary loopback processes and synthetic keys only.
# Requires etcd and etcdctl 3.6.15; no existing cluster is accepted.Three voters plus one learner require two votes. After promotion there are four voters and three are required; two overlapping shutdowns remove the majority.
Common pitfalls
Counting an identity as a vote; accepting a local read as a linearizable contract; removing safeguards to overcome rejection; using a design proposal as an active feature.
Related topics: Failure domains and residual capacity · Quorum and writer isolation · Election, rejoining, and maintenance
Prepare data and confirm the role. Each transition changes the conditions under which the next maintenance action can preserve service.
Reference: etcd learner design · BigSavant HA 2026-09; Pacemaker 3.0, etcd 3.6, PostgreSQL 18 and selected Kubernetes/AWS behavior