LEARN. PRACTICE. GROW.Fix the transaction boundary
A fictional application writes an operation and notification intent in one local transaction. That table of pending intents is the outbox. A separate component handles later delivery. In the lab, an exception between the two inserts causes explicit rollback and neither row remains persisted. After a successful commit, operation and intent exist, but the consumer has received nothing yet. This distinction is central to coordination: local commit is not completion of the whole service. The pattern reduces a gap between persistence and notification without turning two systems into one transaction. The plan should show who observes pending work, who can resume it, and what evidence confirms the final effect. Exercise data are invented and do not represent actual financial instructions.
Rehearse lost acknowledgment
The consumer receives event-A with value 125 and writes receipt and effect in its own transaction. The lab deliberately omits acknowledgment to the sender. The outbox remains pending although the effect exists. On retry with the same ID and content, the receipt identifies the repeat and retains one local effect. If the same ID arrives with value 999, the model signals conflict rather than treating it as an equivalent duplicate. Preserve intent identity between attempts and confirm the boundary in which receipt and effect are recorded. If the receipt committed first and the effect then failed, retry could be skipped without ever producing an outcome. The exercised implementation rolls both back on that failure. This is not a universal exactly-once guarantee.
Reconcile after the recovery point
The sender checkpoint was taken when A existed. The consumer applies A and later B, but only the sender is restored. The restored copy has one operation; the consumer has two effects with a fictional total of 205. Restoration did not undo B. Reconciliation requires knowing post-checkpoint operations, receipts, and effects in each component, with owners to resolve differences. Repeating A can be controlled if its receipt remains available. Two-day retention does not automatically cover permitted seven-day replay. The lab removes one receipt while retaining its effect and encounters a uniqueness violation on replay: this preserves the effect but needs investigation and does not establish normal resumption. Do not delete differences merely to align counts. Business compensations need their own rules and authority.
Practice the resumption decision with APS
Save the complete code below as release-state.py and run python3 release-state.py with Python 3.13 or later. Read its sixteen checks. It uses in-memory SQLite stores, exception-injected failures, a checkpoint copied between isolated databases, and calculations with fictional inputs. There is no broker, concurrent load, network timeout, or production restoration. The professional exercise is to explain to APS what each result establishes and what remains to be validated. Before resumption, identify paused ingress, already-accepted work, worker state, stable identities, retained receipts, and operations without known outcomes. Disabling an API flag does not cancel messages delivered to workers that do not read it. Finish with a decision stating resumption scope, boundaries, owners, and a stop condition. Process availability and outcome reconciliation are distinct evidence.
"""Original isolated SQLite release-state lab. No broker, cloud, bank or production system."""
import sqlite3,json,platform,hashlib
from pathlib import Path
from fractions import Fraction
checks=[]
def check(name,actual,expected):
assert actual==expected,(name,actual,expected)
checks.append(dict(name=name,actual=actual,expected=expected,passed=True))
def db:return sqlite3.connect(':memory:',isolation_level=None)
def count(c,t):return c.execute('SELECT count(*) FROM '+t).fetchone[0]
# Fixture 1: a deliberately explicit-column old writer survives a nullable addition.
schema=db;schema.execute('CREATE TABLE positions(id TEXT PRIMARY KEY, amount INTEGER NOT NULL)')
schema.execute("INSERT INTO positions(id,amount) VALUES('old-1',100)")
schema.execute('ALTER TABLE positions ADD COLUMN currency TEXT')
schema.execute("INSERT INTO positions(id,amount) VALUES('old-2',200)")
schema.execute("INSERT INTO positions(id,amount,currency) VALUES('new-1',300,'EUR')")
check('expanded_schema_accepts_both_writers',schema.execute('SELECT id,currency FROM positions ORDER BY id').fetchall,[('new-1','EUR'),('old-1',None),('old-2',None)])
implicit_failed=False
try:schema.execute("INSERT INTO positions VALUES('implicit',400)")
except sqlite3.OperationalError:implicit_failed=True
check('implicit_column_count_not_compatible',dict(rejected=implicit_failed,rows=count(schema,'positions')),dict(rejected=True,rows=3))
# A separate fixture represents the post-contract shape, not a production migration recipe.
contract=db;contract.execute('CREATE TABLE positions(id TEXT PRIMARY KEY,amount INTEGER NOT NULL,currency TEXT NOT NULL)')
old_failed=False
try:contract.execute("INSERT INTO positions(id,amount) VALUES('old-3',400)")
except sqlite3.IntegrityError:old_failed=True
check('contract_shape_rejects_old_writer',dict(rejected=old_failed,rows=count(contract,'positions')),dict(rejected=True,rows=0))
contract.execute("INSERT INTO positions(id,amount,currency) VALUES('new-2',400,'EUR')")
check('contract_shape_accepts_new_writer',count(contract,'positions'),1)
# Two separate stores: an application-side outbox and a receiver with a local ledger.
sender=db;receiver=db
sender.executescript('CREATE TABLE operations(id TEXT PRIMARY KEY,amount INTEGER NOT NULL); CREATE TABLE outbox(id TEXT PRIMARY KEY,payload TEXT NOT NULL,acked INTEGER NOT NULL DEFAULT 0);')
receiver.executescript('CREATE TABLE receipts(id TEXT PRIMARY KEY,payload TEXT NOT NULL); CREATE TABLE ledger(id TEXT PRIMARY KEY,amount INTEGER NOT NULL);')
def create_operation(key,amount,fail_between=False):
sender.execute('BEGIN')
try:
sender.execute('INSERT INTO operations VALUES(?,?)',(key,amount))
if fail_between:raise RuntimeError('injected-before-outbox')
sender.execute('INSERT INTO outbox(id,payload) VALUES(?,?)',(key,json.dumps(dict(amount=amount),sort_keys=True)))
sender.execute('COMMIT')
except Exception:
sender.execute('ROLLBACK');raise
try:create_operation('aborted',999,True)
except RuntimeError:pass
check('injected_failure_rolls_back_operation_and_outbox',[count(sender,'operations'),count(sender,'outbox')],[0,0])
create_operation('event-A',125)
check('commit_before_delivery_is_pending',[count(sender,'operations'),count(sender,'outbox'),count(receiver,'ledger')],[1,1,0])
checkpoint=db;sender.backup(checkpoint)
def receive(key,payload,fail_before_effect=False):
receiver.execute('BEGIN IMMEDIATE')
try:
prior=receiver.execute('SELECT payload FROM receipts WHERE id=?',(key,)).fetchone
if prior is not None:
if prior[0]!=payload:raise ValueError('same-id-different-payload')
receiver.execute('COMMIT');return 'duplicate'
receiver.execute('INSERT INTO receipts VALUES(?,?)',(key,payload))
if fail_before_effect:raise RuntimeError('injected-before-ledger')
receiver.execute('INSERT INTO ledger VALUES(?,?)',(key,json.loads(payload)['amount']))
receiver.execute('COMMIT');return 'applied'
except Exception:
receiver.execute('ROLLBACK');raise
key,payload=sender.execute('SELECT id,payload FROM outbox').fetchone
try:receive(key,payload,True)
except RuntimeError:pass
check('receipt_and_effect_rollback_together',[count(receiver,'receipts'),count(receiver,'ledger')],[0,0])
result=receive(key,payload) # Deliberately omit acknowledgment to the sender.
check('receiver_committed_sender_ack_missing',[result,sender.execute('SELECT acked FROM outbox').fetchone[0],count(receiver,'ledger')],['applied',0,1])
result=receive(key,payload)
sender.execute('UPDATE outbox SET acked=1 WHERE id=?',(key,))
check('retry_with_retained_receipt_has_one_effect',[result,count(receiver,'ledger'),receiver.execute('SELECT sum(amount) FROM ledger').fetchone[0]],['duplicate',1,125])
conflict=False
try:receive(key,json.dumps(dict(amount=999),sort_keys=True))
except ValueError:conflict=True
check('same_id_changed_payload_is_conflict',[conflict,receiver.execute('SELECT sum(amount) FROM ledger').fetchone[0]],[True,125])
# Restore only a synthetic sender checkpoint. The receiver remains at its later state.
restored=db;checkpoint.backup(restored);oldkey,oldpayload=restored.execute('SELECT id,payload FROM outbox WHERE acked=0').fetchone
check('restored_sender_replays_known_event',[receive(oldkey,oldpayload),count(receiver,'ledger')],['duplicate',1])
# A genuinely later operation is absent from the sender checkpoint, even if accepted downstream.
create_operation('event-B',80);bkey,bpayload=sender.execute("SELECT id,payload FROM outbox WHERE id='event-B'").fetchone;receive(bkey,bpayload)
check('sender_checkpoint_does_not_rewind_receiver',{'restoredOperations':count(restored,'operations'),'receiverEffects':count(receiver,'ledger'),'receiverTotal':receiver.execute('SELECT sum(amount) FROM ledger').fetchone[0]},dict(restoredOperations=1,receiverEffects=2,receiverTotal=205))
# The safety claim fails if deduplication history is removed while effects remain.
receiver.execute("DELETE FROM receipts WHERE id='event-A'")
replay_blocked=False
try:receive(key,payload)
except sqlite3.IntegrityError:replay_blocked=True
check('missing_receipt_needs_reconciliation',[replay_blocked,count(receiver,'ledger'),receiver.execute('SELECT count(*) FROM receipts WHERE id=?',(key,)).fetchone[0]],[True,2,0])
# Atomic version check applies only to this metadata update, not to external deployment commands.
state=db;state.execute('CREATE TABLE target(id TEXT PRIMARY KEY,generation INTEGER NOT NULL,version TEXT NOT NULL)');state.execute("INSERT INTO target VALUES('test',7,'v7')")
fresh=state.execute("UPDATE target SET generation=8,version='v8' WHERE id='test' AND generation=7").rowcount
stale=state.execute("UPDATE target SET generation=8,version='v6' WHERE id='test' AND generation=7").rowcount
check('stale_state_update_changes_zero_rows',[fresh,stale,state.execute('SELECT version FROM target').fetchone[0]],[1,0,'v8'])
check('recovery_deadline_minutes',23*60-(20+15+10),22*60+15)
check('synthetic_segment_rates',{'canary':str(Fraction(6,150)),'control':str(Fraction(9,9850)),'global':str(Fraction(15,10000))},{'canary':'1/25','control':'9/9850','global':'3/2000'})
for conn in [schema,contract,sender,receiver,checkpoint,restored,state]:conn.close
print(json.dumps(dict(python=platform.python_version,sqlite=sqlite3.sqlite_version,scriptSha256=hashlib.sha256(Path(__file__).read_bytes).hexdigest,groups=len(checks),scope='Executed isolated in-memory SQLite transactions, schema shapes, synthetic sender restore and failure injection; arithmetic fixtures use fictional inputs. No real broker, distributed exactly-once guarantee, production restore, rollout, concurrency load or independent specialist review.',checks=checks),indent=2))Restored sender: one operation. Retained consumer: two effects totaling 205. Recovering one component does not automatically rewind the other.
Common pitfalls
Timeout as certain failure; new ID on every retry; receipt separate from effect; local restoration as global consistency; deleting receipts before defining replay.
Related topics: Recovery and transition to APS · Artifacts and evidence
Resumption must distinguish what was requested, what already had an effect, and what still needs reconciliation.
Reference: Transactional outbox pattern · Google SRE release and canary guidance; GitHub immutable releases and GitLab release evidence and deployment safety; DORA five-metric model; inspected 2026-10-01