← Microservices: design boundaries and operate distributed systems
07 / 8 · 60 MIN

Outbox, inbox and failures between commits

Follow an event across local boundaries and verify what persists after interruption.

1. Three different confirmations

A persisted job, an accepted message and an updated projection are three different facts. In the lab, three SQLite files represent producer, queue and consumer. The producer commits the job and outbox intent; the relay transfers that intent; the consumer applies a projection and records processed identity. No transaction spans all three files. This design exposes boundaries that a single-arrow diagram may hide. During an incident, first locate the last evidenced confirmation instead of declaring success merely because the first service responded.

2. Interrupt before and after commit

The runner starts a child process that opens BEGIN IMMEDIATE and inserts job-a. In the first case it exits with os._exit(73) before the outbox; in the second it exits after the outbox but before COMMIT. Reopening the database shows zero jobs and zero events in both cases. In the third, exit occurs after COMMIT and both remain. The exit code is identical but state differs. Relevant evidence includes interruption location and subsequent reads. The experiment uses DELETE journaling and synchronous FULL; it does not simulate power loss or validate every disk behavior.

3. Publishing and marking are separate operations

The relay reads a pending event, records delivery in the local queue and only then marks the outbox published. An exception between those writes leaves accepted delivery and still-pending intent. After logically restarting the relay, publication repeats: the experiment finds two deliveries with the same identity. Reversing the order does not remove the problem; marking first can hide an event never delivered. Consumers must tolerate repetition and operations must monitor stuck intents. The SQLite queue is an original fixture without a commercial broker’s guarantees or acknowledgment mechanisms.

4. Define identity and consumer scope

CloudEvents 1.0.2 combines source and id to identify an event. A transport attempt may change without representing a new occurrence. When the same event feeds independent effects, inbox scope must also reflect the consumer. In the example, projection-v1 and projection-v2 have separate rows for the same source/id because each builds its own projection. The fixture also compares an envelope fingerprint: the same identity with different payload is a conflict, not a silent correction. This control is a lab choice. It neither establishes full CloudEvents compliance nor authenticates the event’s issuer.

5. The completion marker accompanies the effect

The consumer opens a local transaction, checks inbox, inserts the marker and updates the projection. An exception after the marker rolls both back; a subsequent attempt remains possible. After successful commit, repetition returns duplicate and retains total=7. The experiment shows one local application for two deliveries. Adding an external call does not make that call part of the SQLite transaction. A crash after a remote effect may require lookup, idempotent repetition or compensation. Define that behavior before promoting the solution to an integration that sends notifications or changes another system.

6. Diagnose by boundary

Prepare a table for an affected job with producer confirmation, outbox presence, delivery evidence, consumer marker and projection value. For each column, state the evidence source and what remains unknown. With premature relay marking, an outbox without pending work does not prove completion; reconcile affected references before resuming sends. APS work includes controlling recovery scope, retaining original records and obtaining authority for data repair. The lab rehearses concrete hypotheses rather than recommending direct production-database access.

BEGIN IMMEDIATE;
INSERT INTO jobs(id,state) VALUES ('job-a','accepted');
INSERT INTO outbox(id,payload,created) VALUES ('evt-1','{}',10);
COMMIT;
-- Teaching outline: publication happens later, outside this transaction.
IN PRACTICE

Exit 73 before COMMIT: jobs=0/outbox=0. Exit 73 after COMMIT: jobs=1/outbox=1. The relay may create two deliveries while inbox preserves one local application.

Common pitfalls

Mark published before sending; regenerate identity on retry; deduplicate by id alone; commit inbox separately from its effect.

Related topics: Transactions and persistence · Events and identity · Operational recovery

Take this idea with you

Each confirmation has a scope. Preserve identity and persist the processing marker with the effect it establishes.

Create account

Reference: Transactional outbox pattern · Microservice architecture patterns and scoped platform examples; primary guidance consulted 2026-09-30