LEARN. PRACTICE. GROW.Concept and mechanism
A stream has no single completion point like a closed file. Define event identity, required order, business time, retention, and recovery. Consumer groups enable independent consumption; checkpoints save progress that each application must manage coherently. Persisting progress before the target effect can skip incomplete work. Persisting it afterward can repeat work if failure occurs between those actions. With foreachBatch, delivery is at least once: an external target needs idempotency or deduplication. A random key for every attempt is unsuitable because it gives the same operation different identities. Analyze the complete boundary before promising exactly-once.
Guided application
Watermarks bound state and express a lateness policy; they are not backups. In a Structured Streaming query with several inputs, the min policy follows the slower input; changing to max can reduce waiting and drop that input’s data. In Stream Analytics, processing and delivery have distinct guarantees. For SQL output, a natural key and uniqueness constraint can identify the same output, such as account and window end. Historical replay sent through a live path can be dropped or have its time adjusted; plan a suitable path and reconcile results. Test failure after target confirmation, lateness beyond tolerance, and producer semantic changes such as euros becoming cents without a field-name change.
The API confirmed batch91, but the consumer restarts without an updated checkpoint. Reconcile the operation and test replay with persistent identity before authorizing repetition.
Common pitfalls
Checkpoint treated as distributed transaction; arrival time treated as business time; retention treated as unlimited recovery.
Related topics: Storage, distribution, and exploration · Incremental loads and recovery · Lake access and secret management
Define what can repeat, what can arrive late, and how to establish target results.
Reference: Structured Streaming foreachBatch delivery · DP-203 objectives 2024-10-24; retired 2025-03-31