Ingestion, publication, and CI/CD | Professional Data Engineer: pipelines and data decisions

Concept and mechanism

In the Storage Write API gRPC interface, stream type and offsets influence writing guarantees. The default stream makes data queryable with at-least-once semantics. An application-created committed stream can use offsets to control appends within that stream. The client must persist and reconcile positions and handle responses after timeouts. Creating a new stream on every retry does not provide global business-identifier deduplication. To prepare data without partial exposure, pending streams allow writers to finalize and commit together. The publication barrier should be part of the design rather than relying on a comment saying the batch is incomplete.

Guided application

Destination contracts also matter for CDC: Datastream to BigQuery merges using sequence metadata, while a custom file consumer must implement its handling. In orchestration, design idempotent units per logical interval with staging and controlled publication. If a retry uses current time as a new identity and appends rows, it can duplicate an already-processed day. Current documentation presents Managed Service for Apache Airflow, still called Cloud Composer in the guide. In CI/CD, valid syntax is only one gate. A monetary change needs boundary cases and expected results with criteria set by the responsible team. Do not invent a universal financial rule. Retain code version and evidence to connect each result to its producing transformation.

IN PRACTICE

Repeating an append at the same position requires handling the API result; switching streams changes the guarantee’s scope.

Common pitfalls

Default stream as global deduplication; offset without state; syntax success as correctness; current time as retry identity.

Related topics: Design, governance, and identity · Quality, migration, and cutover · Streaming, time, and effects

Take this idea with you

Make logical unit, commit, and promotion gate explicit.

Create account

Reference: BigQuery Storage Write API gRPC · Current linked standard guide (document title v4.2); edition date unconfirmed (2026-09-30 inspection)