LEARN. PRACTICE. GROW.Concept and mechanism
Start with volume, frequency, format, and change behavior. Lakeflow Connect offers managed connectors with source-dependent incremental mechanisms; confirm support and release status. Standard connectors allow more customization. COPY INTO loads files into Delta and normally avoids loading previously processed files again. Auto Loader uses cloudFiles and checkpoint state for incremental discovery and processing; it does not guarantee file arrival order. These guarantees do not eliminate repeated business events inside different files. Retain identifiers, provenance, and reconciliation. For a custom API connector, define pagination, authentication, cursor handling, and failure behavior; completing the first page does not establish full ingestion.
Guided application
In an exercise, a source adds a column to a JSON export. With addNewColumns configured and an inferred schema, Auto Loader updates stored schema and stops the stream, which needs restarting. With rescue, it retains unexpected data in the rescued column without automatically changing schema. The team must interpret the actual configuration, review the contract, and monitor that column. Deleting the checkpoint as a first reaction can cause unwanted reprocessing. Auto Loader inference uses strings by default for JSON and CSV; conversion to amounts and dates belongs in a controlled transformation. Define quarantine and promotion criteria so technical compatibility does not hide changed meaning.
A new file does not necessarily contain a new event.
Common pitfalls
Deleting state to fix schema; ignoring rescued data; assuming ordering; cursor without reconciliation.
Related topics: Platform, compute, and data contracts · Transformation, grain, and quality · Jobs, dependencies, and recovery
Preserve state and provenance, and validate ingested data meaning.
Reference: Auto Loader ingestion and checkpoints · 2026-05-04