LEARN. PRACTICE. GROW.Concept and mechanism
Useful monitoring answers whether the required data product arrived on time and with quality. Aggregate pipeline status is only part of the evidence. In a pattern with a main activity and only a successful Upon Failure branch, the pipeline can succeed despite the handled failure. That status describes configured orchestration outcome; it does not prove the main load delivered data. A run that never started may also produce no Failed event. Define alerts for missing expected runs, data age, rejects, and reconciliation alongside technical errors. Each alert needs an owner, an action, and an observable recovery condition.
Guided application
For performance investigation, break copying into reading, transfer, writing, and waiting. If eighteen of twenty minutes occur at a throttled source, increasing target capacity is not the first supported hypothesis. Correlate the run ID with activities and external services while retaining the data-interval context. In a dedicated SQL pool, stale statistics after loading can distort estimates; update relevant statistics and compare plan and duration. For functional acceptance, connect the report to available lineage and document custom notebook transformations the integration does not capture. Give RUN a procedure distinguishing technical success, delay, partial data, and reconciled data, with suitable business communication.
The pipeline is green and the latest file is yesterday’s. Keep acceptance pending, investigate the load, and confirm the content period before publishing.
Common pitfalls
Green status treated as completeness; absence of error treated as execution; total duration treated as sufficient diagnosis.
Related topics: Storage, distribution, and exploration · Incremental loads and recovery · Streams, time, and external effects
Accept the product against agreed criteria and retain evidence of the complete path.
Reference: Data Factory run monitoring · DP-203 objectives 2024-10-24; retired 2025-03-31