LEARN. PRACTICE. GROW.Concept and mechanism
Optimization starts with an observable hypothesis and a reproducible comparison. In an existing date-partitioned Delta table without liquid clustering, compaction with OPTIMIZE can reorganize small files while preserving logical results. For new designs, current documentation recommends evaluating liquid clustering; do not automatically copy the historical design. VACUUM has a different purpose: removing eligible files no longer needed by retained versions. The log may still mention a version whose files no longer exist. Define data and history retention around recovery, maximum job duration, and delayed consumers. Storage savings do not establish that restoration remains possible.
Guided application
In Spark, one task much slower than its peers can indicate skew, especially when one key dominates rows. Investigate distribution before attributing all delay to insufficient capacity. Semantically valid projection and filtering before shuffle reduce moved data; indiscriminate caching can increase memory pressure. Measure spill, task tails, volume, and results. In serverless SQL, reducing reads from900 to90GB means90% for that run, but does not guarantee the same reduction in latency or total platform cost. In a project, present FinOps with measured savings including optimization cost, and APS with effects on the operating window and recovery. Maintain rollback and reconciliation criteria during each experiment.
Proposed retention is shorter than a job’s duration and the seven-day recovery commitment. Defer the change until a design preserving those requirements is demonstrated.
Common pitfalls
OPTIMIZE treated as backup; VACUUM treated as compaction; fewer bytes treated as equal gains across all metrics.
Related topics: Storage, distribution, and exploration · Incremental loads and recovery · Streams, time, and external effects
An improvement is established when performance, cost, and correctness are measured in the same context.
Reference: Delta compaction and data layout · DP-203 objectives 2024-10-24; retired 2025-03-31