Ingestion, storage, and quality | AWS Machine Learning Engineer: prepare and operate models

Concept and mechanism

An ML pipeline starts with data contracts: origin, schema, meaning, event time, expected delay, and handling of invalid records. Choose formats and storage according to access and transformation. Parquet suits analytical column reads; JSON can retain varying structures that still need validation. Streaming reduces arrival-to-processing delay but requires handling duplicates, late events, and consumers falling behind input. Stable identifiers help recognize repeats. Do not confuse receipt acknowledgement with completed processing. Record counts, rejections, and delays to reconcile arrivals with available output. Vectors add a representation for semantic search while retaining requirements for version, dimension, metadata, and access to associated documents.

Guided application

In a fictional exercise, a source changes an amount from a number to text using a different decimal separator. The job technically succeeds but produces many empty values. A completeness or range rule helps stop promotion of those records and route them to quarantine. AWS Glue Data Quality supports DQDL rules in the catalog or ETL; the team must define failure response, ownership, and reprocessing. An aggregate score does not replace criteria for critical fields. For RAG documents, retain identifiers, versions, permissions, and conditions alongside chunks. If the embedding model changes, check index compatibility and required reindexing. APS handover includes freshness and count metrics, recovery procedures, and a way to distinguish delay from data loss.

IN PRACTICE

A green job can produce a dataset unsuitable for training.

Common pitfalls

Technical success treated as quality; duplicates without keys; average scores without critical fields; incompatible embedding versions.

Related topics: Features, splits, and leakage · Training, tuning, and reproducible experiments · Foundation models and RAG quality

Take this idea with you

Validate data meaning and completeness, and define what happens when they fail.

Create account

Reference: AWS Glue Data Quality · MLA-C02