LEARN. PRACTICE. GROW.Choose the path by the result
A fictional team prepares an operations report by day and currency. First define the result: which events belong to the day, permitted lateness, identities that must appear, and who may query raw or aggregated data. Kinesis Data Streams can support applications following the same retained history with independent progress.
Firehose delivers data to destinations under a format and buffering contract. Glue provides transformation and a catalog; discovering metadata does not execute the transformation. Athena SQL supports analysis of S3 data.
These roles can coexist. Select the necessary ones and trace an event from entry to its queried row rather than accepting the pipeline’s last green status.
Preserve replay and size the right unit
Retention and capacity solve different problems. If an application stops for 32 hours and only 24 hours remain retained, changing retention at restart does not recreate the first eight hours. Plan archival or alternative recovery when requirements exceed effective retention.
For capacity, look for concentration by partition key and read contention. An organization key can concentrate many accounts; if required ordering is only per account, assess distribution by accountId and validate order at producer and consumer.
Enhanced fan-out separates read capacity between consumers but does not repair concentrated writes or a slow processor. The choice requires locating the observed limit.
Distinguish delivery, identity, and progress
A producer timeout leaves uncertainty about whether the service accepted an event. Retrying can create another transport record for the same operation. Retain a stable eventId or operationId and define where the effect becomes confirmed.
A consumer that published 101–110 while retaining checkpoint 100 can process those events again; the destination must recognize completed work. In our model, the destination supplies atomic writing by identity. That premise must be implemented and verified in a real system.
Recording progress before preserving results can lose work; preserving results before recording progress requires controlled replay. Neither boundary is solved simply by adding workers.
Transform without hiding exceptions
Publishing bytes does not mean a reader can interpret them. Two concatenated JSON objects need framing compatible with the parser. During transformation, preserve types and meaning: an unknown amount does not become zero without an authorized business rule.
Record the event in quarantine with identity, reason, and owner. Glue bookmarks help select source work; they do not replace event-level reconciliation. When replaying an interval, plan the destination explicitly: an isolated version can be validated before switching the publication pointer.
If the system appends, repeated inputs can produce repeated outputs. Preserve the available version until the contract permits switching, and retain evidence of the source used.
Query the correct set with bounded access
A partition describes physical organization, which may differ from the business day. If partitioning by arrival, day-10 events can occupy days 10, 11, and 12; read the permitted arrival range and filter event day. Confirm metadata locations: querying raw after transforming curated still returns the old source.
Columnar formats and filters can reduce reading work, but compare equivalent results before interpreting bytes scanned and duration. For access, check every route. Restricting SELECT in Lake Formation does not automatically remove direct S3 authorization.
Test the permitted analytical path and raw access that should be denied using the actual identities involved.
Workshop: rebuild the report without losing identities
Complete local-model inputs: A occurs on day 10, arrives on day 10, and is 1000 EUR cents; B occurs on day 10, arrives on day 11, and is 2000 USD cents; a repeat of A arrives on day 11 with the same content; C occurs on day 11, arrives on day 11, and is 500 EUR cents; D occurs on day 10, arrives on day 12, and has an invalid amount.
The report covers day 10 and permits identified quarantine. Predict IDs, totals, and exceptions before execution. The answer is IDs A and B, totals EUR=1000 and USD=2000, with D quarantined; C is outside the requested day and repeated A does not duplicate the total.
Reading only arrival partition 10 would miss B and D. The program rejects repeats with different content for the same identity instead of silently choosing one. Save as ingestion-model.mjs and run with Node.js.
Change eventDay, arrivalDay, or amountCents one at a time. The model runs only in memory and does not implement checkpoints, concurrent publication, or AWS services.
// In-memory teaching model; no AWS calls, persistence, or concurrent publication.
const events=[
{id:'A',eventDay:10,arrivalDay:10,currency:'EUR',amountCents:1000},
{id:'B',eventDay:10,arrivalDay:11,currency:'USD',amountCents:2000},
{id:'A',eventDay:10,arrivalDay:11,currency:'EUR',amountCents:1000},
{id:'C',eventDay:11,arrivalDay:11,currency:'EUR',amountCents:500},
{id:'D',eventDay:10,arrivalDay:12,currency:'EUR',amountCents:null},
];
function report(rows,day,arrivals){
if(!Number.isInteger(day)||!Array.isArray(rows)||!Array.isArray(arrivals)||!arrivals.every(Number.isInteger))throw Error('invalid input');
const seen=new Map(), totals={}, ids=[], quarantine=[];
for(const e of rows){
if(!e||typeof e.id!=='string'||!e.id||!Number.isInteger(e.eventDay)||!Number.isInteger(e.arrivalDay))throw Error('invalid event');
if(!arrivals.includes(e.arrivalDay)||e.eventDay!==day)continue;
const identity=JSON.stringify([e.eventDay,e.currency,e.amountCents]);
if(seen.has(e.id)){if(seen.get(e.id)!==identity)throw Error('conflicting identity: '+e.id);continue;}
seen.set(e.id,identity);
if(!['EUR','USD'].includes(e.currency)||!Number.isSafeInteger(e.amountCents)){quarantine.push(e.id);continue;}
const total=(totals[e.currency]||0)+e.amountCents;
if(!Number.isSafeInteger(total))throw Error('unsafe total');
ids.push(e.id);totals[e.currency]=total;
}
return {ids:ids.sort(),totals,quarantine:quarantine.sort()};
}
console.log(JSON.stringify(report(events,10,[10,11,12])));Equal row counts can hide a missing ID and a repeated one; retain identity, currency, and time scope.
Common pitfalls
Confusing arrival day with event day; treating metadata as transformation; summing different currencies; accepting a checkpoint as proof of an effect.
Related topics: Events, queues, and processing · Events: permissions, capacity, and costs · Migration, validation, and cutover
Accept data by the set and meaning the consumer requires, with replay and exceptions accounted for.
References
- AWS Glue · AWS documentation consulted 2026-10-12
- Kinesis Data Streams concepts · AWS documentation consulted 2026-10-12
- Kinesis duplicate records · AWS documentation consulted 2026-10-12
- Kinesis enhanced fan-out · AWS documentation consulted 2026-10-12
- Data Firehose delivery · AWS documentation consulted 2026-10-12
- Glue job bookmarks · AWS documentation consulted 2026-10-12
- Athena partitioning · AWS documentation consulted 2026-10-12
- Athena columnar formats · AWS documentation consulted 2026-10-12
- Amazon Athena · AWS documentation consulted 2026-10-12
- Lake Formation underlying data access · AWS documentation consulted 2026-10-12