LEARN. PRACTICE. GROW.1. Contract the source before connecting it
In a fictional middleware migration, the producer sends authentication, configuration-change, and network events. Before creating the integration, identify each field meaning, event class, source clock, and information that may contain sensitive data. A Security Lake custom source must deliver OCSF in Parquet; the crawler does not automatically transform arbitrary JSON into the correct model. Keep each class in a compatible object and separate sources where needed. The contract should contain an accepted example, an invalid record, and responsibility for rejections. During handover, request evidence that invalid records are visible to the operator rather than simply disappearing inside an aggregate success count.
2. Make coverage observable
The eventDay partition uses the event UTC day. A batch near local midnight may belong to the previous UTC day; use the timestamp offset and avoid inferring the day from bucket location. Build an account, Region, source, and consumer matrix with an owner and proof event. A query returning results does not demonstrate that every matrix row works. Distinguish regional configuration from collection status: ListDataLakes describes the data lake and GetDataLakeSources describes sources and accounts. Record rehearsal time and track observed latency through to the consumer. The operational goal is to explain an absence: a quiet source, failed delivery, wrong partition, or incomplete query requires different action.
3. Choose how the consumer accesses data
A SIEM reading new objects can use the S3 subscriber and HTTPS or SQS notifications. An analyst querying tables uses the LAKEFORMATION path. Choose according to the consumer contract without assuming one mode configures the other. Cross-account query sharing includes subscriber-side work: accept the valid RAM share and create the resource link to the shared database. The process also depends on appropriate Lake Formation permissions and configuration. To accept the integration, run the query as the actual consumer identity against an authorized source and demonstrate that an out-of-scope source is inaccessible. Retain share identifiers and ownership for the integration maintenance lifecycle.
4. Consolidate while controlling lifecycle
A rollup can consolidate several Regions but cannot contribute to another rollup. Design direct contributions and confirm approved residency, the key, and replication permissions before accepting the path. Manage retention through Security Lake: a broad S3 rule can remove required metadata. Object Lock on the managed bucket is not a universal solution either; it is unsupported and default retention interrupts normalized delivery. If policy requires an immutable copy, treat it as a separate path to design and validate. Include cost, ownership, recovery, read verification, and decommissioning in the project plan. Long retention without the ability to query does not meet the SOC need.
5. Read the CloudWatch delivery contract
In a CloudWatch subscription, undo base64 and gzip before interpreting the document. When transformation exists, the filter compares the transformed event; changing field names requires reviewing patterns. Use Standard for groups requiring subscriptions. Operations must distinguish retryable errors, with retries bounded to up to 24 hours, from AccessDenied or a missing destination, which can suspend the filter and leave logs skipped during the interval. A permission correction followed by a new event proves the current path, not the past. Define how to reconcile the retained source with the SOC, identify previously received events, and recover what is needed without duplicate alerts. That procedure should exist before the first incident.
6. Rehearse the circuit and hand over to RUN
A Firehose, S3, and Lambda chain can ingest logs produced by its own processing. Identify those groups and configure account-subscription exclusions with selectionCriteria. Do not generalize the documented exception for a Lambda used directly as the destination to every downstream Lambda in the circuit. In the local exercise, compare expected and received IDs: differences expose gaps and repetitions expose duplicates. This model only uses fictional inventory; it does not replace real service reconciliation. Finish handover with a proof event, a controlled failure, recovery, delivery metrics, and an escalation contact. The management summary should state demonstrated scope, limitations, observed recovery time, and what has not yet been rehearsed.
# Original fictional reconciliation model, not a CloudWatch replay tool.
# No credentials, network calls, or AWS changes.
from collections import Counter
def reconcile(expected, received):
counts = Counter(received)
return (set(expected) - set(received), {k for k,v in counts.items if v > 1}, set(received) - set(expected))
assert reconcile(['a','b'], ['a','b']) == (set,set,set)
assert reconcile(['a','b'], ['b'])[0] == {'a'}
assert reconcile(['a'], ['a','a'])[1] == {'a'}
assert reconcile(['a'], ['a','z'])[2] == {'z'}
assert reconcile(['a'], [])[0] == {'a'}
print('five reconciliation cases passed; no logs were recovered')
At 02:09 the SOC receives events again, but the interval since 02:00 remains unreconciled.
Common pitfalls
JSON as Parquet; local day as UTC; partial query as coverage; retries as guaranteed replay.
Related topics: Evidence and operational acceptance · Security and continuity
Demonstrate the path and interval completeness; a created integration still needs operational acceptance.
Reference: Security Lake custom sources · SCS-C03