LEARN. PRACTICE. GROW.1. Assess an optimization effect
A FinOps proposal should state which technical behavior changes. S3 Bucket Keys reduce KMS calls through reuse and use the bucket ARN as encryption context. A previously object-scoped condition can stop working or stop representing the same separation. Assess compatibility and isolation before changing policy. Existing objects do not automatically start using Bucket Keys when the default changes, and DSSE-KMS does not support this optimization. Sampling old and new objects helps verify change scope. In reporting, do not treat fewer KMS events as direct proof of fewer S3 reads: the relationship between those operations has changed.
2. Follow a rotation attempt
Lambda rotation coordinates states in different systems. createSecret prepares the AWSPENDING version identified by the attempt; retrying with the same token should reuse that work. setSecret applies credentials at the target. testSecret checks the pending version and finishSecret promotes the current version. These steps are not interchangeable. A stored value does not prove the database changed, and promoting a label does not replace a target write. During diagnosis, record version identifiers, steps, times, and results without printing passwords. If AWSPENDING is separate from AWSCURRENT, investigate the previous execution before recovery. Removing labels without understanding state can hinder the reconciliation the team needs to perform.
3. Choose a strategy and maintain permissions
Single-user updates the credentials of one user. A short interval can exist between the database change and secret publication; existing connections are not automatically terminated. Plan suitable retries for new connections. Alternating-users maintains two users and favors availability, but later original-user permission changes must also be applied to the clone. Both can remain valid after rotation, so changing AWSCURRENT does not establish revocation of an exposed password. If containment is the concern, identify the compromised credential and validate the measure at the target. At RUN handover, include both cycles when the design alternates users to avoid success that occurs only on every other rotation.
4. Confirm destinations, cache, and functional testing
Lambda needs to communicate with the secrets API and the database. A working private Secrets Manager endpoint does not demonstrate permitted ingress and egress on the database port. If the function test confirms reading, do not infer that a business write was also rehearsed. The consumer adds another state: a cache can retain the old value after rotation. In AWS Workload Credentials Provider, refreshNow=true requests immediate refresh; that behavior should not be generalized to every library. Define how the client reacts to authentication failures and bounds retries. Operational proof is a new connection with the expected version and required operation, without revealing the secret in evidence or logs.
5. Reconcile states before rollback
In a fictional incident, the new database password works but the application uses an old entry. Moving AWSCURRENT to AWSPREVIOUS can publish a password that no longer authenticates. First distinguish published, accepted, and consumed values. The local exercise uses fictional identifiers to show these mismatches; it neither stores passwords nor simulates Secrets Manager. Reconciliation is an operational inference supported by the separation of steps. Real rollback needs an authorized procedure with testing after each relevant change. Do not use an open session as the only proof: it can keep working while new connections fail. Record the decision, owner, and criterion for resuming processing.
6. Plan the window, retirement, and evidence
A rotation window defines an interval rather than an exact execution second. Coordinate teams through observed state and recovery criteria, especially near closing processes or out-of-hours changes. During retirement, a secret recovery window does not preserve availability: scheduling deletion makes it inaccessible before permanent removal. Confirm consumers before that decision and prepare recovery if an unidentified dependency appears. The acceptance package should combine version, new-connection tests, business operations, consumer coverage, and known limits. The lesson summary is to treat the change as state coordination. Saved configuration, promoted label, and recovered application are different events needing their own evidence.
# Original fictional version-state exercise, not a secret rotation implementation.
# No credentials, network calls, or AWS changes.
def diagnosis(published, accepted, consumed):
if published!= accepted:
return "reconcile-publication-and-target"
if consumed!= published:
return "refresh-consumer-and-test"
return "versions-aligned-test-new-connection"
assert diagnosis("v2", "v2", "v1") == "refresh-consumer-and-test"
assert diagnosis("v1", "v2", "v1") == "reconcile-publication-and-target"
assert diagnosis("v2", "v1", "v2") == "reconcile-publication-and-target"
assert diagnosis("v2", "v2", "v2") == "versions-aligned-test-new-connection"
assert diagnosis("v3", "v3", "v2") == "refresh-consumer-and-test"
print("five version-state cases passed; aligned versions still require functional tests")
The new version authenticates, but the consumer retains old cache; the team refreshes retrieval and tests a new connection before considering rollback.
Common pitfalls
AWSPREVIOUS as a still-valid password; rotation as revocation; open connection as new testing; recovery window as a readable period.
Related topics: Federation and KMS authorization · Infrastructure paths and governance
Change closure requires consistency between published version, accepted credential, and consumed value.
Reference: S3 Bucket Keys · SCS-C03