LEARN. PRACTICE. GROW.Define evidence of usable recovery
An APS team at a fictional bank recovers position processing after accidental deletion. A backup exists and its hash matches the available record. The team still needs to demonstrate decryption, destination creation, restricted access and functional reconciliation. This case does not describe internal BNP Paribas procedures. The lesson turns the statement “we have a backup” into dependencies that can be checked, with owners and acceptance criteria. First separate the artifact, backup protection and destination protection. A backup vault configured with CMEK applies that protection to backups, including backups of disks that originally used Google-managed encryption. The key version protecting an older backup remains relevant after rotation. A new primary version does not establish availability of the earlier version. Record full identifiers, states, service identities and observed evidence, without including secret material in the report. Assign an owner to obtain each observation before the recovery window expires. For the guided exercise, draw one arrow from the restore plan to the backup and another to the destination. Add the cryptographic and authorization dependencies of each branch. For every arrow, describe the observation that would confirm the relationship. A hash can detect changed bytes against a reference value; it does not prove that the reference is authentic, that the key works or that the content is functionally correct. The manager tracks these conditions separately and keeps the service in recovery until results cover the agreed scope.
Restore objects within the correct window
The storage owner changes soft-delete retention from seven to thirty days after discovering the loss. That change protects subsequent deletions under the applicable policy; it does not retroactively extend the window of an object already deleted. The incident timeline should record deletion, the policy then applicable and the remaining deadline. A correct current configuration does not replace analysis of the configuration governing the event. Restoring an object creates a new live generation. Do not require that generation to equal the deleted generation, or declare recovery unsuccessful merely because the identifier differs. Keep a mapping between the source resource and the operation result. If the bucket was also deleted, distinguish tools: restoring a bucket through the CLI or JSON API initially returns an empty bucket and requires separate object restoration. The console can offer a flow that includes both. Establish which operation the team actually performed before investigating apparently missing files. In a bucket with hierarchical namespace, some operations can produce deleted objects with the same name and generation. When that combination is ambiguous, the restore API requires the restoreToken identifying the intended object. Do not invent a token or select the first inventory row. Confirm identity against metadata returned by the service. If the entire project was deleted, data availability is also constrained by the project recovery window. Record both dependencies instead of promising that bucket retention alone guarantees recovery.
Preserve concurrency and access scope
The application starts writing to the recovery path while the team prepares restoration. If the approved requirement is “restore only when no live version exists,” an earlier lookup is insufficient: another process can write between that lookup and the change. The ifGenerationMatch=0 precondition attaches the requirement to the restore operation. If a live version already exists, the conditional operation should fail so the team can reconcile state instead of silently replacing recent work. For a bucket without uniform bucket-level access, the ACL choice also matters. With copySourceAcl=true, the operation copies the source object’s ACL. When that parameter is omitted or false, the restored object inherits the bucket’s default object ACL. In this fictional case, the default ACL is broader than the deleted object’s restricted ACL. Restoring identical bytes can therefore change who can read them. Review the intended ACL and effective access policies; copying an ACL does not cancel grants at other levels of the resource hierarchy. Managed folders introduce another dependency: restoring objects does not restore those folders’ IAM policies. Include authorized recovery or reconciliation of these policies in the plan, then check access using the appropriate consuming identity. During a rehearsal, ask the learner to write two acceptance conditions: no concurrent replacement and access limited to the approved audience. Each needs its own evidence. The restored generation, a matching hash and a successful HTTP response do not establish both conditions on their own.
Cancel destruction without reopening the secret
A Secret Manager version has been scheduled for destruction on a secret configured with delayed destruction. During that waiting period, the version is disabled. The incident is subsequently identified as a mistake, but permission to read the secret again has not been granted. Cancellation can use an enable or disable operation on the version. In this case, disable cancels scheduled destruction while keeping reads blocked. The operational requirement determines the appropriate transition. Do not turn that mechanism into a general recovery promise. Without the delay configuration, destruction can be immediate. The plan must identify the exact version, observed state, available deadline and documented cancellation operation. Changing a setting for future actions does not prove that an already scheduled destruction was cancelled. After intervention, collect the effective state and establish separately whether the version should remain disabled or be enabled through an authorized decision. Include the evidence owner and the time of observation in the incident record. Cloud KMS has its own lifecycle and needs a separate analysis. A service-generated version that has been permanently destroyed cannot be recovered through a simple state change. However, there is a documented exception for certain previously imported versions, covered in the next section. At an incident committee, avoid both “all keys are recoverable” and “no destroyed key can return.” Present the version type, eligibility and available material. The exercise asks for a bounded decision: preserve the recovery opportunity without granting access before approval.
Distinguish backup and destination keys
The backup vault depends on a CMEK version that has become inaccessible. The team has a new, enabled key authorized for the destination disk. This permits analysis of destination protection, but does not remove the need to decrypt the backup. A key override during individual-disk restoration does not make an inaccessible vault-protection key usable. Treat the two branches of the initial drawing as separate dependencies and check their corresponding service identities. For individual-disk restoration, using another CMEK requires the destination project’s Compute Engine service agent to have the documented encrypter/decrypter permissions on that key. A human operator’s backup access does not establish this service authorization. Also distinguish whole-VM recovery, whose default behavior retains disk encryption configuration, from the override option in the documented individual-disk flow. A recovery runbook should identify the actual operation being approved rather than merely saying “restore the VM or disk.” For a previously imported Cloud KMS version, inspect reimport_eligible and the reimport conditions. If an earlier import succeeded, the same original material is required; a new key with the same algorithm and size is not a substitute. Documentation distinguishes the situation in which no import ever completed successfully. Do not use that exception to justify different material for a version that already protected data. With Cloud EKM, add external key-service availability to the analysis: available IAM permissions and local data do not prevent read or write failures when the external cryptographic dependency is unavailable.
Restore AI controls within the approved scope
The fictional application uses Model Armor to evaluate requests before forwarding them. During recovery, the team moves the application location and configures the global endpoint to reuse a template. That endpoint supports floor settings only; it does not support the template and sanitization operations used here. A template’s location also cannot be changed after creation. Prepare the supported resource and endpoint for the approved destination, then confirm that the application uses that configuration. The exercise’s fictional policy requires processing in the authorized location. With data residency enforcement, features unavailable locally can be disabled. An expected filter’s absence does not justify automatically disabling the restriction to obtain results resembling the previous environment. Identify the affected feature, lost coverage and options compatible with the agreed policy. The team may need to defer activation, constrain the flow or obtain an explicit decision on a different design. Do not present this case requirement as a universal legal rule or infer compliance merely from a selected region. Filter versions require additional evidence. Stable is an alias that can move to a newer version. To compare rehearsal results, record the effective version, parameters and evaluation examples. Pinning a specific version helps while that version remains available, but does not guarantee unchanged behavior forever. When a version becomes Retired, requests use Stable. Not every filter uses the version mechanism. The report must identify which controls have an established version and which still require validation.
Inventory caches and check local dependencies
The team removes temporary resources after rehearsal and cannot find a cache in the recovery destination. The resource identifier includes its project and creation location. The application’s current location does not automatically change those components. Use the original resource inventory to address deletion of the correct cache, and record the observed response. A location error does not establish that the content no longer exists. The absence of explicit caches also does not establish the absence of implicit caching. When this exercise’s fictional policy prohibits cache retention, the decision must account for both mechanisms: disable implicit caching and avoid explicit caching according to applicable documentation. Do not assume that deleting a source file proves removal of every copy or cache. Define the scope of resources examined and what the collected evidence actually supports before declaring cleanup complete. The lesson’s Python program uses fictional strings and a dependency graph declared by the learner. Each root can represent a recovery plan; requires lists direct dependencies. The program follows transitive dependencies too, reporting missing references, unknown or unavailable states and mismatching hashes. Identifiers are exact: vault-key/v4 does not satisfy a reference to vault-key/v3. A failure outside the analyzed branch does not block that branch. However, an inventory declared incomplete prevents supportedUnderDeclaredEvidence, even when every dependency found appears consistent. This conclusion applies exclusively to supplied data and does not involve querying cloud services.
Run the exercise and prepare the RUN handover
Run python3 run.py locally without credentials. Before reading results, predict the effect of four changes: remove the vault key, add a new version with another identifier, change only the destination key and alter backup bytes. Compare your prediction with requiredClosure, missing, unavailable and hashMismatches. If it differs, identify the dependency missing from your reasoning and correct the drawing before changing the code. Keep the prediction alongside the resulting evidence so another learner can follow your reasoning. The self-consistent-untrusted-manifest case changes both the fictional bytes and the expected hash. The comparison becomes consistent again, but manifestAuthenticityVerified remains false. This limitation is intentional: a manifest controlled by the same source as the data does not establish authenticity of either. The available and unknown states are also exercise declarations rather than actual IAM or KMS responses. The program performs no encryption, decryption or data restoration, authenticates no evidence and authorizes no production operation. Tests cover transitive dependencies, independent roots, state combinations, order permutations and rejection of cycles or invalid fields. Input is preserved. To finish the lesson, write a RUN handover note with the recovered resource, resulting generation, cryptographic dependencies, intended access, functional result and remaining gaps. If only inventory consistency has been established, state exactly that. Incident closure needs the real observations defined at the beginning. Solve the two final cases and explain why the alternatives fail the concurrency or decryption constraint.
"""Original offline recovery-dependency worksheet using fictional strings.
No encryption, decryption, KMS calls or actual restore is performed. States,
references and expected hashes are supplied by the learner, not authenticated.
Exact dependency IDs matter: a new primary key does not satisfy another ID.
"""
from copy import deepcopy
from hashlib import sha256
from itertools import permutations, product
from pathlib import Path
import json
import re
def label(x):
return isinstance(x,str) and bool(x.strip) and x == x.strip
def unique(values, nonempty=False):
return isinstance(values,list) and (bool(values) or not nonempty) and all(label(x) for x in values) and len(values)==len(set(values))
def validate(model):
if not isinstance(model,dict) or set(model)!={'nodes','roots','inventoryComplete'}:
raise ValueError('Expected exact worksheet fields')
if not isinstance(model['nodes'],list) or not unique(model['roots'],True) or type(model['inventoryComplete']) is not bool:
raise ValueError('Nodes, unique roots and inventory flag required')
indexed={}
for n in model['nodes']:
if not isinstance(n,dict) or set(n)!={'id','kind','state','requires','payload','expectedSha256'}:
raise ValueError('Expected exact node fields')
if not label(n['id']) or n['id'] in indexed:
raise ValueError('Unique nonblank node IDs required')
if n['kind'] not in ['artifact','dependency'] or n['state'] not in ['available','unavailable','unknown'] or not unique(n['requires']):
raise ValueError('Recognized kind, state and unique dependencies required')
if n['kind']=='artifact':
if not isinstance(n['payload'],str) or not isinstance(n['expectedSha256'],str) or not re.fullmatch('[0-9a-f]{64}',n['expectedSha256']):
raise ValueError('Artifact requires synthetic text and lowercase SHA256')
elif n['payload'] is not None or n['expectedSha256'] is not None:
raise ValueError('Dependency nodes do not carry artifact bytes')
indexed[n['id']]=n
visiting,done=set,set
def visit(id):
if id not in indexed or id in done:return
if id in visiting:raise ValueError('Dependency cycle')
visiting.add(id)
for dep in indexed[id]['requires']:visit(dep)
visiting.remove(id);done.add(id)
for id in indexed:visit(id)
return indexed
def analyze(model):
nodes=validate(model);results=[]
for root in sorted(model['roots']):
closure=set
def walk(id):
if id in closure:return
closure.add(id)
if id in nodes:
for dep in nodes[id]['requires']:walk(dep)
walk(root)
missing=sorted(closure-nodes.keys)
unavailable=sorted(id for id in closure & nodes.keys if nodes[id]['state']=='unavailable')
unknown=sorted(id for id in closure & nodes.keys if nodes[id]['state']=='unknown')
mismatches=sorted(id for id in closure & nodes.keys if nodes[id]['kind']=='artifact' and sha256(nodes[id]['payload'].encode('utf-8')).hexdigest!=nodes[id]['expectedSha256'])
consistent=not(missing or unavailable or unknown or mismatches)
results.append({'root':root,'requiredClosure':sorted(closure),'missing':missing,
'unavailable':unavailable,'unknown':unknown,'hashMismatches':mismatches,
'declaredDependenciesConsistent':consistent,
'inventoryCoverageUnproven':not model['inventoryComplete'],
'supportedUnderDeclaredEvidence':consistent and model['inventoryComplete'],
'manifestAuthenticityVerified':False,'cryptographicRecoveryExecuted':False,
'cloudAuthorizationVerified':False,'productionRestoreAuthorized':False})
return results
def node(id, requires=, state='available', text=None):
return {'id':id,'kind':'dependency' if text is None else 'artifact','state':state,
'requires':list(requires),'payload':text,
'expectedSha256':None if text is None else sha256(text.encode).hexdigest}
def model:
return {'roots':['restore-plan'],'inventoryComplete':True,'nodes':[
node('vault-key/v3'),node('vault-agent-access'),node('backup', ['vault-key/v3','vault-agent-access'],text='fictional encrypted backup bytes'),
node('target-key/v8'),node('target-agent-access'),
node('restore-plan',['backup','target-key/v8','target-agent-access'])]}
def evidence:
fixtures=[]
def record(id,m):
old=deepcopy(m);r=analyze(m);assert m==old
fixtures.append({'id':id,'results':r});return r
assert record('complete-declared-chain',model)[0]['supportedUnderDeclaredEvidence']
m=model;m['nodes']=[n for n in m['nodes']if n['id']!='vault-key/v3']
assert record('missing-vault-key',m)[0]['missing']==['vault-key/v3']
m['nodes'].append(node('vault-key/v4'))
assert record('new-primary-not-old-version',m)[0]['missing']==['vault-key/v3']
m=model;m['nodes'][0]['state']='unavailable'
assert record('vault-key-unavailable',m)[0]['unavailable']==['vault-key/v3']
m=model;m['nodes'][1]['state']='unknown'
assert record('vault-agent-unknown',m)[0]['unknown']==['vault-agent-access']
m=model;m['nodes'][4]['state']='unavailable'
assert record('target-agent-unavailable',m)[0]['unavailable']==['target-agent-access']
m=model;m['nodes'][2]['payload']='changed bytes'
assert record('backup-hash-mismatch',m)[0]['hashMismatches']==['backup']
m['nodes'][2]['expectedSha256']=sha256(b'changed bytes').hexdigest
r=record('self-consistent-untrusted-manifest',m)[0]
assert r['declaredDependenciesConsistent'] and not r['manifestAuthenticityVerified']
m=model;m['inventoryComplete']=False
r=record('incomplete-inventory',m)[0];assert r['declaredDependenciesConsistent'] and not r['supportedUnderDeclaredEvidence']
m=model;m['nodes'][0]['state']='unavailable'm['nodes'][3]=node('target-key/v9');m['nodes'][-1]['requires'][1]='target-key/v9'
assert record('target-override-does-not-recover-backup',m)[0]['unavailable']==['vault-key/v3']
m=model;m['nodes'].append(node('unused-old-key',state='unavailable'))
assert record('unrelated-failure-outside-closure',m)[0]['supportedUnderDeclaredEvidence']
m=model;m['nodes'].append(node('other-backup',['missing-other-key'],text='other bytes'));m['roots'].append('other-backup')
r=record('independent-root-results',m);assert r[0]['missing']==['missing-other-key'] and r[1]['supportedUnderDeclaredEvidence']
m=model;m['roots']=['absent-root']
assert record('missing-root',m)[0]['missing']==['absent-root']
m=model;m['nodes'][-1]['requires'].append('source-policy');m['nodes'].append(node('source-policy',['acl-evidence'],state='unknown'))
r=record('transitive-policy-gap',m)[0];assert r['missing']==['acl-evidence'] and r['unknown']==['source-policy']
combinations=0
for key,access,inventory in product(['available','unavailable','unknown'],['available','unavailable','unknown'],[False,True]):
m=model;m['nodes'][0]['state']=key;m['nodes'][1]['state']=access;m['inventoryComplete']=inventory
r=analyze(m)[0]
assert r['supportedUnderDeclaredEvidence']==(key==access=='available' and inventory)
combinations+=1
m=model;expected=analyze(m);orders=0
for order in permutations(m['nodes'][:3]):
other=deepcopy(m);other['nodes']=list(order)+other['nodes'][3:]
assert analyze(other)==expected;orders+=1
bad=[]
def altered(fn):
v=model;fn(v);bad.append(v)
altered(lambda m:m.update(roots=[]))
altered(lambda m:m.update(roots=['backup','backup']))
altered(lambda m:m.update(roots=[' ']))
altered(lambda m:m.update(inventoryComplete='yes'))
altered(lambda m:m.update(nodes={}))
altered(lambda m:m['nodes'].append(deepcopy(m['nodes'][0])))
altered(lambda m:m['nodes'][0].update(id=''))
altered(lambda m:m['nodes'][0].update(kind='key'))
altered(lambda m:m['nodes'][0].update(state='enabled'))
altered(lambda m:m['nodes'][0].update(requires=['backup','backup']))
altered(lambda m:m['nodes'][0].update(requires='backup'))
altered(lambda m:m['nodes'][0].update(requires=['vault-key/v3']))
altered(lambda m:m['nodes'][0].update(requires=['restore-plan']))
altered(lambda m:m['nodes'][0].update(payload='secret material'))
altered(lambda m:m['nodes'][0].update(expectedSha256='0'*64))
altered(lambda m:m['nodes'][2].update(payload=None))
altered(lambda m:m['nodes'][2].update(expectedSha256='not-a-digest'))
altered(lambda m:m['nodes'][2].update(expectedSha256='A'*64))
altered(lambda m:m['nodes'][2].update(extra=True))
altered(lambda m:m.update(extra=True))
bad.extend([None,[]])
for value in bad:
try:analyze(value)
except ValueError:pass
else:raise AssertionError('Invalid input accepted')
return {'scriptSha256':sha256(Path(__file__).read_bytes).hexdigest,'fixtures':fixtures,
'stateCombinations':combinations,'inputPermutations':orders,'invalidInputs':len(bad),
'inputPreserved':True,'orderIndependent':True,'network':False,'cloudExecuted':False,'persistentWrites':False}
if __name__=='__main__':
print(json.dumps(evidence,ensure_ascii=False,indent=2))
The backup hash matches, but the vault key version is unavailable. A new destination disk key does not resolve that decryption dependency.
Common pitfalls
Confusing current with retroactive retention; restoring bytes without reviewing ACLs; replacing an old key with the new primary in inventory; treating hashes as authenticity evidence.
Related topics: Data protection and key management · Trust recovery and access rollback · Failover with preserved capacity and trust
Usable restoration requires recoverable data, available cryptographic dependencies, correct access and functional validation within the approved scope.
Reference: Soft delete overview · Current linked guide; edition date unconfirmed (2026-09-30 inspection)