LEARN. PRACTICE. GROW.Define what must survive failure
An APS team at a fictional bank prepares a funds-service recovery rehearsal. The primary path fails while positions are being sent to a partner. The objective is to retain authorized processing within agreed capacity while preserving transport controls. This case does not describe internal BNP Paribas procedures. Before counting tunnels, list each path’s dependencies: cloud interfaces, peer routers, physical links, DNS resolution, TLS termination and security rules. HA VPN topology has specific interface requirements. Two tunnels on the same interface do not replace one tunnel on each interface. Even with that distribution, two IPs on one peer device without internal redundancy remain dependent on that device. Record the cloud topology condition separately from the complete system’s failure assumption. A product availability commitment does not establish application availability. As a guided exercise, draw two paths and mark a shared router. Remove that router from the drawing and identify consumers left without a path. Then add an independent path and check whether it preserves required controls. Avoid proposing features outside supported scope: Cloud Router BFD is documented for Interconnect VLAN attachment sessions rather than this case’s HA VPN sessions. The manager should turn each gap into a decision with an owner, required evidence and an effect on the rehearsal schedule.
Interpret the health signal that actually exists
The dashboard shows healthy DNS and the team wants to close recovery. First ask what that signal measures. For an internal passthrough Network Load Balancer without backend instances, Cloud DNS can consider the endpoint healthy. With external endpoint health checking, when every target is unhealthy, responses can still include those targets. A returned address does not establish a working service, and health check failure is not a universal traffic-blocking mechanism. Hybrid resolution also needs consistent data. Cloud DNS ranks forwarding targets automatically, considering responses and latency. NXDOMAIN counts as a successful response in that context. If one server contains the record and another says the name does not exist, do not treat the targets as a guaranteed sequential search until an address is found. Correct the data divergence and check the path actually used by the workload. During rehearsal, collect three separate observations: DNS response, applicable probe result and the consumer’s functional operation. Identify the queried name, origin and time to avoid comparing different tests as though they were the same. Give the observer one case where a VIP exists without backends and another where a forwarder returns NXDOMAIN. Ask for a supported conclusion and next verification for each. Reporting should retain the distinction between a resolved name, an evaluated endpoint and recovered business processing.
Confirm the certificate presented to the client
The team renewed a wildcard certificate, but the client still receives earlier material. The certificate map contains an exact entry for the client’s SNI. Exact matching takes precedence over the wildcard; updating another resource does not necessarily change the selected certificate. Acceptance should use the actual recovery hostname and check the handshake, chain, validity and name match. Wildcard coverage also has boundaries. A certificate for *.funds.example does not cover api.eu.funds.example. Separate this issue from DNS resolution: pointing both names at one VIP does not change covered names. Also identify who renews the material. Self-managed certificates require renewal and replacement by the responsible team; appearing in Certificate Manager does not turn them into Google-managed certificates. For DNS authorization, preserve the CNAME dependency needed for renewal as well as initial issuance. Prepare an original table containing hostname, test SNI, selected entry, observed certificate and result. Include a row whose HTTP health check succeeds but whose client handshake fails. The purpose is to demonstrate why a health response does not replace TLS validation. If the new destination is still incorrect, present the option of temporarily retaining a previously authorized destination with a deadline and exit condition. Do not disable client validation to obtain a green result that leaves the approved criterion unmet.
Preserve control during traffic transition
A Cloud Armor preview rule matches a request, but evaluation continues to the enforced rule allowing traffic. The rehearsal showed the deny’s potential effect; it did not show actual blocking. In reporting, associate every decision with rule mode and policy version. If import fails because a fingerprint is stale, retrieve current state and reconcile the concurrent change. Pasting the current fingerprint into an old export can erase a legitimate correction. For outbound communication with a partner, the team wants to replace a manually configured Public NAT IP. Draining lets existing sessions continue until natural expiry while new connections use active addresses. At least one active address must remain. The partner needs to accept the new source before transition can be considered functional. A file transferred through an earlier session does not establish that acceptance. When draining through gcloud, moving from the active pool to the drain pool must be part of the same change. Removing first and draining later can terminate active sessions. Build a change sequence covering authorization of the new source, a new-connection test, draining and observation of remaining sessions. State what would trigger reversal and what cannot be recovered merely by adding an address again. In handover, identify which sessions tested the active destination and which demonstrate only continuity of earlier state.
Recover boundaries with explicit scope and versions
Promotion of one perimeter was approved, but the script uses dry-run enforce-all on the access policy. Another team has a prepared experiment in that scope. The broad command can promote that configuration too. Operation selection must match approved scope. An etag identifies a revision; it does not turn an all-perimeters operation into an individual operation. When an explicit dry-run configuration exists, compare spec with status before promotion. A change in enforced mode does not establish that rehearsal configuration became identical. An old proposal can reintroduce a rule removed in production. The change record should show which configurations were reviewed together and who confirmed flows that must remain allowed and denied. In a GKE Standard cluster with the legacy dataplane, enabling network policy enforcement requires node recreation and respects maintenance policies. Acceptance of configuration does not mean transition is complete. During migration to GKE Dataplane V2, also review manifests identifying Pods through ipBlock.cidr and validate suitable labels and selectors. Application Load Balancer probes need to traverse the applicable policy. A successful internal test does not establish that path. As the deliverable, build a short matrix containing resource, intended state, observed state and pending test. Include the node-update timing dependency in the plan instead of announcing coverage before evidence exists.
Distinguish convergence, capacity and packet size
Two independent paths each support six abstract units and demand is nine. Split operation provides enough aggregate capacity. When one fails, six units remain and the shortfall is three. These numbers are teaching assumptions rather than product throughput limits. Defining redundancy without testing surviving capacity can conceal degradation at the very moment failover occurs. Control-plane convergence is another dimension. During Cloud Router software maintenance, graceful restart at the peer helps retain routes. If disabled and CEASE occurs, merely increasing the hold timer does not prevent explicit session termination. Do not confuse missing keepalives with a termination notification. Record expected behavior for each scenario and measure it in an authorized rehearsal instead of inferring recovery time from the final console state. After the path changes, test representative traffic. Small TCP packets can succeed while large UDP packets with DF fail. MSS clamping is a TCP mechanism; for the other traffic, analyze payload MTU, encapsulation and delivery of required ICMP messages. External MTU is not automatically the space available to payload. Prepare three separate pieces of committee evidence: path existence, capacity under load and compatibility with the protocols and sizes used. Positive evidence in one column does not fill a gap in the others. Assign an owner to each remaining uncertainty before proposing acceptance of the tested failure scenario.
Model shared dependencies without double-counting capacity
The local exercise represents a network as a directed graph with abstract integer capacities. Each edge has failure domains and declared states for required controls. A failure removes every edge depending on the affected domain. An edge with a missing, unknown or failed control is excluded from eligible capacity. This is a conservative exercise rule rather than an automatic reading of cloud configuration. First predict common-router-fails. Although two paths are drawn, both lose an essential edge when their shared router fails. Then compare shared-capacity-bottleneck: two six-unit branches pass through one shared seven-unit edge. Summing branches would give twelve, but the shared constraint keeps total capacity at seven. The algorithm finds maximum flow in the eligible graph, and tests compare its result with enumerated cuts on a small graph. The model allows demand units to be split in one direction. It does not represent indivisible sessions, ECMP, latency, loss, routing protocols or convergence time. An edge drawn in the reverse direction does not automatically create a forward path. Mark inventory incomplete and observe that calculated capacity remains visible but no longer supports scenario acceptance. Explain in writing which assumptions would require real measurement before using the result in an operational decision. The calculation helps identify contradictions; it does not replace an application rehearsal.
Deliver a recovery decision that can be checked
Run the program without credentials using the supplied fictional data. Before inspecting output, write the capacity you expect for one-path failure, shared-router failure and an unknown backup control. Compare your prediction with capacityUnderDeclaredModel, shortfall and excludedControlEdges. If it differs, explain which assumption or edge was overlooked. Tests cover capacity, control and failure combinations, input permutations and rejection of invalid models. Repeated IDs, unknown failure domains and negative capacities are not silently converted into useful data. The program preserves input, contacts no services and writes no configuration. Its hash records which code version produced evidence; it does not authenticate declared capacities or control states. At handover, provide a short note with the failure scenario, required demand, removed dependencies, eligible capacity, missing controls and next validation. If modeled capacity is sufficient but inventory is incomplete, keep both statements explicit. Distinguish who observed the rehearsal, who accepts residual risk and who would execute the change. The final decision still needs application tests, timing criteria and actual authorization. Complete the lesson by answering the certificate and shared-dependency cases and explaining why alternatives fail to meet the supplied constraints. Use the explanation to identify the exact evidence that would change your decision, rather than relying on an overall green dashboard.
"""Synthetic directed capacity worksheet, not a cloud routing simulator.
Capacity units are divisible abstract units for one direction and one demand.
Every selected edge must pass every declared control. Failure domains remove
all affected edges. This does not model BGP, ECMP, latency or real throughput.
"""
from collections import deque
from copy import deepcopy
from hashlib import sha256
from itertools import combinations, permutations, product
from pathlib import Path
import json
def label(x):
return isinstance(x, str) and bool(x.strip) and x == x.strip
def unique_labels(values, nonempty=True):
return isinstance(values, list) and (bool(values) or not nonempty) and all(label(x) for x in values) and len(values) == len(set(values))
def validate(model, failed):
if not isinstance(model, dict) or set(model)!= {'nodes','source','sink','demand','controls','edges','inventoryComplete'}:
raise ValueError('Expected exact worksheet model fields')
if not unique_labels(model['nodes']) or not unique_labels(model['controls']):
raise ValueError('Unique nonempty node and control lists required')
if not all(label(model[k]) and model[k] in model['nodes'] for k in ['source','sink']) or model['source'] == model['sink']:
raise ValueError('Source and sink must be distinct declared nodes')
if type(model['demand']) is not int or model['demand'] <= 0:
raise ValueError('Demand must be a positive integer')
if type(model['inventoryComplete']) is not bool or not isinstance(model['edges'], list):
raise ValueError('Explicit inventory flag and edge list required')
ids, pairs, domains = set, set, set
for edge in model['edges']:
if not isinstance(edge, dict) or set(edge)!= {'id','src','dst','capacity','domains','controls'}:
raise ValueError('Expected exact edge fields')
if not label(edge['id']) or edge['id'] in ids:
raise ValueError('Unique nonblank edge ID required')
ids.add(edge['id'])
if not all(label(edge[k]) and edge[k] in model['nodes'] for k in ['src','dst']) or edge['src'] == edge['dst']:
raise ValueError('Edge endpoints must be distinct declared nodes')
pair = (edge['src'], edge['dst'])
if pair in pairs:
raise ValueError('Duplicate directed pair; use explicit intermediate nodes')
pairs.add(pair)
if type(edge['capacity']) is not int or edge['capacity'] < 0:
raise ValueError('Capacity must be a nonnegative integer')
if not unique_labels(edge['domains']):
raise ValueError('Each edge needs unique declared failure domains')
domains.update(edge['domains'])
states = edge['controls']
if not isinstance(states, dict) or any(k not in model['controls'] or v not in ['pass','fail','unknown'] for k,v in states.items):
raise ValueError('Only declared controls and recognized states are accepted')
if not unique_labels(failed, nonempty=False) or not set(failed) <= domains:
raise ValueError('Failure scenarios must name declared domains without duplicates')
def capacity(model, edges):
"""Augmenting-path maximum for this synthetic directed graph only."""
nodes = model['nodes']
residual = {u:{v:0 for v in nodes} for u in nodes}
for e in edges:
residual[e['src']][e['dst']] += e['capacity']
source, sink, value = model['source'], model['sink'], 0
while True:
previous, queue = {source:None}, deque([source])
while queue and sink not in previous:
u = queue.popleft
for v in sorted(nodes):
if v not in previous and residual[u][v] > 0:
previous[v] = u
queue.append(v)
if sink not in previous:
return value
increment, v = None, sink
while previous[v] is not None:
u = previous[v]
increment = residual[u][v] if increment is None else min(increment,residual[u][v])
v = u
v = sink
while previous[v] is not None:
u = previous[v]
residual[u][v] -= increment
residual[v][u] += increment
v = u
value += increment
def analyze(model, failed):
validate(model, failed)
disabled, unproven, eligible = [], [], []
for e in model['edges']:
if set(e['domains']) & set(failed):
disabled.append(e['id'])
elif any(e['controls'].get(c,'unknown')!= 'pass' for c in model['controls']):
unproven.append(e['id'])
else:
eligible.append(e)
maximum = capacity(model, eligible)
return {'failedDomains':sorted(failed),'disabledEdges':sorted(disabled),
'excludedControlEdges':sorted(unproven),'eligibleEdges':sorted(e['id']for e in eligible),
'capacityUnderDeclaredModel':maximum,'demand':model['demand'],
'shortfall':max(0,model['demand']-maximum),
'meetsDeclaredCapacity':maximum >= model['demand'],
'inventoryCoverageUnproven':not model['inventoryComplete'],
'scenarioSupportedUnderAssumptions':maximum >= model['demand'] and model['inventoryComplete'],
'realThroughputMeasured':False,'routingConvergenceVerified':False,
'controlAuthenticityVerified':False,'productionFailoverAuthorized':False}
def edge(id, src, dst, cap=6, domains=None):
return {'id':id,'src':src,'dst':dst,'capacity':cap,'domains':[id] if domains is None else domains,
'controls':{'transport':'pass','boundary':'pass'}}
def model:
return {'nodes':['source','a','b','sink'],'source':'source','sink':'sink','demand':6,
'controls':['transport','boundary'],'inventoryComplete':True,
'edges':[edge('sa','source','a'),edge('at','a','sink'),edge('sb','source','b'),edge('bt','b','sink')]}
def evidence:
fixtures=[]
def record(id,m,failed):
old=deepcopy((m,failed));r=analyze(m,failed);assert(m,failed)==old
fixtures.append({'id':id,**r});return r
m=model;assert record('two-independent-paths',m,[])['capacityUnderDeclaredModel']==12
assert record('one-path-fails',m,['sa'])['capacityUnderDeclaredModel']==6
assert record('two-paths-fail',m,['sa','sb'])['capacityUnderDeclaredModel']==0
m=model;m['demand']=9
assert record('survivor-under-capacity',m,['sa'])['shortfall']==3
m=model;m['edges'][0]['domains'].append('shared-router');m['edges'][2]['domains'].append('shared-router')
assert record('common-router-fails',m,['shared-router'])['capacityUnderDeclaredModel']==0
m=model;m['nodes'].append('shared');m['edges']=[edge('common','source','shared',7),edge('sa','shared','a'),edge('at','a','sink'),edge('sb','shared','b'),edge('bt','b','sink')];m['demand']=9
assert record('shared-capacity-bottleneck',m,[])['capacityUnderDeclaredModel']==7
m=model;m['edges'][2]['controls']['transport']='unknown'
assert record('unknown-backup-control',m,['sa'])['capacityUnderDeclaredModel']==0
m=model;del m['edges'][2]['controls']['transport']
assert record('missing-backup-control',m,['sa'])['excludedControlEdges']==['sb']
m=model;m['edges'][2]['controls']['boundary']='fail'
assert record('failed-backup-boundary',m,['sa'])['scenarioSupportedUnderAssumptions'] is False
m=model;m['inventoryComplete']=False
r=record('incomplete-inventory',m,[]);assert r['meetsDeclaredCapacity'] and not r['scenarioSupportedUnderAssumptions']
m=model;m['edges'][2]['src'],m['edges'][2]['dst']='b','source'
assert record('reverse-edge-not-forward-path',m,['sa'])['capacityUnderDeclaredModel']==0
m=model;m['edges'][2]['capacity']=0
assert record('zero-capacity-backup',m,['sa'])['shortfall']==6
m=model;m['edges'].append(edge('cycle','a','source',3));m['edges'].append(edge('cross','a','b',2))
assert record('cycles-do-not-create-capacity',m,[])['capacityUnderDeclaredModel']==12
m=model;m['edges']=[]
assert record('empty-declared-topology',m,[])['capacityUnderDeclaredModel']==0
# Independent minimum-cut oracle for all 0/1/2 capacities on a five-edge graph.
checked=0
m=model;m['edges'].append(edge('cross','a','b'))
for caps in product(range(3),repeat=5):
for e,c in zip(m['edges'],caps):e['capacity']=c
cut_values=[]
for count in range(3):
for middle in combinations(['a','b'],count):
side={'source',*middle}
cut_values.append(sum(e['capacity']for e in m['edges']if e['src']in side and e['dst']not in side))
assert capacity(m,m['edges'])==min(cut_values)
checked+=1
states=0
for state,failed in product(['pass','fail','unknown'],[[],['sa'],['sb'],['sa','sb']]):
m=model;m['edges'][2]['controls']['boundary']=state
expected=(0 if 'sa'in failed else 6)+(6 if state=='pass'and'sb'not in failed else 0)
assert analyze(m,failed)['capacityUnderDeclaredModel']==expected
states+=1
m=model;expected=analyze(m,['sa']);orders=0
for order in permutations(m['edges']):
candidate=deepcopy(m);candidate['edges']=list(order)
assert analyze(candidate,['sa'])==expected;orders+=1
invalid=[]
def changed(fn):
v=model;fn(v);invalid.append((v,[]))
changed(lambda m:m.update(demand=0))
changed(lambda m:m.update(demand=True))
changed(lambda m:m.update(nodes=['source','source','sink']))
changed(lambda m:m.update(source='absent'))
changed(lambda m:m.update(sink='source'))
changed(lambda m:m.update(controls=[]))
changed(lambda m:m.update(inventoryComplete='yes'))
changed(lambda m:m.update(edges={}))
changed(lambda m:m['edges'][0].update(capacity=-1))
changed(lambda m:m['edges'][0].update(capacity=1.5))
changed(lambda m:m['edges'][0].update(capacity=True))
changed(lambda m:m['edges'][0].update(src='absent'))
changed(lambda m:m['edges'][0].update(dst='source'))
changed(lambda m:m['edges'][0].update(domains=[]))
changed(lambda m:m['edges'][0].update(domains=['same','same']))
changed(lambda m:m['edges'][0].update(controls={'other':'pass'}))
changed(lambda m:m['edges'][0].update(controls={'transport':'yes'}))
changed(lambda m:m['edges'][1].update(id='sa'))
changed(lambda m:m['edges'].append(edge('duplicate','source','a')))
changed(lambda m:m.update(extra='unexpected'))
invalid.extend([(model,['unknown-domain']),(model,['sa','sa']),(model,None),(None,[])])
for bad,failures in invalid:
try:analyze(bad,failures)
except ValueError:pass
else:raise AssertionError('Invalid input accepted')
return {'scriptSha256':sha256(Path(__file__).read_bytes).hexdigest,'fixtures':fixtures,
'capacityCombinations':checked,'controlFailureCombinations':states,'inputPermutations':orders,
'invalidInputs':len(invalid),'inputPreserved':True,'orderIndependent':True,
'network':False,'cloudExecuted':False,'persistentWrites':False}
if __name__=='__main__':
print(json.dumps(evidence,ensure_ascii=False,indent=2))
Two six-unit branches share a seven-unit edge: eligible capacity is seven rather than twelve. If the shared router fails,both paths can disappear.
Common pitfalls
Counting tunnels as independence;summing shared capacity;treating preview as enforcement;testing only old sessions;confusing an updated certificate with the served certificate.
Related topics: Private networks,names and trust boundaries · Connectivity diagnosis and failure evidence · Trust recovery and access rollback
A recovery path needs demonstrated capacity,independence and controls for the scenario being accepted.
Reference: DNS routing policies and health checks · Current linked guide; edition date unconfirmed (2026-09-30 inspection)