LEARN. PRACTICE. GROW.Prepare a change that can be recovered
A fictional payment platform will replace its TLS chain before daily close. The change appears small: publish a Secret and confirm the endpoint responds. Yet consumers have different bundles, two controller replicas serve traffic and persistent sessions remain open. APS needs to know which configuration and trust combination is actually active along each path. One green result may represent only the most recently updated consumer. Before the window, define the starting state, authorized target and criterion for returning. Retain manifest versions, Secret references, public certificates, artifact hashes and recovery procedures. Protect private keys through the approved mechanism without including them in the ticket. Assign owners for trust distribution, server promotion, client validation and continuity decisions. Rollback combines dependencies. An old certificate can remain time-valid while no longer providing a trusted chain for new clients. Previous configuration may depend on a binary retired for known risk. Establish what must remain available and authorized during the window. This lesson’s examples are original and fictional and do not represent internal BNP Paribas procedures. The objective is an operational decision grounded in observed results. A useful change record lets another engineer identify the exact restoration target and explain which consumers are expected to accept it.
Validate policies against the traffic that should change
Build a small matrix of source, destination, port, protocol and expected result. Include a necessary connection and a similar connection that must be denied. If source and destination are isolated in their applicable directions, inspect both. Correcting database ingress alone does not establish that frontend egress permits session initiation. Keep DNS and availability as separate diagnostic hypotheses. A NetworkPolicy accepted by the API does not establish plugin support for every feature. With endPort, the implementation must support ranges; test the beginning, an intermediate point and the boundary, plus a port outside scope. This lesson’s TLS exercise runs neither a CNI nor cluster policies. Network scenarios assess documentation interpretation and design of necessary evidence without presenting those decisions as measured dataplane results. After removing permission, a new connection may fail while an old session continues. Effects on existing connections depend on implementation. If the requirement includes terminating open sessions, define a separate measure and assess impact. During rollback, deleting the sole policy isolating one direction can open access. Restoring connectivity and restoring the security boundary require explicit checks. Record both outcomes so the next operator does not interpret business recovery alone as proof that the intended restrictions remain effective.
Control selection, benchmarks and node exposure
A YAML correction can change who matches permission. A namespaceSelector and podSelector in the same entry require their combination; separate entries represent alternatives. Reviewing label values without reading structure can widen access. Also confirm who can change those labels. Policy may remain intact while a team makes its namespace eligible for a sensitive destination. A platform change requires reassessing benchmark selection and assessment scope. The public CIS page distinguishes Kubernetes from managed-service variants. Use the applicable profile, version and concrete responsibilities; do not reuse an old report solely because the application is unchanged. This lesson reproduces no numbered CIS PDF recommendations and claims no complete CIS assessment. Exercises use original remediation and risk-acceptance decisions. When a new node group uses another template, confirm paths to cloud metadata and administration endpoints. A successful administrative kubelet test does not establish anonymous-access denial. A Ready node does not establish that it loaded the hardening correction. Link evidence to the effective process and repeat relevant tests after the change, including the condition that must be refused and operational access that must remain. If remediation must wait for a controlled window, keep the failed state visible alongside the exception’s owner, compensating measure and review deadline.
Distribute trust before switching the server
In the teaching rotation, the old service uses one root and the new service uses another root with an intermediate CA. The service DNS name remains unchanged. A client knowing only the old root does not accept the new chain. Before promotion, distribute authorized overlapping trust and confirm that relevant consumers can establish new connections along both intended paths. Overlap has a duration and scope. Adding a root widens the set of accepted issuers, so it must correspond to the approved change. After migration, retire the old root only when inventory and tests establish that it is no longer needed, including the recovery strategy. Do not copy the server’s private key to clients: they need public trust material appropriate to their role. The laboratory uses explicit SSLContext objects without loading system trust or disabling hostname verification. It demonstrates four relationships: old trust accepts the old server, new trust rejects the old server, old trust rejects the new server and overlap accepts both. Results belong to the generated certificates and contexts. They demonstrate neither revocation, mTLS, enterprise bundle distribution nor any Ingress controller’s behavior. Those elements need their own validation in the intended environment. Keep acceptance evidence per consumer group instead of assuming every runtime reloads trust identically.
Deliver the chain and validate the candidate
A correct leaf is insufficient when the client cannot build a path to its trusted root. The exercise starts one endpoint delivering only the new leaf and another delivering leaf plus intermediate. The client store contains only the new root. The first is rejected and the second succeeds. The supplied intermediate helps construct the path; it does not authorize trusting arbitrary CAs sent by the server. Prepare a candidate context with matching chain and key before changing the active context. In the trial, loading the old key with the new chain fails. The already-valid service continues responding because the rejected candidate was never promoted. This makes a local failure observable without deliberately introducing it into consumer traffic. An updated Secret still requires controller adoption. Observation and reload timing vary by implementation. If two paths return different fingerprints, correlate replicas, configuration and actually served material. Do not automatically conclude that the CA signed incorrectly. Validate name, chain and connection outcome along the consumer path. Retain the authorized previous state until promotion and recovery criteria are met, without leaving private keys in logs or reports. Where automation writes multiple files, verify the coherent set before activation rather than inferring consistency from individual write success.
Separate files, contexts and sessions
The exercise retains a client with old trust already loaded. It then changes the file to include both roots. That context still rejects the new server; a context constructed from the updated file succeeds. The same pattern appears on the server: replacing disk files does not change the certificate an already-loaded context presents to a new connection. The exercise explicitly promotes a new context. A session opened before promotion continues exchanging messages and retains the fingerprint observed in its old handshake. Meanwhile, a new connection receives the new certificate. Both results can be true. To validate rotation, distinguish existing-session availability, acceptance of new handshakes and any process-required drainage. The probes do not reuse TLS sessions and record sessionReused to avoid confusing the measurement. When the on-disk bundle contains only the new root, an earlier overlap context still accepts the old server. A fresh context rejects it. Removing trust from a file does not establish removal from every in-memory instance. Inventory consumers, reload mechanisms and relevant sessions. Do not generalize these Python-library details to every runtime; use them to formulate tests the real component must pass. Record which process loaded each trust generation so a later investigation can explain differing outcomes.
Run the rotation laboratory
Prerequisites are Python 3.13 or later and OpenSSL 3.x, with permission to open local sockets. Save the code as run.py and execute python3 run.py --output new-result.json. Use --openssl /path/openssl if several executables are installed. The script refuses to overwrite output, creates a private temporary directory and generates its own roots, keys and certificates. The hostname payments.training.test is supplied to TLS verification, but the TCP connection uses loopback only. Read the 16 observations and connect each to its tested hypothesis. Compare acceptance and denial with old, new and overlapping trust; incomplete and complete chains; changed files with old and new contexts; a persistent session and new handshake; a mismatched-key candidate and preserved active context. Fingerprints and ports differ between runs because material is disposable. Compare relationships and predicates rather than expecting identical identifiers. Finally, confirm temporaryKeysRemoved and serversStopped. Final trials use Python 3.13.1 and OpenSSL 3.6.5 as recorded in the reports. This exercise uses no Kubernetes cluster, Ingress controller, CNI, public DNS, production trust store or revocation service. The official CKS page still states runtime 1.35 and a 120-minute practical exam; this local exercise does not reproduce that exam. Repeat with a new output path and inspect any failed predicate before treating the trial as accepted.
Decide close readiness and retain authorized recovery
In the final case, twenty minutes remain before close. Updated clients connect to the new server, but older consumers fail new handshakes. Sessions already open continue working. The decision must not use that continuity as proof of universal acceptance. If consumer distribution and validation cannot fit the window, apply authorized rollback and reconfirm served material, loaded trust and relevant paths. Retain recovery artifacts as verifiable sets. A binary transferred into an isolated zone needs its matching signature and verification material without mixing versions sharing a filename. A valid signature also does not approve a version retired for known exposure. The team needs an authorized target or an explicit exception with documented impact and responsibility. The TLS exercise performs no Kubernetes-binary verification; scenarios distinguish that control from cryptographic certificate validation. Summary: validate intended traffic, effective configuration, trust distribution and new connections. Preserve the relationship between technical recovery and current authorization. Connect this lesson with NetworkPolicy, identities, Secret management, supply chain and operational change. CKS deepening continues with response in the remaining areas, full practical mocks and independent specialist review. Before handing over, ask the next operator to identify a result that proves readiness and a result that only proves continuity of an existing session.
#!/usr/bin/env python3
"""Original TLS trust-rollover lab. Python3.13+ and OpenSSL3.x, loopback only.
Creates disposable CAs/keys; no system trust, Kubernetes controller, CNI,
production endpoint, revocation service or full practical mock is exercised.
"""
import argparse, datetime, hashlib, json, platform, shutil, socket, ssl
import subprocess, tempfile, threading
from pathlib import Path
p=argparse.ArgumentParser(description=__doc__)
p.add_argument('--openssl',default='openssl')
p.add_argument('--output',required=True)
a=p.parse_args;out=Path(a.output).resolve
if out.exists:raise SystemExit('Choose a new output path; existing reports are not overwritten.')
HOST='payments.training.test'observations=[];servers=[];persistent=None
def cmd(root,*args):
r=subprocess.run([a.openssl,*args],cwd=root,text=True,capture_output=True,timeout=20)
if r.returncode:raise RuntimeError('OpenSSL '+args[0]+' failed: '+r.stderr[-700:])
return r.stdout.strip
def observe(name,passed,**data):
observations.append(dict(name=name,passed=bool(passed),observed=data))
if not passed:raise AssertionError(name)
def client(bundle):
c=ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
c.minimum_version=ssl.TLSVersion.TLSv1_2
c.verify_flags|=ssl.VERIFY_X509_STRICT
c.load_verify_locations(cafile=str(bundle))
return c
def server_context(chain,key):
c=ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
c.minimum_version=ssl.TLSVersion.TLSv1_2
c.load_cert_chain(str(chain),str(key))
return c
class Server:
def __init__(self,context):
self.context=context;self.stop=threading.Event;self.workers=[]
self.sock=socket.socket;self.sock.bind(('127.0.0.1',0));self.sock.listen(20)
self.sock.settimeout(.1);self.port=self.sock.getsockname[1]
self.thread=threading.Thread(target=self.accept,daemon=True);self.thread.start;servers.append(self)
def accept(self):
while not self.stop.is_set:
try:raw,_=self.sock.accept
except socket.timeout:continue
except OSError:break
t=threading.Thread(target=self.serve,args=(raw,self.context),daemon=True);self.workers.append(t);t.start
def serve(self,raw,context):
raw.settimeout(8)
try:
with context.wrap_socket(raw,server_side=True)as conn:
while True:
data=conn.recv(64)
if not data:break
conn.sendall(data)
except (ssl.SSLError,OSError):pass
finally:raw.close
def close(self):
self.stop.set;self.sock.close;self.thread.join(2)
for t in self.workers:t.join(9)
return not self.thread.is_alive and not any(t.is_alivefor t in self.workers)
def connect(c,server):
raw=socket.create_connection(('127.0.0.1',server.port),timeout=3)
try:return c.wrap_socket(raw,server_hostname=HOST)
except Exception:raw.close;raise
def probe(c,server):
try:
with connect(c,server)as conn:
conn.sendall(b'check');data=conn.recv(64)
return dict(accepted=data==b'check',leafSha256=hashlib.sha256(conn.getpeercert(binary_form=True)).hexdigest,protocol=conn.version,sessionReused=conn.session_reused)
except ssl.SSLCertVerificationError as e:
return dict(accepted=False,verifyCode=e.verify_code,verifyMessage=e.verify_message)
def check(name,c,server,expected,fingerprint=None):
r=probe(c,server);observe(name,r['accepted']==expected and (fingerprint is None or r.get('leafSha256')==fingerprint),**r);return r
report=dict(startedAt=datetime.datetime.now(datetime.timezone.utc).isoformat,scriptSha256=hashlib.sha256(Path(__file__).read_bytes).hexdigest,versions=dict(python=platform.python_version,pythonTLS=ssl.OPENSSL_VERSION,openssl=cmd(None,'version')),observations=observations,fullPracticalMock=False,independentVerification=False,scope='Actual local TLS handshakes, chain delivery, CA overlap, loaded contexts, live connection and candidate-context rejection. No Kubernetes ingress controller, CNI, public DNS, mTLS, revocation or production trust store.')
temp=tempfile.TemporaryDirectory(prefix='dr-cks-rollover-');root=Path(temp.name)
try:
root.chmod(0o700)
def key(name):
cmd(root,'genpkey','-algorithm','RSA','-pkeyopt','rsa_keygen_bits:2048','-out',name+'.key');(root/(name+'.key')).chmod(0o600)
def ca(name):
key(name);cmd(root,'req','-new','-x509','-key',name+'.key','-out',name+'.pem','-days','2','-subj','/CN='+name,'-addext','basicConstraints=critical,CA:TRUE','-addext','keyUsage=critical,keyCertSign,cRLSign','-addext','subjectKeyIdentifier=hash')
def issue(name,issuer,is_ca=False):
key(name);cmd(root,'req','-new','-key',name+'.key','-out',name+'.csr','-subj','/CN='+name)
ext='basicConstraints=critical,CA:TRUE,pathlen:0\nkeyUsage=critical,keyCertSign,cRLSign\n'if is_ca else 'basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature,keyEncipherment\nextendedKeyUsage=serverAuth\nsubjectAltName=DNS:'+HOST+'\n'
(root/(name+'.ext')).write_text(ext+'subjectKeyIdentifier=hash\nauthorityKeyIdentifier=keyid,issuer\n')
cmd(root,'x509','-req','-in',name+'.csr','-CA',issuer+'.pem','-CAkey',issuer+'.key','-CAcreateserial','-out',name+'.pem','-days','1','-extfile',name+'.ext')
ca('old-root');ca('new-root');issue('old-leaf','old-root');issue('new-intermediate','new-root',True);issue('new-leaf','new-intermediate')
old=root/'old-leaf.pem'new=root/'new-leaf.pem'chain=root/'new-chain.pem'chain.write_bytes(new.read_bytes+(root/'new-intermediate.pem').read_bytes)
bundle=root/'trust.pem'bundle.write_bytes((root/'old-root.pem').read_bytes)
old_client=client(bundle);new_client=client(root/'new-root.pem')
active_cert=root/'active-chain.pem'active_key=root/'active.key'shutil.copy2(old,active_cert);shutil.copy2(root/'old-leaf.key',active_key)
serving=Server(server_context(active_cert,active_key));old_server=Server(server_context(old,root/'old-leaf.key'));new_server=Server(server_context(chain,root/'new-leaf.key'));missing=Server(server_context(new,root/'new-leaf.key'))
old_fp=check('initial-old-trust-accepts-old-server',old_client,serving,True)['leafSha256']
check('new-trust-denies-old-server',new_client,old_server,False)
check('old-trust-denies-new-server',old_client,new_server,False)
bundle.write_bytes((root/'old-root.pem').read_bytes+(root/'new-root.pem').read_bytes);dual=client(bundle)
check('overlap-trust-accepts-old-server',dual,old_server,True,old_fp)
new_fp=check('overlap-trust-accepts-new-server',dual,new_server,True)['leafSha256']
check('missing-intermediate-denied',new_client,missing,False)
check('full-chain-accepted',new_client,new_server,True,new_fp)
check('changed-trust-file-does-not-refresh-old-context',old_client,new_server,False)
check('new-context-loads-updated-trust-file',client(bundle),new_server,True,new_fp)
persistent=connect(old_client,serving);persistent.sendall(b'before');assert persistent.recv(64)==b'before'
shutil.copy2(chain,active_cert);shutil.copy2(root/'new-leaf.key',active_key)
check('changed-server-files-do-not-refresh-context',old_client,serving,True,old_fp)
serving.context=server_context(active_cert,active_key)
check('replacement-context-serves-new-leaf',dual,serving,True,new_fp)
persistent.sendall(b'after');reply=persistent.recv(64);fp=hashlib.sha256(persistent.getpeercert(binary_form=True)).hexdigest
observe('established-session-retains-old-handshake',reply==b'after'and fp==old_fp,echoContinues=reply==b'after',leafSha256=fp,oldLeafSha256=old_fp,newLeafSha256=new_fp)
bundle.write_bytes((root/'new-root.pem').read_bytes);retired=client(bundle)
check('retired-root-denied-by-fresh-context',retired,old_server,False)
check('loaded-overlap-context-still-trusts-old-root',dual,old_server,True,old_fp)
check('fresh-new-only-context-accepts-new-server',retired,serving,True,new_fp)
rejected=False
try:server_context(chain,root/'old-leaf.key')
except ssl.SSLError:rejected=True
r=probe(retired,serving)
observe('mismatched-candidate-rejected-active-service-preserved',rejected and r['accepted']and r['leafSha256']==new_fp,candidateRejected=rejected,activeService=r)
report['passed']=all(x['passed']for x in observations)
finally:
if persistent:persistent.close
closed=[s.closefor s in servers];temp.cleanup
report['cleanup']=dict(temporaryKeysRemoved=not root.exists,serversStopped=all(closed),realCredentialsUsed=False,systemTrustModified=False)
report['finishedAt']=datetime.datetime.now(datetime.timezone.utc).isoformat;out.write_text(json.dumps(report,indent=2)+'\n')
print(json.dumps(dict(passed=report.get('passed',False),observations=len(observations),output=str(out))))
Old sessions continue after rotation, but clients lacking the new CA fail when opening a connection.
Common pitfalls
Confusing files with loaded contexts, old sessions with new handshakes and valid signatures with approved versions.
Related topics: NetworkPolicy and exposure · TLS chains and trust · Change management and rollback
A recoverable change requires evidence of effective state and dependencies that still make rollback usable.
Reference: CKS certification and domains · Kubernetes v1.35; current six-domain CKS outline