LEARN. PRACTICE. GROW.Define what must regain trust
Supply-chain recovery begins by identifying where trust is no longer established. The affected boundary may be a signing key, runner, external action, base image or promotion-control configuration. Each cause needs different evidence. If the runner was compromised, signing its output again does not establish safe bytes. If a key was exposed, rebuilding in a clean environment does not automatically withdraw trust from old verifiers. In a fictional banking case, APS receives a reconciliation candidate thirty minutes before closing. The pipeline is green, but malicious execution occurred on a persistent runner. The technical lead brings development, security and operations together to bound the affected window, identify produced digests and consumers, and review accessible credentials. The aim is to decide which artifacts need quarantine or rebuilding and which business flows need fallback. This case does not describe internal BNP Paribas procedures. Prepare a record containing digest, build origin, signature, provenance, applied policy and decision per consumer. Separate observed facts from hypotheses and pending actions. A tag such as candidate helps locate a release but does not permanently identify its bytes. Retain its digest resolution at decision time. Agree business checkpoints: deadline, data already processed, consequences of deferral and authority to accept a bounded exception. Urgency changes priorities but does not turn missing evidence into approval. Record who must supply each missing fact so the recovery meeting produces actionable decisions rather than an unqualified green status.
Rebuild from trusted inputs
An OCI manifest references configuration and an ordered set of layers. The filesystem visible to a container results from applying those layers; it is not an inventory of every byte still distributed in blobs. A whiteout can hide a file from an earlier layer without erasing it there. Copying a credential and deleting it in a later instruction therefore does not protect secrets. During recovery, rotate the exposed credential, assess where the image circulated and rebuild without embedding that material. A vendor’s base-image update also does not modify the digest of an application already built. Produce a new artifact incorporating the changes, assess its output and confirm consumer adoption. Record input and output digests. If image reduction removed trust certificates or required libraries, a process starting does not demonstrate a working business flow. Test dependencies actually used while preserving TLS verification and other requirements. Restoring the build environment needs its own decision. On a persistent self-hosted runner, fixing the workflow does not establish that processes, files or credentials are no longer compromised. Single-job JIT registration does not automatically clean reused hardware either. Use an environment whose trust has been restored and review accessible credentials. When pinning an action to a full SHA, confirm the commit in the original repository and review code and permissions. Reference immutability helps repeat a choice; it does not establish that the original choice was safe.
Distinguish valid signature from acceptance
A signature binds a payload to a key. In the exercise, Cosign verifies the image signature and its digest binding. When candidate moves from the prior image to an unsigned recovery image, tag verification stops passing while the prior digest remains verifiable. The signature did not transfer to the new bytes. This explains why a promotion decision must retain resolved identity instead of relying on a movable name. The exercise creates two disposable keys. After recovery is signed with the old key, verification using that key passes and verification using only the new key fails. Adding a new signature allows the policy using the new key to pass. The old signature has not stopped being cryptographically valid. The exercise implements no revocation service; it observes explicit trust selection by the verifier. In production, withdrawing trust requires a consumer inventory and confirmation of the policy actually used. Changing the pipeline key does not by itself update environments verifying images. Deleting one private copy also does not remove copies already exposed. Define acceptance and rejection criteria, test both directions and record where adoption remains pending. A release annotation adds an expectation about the signed payload: the correct key may pass while the required release value fails. None of these checks independently establishes absence of vulnerabilities or legitimacy of a compromised build. Preserve those distinctions in the handover so another operator can reproduce the decision without guessing which property was actually checked.
Verify provenance before checking expectations
An image signature and a provenance attestation are not equivalent. The lab confirms that a signed image can still lack the statement required by verify-attestation. Policy must treat absence as an unmet condition. Obtain evidence, verify its signature using approved trust, and only then interpret the authenticated payload. JSON obtained without that binding must not be presented as a validated statement from the signer. Cosign 3.1.3 used in this exercise emitted an in-toto v0.1 Statement with SLSA provenance v1 predicateType. The DSSE envelope declares application/vnd.in-toto+json. These layers have their own identifiers. The lab explicitly recognizes the observed Statement and does not rewrite its _type to suggest another version. The consulted SLSA reference is specification v1.2; the provenance type URI still ends in v1. Record actual identifiers instead of inferring numerical equivalence. Local policy compares subject.digest, predicateType, buildType, builder.id and the exact approved externalParameters set. A signed statement naming an unauthorized builder fails; another adding debug=true also fails; the approved combination passes. The same statement is rejected for a different digest. This exercise demonstrates decisions about synthetic statements. It neither executes an isolated CI platform nor proves that the declared builder produced the bytes, and it establishes no SLSA level. In a real application, trust in the platform and provenance issuance must support interpretation of these fields. Keep that dependency visible when assessing an incident involving the build platform itself.
Review what analysis reports omitted
A report without findings can represent no problems in analyzed scope or a lack of relevant analysis. During recovery, confirm the assessed image, configuration, inputs and filters. In the consulted Grype documentation, only-fixed excludes findings with states such as not-fixed, wont-fix and unknown. The fail-on threshold is evaluated after filtering. Success with that filter therefore does not establish absence of vulnerabilities without fixes. Also inspect ignoredMatches and appliedIgnoreRules to understand what did not participate in the decision. A VEX statement must match the assessed product and version. Under_investigation does not mean not_affected. Do not use uncertainty as evidence of no impact or automatically transfer a justification concerning the prior image to a tag now resolving to another digest. Document the decision basis, owner and reassessment point. These scenarios are documentation-based; this new lab runs neither Grype nor a vulnerability scan. For KubeLinter, confirm effective configuration when a job changes directories. Explicit inclusion does not override a matching checks.exclude. Review ignorePaths and annotations, particularly in the recovery Job and rollback manifest, which may have fallen outside primary analysis. An old ignore-all annotation does not remain justified simply because it appeared in a previously approved commit. Exceptions need current scope and rationale. Prepare an input expected to fail to observe whether the control remains active. A positive result for the main Deployment does not replace assessment of the other workloads required by fallback. Record these coverage checks alongside the findings rather than hiding them in an undocumented pipeline assumption.
Run the local OCI laboratory
The exercise uses Docker with a local Unix-socket context and Cosign 3.1.3. It uses previously obtained images python@sha256:2d9aefe2fef018a7eb2c13064c89c71929800fd2e5dccdbf52ea5da5bb8d929a and registry@sha256:ddf754342cfc8acc51a56d5d0ab6af06826461864460636d8bd5c546dab2a7b8, corresponding to Distribution 3.1.2. The script does not download them: obtain trusted references before execution if missing. Supply the binary path with --cosign and a report path with --output. Displayed code matches the two final runs and is identified by SHA-256 in their reports. The script creates a temporary registry published only on 127.0.0.1 without real credentials. It creates two synthetic images with different markers and confirms marker reads in containers without networking. Each image is exported and its configuration, layers and manifest uploaded through the local HTTP API. The script verifies blob sizes and digests and returned manifest identity. This avoids confusing host loopback with Docker Desktop daemon loopback. Image assembly using commit is a teaching fixture, not a secure or reproducible build pipeline. It then exercises signing, tag movement, trust selection and provenance. Each final run recorded 29 passing observations. Reports contain versions, digests and verified synthetic statements, not private keys. Cleanup removes owned containers and images, the temporary directory and private material. Cached base images remain. The script uses an empty temporary Docker configuration and does not use the user’s registry credentials. Read the exercise boundaries before reusing any command in another environment. A passing report describes these particular inputs and controls; it does not authorize promotion of unrelated production artifacts.
Interpret negative results and boundaries
Before execution, predict four outcomes: an unsigned image should be rejected; an old-key signature should fail when only the new key is accepted; an attestation naming an unapproved builder should fail policy; and a valid statement for another digest should be rejected. Then compare prediction with observation. Expected failure is not a defect in the exercise: it demonstrates a boundary recovery must preserve. Retain necessary successes too, establishing that the permitted path remains usable. Final reports show that the tag moved without transferring its signature, the old key continued producing a cryptographically valid signature, and stricter policy rejected signed statements. The program decodes only output from successful attestation verification, checks DSSE payloadType and applies expectations to the observed Statement. Local acceptance of at least one approved statement does not establish that every available statement describes a legitimate build; deliberately define intended policy when multiple attestations exist. HTTP and --insecure-ignore-tlog are bounded options for this loopback exercise. It supplies no public-transparency proof, OIDC, KMS, admission control, vulnerability scan or actual CI-platform recovery. A production registry needs its own transport, access, retention and availability controls. Initial development runs encountered marker permissions, Docker Desktop loopback and Statement-format issues; these were corrected before two final runs using identical code. Retaining these boundaries makes results useful without turning them into an unsupported coverage claim. Repeat the exercise in a controlled local environment and compare observed behavior rather than assuming identical runtimes from matching filenames.
Hand over a verifiable recovery decision
For the banking-case handover, deliver a small matrix: candidate artifact, assessed digest, accepted origin, required trust, available provenance, security assessment, tested consumers and decision. For each environment, identify the verifier and policy that actually participated in testing. Record positive and negative outcomes, ownership of remaining adoption and exception expiry. If evidence is unavailable before the deadline, the decision may be to defer the release and use an assessed fallback, communicating impact and preserving data for reconciliation. Do not declare recovery complete merely because the pipeline signs with the new key. Confirm consumers accept the approved artifact and reject what policy no longer permits. Check signatures and attestations remain accessible after copying the image to the destination registry. A successful source query does not observe destination access. Reassess rollback: a historically signed digest may remain identifiable while no longer satisfying current risk assessment. As a final exercise, complete this matrix for a candidate created on the compromised runner, another rebuilt in an accepted environment, and a rollback retaining an old analysis exception. Explain which decisions you can make and what evidence you would request for the rest. Relate the result to incident management, change, least privilege and observability. The lesson’s conclusion is operational: recovering trust involves origin, bytes, verified claims, policy and adoption. The 29 local checks support parts of that reasoning; full practical mocks and independent course review remain pending. Keep that distinction visible when communicating readiness to management and to the next support shift.
"""Original local OCI-signature and recovery-policy lab; no production registry or keys.
Requires cached Docker images. Uses a loopback-only disposable registry and
synthetic image variants created by docker commit, not a production build chain.
Public transparency and OIDC are deliberately absent; no SLSA level is claimed.
"""
import argparse,base64,datetime,hashlib,json,os,secrets,shutil,subprocess,tempfile,time,urllib.request,urllib.parse,uuid,tarfile
from pathlib import Path
p=argparse.ArgumentParser(description=__doc__);p.add_argument('--cosign',required=True);p.add_argument('--output',required=True);a=p.parse_args;out=Path(a.output).resolve;assert not out.exists
BASE='python@sha256:2d9aefe2fef018a7eb2c13064c89c71929800fd2e5dccdbf52ea5da5bb8d929a'
REGISTRY='registry@sha256:ddf754342cfc8acc51a56d5d0ab6af06826461864460636d8bd5c546dab2a7b8'
os.umask(0o077);tmp=Path(tempfile.mkdtemp(prefix='dr-cks-oci-'));prefix='dr-cks-oci-'+uuid.uuid4.hex[:8];containers=[];images=[];tags=[];observations=[];report={};env=dict(os.environ)
def run(cmd,ok=True,timeout=90):
r=subprocess.run(cmd,cwd=tmp,env=env,text=True,capture_output=True,timeout=timeout)
if ok and r.returncode:raise RuntimeError('Command failed '+str(cmd[:3])+': '+r.stderr[-1200:])
return r
def docker(*args,ok=True):return run(['docker',*map(str,args)],ok=ok)
def check(name,value,expected):
assert value==expected,(name,value,expected);observations.append(dict(name=name,observed=value,passed=True));print(name,flush=True)
def cos(*args,ok=True):return run([a.cosign,*map(str,args)],ok=ok)
def request(path,method='GET',data=None,headers=None):
req=urllib.request.Request('http://'+address+path,method=method,data=data,headers=headers or {})
with urllib.request.urlopen(req,timeout=10) as res:return res.read,dict(res.headers)
def manifest(tag):
data,headers=request('/v2/funds/reconciler/manifests/'+tag,headers={'Accept':'application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json'})
digest='sha256:'+hashlib.sha256(data).hexdigest;assert headers.get('Docker-Content-Digest')==digest
return digest,data,headers['Content-Type']
def push_archive(local,tag):
archive=tmp/'image.tar'docker('image','save','--platform','linux/arm64','--output',archive,local)
def upload(data):
digest='sha256:'+hashlib.sha256(data).hexdigest
_,headers=request('/v2/funds/reconciler/blobs/uploads/','POST',b'')
location=urllib.parse.urlsplit(headers['Location']);assert location.netloc==address
path=location.path+'?'+location.query+'&digest='+digest
request(path,'PUT',data,{'Content-Type':'application/octet-stream'})
return digest
with tarfile.open(archive,'r') as tar:
entry=json.load(tar.extractfile('manifest.json'))[0];config=tar.extractfile(entry['Config']).read;cd=upload(config);layers=[]
for path in entry['Layers']:
data=tar.extractfile(path).read;digest=upload(data)
layers.append(dict(mediaType='application/vnd.oci.image.layer.v1.tar'+('+gzip' if data[:2]==bytes([31,139]) else ''),digest=digest,size=len(data)))
value=dict(schemaVersion=2,mediaType='application/vnd.oci.image.manifest.v1+json',config=dict(mediaType='application/vnd.oci.image.config.v1+json',digest=cd,size=len(config)),layers=layers)
body=json.dumps(value,separators=(',',':')).encode;request('/v2/funds/reconciler/manifests/'+tag,'PUT',body,{'Content-Type':value['mediaType']})
archive.unlink
def sign(ref,key,label):return cos('sign','--yes','--key',key+'.key','--signing-config','local-signing.json','--allow-http-registry','-a','release='+label,ref)
def verify(ref,key,label=None):
args=['verify','--key',key+'.pub','--allow-http-registry','--insecure-ignore-tlog']
if label:args+=['-a','release='+label]
return cos(*args,ref,ok=False)
def attest(ref,predicate):
(tmp/'predicate.json').write_text(json.dumps(predicate))
return cos('attest','--yes','--key','recovery.key','--signing-config','local-signing.json','--allow-http-registry','--type','slsaprovenance1','--predicate','predicate.json',ref)
def verify_attestation(ref):return cos('verify-attestation','--key','recovery.pub','--allow-http-registry','--insecure-ignore-tlog','--type','slsaprovenance1',ref,ok=False)
def decode(result):
assert result.returncode==0
# Cosign emits one DSSE envelope per line for registry attestations.
objects=[]
try:
value=json.loads(result.stdout);objects=value if isinstance(value,list) else [value]
except json.JSONDecodeError:objects=[json.loads(line) for line in result.stdout.splitlines if line.strip]
assert all(x.get('payloadType')=='application/vnd.in-toto+json' for x in objects)
return [json.loads(base64.b64decode(x['payload'])) for x in objects]
builder='https://ci.example.invalid/funds/recovery-v1'repository='https://git.example.invalid/funds/reconciler'commit='0123456789abcdef0123456789abcdef01234567'build_type='https://dr.pt/training/synthetic-image-assembly/v1'
expected_params={'repository':repository,'commit':commit}
def policy(statement,digest):
predicate=statement.get('predicate',{});definition=predicate.get('buildDefinition',{});details=predicate.get('runDetails',{})
return (statement.get('_type')=='https://in-toto.io/Statement/v0.1' and statement.get('predicateType')=='https://slsa.dev/provenance/v1'
and any(x.get('digest',{}).get('sha256')==digest.removeprefix('sha256:') for x in statement.get('subject',[]))
and definition.get('buildType')==build_type and definition.get('externalParameters')==expected_params
and details.get('builder',{}).get('id')==builder)
try:
# Avoid reading any real registry credentials; retain only the local daemon endpoint.
endpoint=docker('context','inspect','--format','{{(index.Endpoints "docker").Host}}').stdout.strip;assert endpoint.startswith('unix://')
(tmp/'docker-config').mkdir;(tmp/'docker-config/config.json').write_text('{}');env.update(DOCKER_HOST=endpoint,DOCKER_CONFIG=str(tmp/'docker-config'),COSIGN_PASSWORD=secrets.token_urlsafe(24),COSIGN_YES='true');env.pop('DOCKER_CONTEXT',None)
docker('image','inspect',BASE);docker('image','inspect',REGISTRY)
report['cosign']=json.loads(cos('version','--json').stdout);report['dockerVersion']=docker('version','--format','{{.Server.Version}}').stdout.strip
reg=prefix+'-registry'containers.append(reg)
docker('run','-d','--name',reg,'--pull=never','--user','1000:1000','--cap-drop=ALL','--security-opt','no-new-privileges','--read-only','--memory','256m','--cpus','1','--tmpfs','/var/lib/registry:rw,noexec,nosuid,nodev,size=256m,uid=1000,gid=1000','--tmpfs','/tmp:rw,noexec,nosuid,nodev,size=16m,uid=1000,gid=1000','-p','127.0.0.1::5000','-e','OTEL_TRACES_EXPORTER=none',REGISTRY)
inspected=json.loads(docker('inspect',reg).stdout)[0];binding=inspected['NetworkSettings']['Ports']['5000/tcp'][0];assert binding['HostIp']=='127.0.0.1'address='127.0.0.1:'+binding['HostPort'];repo=address+'/funds/reconciler'
report['registryVersion']=docker('exec',reg,'registry','--version').stdout.strip;report['registryImage']=REGISTRY;report['baseImage']=BASE
for attempt in range(30):
try:request('/v2/');break
except OSError:time.sleep(.2)
else:raise AssertionError('Registry did not start')
check('registry-loopback-only',binding['HostIp'],'127.0.0.1')
refs={};digests={};manifests={}
for variant in ['prior','recovery']:
marker=tmp/(variant+'-marker');marker.write_text('synthetic-'+variant+'\n');marker.chmod(0o444);name=prefix+'-'+variant;containers.append(name)
docker('create','--name',name,'--pull=never','--network','none',BASE,'python','-c','print("unused")');docker('cp',marker,name+':/training-marker')
local=prefix+':'+variant;images.append(local);docker('commit','--change','LABEL dr.training='+prefix,name,local)
result=docker('run','--rm','--pull=never','--network','none','--user','1000:1000','--cap-drop=ALL','--security-opt','no-new-privileges','--read-only',local,'python','-c','from pathlib import Path;print(Path("/training-marker").read_text.strip)')
check(variant+'-image-runs-marker',result.stdout.strip,'synthetic-'+variant)
push_archive(local,variant)
digest,body,media=manifest(variant);descriptors=json.loads(body);verified_blobs=True
for blob in [descriptors['config'],*descriptors['layers']]:
content,_=request('/v2/funds/reconciler/blobs/'+blob['digest']);verified_blobs=verified_blobs and len(content)==blob['size'] and 'sha256:'+hashlib.sha256(content).hexdigest==blob['digest']
check(variant+'-registry-content-digests-match',verified_blobs,True)
digests[variant]=digest;manifests[variant]=(body,media);refs[variant]=repo+'@'+digest
check('image-digests-differ',digests['prior']!=digests['recovery'],True)
for key in ['prior','recovery']:cos('generate-key-pair','--output-key-prefix',key)
cos('signing-config','create','--no-default-rekor','--no-default-fulcio','--no-default-oidc','--no-default-tsa','--out','local-signing.json')
check('unsigned-prior-image-rejected',verify(refs['prior'],'prior').returncode!=0,True)
sign(refs['prior'],'prior','prior');check('prior-image-signature-accepted',verify(refs['prior'],'prior').returncode,0)
check('different-key-rejects-prior-image',verify(refs['prior'],'recovery').returncode!=0,True)
for variant in ['prior','recovery']:
body,media=manifests[variant];request('/v2/funds/reconciler/manifests/candidate','PUT',body,{'Content-Type':media})
check('candidate-resolves-'+variant,manifest('candidate')[0],digests[variant])
result=verify(repo+':candidate','prior');check('candidate-'+variant+'-verification',result.returncode==0,variant=='prior')
check('pinned-prior-remains-verifiable',verify(refs['prior'],'prior').returncode,0)
sign(refs['recovery'],'prior','recovery');check('old-key-can-still-produce-valid-signature',verify(refs['recovery'],'prior').returncode,0)
check('new-trust-rejects-old-only-signature',verify(refs['recovery'],'recovery').returncode!=0,True)
sign(refs['recovery'],'recovery','recovery');check('new-key-accepts-recovery-image',verify(refs['recovery'],'recovery','recovery').returncode,0)
check('wrong-release-annotation-rejected',verify(refs['recovery'],'recovery','prior').returncode!=0,True)
check('old-key-signature-still-cryptographically-valid',verify(refs['recovery'],'prior').returncode,0)
check('old-image-not-accepted-by-new-trust',verify(refs['prior'],'recovery').returncode!=0,True)
check('image-signature-does-not-create-provenance',verify_attestation(refs['recovery']).returncode!=0,True)
predicate={'buildDefinition':{'buildType':build_type,'externalParameters':expected_params,'internalParameters':{},'resolvedDependencies':[]},'runDetails':{'builder':{'id':'https://ci.example.invalid/unapproved/runner'},'metadata':{'invocationId':'synthetic-lab'}}}
attest(refs['recovery'],predicate);statements=decode(verify_attestation(refs['recovery']));check('observed-statement-version',sorted({x.get('_type') for x in statements}),['https://in-toto.io/Statement/v0.1']);check('unapproved-builder-attestation-signature-valid',len(statements)>0,True)
check('unapproved-builder-policy-rejected',any(policy(x,digests['recovery']) for x in statements),False)
predicate['runDetails']['builder']['id']=builder;predicate['buildDefinition']['externalParameters']={**expected_params,'debug':True};attest(refs['recovery'],predicate)
statements=decode(verify_attestation(refs['recovery']));check('unknown-build-parameter-policy-rejected',any(policy(x,digests['recovery']) for x in statements),False)
predicate['buildDefinition']['externalParameters']=expected_params;attest(refs['recovery'],predicate)
statements=decode(verify_attestation(refs['recovery']));report['verifiedSyntheticStatements']=statements;accepted=[x for x in statements if policy(x,digests['recovery'])];check('approved-expectations-policy-accepted',len(accepted)>0,True)
check('provenance-subject-other-digest-rejected',any(policy(x,digests['prior']) for x in statements),False)
check('candidate-still-bound-to-recovery',manifest('candidate')[0],digests['recovery'])
check('recovery-signature-still-valid-after-attestations',verify(refs['recovery'],'recovery','recovery').returncode,0)
assert len(observations)==29,len(observations)
report.update(passed=True,digests=digests,verifiedStatementType=accepted[0]['_type'],verifiedPredicateType=accepted[0]['predicateType'],verifiedBuilder=accepted[0]['predicate']['runDetails']['builder']['id'],policyScope='Original local allow-list checks on already cryptographically verified statements; not a SLSA level assessment or cluster admission controller.')
finally:
for name in reversed(containers):docker('rm','-f',name,ok=False)
for name in tags+images:docker('image','rm',name,ok=False)
left=docker('ps','-a','--filter','name='+prefix,'--format','{{.Names}}').stdout.strip;left_images=docker('image','ls','--filter','reference='+prefix+':*','--format','{{.Repository}}:{{.Tag}}').stdout.strip
report['cleanup']=dict(ownedContainersRemoved=not left,ownedImageTagsRemoved=not left_images and all(docker('image','inspect',name,ok=False).returncode!=0 for name in tags+images),temporaryMaterialRemoved=False,realCredentialsUsed=False,publicRegistryWrites=False)
shutil.rmtree(tmp);report['cleanup']['temporaryMaterialRemoved']=not tmp.exists
report.update(executedAt=datetime.datetime.now(datetime.timezone.utc).isoformat,scriptSha256=hashlib.sha256(Path(__file__).read_bytes).hexdigest,observations=observations,scope='Actual local OCI image signatures,tag movement,new-key trust selection,annotation checks and signed provenance expectation checks. HTTP only on loopback with synthetic keys; transparency verification explicitly absent. No OIDC,public transparency,KMS,live admission,vulnerability scan,isolated CI builder,SLSA level certification or production recovery executed.')
out.write_text(json.dumps(report,indent=2)+'\n')
assert report['cleanup']['ownedContainersRemoved'] and report['cleanup']['ownedImageTagsRemoved'];print('PASS',len(observations))
The old key still produces a valid signature, but new trust and provenance policy reject the artifact.
Common pitfalls
Resigning suspect bytes, trusting a movable tag, overlooking old consumers and confusing analysis filters with fixes.
Related topics: Signing and provenance · Incident response · CI/CD and least privilege · Static analysis and exceptions
Recovery needs accepted origin, digest-bound evidence and demonstrated consumer adoption.
Reference: CKS domains and exam details · Kubernetes v1.35; current six-domain CKS outline