LEARN. PRACTICE. GROW.Topology and execution scope
The script creates one container without an external network, containing four namespaces: blue, red and their two corresponding endpoints. Zebra and mgmtd start with -w, FRRouting’s namespace-backed VRF implementation. Blue and red each use 192.0.2.1/24; their respective peer uses 192.0.2.2/24 and has 198.51.100.10/32 on loopback. A static route in each context points to its own peer. The same numeric destination therefore refers to distinct endpoints. This exercise has no Linux l3mdev VRF devices; the initial attempt to create them failed in the available environment. The executed implementation is FRRouting 10.4.5, not IOS XE.
Case: the monitor inspects the wrong table
In phase one, both contexts reach their respective endpoint while the global table remains without a route. A global probe fails while a probe inside blue or red passes. This is not contradictory: each command selects a different context. In a fictional APS scenario, a monitor created without VRF context raises an incident while the client flow works. Correct monitor representativeness before concluding a general failure. The lab uses ip netns exec to run ping in the selected namespace and ip -n to inspect its table. It does not substitute one global ping for both tests.
Withdraw a route while retaining transit
The script removes only the 198.51.100.10/32 route from blue. Lookup in that context fails and the destination stops responding. Peer 192.0.2.2 remains reachable, preserving evidence of transit-segment connectivity. Red retains its own route and response to the same numeric address. Restoring the blue route recovers the probe. The sequence shows why route lookup, peer probing and destination probing should all be retained: the three results answer different questions. Neither restarting red nor importing its route was needed to recover blue.
Withdraw the endpoint while retaining the route
The second failure removes 198.51.100.10/32 from the red peer’s loopback. Red’s static route remains installed and lookup still returns the next hop. The destination nevertheless stops responding; the transit peer remains reachable and blue retains connectivity. Restoring the address recovers red. A valid route does not confirm remote-address presence, service state or application authorization. The exercise observes only IPv4 ICMP. For an actual log collector, transport, authentication and message receipt would still need checking with the source used by agents.
Reproduce, observe and close
Save the complete script below as run.py. With Docker running and the specified image already available, run python3 run.py /tmp/vrf-evidence.json. The script records versions, hash, five phases and 24 checks, then removes the container. Two complete runs passed. NET_ADMIN, NET_RAW and SYS_ADMIN capabilities are used only inside the temporary container to configure namespaces; there are no published ports, volumes or host networking. All contexts share one machine, so physical redundancy is not measured. BGP, MPLS, RT import, route leaking, IPv6, TLS and throughput are also not exercised.
# Save as run.py; execute: python3 run.py /tmp/vrf-evidence.json
"""Original FRR namespace-backed VRF lab; only one disposable container is changed."""
import datetime,hashlib,json,pathlib,subprocess,sys,time,uuid
IMAGE='quay.io/frrouting/frr@sha256:b0faf7c8f3d8b09aea1e38f77603c745a66dbf5e26ef9718527c2fbb53cc3aed'
name='dr-encor-vrf-'+uuid.uuid4.hex[:8];created=False;checks=[];phases=[];DEST='198.51.100.10'
def docker(args,check=True):
r=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=30)
if check and r.returncode:raise RuntimeError(str(args)+'\n'+r.stdout+r.stderr)
return r
def ex(*args,check=True):return docker(['exec',name,*args],check)
def cli(command):
r=ex('vtysh','-c',command).stdout
if '% Unknown' in r or '% Ambiguous' in r:raise RuntimeError(r)
return r
def verify(label,condition):
if not condition:raise AssertionError(label)
checks.append(label)
def result(r):return {'exit':r.returncode,'stdout':r.stdout,'stderr':r.stderr}
def lookup(vrf=None):return ex('ip',*(['-n',vrf]if vrf else []),'-j','route','get',DEST,check=False)
def ping(vrf=None,dest=DEST):return ex(*(['ip','netns','exec',vrf]if vrf else []),'ping','-c','1','-W','1',dest,check=False)
def snapshot(label):
time.sleep(0.3)
d={'name':label,'observedAt':datetime.datetime.now(datetime.timezone.utc).isoformat,'frrVRFs':cli('show vrf'),'main':json.loads(ex('ip','-j','route','show','table','main').stdout),'vrfs':{}}
for vrf in ['blue','red']:
d['vrfs'][vrf]={'routes':json.loads(ex('ip','-n',vrf,'-j','route').stdout),'frrRoutes':json.loads(cli('show ip route vrf '+vrf+' json')),'lookup':result(lookup(vrf)),'probe':result(ping(vrf)),'neighbors':json.loads(ex('ip','-n',vrf,'-j','neigh').stdout),'peerAddresses':json.loads(ex('ip','-n',vrf+'-peer','-j','address').stdout)}
d['globalLookup']=result(lookup);d['globalProbe']=result(ping);phases.append(d);return d
try:
image=json.loads(docker(['image','inspect',IMAGE]).stdout)[0]
docker(['run','-d','--name',name,'--label','dr.lab=encor-vrf','--network','none','--memory','128m','--cpus','0.5','--pids-limit','80','--cap-add','NET_ADMIN','--cap-add','NET_RAW','--cap-add','SYS_ADMIN','--entrypoint','/bin/sh',IMAGE,'-c','sleep 900']);created=True
kernel=ex('uname','-r').stdout.strip;ipversion=ex('ip','-Version').stdout.strip
for vrf in ['blue','red']:
peer=vrf+'-peer'link=vrf+'-if'other=vrf+'-end'
for ns in [vrf,peer]:ex('ip','netns','add',ns);ex('ip','-n',ns,'link','set','lo','up')
ex('ip','link','add',link,'type','veth','peer','name',other);ex('ip','link','set',link,'netns',vrf);ex('ip','link','set',other,'netns',peer)
ex('ip','-n',vrf,'address','add','192.0.2.1/24','dev',link);ex('ip','-n',vrf,'link','set',link,'up')
ex('ip','-n',peer,'address','add','192.0.2.2/24','dev',other);ex('ip','-n',peer,'link','set',other,'up');ex('ip','-n',peer,'address','add',DEST+'/32','dev','lo')
ex('touch','/etc/frr/frr.conf','/etc/frr/vtysh.conf');ex('chown','frr:frr','/etc/frr/frr.conf');ex('/usr/lib/frr/mgmtd','-w','-d','-A','127.0.0.1');ex('/usr/lib/frr/zebra','-w','-d','-A','127.0.0.1','--log','file:/tmp/zebra-vrf.log')
deadline=time.monotonic+20
while time.monotonic<deadline:
r=ex('vtysh','-c','show ip route vrf blue json',check=False)
if r.returncode==0 and '192.0.2.0/24' in r.stdout:break
time.sleep(0.5)
else:raise AssertionError('FRR namespace-backed VRF not ready')
for vrf in ['blue','red']:ex('ip','-n',vrf,'route','add',DEST+'/32','via','192.0.2.2','proto','static')
version=cli('show version');verify('FRR is version 10.4.5','10.4.5' in version)
base=snapshot('overlapping-prefixes')
verify('global main table has no route',base['main']==[]);verify('global lookup cannot resolve endpoint',base['globalLookup']['exit']!=0);verify('global probe cannot reach endpoint',base['globalProbe']['exit']!=0)
for vrf in ['blue','red']:
v=base['vrfs'][vrf];verify(vrf+' scoped probe reaches its endpoint',v['probe']['exit']==0)
route=json.loads(v['lookup']['stdout'])[0];verify(vrf+' lookup uses its own interface',route['dev']==vrf+'-if')
verify(vrf+' lookup resolves the transit next hop',route['gateway']=='192.0.2.2')
verify(vrf+' route exists in the FRR VRF',DEST+'/32' in v['frrRoutes'])
ex('ip','-n','blue','route','del',DEST+'/32');bad=snapshot('blue-route-withdrawn')
verify('blue withdrawal breaks lookup',bad['vrfs']['blue']['lookup']['exit']!=0);verify('blue withdrawal breaks probe',bad['vrfs']['blue']['probe']['exit']!=0)
verify('blue transit peer remains reachable',ping('blue','192.0.2.2').returncode==0)
verify('red survives blue route withdrawal',bad['vrfs']['red']['probe']['exit']==0)
ex('ip','-n','blue','route','add',DEST+'/32','via','192.0.2.2','proto','static');fixed=snapshot('blue-route-restored')
verify('blue route restoration restores probe',fixed['vrfs']['blue']['probe']['exit']==0);verify('red remains reachable after blue recovery',fixed['vrfs']['red']['probe']['exit']==0)
ex('ip','-n','red-peer','address','del',DEST+'/32','dev','lo');endpoint=snapshot('red-endpoint-withdrawn')
verify('red endpoint loss leaves route lookup valid',endpoint['vrfs']['red']['lookup']['exit']==0);verify('red endpoint loss breaks probe',endpoint['vrfs']['red']['probe']['exit']!=0)
verify('red transit peer remains reachable',ping('red','192.0.2.2').returncode==0);verify('blue survives red endpoint loss',endpoint['vrfs']['blue']['probe']['exit']==0)
ex('ip','-n','red-peer','address','add',DEST+'/32','dev','lo');final=snapshot('all-changes-reverted')
verify('both scoped probes recover',all(x['probe']['exit']==0 for x in final['vrfs'].values));verify('global context remains unreachable',final['globalProbe']['exit']!=0)
evidence={'checkedAt':datetime.datetime.now(datetime.timezone.utc).isoformat,'image':IMAGE,'imageId':image['Id'],'architecture':image['Architecture'],'kernel':kernel,'iproute2':ipversion,'frrVersion':version,'backend':'FRRouting zebra -w: Linux network namespaces as VRFs','scriptSha256':hashlib.sha256(pathlib.Path(__file__).read_bytes).hexdigest,'checks':checks,'phases':phases,'limitations':['FRRouting namespace-backed VRFs, not Cisco IOS XE or Linux l3mdev VRF device execution','Two studied routing namespaces plus two endpoint namespaces; Zebra discovers all four','IPv4 ICMP and route lookup only; no TCP, TLS, application identity, IPv6 or throughput claim','No BGP, MPLS, route-target import/export or inter-VRF route-leak execution','One physical host; logical isolation does not demonstrate physical resilience','No external network, ports or mounts; capabilities limited to disposable container'],'cleanup':None}
except Exception:
try:pathlib.Path(sys.argv[1]+'.failure.json').write_text(json.dumps({'checks':checks,'phases':phases,'namespaces':ex('ip','netns','list',check=False).stdout,'vrfs':ex('vtysh','-c','show vrf',check=False).stdout,'routes':ex('vtysh','-c','show ip route vrf all json',check=False).stdout,'interfaces':ex('vtysh','-c','show interface vrf all',check=False).stdout,'log':ex('tail','-n','40','/tmp/zebra-vrf.log',check=False).stdout},indent=2))
except Exception:pass
raise
finally:
cleanup=[{'container':name,'exit':docker(['rm','-f',name],False).returncode}]if created else []
if any(x['exit']for x in cleanup):raise RuntimeError(cleanup)
evidence['cleanup']=cleanup;pathlib.Path(sys.argv[1]).write_text(json.dumps(evidence,indent=2)+'\n');print(json.dumps({'checks':len(checks),'phases':len(phases),'kernel':kernel,'iproute2':ipversion,'cleanup':True}))
Blue without a route: lookup and ping fail. Red without its endpoint address: lookup passes, ping fails.
Common pitfalls
Comparing probes from different contexts; attributing every ping failure to routing; presenting namespaces as l3mdev devices or IOS XE execution.
Related topics: VRF: context and controlled sharing · Virtualization, VRFs and overlays
Relate the injected failure to the table, transit and endpoint before choosing recovery.
Reference: Zebra and Virtual Routing and Forwarding · 350-401 ENCOR v1.2, effective 2026-03-19; core component of CCNP Enterprise