LEARN. PRACTICE. GROW.Two paths for each packet
In a GRE tunnel, distinguish the inner destination from the outer address used to reach the other endpoint. In this lab, 198.51.100.1 and 198.51.100.2 are inner endpoints; 192.0.2.1 and 192.0.2.2 belong to the underlay. The inner-destination route points to gre-lab, while lookup of the outer address should use the wire interface. This separation transports a packet across another network but does not establish confidentiality. Basic GRE over IPv4 uses IP protocol number 47, not TCP or UDP port 47. When requesting a firewall rule, identify protocol, endpoints and direction; also check return traffic and the policy applied after decapsulation.
An enabled tunnel may have the wrong destination
The run first records lookups and probes in both directions. It then changes the right tunnel remote to 192.0.2.99. The original outer addresses still respond, and GRE interfaces retain the administrative UP flag, but the probe between inner endpoints fails. This separates three facts: the interface was enabled, the underlay reaches the original peer, and the overlay delivers useful traffic. Do not substitute one for the others. Compare local and remote settings at both ends before changing application routes. Restoring 192.0.2.1 on the right endpoint recovers inner probes. This behaviour was observed on Linux; it is not an IOS XE line-protocol or keepalive test.
Detect a circular forwarding dependency
The next phase installs a /32 route on left for 192.0.2.2 through gre-lab itself. Because it is more specific, outer-destination lookup now selects the tunnel that depends on that destination to work; the inner probe fails. Only the introduced route is removed, and recovery is checked. In a real incident, retain route origin and preference as well as lookup output: redistribution or an overlay advertisement may have created the dependency. Cisco documentation describes related RECURDOWN situations, but this lab does not produce that Cisco log. A static route may suit an approved design; do not use it to conceal an incorrect advertisement policy indefinitely.
Calculate MTU without confusing payload, ICMP and TCP
With a 1500-byte underlay, a 20-byte outer IPv4 header without options and a 4-byte basic GRE header, the inner limit used is 1476. In iputils ping, -s counts data only: 1448 + 8 ICMP + 20 inner IPv4 = 1476. With -M do, that size passes and 1449 is rejected locally with Message too large. Reducing tunnel MTU to 1400 permits 1372 data bytes and rejects 1448; underlay MTU stays 1500. These results do not establish a PMTUD black hole in a remote network. GRE options, IPv6 or IPsec protection change the calculation. MSS refers to TCP data and requires its own calculation; adjusting MSS does not generally fix UDP traffic or prove an application transfer.
Accept service and protection using their own evidence
In a fictional APS change, a small reconciliation file arrives but the larger batch misses its operational deadline. Keep size, source, destination, protocol and time window in the evidence; compare MTU limits and errors before attributing failure to the application. Classic IPv4 PMTUD uses ICMP feedback when a router cannot forward a DF packet without fragmentation; blocked feedback is a hypothesis to investigate, not the conclusion of this local exercise. If the contract requires confidentiality, GRE alone is insufficient. Validate the IPsec policy applying to the flow, security associations and agreed cryptographic services. An active negotiation session does not prove that the batch followed the protected policy. Collect evidence of batch sending, receipt, integrity and duration.
Reproduce, interpret and roll back the lab
The complete script below requires Python 3 and Docker. Prepare image dr-encor-gre-tools:20240905-r0 using the lab Dockerfile: it uses the digest-pinned FRR image and adds only iputils-ping=20240905-r0. BusyBox ping in the original image does not support -M; that initial tool error was not counted as a network fault. The run creates two namespaces in a container with no external network, published ports or volumes and starts no FRR daemons. It records versions, lookups, probes, seven phases and 24 checks. Two independent runs confirmed recovery and container removal. Final MTU returns to 1476. The package does not validate IOS XE, IPsec, TCP MSS, applications, throughput or physical redundancy. For production, add those criteria according to the contract and assign rollback and RUN handover owners.
# Toolbox preparation (shell commands, run once before this Python script):
# Save the following two lines as Dockerfile:
# FROM quay.io/frrouting/frr@sha256:b0faf7c8f3d8b09aea1e38f77603c745a66dbf5e26ef9718527c2fbb53cc3aed
# RUN apk add --no-cache iputils-ping=20240905-r0
# docker build --pull=false -t dr-encor-gre-tools:20240905-r0.
# python3 run.py /tmp/gre-evidence.json
"""Original Linux GRE lab. Only a disposable networkless Docker container is changed."""
import datetime,hashlib,json,pathlib,subprocess,sys,uuid
IMAGE='dr-encor-gre-tools:20240905-r0'
name='dr-encor-gre-'+uuid.uuid4.hex[:8];created=False;checks=[];phases=[]
def docker(args,check=True):
r=subprocess.run(['docker',*args],capture_output=True,text=True,timeout=25)
if check and r.returncode:raise RuntimeError(str(args)+'\n'+r.stdout+r.stderr)
return r
def ex(*args,check=True):return docker(['exec',name,*args],check)
def ip(ns,*args,check=True):return ex('ip','-n',ns,*args,check=check)
def pack(r):return {'exit':r.returncode,'stdout':r.stdout,'stderr':r.stderr}
def verify(label,value):
if not value:raise AssertionError(label)
checks.append(label)
def probe(ns='left',size=56,outer=False):
dest=('192.0.2.2' if ns=='left' else '192.0.2.1')if outer else ('198.51.100.2'if ns=='left'else'198.51.100.1')
source=('192.0.2.1' if ns=='left'else'192.0.2.2')if outer else ('198.51.100.1'if ns=='left'else'198.51.100.2')
return ex('ip','netns','exec',ns,'ping','-c','1','-W','1','-M','do','-I',source,'-s',str(size),dest,check=False)
def snapshot(label):
d={'name':label,'observedAt':datetime.datetime.now(datetime.timezone.utc).isoformat,'nodes':{}}
for ns,remote in [('left','192.0.2.2'),('right','192.0.2.1')]:
d['nodes'][ns]={'links':json.loads(ip(ns,'-j','-d','link').stdout),'routes':json.loads(ip(ns,'-j','route').stdout),'outerLookup':pack(ip(ns,'-j','route','get',remote,check=False)),'innerLookup':pack(ip(ns,'-j','route','get','198.51.100.2'if ns=='left'else'198.51.100.1',check=False)),'smallProbe':pack(probe(ns)),'underlayProbe':pack(probe(ns,outer=True))}
phases.append(d);return d
try:
image=json.loads(docker(['image','inspect',IMAGE]).stdout)[0]
docker(['run','-d','--name',name,'--label','dr.lab=encor-gre','--network','none','--memory','128m','--cpus','0.5','--pids-limit','80','--cap-add','NET_ADMIN','--cap-add','NET_RAW','--cap-add','SYS_ADMIN','--entrypoint','/bin/sh',IMAGE,'-c','sleep 900']);created=True
kernel=ex('uname','-r').stdout.strip;version=ex('ip','-Version').stdout.strip;ping_version=ex('ping','-V').stdout.strip
for ns in ['left','right']:ex('ip','netns','add',ns);ip(ns,'link','set','lo','up')
ex('ip','link','add','left-wire','type','veth','peer','name','right-wire')
for ns,i,peer in [('left',1,2),('right',2,1)]:
wire=ns+'-wire'ex('ip','link','set',wire,'netns',ns);ip(ns,'link','set',wire,'mtu','1500');ip(ns,'address','add',f'192.0.2.{i}/24','dev',wire);ip(ns,'link','set',wire,'up')
ip(ns,'tunnel','add','gre-lab','mode','gre','local',f'192.0.2.{i}','remote',f'192.0.2.{peer}','ttl','64');ip(ns,'link','set','gre-lab','mtu','1476');ip(ns,'address','add',f'10.240.0.{i}/30','dev','gre-lab');ip(ns,'link','set','gre-lab','up')
ip(ns,'address','add',f'198.51.100.{i}/32','dev','lo');ip(ns,'route','add',f'198.51.100.{peer}/32','via',f'10.240.0.{peer}','dev','gre-lab')
verify('container global table is empty',json.loads(ex('ip','-j','route').stdout)==[])
base=snapshot('baseline')
for ns in ['left','right']:
n=base['nodes'][ns];verify(ns+' underlay responds',n['underlayProbe']['exit']==0);verify(ns+' inner endpoint responds',n['smallProbe']['exit']==0)
verify(ns+' outer destination uses underlay',json.loads(n['outerLookup']['stdout'])[0]['dev']==ns+'-wire')
verify(ns+' inner destination uses GRE',json.loads(n['innerLookup']['stdout'])[0]['dev']=='gre-lab')
size1476=probe(size=1448);size1477=probe(size=1449)
verify('1476-byte inner packet with DF passes',size1476.returncode==0);verify('1477-byte inner packet with DF is rejected',size1477.returncode!=0)
base['boundary']={'payload1448':pack(size1476),'payload1449':pack(size1477)}
ip('right','tunnel','change','gre-lab','mode','gre','local','192.0.2.2','remote','192.0.2.99','ttl','64');bad=snapshot('wrong-remote')
verify('wrong remote breaks overlay',bad['nodes']['left']['smallProbe']['exit']!=0)
verify('wrong remote retains underlay reachability',all(n['underlayProbe']['exit']==0 for n in bad['nodes'].values))
verify('wrong remote leaves administrative UP flags',all('UP' in next(l for l in n['links']if l['ifname']=='gre-lab')['flags']for n in bad['nodes'].values))
ip('right','tunnel','change','gre-lab','mode','gre','local','192.0.2.2','remote','192.0.2.1','ttl','64');fixed=snapshot('remote-restored')
verify('remote restoration recovers both directions',all(n['smallProbe']['exit']==0 for n in fixed['nodes'].values))
ip('left','route','add','192.0.2.2/32','dev','gre-lab');recursive=snapshot('recursive-underlay-route')
verify('bad host route sends outer destination into GRE',json.loads(recursive['nodes']['left']['outerLookup']['stdout'])[0]['dev']=='gre-lab')
verify('recursive underlay route breaks left overlay probe',recursive['nodes']['left']['smallProbe']['exit']!=0)
ip('left','route','del','192.0.2.2/32');fixed=snapshot('underlay-route-restored')
verify('underlay route recovery restores both directions',all(n['smallProbe']['exit']==0 for n in fixed['nodes'].values))
for ns in ['left','right']:ip(ns,'link','set','gre-lab','mtu','1400')
reduced=snapshot('tunnel-mtu-1400');small=probe(size=1372);large=probe(size=1448);reduced['boundary']={'payload1372':pack(small),'payload1448':pack(large)}
verify('1400-byte inner packet passes reduced MTU',small.returncode==0);verify('1476-byte inner packet is rejected at reduced local MTU',large.returncode!=0)
verify('reduced tunnel MTU leaves small probes healthy',all(n['smallProbe']['exit']==0 for n in reduced['nodes'].values))
verify('underlay MTU remains 1500',all(next(l for l in n['links']if l['ifname']==ns+'-wire')['mtu']==1500 for ns,n in reduced['nodes'].items))
for ns in ['left','right']:ip(ns,'link','set','gre-lab','mtu','1476')
final=snapshot('all-changes-reverted');final['largeProbe']=pack(probe(size=1448))
verify('final large DF probe recovers',final['largeProbe']['exit']==0);verify('final bidirectional small probes recover',all(n['smallProbe']['exit']==0 for n in final['nodes'].values))
evidence={'checkedAt':datetime.datetime.now(datetime.timezone.utc).isoformat,'image':IMAGE,'imageId':image['Id'],'kernel':kernel,'iproute2':version,'ping':ping_version,'scriptSha256':hashlib.sha256(pathlib.Path(__file__).read_bytes).hexdigest,'checks':checks,'phases':phases,'limitations':['Linux GRE over IPv4 in two namespaces on one host; no Cisco IOS XE execution','FRR image supplies tools only; no FRR routing daemons started','Basic GRE without key, checksum or sequence options; explicit tunnel MTU 1476 over underlay 1500','MTU failures are local DF/MTU rejection, not an exercised remote PMTUD black hole','No IPsec, IKE, encryption, TCP MSS, IPv6, application, throughput or physical resilience validation','No external network, ports, mounts or host kernel changes'],'cleanup':None}
except Exception:
try:pathlib.Path(sys.argv[1]+'.failure.json').write_text(json.dumps({'checks':checks,'phases':phases,'left':ip('left','-d','link',check=False).stdout,'right':ip('right','-d','link',check=False).stdout},indent=2))
except Exception:pass
raise
finally:
cleanup=[{'container':name,'exit':docker(['rm','-f',name],False).returncode}]if created else []
if any(x['exit']for x in cleanup):raise RuntimeError(cleanup)
evidence['cleanup']=cleanup;pathlib.Path(sys.argv[1]).write_text(json.dumps(evidence,indent=2)+'\n');print(json.dumps({'checks':len(checks),'phases':len(phases),'cleanup':True}))
With MTU 1476, ping -M do -s 1448 passes; -s 1449 receives Message too large.
Common pitfalls
Treating IP protocol 47 as a port; confusing UP with delivery; resolving the outer endpoint through its own tunnel; confusing -s with MTU; assuming GRE encrypts traffic.
Related topics: VRF: context and controlled sharing · Network assurance and diagnosis
Check both lookups, test known sizes and demonstrate recovery and protection separately.
Reference: Generic Routing Encapsulation · 350-401 ENCOR v1.2, effective 2026-03-19; core component of CCNP Enterprise