← SSH: secure access and production diagnosis
09 / 12 · 60 MIN

SSH sessions and observed failure stages

Run controlled local sessions and distinguish host trust, authentication, and remote-command results.

Prepare a session independent of personal keys

The runner below creates disposable keys and starts eight temporary sshd instances on loopback. It uses the executing local account without creating accounts or changing the system SSH service. Authorization points only to temporary files; passwords, agent, PTY, forwarding, and user rc are disabled. The client receives explicit configuration and known_hosts built from the host key generated by the experiment itself. The server applies a fixed Python command that records the original request without executing it as code. Provide Python 3 and five matching OpenSSH 10.5p1 executables: ssh, sshd, sshd-session, sshd-auth, and ssh-keygen. Recorded execution compiled that version with OpenSSL 3.6.1 and without PAM. This is not a production configuration template and does not test enterprise account integration.

Compare three paths with different outcomes

Start with rawAccepted, wrongKey, and wrongHost observations. The first confirms the host, authenticates the authorized key, and returns fixed-command JSON. The second reaches the server but offers an unauthorized key and does not run the command. The third contains a deliberately different host key in known_hosts; the client stops before authentication. Two rejections end with status 255 but do not have the same cause. Read client diagnostics and server logs before choosing a change. During real investigation, record destination, account, offered identity, configuration, and reached stage. Do not replace a user key to address an unknown host identity. The lab knows its host key’s provenance; a real rebuild needs a trusted reference and a scoped trust update.

Separate accepted authentication from operation failure

In the positive control’s second session, the client requests fail-seven. The fixed command prints that request and deliberately exits with seven. The log still shows accepted publickey authentication. This demonstrates that execution can fail after access has been established. At fictional bank Cais, proposed key rotation did not address the batch error’s cause. Support should confirm the operation contract, effects already produced, and whether retrying could duplicate delivery. The lab script processes no files or transactions; it creates only a bounded, reproducible observation. Client-submitted text is data, not an executed instruction. Preserve this distinction when documenting ForceCommand access, where an authorized session does not mean permission to open a shell or invoke any operation.

Repeat the experiment and record scope

Save the code as ssh-sessions.py and use the next guide’s command, supplying paths to matching-version executables. JSON contains configuration, certificate fields, diagnostics, and forty checks. The local account name is replaced with lab-account in textual evidence; public keys and temporary paths remain identifiable as experiment artifacts. Private keys are removed with the temporary directory. Repetition generates different keys and ports, so comparison should use acceptance, rejection, and reasons. If a condition fails, retain diagnostics instead of turning failure into an expected result. Summary: configuration, trust, authentication, and operation are related but distinct stages. Connect this lesson to effective configuration, host identity, and incident management. Bastions, tunnels, production accounts, and host certificates remain outside these sessions.

#!/usr/bin/env python3
"""Original loopback SSH authentication lab. OpenSSH 10.5p1 tools, disposable keys, fixed forced command."""
import argparse,hashlib,json,os,platform,pwd,re,shlex,shutil,signal,socket,subprocess,sys,tempfile,time
from pathlib import Path

def main:
 p=argparse.ArgumentParser(description=__doc__)
 for name in ['ssh','sshd','sshd-session','sshd-auth','keygen']:p.add_argument('--'+name,required=True)
 p.add_argument('--output',default='ssh-sessions-evidence.json');args=p.parse_args
 user=pwd.getpwuid(os.getuid).pw_name
 def call(argv,expected=0):
 x=subprocess.run([str(a)for a in argv],text=True,capture_output=True,timeout=20)
 if expected is not None:assert x.returncode==expected,(argv,x.returncode,x.stdout,x.stderr)
 return x
 v=call([args.ssh,'-V']);version=(v.stdout+v.stderr).strip;assert 'OpenSSH_10.5p1' in version,version
 checks=[];observations={};configs=[]
 def check(label,condition):assert condition,label;checks.append(label)
 with tempfile.TemporaryDirectory(prefix='dr-ssh-sessions-') as temp:
 root=Path(temp);root.chmod(0o700)
 def key(name):
 path=root/name;call([args.keygen,'-q','-t','ed25519','-N','','-C','dr-disposable-'+name,'-f',path]);return path
 host=key('host');wronghost=key('wronghost');identity=key('identity');wrong=key('wrong');ca=key('ca')
 force=root/'forced.py'force.write_text('import json,os,sys\nrequest=os.environ.get("SSH_ORIGINAL_COMMAND","")\nprint(json.dumps({"forced":True,"requested":request}))\nsys.exit(7 if request=="fail-seven" else 0)\n')
 forced_command=shlex.quote(sys.executable)+' '+shlex.quote(str(force))
 certs={}
 for label,principal,validity,extras,serial in [
 ('valid',user,'-5m:+1h',[],101),('principal','other-lab-principal','-5m:+1h',[],102),
 ('expired',user,'-2h:-1h',[],103),('critical',user,'-5m:+1h',['-O','critical:dr-unknown=bounded-lab'],104)]:
 path=root/('cert-'+label+'.pub');shutil.copy2(str(identity)+'.pub',path)
 call([args.keygen,'-q','-s',ca,'-I','dr-'+label,'-z',str(serial),'-n',principal,'-V',validity,'-O','clear',*extras,path])
 certs[label]=path.with_name(path.stem+'-cert.pub')
 assert certs[label].exists
 krl=root/'revoked.krl'call([args.keygen,'-k','-f',krl,certs['valid']])
 inspect=call([args.keygen,'-Q','-f',krl,certs['valid']],expected=1)
 check('KRL identifies valid certificate as revoked','REVOKED' in inspect.stdout+inspect.stderr)
 def experiment(label,offered,cert=None,trust_ca=False,bad_host=False,revoked=False,success=False):
 folder=root/("run-"+label);folder.mkdir;authorized=folder/'authorized_keys'authorized.write_text(Path(str(identity)+'.pub').read_textif not trust_ca else '');authorized.chmod(0o600)
 with socket.socketas sock:sock.bind(('127.0.0.1',0));port=sock.getsockname[1]
 known=folder/'known_hosts'pub=Path(str(wronghost if bad_host else host)+'.pub').read_text.split;known.write_text('[127.0.0.1]:'+str(port)+' '+pub[0]+' '+pub[1]+'\n')
 conf=folder/'sshd.conf'log=folder/'server.log'
 text=f'''ListenAddress 127.0.0.1
Port {port}
HostKey {host}
PidFile {folder}/pid
SshdSessionPath {args.sshd_session}
SshdAuthPath {args.sshd_auth}
AuthorizedKeysFile {authorized}
AuthorizedPrincipalsFile none
TrustedUserCAKeys {str(ca)+'.pub' if trust_ca else 'none'}
RevokedKeys {str(krl) if revoked else 'none'}
AllowUsers {user}
AuthenticationMethods publickey
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
PermitUserEnvironment no
PermitUserRC no
PermitTTY no
X11Forwarding no
AllowAgentForwarding no
AllowTcpForwarding no
AllowStreamLocalForwarding no
PermitTunnel no
DisableForwarding yes
StrictModes yes
UseDNS no
LogLevel VERBOSE
ForceCommand {forced_command}
SetEnv ZDOTDIR={folder}
'''
 conf.write_text(text);call([args.sshd,'-t','-f',conf]);effective=call([args.sshd,'-T','-f',conf]).stdout
 configs.append({'label':label,'configuration':text,'effective':effective})
 argv=[args.ssh,'-F','/dev/null','-vv','-p',str(port),'-l',user,'-i',str(offered),'-o','BatchMode=yes','-o','IdentitiesOnly=yes','-o','IdentityAgent=none','-o','UserKnownHostsFile='+str(known),'-o','GlobalKnownHostsFile=/dev/null','-o','StrictHostKeyChecking=yes','-o','UpdateHostKeys=no','-o','PreferredAuthentications=publickey','-o','PasswordAuthentication=no','-o','KbdInteractiveAuthentication=no','-o','ConnectTimeout=5','-o','ConnectionAttempts=1','-o','ControlMaster=no','-o','ControlPath=none','-o','ProxyCommand=none','-o','ProxyJump=none','-o','RequestTTY=no','-o','CertificateFile='+str(cert or 'none'),'127.0.0.1']
 proc=None
 with log.open('w')as lf:
 proc=subprocess.Popen([args.sshd,'-D','-e','-f',str(conf)],stdout=lf,stderr=lf,start_new_session=True)
 try:
 for _ in range(100):
 assert proc.pollis None,log.read_text
 try:
 with socket.create_connection(('127.0.0.1',port),timeout=.1):break
 except OSError:time.sleep(.03)
 else:raise AssertionError('sshd did not listen')
 result=call(argv+['requested-check'],expected=None)
 check(label+': expected client exit',result.returncode==(0 if success else 255))
 if success:
 check(label+': forced command executed',json.loads(result.stdout)=={'forced':True,'requested':'requested-check'})
 check(label+': publickey authentication observed','Authenticated to 'in result.stderr and 'using "publickey"'in result.stderr)
 if label=='rawAccepted':
 failed=call(argv+['fail-seven'],expected=7)
 check('remote exit 7 follows successful authentication',json.loads(failed.stdout)=={'forced':True,'requested':'fail-seven'}and'Authenticated to 'in failed.stderr)
 observations['remoteCommandFailure']={'exit':failed.returncode,'stdout':failed.stdout,'stderr':failed.stderr}
 else:
 check(label+': no forced command output',result.stdout=='')
 check(label+': no authentication success','Authenticated to 'not in result.stderr)
 observations[label]={'exit':result.returncode,'stdout':result.stdout,'stderr':result.stderr}
 except Exception:
 print(json.dumps({'experiment':label,'serverLog':log.read_text}));raise
 finally:
 if proc.pollis None:
 os.killpg(proc.pid,signal.SIGTERM)
 try:proc.wait(timeout=5)
 except subprocess.TimeoutExpired:os.killpg(proc.pid,signal.SIGKILL);proc.wait(timeout=5)
 check(label+': daemon stopped',proc.pollis not None)
 observations[label]['serverLog']=log.read_text
 experiment('rawAccepted',identity,success=True)
 experiment('wrongKey',wrong)
 experiment('wrongHost',identity,bad_host=True)
 experiment('certificateAccepted',identity,certs['valid'],trust_ca=True,success=True)
 experiment('wrongPrincipal',identity,certs['principal'],trust_ca=True)
 experiment('expiredCertificate',identity,certs['expired'],trust_ca=True)
 experiment('unknownCriticalOption',identity,certs['critical'],trust_ca=True)
 experiment('revokedCertificate',identity,certs['valid'],trust_ca=True,revoked=True)
 check('wrong host fails host verification','REMOTE HOST IDENTIFICATION HAS CHANGED' in observations['wrongHost']['stderr'])
 check('wrong principal is diagnosed','name is not a listed principal' in observations['wrongPrincipal']['serverLog'])
 check('expired certificate is diagnosed','Certificate invalid: expired' in observations['expiredCertificate']['serverLog'])
 check('unknown critical option is diagnosed','Certificate critical option "dr-unknown" is not supported' in observations['unknownCriticalOption']['serverLog'])
 check('revocation is diagnosed','revoked' in observations['revokedCertificate']['serverLog'].lower)
 certificate_details={k:call([args.keygen,'-L','-f',path]).stdout for k,path in certs.items}
 check('temporary keys and configuration removed',not root.exists)
 report={'sshVersion':version,'pythonVersion':platform.python_version,'checks':checks,'observations':observations,'configurations':configs,'certificateDetails':certificate_details,'runnerSha256':hashlib.sha256(Path(__file__).read_bytes).hexdigest,'scope':'Eight loopback sshd instances and nine bounded SSH sessions using disposable keys and a fixed ForceCommand. Existing local account, no account creation or system SSH changes. No passwords, personal authorized_keys, agent, user rc, PTY or forwarding. One remote command intentionally exits 7 after successful authentication. Certificates: trusted principal, mismatched principal, expired, unknown critical option and KRL rejection. No production, bastion, host-certificate session, CA rotation or human workshop.','redactions':'Local account name replaced by lab-account in textual evidence; temporary paths and disposable public keys retained.'}
 encoded=json.dumps(report,indent=2).replace(user,'lab-account')
 Path(args.output).write_text(encoded+'\n');print(json.dumps({'instances':8,'sessions':9,'checks':len(checks),'output':args.output}))
if __name__=='__main__':main
IN PRACTICE

Cais has accepted authentication and remote status seven. Investigation moves to the batch operation while retaining SSH session correlation.

Common pitfalls

Rotating keys for any nonzero status, removing trust to bypass a warning, or assuming ForceCommand freely executes the client request.

Related topics: Server and user identity · Job diagnosis

Take this idea with you

A status code alone does not identify cause. Stages, logs, and command results together support a proportionate intervention.

Create account

Reference: OpenSSH daemon operation · OpenSSH concepts and OpenBSD-current manuals consulted 2026-09-29; distribution defaults vary