LEARN. PRACTICE. GROW.What passing means
Start by writing the functional outcome that delivery must demonstrate. In the exercise, two IDs are required: one checks a sum and another checks rejection of invalid input. This is an explicit teaching policy, not a universal CI/CD rule. The runner supplies observations; the team decides which satisfy each requirement. A favorable overall result can coexist with skipped tests or failures classified as expected. For a real application, add platform context, inputs, configuration and the exercised version. Do not invent a pass percentage to replace discussion of critical behaviors. An exception changes the risk decision but does not turn the original observation into a passed test.
Discovery and execution
Run the lab in a local copy and compare required-tests-pass with wrong-pattern-empty-suite. The first run finds two methods in test_case.py; the second uses checks_*.py and finds none. Both exit zero in the recorded runtime, but only the first satisfies the required set. Before correcting application code, review the starting directory, pattern and selected IDs. Unittest discovery imports modules: the import fixture deliberately raises ImportError and produces an error result. Discovering tests from an unknown contribution therefore already requires respecting the executor trust boundary. This exercise uses only controlled original code, without external contributions or credentials. Keep the recorded interpreter version with the result; another runner can classify empty selection differently.
Interpret each outcome
Compare the skip, expected, unexpected, fail and error fixtures. A skipped test does not execute its body; an expected failure retains a known divergence; both leave passedIds empty. When an assertion marked expectedFailure passes, the runtime records unexpectedSuccesses and returns failure: review the annotation and functional change. The fail fixture executes a failing comparison. The error fixture fails in setUp before executing that comparison. Do not declare functionality correct or incorrect solely from a setup problem. Give the states clear names in the operational report and identify who resolves each condition before verification is repeated. Preserve the original result when a later run changes the outcome.
Counts and coverage
The subtests fixture evaluates three values in one method and records testsRun as one. The example shows why a count needs a unit and context. Refactoring can split one method into several without adding evaluated behaviors; it can also reduce methods without losing the same inputs. At work, maintain a map between important requirements and representative checks. If the only rounding test disappears and two formatting tests are added, the total grows while financial coverage gets worse. Review changes to the required set as part of the software change. Use test count as an investigation signal, alongside results and scope, instead of treating it as demonstrated quality.
Decide during a window
In a fictional APS situation, a supplier moves tests fifteen minutes before a batch window. The runner finishes quickly and the manager receives a green message. Request the detailed report: zero tests means the reconciliation criterion was not observed. Coordinate selection correction and a new run; RUN retains diagnostics and informs the window owner. If time is insufficient, propose another window or a formal exception to the authorized owner, with impact, measures, expiry and a review condition. Do not replace the gap with an administrative test that merely looks for a file. These roles and decisions are learning examples and do not describe any bank procedures.
Workshop and summary
For a twenty-minute workshop, divide responsibilities among development, QA and RUN. Development explains the selector; QA defines required IDs and behaviors; RUN interprets results and recommends whether to proceed. Use three cards: empty suite, skipped critical test and setup failure. Each participant must distinguish observation, hypothesis and missing evidence without editing the report to gain approval. Compare responses with the guide and record interpretation differences. This workshop is proposed but has not been performed by a human group. Retain the sequence: define requirements, confirm selection, interpret outcomes and decide. Connect this lesson with jobs and dependencies, incidents, artifact evidence and release management.
python3 content/labs/cicd-results/run.py --output /tmp/dr-cicd-results-first.json
# Choose a new output path; existing output files are not overwritten.
# CPython 3.13.1 was executed; see the recorded environment.
# wrong-pattern-empty-suite: exitCode=0, testsRun=0
# skip-outcome: exitCode=0, passedIds=[]
# expected-outcome: exitCode=0, expectedFailures contains one ID
# unexpected-outcome: exitCode=1
# These are actual local unittest results, not hosted CI execution.In a fictional batch, moving a directory leaves the pipeline green with zero tests. Required reconciliation still lacks evidence.
Common pitfalls
Counting skips as passed tests, using an expected failure as a functional waiver or compensating for a lost critical test with irrelevant checks.
Related topics: Job contracts · Test evidence · Release management
A successful runner answers according to its own semantics. Delivery needs a policy confirming relevant behaviors in the correct context.
Reference: unittest: discovery, outcomes and test results · CI/CD practices 2026-09; scoped GitHub Actions GitLab Jenkins and Azure DevOps Services documentation