LEARN. PRACTICE. GROW.1. Zero observed does not mean zero in the population
The lab uses the hypergeometric distribution for a fixed simple random sample without replacement. With 100 items and zero deviations observed among 30, exact distribution inversion gives a one-sided upper bound of eight population deviations at the nominal 95% confidence level, using the conservative convention declared in the code. This is neither an estimate that exactly eight exist nor a 95% probability that the control works. It is a property of the procedure under repeated sampling and the model assumptions. When all 100 are examined and correctly classified, the observed count equals the count in the defined population. Errors can still affect the universe definition or the testing procedure itself.
2. Observed rate and bound answer different questions
One deviation in 30 gives an observed rate of approximately 3.33%. In the N=100 example, the 95% upper bound is 13 items, or 13% of the population. The difference represents sampling uncertainty rather than a correction to observed data. If the previously agreed assurance criterion requires a bound no higher than 5%, this sample does not establish it even though the observed rate is lower. Three deviations in 30 increase the bound to 21. Record the criterion before knowing the outcome and apply it without changing the denominator to favor a conclusion. An exception’s severity can also require its own investigation even when an aggregate rate appears small.
3. Missing evidence is not a pass
The selected list contains 30 items: 25 have evidence of compliance, one has a deviation and four still lack sufficient evidence. The summary retains 25 passes, one deviation and four unknowns. One in 26 describes only known results and does not eliminate the four selected items. Replacing them with easier cases changes selection and may hide precisely the most problematic cases. The code also shows a conservative sensitivity calculation treating the four as deviations to explore the worst case; it does not claim that classification was established. The auditor should seek appropriate alternative evidence or report the limitation under the engagement methodology. APS should explain unavailable evidence without reconstructing retrospective approvals as though they were contemporaneous.
4. Investigate exceptions before expanding conclusions
A missing approval can reflect operating failure, lost evidence, incorrect deployment linkage or a different emergency process. Define the attribute and applicable criterion before classification. A current screenshot does not establish approval at execution time; conversely, an empty extract field alone does not establish that approval never existed. Confirm facts, scope and consequences. If new groups or process changes emerge, assess their effect on the plan. Extending the sample may be appropriate, but stopping as soon as results look favorable does not automatically retain the confidence of the original fixed design. Document the new decision and use a method matching the sequence actually followed.
5. Complete the conclusion and follow up correction
The report should distinguish population, selection, performed procedure, results, uncertainty and limitations. Avoid saying “100% compliant” when only part was inspected or when the test does not cover complete operation. A recovery control needs evidence of the recovered outcome rather than merely closed tickets; an access review needs criteria about effective rights and revocation rather than only a signature. Management decides remediation and owns risk; the auditor assesses sufficient evidence for the conclusion and follows up treatment. Repeating the lab checks calculations and synthetic selection, including 818 small configurations of bound coverage. It does not validate a client population, production controls or an actual audit opinion.
# From content/labs/cisa-sampling-decisions
from sampling import upper_count, summarize
print(upper_count(100, 30, 0)) # 8, not a known true count
print(upper_count(100, 30, 1)) # 13, not a posterior probability
print(summarize([True] * 25 + [False] + [None] * 4))
# Four unknown outcomes remain explicit.
With zero among 30 from a population of 100, the teaching upper bound is eight; with one deviation, it becomes 13. A low observed rate may not satisfy the defined assurance criterion.
Common pitfalls
Confusing confidence with compliance probability; removing unknowns; extrapolating targeted selections; ignoring individual severity; extending a sample until an unfavorable conclusion disappears.
Related topics: Design audit samples
Report what was observed, the uncertainty remaining and the decision supported by evidence, keeping scope explicit.
Reference: DATAPLOT hypergeometric cumulative distribution reference · CISA outline effective August 1, 2024