LEARN. PRACTICE. GROW.1. An audit question needs a population
A team wants to assess whether production changes received approval before execution. The unit could be a change, an execution or a change-environment pair. Choosing a different unit changes both the denominator and what will be tested. Record the period, systems, change types, process exceptions and list provenance. Random selection from an incomplete list still excludes work that never entered that list. Before calculating sample size, reconcile the universe against evidence appropriate to the objective and check duplicate identifiers. The lab contains 100 distinct synthetic identifiers. This known population supports a mathematical demonstration; it does not establish that an actual ticket inventory contains every executed change.
2. Choose the method for the intended conclusion
A selection targeted at emergency changes can help investigate a specific risk. It does not become a random sample of all changes merely because it contains many items. If the objective is to infer a rate for a finite population, define an appropriate probability design and retain its logic in analysis. Simple random sampling without replacement gives each set of the planned size the same probability in the ideal model. Stratification can improve coverage of relevant groups, but analysis must consider group sizes and selection probabilities. The method must also respect independence, evidence protection and available resources. Mathematics alone does not choose an acceptable sampling risk for the engagement.
3. Detection planning with explicit assumptions
In the exercise, the owner defines the condition to detect as a population of 100 items containing at least five deviations. The goal is at least a 95% probability of finding one or more using a fixed simple random sample. The code calculates the probability of finding none by counting combinations and obtains a minimum sample of 45. With 44, the probability of missing detection still exceeds 5%; with 45, it is approximately 4.62%. Five is a planning assumption rather than an observed estimate. The plan assumes correct classification and the declared design. It is not a universal rule requiring 45 items for every control, population or audit.
4. Select and retain the selected units
The lab sorts unique identifiers and uses random.sample with a fixed seed to reproduce the example in the recorded runtime. It retains selected IDs, population, size, seed and Python version. The function leaves the original list unchanged and rejects duplicated units. The public seed serves learning; it does not establish that an auditor selected a sample independently. Repeating draws until fewer exceptions appear would introduce outcome-driven selection. The same seed also does not guarantee the same sample across every future Python version. In an actual engagement, document the approved selection process, protect it from interference and preserve the actual selection rather than relying only on regenerating it.
5. Examine strata and periodic patterns
The second example separates 20 emergency changes from 180 standard changes. Ten and 30 are examined respectively, finding two and one deviations. The weighted estimate is ten deviations in 200, or 5%; simply pooling three in 40 gives 7.5%, overrepresenting emergencies. The lab does not calculate a confidence interval for this estimate. Another example has ten deviations spaced twelve positions apart in an ordered population of 120. Selecting every twelfth item with a random start detects deviations in only one of twelve possible starts. Every item has the same inclusion probability, but the distribution differs from simple random sampling. Use analysis matching the design and examine patterns before choosing a systematic interval.
# From content/labs/cisa-sampling-decisions
# Full exercise: python3 run.py --output evidence-local.json
from fractions import Fraction
from sampling import zero_detection_size, cdf
n = zero_detection_size(100, 5, Fraction(1, 20))
print(n) # 45 under these specific assumptions
print(float(cdf(100, 5, n, 0))) # about 0.0462
For N=100 with five assumed deviations, 45 random selections without replacement reduce the probability of finding none to about 4.62%. This is a design assumption, not a conclusion about a client.
Common pitfalls
Using 45 as a universal rule; drawing from an incomplete list; swapping missing items; retrying seeds for a favorable result; applying simple-random analysis to a different design.
Related topics: Interpret samples and deviations
Selection supports a conclusion only when population, unit, design and evidence are aligned and documented.
Reference: ISACA glossary: sampling, population, evidence and tolerable error · CISA outline effective August 1, 2024