LEARN. PRACTICE. GROW.Concept and mechanism
Useful evidence needs to be sufficient for the objective and appropriate in relevance and reliability. Examining records, interviewing owners, and testing mechanisms produce different information and may complement each other. NIST SP 800-53A describes these methods and selecting depth and coverage. Do not choose a method solely for convenience. To test change approval, selecting only approved tickets excludes possible changes without tickets or approval. Increasing the sample within the wrong set does not correct that exclusion. Define the population, reconcile its source, and explain selection before extrapolating results.
Guided application
Attribute tests may assess whether required approval occurred; substantive tests may directly verify amounts and transactions. Technique and sampling depend on the objective, not a universal item count. An analysis of 100% of a CSV remains incomplete if it omits 28 of the expected 10,000 transactions. Also validate script logic and retain filter traceability. In a finding, separate criterion, observed condition, consequence, and recommendation. Evaluate management response and correct factual errors, but do not turn a promised fix into an already effective control. At follow-up, assess proportionate outcome evidence rather than closing solely because a ticket is complete.
No exceptions among 9,972 analyzed records does not mean no exceptions among the expected 10,000.
Common pitfalls
Biased population; unsupported statistical precision; no finding as coverage proof; administrative closure as effectiveness.
Related topics: Governance, resources, and suppliers · Delivery, migration, and acceptance
The conclusion should be no broader than the evidence.
Reference: Assessing Security and Privacy Controls · CISA outline effective August 1, 2024