LEARN. PRACTICE. GROW.An evidence chain
Attendance, satisfaction, learning, application and outcomes answer different questions. Positive session feedback says something about the experience; a solved task demonstrates performance in that assessment. To observe transfer, look for later task execution in the work context. Organizational outcomes, such as reduced incident impact, also depend on technology, exposure and processes. Plan measures before launch and identify the conclusions each supports. Do not convert a completion rate into an estimate of risk reduction. In committee reporting, present the proposed action and missing supporting evidence rather than one traffic-light indicator that blends every level.
Population, version and duplicates
The lab contains ten fictional people; eight are active and assigned the learning element. Nine completion events include a repeat, an old version and people outside scope. To measure current coverage, filter version and population, then count distinct people. The result is five of eight, or 62.5%. Three outstanding people remain identifiable for follow-up. APS has three of four and Business two of four, showing that an aggregate can hide different needs. Preserve the population date and eligibility rule. If someone leaves, the historical snapshot should not be silently rewritten to improve last month’s percentage.
Before and after comparisons
Two participants have first attempts before and after: 40 to 60 and 60 to 80. The paired mean moves from 50 to 70, a difference of 20 percentage points. One participant has only a baseline and another only a follow-up; combining them changes the comparison. Including a second attempt scored at 100 also raises the follow-up mean. Decide in advance whether the measure uses first attempts, best scores or later retention and disclose that rule. Even a paired comparison does not isolate the causal effect of learning: prior practice, technical changes and participant selection can affect results. The exercise demonstrates descriptive calculations without measuring actual people or proving effectiveness.
Difficulty, delivery and unknown data
A decline in clicks between simulations does not by itself demonstrate improvement if examples, audiences or delivery mechanisms changed. Detection difficulty also depends on how a message relates to a recipient’s tasks. Use that context to interpret results and maintain consistent criteria. The lab sends no messages: six synthetic deliveries are confirmed, one bounced and another is unknown. Two recipients with confirmed delivery reported, despite repeated report events. The defined rate is 2/6, approximately 33.33%. A report associated with unknown delivery requires reconciliation and remains separate. Missing telemetry counts neither as safe behavior nor as participant failure.
Decide and improve without hiding limits
A useful program turns analysis into actions: fix the population list, adapt a task, make practice time available or remove a technical obstacle. Assign owners and a date for follow-up observation. Also track adverse effects: a campaign that reduces clicks but discourages reporting can weaken detection. Avoid public rankings of individuals or small groups and restrict access to data needed for follow-up. In the lab, the aggregate CSV reconciles the five valid completions without exporting identifiers. Their absence does not guarantee anonymity in every context. The final report should distinguish what was calculated, what was observed and what remains a hypothesis.
python3 content/labs/cism-learning-program/run.py --output /tmp/cism-learning-evidence.json
# Synthetic SQLite/CSV only; no campaign or real workforce data.Nine events do not mean nine covered people: reconciliation produces five valid completions among eight eligible people.
Common pitfalls
Events as people; absence as success; means from different groups as individual improvement; fewer clicks as proven causality.
Related topics: Risk management · Incident readiness · Metrics and reporting
A reliable metric needs a population, calculation rule, context and a conclusion proportionate to the evidence.
Reference: Building a Cybersecurity and Privacy Learning Program · CISM current outline before November 3, 2026