LEARN. PRACTICE. GROW.Compare the same question over time
The exercise’s hour-twelve report has eight requests with complete windows and three within-window outcomes. By hour sixteen, all eleven requests admitted by the first observation have had enough time; five meet the rule. Moving from 37.5% to about 45.45% does not demonstrate that a change improved the service. The eligible set grew and additional observations of the same requests became available. The script can fix maximum admission at hour twelve, excluding L from follow-up, and separately show the dynamic population. Define in advance whether to follow a fixed cohort, compare periods with equivalent exposure or inspect the queue’s current state. Each question can justify a view, but giving every view the same title hides important differences.
Separate observed outcome from causal explanation
A team installs a new tool in the same week volume decreases and night coverage expands. Mean time falls. The result merits analysis, but temporal sequence does not identify how much change came from the tool. Record concurrent interventions, seasonality, request mix and measurement changes. Plan evaluation while a comparable reference can still be collected. Depending on the question and conditions, a bounded pilot, a comparison designed with specialist support or qualitative investigation of the mechanism may help. No method becomes valid merely through its name. The Magenta Book provides contextual evaluation guidance; it does not impose a process on this fictional service or define the CBAP exam. The lab has no actual intervention, comparison group or estimated causal effect.
Locate the limitation preventing value
A portal can receive instructions correctly while requests still wait for approval outside the application. Analyze the activity chain before recommending a rewrite. Defective validation or an internal dependency may indicate a solution limitation. A policy requiring a signature unavailable at night introduces an organizational condition to assess with its owners. Both causes can coexist. Use request examples, timing and decisions to examine hypotheses without assigning blame from a chart. The technical team may fix a defect but does not automatically receive authority to change a policy. Expose who must decide, which risk changes and how the action’s outcome will be observed. The recommendation should match the demonstrated limitation rather than the team that owns the dashboard.
Recommend an action with checkable conditions
Evidence may justify investigation before it justifies expansion. If reopened requests account for much of the waiting, propose an authorized review of causes and closure criteria. Define an owner, response date, needed information and the decision that investigation will support. If the hypothesis is confirmed, compare technical correction, process change, training or combined measures with costs and side effects. Improving the metric should not encourage premature ticket closure. Include quality measures and work transferred to other teams. Evaluation budget matters too: collecting data nobody will use adds cost without improving decisions. Ensure capacity to follow the service after the change and record when the recommendation should be reconsidered. An approved action still needs implementation and outcome evidence.
Publish a conclusion that can be reconstructed
A useful report identifies definition version, observation instant, population, exclusions and source. In the example, present eight requests with complete windows: three within deadline, three completed late and two still open at hour twelve. Add the three recent requests separately from the denominator and explain C’s reopening. Retain the historical calculation when new events arrive; a later update needs another observation instant or an identified revision. If a source defect is discovered, correct the record and communicate effects on decisions using the old number. The script’s 36 checks demonstrate local rules and arithmetic, not stakeholder approval, actual data quality or commercial benefit. This distinction lets people use the result without assigning it more scope than the evidence supports.
Original exercise: events and question
Admission by request, in hours from start: A=0, B=1, C=2, D=3, E=4, F=5, G=6, H=8, I=9, J=10, K=11, L=13. Events: A completes at 3; B at 7; C completes at 3, reopens at 4 and completes at 14; D completes at 8; E has no completion; F at 8; G at 11; H at 12; I at 11; J at 14; K at 16. L has no events yet. The window is four elapsed hours, inclusive, from admission. Completion must be the current state at the observed instant. The dataset is fictional and complete by construction. Before running: at hour sixteen, follow only requests admitted by hour twelve. Identify denominator and outcome. Compare with the earlier report and write a conclusion without causal attribution. Propose a next investigation if there is pressure to announce improvement.
Sample analysis and limitations
Sample analysis: the followed cohort has eleven requests, all with complete windows at hour sixteen. A, F, H, I and J meet the rule: 5/11 = about 45.45%. L is outside the fixed cohort. C completed twelve hours after admission and remains late. The difference from 3/8 reflects different observation and eligibility; no intervention was modeled. To investigate actual improvement, define the change, intended outcomes, comparable periods, other changes and missing information. Confirm the rule and source with their owners before recommending expansion.
python3 content/labs/cbap-outcome-windows/run.py --output /tmp/cbap-outcomes.json
# Compare observations.at12, at16 and followed12CohortAt16.
# A changing rate does not establish a causal effect.Following the same eleven admissions for four more hours reveals additional completions. This is not a comparison between an old and a new service.
Common pitfalls
Attributing all improvement to the latest release; comparing cohorts with different observation times; confusing application defects with organizational constraints; silently changing historical indicators.
Related topics: Solution evaluation · Strategy analysis · Benefits and organizational limitations
Measurement, explanation and recommendation are related steps requiring different evidence. Keep conditions and limitations visible in each decision.
Reference: CBAP competencies · CBAP six-knowledge-area blueprint, May 2026 handbook