LEARN. PRACTICE. GROW.From completed work to user experience
A team published three versions of a reconciliation portal. That confirms delivery, but does not yet show whether operators can finish their work with less assistance. Distinguish resources invested, activities performed, outputs produced, outcomes experienced by users, and impacts on the organization. These are different observations that can complement each other. Define the task, population, and observation period before selecting a measure. In this exercise, the proportion of tasks completed without assistance observes a user capability. Deployment count informs delivery. A favorable result in one measure does not automatically determine the result in the others.
Four perspectives for framing questions
In Evidence-Based Management, Current Value helps observe value already experienced; Unrealized Value guides investigation of unmet needs. Time-to-Market considers how quickly the organization can deliver and learn; Ability-to-Innovate helps investigate obstacles to creating new capabilities. In an internal service, twenty days waiting for feedback may limit learning while three old variants consume maintenance capacity. These observations may be related. They are neither four mandatory scores to add together nor a fixed metric checklist. Choose measures that answer the hypotheses and keep attention on outcomes for users. Improving internal speed alone does not establish improved customer experience.
Designing an experiment that supports a decision
Write a concrete hypothesis: an assisted query might enable new operators to identify exceptions without asking for help. Before starting, identify participants, task, period, measure, and quality conditions. Also define when to stop the experiment if adverse effects appear. A small experiment may be enough to investigate uncertainty; building the complete feature is not always necessary. However, a learning prototype should not be presented as a Done production increment without meeting the Definition of Done. After observation, compare evidence with the prediction, explain limitations, and decide whether to adjust the solution, investigate another hypothesis, or cautiously broaden the scope.
Rates, populations, and limits of inference
In pilot A, 18 of 20 tasks were completed without assistance: 90%. In B, 32 of 40 were completed: 80%. A has an observed rate ten percentage points higher, although B has more completions in absolute terms. This establishes neither causality, permanence, nor statistical significance. Compare definitions and population composition. A pilot involving only experienced operators does not automatically represent newcomers. If the mix of groups changes, the aggregate rate can improve even when every group worsens. Show numerators, denominators, and results by group before advocating expansion. Arithmetic helps describe observations; it does not replace suitable experimental design.
Freed capacity and financial effect
An automation avoids twenty weekly hours of manual work but requires six hours of support and four of additional controls. Assuming categories do not overlap, it frees ten net hours per week. This does not establish an equivalent budget reduction: people may use that time for other tasks, costs may be fixed, and additional expenses may exist. Record the horizon, units, assumptions, and quality effects. Do not add saved hours to monetary amounts or count the same benefit twice. To decide the next investment, compare the observed benefit with the unmet need and the effort required to keep the solution operating.
Connecting the experiment to product direction
A strategic direction might seek greater operational autonomy. An intermediate goal brings that ambition closer, and a tactical goal guides an immediate experiment. These EBM horizons support learning; they do not replace Scrum commitments. Retain clarity about the Product Goal and Sprint Goal. In a simple record, write the hypothesis, prediction, observation, limitations, and next decision. If the result contradicts the prediction, investigate before changing the metric to declare success. Deciding to stop one solution can save investment and enable another hypothesis. Reporting should show what changed in understanding the problem as well as the work delivered.
Pilot A: 18/20 = 90%; pilot B: 32/40 = 80%. The observed gap is 10 percentage points. Identify what remains unknown before attributing the difference to the solution.
Common pitfalls
Comparing counts without denominators; selecting only the favorable metric; extrapolating experienced operators to everyone; automatically converting freed hours into budget reductions.
Related topics: Value and Product Goal · Stakeholders and learning
A useful measure supports a decision with visible assumptions. Observed outcomes guide learning but do not alone prove causality or financial savings.
Reference: Online Evidence-Based Management Guide | Scrum.org · PSPO I; Scrum Guide November 2020; no public numbered exam revision