Experiments, feedback and adaptation

An uncertainty that changes a decision

An experiment starts with the decision the team needs to make. “Should we automate the recommendation?” contains at least two uncertainties: does the recommendation help someone act, and can the mechanism produce it under operating conditions?

A manually prepared prototype can investigate the first before the engine is built. A volume trial can investigate the second. Record which uncertainty is being addressed, the observed population and the investment depending on the answer.

Choosing the smallest trial is useful only when it addresses the relevant uncertainty. A quick test with colleagues who already know the interface may provide little information about new customers. Also define what happens after favorable, unfavorable or insufficient results.

The experiment then preserves an investment option instead of merely justifying an already settled plan.

Criteria, denominators and joint conditions

Write an observable hypothesis: clearer confirmation should reduce status contacts per confirmed order over seven days. Contact totals without order counts may fall simply because sales fell. Define when the window starts and how to handle orders without seven complete observation days.

Distinguish the outcome measure from acceptance conditions. If the pilot requires at least 80% unaided completion and zero critical errors, both conditions must hold. An 86% rate does not compensate for a critical error.

Retain failures in the record and investigate them. If the measure turns out to be poorly defined, document the revision and its reason; do not silently reclassify an earlier trial as accepted using a rule chosen after seeing its result.

Comparisons that support the conclusion

Before attributing a difference to the solution, compare the conditions that produced the results. If A receives simple requests and B receives complex requests, B’s longer time may reflect that composition. Increasing both samples while retaining the allocation does not remove the limitation.

If price, messaging and channel changed together, increased sales do not identify which change contributed most. The record can preserve the observed improvement and state that attribution remains unestablished. Participant selection also limits conclusions: nine successes among ten experienced volunteers do not automatically estimate performance among first-time users.

Add observations relevant to the missing population and retain the result within its observed scope. A small sample can guide investigation without establishing a population effect.

Close the loop from observation to decision

A usable slice crosses the layers needed to complete a small task. Delivering database, API and interface separately can produce three components without an opportunity to observe the task. Use the slice to obtain feedback while it can still guide subsequent work.

Collection speed is only part of the loop: notes received today but read by the priority decision-maker three months later still delay adaptation. Arrange a review with authority to change priorities and record the decision. Not every suggestion enters the backlog.

Compare suggestions with the objective, evidence and constraints. Continue observation when an outcome needs time: satisfaction in the first 24 hours does not confirm fewer repeat requests over 30 days. An incomplete window calls for follow-up, not automatic acceptance or rejection.

Adapt commitments using actual capacity

Learning can change the solution, sequence or decision to continue. Unrecoverable costs deserve records and learning but do not make further spending mandatory. Compare remaining effort and expected value among feasible options.

If an option needs twelve days, ten are available and partial delivery is unavailable, greater absolute benefit does not make it executable in that period. Check dependencies too: reordering an integration in the backlog does not reserve another team’s review.

Agree timing, update the forecast and present options to stakeholders. When change affects consumers on different schedules, temporary coexistence can support learning without interrupting service. Define migration and acceptance evidence before retiring the previous version.

In the retrospective, examine adverse effects of a process change alongside its intended benefit.

Common pitfalls

Changing criteria after results; confusing observation with causal attribution; collecting without deciding; treating technical availability as completed migration.