Plan collection and interpret evidence

Start with the question and the unit

Business investigation needs a question that collection can answer. “We need automation” already chooses a solution; “under which conditions does an instruction require manual lookup?” leaves room to discover work, constraints and exceptions.

Before scheduling interviews, specify what each record represents. A person, an instruction, a lookup and an observation session are different units. One operator might process fifty instructions and repeat a lookup three times within one of them.

None of these counts automatically substitutes for another. In a fictional service, Business asks how many operators used manual review at least once in week W. An individual questionnaire can collect that reported experience.

To find the proportion of instructions reviewed, we need eligible instructions and each outcome, with a different observation boundary. Understanding why a review occurred may require following the work sequence and confirming the interpretation with the participant.

Write the question, unit, period and permissible conclusion before choosing the instrument.

Who can enter the collection

A contact list is a starting point, not proof that every relevant perspective can participate. If the process includes day and night operations, an invitation sent only to the day team leaves a gap even when every recipient responds. First identify the roles and conditions covered by the decision; compare them with the people actually reachable.

Record exclusions and seek an appropriate way to involve missing groups. Do not assign answers to absent people to complete a spreadsheet. To explore behavior, you can deliberately select normal, corrected and rejected instructions from each shift.

This helps discover differences; by itself, it does not estimate how often each behavior occurs in the service. Distinguish coverage of planned conditions from the representativeness required for an estimate. A six-cell matrix, two shifts and three instruction types, makes visited and missing cells visible.

Another example from an already observed cell may deepen the explanation, but cannot fill a different cell.

Prepare an instrument people can use

Choose the instrument for the information needed and the actual access conditions. An interview may reveal criteria absent from logs; a technical record may confirm event order without explaining intent; authorized observation may expose steps omitted from a procedure description. Before the main session, try the guide with an example. Confirm that the participant distinguishes week W from the current week, and manual review from automatic resubmission. If two people interpret the question differently, clarifying the term and repeating collection may be more useful than adding more ambiguous responses. During work observation, agree its scope and when questions will be asked. Interrupting every step can alter the behavior being investigated. An alternative is to record observable events and request clarification at an agreed point. Distinguish the observed description from the participant’s later explanation. If an interruption changed the path, identify that trial accordingly. A completed form cannot compensate for an instrument that collected something else.

A success-only log cannot locate missing rejection events, even when it covers every shift. Confirm what the source includes before extending the search period. In a pilot, two observers can apply the same wrong category: agreement on nine of ten cases measures agreement on those cases, not correctness of meaning. Compare categories with confirmed examples and review both disagreements and possible shared errors.

Count responses without changing the unit

In the workshop, the defined population contains eight people: D1 through D4 on the day shift and N1 through N4 on the night shift. The individual question concerns reported manual-review use in week W. Five responses are confirmed: D1 yes, D2 yes, D3 no, D4 yes and N1 no.

Thus three of the five respondents reported use, 60%. This fraction describes respondents. It does not establish that 60% of all eight people used review, or the percentage of instructions reviewed.

The contract permits one response per person. Two rows with the same identity need source resolution: they might be transport repetition, a changed answer or a wrong association. The program rejects repeated identities instead of selecting the latest row without a rule.

If a technical repeat is confirmed, retain the correction record and produce valid input with one response. Do not confuse this control with work observations: two different instructions handled by the same person can be two valid cases while representing only one participant.

Represent what remains unknown

Responses from N2, N3 and N4 are missing. Without additional information, the number of yes responses in the defined population could be three if all missing people answer no, or six if all answer yes. The logical range is 3/8 to 6/8, or 37.5% to 75%.

It is not a confidence interval, a forecast or a statistical adjustment for non-response. It merely enumerates the extremes compatible with the five supplied responses and the complete eight-person list. If the local objective is to establish that at least half this population reported use, current evidence is undecided: both three and six remain possible.

One additional yes raises the lower bound to four; one additional no lowers the upper bound to five. The value of further collection therefore depends on its result and the decision question. Absence stays explicit until information arrives.

If the initial list excludes other eligible people, these bounds no longer describe the wider population. Population size and completeness are assumptions, not calculation results.

Situation coverage is not a person count

The workshop contains six observations of different instructions: D1 normal, D1 corrected, D2 rejected, N1 normal, N1 normal and N2 corrected. These are six instructions and four participants. The planned matrix combines day and night with normal, corrected and rejected.

Five of the six cells have observations; a rejected instruction at night is missing. N1’s two normal cases may help explain variation within that cell, but add neither another participant nor another cell. Adding a rejected night observation from N2 fills the sixth cell while retaining four participants.

Adding a normal observation from N3 increases participants to five while retaining five cells. These are different kinds of progress. Even full matrix coverage does not establish that every rejection reason, experience level or configuration was investigated.

The matrix tracks the selected scope. Revisit that scope when a relevant condition appears that was not represented.

Confirm interpretations and adapt the plan

After collection, separate record quality from the business conclusion. Confirm with participants whether the description reflects what happened and which points remain unexplained. If someone corrects “I did not use review in week W” to “I did not use it on the observed day,” the issue is not choosing the more convenient sentence. Clarify the period, retain the change and obtain an answer to the original question if possible. A missing answer can remain missing after a useful conversation. Plan the next activity around what it must clarify. To investigate a missing cell, seek a suitable case in that condition. To reduce uncertainty about an individual statement, seek the person or source able to confirm it. To understand differing interpretations, try the vocabulary and guide. Further collection needs an objective, access, owner, deadline and expected result. Record what could not be obtained and how that limits the decision. Participant agreement with a description does not replace requirement approval by the agreed authority.

Retain the guide version associated with each answer. Changing “during the month” to “in week W” does not make old records comparable by relabeling them. Reported monthly use may concern another week. Use an explicit period and confirm missing information without assigning the new guide’s meaning to earlier answers. Treat collection objectives separately too: reaching four yes answers can decide the local threshold while a rejection path still needs investigation.

Run the workshop and communicate its limits

The downloadable package contains the fictional dataset, a Python analyzer and instructions with expected results. It requires no account, network connection or production data. First calculate by hand the five respondents, three yes answers, 3/8 and 6/8 bounds, four observed participants and missing cell.

Then run the command in the README and compare results. The program checks the file contract; it does not confirm that answers are true or that the investigation plan is adequate. Change one thing at a time in a copy of the data: add a yes response, a no response, a case in the missing cell and a repeated identity.

Before running, predict which result should change and which should remain the same. When presenting a recommendation to Business and APS, distinguish what people reported, what was observed and what remains unknown. State the scope, assumptions and next useful action.

A finite workshop supports learning; it does not certify a real process or exam readiness.

Common pitfalls

Confusing rows with people; turning missing into no; generalizing a respondent proportion; counting repetition as new coverage; treating logical bounds as a confidence interval; changing an answer without preserving the correction.