Criteria, scales and choice sensitivity

1. Separate eligibility from preference

Comparison starts with conditions determining which options may be considered. In an exercise, every option must support agreed recovery and fit the initial funding limit. Only then are performance and support convenience compared as preferences.

A high score does not compensate for failure of an explicitly mandatory condition. Missing information is not evidence of compliance either, but should be distinguished from an established failure: the former calls for evidence, the latter for resolving incompatibility.

Record all three states and what would permit each to change. Avoid hiding mandatory conditions inside an average where an attractive interface compensates for absent recovery. If the authority changes a condition, that is an explicit decision with consequences; the model should not silently change it to produce a convenient winner.

2. Give scales meaning before combining them

A millisecond column and a euro column cannot be added as though they measured the same thing. Define preference direction and an agreed conversion function. In an example limited to latencies from 100 to 300 ms, the score is 100 × (300 − latency) / 200.

Thus, 100 ms scores one hundred, 300 ms zero and 180 ms sixty. This linear function is an explicit exercise choice, not a law about user satisfaction. If categories are merely low, medium and high, assigning numbers one, two and three does not establish equal value differences.

The team must explain each score’s meaning and why comparison is appropriate. It should also define treatment of out-of-range values rather than improvising a rule after seeing proposals.

3. Distinguish changing units from changing preference

Converting latency from milliseconds to seconds, while also converting anchors of 100 and 300 ms to 0.1 and 0.3 seconds, preserves the score of sixty for 0.18 seconds. By contrast, retaining the 180 ms observation while moving the worse anchor from 300 to 500 ms yields eighty points.

This changes the relationship between performance and score even if the weights in the header remain identical. A scale change can therefore change the matrix result without changing any service. Record units, anchors, direction and the function version used.

When someone says they only normalized the data, establish whether units were transformed equivalently or preference meaning was changed. Review should make that difference understandable to the people approving the criteria.

4. Calculate scores and the point where ranking changes

Consider two already defined zero-to-one-hundred scores where higher is better: performance and support. A scores ninety and fifty; B scores sixty-five and eighty-five. With weight w for performance and 1 − w for support, A obtains 50 + 40w and B obtains 85 − 20w.

At w=0.6, A scores seventy-four and B seventy-three. At w=0.5, A scores seventy and B seventy-five. Equality occurs at w=7/12, approximately 0.5833.

Above that value, the model prefers A; below it, B. Weights represent choices about trading gains in one dimension against losses in another within agreed scales. A one-point difference is not automatically an important or statistically significant advantage.

Data precision, scale meaning and decision policy need to be understood.

5. Recognize dominance within the model

Add C, scoring sixty for performance and forty for support. B is at least as good on both dimensions and better on each. If B and C are equally eligible, these are all considered dimensions and both weights are strictly positive, B scores higher for every combination of those weights.

Finding the exact weight is unnecessary to exclude C as the best option in this model. The conclusion has limits: an important compatibility difference not yet modeled may require another dimension or condition. Do not treat partial dominance as superiority in everything.

If a weight may be zero, improvement only in that dimension might no longer give a strict advantage. Explicit assumptions distinguish a general model conclusion from a preference existing at only one particular weight.

6. Propagate uncertainty without inventing an observed mean

Return to A and B with w=0.6. A’s support score may still lie between forty-five and fifty-five; performance remains ninety. A’s total therefore ranges from seventy-two to seventy-six.

B scores seventy-three with the fixed values already supplied. A’s midpoint is seventy-four, but preference for A does not hold throughout its admissible interval. Do not assign a winning probability merely because more of the drawn interval sits above seventy-three: no distribution was supplied.

If new evidence bounds A’s support score between fifty-one and fifty-three, its total becomes 74.4 to 75.2, exceeding B throughout that interval. This stability concerns modeled conditions; it removes neither omitted risks, funding constraints nor the approval decision.

7. Review repeated criteria and choose the information needed

A team assigns performance weight 0.6 and support weight 0.4. It then splits support into “support convenience” and “operational simplicity”, both calculated from the same data, retaining 0.4 in each column. Normalizing the resulting total of 1.4 gives support 0.8/1.4, approximately 57.1%, instead of forty percent.

The change may be deliberate, but it is not neutral. If the intention is only to split presentation without changing preference, distributing the original weight between equivalent copies can preserve the result. Also distinguish measurement uncertainty from disagreement about values: additional performance tests do not themselves resolve disagreement about the weight Business wants to assign support.

The next activity should address the specific reason why the recommendation can still change.

8. Exercise: present what the matrix establishes

Prepare the decision for three eligible options using performance weight 0.6 and support weight 0.4. A has performance ninety and support between forty-five and fifty-five. B has sixty-five and eighty-five.

C has sixty and forty. These are the model’s only preferences. Solution: B dominates C; A ranges from seventy-two to seventy-six and B scores seventy-three.

Neither A nor B wins for every admissible value of A’s support. Support equality occurs at 47.5 because 0.6 × 90 + 0.4 × 47.5 = 73. Identify evidence collection capable of clarifying that dimension if its cost and time are acceptable.

Retain mandatory conditions, scales, weights, intervals, the assumption capable of changing ranking and decision authority in the report. The matrix organizes a recommendation; calculation does not grant authority to the analyst.

Common pitfalls

Compensating for a mandatory constraint with points; adding different units; treating ordinal categories as demonstrated value differences; changing anchors without reviewing weights; duplicating criteria; interpreting intervals as probabilities; confusing ranking with authorization.