The availability of reliable and proper data is indispensable in manufacturing. If we cannot trust the data set we measured, we are not able to improve our processes, and cannot define appropriate actions. Such as we validate our manufacturing processes with capability studies, testing and measurement, so we have to validate our measurement processes as well. Proper measurement and testing is the basis of quality and Statistical Process Control (SPC). The variation of products have two factors, one is manufacturing process variation, while the other one is the variation of the measurement. In other words, the influencing factors of the final value being measured are the measured sample (true value) and the measurement uncertainty (measurement system variation).

Every measurement and test system has a so called "measurement uncertainty", and this can bring measurement errors. Our intention must be to decrease this uncertainty as small as possible, to increase the quality of measurement data we gathered. In order to validate our measurement systems, we need to be sure if our system is capable to measure, hence we need to perform Measurement System Analysis (MSA).

Generally, the MSA is focusing on variable characteristics (continuous data), but there are methods to analyse a system, that evaluates attribute characteristics (e.g. OK / NOK or GO / NO-GO). During an attribute inspection / testing, appraisers and test systems need to classify parts (see difference of variable and attribute characteristics in the following table).

Our intention is always to reach the highest level of reliability of these classifications (e.g. all appraisers classify good parts as good ones, bas parts as bad ones, and don’t misclassify any of them).

In order to answer the following questions, we need to perform an Attribute Agreement Analysis (AAA) or an Attribute RR:

To answer these questions, we need to evaluate mathematically the connection between the decisions.

Complex Attribute Agreement Analysis (Example 1): we have 3 operators (m), evaluating 10 parts (n) with 2 trials (r). Remark: for an effective AAA we should use at least 20 parts, and the number of trials should be at least 2.

Step 1: Collect the decision results of each evaluation on a data collection sheet for each operator, for each part and for each trial. The reference value should be also indicated in the right column. Number 1 means pass, while number 0 means fail. A-1 means the first trial of operator A.

Step 2: Calculate repeatability for each appraiser (operator).

%Appraiser Repeatability = 100 * Number of parts, where all trials agree / n

where n = the number of parts.

In the case of appraiser A, the %Repeatability is 90% (100 * 9 / 10), as the operator classified 9 parts with the same result during the two trials.

Step 3: Calculate reproducibility. First, we create the cross tabulation for each observer pair (A-B, B-C and A-C).

We fill up the table based on the inspected results:

Next, we summarize the number, receiving the total row and column (a + b), (a + c), (c + d) and (b + d). The total sum is N. We need these numbers for calculating proportion numbers (p0 and pE), and the Kappa.

p0 = the sum of observed proportion, pE = the sum of expected proportions.

The kappa value gives us the statistical inter-rater agreement (proposed by Cohen), in other words it is the proportion of agreement corrected for chance. When K = -1, the appraisers totally disagree, K = 0 means no agreement above expected (only by chance), while K = 1 means perfect (or total) agreement.

The following thresholds are key to decide about inter-rater agreement:

Step 4: Calculate the appraiser effectiveness for each appraiser (operator) with the following formulae:

%Effectiveness = (Number of correct decisions) / (Total opportunities for decision)

In case of appraiser A the %Effectiveness is 95% (100 * 19 / 20), as the appraiser classified the part correctly 19 times, compared to the overall 20 trial opportunities (10 parts * 2 trials).

Simple Attribute RR (Example 2): we have 2 operators (m), evaluating 10 parts (n) with 2 trials (r).

Meaning of variables and their calculations:

What do the numbers tell us?

Before conducting any improvement or intervention in your process, be sure, that the noticed deviation is the result of the process, and not your measurement system. In addition, SPC is not efficient if your measurement system is incapable, as the measured results are distorted, and may be far away from the true value.

Prior to the AAA or Attribute R&R study, select parts from the true variation range, otherwise your study will become worthless. Why? Because we analyse and evaluate the measurement system, and not the parts. If you choose very similar products (with very low variation), the measurement system variation will have a much higher contribution to the whole variation, despite it is not being the truth.

Operators must not know which part they measure or evaluate, to avoid any prejudice that would affect their decision making.