Absolute Agreement Or Consistency

In addition, the CCI estimate from a reliability study is only an expected value of the true ICC. It makes sense to determine the degree of reliability (i.e. mediocre, moderate, good and excellent) by testing whether the value of the ICC obtained is significantly higher, using statistical conclusions, than the values proposed above. This type of analysis can be easily implemented using SPSS or other statistical software. As part of the reliability analysis, SPSS calculates not only an ICC value, but also its 95% confidence interval. Table 4 shows an example of a background analysis of SPSS. In this hypothetical example, the CCI obtained was calculated by a single scoring model with absolute match and 2-way random effects with 3 advisors in 30 subjects. Although the ICC value obtained is 0.932 (which indicates excellent reliability), its 95% confidence interval is between 0.879 and 0.965, which means that the probability that the real ICC will be at each point between 0.879 and 0.965 is 95%. Therefore, on the basis of statistical conclusions, it would be preferable to conclude the degree of reliability as “good” to “excellent”.” The only difference with Eq (9) is that the nc2 variance was omitted because of the distortion in the measurements.

One can notice the resemblance to Model 1 (see figure 1 or Eq (2)). The coefficient defined by Eq (13) is generally referred to as a measure of consistency between measures [6], i.e. the extent to which subjects retain their hierarchy and internal differences. It should be noted that Model 2, given by Eq (8), is still adopted. Thus, although the same model is used and the same matrix of measurement data is analyzed, the question asked (coherence rather than concordance) is different. As a term is missing in the denominator, compared to Eq (9), we expect a higher ICC with Eq (13). It is reasonable from the unatithivable point of view that if we are satisfied with fair consistency and not absolute agreement, the method could also be considered satisfactory, that is, reliable, but in a limited sense. It was noted that Eq (13) is not correctly an intra-class correlation coefficient, as variance in the denominator is not the total variant [26]. Nevertheless, it has been suggested that it could be useful in measuring consistency [5]. The intra-class correlation coefficient was calculated to assess the agreement between three physicians to assess the anxiety levels of 20 people.

There was absolute misreprescing between the three doctors, with the two-way random effect models and the “single-rater” unit, kappa – 0.2, p – 0.056. We can understand the details of Model 3 distributions in Figure 10 as follows. In the case (a), the bias values give c1 – 1, c2 – 6 and c3 – -1, with Eq (23), 2 – 13. With Eq (24), the ICC absolute match population is 3A – 102/(102 – 13 – 52) ≈ 0.725.