# Intraclass Correlation Coefficient

It can run on SPSS, but data cannot be interpreted about kappa, weighted kappa, ICC, consistency, and agreement.

• FIM_TOTAL1 and FIM_TOTAL1R
 Correlations FIM_TOTAL1 FIM_TOTAL1R FIM_TOTAL1 Pearson Correlation 1 .912** Sig. (2-tailed) .000 N 209 209 FIM_TOTAL1R Pearson Correlation .912** 1 Sig. (2-tailed) .000 N 209 209 **. Correlation is significant at the 0.01 level (2-tailed).

There is a significant strong positive correlation between FIM_TOTAL1and FIM_TOTAL1R

• RAI_TOTAL1 and RAI_TOTAL1R
 Correlations RAI_TOTAL1 RAI_TOTAL1R RAI_TOTAL1 Pearson Correlation 1 .935** Sig. (2-tailed) .000 N 209 209 RAI_TOTAL1R Pearson Correlation .935** 1 Sig. (2-tailed) .000 N 209 209 **. Correlation is significant at the 0.01 level (2-tailed).

There is a significant strong positive correlation between RAI_TOTAL1and RAI_TOTAL1R

SAMP WEEK 6

Calculate Intraclass Correlation Coefficient (ICC) and Kappa and Weighted Kappa:

Calculate the ICC for the following pairs of measures (In SPSS, use: Analyze > Scale > Reliability Analysis):

• FIM_TOTAL1 and FIM_TOTAL1R
• RAI_TOTAL1 and RAI_TOTAL1R
• SRH1 and SRH1R
• QoL1 and QoL1R
• In these calculations, consider whether you are interested in “consistency” or “agreement” as discussed in the course text (Streiner, Norman and Cairney, p. 167-169), and whether you are considering the times of the ratings (time 1 and time 2) as fixed or random factors.
• Compare the ICC values with the Pearson’s correlations obtained in Week 1; discuss how would you interpret the differences in the coefficients.

Calculate Kappa:

• In SPSS, for Kappa, use: Analyze > Crosstabs
• Take the admission and discharge scores for two individual items from the RAI or FIM scales (e.g., RAI_WALK1 and RAI_WALK2) and calculate kappa.
• Calculate kappa for SRH1 and SRH1R
• Use quadratic weights for weighted kappa. Briefly discuss differences in results between weighted and unweighted kappa.

Calculate Weighted Kappa:

• Calculate weighted kappa for the same items used above (RAI and/or FIM items, SRH)
• In SPSS, use: Analyze > Crosstabs, to create a table
• Using the values from the table, and using a quadratic weighting scheme, calculate weighted kappa using the utility in the VassarStats website.

Part 1 calculations

• FIM_TOTAL1 and FIM_TOTAL1R
 Correlations FIM_TOTAL1 FIM_TOTAL1R FIM_TOTAL1 Pearson Correlation 1 .912** Sig. (2-tailed) .000 N 209 209 FIM_TOTAL1R Pearson Correlation .912** 1 Sig. (2-tailed) .000 N 209 209 **. Correlation is significant at the 0.01 level (2-tailed).

There is a significant strong positive correlation between FIM_TOTAL1and FIM_TOTAL1R

• RAI_TOTAL1 and RAI_TOTAL1R
 Correlations RAI_TOTAL1 RAI_TOTAL1R RAI_TOTAL1 Pearson Correlation 1 .935** Sig. (2-tailed) .000 N 209 209 RAI_TOTAL1R Pearson Correlation .935** 1 Sig. (2-tailed) .000 N 209 209 **. Correlation is significant at the 0.01 level (2-tailed).

There is a significant strong positive correlation between RAI_TOTAL1and RAI_TOTAL1R

Solution

SAMP WEEK 6

Part 1 calculations

• FIM_TOTAL1 and FIM_TOTAL1R
 Correlations FIM_TOTAL1 FIM_TOTAL1R FIM_TOTAL1 Pearson Correlation 1 .912** Sig. (2-tailed) .000 N 209 209 FIM_TOTAL1R Pearson Correlation .912** 1 Sig. (2-tailed) .000 N 209 209 **. Correlation is significant at the 0.01 level (2-tailed).

There is a significant strong positive correlation between FIM_TOTAL1and FIM_TOTAL1R

 Intraclass Correlation Coefficient Intraclass Correlationb 95% Confidence Interval F Test with True Value 0 Lower Bound Upper Bound Value df1 df2 Sig Single Measures ,909a ,882 ,930 21,145 208 208 ,000 Average Measures ,952c ,938 ,964 21,145 208 208 ,000 Two-way mixed effects model where people effects are random and measures effects are fixed. a. The estimator is the same, whether the interaction effect is present or not. b. Type A intraclass correlation coefficients using an absolute agreement definition. c. This estimate is computed assuming the interaction effect is absent, because it is not estimable otherwise.

Two-way mixed model as observations are random, but raters are not.

Type is absolute agreement, as systematic differences are relevant.

The Pearson’s correlation coefficient is pretty close to single measure ICC, which is significant and excellent as its far above 0.75.

• RAI_TOTAL1 and RAI_TOTAL1R
 Correlations RAI_TOTAL1 RAI_TOTAL1R RAI_TOTAL1 Pearson Correlation 1 .935** Sig. (2-tailed) .000 N 209 209 RAI_TOTAL1R Pearson Correlation .935** 1 Sig. (2-tailed) .000 N 209 209 **. Correlation is significant at the 0.01 level (2-tailed).

There is a significant strong positive correlation between RAI_TOTAL1and RAI_TOTAL1R

 Intraclass Correlation Coefficient Intraclass Correlationb 95% Confidence Interval F Test with True Value 0 Lower Bound Upper Bound Value df1 df2 Sig Single Measures ,930a ,903 ,949 29,764 208 208 ,000 Average Measures ,964c ,949 ,974 29,764 208 208 ,000 Two-way mixed effects model where people effects are random and measures effects are fixed. a. The estimator is the same, whether the interaction effect is present or not. b. Type A intraclass correlation coefficients using an absolute agreement definition. c. This estimate is computed assuming the interaction effect is absent, because it is not estimable otherwise.

Two-way mixed model as observations are random, but raters are not.

Type is absolute agreement, as systematic differences are relevant.

The Pearson’s correlation coefficient is pretty close to single measure ICC, which is significant and excellent as its far above 0.75.

(single measure is for single rater, average measure is higher and is for reliability of different raters averaged together)

 Correlations SRH1 SRH1R SRH1 Pearson Correlation 1 ,837** Sig. (2-tailed) ,000 N 209 209 SRH1R Pearson Correlation ,837** 1 Sig. (2-tailed) ,000 N 209 209 **. Correlation is significant at the 0.01 level (2-tailed).
 Intraclass Correlation Coefficient Intraclass Correlationb 95% Confidence Interval F Test with True Value 0 Lower Bound Upper Bound Value df1 df2 Sig Single Measures ,834a ,788 ,871 11,147 208 208 ,000 Average Measures ,910c ,881 ,931 11,147 208 208 ,000 Two-way mixed effects model where people effects are random and measures effects are fixed. a. The estimator is the same, whether the interaction effect is present or not. b. Type A intraclass correlation coefficients using an absolute agreement definition. c. This estimate is computed assuming the interaction effect is absent, because it is not estimable otherwise.

Two-way mixed model as observations are random, but raters are not.

Type is absolute agreement, as systematic differences are relevant.

The Pearson’s correlation coefficient is pretty close to single measure ICC, which is significant and excellent as its far above 0.75.

 Correlations QoL1 QoL1R QoL1 Pearson Correlation 1 ,905** Sig. (2-tailed) ,000 N 209 209 QoL1R Pearson Correlation ,905** 1 Sig. (2-tailed) ,000 N 209 209 **. Correlation is significant at the 0.01 level (2-tailed).
 Intraclass Correlation Coefficient Intraclass Correlationb 95% Confidence Interval F Test with True Value 0 Lower Bound Upper Bound Value df1 df2 Sig Single Measures ,890a ,847 ,919 18,495 208 208 ,000 Average Measures ,942c ,917 ,958 18,495 208 208 ,000 Two-way mixed effects model where people effects are random and measures effects are fixed. a. The estimator is the same, whether the interaction effect is present or not. b. Type A intraclass correlation coefficients using an absolute agreement definition. c. This estimate is computed assuming the interaction effect is absent, because it is not estimable otherwise.

Two-way mixed model as observations are random, but raters are not.

Type is absolute agreement, as systematic differences are relevant.

The Pearson’s correlation coefficient is pretty close to single measure ICC, which is significant and excellent as its far above 0.75.

 RAI_WALK1 * RAI_WALK2 Crosstabulation Count RAI_WALK2 Total 0 1 2 3 4 5 6 RAI_WALK1 0 25 0 0 1 0 0 0 26 1 6 1 0 0 0 0 0 7 2 67 9 7 0 1 0 1 85 3 8 5 2 0 0 0 0 15 4 23 4 4 0 1 0 1 33 5 9 5 6 2 0 3 0 25 6 7 2 4 1 2 0 2 18 Total 145 26 23 4 4 3 4 209
 Symmetric Measures Value Asymptotic Standardized Errora Approximate Tb Approximate Significance Measure of Agreement Kappa ,051 ,022 2,418 ,016 N of Valid Cases 209 a. Not assuming the null hypothesis. b. Using the asymptotic standard error assuming the null hypothesis.

Unweighted Kappa is poor, but significant 0.051.

Kappa with Quadratic Weighting based on atatched webpage is 0.1852.

Weighted kappa are meaningful only if the categories are ordinal and if the weightings ascribed to the categories faithfully reflect the reality of the situation.

Whereas unweighted kappa does not distinguish among degrees of disagreement, weighted kappa incorporates the magnitude of each disagreement and provides partial credit for disagreements when agreement is not complete.

 SRH1 * SRH1R Crosstabulation Count SRH1R Total 1 2 3 4 5 SRH1 1 8 11 2 0 0 21 2 3 40 9 0 0 52 3 1 8 33 15 1 58 4 0 1 8 31 9 49 5 0 0 0 12 17 29 Total 12 60 52 58 27 209
 Symmetric Measures Value Asymptotic Standardized Errora Approximate Tb Approximate Significance Measure of Agreement Kappa ,503 ,043 13,679 ,000 N of Valid Cases 209 a. Not assuming the null hypothesis. b. Using the asymptotic standard error assuming the null hypothesis.

Unweighted Kappa is moderate, and strongly significant.

Kappa with Quadratic Weighting is 0.8338, which is very good.

 Value of K Strength of agreement < 0.20 Poor 0.21 – 0.40 Fair 0.41 – 0.60 Moderate 0.61 – 0.80 Good 0.81 – 1.00 Very good