Variance estimates and confidence intervals for the Kappa measure of classification accuracy
|Authors:||M. A. Kalkhan, R. M. Reich, R. L. Czaplewski|
|Type:||Scientific Journal (JRNL)|
|Station:||Rocky Mountain Research Station|
|Source:||Canadian Journal of Remote Sensing. 23(3): 210-216.|
The Kappa statistic is frequently used to characterize the results of an accuracy assessment used to evaluate land use and land cover classifications obtained by remotely sensed data. This statistic allows comparisons of alternative sampling designs, classification algorithms, photo-interpreters, and so forth. In order to make these comparisons, it is important to know how far in error the estimate might reasonably be. This is accomplished by constructing confidence intervals around the point estimate. The decision to use either the asymptotic variance formulae or bootstrapping variance estimates in constructing confidence intervals for the Kappa statistic is not a simple task. This study was designed to help answer this question. Nine error matrices representing three levels of accuracy (poor, average, and good) of Landsat TM Data consisting of 4, 8 and 16 land cover types in North Carolina were used in this study. Each error matrix was sampled, with replacement, using sample sizes of 50, 100, 150, 300 and 800 pixels to obtain estimates of the Kappa statistic and sample variance. Each of the sample error matrices were resampled 500 times to obtain bootstrap estimates of the variance. The asymptotic variance formula for the Kappa statistic and bootstrap variance provided unbiased estimates of the sample variance. In general, the asymptotic variance estimates were larger than those obtained using bootstrapping, even though the differences were not significant. Confidence intervals based on percentiles of the bootstrap distribution provided the best 95 percent coverage rates (92 to 96 percent with a median of 95 percent). The lowest 95 percent coverage rates were obtained using the bootstrap variance estimate (median of 83 percent).