Ординатура / Офтальмология / Английские материалы / Using and Understanding Medical Statistics_Matthews, Farewell_2007

same confidence encompasses the belief that had the measurement been taken by another observer, or indeed repeated by the same observer, the observed temperature would have been similar.

The terms ‘observer reliability’ and ‘observer agreement’ are often used interchangeably; in theory, they are different concepts. Reliability coefficients indicate the ability of the corresponding measurements to differentiate among subjects. For continuous measures, these coefficients are typically ratios of variances: in general, the variance attributed to the differences in the measurement of interest among subjects divided by the total variance in the same measurement. For example, if a particular test was administered at two different time points and the data from those successive determinations were available, in an ideal world the results would be the same. However, variation in the method and location of sampling, as well as variation in other aspects of the measurement process, may give rise to different outcomes. In this context, if both occasions when the test is administered provide the same capability to discriminate between patients with respect to the measurement under investigation, we might claim to have empirical evidence of test measurement reliability.

In contrast, agreement refers to conformity. Agreement parameters assess whether the same observation results if a particular measurement is carried out more than once, either by the same observer or by different observers. Typically, observers are said to exhibit a high degree of agreement if a large percentage of the measurements they carried out actually concurred. Poor agreement corresponds to a situation where observers often made substantially different measurements.

In a more heterogeneous population (with wider ranges of observed measurements), the value of a reliability coefficient will tend to be larger. This reflects the fact that, in heterogeneous populations, subjects are easier to distinguish than in homogeneous populations. In general, one might expect that, in a heterogeneous population, reliability and agreement measures will correspond well. On the other hand, in homogeneous populations reliability is generally low since it is difficult to distinguish between patients; however, agreement may well be high.

There is a considerable literature about assessing reliability and agreement, and that literature reflects different opinions on the best way to characterize these two concepts. Here we adopt the limited aim of presenting some methods that are commonly used to study reliability and agreement, and discussing their various properties. In doing so, we hope to highlight some important issues for the reader.

Table 23.1. BILAG scores calculated for eight lupus patients evaluated by eight different physicians

23.2. Intraclass Correlation Coefficient

Consider first the situation when the measurement of interest, Y, is a continuous variable. For each of N subjects, we assume there are J measurements made by J different observers. Table 23.1 displays an example of such measurements of a total disease activity score, called BILAG, for patients with systemic lupus erythematosus. In this example, eight patients with lupus were each examined by eight physicians who completed activity questionnaires from which the actual BILAG values were calculated. Thus, for this example, N and J happen to have the same value; however, their equality is not a universal requirement in measurement reliability studies.

It is helpful to represent this measurement study involving N subjects and J observers in terms of the statistical model

Yij = a + bi + cj + eij,

which is simply another way of representing an analysis of variance model like those we previously encountered in chapter 15. Here we need to use subscripts to be very explicit about the model. The subscript i indexes the N subjects and j indexes the J observers. There are NJ observations in total, J for each of the N subjects, and the subscript pair, ij, uniquely identifies a single observation. The symbols b1, ..., bN, which are called subject effects, are similar to regression coefficients associated with the subjects labelled 1, ..., N. Likewise, the symbols c1, ..., cJ – so-called observer effects – are akin to regression coefficients that correspond to observers 1, ..., J. The symbol a is the analog of an intercept term,

Table 23.2. ANOVA table for BILAG reliability/agreement experiment

and eij represents the residual value that makes the leftand right-hand sides of the model equation equal to each other.

Table 23.2 is the ANOVA table that corresponds to fitting this model to the measurement study data summarized in table 23.1.

The N patients involved in such a measurement reliability study are assumed to be a random sample of possible subjects on whom the measurements of interest might have been observed. The ‘effect’ associated with the subject labelled i, i.e., bi, is assumed to be one observation from a normal distribution with mean zero and variance b2. Thus, b2 represents the variation in the measurement of interest by the same observer from subject to subject. In addition, in the context of measurement agreement and reliability, it is common to think of the J observers as a random sample of possible observers, and to assume that the ‘effect’ associated with observer j, i.e., cj, is one observation from a normal distribution with mean zero and variance c2. Then c2 represents the variation in the measurement of interest from observer to observer when measuring the same subject. Finally, the NJ residuals, which are represented by eij, are assumed to be observations from a normal distribution with mean zero and variance e2.

If readers find this measurement study model confusing, it is sufficient simply to realize that the model involves three sources of variation – subject (denoted by b2), observer (denoted by c2) and residual (denoted by e2).

While the ANOVA summary displayed in table 23.2 has the same form as those we encountered in chapters 10 and 15, it is used in a different way for reliability and agreement studies. We will not discuss the different uses in general but simply say that the entries in the column labelled Mean square are used to estimate variances such as b2, c2 and e2. For example, in the particular case of table 23.2, it turns out that the mean square for subjects is an estimate of J b2 + e2, the mean square for observers is an estimate of N c2 + e2, and the mean square for residuals is an estimate of e2. By suitably manipulating these expres-

sions, we obtain estimates of b2, c2 and e2 that are equal to 20.68, 2.75 and 15.40, respectively.

The total variation in the observations carried out in the study is the sum of the three variances, i.e., b2 + c2 + e2. If the measurement under investigation is to have any value for discriminating between subjects, then the variation associated with the subject effects should represent the largest fraction of this total variation. An intraclass correlation coefficient (ICC), which is defined as the ratio

2

b ,

b2 c2 e2

measures this fraction. Thus, the ICC compares the component of variation in BILAG associated with subjects to the total variance in BILAG.

As we indicated earlier, the full details of how to estimate an ICC are beyond the scope of this book. Note, also, that there is no particular interest in testing the hypothesis that the ICC is equal to any particular value. However, from an appropriate statistical analysis, an estimate and confidence interval for an ICC can be calculated. These quantities can be used in the usual way to quantify what can be said about reliability based on the available data. For the example summarized in table 23.1, the estimated ICC is 20.68/(20.68 + 2.75 + 15.40), which equals 0.53. With only 53% of the total variation in BILAG scores associated with the subjects, the reliability of the activity score measurement appears to be moderate at best. The 95% confidence interval associated with this estimate is (0.28, 0.84), and the rather considerable width of this interval reflects the relatively small sample size. Therefore the true value of the ICC is rather uncertain. The reliability of the BILAG measurement could be as low as 0.28, which is quite small; on the other hand, it might also be as large as 0.84, which would indicate quite good reliability.

There are other experimental designs that one can use to undertake a measurement reliability study, and the definition of the intraclass correlation coefficient varies somewhat with different designs. We have chosen to limit our discussion to the case we described because, although it is not the simplest statistically, this design with multiple observers is a very common one that amply demonstrates the general concept that an ICC conveys.

Finally, note that the ICC is primarily an evaluation of measurement reliability, that is of the ability of a measurement to discriminate between subjects or patients. It does not explicitly address the question of agreement between measurements taken on the same subject by different observers.

23.3. Assessing Agreement

For continuous measurements, the use of ANOVA to study reliability and agreement is the basis of ‘generalizability’ studies [82]. In the corresponding literature, it is recommended that investigators undertake and present a full analysis of variance, including the estimation of multiple sources of variance. The square roots of the estimated variance components, which have the same dimension as the original measurements, estimate standard deviations that can be compared to evaluate the relative size of the different sources of variation on a meaningful scale.

While careful examination of a full ANOVA table is generally recommended, specific measures of reliability and agreement are designed to extract the most relevant information from the ANOVA for the measurement study. For example, reliability coefficients are often ratios of variance estimates. Since these quantities are widely known and convey useful information, they can be used as convenient summary measures of reliability when one is examining a large number of outcomes. The gain from attempting to develop an alternative measure based on standard deviations would likely be minimal.

In contrast to the question of measurement reliability, there is more debate as to what constitutes a suitable summary of measurement agreement, and there is, perhaps, a need to consider quantities that are clearly distinguishable from measures of reliability. Following Isenberg et al. [83], we suggest using the ratio of the standard deviation of measurement attributable to the observers and the standard deviation of measurement attributable to the subjects as a measure of agreement. For the measurement model for BILAG scores that we used in the previous section, this ratio of standard deviations is equal toc/ b = r. A small value of r is associated both with a small value of c, which indicates a high level of agreement between the observers, and a large value ofb. Since this summary indicator of agreement does not involve e, it provides a way of assessing the level of observer/rater concurrence irrespective of the magnitude of the residual variation.

For the example we considered in §23.2, the estimates of c and b are 1.66 and 4.55, respectively. This gives an estimated value for r of 0.36. The corresponding 95% confidence interval for r that can be calculated from this estimate is (0, 0.97). While the small values in this confidence interval suggest that measurement agreement might be good, the larger values of r that are consistent with the study data indicate that the standard deviation for variation due to observers could have a magnitude comparable to the standard deviation associated with the patients, i.e., a value of r that is approximately 1. A larger study would be required to provide more precise information.

Table 23.3. Assessments of musculoskeletal disease activity for 80 subjects provided by two physicians

Table 23.4. The table of expected counts corresponding to the observed data summarized in table 23.3

23.4. The Kappa Coefficient

For data that are not continuous measurements, the statistical method most commonly associated with reliability and agreement is called , the kappa coefficient. It is usually motivated in the following fashion.

Consider first the situation where two observers each assess the same study subjects using a common binary classification scheme. Table 23.3 summarizes the assessments of musculoskeletal disease activity for 80 subjects made by two physicians.

From table 23.3 we can see that the two physicians agree on 67/80 = 83.8% of their assessments. However, even if each physician randomly chose a category for each subject, some minimal level of agreement would be observed. Therefore, is designed to compare observed agreement with what could be expected purely by chance.

Table 23.5. The 4 ! 4 table summarizing two physicians’ assessments of disease activity in 80 lupus patients

Calculating the required expected values for the entries in table 23.3 is no different than the calculations we first outlined in chapter 4 for the 2 ! 2 table. This is because we are assuming that the physicians’ assessments are mutually independent, and therefore agreement arises by chance. The corresponding expected values are displayed in table 23.4.

In the current situation, we are only interested in the diagonal entries in the table, since these represent subjects on whom both physicians’ assessments agree. For example, Physician 1 classified 15/80 (18.8%) of the patients as having active disease; for Physician 2 the corresponding result was 16/80 (20%). Therefore, the probability that both physicians would classify a given patient as having active disease, purely by chance, would be 0.188 ! 0.20 = 0.038. For a group of 80 patients, the corresponding expected value appearing in the upper left corner entry of the table would be 0.038 ! 80 = 3.

From table 23.4, we see that 55 agreements in the 80 patients could be expected by chance. The observed number of agreements was 67 so that 67 – 55 = 12 ‘extra’ agreements were observed. Kappa is equal to the ratio of the observed extra agreements to the maximum possible number of extra agreements which, in this case, would equal 80 – 55 = 25. Thus the value of for table 23.3 would be 12/25 = 0.48, indicating that 48% of the possible increase in agreement, in excess of that due to chance, was observed.

The definition of is easily extended to classification schemes with more than two categories. The data in table 23.3 came from an experiment in which lupus patients were assigned to one of four categories of disease activity. The original data from the experiment are summarized in table 23.5, using the labels A, B, C, and D for the disease activity categories; A represents the highest level of activity.

If we use the same logic in this case that we did for table 23.3 and calculate expected values for the diagonal entries using the rule we outlined in §4.3, the required values will be 0.15, 1.80, 9.63 and 15.75, giving a total of 27.33 agreements that might be expected by chance. The total observed number of agreements is 2 + 5 + 14 + 25 = 46, and therefore the value of for table 23.5 is (46 – 27.33)/(80 – 27.33) = 0.35.

If we represent the observed and expected entries for classification categories 1, ..., K by O11, ..., OKK and e11, ..., eKK, respectively, then

= iK=1 Oii iK=1 eii ,

N K= e

i 1 ii

where N represents the number of patients or subjects.

The value of lies between –1 and 1, although in some tables the lower limit is greater than –1. Negative values indicate that observed agreement is less than that expected by chance, suggesting that the observers tend to disagree rather than to agree.

The values of that represent good and poor agreement can only be assessed subjectively. A common guideline is to say that 1 0.75 represents good agreement, and ! 0.4 represents poor agreement. We do not recommend adopting any particular values as canonical, especially given some of the discussion in §§23.5–23.7. However, the guidelines we just mentioned may be useful in some contexts.

23.5. Weighted Kappa

The ordinary kappa statistic only counts those subjects to whom both raters assign the same classification, and any disagreements are treated with equal severity. However, in a summary such as table 23.4, the categories are naturally ordered and some investigators might claim that disagreements which belong to adjacent categories are less serious than those assigned to categories that are farther apart. This notion gives rise to a summary measure of agreement called a weighted kappa.

To calculate a weighted kappa, it is first necessary to assign ‘agreement weights’ to all cells in the K ! K summary table. Exact agreement, which corresponds to observations recorded on the diagonal of the summary table, is assigned a weight, which we can denote by wii, of 1. Other cells will have weights that represent a fractional value. Thus, the entry belonging to row i and column j is assigned the agreement weight wij between 0 and 1. The weighted measure of observed agreement will then involve a sum over all cells of the table and can be represented by

A = K= K= w O .

O i 1 j 1 ij ij

If we use the same set of assigned weights, the weighted value of expected agreements is equal to

A = K= K= w e .

e i 1 j 1 ij ij

Then the weighted value of is equal to

and is interpreted in the same way as the unweighted kappa. Readers who want to be sure that they understand this formula should check that the value of w is the same as that discussed in §23.4 if wij = 0 when i is different from j.

A common choice of weights is the so-called quadratic weights, which are equal to

Another commonly-used set of weights is called the Cicchetti weights. Normally, any of the weights available in computer programs that calculate weighted kappa values will be adequate. If there is any concern about the effect of choosing a particular set of weight values, w can be calculated with different sets of weights to see if any change in qualitative conclusions arises.

If we use the quadratic weights for the data summarized in table 23.4,w = 0.47. This value is larger than = 0.35 that we calculated in §23.4, reflecting the fact that disagreements between adjacent categories are not treated as seriously as the disagreements concerning a particular patient that span three categories.

It turns out that w, using quadratic weights, is equivalent to an intraclass correlation coefficient. Since the ICC is used to assess measurement reliability, it would seem that w also has properties that are appropriate for a measure of reliability. This will become further apparent in light of our discussion of a particular feature of in §23.7.

23.6. Measures of Agreement for Discrete Data

The quantity r that we introduced in §23.3 can be adapted for use with discrete data, but the details are beyond the scope of this book. Alternatively, we can define agreement measures that are based on the concept of odds ratios – natural characterizations of 2 ! 2 tables. But any discussion of this