Ординатура / Офтальмология / Английские материалы / Principles Of Medical Statistics_Feinstein_2002

insulin concentrations than the whites or blacks”; and (2) in whites, but not in Pima Indians or blacks, “mean blood pressure ... was significantly correlated with fasting plasma insulin concentration (r = .42) and [with] the rate of glucose disposal during the low dose (r = − 0.41) and high-dose (r = − 0.49) insulin infusions.” The authors concluded further that “a common mechanism, genetic or acquired, such as enhanced adrenergic tone, or a cellular or structural defect may constitute the link between insulin resistance and blood pressure in whites but not in other racial groups.”

FIGURE E.19.4

Relation between Mean Blood Pressure and Fasting Plasma Insulin Concentration (Top Panel) and Insulin-Mediated Glucose Disposal during Low-Dose (Middle Panel) and High-Dose (Bottom Panel) Insulin Infusions in Pima Indians, Whites, and Blacks, after Adjustment for Age, Sex, Body Weight, and Percentage of Body Fat. The differences among the three groups in the slopes of the regression lines between mean blood pressure and the fasting plasma insulin concentration and insulinmediated glucose disposal during low-dose and high-dose insulin infusions were statistically significant (P = 0.001, 0.017, and 0.025 for Pima Indians, whites, and blacks, respectively). [Figure and legend taken from Chapter Reference 25.]

Considering only the graphic evidence, and avoiding discussion of the proposed biologic or physiologic mechanisms, do you think the investigators’ conclusions are justified?

19.5. The four figures marked E.19.5.1 through E.19.5.4 are taken directly from published medical reports. In each instance, the author(s) claimed that a significant relationship had been demonstrated. Except for methodologic details describing the measurements and groups, the text offered no statistical

© 2002 by Chapman & Hall/CRC

information about these relationships, beyond what is shown in the figures. (In one instance, r values reported in the text have been added to the figures.)

For Exercises 19.5.1 through 19.5.4, comment on each of these four analyses, figures, and conclusions. Do you think the authors are justified or unjustified? If you do not like what they did, what would you offer as an alternative?

DOMESTICALLY PRODUCED HANDGUNS (Millions)

FIGURE E.19.5.1

Handgun availability and firearm mortality: United States, 1946–85. [Figure and legend taken from Chapter Reference 26.]

ˆ

19.6. Figure E.19.6 shows a scatter-plot of residuals in which the values of Yi – Yi have been plotted

ˆ

against the values of Yi for a group of 12 bivariate points. What conclusion would you draw from this pattern? If you are unhappy with the problem, what solution would you propose?

19.7. Figure E.19.7 shows the published points and corresponding regression lines for a plot of serum copper vs. pleural fluid copper in three diagnostic categories of patients.

19.7.1.In the text, the authors stated that for the 120 patients with malignant disease the value of r = 0.19 was statistically significant at P < 0.05. Are you convinced that this

significance is biologically important? If so, why? If not, why not?

19.7.2.A regression line has been drawn for each of the three cited equations. How can you quickly (by eye test alone) get supporting evidence that these lines correctly depict the stated equation?

© 2002 by Chapman & Hall/CRC

Plasma Glucose (mg/dl)

FIGURE E.19.5.2

Relationship between fasting plasma glucose and FFA concentrations in nonobese and obese persons with normal glucose tolerance (open circles) or NIDDM (solid circles). [Figure and legend taken from Chapter Reference 27.]

Correlation of mean arterial blood pressure (MABP) and measured PGI2 metabolite excretion (PGIM) in seven essential hypertensive patients receiving no medications. Each patient underwent three separate determinations for MABP and PGIM (during control periods 1 and 2 and during the placebo period) for a total of seven independent observations. Regression analysis was performed with the Statistical Analysis System using analysis of variance. [Figure and legend taken from Chapter Reference 28.]

© 2002 by Chapman & Hall/CRC

SPI

FIGURE E.19.5.4

Interleukin-2 serum level as measured by enzyme-linked immunosorbent assay in 81 sera samples. Data represent the mean of quadruplicate values. [Figure and legend derived from Chapter Reference 29.]

FIGURE E.19.6

[Figure and legend taken from Chapter Reference 30.]

© 2002 by Chapman & Hall/CRC

Pleural fluid Cu, g/dl

FIGURE E.19.7

Positive relationship between pleural fluid copper and serum copper in the group with malignant disease. [Figure and legend taken from Chapter Reference 31.]

© 2002 by Chapman & Hall/CRC

20

Evaluating Concordances

CONTENTS

20.1Distinguishing Trends from Concordances

20.2Conformity vs. Agreement

20.3Challenges in Appraising Agreement

20.3.1Goals of the Research

20.3.2Process, Rater, and Observer

20.3.3Number of Raters

20.3.4Types of Scale

20.3.5Individual and Total Discrepancies

20.3.6Indexes of Directional Disparity

20.3.7“Adjustment” for Chance Agreement

20.3.8Stability and Stochastic Tests

20.4Agreement in Binary Data

20.4.1Proportion (Percentage) of Agreement

20.4.2φ Coefficient

20.4.3Kappa

20.4.4Directional Problems of an “Omnibus” Index

20.5Agreement in Ordinal Data

20.5.1Individual Discrepancies

20.5.2Weighting of Disagreements

20.5.3Proportion of Weighted Agreement

20.5.4Demarcation of Ordinal Categories

20.5.5Artifactual Arrangements

20.5.6Weighted Kappa

20.5.7Correlation Indexes

20.6Agreement in Nominal Data

20.6.1Substantively Weighted Disagreements

20.6.2Choice of Categories

20.6.3Conversion to Other Indexes

20.6.4Biased Agreement in Polytomous Data

20.7Agreement in Dimensional Data

20.7.1Analysis of Increments

20.7.2Analysis of Correlation

20.7.3Analysis of Intraclass Correlation

20.8Stochastic Procedures

20.9Multiple Observers

20.9.1Categorical Data

20.9.2Dimensional Data

References

Exercises

A striking feature of modern medical statistics has been the relative absence of attention to the scientific quality of raw data. Despite the many methods developed for sampling, receiving, and analyzing data, and for drawing conclusions about importance or “significance,” the scientific suitability, accuracy, and reproducibility of the basic information has not been a major focus of concern.

© 2002 by Chapman & Hall/CRC

The contemporary statistical emphasis on inference rather than evidence is particularly ironic because problems of “observer variability” were the stimulus about 150 years ago that made C. F. Gauss develop his “theory of errors” in describing the “normal” distribution of deviations from the “correct” value (i.e., the mean) of a measurement. Analogous challenges in “quality control” for chemical variations in beer at the Guinness brewery were the stimulus almost 100 years ago for W. S. Gosset’s activities that are now famous as the Student t test. After Gaussian and Gossetian theory were established, however, statistical creativity became devoted more to quantity in mathematical variance than to quality in measurement variability.

Until about two decades ago, W. Edwards Deming, a statistician who gave imaginative attention to industrial methods of achieving “quality control,” was generally little known or heralded in Englishspeaking academic enclaves. Nevertheless, Deming’s methods are particularly famous in Japan, where they were adopted after World War II and became a keystone of Japanese success in developing high quality products with modern industrial technology.

Because variability in scientific measurement1 is still distressingly alive, well, and flourishing, the statistical methods of describing the problems have often come from investigators working directly in the pertinent scientific domain. Examining observer variability among radiologists,2 J. Yerushalmy, an epidemiologist, devised the indexes of sensitivity and specificity now commonly used in the literature of diagnostic tests. Indexes of biased observation,3 a special chi-square test for observer variability,4 and the commonly used kappa coefficient of concordance5 were contributed by two psychologists, Quinn McNemar and Jacob Cohen. Statistical methods for describing accuracy and reproducibility in laboratory data have also been developed mainly by workers in that field,6 although R. A. Fisher’s intraclass correlation coefficient has sometimes been applied for laboratory measurements. [Fisher originally proposed7 the intraclass coefficient, however, to compare results in pairs of brothers, not to analyze diverse measurements of the same entity.]

Lacking the mathematical “clout” of statistical theory, the pragmatic challenges of measurement variability are omitted from many textbooks of medical, biologic, and epidemiologic statistics. In many textbook discussions of “association,” in fact, trends in measurements of different variables are not distinguished from concordances in different measurements of the same entity.

This chapter is intended to outline some of the main distinctions and challenges in statistical appraisals of concordance and to discuss the diverse indexes of concordance that appear in medical literature.

20.1 Distinguishing Trends from Concordances

The idea of interchangeability is what separates concordance from trend. Trends are assessed to determine whether two different variables are co-related, i.e., whether they “go along together.” For concordances, however, the goal is to see whether Variable A can be interchangeably substituted for Variable B. An index of trend may have a high correlation value when derived from the proportionate

ˆ

reduction in group variance for sums of squared errors, but the estimates of Yi may not be good enough to be used as direct substitutes for Yi.

To illustrate the problem, consider the four data sets in the table below. The values of X are the same in each set, but the corresponding values of Y — in the columns marked YA,YB,YC , and YD — represent different methods of measuring X.

© 2002 by Chapman & Hall/CRC

The graphs of the four sets of points and the corresponding fitted lines are shown in Figure 20.1.

If assessed for trend, the regression line fits the points perfectly for each of groups YA , YB, and YC, and the line for Group YD also fits very well. The r values are 1 for the first three lines, and close to 1 for the fourth. Despite the almost identical high marks for trend, however, only the YA line has close agreement; and in the three other lines, the corresponding Y values never agree with those of X. InY B , the values are always 4 units higher; in YC, they are doubled; and in Y D , they are alternatingly one unit higher or lower.

“gold standard” may sometimes change. Thus, the “cor - rect” answer to the question in a certifying examination ten years ago may no longer be correct today.

Agreement, however, is assessed without a “gold-standard” criterion. We determine how closely two observations agree, but not whether they are correct. For example, suppose two radiologists independently decide whether pulmonary embolism is present or absent in each of a series of chest films. If no information is available about the patients’ true conditions, we can assess only the radiologists’ agreement. If additional data indicate whether each patient did or did not actually have pulmonary embolism, we can determine each radiologist’s accuracy (i.e., conformity with the correct diagnosis) as well as the agreement between them.

When pathologists provide “readings” of histologic specimens, we can assess only the agreement of the observers, unless one of the pathologists is accorded the deified status of always being correct. Two pathologists’ readings of cytologic specimens (such as pap smears) would usually be assessed for agreement, but accuracy could also be checked if an appropriate histologic decision were available for each smear.

Although studies of observer variability will indicate disagreements among “equal” observers, studies of conformity are done with a “gold standard.” For laboratory measurements, the gold standard is provided by a selected reference laboratory or a national Bureau of Standards. For the categorical measurements used in certifying examinations or audits of health care, the “gold standard” is provided by an individual expert or consensus of designated experts.

© 2002 by Chapman & Hall/CRC

In a particularly common type of conformity research, the gold standard for a diagnostic marker test is the definitive diagnosis of the selected disease. Diagnostic marker tests and other aspects of conformity are appraised in so many different ways that the topic will receive a separate discussion in Chapter 21. The rest of this chapter is devoted to evaluating agreement.

In the specialized jargon developed for describing the ideas, the results of a study of agreement are sometimes called reproducibility, repeatability, reliability, or consistency. The first two terms are inadequate if each rater has given only a single rating, and the third term is unsatisfactory because reliability is an idea that generally connotes “trustworthiness” beyond mere agreement alone. To describe the general concept, consistency is probably the best of the four terms, but fortunately, the assessment of individual (rather than group) agreements is usually called agreement (or concordance). Indexes of agreement or disagreement will be needed for the individual results and total group patterns that can occur in the four main types of rating scales.

20.3 Challenges in Appraising Agreement

Appraising agreement involves a new set of statistical challenges that arise uniquely when discrepancies are noted and summarized for measurements of the same entities.

20.3.1Goals of the Research

An important consideration before the work is done is to establish the goals of the research. Is it intended to expose and quantify the state of disagreement among the raters, or is the main goal to improve the state of the observational art? In most published studies of concordance, the work seems aimed merely at quantifying disagreement. The investigator does the research, presents the report, and departs in an air of sagacious revelation — but nothing happens thereafter. The disagreement itself becomes exposed, but whatever was causing it is neither discovered nor repaired.

The distinction between “diagnostic” demonstration and “therapeutic” improvement is shown in the names commonly used for the research. When disagreement is investigated for laboratory measurements, the work is usually called quality control. The revealed disagreements are confronted and carefully explored; the methods of measurement are checked and improved; and the eventual improvements elevate the quality of the measurement process. In most investigations of disagreement among clinicians, radiologists, and pathologists, however, the work is usually called observer variability. Little or no effort is made afterward to remove the defects that have been revealed.

This apparent complacency is probably due to the difficulty of arranging suitable analytic confrontations. In laboratory measurements, the procedures can usually be clearly delineated, so that sources of variation can easily be sought among the component steps of the observational process. For clinicians, radiologists, and pathologists, however, the component steps of the procedure are not clearly discerned when the result is merely stated as a rating — such as systolic murmur, 1+ enlargement, or poorly differentiated adenocarcinoma — that emerges from a complex act of observation and decision. To determine component steps and criteria for the observational decisions, the observers must meet together, confront the disagreements, and identify the constituent steps of the process. These analytic confrontations may be difficult to arrange because of problems in getting the observers assembled, and also because many observers may dislike the confrontational process and its possible departures from “diplomacy.”

If the goal is to improve rather than merely document the problems of observer variability, however, the investigator may have to plan for much more than just getting and comparing the stated ratings. If the observers cannot be assembled for direct confrontation, the solicited information should include attention not only to the “final” ratings, but particularly to the intricate components and criteria of the observational process.

20.3.2Process, Rater, and Observer

If the entity being “measured” is a white blood count, serum calcium, chest film, liver biopsy, answer to an examination question, or clinical decision, the person or apparatus that produces the actual

© 2002 by Chapman & Hall/CRC