Publication and other biases

Figure 20.2 Funnel plot used to check for publication bias in a systematic review of the effectiveness of topically applied non-steroidal anti-inflammatory drugs. The asymmetry of the funnel is an indication of publication bias (see text). Reproduced from BMJ, Jan 1998; 316: 333–338, courtesy of BMJ Publishing Group

As an example, Figure 20.2 is a funnel plot from a systematic review of the effectiveness of topically applied non-steroidal anti-inflammatory drugs in acute and chronic pain conditions (Moore et al. 1998). Relative benefit (risk ratio) is shown on the horizontal axis. Each point in the figure represents one of the studies. Values to the left of the value of 1 on the horizontal axis show negative ‘benefit’, values to the right, positive benefit.

The asymmetry in the funnel is quite marked, with a noticeable absence of small studies showing negative ‘benefit’ (risk ratio less than 1). The authors comment:

The funnel plot might be interpreted as showing publication bias. The tendency for smaller trials to produce a larger analgesic effect might be construed as supporting the absence of trials showing no difference between topical non-steroidal and placebo. We made strenuous efforts to unearth unpublished data and contacted all pharmaceutical companies in the United Kingdom that we identified as producing non-steroidal products. One company made unpublished data available to us, but the others did not feel able to do so.

Exercise 20.3 a) Outline the major sources of publication bias. (b) Figure 20.3 shows a funnel plot from a systematic review of trials of beta blockers in secondary prevention after myocardial infarction (Egger and Davey Smith 1998). The plot has odds ratio (horizontal axis) against sample size. Comment on the evidence for publication bias.

Figure 20.3 Funnel plot from a systematic review of trials of beta blockers in secondary prevention after myocardial infarction. Reproduced from BMJ 1998, 316: 61–6. Figure 2, p. 64, courtesy of BMJ Publishing Group

Combining the studies

Meta-analysis is the process of combining a number of separate studies to produce one ‘superstudy’. So, for example, we might have three separate studies, with sample sizes of 40, 80 and 150. When combined, we get a super-study with a sample size of 270. The assumption of the meta-analysis is that this super-study will provide a more reliable and precise overall result for the output variable in question, than do any of the smaller individual studies. We can use the Mantel-Haenszel procedure to combine the studies.2 Before studies can be combined, however, they must satisfy the homogeneity criterion. A few words about that first, before we look at an example of meta-analysis.

Homogeneity among studies

Even when a set of potentially similar studies has been identified, authors have to make sure they are similar, or homogeneous, enough to be combined. For example, they should have similar subjects, have the same type and level of intervention, the same output measure, the same treatment effect and so on. Only if studies are homogeneous in this way can they be properly combined. Studies which don’t have this quality are said to suffer from heterogeneity. The underlying assumption (i.e. the null hypothesis) of meta-analysis is that all of the studies measure the same effect in the same population, and that any differences between them is due to chance alone. When the results are combined the chance element cancels out.

2 Note that this is not to be confused with the Mantel-Haenszel test for heterogeneity.

You might find the comments on heterogeneity by the authors of the diet and cholesterol study quoted earlier illuminating (Tang, et al. 1998):

Heterogeneity between study effects

The design and results of these dietary studies differed greatly. They were conducted over 30 years and varied in their aims, in the intensity and type of intervention, and in the different baseline characteristics of the subjects included. Completeness and duration of follow up also differed. Unsurprisingly, the heterogeneity between their effects on blood cholesterol concentration was also significant. Among the longer trials some, but not all, of the heterogeneity between the effects on blood cholesterol concentration seemed to be due to the type of diet recommended. Deciding which trials should be included in which groups is open to different interpretation and, although we tried to be consistent, for some trials the target diets either were not clearly stated or did not fit neatly into recognised categories such as the step 1 and 2 diets. It is important to be cautious in interpreting meta-analysis when there is evidence of significant heterogeneity; although there was no evidence that the overall results were influenced by trials with outlying values.

The homogeneity assumption should be tested. One possibility is for the authors to provide readers with a L’Abb´e plot. The L’Abbe´ plot displays outcomes from a number of studies, with the percentage of successes (or reduction in risk, etc.) with the treatment group on the vertical axis, and same measure for the control/placebo group on the horizontal axis. The 45◦ line is thus the boundary between effective and non-effective treatment. Values above the line show beneficial results. If possible, varying sized plotting points proportional to sample size should be shown. The more compact the plot, the more homogeneous the studies.

As an example, Figure 20.4 is a L’Abbe´ plot showing outcomes from 37 placebo-controlled trials of topical non-steroidal anti-inflammatory drugs in acute (•), and chronic ( ), pain

Figure 20.4 L’Abbe´ plot showing outcomes from 37 placebo-controlled trials of topical non-steroidal anti-inflammatory drugs in acute (•) and chronic ( ) pain conditions. The compactness of the plotted points is a measure of homogeneity across the studies. Reproduced from BMJ, Jan 1998; 316: 333–338, courtesy of BMJ Publishing Group

conditions (Moore et al. 1998). In this plot, the authors have not plotted the points in proportion to sample size. Whether the degree of spread of the points in Figure 20.4 is indicative of homogeneity among the studies is a matter of judgement, which can only be made by those experienced in the interpretation of these charts. Note that the overall meta-analytic result can also be plotted on this same plot (but is not shown in Figure 20.4).

Mantel-Haenszel test for heterogeneity

A more commonly used alternative is the Mantel-Haenszel test for heterogeneity, which uses the chi-squared distribution (see Chapter 14). The null hypothesis is that the studies are homogeneous. An example of its use is given in Table 20.2, which is taken from a study that ‘aimed to identify and evaluate all published randomised trials of hospital versus general practice care for people with diabetes’ (Griffin 1998). The author’s Table 2 presents a summary of the weighted (by sample size) mean differences, for a number of different outcomes. The author’s Table 3 presents similar information for different outcomes in terms of the odds ratio. The p-values for the Mantel-Haenszel test (using chi-squared) are given in the last column. Only one set of studies (Referral to chiropody, p-value < 0.005) displays evidence of heterogeneity, but since this comprised only two studies, the result is somewhat meaningless.

Meta-analysis and the Mantel-Haenszel procedure

If the studies pass the homogeneity test then we can combine them using the Mantel-Haenszel procedure, to produce the meta-analysis; this will give us an overall value for the outcome in question. The procedure is often accompanied by a forest plot, showing the individual studies, together with the combined result, as in the next example.

This is a report of a meta-analysis of randomised controlled trials to compare antibiotic with placebo, for acute cough in adults, referred to above (Fahey et al. 1998). The focus was on placebo-controlled trials, which reported two specific outcomes: the proportion of subjects reporting productive cough; and the proportion of subjects reporting no improvement at follow-up.3 Figure 20.5 shows the forest plots for these two acute cough outcomes, in terms of the risk ratios (called by the authors ‘relative risks’) in favour of the specific outcome.

The overall net outcome effect is shown with a diamond shape here (one for each of the two outcomes). The area of the diamond is proportional to the total number of studies represented, and the width the 95 per cent confidence interval. Values to the left of an odds ratio of 1 (bottom axis) show reductions in fatalities among cases, those to the right an increase in fatality (compared to control groups).

The Mantel-Haenszel procedure was used to produce the final result shown at the bottom of the forest plot in Figure 20.5. The aggregate relative risks are 0.85 for productive cough and 0.62 for no improvement at follow-up. These appear to show reductions in the risk for both conditions and favour the antibiotic over the placebo. However, since both have 95 per cent confidence intervals which include 1, neither is in fact significant, confirmed by the fact that

3 There was a third outcome concerned with side-effects which is not considered here. See Exercise 20.2 above.

Appendix

Table of random numbers

Medical Statistics from Scratch, Second Edition David Bowers

C 2008 John Wiley & Sons, Ltd