20
Systematic review and meta-analysis
Learning objectives
When you have finished this chapter you should be able to:
Provide a broad outline of the idea of systematic review.
Outline a typical search procedure.
Describe what is meant by publication bias and its implications.
Describe how we can use the funnel plot to examine for the presence of publication bias.
Explain the importance of heterogeneity across studies and how the L’Abbe´ plot can be used in this context.
Explain the meaning of meta-analysis.
Outline the role of the Mantel-Haenszel procedure in combining studies. Describe what a forest plot is and how it is used.
Medical Statistics from Scratch, Second Edition David Bowers
C 2008 John Wiley & Sons, Ltd
240 |
CH 20 SYSTEMATIC REVIEW AND META-ANALYSIS |
Introduction
If you have a patient with a particular condition and you want to know the current consensus on the most effective treatment, then you could perhaps ask the opinions of colleagues (although they may know no more than you) or maybe look through some pharmaceutical publicity material. Or read all the relevant journals lying around your clinic or office. Better still, if you have access to one of the clinical databases, such as Medline, then the job will be that much easier; in fact, anything like an adequate search is almost impossible otherwise. If you want your search to capture everything written on your topic then you will need a systematic approach. This process of searching for all relevant studies (or trials) is known as a systematic review.
However you do your systematic review, you are likely to encounter some difficulties:
Many of the studies you turn up will be based on smallish samples. As you know, small samples may well produce unreliable results.
Partly as a consequence of the above problem, many of the studies come to different and conflicting conclusions.
There will be some studies that you simply do not find. Perhaps because they are published in obscure and/or non-English-language journals, or are not published at all (for example, internal pharmaceutical company reports, or research dissertations). This shortfall may lead to what is known as publication bias.
To some extent you can address the first two of these problems by combining all of these individual studies into one large study, as you will see later (a process called meta-analysis), and you will also want to deal with the potential for publication bias. But let’s start with a brief description of systematic review.
Systematic review
The basis of a systematic review is a comprehensive search that aims to identify all similar and relevant studies that satisfy a pre-defined set of inclusion and exclusion criteria. As an example, the following extract from a systematic review and meta-analysis of studies of dietary intervention to lower blood cholesterol, shows the inclusion and exclusion criteria, together with a brief description of the search procedure (Tang et al. 1998).
Methods
Identification of trials and extraction of data
We aimed to identify all unconfounded randomised trials |
The object of |
|
of dietary advice to lower cholesterol concentration in free- |
||
the search... |
||
living subjects published before 1996. Trials were eligible for |
|
SYSTEMATIC REVIEW
inclusion if there were at least two groups, of which one could be considered a control group; treatment assignment was by random allocation; the intervention was a global dietary modification (changes to various food components of the diet to achieve the desired targets); and lipid concentration were measured before and after the intervention.
Trials of diets to reduce fat intake in women considered to be at risk of breast cancer were included because the diets were similar to those aimed at lowering cholesterol concentration. We excluded trials of specific supplementation diets (such as those with particular oils or margarine, garlic, plant sterol, or fibre supplements, etc.), multifactorial intervention trials, trials aimed primarily at lowering body weight or blood pressure, and trials whose interventions lasted less than four weeks. Trials based on randomisation of workplace or general practice were also excluded.
241
...the inclusion criteria...
...the exclusion criteria...
To identify these trials we identified four electronic databases (Medline, Human Nutrition, EMBASE, and Allied and Alternative Medicine). These databases included trials published after 1966. We also identified trials by hand searching the American Journal of Human Nutrition by scrutinising the references of review articles and of each relevant randomised trial, and by consulting experts on the subject.
...and the search strategy.
Reports that appeared only in non-English language journals were examined with the help of translators. Trials were categorised according to their approximate target diet into four groups.
The end result of a systematic review then, is a list of studies, each one of which provides a value for the specified outcome measure. In the above example, this outcome measure was the percentage difference in mean total blood cholesterol between the intervention (dietary advice) group and the control group. Examination of this list of outcome values may provide the required insights into treatment effectiveness.
Exercise 20.1 Briefly outline the systematic review procedure and some of the problems that may arise.
The forest plot
The list of studies produced by the systematic review is often accompanied by what is known as a forest plot. This plot has study outcome on the vertical axis, usually arranged by size of study (i.e. by sample size), and the outcome measure on the horizontal axis. The outcome measure
242 |
CH 20 SYSTEMATIC REVIEW AND META-ANALYSIS |
might be odds or risk ratios, means or proportions, or their differences, and so on. There are a number of ways of displaying the data. For example, by using a box with a horizontal line through it, whose length represents the width of the 95 per cent confidence interval for whatever outcome measure is being used. Or with a diamond, whose width represents the 95 per cent confidence interval. The area of each box or diamond should be proportional to its sample size. As an example, the forest plot for the cholesterol study referred to above is shown in Figure 20.1.
Here the 22 individual studies, each represented by a black square whose size is proportional to sample size, are divided into four groups according to their approximate target diet (we don’t need to go into the details). The aggregated mean percentage reduction in cholesterol (with a 95 per cent confidence interval) for each of these groups is represented by a white square, whose size is proportional to the sample size of the aggregated individual studies. The large white square at the bottom of the plot is the aggregated value for all the studies combined. I’ll come back to this shortly.
The horizontal axis represents mean percentage change in blood cholesterol. As you can see, 21 of the 22 studies show a reduction in percentage cholesterol (the study fourth from the top lies exactly on the zero, or no difference, line). However, in seven of the studies the confidence interval crosses the zero line, indicating that the reduction in cholesterol is not statistically significant. The remaining 15 studies show a statistically significant reduction (95 per cent confidence interval does not cross the zero line), as do all four group summary values. Thus there appears to be plenty of evidence that dietary interventions of the type included here do manage to achieve statistically significant reductions in total blood cholesterol.
Exercise 20.2 The results in Table 20.1 show the outcomes (relative risk for proportion of subjects with side effects), from each of six randomised trials comparing antibiotic with placebo for treating acute cough in adults (Fahey et al. 1998). Draw a forest plot of this data and comment briefly on what it shows. Note: relative risks greater than 1 favour the placebo (i.e. fewer side effects).
Table 20.1 The outcomes (relative risk for proportion of subjects with side effects), from each of six randomised trials comparing antibiotic with placebo for treating acute cough in adults. Reproduced from BMJ 1998, 316: 906–10. Figure 4, p. 909. Figures 2 and 3, p. 908, courtesy of BMJ Publishing Group
Study |
Sample size |
Relative risk (95 % CI) |
|
|
|
|
|
Briskfield et al. |
50 |
0.51 |
(0.20 to 1.32) |
Dunlay et al. |
57 |
7.59 |
(0.43 to 134.81) |
Franks and Gleiner |
54 |
3.48 |
(0.39 to 31.38) |
King et al. |
71 |
2.30 |
(0.93 to 5.70) |
Stott and West |
207 |
1.49 |
(0.63 to 3.48) |
Verheij et al. |
158 |
1.71 |
(0.80 to 3.67) |
Total |
597 |
1.51 |
(0.86 to 2.64) |
|
|
|
|
The list of individual studies.
The width of the horizontal line is the 95 % CI for the study.
This white square is the combined result for all of the studies (see the meta-analysis section on
p. 248 below).
American Heart Association step 1 diets
Bloemberg et al33
Sarkkinen et al (A)34 Burr et al35
American diet-heart study (2C)37 Baron et al39
Any American Heart Association step 1 diet
American Heart Association step 2 diets
Watts et al40 Anderson et al (1)42 Anderson et al (2)42
American diet-heart study (1A)37 American diet-heart study (1C)37 American diet-heart study (2A)37 American diet-heart study (2D)37 Dietary intervention study in children43
Any American Heart Association step 2 diet
Diet to Increase polyunsaturated to
unsaturated fat ratio
Leren44 45 46
Woodhill et al47 Sarkkinen et al (B)37
American diet-heart study (1B)37 American diet-heart study (2B)37 Any diet to increase P/S ratio
Low fat diets
Research Committee50
Sarkkinen et al (C)34
Insull et al51
Boyd et al52
Any low fat diet
Any diet
–20 –18
Weighted mean reduction in cholesterol by group and test for differences between effects in different comparisons
3.0% (1.8% to 4.1%) χ24 = 6, P>0.1
5.6% (4.7% to 6.5%) χ27= 45, P<0.001
7.6% (6.2% to 9.0%) χ24= 24, P<0.001
5.8% (3.8% to 7.8%) χ23= 2, P>0.5
5.3% (4.7% to 5.9%) χ221= 104, P<0.0001
–16 –14 –12 –10 –8 –6 –4 –2 0 2 4 6
Mean percentage change in blood cholesterol
Each black square represents an individual study...
... the size of which is proportional to the sample size of that study
The four white squares show the combined results for the studies in each of the four groups.
Figure 20.1 Forest plot for the dietary intervention and blood cholesterol study. Mean percentage changes (with 95 per cent confidence intervals) in blood total cholesterol concentration. Reproduced from BMJ 1998, 316: 1213–20, courtesy of BMJ Publishing Group