2 Synthesising the evidence

The output of the biomedical publication industry is huge. It has been estimated that individual clinicians would need to read nearly 17 articles a day to keep up to date with original articles published in their field.1 When we prepare systematic reviews we aim to combine the available evidence into a whole in order to understand better and, hopefully, to answer a clinical question.

Systematic reviews avoid bias and improve precision

The main aim of a systematic review is to avoid bias and to improve precision. Bias is a broad term encompassing all the possible types of systematic error that may lead to incorrect conclusions being drawn from observational and experimental studies. In this context we can view the systematic review as an observational study of the experimental studies on a particular topic. Since systematic reviews are conducted retrospectively, they are prone to the effects of bias.

When we talk about precision, we are referring to random or biological variation. The main way that we can improve precision, or reduce random error, is to increase the size of the study. Within systematic reviews, it is often possible, and desirable, to pool the results of the studies included in the review to obtain an overall measure of treatment effect. This statistical pooling is termed metaanalysis. The pooled estimate will be more precise than individual estimates from the contributing studies.

Steps involved in doing a systematic review

Box 2.1 sets out the steps involved in conducting a systematic review. These structured steps have been developed in order to minimise bias and improve precision. It is important to see a systematic review as a research project in its own right. As in primary research, a protocol should be prepared, setting out all the work that is to be done. The protocol should state clearly the problem to be addressed, and define the population, interventions and outcome measures to be used. Inclusion and exclusion criteria should be described and the analysis strategy should be developed in detail. The purpose of this detailed protocol

Box 2.1 Steps in conducting a systematic review (Source: adapted from Egger and Smith, 20012)

1Formulate review question.

2Define inclusion and exclusion criteria:

●participants

●interventions and outcomes

●study designs and methodological quality.

3Locate studies (see Chapter 1).

4Select studies:

●have eligibility checked by more than one observer

●develop strategy to resolve disagreements

●keep log of excluded studies, with reasons for exclusions.

5Assess study quality:

●consider assessment by more than one observer

●use simple checklists rather than quality scales

●always assess concealment of treatment allocation, blinding and handling of patient attrition

●consider masking of observers to authors, institutions and journals.

6Extract data:

●design and pilot data extraction form

●consider data extraction by more than one observer

●consider masking of observer to authors, institutions and journals.

7Analyse and present results:

●tabulate results from individual studies

●examine forest plot

●explore possible sources of heterogeneity

●consider meta-analysis of all trials or subgroups of trials

●perform sensitivity analyses, examine funnel plots

●make list of excluded studies available to interested readers.

8Interpret results:

●consider limitations, including publication and related biases

●consider strength of evidence

●consider applicability

●consider numbers needed to treat to benefit/harm

●consider economic implications

●consider implications for future research.

is to prevent decisions about the conduct of the review being influenced by the results.

Location and appraisal of relevant studies

The individual studies that are included in the systematic review form the “data” to be collected. A major task of the reviewer is to identify all the studies that might be relevant to the review (see Chapter 1). Having identified the potentially relevant studies, the reviewer has to decide whether or not to include them in the review. It is important that this process is independent of the results of the study. The best way to ensure this is to have clear criteria for inclusion and exclusion. Another method is to have studies selected independently by at least two people. Masking the study results and identity of the authors is very time-consuming and empirical studies of their usefulness have been contradictory.3,4 Each study also has to be appraised critically in order to assess how likely it is that its results are biased.

Data extraction

Data extraction should be done with as much care as would be taken for data collection in a primary research study. In many cases it will be necessary to contact study authors for clarification. Double data entry and other common sense checks to ensure the accuracy of the data entered into the review are useful.

Individual patient data

In some situations it is feasible and desirable to obtain individual patient data from all studies that contribute to the review. This is particularly useful in studies where time to an event is the outcome of interest. The reviewer can also perform simple checks on the data to improve the validity of the review and can explore appropriate subgroups in more depth. What the reviewer cannot do is pool the data ignoring the studies from which the data have been drawn. The methods of combining the results of the studies are similar to those where individual patient data are not available.

Ways of presenting data

There are several different ways of presenting data drawn from individual studies. Table 2.1 summarises the advantages and disadvantages of the different measures.

Deeks et al. have set out criteria for use when deciding which measure of effect to use, including consistency, mathematical properties and ease of interpretation.5 In general people find it easier to interpret a risk than an odds ratio. Conveniently, the odds ratio is a good estimate of the risk ratio when events are rare, which is often the case in healthcare studies. In these situations, the odds ratio can be analysed and the risk ratio can be interpreted. However, in situations when events are common, odds ratios will overestimate the benefits and harms of a treatment.

Graphical presentation

The forest plot is the traditional way of presenting the data in a systematic review. Figure 7.1 on page x shows an example of a forest plot, in which the results of individual studies are plotted horizontally with a line representing the confidence intervals. All the studies are plotted one after another and, if appropriate, a combined estimate is presented at the bottom of the diagram. The order in which the studies are plotted can be varied, but often they are presented chronologically.

When not to combine data from different studies

A statistical synthesis, or meta-analysis, is not always recommended in a review. If the studies give very different results then a summary measure will hide the different results and will be meaningless. It is possible to do a statistical test to measure the level of heterogeneity, although this test is not very powerful. In some situations, the reviewer might decide that a summary measure is not indicated because of important differences between the individual studies. However, these judgements may vary from reviewer to reviewer and might explain differences that are sometimes found between different reviews of the same research.

Interpreting data that has not been combined

A common pitfall in interpreting data is counting positive studies. For example, if none of the contributing studies is statistically significant, this is commonly interpreted as evidence of no effect. However, if the results were to be combined, the overall pooled estimate could be statistically significant. It is always important to look at the level of the effect in each contributory study. In situations when pooling is not advisable the results need to be discussed in a way that avoids counting positive studies.

Fixed effect versus random models for combining data

Broadly, there are two models for pooling study results. Fixed effect models weight the summary statistics according

to some attribute, usually size, of the contributing studies (inverse variance, Mantel-Haenszel and Peto). Random effects models include an estimate of between-study variation in the estimate of the overall effect (DerSimonian and Laird). There is no consensus on which model is better

and they will give the same results unless there is considerable heterogeneity, in which case the random effects model will give a more conservative estimate of effect size.5

Sensitivity analyses

Sensitivity analyses aim to evaluate the extent to which the results of the review are dependent upon key assumptions. Usually this involves excluding or including a group of trials of lesser quality.

Subgroup analyses

Subgroup analyses are subject to bias. For this reason it is recommended that the subgroups of interest be set out before the review is started. Post hoc analyses are particularly difficult to interpret because significant differences may have arisen due to chance. However, some would argue that one role of the systematic review is to explore possible differences of effect in different populations.

Statistical issues relating to eyes

There are particular statistical issues in randomised controlled trials relating to eyes because each individual has two eyes and the outcome of the two eyes is related, i.e. not statistically independent. As statistical methods assume statistical independence, in general, the analysis of studies where the results from both eyes are included is not straightforward. An analogous situation is the case of

cluster-randomised trials, where the extra variation introduced by the clustering needs to be taken into account when calculating the confidence intervals. Currently the methods for dealing with the meta-analysis of such data are not well developed. One strategy commonly used in surgical trials is to apply the intervention to one eye only, in which case the problem does not arise.

Summary

The statistical pooling (meta-analysis) of study results forms a very small part of the process of synthesising the evidence. It reduces the effect of random error and therefore improves the precision of the estimate of the effect of an intervention. The value of systematic reviews lies in the efforts put into locating and appraising all the available evidence, in order to reduce the effects of bias.

References

1.Davidoff F, Haynes B, Sackett D, Smith R. Evidence-based medicine. BMJ 1995;310:1085–6.

2.Egger M, Smith GD. Principles of and procedures for systematic reviews. In: Egger M, Smith GD, Altman DG, eds. Systematic Reviews in Health Care: Meta-analysis in context. London: BMJ Publishing Group, 2001, pp. 23–42.

3.Moher D, Pham B, Jones A et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352:609–13.

4.Berlin JA. Does blinding of readers affect the results of meta-analyses? Lancet 1998;350:185–6.

5.Deeks JJ, Altman DG, Bradburn MJ. Statistical methods for examining heterogeneity and combining results from several studies in metaanalysis. In: Egger M, Smith GD, Altman DG, eds. Systematic Reviews in Health Care: Meta-analysis in context. London: BMJ Publishing Group, 2001, pp. 285–312.