Preface

In an ideal world, every clinician would have easy access to up to date information on the safety and effectiveness of all treatments offered to their patients. But the volume of medical literature is immense, and growing rapidly. Therefore a systematic and unbiased method for summarising and disseminating the evidence is necessary. Hence the efforts of the international Cochrane Collaboration and its Database of Systematic Reviews and others such as the Centre for Reviews and Dissemination in York, UK and the Agency for Healthcare Research and Quality in the US.

Ophthalmology has a long way to go before the implementation of evidence-based practice is established but in this, ophthalmology is similar to many other surgically dominated subjects. There are many who vociferously reject its principles, particularly in the valuation of experimental versus experiential dogma. The traditional practice of medicine is based on the individual, and opinion,

but modern practice requires a more solid foundation. In order to form opinions and make judgements, clinicians require access to reliable sources of evidence.

This book does not describe the whole of the evidence base and the chapters do not have the rigour of systematic reviews but they illustrate where reviews have been conducted, which trials have been identified, and, perhaps more important, where the gaps in the evidence exist. It is inevitable that by the time this text is published, it will be out of date. Despite this, there is a wealth of information that has the potential to change our practice, not just because trial findings contradict what is common practice, but because it might help generate a more inquisitive approach to our routine clinical activity.

Richard Wormald

Liam Smeeth

Introduction

For more than a decade healthcare practitioners have been discussing the merits of evidence-based medicine. But what is evidence? Most use this term to mean the best available research evidence. And when one wants to know how well a particular diagnostic, therapeutic or preventive intervention works, the preferred type of research is that using randomised controlled trials (RCTs). The Canadian Task Force on Preventive Health Care, which formally introduced the concept of “levels of evidence”, has assigned RCTs the highest level of evidence (see Tables 1.1 and 1.2).1 Other types of research evidence, for example, controlled observational studies and case series, are considered to be at a lower level, and expert opinion is lowest of all. What this means is that reviewers should give the greatest weight to evidence from RCTs and progressively less to studies that are likely to produce less valid or reliable results.

Systematic reviews, even when they include only randomised controlled trials, are a type of observational

study. Instead of studying human participants, however, the systematic review includes research studies. As in human studies, if the population gathered for study is biased or not generalisable, the integrity of the entire project is brought into question. Having a predefined, well-constructed, comprehensive search for eligible studies is one of the ways in which a systematic review differs from a traditional narrative review.

One of the most challenging aspects of performing a search for studies for a systematic review is the requirement for comprehensiveness. If we could assume that identified studies were a random sample of all relevant studies, the absence of unidentified studies would not cause a problem. Unfortunately, evidence indicates that those studies with the potential for being missed by a standard search (for example, studies that are unpublished) are systematically different from those that are more likely to be found.

Thus, an entire area of research has grown up around how best to identify studies, and in particular clinical trials, for systematic reviews. Along the way we have learned

some important facts: a comprehensive search to identify published research is surprisingly hard to do, and a search for unpublished studies is even harder. This chapter will outline what we know about both of these challenges and how to use what we know to ensure comprehensive reviews.

Searching for published studies

Electronic databases

Anyone who has embarked on research in a new area knows that it is not easy to gather “from scratch” all of the state-of-the-art literature. Electronic searching of bibliographic databases is used more and more for searching the extant literature, in part because it is easier than going to a library, but also because, over time, it is increasingly what we know best. What is not generally known is just how much of the literature each of these databases covers, and what might be missed if searches are limited so that they produce manageable numbers of citations.

There are probably hundreds or even thousands of bibliographic databases available for searching. Prominent English language databases include MEDLINE, EMBASE, Healthstar, CINAHL (Cumulative Index of Nursing and Allied Health Literature), PsycLIT, CancerLIT, BIOSIS, AMI (Australasian Medical Index), SciSearch and Dissertation Abstracts. In addition, there are databases for the Japanese literature, the Latin-American literature, the Chinese literature and many others. These databases each only cover a fraction of the literature, and sometimes there are large overlaps in coverage. Well over 20,000 biomedical periodicals are being published around the world, yet MEDLINE, one of the primary bibliographic resources, includes only about 4600 of these.2

While journals from a diverse group of countries and representing numerous languages are included in MEDLINE, the majority are US-based and in English. This would not be a problem if one could assume that publishing in English or in a US journal was a random decision. Egger has shown that at least for investigators in German-speaking countries, there is a tendency to publish statistically significant positive results in international English language journals and negative results in German language journals.3 Furthermore, there is reasonable evidence that journals tend to publish articles from their home countries,4 although there is no direct evidence as yet that there is an author or editorial bias favouring submission or acceptance of articles from a home country over others. Thus, limiting a search to articles in English or in journals accessible through the investigator’s library may introduce bias to a review.

Limiting a search to a single or a few databases may result in a failure to identify any publications if the topic is not covered well by the journals in the database. For example, studies of complementary and alternative medicine have only recently been considered part of mainstream US medicine, and represent a small proportion of articles in MEDLINE-indexed journals.5

Even when studies are included in MEDLINE, they may be difficult to find. Most investigators know that a broad search using a few terms may be more likely to find most of the relevant records, but at the same time it may also retrieve thousands of citations. Use of additional indexing terms that narrow the search and reduce the number of retrieved citations runs the risk of missing relevant articles. Studies in the 1980s and early 1990s indicated that it was particularly difficult at that time to identify reports of RCTs in MEDLINE.6

The Cochrane Collaboration is an international organisation that aims to help people make well-informed decisions about health care by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of healthcare interventions. It has developed methods to overcome some of the difficulties in producing a comprehensive search. The Collaboration has developed an electronic, centralised register of studies potentially eligible for systematic reviews of healthcare interventions. This database, called CENTRAL, includes published and unpublished controlled trials, in any language, from any country, performed at any point in time.7 CENTRAL is available as part of the Cochrane Library, which is available through libraries and by subscription. As of the end of 2002, CENTRAL included over 345 378 citations to controlled clinical trials – about 120 000 more than contained in MEDLINE.

Members of the Collaboration, including review groups such as the Cochrane Eyes and Vision Group, contribute trials they have identified to CENTRAL. In turn, the review groups use CENTRAL to create their own “specialised registers” that they use to help their reviewers find the best evidence for the group’s systematic reviews. Review groups use all the methods described in this chapter, and more, to identify trials that are potentially relevant for their reviews.

Handsearching

Studies have repeatedly shown that even using multiple databases and the best search strategies, searching of electronic databases does not identify all relevant published studies for a systematic review.8,9 There are a number of reasons for this. First, not all study reports are included in at least one of the existing electronic databases. For example, conference abstracts and letters to journal editors often

contain information about trial results, but are not centrally electronically indexed. In many cases, the only way to find trial reports presented solely in abstracts is by handsearching conference proceedings.

Systematic reviews that fail to include data from conference abstracts may be excluding results that are different from the types of data appearing in full reports. Indeed, less than half (44·8%) of conference abstracts and other short reports ever reach full publication.10 Furthermore, there appears to be an association between statistically significant results and full publication (relative risk (RR) = 1·51; 95% confidence interval (CI) 1·27–1·79), although no such association was detected between direction of results (i.e. favouring the experimental arm) and full publication (RR = 0·97; 95% CI 0·81–1·18). In addition, McAuley has shown that results presented in the “grey” literature, including conference abstracts, tend more often to be “negative” than results published in full journal reports.11

A second reason why searching of electronic databases tends not to identify all relevant studies for a systematic review is that most electronic databases are not indexed with systematic reviewers in mind, that is, indexing is not done to ensure comprehensiveness of a search. And, although there are almost always explicit rules for application of the index terms, these rules are not always followed by the indexers. Studies of MEDLINE and EMBASE searching conducted in the past have shown that on average MEDLINE searches identify 77% of the randomised trials that they index,12 and in one study at least EMBASE identified 85% of controlled trials.9 Other databases almost certainly have similar or worse problems. The percentage of total relevant citations retrieved by MEDLINE is probably improved today because of newer indexing terms that became available in the 1990s and a massive “retagging” effort the National Library of Medicine has undertaken in collaboration with the United States in Cochrane Center Providence, to ensure that all citations in the MEDLINE database are assigned the newer terms.7 Still, recent studies have shown that even with these improvements, an additional handsearch of the literature is warranted.13

Thus, the Cochrane Collaboration has elected to handsearch journals, conference proceedings and other documents to ensure comprehensive systematic reviews. As of March 2001, over 2100 periodicals have been or are being handsearched, page by page, from 1948 onward.7 This handsearching has contributed to CENTRAL citations to thousands of abstracts that might otherwise not be accessible through electronic means. On average, as of Issue 1 2001 of the Cochrane Library, 10% of the trials in Cochrane reviews are referenced to abstracts, dissertations or other “grey” sources only.14

Perusal of reference lists

Perusal of reference lists is a common method used to search for relevant studies for systematic reviews. Used less often, SciSearch, the Institute for Scientific Information’s electronic database for citation searching, provides a similar opportunity.15 The major problem with either approach is that citation is subjective and may be biased. Several studies have shown a tendency of investigators to cite articles supporting effectiveness of an intervention,16,17 or conversely, articles failing to support effectiveness.18 This is not to say that reference lists should not be searched, rather one should not consider that an electronic search supplemented by a search of reference lists would lead to comprehensive identification of all relevant work.

Searching for unpublished studies

Up to now, we have assumed that all important research can be accessed through the published literature, but this is not true. A systematic review of studies following all initiated research approved by local ethics committees showed that 25–50% of all initiated studies are never published in full.19

Follow up of clinical trials funded by the National Institutes of Health showed that a lower percentage remained unpublished (only 7%).20

Failure to publish is a problem because, as several studies have now shown, publication is associated with positive or statistically significant study outcomes. That is, there is a publication bias such that investigators are less likely to publish study findings if the results are “negative”.19 Those who perform systematic reviews without making an effort to identify unpublished data thus run the risk of overestimating a beneficial effect.21 Publication bias is not only a matter of selective publication of entire research studies, but also relates to selective publication of outcomes. We currently have no estimates of how often this happens, but suffice to say that the tendency for preferential reporting of statistically significant results in journal articles contributes to the problem.

Good methods for identifying unpublished studies and outcomes have not been developed. There is some evidence that publication bias may be associated with industry funding sources,22 and that information of this type may be particularly hard to come by.23 Cochrane reviewers have specifically reported on the difficulties of obtaining unpublished information for their reviews, regardless of funding source.24,25 Hetherington and colleagues tried writing to 42 000 perinatal researchers internationally to identify unpublished trials conducted in the past, but this effort was essentially unfruitful.26

There is widespread agreement that the best way to ensure comprehensive identification of clinical trials,

regardless of funding source, is through prospective registration at study initiation. Though progress has been made in this area,27 coverage tends to favour government supported trials, with industry trials less well identified.28

The Cochrane Collaboration is only beginning to address how best to identify and include all trials in its databases, regardless of publication status. Until a comprehensive register of all initiated studies exists, individual reviewers will need to assess the likelihood that publication bias could influence their findings.

Conclusion

It is perhaps presumptuous to think we can have truly evidence-based medicine if finding the evidence is so difficult. The Cochrane Collaboration has made a concerted effort to ensure that resources are available to assist with the process, but some problems remain, especially those related to identification of unpublished results. Interestingly, very little parallel effort has taken place related to ensuring complete identification of the evidence for systematic reviews of observational and epidemiological data.29 This fact lends further support to the decision of the Cochrane Collaboration to focus on summarising evidence on interventions using the highest possible evidence, preferably randomised controlled trials.

References

1.Canadian Task Force on Preventive Health Care. http://www. ctfphc.org, accessed on 26 May 2003.

2.US National Library of Medicine MEDLINE Fact Sheet, 2000. http:// www.nlm.nih.gov/pubs/factsheets/medline.html, accessed on 26 May 2003.

3.Egger M, Zellweger-Zahner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997;350:326–9.

4.Joyce J, Rabe Hesketh S, Wessely S. Reviewing the reviews: the example of chronic fatigue syndrome. JAMA 1998;280:264–6.

5.Barnes J, Abbot NC, Harkness EF, Ernst E. Articles on complementary medicine in the mainstream medical literature: an investigation of MEDLINE, 1966 through 1996. Arch Int Med 1999;159:1721–5.

6.Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994;309:1286–91.

7.Dickersin K, Manheimer E, Wieland LS et al. Development of a Centralized Register of Controlled Clinical Trials: The Cochrane Collaboration’s CENTRAL Database. Eval Health Professions 2002;25: 38–64.

8.Avenell A, Handoll HH, Grant AM. Lessons for search strategies from a systematic review, in the Cochrane Library, of nutritional supplementation trials in patients after hip fracture. Am J Clin Nutr 2001;73:505–10.

9.Suarez Almazor ME, Belseck E, Homik J, Dorgan M, Ramos Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials 2000; 21:476–87.

10.Scherer RW, Langenberg P. Full publication of results initially presented in abstracts (Cochrane Methodology Review). In: Cochrane Collaboration: Cochrane Library. Issue 1. Oxford: Update Software, 2002.

11.McAuley L, Pham B, Tugwell P, Moher D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet 2000;356:1228–31.

12.Dickersin K, Scherer R, Lefebvre C. Identification of relevant studies for systematic reviews. BMJ 1994;309:1286–91.

13.Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M. A comparison of handsearching versus electronic searching to identify reports of randomized trials. Stat Med 2002;21:1625–34.

14.Mallett S, Hopewell S, Clarke M. Grey literature in systematic reviews: The first 1000 Cochrane systematic reviews. In: 4th Symposium on Systematic Reviews: Pushing the Boundaries, 2–4 July 2002, Oxford, UK. Abstract no 5.

15.Pao ML. Perusing the literature via citation links [published erratum].

Comput Biomed Res 1993;26:143–56.

16.Gotzsche PC. Reference bias in reports of drug trials. BMJ (Clin Res Ed).1987;295:654–6.

17.Ravnskov U. Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome. BMJ 1992;305:15–19.

18.Hutchinson BG, Oxman AD, Lloyd S. Comprehensiveness and bias in reporting clinical trials. Can Fam Physician 1995;41:1356–60.

19.Dickersin K. How important is publication bias? A synthesis of available data. AIDS Educ Prev 1997;9(Suppl 1):15–21.

20.Dickersin K, Min YI. NIH clinical trials and publication bias. Online J Curr Clin Trials 1993;Doc No 50.

21.Simes RJ. Publication bias: the case for an international registry of clinical trials. J Clin Oncol 1986;4:1529–41.

22.Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337:867–72.

23.Herxheimer A. Data on file cited in pharmaceutical advertisements: What are they? The International Congress on Biomedical Peer Review and Global Communications, September 1997, Prague, Czech Republic, 1997.

24.Hadhazy V, Ezzo J, Berman B. How valuable is effort to contact authors to obtain missing data in systematic reviews? 7th Annual Cochrane Colloquium Abstracts, October 1999, Rome, Italy, 1999.

25.Walters EH, Walters JA. Many reports of RCTs give insufficient data for Cochrane reviewers. BMJ 1999;319:257.

26.Hetherington J, Dickersin K, Chalmers I, Meinert CL. Retrospective and prospective identification of unpublished controlled trials: lessons from a survey of obstetricians and pediatricians. Pediatrics 1989;84: 374–80.

27.McCray AT. Better access to information about clinical trials. Ann Intern Med 2000;133:609–14.

28.Manheimer E, Anderson D. Survey of public information about ongoing clinical trials funded by industry: evaluation of completeness and accessibility. BMJ 2002;325:528–31.

29.Dickersin K. Systematic reviews in epidemiology: Why are we so far behind? Int J Epidemiol 2002;31:6–12.