October 2008, Volume 58, Issue 10

Review Articles

Systematic Reviews and their role in Evidence - Informed Health Care

Zaigham Abbas  ( Associate Professor Gastroenterology and Hepatology, SIUT )
Sajjad Raza  ( Dow Medical College, DUHS )
Kiran Ejaz  ( Aga Khan University, Karachi, Pakistan. )


 Systematic reviews are defined as convenient evidential summaries for clinicians which serve as a powerful tool for patient care decisions. They are considered to be the basis for guidelines of medical practice suggesting directions for new research. These scientific reviews, constructed through well-defined methods, have key role in Evidence-Informed Health Care which means to apply new therapies and treatments, which are proved to be effective for a particular treatment by authentic and quality researches, in clinical practice.
The art of medicine requires for practice a very sound and up to date clinical knowledge by means of which clinicians can conquer the different challenges. However, a clinician can not depend on the result of a single study for making decisions in clinical practice. The solution to this dilemma is systematic reviews as they provide a precise summary of a topic and have many advantages over narrative reviews. Meta-analysis is a technique which plays an important role in increasing the precision of systematic reviews. Steps of preparation of a systematic review include:
1- Defining the review question:
2- Search and selection of studies:
3- Critical evaluation of studies for biases:
4- Data Collection:
5- Data analysis and presentation of results:
6- Interpretation of results and drawing conclusions:
7- Improving and updating reviews:
A reader should first be able to evaluate the quality of a systematic review before practicing the evidence provided by it as it helps to practice the provided evidence more effectively.


The latter half of the 20th century shows an outbreak of biomedical research publications making it impossible for the clinicians to be updated on the topics of interest. However, clinicians and medical practitioners need to have a very sound and up to date clinical knowledge in order to face the new challenges in medical practice. Firstly, in context of the large number of medical publications it is almost impossible for a physician to read those articles for latest updates. Secondly, a physician cannot depend on the result of a single study for making decisions in clinical practice as they may have certain biases and the results may also not be clear.
Thus in order to have a bigger view of the picture, a clinician needs to have strong evidence to make decisions. Here comes the need of reviews that may summarize the results of different studies on a topic to provide a clear, authentic and practicable result.
Systematic reviews:
In the last decade at least 500-fold increase in the number of systematic reviews published annually has been observed.1 Systematic reviews are defined as convenient evidential summaries for clinicians which serve as a powerful tool for patient care decisions.2,3 Thus they are considered to be the basis for guidelines of medical practice suggesting directions for new research.2
As Cooper and Hedges point out that the systematic reviews,
“attempt to discover the consistencies and account for the variability in similar-appearing studies”.4
For example if a study shows that the drug X is effective for the treatment of a particular disease and another research shows that the drug Y is also effective against the same disease, the physician becomes confused that which one is more effective. However, if we conduct a systematic review of different studies to see which one is more effective, then the result provided by systematic review will provide a better option to the physician for practice.5
The growing number of publications on different topics is the cause of publication of more systematic reviews on a particular topic. Figure 1 shows growing number of systematic reviews as indicated by the MEDLINE search results for the word systematic review at the website www.pubmed.gov. PubMed is a service of the U.S. National Library of Medicine.[(f1)]
Qualitative systematic review: A systematic review is termed as a qualitative systematic review if the results of the primary studies included are summarized without being combined statistically.1
Quantitative systematic review: A systematic review is termed as a quantitative systematic review if the results of the primary studies are statistically combined. This is also called as meta-analysis.1,6,7 Figure 2 shows the increasing use of meta-analysis in systematic reviews.
Importance of Evidence - Informed Health Care:
"Evidence - Informed Health care" (EIHC) or "Evidence Based Medicine" (EBM) means to apply new therapies and treatments, which are proved to be effective for a particular treatment by authentic and quality researches, in clinical practice. According to the report of a Nordic workshop,
"Evidence-based healthcare is the conscientious use of current best evidence in making decisions about the care of individual patients or the delivery of health services. Current best evidence is up-to-date information from relevant, valid research about the effects of different forms of health care, the potential for harm from exposure to particular agents, the accuracy of diagnostic tests, and the predictive power of prognostic factors".8[(f2)]
However, a clinician cannot depend on the result of a single study for applying a new therapy. The solution to this dilemma is what we call "Systematic Reviews".
In the context of rapidly changing healthcare system, it is very important for a health care practitioner to be aware of the latest developments and advancements so that he is able to suggest the best available treatment for the disease. In a usual week, a clinician sees a variety of patients coming with different complaints and implements numerous interpersonal, technical and clinical skills. So he has to take a number of decisions using his present knowledge. Complex are the factors that affect these decisions.1
It is seen that in order to prevent inapt care, there is a growing trend in clinical practice to use the practice guidelines. This helps to make the use of health care resources more effective and thus controlling the regional differences in practice patterns.9
Latest, authentic and useful evidence provided by the medical literature should be used as an important component in decision making during medical practice.1 If a systematic review shows that a particular drug is better than the other for a particular treatment, it means that the author has reviewed all the relevant primary studies and then given a final statement. So, clinicians can rely on the results provided by the systematic reviews.
It is seen that small trials fail to provide definitive answers to many questions in clinical research. Hence small trials are also considered misleading.10 Small trials are usually conducted at a single centre and hence the regional differences of the results are also limitised.10 Another drawback of small trials is that they miss some of the treatment effects that are important clinically.11 It is also noteworthy that the results of large trials also usually fail to provide definitive answers.10 Secondly, sometimes the results of very large trials produce contradictory results confusing the reader.12 Hence the systematic reviews in which the combination of high-quality evidences can be seen are the best solution for this dilemma.10 Thus one can say that properly conducted systematic reviews provide the best quality evidence for decision making in clinical practice.13,14
Advantages of Systematic reviews:
Systematic reviews provide a precise summary of a topic and have many advantages over narrative reviews.3,15
 * In systematic reviews, one can quickly assimilate a large amount of  information on the relevant topic.
 * Bias can be limited in the detection and rejection of studies by the use of plain, precise and unequivocal methods in systematic reviews.
 * The methods used in the systematic reviews play an important role in producing accurate and authentic conclusions.
 * The consistency and generalisibility of the results can be created by the proper and formal comparison of results of different studies involved in the systematic review.
 * In systematic reviews, new hypothesis can be established about particular sub groups and causes of inconsistency (heterogeneity) in the selected studies can be recognized. The systematic reviews play a significant role in answering the new questions which individual studies fail to answer.
 * Results of different studies can be formally compared to establish generalisibility of findings and consistency (lack of heterogeneity) of results.
 * Systematic reviews may play an important role in reduction of the delay between the new discoveries in research and the execution of efficient therapeutic and diagnostic approaches and point out the areas in which further research is needed.
The following are advantages of using meta-analysis in systematic reviews
 * The accuracy and precision of over all result can be increased by the use of meta-analysis in systematic reviews
 * It increases the arithmetical power of comparison of different studies included in systematic reviews.
 * The evaluation and estimation of the effectiveness of a particular therapy can be made better.
 * The results of various studies that are different can be united.
 * By the use of meta-analysis, various subgroups of subjects and trends can be effectively analyzed to improve the quality of a systematic review.
Weaknesses of Systematic reviews:
Inspite of the fact that systematic reviews are considered to be the best evidence for decision making in clinical practice it has got some weaknesses and pitfalls16 as well which need to be removed. According to a research, in which 300 studies were reviewed, shows that all systematic reviews are not reliable and inconsistency is observed in their reporting quality. An improvement in this condition can be seen by more universally agreed upon evidence-based guidelines. Hence there shouldn't be acceptance of systematic reviews uncritically by the readers.17
A study by the same research group evaluated the time period by which the systematic reviews become out dated.18 The results show that out of the 100 reviewed guidelines, 7% of the systematic reviews required updating at the time of their publication, 4% were in need to be updated within a year and 11% needed updating after the time period of two years.18
It is noteworthy that even all of the scientists are not influenced by the opinions in favour of systematic reviews. A psychologist, Eysenck says,
“Meta-analyses are often used to recover something from poorly designed studies, studies of insignificant statistical power, studies that give erratic results and those resulting in apparent contradictions”.19
He further states,
“If a medical treatment has an effect so recondite and obscure as to require meta-analysis to establish it, I would not be happy to have it used on me. It would seem better to improve treatment, and the theory underlying the treatment”.19
In the scientific world of today, there are two particular dilemmas used to enlighten the drawbacks of systematic reviews
1) There are certain conditions in which the results of systematic reviews conflict with that of large clinical trials.20
2) Sometimes the results of two systematic reviews on the same topic are found to be opposing.21
However in spite of the above mentioned drawbacks, systematic reviews are considered to be the best form of evidence for the clinical practice13,14 and these drawbacks can be reduced to some extent by eliminating the reasons causing them.
Reviews other than systematic Reviews:
Reviews other than the systematic reviews that perform research synthesis include4:
1. Narrative reviews.
2. Vote counting Reviews.
Narrative reviews: Narrative reviews are the traditional and simplest forms of research synthesis that provide a qualitative review of the literature.
Vote Counting Reviews: Vote counting reviews as the name implies are the forms of research synthesis which tend to be cumulative in nature. They combine the results of a collection of studies on a relevant topic pointing out that out of the total available collection how many of the results are statistically important in one direction, how many are neutral and how many are significantly pointing in other direction. The group or category having the highest votes or counts is considered to be effective.
Some of the major differences between systematic and narrative reviews that highlight the significance of systematic reviews in evidence informed health care are summarized in Table-1.1,4,22,23
Steps for Preparation of Systematic Reviews:
Systematic reviews are well structured scientific methods which have a key role in Evidence-informed Health Care. The following are the steps used in the construction of a systematic review:24-26
1. Defining the review question.
2. Search and selection of studies.
3. Critical evaluation of studies for biases.
4. Data Collection.
5. Data Analysis and presentation of results.
6. Interpretation of results and drawing conclusions.
7. Improving and updating reviews.[(t1)]
1- Defining the review question:
The focus of systematic reviews is determined by clearly formulated question or problem. This is because the inclusion and exclusion criteria for the study, locating and selecting the studies or data for the review, critical evaluation of the validity of the relevant studies and the analysis of different variations among the results of the included studies, all depend on formulating a particular question or problem for which the answer or solution is needed.
2- Search and selection of studies:
Search for the relevant literature is an essential step in the construction of a systematic review.27 It is estimated that perhaps more than 2 million articles and 20,000 biomedical journals are published each year.25 A thorough search of both published and unpublished data is needed to locate and search the studies of interest28 as one of the main differences between a systematic and traditional review is comprehensive search of relevant studies. Also the selection of studies is criterion-based to prevent selection bias in the review. The prevention of publication bias can be done by reviewing only the leading journals which provide an over optimistic view of the effectiveness of a particular therapy. As Greenhalgh mentioned in one of his papers29,
"I’ve conducted meta-analyses…may be biased owing to exclusion of relevant studies or inclusion of inadequate studies."
Selection of good quality studies is the feature that contributes significantly towards the quality of the systematic reviews. Since systematic reviews produce results which have importance in EIHC, the selection of studies should be criterion based. Preferentially, randomized clinical trials (RCTS) are included in the review. Masking (knowledge of intervention assignments) is also an important feature which should be looked for in the studies included.
3- Critical evaluation of studies for biases:
An ideal systematic review should minimize the bias and maximize precision.30 Thus critical evaluation of the searched studies is needed in which each study has to be evaluated for its eligibility, quality and reported findings for preventing biases in the review. A bias is defined as a systematic error in results. Table 2 summarizes certain types of biases and their brief explanation.24
4- Data Collection:
Data collection is considered to be a bridge between the report of a primary investigator and the report of the reviewer. The three common sources of data collection include
1. Data which is extracted from published studies.
2. Data which is obtained by corresponding with the investigators.
3. Data of the individual participant which can be obtained as a result of joint venture with the investigators.
As Mulrow and his colleagues have highlighted:
"Of course, the concept of reviews….is not new. Preparation of reviews has traditionally depended on implicit, idiosyncratic methods of data collection and interpretation".31
5- Data analysis and presentation of results:
The next step is to analyze, combine and present the results from the individual studies. This aggregation is necessary to produce a bottom line in response to the question or problem for which the systematic review is conducted. This aggregation can be qualitative and quantitative (meta-analysis) as already mentioned in the preceding text.[(t2)]
6- Interpretation of results and drawing conclusions:
In this step, the findings that are aggregated as a result of conducting a review from the selected studies are discussed. Issues such as the heterogeneity and quality of the selected studies, possible effects of bias and the probability of applicability of the findings are addressed.
7- Improving and updating reviews:
As we already know that "the best can be the enemy of the good" there is a need to update systematic reviews with the new researches as the systematic reviews are most helpful if they are up-to-date32 and the reader may get the best available up to date knowledge form the review and the clinician may have an authentic up to date evidence to make decisions in clinical practice.
Meta-analysis; a tool to synthesize evidence:
As already mentioned, systematic reviews may or may not use meta-analysis to analyze the data. However, it plays an important role in increasing the accuracy and precision of results. Perhaps meta-analysis can be considered the best way of combining the randomized clinical trials though they are also usually used or the non-randomized clinical trials.4
It is a technique by which the findings of comparable studies in a systematic review can be aggregated.4 The results of meta-analysis may not be authentic if the included studies are small in number and dissimilar or if the data available form the studies is too limited to allow an appropriate and proper investigation of heterogeneity.26
The results of the meta-analysis are presented in somewhat standard form nowadays which can be produced by using computer software such as MetaView.29
Evaluation of Systematic Reviews:
All the systematic reviews are not unbiased and of high quality.14 A reader should first be able to evaluate the quality of a systematic review before practicing the evidence provided by it as it helps to practice the provided evidence more effectively.33 Following is the list of some of the questions which should be positively answered by good quality systematic review25,29
1 Does the review present a well defined clinical question addressed by it?
2 Was the search of the relevant literature thorough and all the appropriate databases and important sources explored?
3 Was there a proper assessment of the methodological quality?
4 Was the inclusion criteria of the studies included in the review clearly described and applied?
5 Was the assessment of the studies done by blinded or independent reviewers?
6 Was there a retrieval of the missing information by the study investigators?
7 Was there a proper evaluation for the play of chance?
8 Is there an indication of similar effects by the included studies?
9 Are the given suggestions and proposals firmly based on the quality of the presented evidence? 


On the basis of the preceding discussion one can say that systematic reviews are the research evidences upon which a clinician can rely for taking decisions in clinical practice. Thus they should be prepared carefully to prevent different types of biases and should be improved and updated whenever required.
Clinicians should take full advantage from well performed systematic reviews so that they may have better evidence for decision making in their clinical practice in order to make the treatments more effective for the betterment of mankind.


1. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis of best evidence for clinical decisions. Ann Intern Med. 1997; 126:376-80.
2. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M, et al. A systematic review identified few methods and strategies describing when and how to update systematic reviews. J Clin Epidemiol. 2007; 60:1095-1104. Epub 2007 Aug 3.
3. Feldstein DA. Clinician's guide to systematic reviews and meta-analyses. WMJ. 2005; 104:25-9.
4. The Magenta Book: guidance notes for policy evaluation and analysis. London. Government Social Research Unit, HM Treasury. Published July 2003. Updated October 2007.
5. Shojania KG, Bero LA. Taking advantage of the explosion of systematic reviews: an efficient MEDLINE search strategy. Eff Clin Pract. 2001; 4:157-62.
6. Margaliot Z, Chung KC. Systematic reviews: a primer for plastic surgery research. Plast Reconstr Surg. 2007; 120:1834-41.
7. Varandas T, Carneiro AV. Types of clinical studies. Systematic reviews. Rev Port Cardiol. 2006; 25:233-46.
8. First Annual Nordic Workshop on how to critically appraise and use evidence in decisions about healthcare, National Institute of Public Health, Oslo, Norway, 1996. available on http://www.shef.ac.uk/scharr/ir/def.html. accessed on 10 March 2008.
9. Woolf SH. Practice guidelines: a new reality in medicine. I. Recent developments. Arch Intern Med. 1990; 150:1811-8.
10. Guyatt GH, Mills EJ, Elbourne D. In the era of systematic reviews, does the size of an individual trial still matter. PLoS Med. 2008 3; 5:e4.
11. Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995; 311:485.
12. Furukawa TA, Streiner DL, Hori S. Discrepancies among megatrials. J Clin Epidemiol. 2000; 53:1193-9.
13. Haines T, McKnight L, Duku E, Perry L, Thoma A. The role of systematic reviews in clinical research and practice. Clin Plast Surg. 2008; 35:207-14.
14. Horvath AR, Pewsner D. Systematic reviews in laboratory medicine: principles, processes and practical considerations. Clin Chim Acta. 2004; 342:23-39.
15. Chalmers I, Altman DG, eds. Systematic reviews. London: BMJ Publishing Group, 1995.
16. Farquhar C, Vail A. Pitfalls in systematic reviews. Curr Opin Obstet Gynecol. 2006; 18:433-9.
17. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. : PLoS Med. 2007; 27:4:e78.
18. Shojania KG, Sampson M, Ansari MT, Ji J, Doucette S, Moher D. How quickly do systematic reviews go out of date? A survival analysis. Ann Intern Med. 2007; 147:224-33. Epub 2007 Jul 16.
19. Eysenck HJ. Problems with meta-analysis. In: Chalmers I, Altman DG. Systematic reviews. London: BMJ Publishing Group, 1995:64-74.
20. LeLorier J, Grégoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med. 1997; 337:536-42.
21. Hopayian K, Mugford M. Conflicting conclusions from two systematic reviews of epidural steroid injections for sciatica: which evidence should general practitioners heed? Br J Gen Pract. 1999; 49:57-61.
22. Collins JA, Fauser BC. Balancing the strengths of systematic and narrative reviews. Hum Reprod Update. 2005; 11:103-4.
23. Montori VM, Swiontkowski MF, Cook DJ. Methodologic issues in systematic reviews and meta-analyses. Clin Orthop Relat Res. 2003; 413:45-54.
24. Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.0 [updated February 2008]. The Cochrane Collaboration, 2008. Available from www.cochrane-handbook.org. accessed on 10 March 2008.
25. Davies HTO, Crombie KI. What is a systematic review? Volume 1, number 5. Published by Hayward Medical Communications, a division of Hayward Group plc. May 2001. www.evidence-based-medicine.co.uk. accessed on 10 March 2008.
26. Little J, Higgins JPT (editors). The HuGENet™ HuGE Review Handbook, version 1.0. http://www.hugenet.ca (accessed 28 February 2008).
27. Guba B. Systematic literature search. Wien Med Wochenschr. 2008; 158:62-9.
28. Hopewell S, Clarke M, Lefebvre C, Scherer R. Handsearching versus electronic searching to identify reports of randomized trials. Cochrane Database Syst Rev. 2007; 18; MR 00000
29. Greenhalgh T. Papers that summarise other papers (systematic reviews and meta-analyses). BMJ. 1997; 315:672-5.
30. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998; 280:278-80.
31. Mulrow CD, Cook DJ, Davidoff F. Systematic reviews: critical links in the great chain of evidence. Ann Intern Med. 1997; 126:389-91.
32. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M, et al. When and how to update systematic reviews. Cochrane Database Syst Rev. 2008; 23.
33. Chan KS, Morton SC, Shekelle PG. Systematic reviews for evidence-based management: how to find them and what to do with them. Am J Manag Care. 2004; 10:806-12.
34. Stevens KR. Systematic reviews: the heart of evidence-based practice. AACN Clin Issues. 2001; 12:529-38.
35. Crowther MA, Cook DJ. Trials and tribulations of systematic reviews and meta-analyses. Hematology Am Soc Hematol Educ Program. 2007; 493-7.

Journal of the Pakistan Medical Association has agreed to receive and publish manuscripts in accordance with the principles of the following committees: