a systematic literature review medicine

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.112(1); Jan-Feb 2015

Systematically Reviewing the Literature: Building the Evidence for Health Care Quality

There are important research and non-research reasons to systematically review the literature. This article describes a step-by-step process to systematically review the literature along with links to key resources. An example of a graduate program using systematic literature reviews to link research and quality improvement practices is also provided.

Introduction

Systematic reviews that summarize the available information on a topic are an important part of evidence-based health care. There are both research and non-research reasons for undertaking a literature review. It is important to systematically review the literature when one would like to justify the need for a study, to update personal knowledge and practice, to evaluate current practices, to develop and update guidelines for practice, and to develop work related policies. 1 A systematic review draws upon the best health services research principles and methods to address: What is the state of the evidence on the selected topic? The systematic process enables others to reproduce the methods and to make a rational determination of whether to accept the results of the review. An abundance of articles on systematic reviews exist focusing on different aspects of systematic reviews. 2 – 9 The purpose of this article is to describe a step by step process of systematically reviewing the health care literature and provide links to key resources.

Systematic Review Process: Six Key Steps

Six key steps to systematically review the literature are outlined in Table 1 and discussed here.

Systematic Review Steps

1. Formulate the Question and Refine the Topic

When preparing a topic to conduct a systematic review, it is important to ask at the outset, “What exactly am I looking for?” Hopefully it seems like an obvious step, but explicitly writing a one or two sentence statement of the topic before you begin to search is often overlooked. It is important for several reasons; in particular because, although we usually think we know what we are searching for, in truth our mental image of a topic is often quite fuzzy. The act of writing something concise and intelligible to a reader, even if you are the only one who will read it, clarifies your thoughts and can inspire you to ask key questions. In addition, in subsequent steps of the review process, when you begin to develop a strategy for searching the literature, your topic statement is the ready raw material from which you can extract the key concepts and terminology for your strategies. The medical and related health literature is massive, so the more precise and specific your understanding of your information need, the better your results will be when you search.

2. Search, Retrieve, and Select Relevant Articles

The retrieval tools chosen to search the literature should be determined by the purpose of the search. Questions to ask include: For what and by whom will the information be used? A topical expert or a novice? Am I looking for a simple fact? A comprehensive overview on the topic? Exploration of a new topic? A systematic review? For the purpose of a systematic review of journal research in the area of health care, PubMed or Medline is the most appropriate retrieval tool to start with, however other databases may be useful ( Table 2 ). In particular, Google Scholar allows one to search the same set of articles as PubMed/MEDLINE, in addition to some from other disciplines, but it lacks a number of key advanced search features that a skilled searcher can exploit in PubMed/MEDLINE.

Examples of Electronic Bibliographic Databases Specific to Health Care

Note: These databases may be available through university or hospital library systems.

An effective way to search the literature is to break the topic into different “building blocks.” The building blocks approach is the most systematic and works the best in periodical databases such as PubMed/MEDLINE. The “blocks” in a “building blocks” strategy consist of the key concepts in the search topic. For example, let’s say we are interested in researching about mobile phone-based interventions for monitoring of patient status or disease management. We could break the topic into the following concepts or blocks: 1. Mobile phones, 2. patient monitoring, and 3. Disease management. Gather synonyms and related terms to represent each concept and match to available subject headings in databases that offer them. Organize the resulting concepts into individual queries. Run the queries and examine your results to find relevant items and suggest query modifications to improve your results. Revise and re-run your strategy based on your observations. Repeat this process until you are satisfied or further modifications produce no improvements. For example in Medline, these terms would be used in this search and combined as follows: cellular phone AND (ambulatory monitoring OR disease management), where each of the key word phrases is an official subject heading in the MEDLINE vocabulary. Keep detailed notes on the literature search, as it will need to be reported in the methods section of the systematic review paper. Careful noting of search strategies also allows you to revisit a topic in the future and confidently replicate the same results, with the addition of those subsequently published on your topic.

3. Assess Quality

There is no consensus on the best way to assess study quality. Many quality assessment tools include issues such as: appropriateness of study design to the research objective, risk of bias, generalizability, statistical issues, quality of the intervention, and quality of reporting. Reporting guidelines for most literature types are available at the EQUATOR Network website ( http://www.equator-network.org/ ). These guidelines are a useful starting point; however they should not be used for assessing study quality.

4. Extract Data and Information

Extract information from each eligible article into a standardized format to permit the findings to be summarized. This will involve building one or more tables. When making tables each row should represent an article and each column a variable. Not all of the information that is extracted into the tables will end up in the paper. All of the information that is extracted from the eligible articles will help you obtain an overview of the topic, however you will want to reserve the use of tables in the literature review paper for the more complex information. All tables should be introduced and discussed in the narrative of the literature review. An example of an evidence summary table is presented in Table 3 .

Example of an evidence summary table

Notes: BP = blood pressure, HbA1c = Hemoglobin A1c, Hypo = hypoglycemic, I = Internet, NS = not significant, PDA = personal digital assistant, QOL = quality of life, SMBG = self-monitored blood glucose, SMS = short message service, V = voice

5. Analyze and Synthesize Data and information

The findings from individual studies are analyzed and synthesized so that the overall effectiveness of the intervention can be determined. It should also be observed at this time if the effect of an intervention is comparable in different studies, participants, and settings.

6. Write the Systematic Review

The PRISMA 12 and ENTREQ 13 checklists can be useful resources when writing a systematic review. These uniform reporting tools focus on how to write coherent and comprehensive reviews that facilitate readers and reviewers in evaluating the relative strengths and weaknesses. A systematic literature review has the same structure as an original research article:

TITLE : The systematic review title should indicate the content. The title should reflect the research question, however it should be a statement and not a question. The research question and the title should have similar key words.

STRUCTURED ABSTRACT: The structured abstract recaps the background, methods, results and conclusion in usually 250 words or less.

INTRODUCTION: The introduction summarizes the topic or problem and specifies the practical significance for the systematic review. The first paragraph or two of the paper should capture the attention of the reader. It might be dramatic, statistical, or descriptive, but above all, it should be interesting and very relevant to the research question. The topic or problem is linked with earlier research through previous attempts to solve the problem. Gaps in the literature regarding research and practice should also be noted. The final sentence of the introduction should clearly state the purpose of the systematic review.

METHODS: The methods provide a specification of the study protocol with enough information so that others can reproduce the results. It is important to include information on the:

• Eligibility criteria for studies: Who are the patients or subjects? What are the study characteristics, interventions, and outcomes? Were there language restrictions?
• Literature search: What databases were searched? Which key search terms were used? Which years were searched?
• Study selection: What was the study selection method? Was the title screened first, followed by the abstract, and finally the full text of the article?
• Data extraction: What data and information will be extracted from the articles?
• Data analysis: What are the statistical methods for handling any quantitative data?

RESULTS: The results should also be well-organized. One way to approach the results is to include information on the:

• Search results: What are the numbers of articles identified, excluded, and ultimately eligible?
• Study characteristics: What are the type and number of subjects? What are the methodological features of the studies?
• Study quality score: What is the overall quality of included studies? Does the quality of the included studies affect the outcome of the results?
• Results of the study: What are the overall results and outcomes? Could the literature be divided into themes or categories?

DISCUSSION: The discussion begins with a nonnumeric summary of the results. Next, gaps in the literature as well as limitations of the included articles are discussed with respect to the impact that they have on the reliability of the results. The final paragraph provides conclusions as well as implications for future research and current practice. For example, questions for future research on this topic are revealed, as well as whether or not practice should change as a result of the review.

REFERENCES: A complete bibliographical list of all journal articles, reports, books, and other media referred to in the systematic review should be included at the end of the paper. Referencing software can facilitate the compilation of citations and is useful in terms of ensuring the reference list is accurate and complete.

The following resources may be helpful when writing a systematic review:

CEBM: Centre for Evidence-based Medicine. Dedicated to the practice, teaching and dissemination of high quality evidence based medicine to improve health care Available at: http://www.cebm.net/ .

CITING MEDICINE: The National Library of Medicine Style Guide for Authors, Editors, and Publishers. This resource provides guidance in compiling, revising, formatting, and setting reference standards. Available at http://www.ncbi.nlm.nih.gov/books/NBK7265/ .

EQUATOR NETWORK: Enhancing the QUAlity and Transparency Of health Research. The EQUATOR Network promotes the transparent and accurate reporting of research studies. Available at: http://www.equator-network.org/ .

ICMJE RECOMMENDATIONS: International Committee of Medical Journal Editors Recommendations for the Conduct, Reporting, Editing and Publication of Scholarly Work in Medical Journals. The ICJME recommendations are followed by a large number of journals. Available at: http://www.icmje.org/about-icmje/faqs/icmje-recommendations/ .

PRISMA STATEMENT: Preferred Reporting Items for Systematic Reviews and Meta-Analyses. Authors can utilize the PRISMA Statement checklist to improve the reporting of systematic reviews and meta-analyses. Available at: http://prisma-statement.org .

THE COCHRANE COLLABORATION: A reliable source for making evidence generated through research useful for informing decisions about health. Available at: http://www.cochrane.org/ .

Examples of Systematic Reviews To Link Research and Quality Improvement

Over the past 17 years more than 300 learners, including physicians, nurses, and health administrators have completed a course as part of a Master of Health Administration or a Master of Science in Health Informatics degree at the University of Missouri. An objective of the course is to educate health informatics and health administration professionals about how to utilize a systematic, scientific, and evidence-based approach to literature searching, appraisal, and synthesis. Learners in the course conduct a systematic review of the literature on a health care topic of their choosing that could suggest quality improvement in their organization. Students select topics that make sense in terms of their core educational competencies and are related to their work. The categories of topics include public health, leadership, information management, health information technology, electronic medical records, telehealth, patient/clinician safety, treatment/screening evaluation cost/finance, human resources, planning and marketing, supply chain, education/training, policies and regulations, access, and satisfaction. Some learners have published their systematic literature reviews 14 – 15 . Qualitative comments from the students indicate that the course is well received and the skills learned in the course are applicable to a variety of health care settings.

Undertaking a literature review includes identification of a topic of interest, searching and retrieving the appropriate literature, assessing quality, extracting data and information, analyzing and synthesizing the findings, and writing a report. A structured step-by-step approach facilitates the development of a complete and informed literature review.

Suzanne Austin Boren, PhD, MHA, (above) is Associate Professor and Director of Academic Programs, and David Moxley, MLIS, is Clinical Instructor and Associate Director of Executive Programs. Both are in the Department of Health Management and Informatics at the University of Missouri School of Medicine.

Contact: ude.iruossim.htlaeh@snerob

None reported.

• University of Texas Libraries
• UT Libraries
• Systematic Reviews
• Dell Medical School Library
• Access our Services
• Clinical Practice Guidelines
• Clinical Trials
• Drug Information
• Health and Medical Law
• Point-of-Care Tools
• Test Prep Resources
• Video, Audio, and Images
• Get Full Text with LibKey Nomad
• What's New?
• Search Tips
• PubMed Guide This link opens in a new window
• Artificial Intelligence This link opens in a new window
• Artificial Intelligence
• Ask the Question
• Acquire the Evidence
• Appraise the Evidence
• Evidence Hierarchy
• EBM Bibliography
• Child Neurology
• Dermatology
• Emergency Medicine
• Family Medicine
• Internal Medicine
• Obstetrics & Gynecology
• Ophthalmology
• Orthopaedic Surgery
• Physical Medicine & Rehabilitation
• Mobile Apps
• Citation Managers This link opens in a new window
• Citation Manuals
• General Resources
• Study Types
• Literature Reviews
• Scoping Reviews
• Rapid Reviews
• Integrative Reviews
• Technical Reports
• Case Reports
• Getting Published
• Open Access Publishing This link opens in a new window
• Selecting a Journal
• Open Access Publishing
• Avoiding Low Quality Open Access
• High Quality Open Access Journals
• Keeping Up with the Literature
• Health Statistics
• Research Funding This link opens in a new window
• Author Metrics
• Article Metrics
• Journal Metrics
• Scholarly Profile Tools
• Health Humanities This link opens in a new window
• Health Equity This link opens in a new window
• UT-Authored Articles
• Resources for DMS COVID-19 Elective

What is a Systematic Review?

The PRISMA statement defines a systematic review as "a review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyze data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyze and summarize the results of the included studies. Meta-analysis refers to the use of statistical techniques in a systematic review to integrate the results of included studies." 1

In the Cochrane Handbook for Systematic Reviews of Interventions, the key characteristics of a systematic review are listed as:

• "a clearly stated set of objectives with pre-defined eligibility criteria for studies
• an explicit, reproducible methodology
• an assessment of the validity of the findings of the included studies, for example through the assessment of risk of bias; and
• a systematic presentation, and synthesis, of the characteristics and findings of the included studies." 2

1. Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6(7):e1000097. https://doi.org/10.1371/journal.pmed.1000097

2. higgins jpt, thomas j, chandler j, cumpston m, li t, page mj, welch va (editors). cochrane handbook for systematic reviews of interventions , version 6.1 (updated september 2020). cochrane, 2020. available from https://training.cochrane.org/handbook/current, data extraction.

• Web Resources
• Other Library Guides
• PRISMA PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. PRISMA focuses on the reporting of reviews evaluating randomized trials, but can also be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions.

• National Academies of Sciences Engineering Medicine. Finding What Works in Health Care: Standards for Systematic Reviews Knowing what works in health care is of highest importance for patients, healthcare providers, and other decision makers. The most reliable way to identify benefits and harms associated with various treatment options is a systematic review of comparative effectiveness research. . . the Institute of Medicine (IOM) undertook this study to develop a set of standards for conducting systematic reviews of comparative effectiveness research.
• Centre for Reviews and Dissemination. Systematic Reviews: CRD's Guidance for Undertaking Reviews in Health Care. "Our aim is to promote high standards in commissioning and conduct, by providing practical guidance for undertaking systematic reviews evaluating the effects of health interventions."
• Joanna Briggs Institute. JBI Reviewer's Manual. "The JBI Reviewer's Manual is designed to provide authors with a comprehensive guide to conducting JBI systematic reviews. It describes in detail the process of planning, undertaking and writing up a systematic review using JBI methods."
• Cochrane Handbook for Systematic Reviews of Interventions Cochrane is an international network of over 130 countries that is recognized as the benchmark for evidence synthesis. "It promotes evidence-informed health decision-making by producing high-quality, relevant, accessible systematic reviews and other synthesized research evidence."

Protocol Templates

• PRISMA Systematic Review Protocol Template
• Evidence Synthesis Protocol Template "This document is based on the PRISMA Statement (evidence-based minimum set of items for reporting in systematic reviews and meta-analyses) extensions for systematic review protocols and scoping reviews, and materials developed by The Campbell Collaboration"
• Systematic Review Accelerator - Methods Wizard

Guidelines to Writing a Protocol

• Guidance Notes for Registering a Systematic Review Protocol with PROSPERO This manual includes guidance on filling out the 22 required fields and 18 optional fields of the PROSPERO protocol registration form.
• A Guide to Writing a Qualitative Systematic Review Protocol to Enhance Evidence-Based Practice in Nursing and Health Care | Worldviews Evid Based Nurs. 2016
• PRISMA for systematic review protocols (PRISMA-P) aims "to facilitate the development and reporting of systematic review protocols."

Protocol Registration

Register your protocol. Several options include:

• Prospero PROSPERO accepts registrations for systematic reviews, rapid reviews and overviews/umbrella reviews. PROSPERO does not accept scoping reviews.
• Open Science Framework Registry
• Research Registry "The Research Registry is a one-stop shop for registering all types of research studies, from ‘first in man’ case reports to observational/interventional studies to systematic reviews and meta-analyses." This particular registry service charges a fee.

Search Development Tools

• PubMed Search Tester An application that can be used to construct and validate a PubMed search strategy
• Yale MeSH Analyzer Enter PubMed record ID numbers (PMIDs) to generate a chart showing the MeSH (Medical Subject Headings) terms assigned to all of the articles.
• SR-Accelerator. Polyglot This tool translate search strings across databases. It translates either a PubMed or Ovid MEDLINE search string into the following database platforms; the Cochrane Library; Embase (via Elsevier); Web of Science; Scopus; EBSCO (e.g. CINAHL); Ovid (e.g. PsycINFO or Embase via Ovid) and PubMed. One of its limitations is that it does not automatically map subject terms across databases.

Search Hedges/Filters

• CADTH Search Filters Database
• The ISSG Search Filter Resource
• UAB Libraries. Hedges
• Systematic Reviews: Tracking Results A guide from MD Anderson Cancer Center showing how to organize search results and remove duplications with EndNote.
• Systematic Review_Search Tracking A spreadsheet from MD Anderson Cancer Center for tracking search strategy, search dates, and duplicates.
• Database + EndNote = Organized Search Results A step-by-step guide to using databases (in this example, PubMed) with EndNote to organize search results
• Rathbone J, Carter M, Hoffmann T, Glasziou P. Better duplicate detection for systematic reviewers: evaluation of Systematic Review Assistant-Deduplication Module. Syst Rev. 2015;4(1):6.
• René Otten, Ralph de Vries, and Linda Schoonmade. Amsterdam Efficient Deduplication (AED) Method. 2019

Screening Tools

• Abstrackr A free online tool for the task of citation screening for systematic reviews.
• Bond University, Institute for Evidence-Based Healthcare. SR-Accelerator "The IEBH SR-Accelerator is a suite of tools to speed up steps in the Systematic Review (SR) process. It is freely available. It is a modular design which means the tools can be incorporated into existing SR workflows and combined with other automation tools."
• Cadima "CADIMA is a free web tool facilitating the conduct and assuring for the documentation of systematic reviews, systematic maps and further literature reviews."
• Colandr Free machine-learning assisted tool for screening and extraction
• Covidence Subscription-based software for citation screening, full text review, risk of bias assessment, and extraction of data for systematic reviews.
• DistillerSR A subscription-based online tool for reference screening, data extraction and reporting solutions for systematic reviews.
• EPPI-Reviewer A subscription-based tool for managing systematic reviews through all stages of the process from bibliographic management, screening, and coding to synthesis.
• Rayyan Free web application for screening and coding of studies in a systematic review.
• RevMan (free for individuals)

Reviews of Screening Tools

• Software Tools for Literature Screening in Systematic Reviews in Biomedical Research | 2019 "a feature analysis was performed to compare different reference screening tools"
• Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation | BMC Med Res Methodol. 2020 A review and comparison of 15 screening tools
• Covidence and Rayyan | J Med Libr Assoc. 2018

Data extraction is the process by which one extracts and records study characteristics and findings from each study included in the systematic review.  This information can be recorded in a spreadsheet or word processor. Or, use a software program such as Covidence, RevMan, or DistillerSR, or survey program such as Qualtrics or Google Forms.

What data, study characteristics, and information to extract from each study is unique to each review, but extracted data may include:

• Article citation and funding sources
• Study participants - demographics, characteristics of participants such as comorbidity, socio-economic status, ethnicity, etc. number of participants, recruitment procedures, details of randomization or blinding, etc.
• Study methodology - study design, objectives of study, study inclusion and exclusion criteria, recruitment procedures, randomization and blinding, if applicable, etc.
• Intervention and setting - description of the intervention, route of delivery, timing, equipment, dose, description of control group, etc.
• Outcomes and results, quantitative and/or qualitative - whether the outcome was assessed and reported, unit of measurement, method of aggregation, tool used to measure outcomes,  adverse outcomes, costs, additional relevant outcomes

Data Extraction Tools

• SRDR+ | AHRQ Developed by the Brown University Evidence-Based Practice Center, this tool is both an archive and data extraction tool. All systematic review projects are made open access when complete.

Useful Readings

• Cochrane Handbook for Systematic Reviews of Interventions. 2020. Chapter 5.3: What data to collect
• Finding What Works in Health Care: Standards for Systematic Reviews. 2011. Chapter 3: Sections: Managing Data Collection; Recommended Standards for Extracting Data
• Systematic Reviews: CDR's Guidance for Undertaking Reviews in Health Care. 2009. Section 1.3.3: Data Extraction
• PubMed articles about Extraction

A requirement of every systematic review is to appraise the quality of each included study. The tool used will depend on the study methodology used. You will see the following phrases used for this phase of the systematic review: critical appraisal, quality assessment, assessment of risk of bias. 

• Critical Appraisal Skills Programme (CASP) Offers skill training workshops, and checklists for assessing the trustworthiness, relevance and results of published papers.
• Centre for Evidence-Based Medicine (CEBM). Critical Appraisal Tools "contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples."
• Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52(6):377‐384. doi:10.1136/jech.52.6 The checklist can be found in the Appendix.
• GRADE The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group "has developed a common, sensible and transparent approach to grading quality (or certainty) of evidence and strength of recommendations."
• Joanna Briggs Institute (JBI). Critical Appraisal Tools "JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers." Includes checklists for 13 types of articles.
• NIH, National Heart, Lung, and Blood Institute. Study Quality Assessment Tools These tools assist users with critical appraisal of the internal validity of studies, including Controlled Intervention studies, Systematic Reviews, Meta-Analyses, Observational Cohort and Cross-Sectional studies, Case-Control studies, Case Series studies.
• Scottish Intercollegiate Guidelines Network. Critical Appraisal Notes and Checklists "Methodological assessment of studies selected as potential sources of evidence is based on a number of criteria that focus on those aspects of the study design that research has shown to have a significant effect on the risk of bias in the results reported and conclusions drawn. These criteria differ between study types, and a range of checklists is used to bring a degree of consistency to the assessment process."
• Mixed Methods Appraisal Tool (MMAT) The MMAT is intended to be used as a checklist for concomitantly appraising and/or describing studies included in systematic mixed studies reviews (reviews including original qualitative, quantitative and mixed methods studies). The MMAT was first published in 2009. Since then, it has been validated in several studies testing its interrater reliability, usability and content validity. The latest version of the MMAT was updated in 2018.

Risk of Bias

• Cochrane Risk-of-Bias tool for Randomized Trials RoB2 - RoB 2: A revised Cochrane risk-of-bias tool for randomized trials. It is the recommended tool to assess the risk of bias in randomized trials included in Cochrane Reviews. RoB 2 is structured into a fixed set of domains of bias, focussing on different aspects of trial design, conduct, and reporting.
• Latitudes Network This is a searchable library of validity assessment tools for use in evidence syntheses. This website also provides access to training on the process of validity assessment.

• A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research. Eur J Epidemiol. 2020
• Meta-Analysis in Clinical Research. Anesth Analg. 2020.
• Systematic Review in Clinical Research. Anesth Analg. 2020
• Developing, Conducting, and Publishing Appropriate Systematic Review and Meta-Analysis Articles | Plastic and Reconstructive Surgery. 2018
• How to do a systematic review. Int J Stroke. 2018.
• Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach | BMC Med Res Methodol. 2018
• Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review | Am J Nurs, 2014
• Systematic reviews and meta-analysis | Plastic and Reconstructive Surgery. 2011
• What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018.

The Brown University School of Public Health  Evidence Synthesis Academy has developed a series of tutorials and videos about conducting systematic reviews.  Here is the introductory video:

 According to the Cochrane  website , "Our volunteers and contributors are researchers, health professionals, patients, carers, and people passionate about improving health outcomes for everyone, everywhere. Our global independent network gathers and summarizes the best evidence from research to help you make informed choices about treatment and we have been doing this for 25 years." This video defines a systematic review:

Library Guides

• University of Texas at Austin | Systematic Reviews
• University of Alabama at Birmingham | Systematic Reviews: An Introduction
• University of Texas, Health Science Center at Houston | Systematic Review Resources.

The Systematic Review Process

• Define your research question
• Check to make sure a systematic review has not already been done on your topic. If one has been done, but it is old, contact the authors to learn whether they plan to update it
• Assemble your team. Team members may include subject specialists, a systematic review method specialist, a biostatistician (if a meta-analysis), and a librarian
• Develop your protocol and register it
• Search the literature and collect the results
• Screen titles and abstracts using your inclusion and exclusion criteria to identify potentially relevant studies
• Review the full text  of the articles chosen in step 6 to confirm eligibility 
• Appraise the quality of the eligible studies
• Extract the qualitative and/or quantitative data and synthesize
• Report the findings

Useful Articles

• Last Updated: May 24, 2024 6:13 AM
• URL: https://guides.lib.utexas.edu/medicine

• Open access
• Published: 01 August 2019

A step by step guide for conducting a systematic review and meta-analysis with simulation data

• Gehad Mohamed Tawfik 1 , 2 ,
• Kadek Agus Surya Dila 2 , 3 ,
• Muawia Yousif Fadlelmola Mohamed 2 , 4 ,
• Dao Ngoc Hien Tam 2 , 5 ,
• Nguyen Dang Kien 2 , 6 ,
• Ali Mahmoud Ahmed 2 , 7 &
• Nguyen Tien Huy 8 , 9 , 10  

Tropical Medicine and Health volume  47 , Article number:  46 ( 2019 ) Cite this article

807k Accesses

193 Citations

94 Altmetric

Metrics details

The massive abundance of studies relating to tropical medicine and health has increased strikingly over the last few decades. In the field of tropical medicine and health, a well-conducted systematic review and meta-analysis (SR/MA) is considered a feasible solution for keeping clinicians abreast of current evidence-based medicine. Understanding of SR/MA steps is of paramount importance for its conduction. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, this methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly conduct a SR/MA, in which all the steps here depicts our experience and expertise combined with the already well-known and accepted international guidance.

We suggest that all steps of SR/MA should be done independently by 2–3 reviewers’ discussion, to ensure data quality and accuracy.

SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing.

Introduction

The amount of studies published in the biomedical literature, especially tropical medicine and health, has increased strikingly over the last few decades. This massive abundance of literature makes clinical medicine increasingly complex, and knowledge from various researches is often needed to inform a particular clinical decision. However, available studies are often heterogeneous with regard to their design, operational quality, and subjects under study and may handle the research question in a different way, which adds to the complexity of evidence and conclusion synthesis [ 1 ].

Systematic review and meta-analyses (SR/MAs) have a high level of evidence as represented by the evidence-based pyramid. Therefore, a well-conducted SR/MA is considered a feasible solution in keeping health clinicians ahead regarding contemporary evidence-based medicine.

Differing from a systematic review, unsystematic narrative review tends to be descriptive, in which the authors select frequently articles based on their point of view which leads to its poor quality. A systematic review, on the other hand, is defined as a review using a systematic method to summarize evidence on questions with a detailed and comprehensive plan of study. Furthermore, despite the increasing guidelines for effectively conducting a systematic review, we found that basic steps often start from framing question, then identifying relevant work which consists of criteria development and search for articles, appraise the quality of included studies, summarize the evidence, and interpret the results [ 2 , 3 ]. However, those simple steps are not easy to be reached in reality. There are many troubles that a researcher could be struggled with which has no detailed indication.

Conducting a SR/MA in tropical medicine and health may be difficult especially for young researchers; therefore, understanding of its essential steps is crucial. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, we recommend a flow diagram (Fig. 1 ) which illustrates a detailed and step-by-step the stages for SR/MA studies. This methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly and succinctly conduct a SR/MA; all the steps here depicts our experience and expertise combined with the already well known and accepted international guidance.

Detailed flow diagram guideline for systematic review and meta-analysis steps. Note : Star icon refers to “2–3 reviewers screen independently”

Methods and results

Detailed steps for conducting any systematic review and meta-analysis.

We searched the methods reported in published SR/MA in tropical medicine and other healthcare fields besides the published guidelines like Cochrane guidelines {Higgins, 2011 #7} [ 4 ] to collect the best low-bias method for each step of SR/MA conduction steps. Furthermore, we used guidelines that we apply in studies for all SR/MA steps. We combined these methods in order to conclude and conduct a detailed flow diagram that shows the SR/MA steps how being conducted.

Any SR/MA must follow the widely accepted Preferred Reporting Items for Systematic Review and Meta-analysis statement (PRISMA checklist 2009) (Additional file 5 : Table S1) [ 5 ].

We proposed our methods according to a valid explanatory simulation example choosing the topic of “evaluating safety of Ebola vaccine,” as it is known that Ebola is a very rare tropical disease but fatal. All the explained methods feature the standards followed internationally, with our compiled experience in the conduct of SR beside it, which we think proved some validity. This is a SR under conduct by a couple of researchers teaming in a research group, moreover, as the outbreak of Ebola which took place (2013–2016) in Africa resulted in a significant mortality and morbidity. Furthermore, since there are many published and ongoing trials assessing the safety of Ebola vaccines, we thought this would provide a great opportunity to tackle this hotly debated issue. Moreover, Ebola started to fire again and new fatal outbreak appeared in the Democratic Republic of Congo since August 2018, which caused infection to more than 1000 people according to the World Health Organization, and 629 people have been killed till now. Hence, it is considered the second worst Ebola outbreak, after the first one in West Africa in 2014 , which infected more than 26,000 and killed about 11,300 people along outbreak course.

Research question and objectives

Like other study designs, the research question of SR/MA should be feasible, interesting, novel, ethical, and relevant. Therefore, a clear, logical, and well-defined research question should be formulated. Usually, two common tools are used: PICO or SPIDER. PICO (Population, Intervention, Comparison, Outcome) is used mostly in quantitative evidence synthesis. Authors demonstrated that PICO holds more sensitivity than the more specific SPIDER approach [ 6 ]. SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) was proposed as a method for qualitative and mixed methods search.

We here recommend a combined approach of using either one or both the SPIDER and PICO tools to retrieve a comprehensive search depending on time and resources limitations. When we apply this to our assumed research topic, being of qualitative nature, the use of SPIDER approach is more valid.

PICO is usually used for systematic review and meta-analysis of clinical trial study. For the observational study (without intervention or comparator), in many tropical and epidemiological questions, it is usually enough to use P (Patient) and O (outcome) only to formulate a research question. We must indicate clearly the population (P), then intervention (I) or exposure. Next, it is necessary to compare (C) the indicated intervention with other interventions, i.e., placebo. Finally, we need to clarify which are our relevant outcomes.

To facilitate comprehension, we choose the Ebola virus disease (EVD) as an example. Currently, the vaccine for EVD is being developed and under phase I, II, and III clinical trials; we want to know whether this vaccine is safe and can induce sufficient immunogenicity to the subjects.

An example of a research question for SR/MA based on PICO for this issue is as follows: How is the safety and immunogenicity of Ebola vaccine in human? (P: healthy subjects (human), I: vaccination, C: placebo, O: safety or adverse effects)

Preliminary research and idea validation

We recommend a preliminary search to identify relevant articles, ensure the validity of the proposed idea, avoid duplication of previously addressed questions, and assure that we have enough articles for conducting its analysis. Moreover, themes should focus on relevant and important health-care issues, consider global needs and values, reflect the current science, and be consistent with the adopted review methods. Gaining familiarity with a deep understanding of the study field through relevant videos and discussions is of paramount importance for better retrieval of results. If we ignore this step, our study could be canceled whenever we find out a similar study published before. This means we are wasting our time to deal with a problem that has been tackled for a long time.

To do this, we can start by doing a simple search in PubMed or Google Scholar with search terms Ebola AND vaccine. While doing this step, we identify a systematic review and meta-analysis of determinant factors influencing antibody response from vaccination of Ebola vaccine in non-human primate and human [ 7 ], which is a relevant paper to read to get a deeper insight and identify gaps for better formulation of our research question or purpose. We can still conduct systematic review and meta-analysis of Ebola vaccine because we evaluate safety as a different outcome and different population (only human).

Inclusion and exclusion criteria

Eligibility criteria are based on the PICO approach, study design, and date. Exclusion criteria mostly are unrelated, duplicated, unavailable full texts, or abstract-only papers. These exclusions should be stated in advance to refrain the researcher from bias. The inclusion criteria would be articles with the target patients, investigated interventions, or the comparison between two studied interventions. Briefly, it would be articles which contain information answering our research question. But the most important is that it should be clear and sufficient information, including positive or negative, to answer the question.

For the topic we have chosen, we can make inclusion criteria: (1) any clinical trial evaluating the safety of Ebola vaccine and (2) no restriction regarding country, patient age, race, gender, publication language, and date. Exclusion criteria are as follows: (1) study of Ebola vaccine in non-human subjects or in vitro studies; (2) study with data not reliably extracted, duplicate, or overlapping data; (3) abstract-only papers as preceding papers, conference, editorial, and author response theses and books; (4) articles without available full text available; and (5) case reports, case series, and systematic review studies. The PRISMA flow diagram template that is used in SR/MA studies can be found in Fig. 2 .

PRISMA flow diagram of studies’ screening and selection

Search strategy

A standard search strategy is used in PubMed, then later it is modified according to each specific database to get the best relevant results. The basic search strategy is built based on the research question formulation (i.e., PICO or PICOS). Search strategies are constructed to include free-text terms (e.g., in the title and abstract) and any appropriate subject indexing (e.g., MeSH) expected to retrieve eligible studies, with the help of an expert in the review topic field or an information specialist. Additionally, we advise not to use terms for the Outcomes as their inclusion might hinder the database being searched to retrieve eligible studies because the used outcome is not mentioned obviously in the articles.

The improvement of the search term is made while doing a trial search and looking for another relevant term within each concept from retrieved papers. To search for a clinical trial, we can use these descriptors in PubMed: “clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH terms] OR “clinical trial”[All Fields]. After some rounds of trial and refinement of search term, we formulate the final search term for PubMed as follows: (ebola OR ebola virus OR ebola virus disease OR EVD) AND (vaccine OR vaccination OR vaccinated OR immunization) AND (“clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH Terms] OR “clinical trial”[All Fields]). Because the study for this topic is limited, we do not include outcome term (safety and immunogenicity) in the search term to capture more studies.

Search databases, import all results to a library, and exporting to an excel sheet

According to the AMSTAR guidelines, at least two databases have to be searched in the SR/MA [ 8 ], but as you increase the number of searched databases, you get much yield and more accurate and comprehensive results. The ordering of the databases depends mostly on the review questions; being in a study of clinical trials, you will rely mostly on Cochrane, mRCTs, or International Clinical Trials Registry Platform (ICTRP). Here, we propose 12 databases (PubMed, Scopus, Web of Science, EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov , mRCTs, POPLINE, and SIGLE), which help to cover almost all published articles in tropical medicine and other health-related fields. Among those databases, POPLINE focuses on reproductive health. Researchers should consider to choose relevant database according to the research topic. Some databases do not support the use of Boolean or quotation; otherwise, there are some databases that have special searching way. Therefore, we need to modify the initial search terms for each database to get appreciated results; therefore, manipulation guides for each online database searches are presented in Additional file 5 : Table S2. The detailed search strategy for each database is found in Additional file 5 : Table S3. The search term that we created in PubMed needs customization based on a specific characteristic of the database. An example for Google Scholar advanced search for our topic is as follows:

With all of the words: ebola virus

With at least one of the words: vaccine vaccination vaccinated immunization

Where my words occur: in the title of the article

With all of the words: EVD

Finally, all records are collected into one Endnote library in order to delete duplicates and then to it export into an excel sheet. Using remove duplicating function with two options is mandatory. All references which have (1) the same title and author, and published in the same year, and (2) the same title and author, and published in the same journal, would be deleted. References remaining after this step should be exported to an excel file with essential information for screening. These could be the authors’ names, publication year, journal, DOI, URL link, and abstract.

Protocol writing and registration

Protocol registration at an early stage guarantees transparency in the research process and protects from duplication problems. Besides, it is considered a documented proof of team plan of action, research question, eligibility criteria, intervention/exposure, quality assessment, and pre-analysis plan. It is recommended that researchers send it to the principal investigator (PI) to revise it, then upload it to registry sites. There are many registry sites available for SR/MA like those proposed by Cochrane and Campbell collaborations; however, we recommend registering the protocol into PROSPERO as it is easier. The layout of a protocol template, according to PROSPERO, can be found in Additional file 5 : File S1.

Title and abstract screening

Decisions to select retrieved articles for further assessment are based on eligibility criteria, to minimize the chance of including non-relevant articles. According to the Cochrane guidance, two reviewers are a must to do this step, but as for beginners and junior researchers, this might be tiresome; thus, we propose based on our experience that at least three reviewers should work independently to reduce the chance of error, particularly in teams with a large number of authors to add more scrutiny and ensure proper conduct. Mostly, the quality with three reviewers would be better than two, as two only would have different opinions from each other, so they cannot decide, while the third opinion is crucial. And here are some examples of systematic reviews which we conducted following the same strategy (by a different group of researchers in our research group) and published successfully, and they feature relevant ideas to tropical medicine and disease [ 9 , 10 , 11 ].

In this step, duplications will be removed manually whenever the reviewers find them out. When there is a doubt about an article decision, the team should be inclusive rather than exclusive, until the main leader or PI makes a decision after discussion and consensus. All excluded records should be given exclusion reasons.

Full text downloading and screening

Many search engines provide links for free to access full-text articles. In case not found, we can search in some research websites as ResearchGate, which offer an option of direct full-text request from authors. Additionally, exploring archives of wanted journals, or contacting PI to purchase it if available. Similarly, 2–3 reviewers work independently to decide about included full texts according to eligibility criteria, with reporting exclusion reasons of articles. In case any disagreement has occurred, the final decision has to be made by discussion.

Manual search

One has to exhaust all possibilities to reduce bias by performing an explicit hand-searching for retrieval of reports that may have been dropped from first search [ 12 ]. We apply five methods to make manual searching: searching references from included studies/reviews, contacting authors and experts, and looking at related articles/cited articles in PubMed and Google Scholar.

We describe here three consecutive methods to increase and refine the yield of manual searching: firstly, searching reference lists of included articles; secondly, performing what is known as citation tracking in which the reviewers track all the articles that cite each one of the included articles, and this might involve electronic searching of databases; and thirdly, similar to the citation tracking, we follow all “related to” or “similar” articles. Each of the abovementioned methods can be performed by 2–3 independent reviewers, and all the possible relevant article must undergo further scrutiny against the inclusion criteria, after following the same records yielded from electronic databases, i.e., title/abstract and full-text screening.

We propose an independent reviewing by assigning each member of the teams a “tag” and a distinct method, to compile all the results at the end for comparison of differences and discussion and to maximize the retrieval and minimize the bias. Similarly, the number of included articles has to be stated before addition to the overall included records.

Data extraction and quality assessment

This step entitles data collection from included full-texts in a structured extraction excel sheet, which is previously pilot-tested for extraction using some random studies. We recommend extracting both adjusted and non-adjusted data because it gives the most allowed confounding factor to be used in the analysis by pooling them later [ 13 ]. The process of extraction should be executed by 2–3 independent reviewers. Mostly, the sheet is classified into the study and patient characteristics, outcomes, and quality assessment (QA) tool.

Data presented in graphs should be extracted by software tools such as Web plot digitizer [ 14 ]. Most of the equations that can be used in extraction prior to analysis and estimation of standard deviation (SD) from other variables is found inside Additional file 5 : File S2 with their references as Hozo et al. [ 15 ], Xiang et al. [ 16 ], and Rijkom et al. [ 17 ]. A variety of tools are available for the QA, depending on the design: ROB-2 Cochrane tool for randomized controlled trials [ 18 ] which is presented as Additional file 1 : Figure S1 and Additional file 2 : Figure S2—from a previous published article data—[ 19 ], NIH tool for observational and cross-sectional studies [ 20 ], ROBINS-I tool for non-randomize trials [ 21 ], QUADAS-2 tool for diagnostic studies, QUIPS tool for prognostic studies, CARE tool for case reports, and ToxRtool for in vivo and in vitro studies. We recommend that 2–3 reviewers independently assess the quality of the studies and add to the data extraction form before the inclusion into the analysis to reduce the risk of bias. In the NIH tool for observational studies—cohort and cross-sectional—as in this EBOLA case, to evaluate the risk of bias, reviewers should rate each of the 14 items into dichotomous variables: yes, no, or not applicable. An overall score is calculated by adding all the items scores as yes equals one, while no and NA equals zero. A score will be given for every paper to classify them as poor, fair, or good conducted studies, where a score from 0–5 was considered poor, 6–9 as fair, and 10–14 as good.

In the EBOLA case example above, authors can extract the following information: name of authors, country of patients, year of publication, study design (case report, cohort study, or clinical trial or RCT), sample size, the infected point of time after EBOLA infection, follow-up interval after vaccination time, efficacy, safety, adverse effects after vaccinations, and QA sheet (Additional file 6 : Data S1).

Data checking

Due to the expected human error and bias, we recommend a data checking step, in which every included article is compared with its counterpart in an extraction sheet by evidence photos, to detect mistakes in data. We advise assigning articles to 2–3 independent reviewers, ideally not the ones who performed the extraction of those articles. When resources are limited, each reviewer is assigned a different article than the one he extracted in the previous stage.

Statistical analysis

Investigators use different methods for combining and summarizing findings of included studies. Before analysis, there is an important step called cleaning of data in the extraction sheet, where the analyst organizes extraction sheet data in a form that can be read by analytical software. The analysis consists of 2 types namely qualitative and quantitative analysis. Qualitative analysis mostly describes data in SR studies, while quantitative analysis consists of two main types: MA and network meta-analysis (NMA). Subgroup, sensitivity, cumulative analyses, and meta-regression are appropriate for testing whether the results are consistent or not and investigating the effect of certain confounders on the outcome and finding the best predictors. Publication bias should be assessed to investigate the presence of missing studies which can affect the summary.

To illustrate basic meta-analysis, we provide an imaginary data for the research question about Ebola vaccine safety (in terms of adverse events, 14 days after injection) and immunogenicity (Ebola virus antibodies rise in geometric mean titer, 6 months after injection). Assuming that from searching and data extraction, we decided to do an analysis to evaluate Ebola vaccine “A” safety and immunogenicity. Other Ebola vaccines were not meta-analyzed because of the limited number of studies (instead, it will be included for narrative review). The imaginary data for vaccine safety meta-analysis can be accessed in Additional file 7 : Data S2. To do the meta-analysis, we can use free software, such as RevMan [ 22 ] or R package meta [ 23 ]. In this example, we will use the R package meta. The tutorial of meta package can be accessed through “General Package for Meta-Analysis” tutorial pdf [ 23 ]. The R codes and its guidance for meta-analysis done can be found in Additional file 5 : File S3.

For the analysis, we assume that the study is heterogenous in nature; therefore, we choose a random effect model. We did an analysis on the safety of Ebola vaccine A. From the data table, we can see some adverse events occurring after intramuscular injection of vaccine A to the subject of the study. Suppose that we include six studies that fulfill our inclusion criteria. We can do a meta-analysis for each of the adverse events extracted from the studies, for example, arthralgia, from the results of random effect meta-analysis using the R meta package.

From the results shown in Additional file 3 : Figure S3, we can see that the odds ratio (OR) of arthralgia is 1.06 (0.79; 1.42), p value = 0.71, which means that there is no association between the intramuscular injection of Ebola vaccine A and arthralgia, as the OR is almost one, and besides, the P value is insignificant as it is > 0.05.

In the meta-analysis, we can also visualize the results in a forest plot. It is shown in Fig. 3 an example of a forest plot from the simulated analysis.

Random effect model forest plot for comparison of vaccine A versus placebo

From the forest plot, we can see six studies (A to F) and their respective OR (95% CI). The green box represents the effect size (in this case, OR) of each study. The bigger the box means the study weighted more (i.e., bigger sample size). The blue diamond shape represents the pooled OR of the six studies. We can see the blue diamond cross the vertical line OR = 1, which indicates no significance for the association as the diamond almost equalized in both sides. We can confirm this also from the 95% confidence interval that includes one and the p value > 0.05.

For heterogeneity, we see that I 2 = 0%, which means no heterogeneity is detected; the study is relatively homogenous (it is rare in the real study). To evaluate publication bias related to the meta-analysis of adverse events of arthralgia, we can use the metabias function from the R meta package (Additional file 4 : Figure S4) and visualization using a funnel plot. The results of publication bias are demonstrated in Fig. 4 . We see that the p value associated with this test is 0.74, indicating symmetry of the funnel plot. We can confirm it by looking at the funnel plot.

Publication bias funnel plot for comparison of vaccine A versus placebo

Looking at the funnel plot, the number of studies at the left and right side of the funnel plot is the same; therefore, the plot is symmetry, indicating no publication bias detected.

Sensitivity analysis is a procedure used to discover how different values of an independent variable will influence the significance of a particular dependent variable by removing one study from MA. If all included study p values are < 0.05, hence, removing any study will not change the significant association. It is only performed when there is a significant association, so if the p value of MA done is 0.7—more than one—the sensitivity analysis is not needed for this case study example. If there are 2 studies with p value > 0.05, removing any of the two studies will result in a loss of the significance.

Double data checking

For more assurance on the quality of results, the analyzed data should be rechecked from full-text data by evidence photos, to allow an obvious check for the PI of the study.

Manuscript writing, revision, and submission to a journal

Writing based on four scientific sections: introduction, methods, results, and discussion, mostly with a conclusion. Performing a characteristic table for study and patient characteristics is a mandatory step which can be found as a template in Additional file 5 : Table S3.

After finishing the manuscript writing, characteristics table, and PRISMA flow diagram, the team should send it to the PI to revise it well and reply to his comments and, finally, choose a suitable journal for the manuscript which fits with considerable impact factor and fitting field. We need to pay attention by reading the author guidelines of journals before submitting the manuscript.

The role of evidence-based medicine in biomedical research is rapidly growing. SR/MAs are also increasing in the medical literature. This paper has sought to provide a comprehensive approach to enable reviewers to produce high-quality SR/MAs. We hope that readers could gain general knowledge about how to conduct a SR/MA and have the confidence to perform one, although this kind of study requires complex steps compared to narrative reviews.

Having the basic steps for conduction of MA, there are many advanced steps that are applied for certain specific purposes. One of these steps is meta-regression which is performed to investigate the association of any confounder and the results of the MA. Furthermore, there are other types rather than the standard MA like NMA and MA. In NMA, we investigate the difference between several comparisons when there were not enough data to enable standard meta-analysis. It uses both direct and indirect comparisons to conclude what is the best between the competitors. On the other hand, mega MA or MA of patients tend to summarize the results of independent studies by using its individual subject data. As a more detailed analysis can be done, it is useful in conducting repeated measure analysis and time-to-event analysis. Moreover, it can perform analysis of variance and multiple regression analysis; however, it requires homogenous dataset and it is time-consuming in conduct [ 24 ].

Conclusions

Systematic review/meta-analysis steps include development of research question and its validation, forming criteria, search strategy, searching databases, importing all results to a library and exporting to an excel sheet, protocol writing and registration, title and abstract screening, full-text screening, manual searching, extracting data and assessing its quality, data checking, conducting statistical analysis, double data checking, manuscript writing, revising, and submitting to a journal.

Availability of data and materials

Not applicable.

Abbreviations

Network meta-analysis

Principal investigator

Population, Intervention, Comparison, Outcome

Preferred Reporting Items for Systematic Review and Meta-analysis statement

Quality assessment

Sample, Phenomenon of Interest, Design, Evaluation, Research type

Systematic review and meta-analyses

Bello A, Wiebe N, Garg A, Tonelli M. Evidence-based decision-making 2: systematic reviews and meta-analysis. Methods Mol Biol (Clifton, NJ). 2015;1281:397–416.

Article   Google Scholar  

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Rys P, Wladysiuk M, Skrzekowska-Baran I, Malecki MT. Review articles, systematic reviews and meta-analyses: which can be trusted? Polskie Archiwum Medycyny Wewnetrznej. 2009;119(3):148–56.

PubMed   Google Scholar  

Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Gross L, Lhomme E, Pasin C, Richert L, Thiebaut R. Ebola vaccine development: systematic review of pre-clinical and clinical studies, and meta-analysis of determinants of antibody response variability after vaccination. Int J Infect Dis. 2018;74:83–96.

Article   CAS   Google Scholar  

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, ... Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

Giang HTN, Banno K, Minh LHN, Trinh LT, Loc LT, Eltobgy A, et al. Dengue hemophagocytic syndrome: a systematic review and meta-analysis on epidemiology, clinical signs, outcomes, and risk factors. Rev Med Virol. 2018;28(6):e2005.

Morra ME, Altibi AMA, Iqtadar S, Minh LHN, Elawady SS, Hallab A, et al. Definitions for warning signs and signs of severe dengue according to the WHO 2009 classification: systematic review of literature. Rev Med Virol. 2018;28(4):e1979.

Morra ME, Van Thanh L, Kamel MG, Ghazy AA, Altibi AMA, Dat LM, et al. Clinical outcomes of current medical approaches for Middle East respiratory syndrome: a systematic review and meta-analysis. Rev Med Virol. 2018;28(3):e1977.

Vassar M, Atakpo P, Kash MJ. Manual search approaches used by systematic reviewers in dermatology. Journal of the Medical Library Association: JMLA. 2016;104(4):302.

Naunheim MR, Remenschneider AK, Scangas GA, Bunting GW, Deschler DG. The effect of initial tracheoesophageal voice prosthesis size on postoperative complications and voice outcomes. Ann Otol Rhinol Laryngol. 2016;125(6):478–84.

Rohatgi AJaiWa. Web Plot Digitizer. ht tp. 2014;2.

Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005;5(1):13.

Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014;14(1):135.

Van Rijkom HM, Truin GJ, Van’t Hof MA. A meta-analysis of clinical studies on the caries-inhibiting effect of fluoride gel treatment. Carries Res. 1998;32(2):83–92.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Tawfik GM, Tieu TM, Ghozy S, Makram OM, Samuel P, Abdelaal A, et al. Speech efficacy, safety and factors affecting lifetime of voice prostheses in patients with laryngeal cancer: a systematic review and network meta-analysis of randomized controlled trials. J Clin Oncol. 2018;36(15_suppl):e18031-e.

Wannemuehler TJ, Lobo BC, Johnson JD, Deig CR, Ting JY, Gregory RL. Vibratory stimulus reduces in vitro biofilm formation on tracheoesophageal voice prostheses. Laryngoscope. 2016;126(12):2752–7.

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355.

RevMan The Cochrane Collaboration %J Copenhagen TNCCTCC. Review Manager (RevMan). 5.0. 2008.

Schwarzer GJRn. meta: An R package for meta-analysis. 2007;7(3):40-45.

Google Scholar  

Simms LLH. Meta-analysis versus mega-analysis: is there a difference? Oral budesonide for the maintenance of remission in Crohn’s disease: Faculty of Graduate Studies, University of Western Ontario; 1998.

Download references

Acknowledgements

This study was conducted (in part) at the Joint Usage/Research Center on Tropical Disease, Institute of Tropical Medicine, Nagasaki University, Japan.

Author information

Authors and affiliations.

Faculty of Medicine, Ain Shams University, Cairo, Egypt

Gehad Mohamed Tawfik

Online research Club http://www.onlineresearchclub.org/

Gehad Mohamed Tawfik, Kadek Agus Surya Dila, Muawia Yousif Fadlelmola Mohamed, Dao Ngoc Hien Tam, Nguyen Dang Kien & Ali Mahmoud Ahmed

Pratama Giri Emas Hospital, Singaraja-Amlapura street, Giri Emas village, Sawan subdistrict, Singaraja City, Buleleng, Bali, 81171, Indonesia

Kadek Agus Surya Dila

Faculty of Medicine, University of Khartoum, Khartoum, Sudan

Muawia Yousif Fadlelmola Mohamed

Nanogen Pharmaceutical Biotechnology Joint Stock Company, Ho Chi Minh City, Vietnam

Dao Ngoc Hien Tam

Department of Obstetrics and Gynecology, Thai Binh University of Medicine and Pharmacy, Thai Binh, Vietnam

Nguyen Dang Kien

Faculty of Medicine, Al-Azhar University, Cairo, Egypt

Ali Mahmoud Ahmed

Evidence Based Medicine Research Group & Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Nguyen Tien Huy

Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Department of Clinical Product Development, Institute of Tropical Medicine (NEKKEN), Leading Graduate School Program, and Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto, Nagasaki, 852-8523, Japan

You can also search for this author in PubMed   Google Scholar

Contributions

NTH and GMT were responsible for the idea and its design. The figure was done by GMT. All authors contributed to the manuscript writing and approval of the final version.

Corresponding author

Correspondence to Nguyen Tien Huy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:.

Figure S1. Risk of bias assessment graph of included randomized controlled trials. (TIF 20 kb)

Additional file 2:

Figure S2. Risk of bias assessment summary. (TIF 69 kb)

Additional file 3:

Figure S3. Arthralgia results of random effect meta-analysis using R meta package. (TIF 20 kb)

Additional file 4:

Figure S4. Arthralgia linear regression test of funnel plot asymmetry using R meta package. (TIF 13 kb)

Additional file 5:

Table S1. PRISMA 2009 Checklist. Table S2. Manipulation guides for online database searches. Table S3. Detailed search strategy for twelve database searches. Table S4. Baseline characteristics of the patients in the included studies. File S1. PROSPERO protocol template file. File S2. Extraction equations that can be used prior to analysis to get missed variables. File S3. R codes and its guidance for meta-analysis done for comparison between EBOLA vaccine A and placebo. (DOCX 49 kb)

Additional file 6:

Data S1. Extraction and quality assessment data sheets for EBOLA case example. (XLSX 1368 kb)

Additional file 7:

Data S2. Imaginary data for EBOLA case example. (XLSX 10 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Tawfik, G.M., Dila, K.A.S., Mohamed, M.Y.F. et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health 47 , 46 (2019). https://doi.org/10.1186/s41182-019-0165-6

Download citation

Received : 30 January 2019

Accepted : 24 May 2019

Published : 01 August 2019

DOI : https://doi.org/10.1186/s41182-019-0165-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Tropical Medicine and Health

ISSN: 1349-4147

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• Langson Library
• Science Library
• Grunigen Medical Library
• Law Library
• Connect From Off-Campus
• Accessibility
• Gateway Study Center

Email this link

Systematic reviews & evidence synthesis methods.

• Schedule a Consultation / Meet our Team
• What is Evidence Synthesis?
• Types of Evidence Synthesis
• Evidence Synthesis Across Disciplines
• Finding and Appraising Existing Systematic Reviews
• 1. Develop a Protocol
• 2. Draft your Research Question
• 3. Select Databases
• 4. Select Grey Literature Sources
• 5. Write a Search Strategy
• 6. Register a Protocol
• 7. Translate Search Strategies
• 8. Citation Management
• 9. Article Screening
• 10. Risk of Bias Assessment
• 11. Data Extraction
• 12. Synthesize, Map, or Describe the Results
• Open Access Evidence Synthesis Resources

About This Guide

This research guide provides an overview of the evidence synthesis process, guidance documents for conducting evidence synthesis projects, and links to resources to help you conduct a comprehensive and systematic search of the scholarly literature. Navigate the guide using the tabs on the left.

"Evidence synthesis" refers to rigorous, well-documented methods of identifying, selecting, and combining results from multiple studies. These projects are conducted by teams and follow specific methodologies to minimize bias and maximize reproducibility. A systematic review is a type of evidence synthesis. We use the term evidence synthesis to better reflect the breadth of methodologies that we support, including systematic reviews, scoping reviews , evidence gap maps, umbrella reviews, meta-analyses and others.

Note: Librarians at UC Irvine Libraries have supported systematic reviews and related methodologies in STEM fields for several years. As our service has evolved, we have added capacity to support these reviews in the Social Sciences as well.

Systematic Review OR Literature Review Conducted Systematically?

There are many types of literature reviews. Before beginning a systematic review, consider whether it is the best type of review for your question, goals, and resources. The table below compares systematic reviews, scoping reviews, and systematized reviews (narrative literature reviews employing some, but not all elements of a systematic review) to help you decide which is best for you. See the Types of Evidence Synthesis page for a more in-depth overview at types of reviews.

• Next: UC Irvine Libraries Evidence Synthesis Service >>
• Last Updated: May 25, 2024 9:20 AM
• URL: https://guides.lib.uci.edu/evidence-synthesis

Off-campus? Please use the Software VPN and choose the group UCIFull to access licensed content. For more information, please Click here

Software VPN is not available for guests, so they may not have access to some content when connecting from off-campus.

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Performing a...

Performing a literature review

• Related content
• Peer review
• Gulraj S Matharu , academic foundation doctor ,
• Christopher D Buckley , Arthritis Research UK professor of rheumatology
• 1 Institute of Biomedical Research, College of Medical and Dental Sciences, School of Immunity and Infection, University of Birmingham, UK

A necessary skill for any doctor

What causes disease, which drug is best, does this patient need surgery, and what is the prognosis? Although experience helps in answering these questions, ultimately they are best answered by evidence based medicine. But how do you assess the evidence? As a medical student, and throughout your career as a doctor, critical appraisal of published literature is an important skill to develop and refine. At medical school you will repeatedly appraise published literature and write literature reviews. These activities are commonly part of a special study module, research project for an intercalated degree, or another type of essay based assignment.

Formulating a question

Literature reviews are most commonly performed to help answer a particular question. While you are at medical school, there will usually be some choice regarding the area you are going to review.

Once you have identified a subject area for review, the next step is to formulate a specific research question. This is arguably the most important step because a clear question needs to be defined from the outset, which you aim to answer by doing the review. The clearer the question, the more likely it is that the answer will be clear too. It is important to have discussions with your supervisor when formulating a research question as his or her input will be invaluable. The research question must be objective and concise because it is easier to search through the evidence with a clear question. The question also needs to be feasible. What is the point in having a question for which no published evidence exists? Your supervisor’s input will ensure you are not trying to answer an unrealistic question. Finally, is the research question clinically important? There are many research questions that may be answered, but not all of them will be relevant to clinical practice. The research question we will use as an example to work through in this article is, “What is the evidence for using angiotensin converting enzyme (ACE) inhibitors in patients with hypertension?”

Collecting the evidence

After formulating a specific research question for your literature review, the next step is to collect the evidence. Your supervisor will initially point you in the right direction by highlighting some of the more relevant papers published. Before doing the literature search it is important to agree a list of keywords with your supervisor. A source of useful keywords can be obtained by reading Cochrane reviews or other systematic reviews, such as those published in the BMJ . 1 2 A relevant Cochrane review for our research question on ACE inhibitors in hypertension is that by Heran and colleagues. 3 Appropriate keywords to search for the evidence include the words used in your research question (“angiotensin converting enzyme inhibitor,” “hypertension,” “blood pressure”), details of the types of study you are looking for (“randomised controlled trial,” “case control,” “cohort”), and the specific drugs you are interested in (that is, the various ACE inhibitors such as “ramipril,” “perindopril,” and “lisinopril”).

Once keywords have been agreed it is time to search for the evidence using the various electronic medical databases (such as PubMed, Medline, and EMBASE). PubMed is the largest of these databases and contains online information and tutorials on how to do literature searches with worked examples. Searching the databases and obtaining the articles are usually free of charge through the subscription that your university pays. Early consultation with a medical librarian is important as it will help you perform your literature search in an impartial manner, and librarians can train you to do these searches for yourself.

Literature searches can be broad or tailored to be more specific. With our example, a broad search would entail searching all articles that contain the words “blood pressure” or “ACE inhibitor.” This provides a comprehensive list of all the literature, but there are likely to be thousands of articles to review subsequently (fig 1). ⇓ In contrast, various search restrictions can be applied on the electronic databases to filter out papers that may not be relevant to your review. Figure 2 gives an example of a specific search. ⇓ The search terms used in this case were “angiotensin converting enzyme inhibitor” and “hypertension.” The limits applied to this search were all randomised controlled trials carried out in humans, published in the English language over the last 10 years, with the search terms appearing in the title of the study only. Thus the more specific the search strategy, the more manageable the number of articles to review (fig 3), and this will save you time. ⇓ However, this method risks your not identifying all the evidence in the particular field. Striking a balance between a broad and a specific search strategy is therefore important. This will come with experience and consultation with your supervisor. It is important to note that evidence is continually becoming available on these electronic databases and therefore repeating the same search at a later date can provide new evidence relevant to your review.

Fig 1 Results from a broad literature search using the term “angiotensin converting enzyme inhibitor”

• Download figure
• Open in new tab
• Download powerpoint

Fig 2 Example of a specific literature search. The search terms used were “angiotensin converting enzyme inhibitor” and “hypertension.” The limits applied to this search were all randomised controlled trials carried out in humans, published in English over the past 10 years, with the search terms appearing in the title of the study only

Fig 3 Results from a specific literature search (using the search terms and limits from figure 2)

Reading the abstracts (study summary) of the articles identified in your search may help you decide whether the study is applicable for your review—for example, the work may have been carried out using an animal model rather than in humans. After excluding any inappropriate articles, you need to obtain the full articles of studies you have identified. Additional relevant articles that may not have come up in your original search can also be found by searching the reference lists of the articles you have already obtained. Once again, you may find that some articles are still not applicable for your review, and these can also be excluded at this stage. It is important to explain in your final review what criteria you used to exclude articles as well as those criteria used for inclusion.

The National Institute for Health and Clinical Excellence (NICE) publishes evidence based guidelines for the United Kingdom and therefore provides an additional resource for identifying the relevant literature in a particular field. 4 NICE critically appraises the published literature with recommendations for best clinical practice proposed and graded based on the quality of evidence available. Similarly, there are internationally published evidence based guidelines, such as those produced by the European Society of Cardiology and the American College of Chest Physicians, which can be useful when collecting the literature in a particular field. 5 6

Appraising the evidence

Once you have collected the evidence, you need to critically appraise the published material. Box 1 gives definitions of terms you will encounter when reading the literature. A brief guide of how to critically appraise a study is presented; however, it is advisable to consult the references cited for further details.

Box 1: Definitions of common terms in the literature 7

Prospective—collecting data in real time after the study is designed

Retrospective—analysis of data that have already been collected to determine associations between exposure and outcome

Hypothesis—proposed association between exposure and outcome. If presented in the negative it is called the null hypothesis

Variable—a quantity or quality that changes during the study and can be measured

Single blind—subjects are unaware of their treatment, but clinicians are aware

Double blind—both subjects and clinicians are unaware of treatment given

Placebo—a simulated medical intervention, with subjects not receiving the specific intervention or treatment being studied

Outcome measure/endpoint—clinical variable or variables measured in a study subsequently used to make conclusions about the original interventions or treatments administered

Bias—difference between reported results and true results. Many types exist (such as selection, allocation, and reporting biases)

Probability (P) value—number between 0 and 1 providing the likelihood the reported results occurred by chance. A P value of 0.05 means there is a 5% likelihood that the reported result occurred by chance

Confidence intervals—provides a range between two numbers within which one can be certain the results lie. A confidence interval of 95% means one can be 95% certain the actual results lie within the reported range

The study authors should clearly define their research question and ideally the hypothesis to be tested. If the hypothesis is presented in the negative, it is called the null hypothesis. An example of a null hypothesis is smoking does not cause lung cancer. The study is then performed to assess the significance of the exposure (smoking) on outcome (lung cancer).

A major part of the critical appraisal process is to focus on study methodology, with your key task being an assessment of the extent to which a study was susceptible to bias (the discrepancy between the reported results and the true results). It should be clear from the methods what type of study was performed (box 2).

Box 2: Different study types 7

Systematic review/meta-analysis—comprehensive review of published literature using predefined methodology. Meta-analyses combine results from various studies to give numerical data for the overall association between variables

Randomised controlled trial—random allocation of patients to one of two or more groups. Used to test a new drug or procedure

Cohort study—two or more groups followed up over a long period, with one group exposed to a certain agent (drug or environmental agent) and the other not exposed, with various outcomes compared. An example would be following up a group of smokers and a group of non-smokers with the outcome measure being the development of lung cancer

Case-control study—cases (those with a particular outcome) are matched as closely as possible (for age, sex, ethnicity) with controls (those without the particular outcome). Retrospective data analysis is performed to determine any factors associated with developing the particular outcomes

Cross sectional study—looks at a specific group of patients at a single point in time. Effectively a survey. An example is asking a group of people how many of them drink alcohol

Case report—detailed reports concerning single patients. Useful in highlighting adverse drug reactions

There are many different types of bias, which depend on the particular type of study performed, and it is important to look for these biases. Several published checklists are available that provide excellent resources to help you work through the various studies and identify sources of bias. The CONSORT statement (which stands for CONsolidated Standards Of Reporting Trials) provides a minimum set of recommendations for reporting randomised controlled trials and comprises a rigorous 25 item checklist, with variations available for other study types. 8 9 As would be expected, most (17 of 25) of the items focus on questions relating to the methods and results of the randomised trial. The remaining items relate to the title, abstract, introduction, and discussion of the study, in addition to questions on trial registration, protocol, and funding.

Jadad scoring provides a simple and validated system to assess the methodological quality of a randomised clinical trial using three questions. 10 The score ranges from zero to five, with one point given for a “yes” in each of the following questions. (1) Was the study described as randomised? (2) Was the study described as double blind? (3) Were there details of subject withdrawals, exclusions, and dropouts? A further point is given if (1) the method of randomisation was appropriate, and (2) the method of blinding was appropriate.

In addition, the Critical Appraisal Skills Programme provides excellent tools for assessing the evidence in all study types (box 2). 11 The Oxford Centre for Evidence-Based Medicine levels of evidence is yet another useful resource for assessing the methodological quality of all studies. 12

Ensure all patients have been accounted for and any exclusions, for whatever reason, are reported. Knowing the baseline demographic (age, sex, ethnicity) and clinical characteristics of the population is important. Results are usually reported as probability values or confidence intervals (box 1).

This should explain the major study findings, put the results in the context of the published literature, and attempt to account for any variations from previous work. Study limitations and sources of bias should be discussed. Authors’ conclusions should be supported by the study results and not unnecessarily extrapolated. For example, a treatment shown to be effective in animals does not necessarily mean it will work in humans.

The format for writing up the literature review usually consists of an abstract (short structured summary of the review), the introduction or background, methods, results, and discussion with conclusions. There are a number of good examples of how to structure a literature review and these can be used as an outline when writing your review. 13 14

The introduction should identify the specific research question you intend to address and briefly put this into the context of the published literature. As you have now probably realised, the methods used for the review must be clear to the reader and provide the necessary detail for someone to be able to reproduce the search. The search strategy needs to include a list of keywords used, which databases were searched, and the specific search limits or filters applied. Any grading of methodological quality, such as the CONSORT statement or Jadad scoring, must be explained in addition to any study inclusion or exclusion criteria. 6 7 8 The methods also need to include a section on the data collected from each of the studies, the specific outcomes of interest, and any statistical analysis used. The latter point is usually relevant only when performing meta-analyses.

The results section must clearly show the process of filtering down from the articles obtained from the original search to the final studies included in the review—that is, accounting for all excluded studies. A flowchart is usually best to illustrate this. Next should follow a brief description of what was done in the main studies, the number of participants, the relevant results, and any potential sources of bias. It is useful to group similar studies together as it allows comparisons to be made by the reader and saves repetition in your write-up. Boxes and figures should be used appropriately to illustrate important findings from the various studies.

Finally, in the discussion you need to consider the study findings in light of the methodological quality—that is, the extent of potential bias in each study that may have affected the study results. Using the evidence, you need to make conclusions in your review, and highlight any important gaps in the evidence base, which need to be dealt with in future studies. Working through drafts of the literature review with your supervisor will help refine your critical appraisal skills and the ability to present information concisely in a structured review article. Remember, if the work is good it may get published.

Originally published as: Student BMJ 2012;20:e404

Competing interests: None declared.

Provenance and peer review: Not commissioned; externally peer reviewed.

• ↵ The Cochrane Library. www3.interscience.wiley.com/cgibin/mrwhome/106568753/HOME?CRETRY=1&SRETRY=0 .
• ↵ British Medical Journal . www.bmj.com/ .
• ↵ Heran BS, Wong MMY, Heran IK, Wright JM. Blood pressure lowering efficacy of angiotensin converting enzyme (ACE) inhibitors for primary hypertension. Cochrane Database Syst Rev 2008 ; 4 : CD003823 , doi: 10.1002/14651858.CD003823.pub2. OpenUrl PubMed
• ↵ National Institute for Health and Clinical Excellence. www.nice.org.uk .
• ↵ European Society of Cardiology. www.escardio.org/guidelines .
• ↵ Geerts WH, Bergqvist D, Pineo GF, Heit JA, Samama CM, Lassen MR, et al. Prevention of venous thromboembolism: American College of Chest Physicians evidence-based clinical practice guidelines (8th ed). Chest 2008 ; 133 : 381 -453S. OpenUrl CrossRef
• ↵ Wikipedia. http://en.wikipedia.org/wiki .
• ↵ Moher D, Schulz KF, Altman DG, Egger M, Davidoff F, Elbourne D, et al. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 2001 ; 357 : 1191 -4. OpenUrl CrossRef PubMed Web of Science
• ↵ The CONSORT statement. www.consort-statement.org/ .
• ↵ Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996 ; 17 : 1 -12. OpenUrl CrossRef PubMed Web of Science
• ↵ Critical Appraisal Skills Programme (CASP). www.sph.nhs.uk/what-we-do/public-health-workforce/resources/critical-appraisals-skills-programme .
• ↵ Oxford Centre for Evidence-based Medicine—Levels of Evidence. www.cebm.net .
• ↵ Van den Bruel A, Thompson MJ, Haj-Hassan T, Stevens R, Moll H, Lakhanpaul M, et al . Diagnostic value of laboratory tests in identifying serious infections in febrile children: systematic review. BMJ 2011 ; 342 : d3082 . OpenUrl Abstract / FREE Full Text
• ↵ Awopetu AI, Moxey P, Hinchliffe RJ, Jones KG, Thompson MM, Holt PJ. Systematic review and meta-analysis of the relationship between hospital volume and outcome for lower limb arterial surgery. Br J Surg 2010 ; 97 : 797 -803. OpenUrl CrossRef PubMed

Health (Nursing, Medicine, Allied Health)

• Find Articles/Databases
• Reference Resources
• Evidence Summaries & Clinical Guidelines
• Drug Information
• Health Data & Statistics
• Patient/Consumer Facing Materials
• Images and Streaming Video
• Grey Literature
• Mobile Apps & "Point of Care" Tools
• Tests & Measures This link opens in a new window
• Citing Sources
• Selecting Databases
• Framing Research Questions
• Crafting a Search
• Narrowing / Filtering a Search
• Expanding a Search
• Cited Reference Searching
• Saving Searches
• Term Glossary
• Critical Appraisal Resources
• What are Literature Reviews?
• Conducting & Reporting Systematic Reviews
• Finding Systematic Reviews
• Tutorials & Tools for Literature Reviews
• Finding Full Text

What are Systematic Reviews? (3 minutes, 24 second YouTube Video)

Systematic Literature Reviews: Steps & Resources

These steps for conducting a systematic literature review are listed below . 

Also see subpages for more information about:

• The different types of literature reviews, including systematic reviews and other evidence synthesis methods
• Tools & Tutorials

Literature Review & Systematic Review Steps

• Develop a Focused Question
• Scope the Literature  (Initial Search)
• Refine & Expand the Search
• Limit the Results
• Download Citations
• Abstract & Analyze
• Create Flow Diagram
• Synthesize & Report Results

1. Develop a Focused   Question 

Consider the PICO Format: Population/Problem, Intervention, Comparison, Outcome

Focus on defining the Population or Problem and Intervention (don't narrow by Comparison or Outcome just yet!)

"What are the effects of the Pilates method for patients with low back pain?"

Tools & Additional Resources:

• PICO Question Help
• Stillwell, Susan B., DNP, RN, CNE; Fineout-Overholt, Ellen, PhD, RN, FNAP, FAAN; Melnyk, Bernadette Mazurek, PhD, RN, CPNP/PMHNP, FNAP, FAAN; Williamson, Kathleen M., PhD, RN Evidence-Based Practice, Step by Step: Asking the Clinical Question, AJN The American Journal of Nursing : March 2010 - Volume 110 - Issue 3 - p 58-61 doi: 10.1097/01.NAJ.0000368959.11129.79

2. Scope the Literature

A "scoping search" investigates the breadth and/or depth of the initial question or may identify a gap in the literature. 

Eligible studies may be located by searching in:

• Background sources (books, point-of-care tools)
• Article databases
• Trial registries
• Grey literature
• Cited references
• Reference lists

When searching, if possible, translate terms to controlled vocabulary of the database. Use text word searching when necessary.

Use Boolean operators to connect search terms:

• Combine separate concepts with AND  (resulting in a narrower search)
• Connecting synonyms with OR  (resulting in an expanded search)

Search:  pilates AND ("low back pain"  OR  backache )

Video Tutorials - Translating PICO Questions into Search Queries

• Translate Your PICO Into a Search in PubMed (YouTube, Carrie Price, 5:11) 
• Translate Your PICO Into a Search in CINAHL (YouTube, Carrie Price, 4:56)

3. Refine & Expand Your Search

Expand your search strategy with synonymous search terms harvested from:

• database thesauri
• reference lists
• relevant studies

Example: 

(pilates OR exercise movement techniques) AND ("low back pain" OR backache* OR sciatica OR lumbago OR spondylosis)

As you develop a final, reproducible strategy for each database, save your strategies in a:

• a personal database account (e.g., MyNCBI for PubMed)
• Log in with your NYU credentials
• Open and "Make a Copy" to create your own tracker for your literature search strategies

4. Limit Your Results

Use database filters to limit your results based on your defined inclusion/exclusion criteria.  In addition to relying on the databases' categorical filters, you may also need to manually screen results.  

• Limit to Article type, e.g.,:  "randomized controlled trial" OR multicenter study
• Limit by publication years, age groups, language, etc.

NOTE: Many databases allow you to filter to "Full Text Only".  This filter is  not recommended . It excludes articles if their full text is not available in that particular database (CINAHL, PubMed, etc), but if the article is relevant, it is important that you are able to read its title and abstract, regardless of 'full text' status. The full text is likely to be accessible through another source (a different database, or Interlibrary Loan).  

• Filters in PubMed
• CINAHL Advanced Searching Tutorial

5. Download Citations

Selected citations and/or entire sets of search results can be downloaded from the database into a citation management tool. If you are conducting a systematic review that will require reporting according to PRISMA standards, a citation manager can help you keep track of the number of articles that came from each database, as well as the number of duplicate records.

In Zotero, you can create a Collection for the combined results set, and sub-collections for the results from each database you search.  You can then use Zotero's 'Duplicate Items" function to find and merge duplicate records.

• Citation Managers - General Guide

6. Abstract and Analyze

• Migrate citations to data collection/extraction tool
• Screen Title/Abstracts for inclusion/exclusion
• Screen and appraise full text for relevance, methods, 
• Resolve disagreements by consensus

Covidence is a web-based tool that enables you to work with a team to screen titles/abstracts and full text for inclusion in your review, as well as extract data from the included studies.

• Covidence Support
• Critical Appraisal Tools
• Data Extraction Tools

7. Create Flow Diagram

The PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) flow diagram is a visual representation of the flow of records through different phases of a systematic review.  It depicts the number of records identified, included and excluded.  It is best used in conjunction with the PRISMA checklist .

Example from: Stotz, S. A., McNealy, K., Begay, R. L., DeSanto, K., Manson, S. M., & Moore, K. R. (2021). Multi-level diabetes prevention and treatment interventions for Native people in the USA and Canada: A scoping review. Current Diabetes Reports, 2 (11), 46. https://doi.org/10.1007/s11892-021-01414-3

• PRISMA Flow Diagram Generator (ShinyApp.io, Haddaway et al. )
• PRISMA Diagram Templates  (Word and PDF)
• Make a copy of the file to fill out the template
• Image can be downloaded as PDF, PNG, JPG, or SVG
• Covidence generates a PRISMA diagram that is automatically updated as records move through the review phases

8. Synthesize & Report Results

There are a number of reporting guideline available to guide the synthesis and reporting of results in systematic literature reviews.

It is common to organize findings in a matrix, also known as a Table of Evidence (ToE).

• Reporting Guidelines for Systematic Reviews
• Download a sample template of a health sciences review matrix  (GoogleSheets)

Steps modified from: 

Cook, D. A., & West, C. P. (2012). Conducting systematic reviews in medical education: a stepwise approach.   Medical Education , 46 (10), 943–952.

• << Previous: Critical Appraisal Resources
• Next: What are Literature Reviews? >>
• Last Updated: May 22, 2024 1:35 PM
• URL: https://guides.nyu.edu/health

Systematic Reviews: Medical Literature Databases to search

• Types of literature review, methods, & resources
• Protocol and registration
• Search strategy
• Medical Literature Databases to search
• Study selection and appraisal
• Data Extraction/Coding/Study characteristics/Results
• Reporting the quality/risk of bias
• Manage citations using RefWorks This link opens in a new window
• GW Box file storage for PDF's This link opens in a new window

How to document your literature search

You should always  document how you have searched each database, what keywords or index terms were used, the date on which the search was performed, how many results you retrieved, and if you use RefWorks to deduplicate results record how many were removed as duplicates and the final number of discrete studies you subjected to your first sift through of study selection.  Here is an example of how to document a literature search on an Excel spreadsheet , this example records a search of the hematology literature for articles about sickle cell disease. Here is another example of  how to document a literature search, this time on one page of a Word document , this example records a search of the medical literature for a poster on Emergency Department throughput.  The numbers recorded can then be used to populate the PRISMA flow diagram summarizing the literature search.

In the final report add as an appendix the full electronic search strategy for each database searched for the literature review e.g. MEDLINE with MeSH terms, keywords & limits

In the final report in the methods section:

PRISMA checklist Item 7 information sources will be reported as:

• What databases/websites you searched, the name of the database search platform and the start/end dates the index covers if relevant e.g. OVID MEDLINE (1950-present, or just PubMed
• Who developed & conducted the searches
• Date each database/website was last searched
• Supplementary sources - what other websites did you search? What journal titles were hand searched, whether reference lists were checked, what trial registries or regulatory agency websites were searched, were manufacturers or other authors contacted to obtain unpublished or missing information on study methods or results.

PRISMA checklist Item 8 search will be reported as:

• In text: describe the principal keywords used to search databases, websites & trials registers

What databases/indexes should you search?

At a minimum you need to search MEDLINE ,  EMBASE , and the  Cochrane CENTRAL  trials register .  This is the recommendation of three medical and public health research organizations: the U.S.  Agency for Healthcare Research and Quality ( AHRQ ), the U.K. Centre for Reviews and Dissemination ( CRD ), and the International Cochrane Collaboration (Source:  Institute of Medicine (2011) Finding What Works in Healthcare: Standards for Systematic Reviews  Table E-1, page 267).  Some databases have an alternate version, linked in parentheses below, that search the same records sets, ie the content of MEDLINE is in PubMed and Scopus, while the content of EMBASE is in Scopus. You should reformat your search for each database as appropriate, contact your librarian if you want help on how to search each database.  

Begin by searching:

1.        MEDLINE  (or  PubMed )

2.       EMBASE (or  Scopus )  Please note Himmelfarb Library does not have a subscription to EMBASE. The content is in the Scopus  database that you can search using keywords, but it is not possible to perform an EMTREE theasaurus search in Scopus.

3.        Cochrane Central Trials Register  (or  Cochrane Library ). In addition Cochrane researchers recommend you search the clinicaltrials.gov and ICTRP clinical trial registries due to the low sensitivity of the Cochrane CENTRAL index because according to Hunter et al (2022) "register records as they appear in CENTRAL are less comprehensive than the original register entry, and thus are at a greater risk than other systems of being missed in a search."

The Polyglot Search Translator is a very useful tool for translating search strings from PubMed or Medline via Ovid across multiple databases, developed by the Institute for Evidence-Based Healthcare at Bond University. But please note Polyglot does not automatically map subject terms across databases (e.g. MeSH terms to Emtree terms) so you will need to manually edit the search syntax in a text editor to change to the actual subject terms used by another database.

The Yale Mesh Analyzer is another very useful tool you can copy and paste in a list of up to 20 PMID numbers for records in the PubMed database, the Yale Mesh Analyzer will then display the Mesh Medical Subject Headings for those 20 articles as a table so you can identify and compare what Mesh headings they have in common, this can suggest additional search terms for your PubMed search.

The MedSyntax tool is another useful tool, for parsing out very long searches with many levels of brackets. This would be useful if you are trying to edit a pre-existing search strategy with many levels of parentheses.

Some sources for pre-existing database search filters or "hedges" include:

• CADTH Search Filters Database ,
• McMaster University Health Information Research Unit ,
• University of York Centre for Reviews and Dissemination InterTASC Information Specialists' Sub-Group ,
• InterTASC Population Specific search filters  (particularly useful for identifying Latinx, Indigenous people's, LGBTQ, Black & Minority ethnic)
• CareSearch Palliative Care PubMed search filters  (bereavement, dementia, heart failure, lung cancer, cost of care, and Palliative Care)
• Low and Middle Income countries filter at https://epoc.cochrane.org/lmic-filters . 
• Search Pubmed for another validated search filter using some variation of a search like this, possibly adding your discipline or search topic keywords: ("Databases, Bibliographic"[Mesh] OR "Search Engine"[Mesh]) AND ("Reproducibility of Results"[Mesh] OR "Sensitivity and Specificity"[Mesh] OR validat*) AND (filter OR hedge) .
• Search MEDLINE (or PubMed), preferably using a peer reviewed search strategy per protocol and apply any relevant methodology filters.
• Search EMBASE (or Scopus) and the Cochrane Central trials register using appropriately reformatted search versions for those databases, and any other online resources. 
• You should also search other subject specific databases that index the literature in your field.  Use our Himmelfarb Library  research guides  to identify other  subject specific databases . 
• Save citations in Covidence to deduplicate citations prior to screening.
• After screening export citations to  RefWorks database when you are ready to write up your manuscript. The Covidence and Refworks databases should be shared with all members of the investigative team.

Supplementary resources to search

Other member of your investigative team may have ideas about databases, websites, and journals they think you should search. Searching these sources is not required to perform a systematic review. You may need to reformat your search keywords.

Researchers at GW should check our subject research guides for suggestions, or check the libguides community for a guide on your subject.

In addition you may wish to search one or more of the following resources:

• Google Scholar
• BASE  academic search engine is useful for searching in University Institutional Repositories
• Cochrane Database of Systematic Reviews  to search for a pre-existing systematic review on your topic
• Epistemonikos database, has a matrix of evidence table so you can see what citations are shared in common across existing systematic reviews of the same topic. This feature might help identify sentinel or 'don't miss' articles.

You might also consider searching one or more of the following websites depending on your topic:

Clinical trial registers. The Cochrane Collaboration recommends for a systematic review to search both clinicaltrials.gov and the WHO ICTRP (See http://handbook.cochrane.org/ section 4.3):

• ClinicalTrials.gov  - also contains study population characteristics and results data of FDA regulated drugs and medical devices in NIH funded studies produced after January 18, 2017.
• WHO ICTRP  - trials register
• TRIP  - searchable index of clinical trials, guidelines,and regulatory guidance
• CenterWatch
• Current Controlled Trials
• European Clinical Trials Register
• ISRCTN Register
• COMPARE - tracks outcome switching in clinical trials
• OpenTrials - aims to match published trials with the underlying data where this is publicly available in an open source 
• ECRI Guidelines Trust

Grey literature resources:

• WONDER - CDC data and reports
• FDSys - search federal government publications
• Science.gov
• NRR Archive
• NIH Reporter
• re3data registry of data repositories
• Data Repositories (listed by the Simmons Open Access Directory)
• OpenDOAR  search academic open access research repositories
• f1000research search open access repositories of articles, slides, and research posters, in the life sciences, public health, education, and communication.
• RAND Health Reports
• National Academy of Medicine Publications
• Kaiser Family Foundation 
• Robert Wood Johnson Foundation health and medical care data archive
• Milbank Memorial Fund reports and issue briefs
• Also search the resources listed in the CADTH (2019) Grey Matters checklist.

Preprints 

• See our Himmelfarb preprints guide page on finding preprints , a useful database for searching Health Sciences preprints is  Europe PMC

Dissertations and Theses:

• Proquest Dissertations and Theses Online 
• Networked Digital Library of Theses and Dissertations
• Open Access Theses and Dissertations
• WorldCat and change Content: from Any Content to Thesis/dissertations

Conference proceedings:

Most conference proceedings are difficult to find because they may or may not be published. Only select individual papers may be made available in print as a book, journal, or series, rather than all of the presented items. Societies and Associations may only publish abstracts, or extended abstracts, from a conference, often in an annual supplement to an issue of the journal of record of that professional society.  Often posters are not published, if they are they may be made available only to other conference registrants at that meeting or online. Authors may "publish" their conference papers or posters on personal or institutional websites.  A limited set of conference proceedings databases include the following:

• BASE  academic search engine, has an Advanced Search feature with a Limit by Type to 'Conference Objects', this is useful for searching for conference posters and submissions stored in University Institutional Repositories.
• Web of Science - click All Databases and select Core Collection - under More Settings limit to the Conference Proceedings Citation Index (CPCI) - searches a limited set of conferences on Science, Social Science and Humanities from 1990-present.
• Scopus - Limit Document Type to Conference Paper or Conference Review.
• Proquest  - Limit search results to conference papers &/or proceedings under Advanced Search.
• BioMed Central Proceedings  - searches a limited set of biomedical conference proceedings, including bioinformatics, genetics, medical students, and data visualization.
• F1000 Research - browse by subject and click the tabs for articles, posters, and slides - which searches a limited number of biology and medical society meetings/conferences. This is a voluntary self-archive repository.

Individual Journals 

• You may choose to "hand search" select journals where the research team reads the Table of Contents of each issue for a chosen period of time.  You can look for the names of high impact journal titles in a particular field indexed in Journal Citation Reports  (JCR). Please note as of August 2021 ISI are linking to a new version of JCR that currently does not have the particularly helpful 'Browse by Category' link working, so I recommend you click the Products link in the top right corner and select Journal Citation Reports (Classic) to switch back to the old version to get that functionality back.
• The AllTrials petition aims to motivate health care researchers to petition regulators and research bodies to require the results and data of all clinical trials be published.
• << Previous: Search strategy
• Next: Study selection and appraisal >>

• Last Updated: May 8, 2024 11:07 AM
• URL: https://guides.himmelfarb.gwu.edu/systematic_review

• Himmelfarb Intranet
• Privacy Notice
• Terms of Use
• GW is committed to digital accessibility. If you experience a barrier that affects your ability to access content on this page, let us know via the Accessibility Feedback Form .
• Himmelfarb Health Sciences Library
• 2300 Eye St., NW, Washington, DC 20037
• Phone: (202) 994-2850
• [email protected]
• https://himmelfarb.gwu.edu

Literature Reviews

• Getting started

What is a literature review?

Why conduct a literature review, stages of a literature review, lit reviews: an overview (video), check out these books.

• Types of reviews
• 1. Define your research question
• 2. Plan your search
• 3. Search the literature
• 4. Organize your results
• 5. Synthesize your findings
• 6. Write the review
• Artificial intelligence (AI) tools
• Thompson Writing Studio This link opens in a new window
• Need to write a systematic review? This link opens in a new window

Contact a Librarian

Ask a Librarian

Definition: A literature review is a systematic examination and synthesis of existing scholarly research on a specific topic or subject.

Purpose: It serves to provide a comprehensive overview of the current state of knowledge within a particular field.

Analysis: Involves critically evaluating and summarizing key findings, methodologies, and debates found in academic literature.

Identifying Gaps: Aims to pinpoint areas where there is a lack of research or unresolved questions, highlighting opportunities for further investigation.

Contextualization: Enables researchers to understand how their work fits into the broader academic conversation and contributes to the existing body of knowledge.

tl;dr  A literature review critically examines and synthesizes existing scholarly research and publications on a specific topic to provide a comprehensive understanding of the current state of knowledge in the field.

What is a literature review NOT?

❌ An annotated bibliography

❌ Original research

❌ A summary

❌ Something to be conducted at the end of your research

❌ An opinion piece

❌ A chronological compilation of studies

The reason for conducting a literature review is to:

Literature Reviews: An Overview for Graduate Students

While this 9-minute video from NCSU is geared toward graduate students, it is useful for anyone conducting a literature review.

Writing the literature review: A practical guide

Available 3rd floor of Perkins

Writing literature reviews: A guide for students of the social and behavioral sciences

Available online!

So, you have to write a literature review: A guided workbook for engineers

Telling a research story: Writing a literature review

The literature review: Six steps to success

Systematic approaches to a successful literature review

Request from Duke Medical Center Library

Doing a systematic review: A student's guide

• Next: Types of reviews >>
• Last Updated: May 17, 2024 8:42 AM
• URL: https://guides.library.duke.edu/litreviews

Services for...

• Faculty & Instructors
• Graduate Students
• Undergraduate Students
• International Students
• Patrons with Disabilities

• Harmful Language Statement
• Re-use & Attribution / Privacy
• Support the Libraries

• Search Menu
• Sign in through your institution
• Advance articles
• Themed Collections
• Editor's Choice
• Ilona Kickbusch Award
• Supplements
• Author Guidelines
• Submission Online
• Open Access Option
• Self-Archiving Policy
• About Health Promotion International
• Editorial Board
• Advertising and Corporate Services
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, supplementary material.

• < Previous

Advertising appeals effectiveness: a systematic literature review

• Article contents
• Figures & tables
• Supplementary Data

Murooj Yousef, Sharyn Rundle-Thiele, Timo Dietrich, Advertising appeals effectiveness: a systematic literature review, Health Promotion International , Volume 38, Issue 4, August 2023, daab204, https://doi.org/10.1093/heapro/daab204

• Permissions Icon Permissions

Positive, negative and coactive appeals are used in advertising. The evidence base indicates mixed results making practitioner guidance on optimal advertising appeals difficult. This study aims to identify the most effective advertising appeals and it seeks to synthesize relevant literature up to August 2019. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework a total of 31 studies were identified and analyzed. Emotional appeals, theory utilization, materials, results and quality were examined. Across multiple contexts, results from this review found that positive appeals were more often effective than coactive and negative appeals. Most studies examined fear and humour appeals, reflecting a literature skew towards the two emotional appeals. The Effective Public Health Practice Project framework was applied to assess the quality of the studies and identified that there remains opportunity for improvement in research design of advertising studies. Only one-third of studies utilized theory, signalling the need for more theory testing and application in future research. Scholars should look at increasing methodological strength by drawing more representative samples, establishing strong study designs and valid data collection methods. In the meantime, advertisers are encouraged to employ and test more positive and coactive advertising appeals.

Advertising appeals have witnessed an increase in research interest and scholarly attention in recent years. Studies investigate appeal effectiveness [e.g. ( Jordan et al. , 2015 ; Lee, 2018 )] and to a lesser extent systematic and meta-analytic studies attempting to synthesize results are evident ( O’Keefe and Jensen, 2009 ; Jenkin et al. , 2014 ; Hornik et al. , 2016 ). These studies however are limited in their focus [e.g. fear appeals ( Tannenbaum et al. , 2015 ; Esrick et al. , 2019 )], context [e.g. disease detection behaviours ( O’Keefe and Jensen, 2009 )], media type, [e.g. mass media ( Elder et al. , 2004 )] and comparison of general advertising appeal types [e.g. rational vs. emotional (fear and humour) vs. metaphor appeals ( Hornik et al. , 2017 )]. Taken together, a review of the literature indicates clear gaps requiring an evidence review focussed on synthesizing studies seeking to examine positive versus negatively framed advertising appeal effectiveness that are context free, not media specific, includes rational as well as emotional studies of different emotional valances (positive, negative and coactive), and extends the range of emotions examined beyond fear and humour which is heavily investigated in the literature. Given that negatively framed appeals dominate behaviour change and prevention studies, a systematic literature review that explores the effectiveness of different advertising approaches is important, timely and called for [e.g. ( Williams et al. , 2004 ; Armstrong, 2010 ; Hornik et al. , 2016 )].

Hornik et al. (Hornik et al. , 2016 ) based their meta-analytic review on rational, emotional (i.e. fear, humour and sex) and metaphor advertising appeals, limiting their results to specific appeal types. The current study seeks to build on their study, extending investigation to other appeals (e.g. coactive) to ascertain the extent these have been used effectively to deliver behaviour change. Following Hornik et al. (Hornik et al. , 2016 ), we argue that positive emotional advertising appeals are more effective in changing behaviour than negative and rational advertising appeals. However, in contrast to their study, we do not follow their general classification of appeals (i.e. rational, emotional and metaphor), but rather we include a wider set of studies that look at rational, emotional, positive, negative and coactive advertising appeals in different campaign contexts (e.g. social and commercial).

Advertising appeals

An advertising appeal refers to the use of persuasion strategies to attract attention, create relevance and memorability, raise awareness and induce action ( Armstrong, 2010 ). An advertising message can appeal to one’s cognition (i.e. rational appeals), emotions (i.e. emotional appeals) or both. Rational appeals rely on arguments, reason and facts to create persuasion ( Dahlen et al. , 2010 ). In contrast, emotional appeals seek to induce certain emotions in the audience to make the message memorable and more persuasive to take action ( Dahlen et al. , 2010 ). The emotional versus rational debate has been widely discussed with scholars exploring effectiveness in different advertising aims, contexts, business types and target audiences [see, e.g. ( Mattila, 1999 ; Matthes and Wonneberger, 2014 ; Akpinar and Berger, 2017 ; Moran and Bagchi, 2019 )]. Two recent meta-analytic studies identified that consumers respond more favourably to emotional appeals than they do to rational appeals ( Hornik et al. , 2016 , 2017 ).

Effectiveness of different emotional appeals utilized in advertising messages has also received attention. Emotional appeals can be classified as positive, negative or coactive based on the valance of emotion employed. Each emotional valence exerts different effects on judgement and therefore affects perceptions and behaviours differently ( Lerner and Keltner, 2000 ). The literature reports mixed results for advertising effectiveness when it comes to positive versus negative emotional appeals. For example, while fear appeals were found to generate defensive reactions ( Witte and Allen, 2000 ) and result in a boomerang effect for young adults ( Lennon et al. , 2010 ), other studies found negative appeals to be effective in creating behaviour change when compared to positive and neutral appeals ( Struckman‐Johnson et al. , 1994 ; Small and Verrochi, 2009 ; Tay, 2011 ; Sun, 2015 ). Neutral appeals are discussed mainly in charity advertising [see, e.g. ( Small and Verrochi, 2009 )], where positive and negative appeals are compared to neutral (no emotion) ads.

Positive emotional appeals are explored in the literature to a lesser extent reflecting their limited use in advertising campaigns focussed on health prevention and related contexts ( Tay, 2005 ; Dunstone et al. , 2017 ). Inducing positive emotions through advertising messages was found to yield more positive attitudes to the advertisement ( Lau-Gesk and Meyers-Levy, 2009 ), higher liking of the message ( Hornik et al. , 2017 ) and a stronger impact on behaviour than negative emotional appeals in multiple contexts such as safe driving ( Plant et al. , 2017 ), reducing binge drinking among college students ( Lee, 2018 ), encouraging environmental friendly behaviour ( Wang et al. , 2017 ; Skurka et al. , 2018 ), health behaviour ( Jordan et al. , 2015 ; Vaala et al. , 2016 ) and anti-cyber bullying ( Alhabash et al. , 2013 ). However, positive emotional appeals were found to be less effective for highly involved consumers ( Yoon and Tinkham, 2013 ) and for female audiences ( Noble et al. , 2014 ) when compared to low involved and male audiences respectively.

Recently there has been an interest in the literature in the use of coactive emotional appeals that seek to induce both positive and negative emotions simultaneously ( Nabi, 2015 ; Yoon, 2018 ). It is hypothesized that the use of a threat-relief emotional message by combining emotions like fear and humour will result in a stronger persuasion outcome ( Nabi, 2015 ). Positive emotions have the ability to reduce the defensive reactions that negative appeals generate, making them more effective in changing behaviour ( Mukherjee and Dubé, 2012 ; Bennett, 2015 ). Eckler and Bolls and Alhabash et al. found coactive appeals ( Eckler and Bolls, 2011 ; Alhabash et al. , 2013 ) to have a stronger impact than negative appeals but their work also indicated that coactive appeals are weaker than positive appeals. No known systematic or meta-analytic review has synthesized the effectiveness of coactive advertising appeals, signalling the need for a review study.

Emotion can be defined as the psychological reaction to an event, a memory and specific types of media ( Allen et al. , 2005 ). Emotions are usually provoked by an internal stimulus that generates a strong short-term reaction influencing one’s attitudes towards something ( Scherer, 2005 ). Wu et al. report that being exposed to an advertisement ( Wu et al. , 2018 ), even a very short exposure, will induce both strong and weak emotions. The type of emotions used in an advertisement will have different results for the audience. Using neural signal tools like heartrate monitors, Kaye et al. (Kaye et al. , 2016 ) found that negative advertisements stimulate respondents while positive advertisements result in a more relaxed feeling.

RQ1. Which emotional advertising appeal is more effective in creating behaviour change across different contexts?

There is no recent systematic review that looks beyond the context of advertising (e.g. health) and valance of emotions (e.g. fear appeals) to understand the effectiveness of positive versus negative and coactive advertising appeals. The aims of this systematic review study are two-fold. First, to highlight the most effective advertising appeal based on empirical research findings utilizing behavioural (e.g. driving speed) or behavioural proxy (e.g. intentions) measures up to August 2019. Second, this review analyses the quality of published studies in the field based on the Effective Public Health Practice Project (EPHPP) protocol to guide future research.

emotional appeals or emotion* based advertis* AND appeal* AND advertis* or public service announcement or psa or message or communication or strategy or promot* or campaign or experiment

In total, 2384 records were initially identified (see Figure 1 for a flowchart of the search process adopted). Due to the magnitude and focus of each database and its alignment with the search terms, there was variance in the number of records produced from each database. The downloaded records were collated using Endnote. First, all duplicate records were removed leaving a total of 1507 unique records. Second, unqualified records including conference and government reports, unidentifiable full text, as well as records not in English were removed. Finally, titles and abstracts of remaining records were assessed and classified into the exclusion criteria categories: studies using non advertising materials (e.g. news articles), non-emotional based advertising, non-experimental studies (e.g. content analysis and literature reviews), studies exploring only one type of appeal (i.e. negative, positive, mixed or rational), rational versus emotional appeal studies, message framing studies (e.g. gain vs. loss frame), studies lacking behaviour or intention measures of effectiveness.

Systematic search diagram using PRISMA process.

After application of the exclusion criteria, a total of 25 articles undertaking a direct comparative evaluation of the effectiveness of positive and negative appeals were identified. Next, backward and forward searching using authors’ names, Google Scholar and reference lists were completed. A further six articles were identified. In total, 31 articles were analysed. The full list of papers can be found in Supplementary Appendix A .

Data extraction and analysis

The included studies were analysed in terms of (i) the employed materials, (ii) study characteristics and results and, (iii) study quality.

Employed materials and media

Each study’s stimulus was screened to determine the type of media (e.g. video, print, audio), the type of emotion (e.g. fear, guilt, happiness), the target issue (e.g. health behaviour, safe driving, environmental behaviour) and the type of appeals tested (e.g. positive, negative, coactive, rational appeals). This categorized studies based on the type of stimulus used to identify patterns and examine appeal effectiveness.

Study characteristics and results

The 31 identified studies were analysed based on their sample size, sample characteristics (e.g. age and gender), data collection methods (e.g. self-report or objective measures), data collection time points (e.g. post exposure only, pre and post exposure or after a delayed period of time), the employed theory (if any) and mediators and moderator measures of effectiveness. Study outcome measures that were set to warrant inclusion in the review were restricted to behaviour or behavioural intention measures. Studies were excluded if an outcome evaluation was not undertaken to examine advertising effectiveness. For included studies, results were categorized based on the most effective appeal, namely (i) positive, (ii) negative, (iii) no difference/inconclusive or (iv) mixed if positive and negative appeals were found to be effective for different cohorts.

Quality assessment

The quality of the included studies was assessed using the EPHPP quality assessment tool for quantitative studies ( Effective Public Health Practice Project, 2019 ). The EPHPP tool is suitable for evaluating multiple study designs ( Deeks et al. , 2003 ) and has been used to assess the quality of advertising studies in previous reviews ( Becker and Midoun, 2016 ). The assessment tool is valid ( Thomas et al. , 2004 ; Jackson and Waters, 2005 ) and suitable for use in systematic reviews examining effectiveness ( Deeks et al. , 2003 ). Each study was rated using six EPHPP criteria: (i) selection bias—how representative the sample is of the target population; (ii) study design—the likelihood of bias due to the allocation process in the study; (iii) confounders—the extent to which groups were balanced at baseline with respect to confounding variables; (iv) blinding—whether participants were aware of the study objectives and researchers participating in the study were aware of each group participation status; (v) data collection—whether study measures were valid and reliable and (vi) withdrawals and drop outs—the percentage of participants remaining in the study at the final data collection period in all groups ( Thomas et al. , 2004 ). Each individual aspect is rated as weak, moderate or strong and an overall rating is applied to each study ( Thomas et al. , 2004 ). All studies assessed through the EHPHH tool were rated by two researchers and inter-reliability scores exceeded the 80% threshold. Discrepancies were discussed and resolved with all three authors.

Description of included studies

In total, 31 studies qualified for inclusion. More than half of studies were from the United States [ n = 18; e.g. ( Alhabash et al. , 2013 ; Bleakley et al. , 2015 )], followed by Australia [ n = 5; e.g. ( Noble et al. , 2014 ; Kaye et al. , 2016 )], the rest ( n = 8) were from Canada ( Tay, 2011 ), United Kingdom ( Eckler and Bolls, 2011 ), Germany ( Jäger and Eisend, 2013 ), Belgium ( Faseur and Geuens, 2010 ), Netherlands ( Hendriks et al. , 2014 ), China ( Wang et al. , 2017 ), Taiwan ( Wu et al. , 2018 ) and South Korea ( Sun, 2015 ) (see Figure 2 for study locations). Most studies addressed social issues ( n = 28) such as safe driving ( Lewis et al. , 2008 ; Taute et al. , 2011 ; Tay, 2011 ; Jäger and Eisend, 2013 ; Previte et al. , 2015 ; Kaye et al. , 2016 ; Plant et al. , 2017 ), charity donations ( Small and Verrochi, 2009 ; Faseur and Geuens, 2010 ; Kemp et al. , 2013 ; Cao and Jia, 2017 ; Zemack-Rugar and Klucarova-Travani, 2018 ), health ( Struckman‐Johnson et al. , 1994 ; Lee and Ferguson, 2002 ; Passyn and Sujan, 2006 ; Hendriks et al. , 2014 ; Bleakley et al. , 2015 ; Jordan et al. , 2015 ; Vaala et al. , 2016 ; Thainiyom and Elder, 2017 ; Lee, 2018 ), the environment ( Yoon and Tinkham, 2013 ; Noble et al. , 2014 ; Wang et al. , 2017 ; Skurka et al. , 2018 ), organ donation ( Rodrigue et al. , 2014 ; Sun, 2015 ) and cyberbullying ( Alhabash et al. , 2013 ). Three studies were undertaken in commercial settings with authors examining toothbrush, influenza vaccine, alcohol, cars and insurance advertisements ( Brooker, 1981 ; Eckler and Bolls, 2011 ; Wu et al. , 2018 ) (see Figure 3 ).

Location of included studies.

Studies by targeted issue.

Most studies looked at positive versus negative advertising appeals [ n = 19; e.g. ( Kemp et al. , 2013 ; Kaye et al. , 2016 )], only two studies included positive, negative and coactive appeals ( Eckler and Bolls, 2011 ; Alhabash et al. , 2013 ), while the rest incorporated a rational [ n = 8; e.g. ( Sun, 2015 ; Skurka et al. , 2018 )] or neutral appeal [ n = 2 ( Small and Verrochi, 2009 ; Zemack-Rugar and Klucarova-Travani, 2018 ); see Figure 4 ] in their tests. In terms of emotions, fear versus humour was most frequently examined with 12 (38%) studies comparing the two emotions [e.g. ( Tay, 2011 ; Vaala et al. , 2016 )]. Of all tested emotional appeals, fear was the most studied appeal (48%) followed by humour (45%). Positive emotions such as pride ( Kemp et al., 2013 ; Noble et al. , 2014 ; Wang et al. , 2017 ), hope ( Rodrigue et al. , 2014 ; Thainiyom and Elder, 2017 ), love ( Previte et al. , 2015 ) and a range of negative emotions such as disgust ( Hendriks et al. , 2014 ), anger ( Rodrigue et al. , 2014 ), shame ( Previte et al. , 2015 ), regret ( Taute et al. , 2011 ) and guilt ( Noble et al. , 2014 ) were also considered. Seven studies did not specify which positive and negative emotions were tested ( Faseur and Geuens, 2010 ; Eckler and Bolls, 2011 ; Alhabash et al. , 2013 ; Sun, 2015 ; Kaye et al. , 2016 ; Plant et al. , 2017 ; Wu et al. , 2018 ).

Studies by the tested appeals.

Only three studies utilized objective data collection tools. Objective outcome data included GPS speed trackers ( Kaye et al. , 2016 ), driving stimulators ( Plant et al. , 2017 ) and donation amounts ( Small and Verrochi, 2009 ). The rest of the studies relied on self-reported measures [ n = 28; e.g. ( Jäger and Eisend, 2013 ; Skurka et al. , 2018 ; Wu et al. , 2018 )]. The majority of studies ( n = 24) collected data post exposure only [e.g. ( Taute et al. , 2011 ; Sun, 2015 )]. Four studies included a post exposure and a follow-up data collection time point after a delayed period of time ( Passyn and Sujan, 2006 ; Lewis et al. , 2008 ; Hendriks et al. , 2014 ; Plant et al. , 2017 ). Two studies collected data pre and post exposure ( Rodrigue et al. , 2014 ; Previte et al. , 2015 ) and only one study collected data at pre, post and follow-up time points ( Kaye et al. , 2016 ).

Only 35% of studies were guided by theories. Theories that were reported included the Elaboration Likelihood Model ( Lewis et al. , 2008 ), Extended Parallel Process Model ( Tay, 2011 ), Theory of Planned Behaviour ( Hendriks et al. , 2014 ), Affect as Information Theory ( Taute et al. , 2011 ) and other theories (see Supplementary Appendix A ).

Study outcomes

The aim of this systematic review was to highlight effective advertising appeals. This is based on the ability of the appeal to influence behaviour or behavioural intentions significantly ( P < 0.05) in the desired direction (e.g. reduce drink driving). The results of the 31 included studies indicate that positive advertising appeals are slightly more effective than negative and coactive advertising appeals. It is important to note there is evidence of effectiveness for all appeal types and each context and target audience differ in appeal effectiveness requiring pre-testing and examination prior appeal consideration. Thirty-five per cent ( n = 11) of studies reported positive appeals to be more effective ( Brooker, 1981 ; Eckler and Bolls, 2011 ; Alhabash et al. , 2013 ; Rodrigue et al. , 2014 ; Previte et al. , 2015 ; Sun, 2015 ; Plant et al. , 2017 ; Wang et al. , 2017 ; Lee, 2018 ; Wu et al. , 2018 ; Zemack-Rugar and Klucarova-Travani, 2018 ), while 26% ( n = 8) reported negative appeals to have a stronger persuasion effect than positive appeals ( Struckman‐Johnson et al. , 1994 ; Small and Verrochi, 2009 ; Tay, 2011 ; Hendriks et al. , 2014 ; Noble et al. , 2014 ; Bleakley et al. , 2015 ; Kaye et al. , 2016 ). Nineteen per cent of studies ( n = 6) showed mixed results. Where mixed results were reported the mixed outcomes occurred as a result of range of factors including gender ( Kemp et al. , 2013 ; Thainiyom and Elder, 2017 ), connection to others ( Faseur and Geuens, 2010 ), prior attitudes ( Jäger and Eisend, 2013 ), time of assessment after exposure ( Lewis et al. , 2008 ), issue involvement ( Yoon and Tinkham, 2013 ) and psychological involvement ( Cao and Jia, 2017 ). Five studies (16%) did not find any significant differences in effectiveness between positive and negative appeals ( Passyn and Sujan, 2006 ; Thainiyom and Elder, 2017 ; Skurka et al. , 2018 ). Finally, only one study reported inconclusive results due to unrepresentative sample ( Lee and Ferguson, 2002 ). Figure 5 showcase results of the included studies.

Results supporting different appeals effectiveness or reporting mixed, indifferent or inconclusive results.

A quality assessment of the identified papers was conducted using the EPHPP tool (see Supplementary Appendix B ). Of the 31 included studies, 26 were assessed as weak in the global rating, five were assessed as moderate and none were assessed as strong. Selection bias was likely in many studies due to the use of student samples or bias to a geographical area. Only one study was somewhat likely to have a representative sample ( Skurka et al. , 2018 ). Five studies included a control group and randomly allocated participants into experimental groups (e.g. positive and negative stimuli) therefore these were assessed as strong in terms of study design ( Struckman‐Johnson et al. , 1994 ; Bleakley et al. , 2015 ; Jordan et al. , 2015 ; Vaala et al., 2016 ; Skurka et al. , 2018 ). Six were assessed as moderate ( Lewis et al. , 2008 ; Hendriks et al. , 2014 ; Previte et al. , 2015 ; Sun, 2015 ; Kaye et al. , 2016 ; Plant et al. , 2017 ), while the rest ( n = 20) were weak due to their cross sectional nature [e.g. ( Alhabash et al. , 2013 ; Jäger and Eisend, 2013 )].

In terms of confounders, about one-third of studies ( n = 10, 32%) reported either no baseline differences between groups or controlled for at least 80% of relevant confounders resulting in a strong rating. The rest of the studies ( n = 21) did not report potential confounders or account for confounds during analysis and were therefore assessed as weak [e.g. ( Alhabash et al. , 2013 ; Bleakley et al. , 2015 ; Lee, 2018 )]. Only two studies (10%) clearly reported that both the assessors and participants were not blinded in the experiment resulting in a weak rating ( Rodrigue et al. , 2014 ; Plant et al. , 2017 ). The rest of the studies ( n = 29, 87%) were rated as moderate as it was not clear if the participants and assessors were blinded or not. In terms of data collection methods, over half of the included studies ( n = 19, 61%) did not provide evidence of the validity of the reported measures and were therefore assessed as weak.

Two studies were assessed as moderate in their data collection method as they reported on validity but not reliability of the measures ( Jordan et al. , 2015 ; Plant et al. , 2017 ), while the rest ( n = 10, 32%) were rated strong for providing evidence of the validity and reliability of the reported outcomes measures [e.g. ( Kemp et al. , 2013 ; Noble et al. , 2014 ; Sun, 2015 ; Kaye et al. , 2016 )]. For the retention rate of participants, only two programs were assessed as strong with more than 80% completing the experiment ( Kaye et al. , 2016 ; Plant et al. , 2017 ). The rest were rated as moderate due to the lack of retention rate reporting [e.g. ( Jäger and Eisend, 2013 )], low completion rate [e.g. ( Rodrigue et al. , 2014 ; Jordan et al. , 2015 )] or due to the post exposure nature of studies where retention rate is not applicable [e.g. ( Faseur and Geuens, 2010 )].

The aims of this study were two-fold. This study aimed to identify which appeal type (positive, negative and/or coactive) was most likely to change social and commercial behaviour and to assess the quality of studies reported in peer review literature. This is the first known systematic review that is not limited to an emotion, appeal type, context or media. Our findings extend understanding in three key ways. First, this article extends understanding of appeal effectiveness with consideration of the effectiveness of coactive appeals. Second, it examines the extent of theory and emotion use in the included studies. Third, it assesses study quality identifying how researchers can enhance the evidence base by improving study quality.

Positive, negative or coactive?

Consistent with the literature ( Jenkin et al. , 2014 ; Hornik et al. , 2016 ) our findings confirm a slight persuasive advantage of positive advertising appeals over negative appeals. Positive appeals are able to increase consumers’ perceived response efficacy more than negative appeals ( Zemack-Rugar and Klucarova-Travani, 2018 ); help consumers realize the rewards of the promoted behaviour [e.g. moderate alcohol consumption ( Previte et al. , 2015 )]; induce positive attitudes—more than negative and coactive appeals—and therefore affect behavioural intentions positively ( Eckler and Bolls, 2011 ; Wang et al. , 2017 ). According to studies synthesized in the present review, positive appeals yield higher acceptance of the advertising message ( Alhabash et al. , 2013 ) by creating a positive climate in which messages may be received ( Brooker, 1981 ), reducing reactance [e.g. skipping, ignoring, backlash or resisting ( Wu et al. , 2018 )] and increasing message liking ( Lee, 2018 ). Further outcomes accruing from positive appeals include illustration of positive benefits of the promoted behaviour by inducing empathy and reducing guilt ( Rodrigue et al. , 2014 ).

Negative appeals dominate social change practice and while evidence for effectiveness exists, there appears to be less support in comparison to positive and coactive appeals based on this study’s findings. Mixed results were also evident in other studies. For example, Kemp et al. argued that positive appeals are ( Kemp et al. , 2013 ) more persuasive with a male audience than a female audience, while Jäger and Eisend found participants with less ( Jäger and Eisend, 2013 ) favourable prior attitudes produce higher change in intentions to drink drive when exposed to positive emotional appeals.

The effectiveness of coactive appeals compared to single appeals was examined by 2 of the 31 included studies. Their findings suggest coactive appeals are less effective than positive appeals and more effective than negative appeals ( Eckler and Bolls, 2011 ; Alhabash et al. , 2013 ). Positive appeals require less cognitive processing, generate a general sense of pleasantness, are more likable and facilitate positive attitudes towards the advertisement making the advertised behaviour more appealing and taking action more tempting. On the contrary, the more negative an ad is, the less likable it is and the less likely viewers are to take action (i.e. share on social media). Therefore, coactive emotional appeals come in the middle, they are more effective than negative appeals but less effective than positive appeals ( Alhabash et al. , 2013 ). Interestingly, the two studies including coactive appeals in their experiments focused on viral sharing behaviour. Taking the target behaviour in consideration, their results can be interpreted more specifically. Previous studies found both emotional valence and arousal to affect content sharing and virality of advertisements ( Berger, 2011 ; Berger and Milkman, 2012 ). More specifically, content that are emotionally arousing (either positive or negative) are more likely to be shared with others than those less arousing. Furthermore, ads that are more positive in nature are more likely to be shared than negative ads ( Berger and Milkman, 2012 ). Moreover, the use of positive emotions along with negative emotions helps reduce the defensive responses of the audience resulting in a higher persuasion effect ( Mukherjee and Dubé, 2012 ). Hence, the studies included in this systematic review found coactive appeals to be more effective than negative appeals. When testing behaviour beyond sharing and virality, Yousef et al (2021) found positive appeals and coactive appeals to have similar effect on behaviour. Target audience plays a role in different appeals effectiveness, including coactive appeals. Studying advertising effect on young adults road safety perceptions and behaviour intentions, Yousef et al (2021) found coactive appeals to be more effective than single emotional appeals. The limited and mixed evidence for coactive appeals effectiveness is mainly due to the limited studies including such appeals in their experiments. More evidence is needed to determine coactive appeals effectiveness in other contexts and behaviours.

Applying theories and moving beyond fear and humour appeals

Over the years, advertising researchers have been under pressure to deliver relevant and practical findings that practitioners can follow and utilize ( Pitt et al. , 2005 ). It is argued that advertising research has formulated theories with ‘a high level of generality’ which makes them difficult to apply in practice ( Cornelissen and Lock, 2002 ). As a corollary, and due to the empirical nature of the included studies, these issues may have led to the limited application of theoretical frameworks. Pitt et al. (Pitt et al. , 2005 ) found only a minority of papers published in an 11-year period made explicit use of theories. Our findings confirm their research with more than half of the included studies lacking a theoretical base. Examples exist indicating how and where theory has been applied by researchers in intervention design, recruitment, implementation and evaluation [see ( Willmott et al. , 2019 )]. For example, Wadsworth and Hallam (Wadsworth and Hallam, 2010 ) applied social cognitive theory to an e-communication intervention identifying which theoretical constructs led to a physical activity increase. Theory did not only inform their study but was tested, refined and built on by the authors. This type of theory application can enhance study outcomes, better inform future research and systematically identify which theories are effective and for which audiences ( Willmott et al. , 2019 ).

Similarly, limited studies explored emotions beyond the heavily investigated emotions of fear and humour. Little is known about how other emotions effectiveness such as anger, disgust, guilt, love, joy and pride appeals deliver (or not) behavioural change. This reinforces past studies which have identified the limited use of emotions in advertising messages ( Tay, 2005 ; Dunstone et al. , 2017 ), not because other emotions are less effective but because there is limited evidence of effectiveness. When studies explore more emotions, new evidence emerges enabling practitioners to innovate and capture the attention of their audience. For example, Previte et al. ( Previte et al. , 2015 ) found a persuasive advantage for love and happiness (two emotions that are rarely examined in the advertising literature) over fear and shame appeals in moderate drinking advertising message, highlighting the potential of other emotions to yield desired results.

Enhancing study rigour to deliver a stronger evidence base for advertising effectiveness

Study quality assessment frameworks provide tools to assess the quality of research. The stronger the study, the more the policy, practitioner and research community can rely on the study findings. This study applied the EPHPP quality assessment tool ( Effective Public Health Practice Project, 2019 ) to assess study quality. Of particular concern is that no one study overall was rated as strong in the current review. In general, the methodological quality of the included studies was low. In the absence of strong evidence any conclusions drawn in the present evidence review and earlier meta-analytic and systematic literature reviews should be interpreted with caution until stronger study designs emerge. Within the present review notable, methodological problems included selection biases, weak study designs and invalid data collection methods.

A common issue with sampling is the use of student samples and samples from a specific region for convenience, resulting in selection bias. While calls for adoption of probability sampling procedures in the academic literature have been made ( Plant et al. , 2011 ; Sarstedt et al. , 2018 ), limited adoption of non-probability sampling is evident. In the absence of replication across samples or regions his reduces the generality of these studies making them bound to their sample and regional characteristics. Furthermore, the use of cross-sectional study designs contributed to the overall weak rating for most studies in this review. Including only a post-test immediately after exposure to the tested advertisements can lead to different result compared to testing over a delayed period of time ( Lewis et al. , 2008 ) making the results incomplete and the findings less comprehensive. Researchers are encouraged to include more than one time point for data collection to measure behaviour change over time. Finally, the validity and reliability of data collection methods used in the included studies are mostly weak. This is a reflection of the limited use of theories, with more studies bringing in their own measures without testing their validity or reliability before conducting their evaluations. Future research should focus on increasing the validity of their studies by utilizing previously validated measures from the literature ( David and Rundle-Thiele, 2018 ). This makes the study easier to replicate and its findings more reliable. Taken together, future research should aim to address these issues and improve the methodological quality of advertising evaluation studies to enhance empirical evidence.

Limitations

This study is restricted by several important limitations, which should be considered when interpreting the findings. First, the study is limited by the search parameters utilized and the study quality frameworks applied. For example, the review only includes studies that empirically test advertising appeal effectiveness (positive, negative, coactive), using behavioural measures (e.g. purchase intentions) that have been published in peer-reviewed English literature. Hence studies that rely on other measures (e.g. attitudes) or evaluate other message tactics (e.g. framing) and non-English and non-peer-reviewed studies, were excluded. Grey literature may contribute important information and future studies may benefit from examining these sources. The study focused mainly on emotional appeals, hence rational appeals were not included. Future reviews should compare rational and emotional appeals for more comprehensive findings. Second, due to the heterogeneity in the tested appeals, study populations and reporting of results, a meta-analysis was not possible, and a qualitative description of study outcomes was provided. Few studies included effect sizes and odds ratios, limiting our ability to compare effectiveness for the different advertising appeals. Third, results of the current review are collected from different contexts and behaviours and generalization of findings cannot be extended beyond this review. Moreover, pre-tests should be carried out before adopting any advertising appeal for any specific context, behaviour and target audience. Finally, based on the quality assessment of the included studies there is a clear absence of strong rigour experiments, hence any conclusions drawn in the present review should be interpreted with caution.

Future research

Future research should examine appeals effectiveness by utilising and applying advertising theories, investigating emotions beyond fear and humour in advertising appeals, increase the strength of their studies by following EPHPP guidelines, or other study quality frameworks, to design rigorous experiments and ensure that valid replicable analysis is reported. More effort should be made to draw representative samples, ensuring valid data collection methods and designing strong experiments that test effectiveness pre, post and after a delayed period of time following exposure. Furthermore, more studies should include coactive appeals in their evaluations to confirm their effectiveness compared to single appeal use as only a limited number of studies explored this type of appeal.

Future systematic literature reviews should build on this study by including other advertising tactics such as non-emotional appeals and gain and loss framing which can provide a wider picture of advertising effectiveness. Moving forward, consensus on advertising effectiveness outcome measures should be generated by the advertising research community. By agreeing on standard outcome measures, as occurs in tobacco control research, the research community could then advance understanding further via meta-analyses. Any effort that can reduce data transformation practices will serve to ensure synthesis studies can advance knowledge through delivery of the highest quality research that can inform policy and advertising practices.

This systematic review examined advertising appeals effectiveness based on the literature up to August 2019. Our findings support previous meta-analytic reviews in confirming positive appeals effectiveness over negative appeals. We extend on their findings however by including coactive advertising appeals. Across different contexts and behaviours, this review found positive appeals to be effective more often than negative appeals and coactive appeals. When all three appeals are studied, evidence suggest coactive appeals are more effective than negative appeals and less effective than positive appeals. Specifically, this review highlighted the scarce of theory use in advertising research signalling the need for more attention to embed theory into advertising design and evaluation. Moreover, a major concern raised by this review is the quality of the published papers. A greater focus should be made by authors to utilize valid data collection methods, representative samples and strong study designs. This research has contributed to a better understanding of advertising appeal effectiveness and may be of interest to policy makers, advertising professionals and designers and researchers who are interested in maximizing return on investment.

Supplementary material is available at Health Promotion International online.

Akpinar E. , Berger J. ( 2017 ) Viral marketing works best with emotional appeals . Journal of Marketing Research , 54 , 318 – 330 .

Google Scholar

Alhabash S. , McAlister A. R. , Hagerstrom A. , Quilliam E. T. , Rifon N. J. , Richards J. I. ( 2013 ) Between likes and shares: effects of emotional appeal and virality on the persuasiveness of anticyberbullying messages on Facebook . Cyberpsychology, Behavior and Social Networking , 16 , 175 – 182 .

Allen C. T. , Machleit K. A. , Schultz Kleine S. , Sahni Notani A. ( 2005 ) A place for emotion in attitude models . Journal of Business Research , 58 , 494 – 499 .

Armstrong J. S. ( 2010 ) Persuasive Advertising: Evidence-Based Principles . Palgrave Macmillan, London .

Google Preview

Becker S. J. , Midoun M. M. ( 2016 ) Effects of direct-to-consumer advertising on patient prescription requests and physician prescribing: a systematic review of psychiatry-relevant studies . The Journal of Clinical Psychiatry , 77 , e1293 – e1300 .

Bennett R. ( 2015 ) Individual characteristics and the arousal of mixed emotions: consequences for the effectiveness of charity fundraising advertisements . International Journal of Nonprofit & Voluntary Sector Marketing , 20 , 155 – 209 .

Berger J. ( 2011 ) Arousal increases social transmission of information . Psychological Science , 22 , 891 – 893 .

Berger J. , Milkman K. L. ( 2012 ) What makes online content viral? Journal of Marketing Research , 49 , 192 – 205 .

Bleakley A. , Jordan A. B. , Hennessy M. , Glanz K. , Strasser A. , Vaala S. ( 2015 ) Do emotional appeals in public service advertisements influence adolescents’ intention to reduce consumption of sugar-sweetened beverages? Journal of Health Communication , 20 , 938 – 948 .

Brooker G. ( 1981 ) A comparison of the persuasive effects of mild humor and mild fear appeals . Journal of Advertising , 10 , 29 – 40 .

Cao X. , Jia L. ( 2017 ) The effects of the facial expression of beneficiaries in charity appeals and psychological involvement on donation intentions . Nonprofit Management and Leadership , 27 , 457 – 473 .

Cornelissen J. P. , Lock A. R. ( 2002 ) Advertising research and its influence on managerial practice . Journal of Advertising Research , 42 , 50 – 55 .

Dahlen M. , Lange F. , Smith T. ( 2010 ) Marketing communications: a brand narrative approach. John Wiley & Sons, United States, New Jersey.

David P. , Rundle-Thiele S. ( 2018 ) Social marketing theory measurement precision: a theory of planned behaviour illustration . Journal of Social Marketing , 8 , 182 – 201 .

Deeks J. J. , Dinnes J. , D’Amico R. , Sowden A. J. , Sakarovitch C. , Song F. , et al. ; European Carotid Surgery Trial Collaborative Group . ( 2003 ) Evaluating non-randomised intervention studies . Health Technology Assessment (Winchester, England) , 7 , iii, 1 – 173 .

Dunstone K. , Brennan E. , Slater M. D. , Dixon H. G. , Durkin S. J. , Pettigrew S. et al.  ( 2017 ) Alcohol harm reduction advertisements: a content analysis of topic, objective, emotional tone, execution and target audience . BMC Public Health , 17 , 13 .

Eckler P. , Bolls P. ( 2011 ) Spreading the virus: emotional tone of viral advertising and its effect on forwarding intentions and attitudes . Journal of Interactive Advertising , 11 , 1 – 11 .

Effective Public Health Practice Project . ( 2019 ) Quality assessment tool for quantitative studies. http://www.ephpp.ca/tools.html .

Elder R. W. , Shults R. A. , Sleet D. A. , Nichols J. L. , Thompson R. S. , Rajab W , et al.  ( 2004 ) Effectiveness of mass media campaigns for reducing drinking and driving and alcohol-involved crashes: a systematic review . American Journal of Preventive Medicine , 27 , 57 – 65 .

Esrick J. , Kagan R. G. , Carnevale J. T. , Valenti M. , Rots G. , Dash K. ( 2019 ) Can scare tactics and fear-based messages help deter substance misuse: a systematic review of recent (2005–2017) research, Drugs-Education Prevention and Policy , 26 , 209 – 218 .

Estrada Y. , Lee T. K. , Huang S. , Tapia M. I. , Velazquez M. R. , Martinez M. J. et al.  ( 2017 ) Parent-centered prevention of risky behaviors among hispanic youths in Florida . American Journal of Public Health , 107 , 607 – 613 .

Faseur T. , Geuens M. ( 2010 ) Communicating the right emotion to generate help for connected versus unconnected others . Communication Research , 37 , 498 – 521 .

Hendriks H. , van den Putte B. , de Bruijn G. J. ( 2014 ) Changing the conversation: the influence of emotions on conversational valence and alcohol consumption . Prevention Science , 15 , 684 – 693 .

Hornik J. , Ofir C. , Rachamim M. ( 2016 ) Quantitative evaluation of persuasive appeals using comparative meta-analysis . The Communication Review , 19 , 192 – 222 .

Hornik J. , Ofir C. , Rachamim M. ( 2017 ) Advertising appeals, moderators, and impact on persuasion: a quantitative assessment creates a hierarchy of appeals . Journal of Advertising Research , 57 , 305 – 318 .

Jackson N. , Waters E ; Promotion Guidelines for Systematic Reviews in Health and Taskforce Public Health . ( 2005 ) Criteria for the systematic review of health promotion and public health interventions . Health Promotion International , 20 , 367 – 374 .

Jäger T. , Eisend M. ( 2013 ) Effects of fear-arousing and humorous appeals in social marketing advertising: the moderating role of prior attitude toward the advertised behavior . Journal of Current Issues & Research in Advertising , 34 , 125 – 134 .

Jenkin G. , Madhvani N. , Signal L. , Bowers S. ( 2014 ) A systematic review of persuasive marketing techniques to promote food to children on television . Obesity Reviews , 15 , 281 – 293 .

Jordan A. , Bleakley A. , Hennessy M. , Vaala S. , Glanz K. , Strasser A. A. ( 2015 ) Sugar-sweetened beverage-related public service advertisements and their influence on parents . American Behavioral Scientist , 59 , 1847 – 1865 .

Kaye S. A. , Lewis I. , Algie J. , White M. J. ( 2016 ) Young drivers’ responses to anti-speeding advertisements: comparison of self-report and objective measures of persuasive processing and outcomes . Traffic Injury Prevention , 17 , 352 – 358 .

Kemp E. , Kennett-Hensel P. A. , Kees J. ( 2013 ) Pulling on the heartstrings: examining the effects of emotions and gender in persuasive appeals . Journal of Advertising , 42 , 69 – 79 .

Lau-Gesk L. , Meyers-Levy J. ( 2009 ) Emotional persuasion: when the valence versus the resource demands of emotions influence consumers’ attitudes . Journal of Consumer Research , 36 , 585 – 599 .

Lee M. J. ( 2018 ) College students’ responses to emotional anti-alcohol abuse media messages: should we scare or amuse them? Health Promotion Practice , 19 , 465 – 474 .

Lee M. J. , Ferguson M. A. ( 2002 ) Effects of anti-tobacco advertisements based on risk-taking tendencies: realistic fear vs. vulgar humor . Journalism & Mass Communication Quarterly , 79 , 945 – 963 .

Lennon R. , Rentfro R. , Bay O. ( 2010 ) Social marketing and distracted driving behaviors among young adults: the effectiveness of fear appeals . Academy of Marketing Studies Journal , 14 , 95 – 113 .

Lerner J. S. , Keltner D. ( 2000 ) Beyond valence: Toward a model of emotion-specific influences on judgement and choice . Cognition & Emotion , 14 , 473 – 493 .

Lewis I. , Watson B. , White K. M. ( 2008 ) An examination of message-relevant affect in road safety messages: should road safety advertisements aim to make us feel good or bad? Transportation Research Part F: Traffic Psychology and Behaviour , 11 , 403 – 417 .

Matthes J. , Wonneberger A. ( 2014 ) The skeptical green consumer revisited: testing the relationship between green consumerism and skepticism toward advertising . Journal of Advertising , 43 , 115 – 127 .

Mattila A. S. ( 1999 ) Do emotional appeals work for services? International Journal of Service Industry Management , 10 , 292 – 306 .

Moher D. , Liberati A. , Tetzlaff J. , Altman D. G. ; PRISMA Group . ( 2009 ) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement . Annals of Internal Medicine , 151 , 264 – 269 .

Moran N. , Bagchi R. ( 2019 ) The power of emotional benefits: examining the role of benefit focus on donation behavior . Journal of Advertising , 48 , 284 – 291 .

Mukherjee A. , Dubé L. ( 2012 ) Mixing emotions: the use of humor in fear advertising . Journal of Consumer Behaviour , 11 , 147 – 161 .

Nabi R. L. ( 2015 ) Emotional flow in persuasive health messages . Health Communication , 30 , 114 – 124 .

Noble G. , Pomering A. , W. Johnson L. ( 2014 ) Gender and message appeal: their influence in a pro-environmental social advertising context . Journal of Social Marketing , 4 , 4 – 21 .

O’Keefe D. J. , Jensen J. D. ( 2009 ) The relative persuasiveness of gain-framed and loss-framed messages for encouraging disease detection behaviors: a meta-analytic review . Journal of Communication , 59 , 296 – 316 .

Passyn K. , Sujan M. ( 2006 ) Self accountability emotions and fear appeals: motivating behavior . Journal of Consumer Research , 32 , 583 – 589 .

Pitt L. F. , Berthon P. , Caruana A. , Berthon J.-P. ( 2005 ) The state of theory in three premier advertising journals: a research note . International Journal of Advertising , 24 , 241 – 249 .

Plant B. , Reeza F. , Irwin J. D. ( 2011 ) A systematic review of how anti-speeding advertisements are evaluated . Journal of the Australasian College of Road Safety , 22 , 18 .

Plant B. R. C. , Irwin J. D. , Chekaluk E. ( 2017 ) The effects of anti-speeding advertisements on the simulated driving behaviour of young drivers . Accident; Analysis and Prevention , 100 , 65 – 74 .

Previte J. , Russell-Bennett R. , Parkinson J. ( 2015 ) Shaping safe drinking cultures: evoking positive emotion to promote moderate-drinking behaviour . International Journal of Consumer Studies , 39 , 12 – 24 .

Rodrigue J. R. , Fleishman A. , Vishnevsky T. , Fitzpatrick S. , Boger M. ( 2014 ) Organ donation video messaging: differential appeal, emotional valence, and behavioral intention . Clinical Transplantation , 28 , 1184 – 1192 .

Sarstedt M. , Bengart P. , Shaltoni A. M. , Lehmann S. ( 2018 ) The use of sampling methods in advertising research: a gap between theory and practice . International Journal of Advertising , 37 , 650 – 663 .

Scherer K. R. ( 2005 ) What are emotions? And how can they be measured? Social Science Information , 44 , 695 – 729 .

Skurka C. , Niederdeppe J. , Romero-Canyas R. , Acup D. ( 2018 ) Pathways of influence in emotional appeals: benefits and tradeoffs of using fear or humor to promote climate change-related intentions and risk perceptions . Journal of Communication , 68 , 169 – 193 .

Small D. A. , Verrochi N. M. ( 2009 ) The face of need: facial emotion expression on charity advertisements . Journal of Marketing Research , 46 , 777 – 787 .

Struckman‐Johnson C. , Struckman‐Johnson D. , Gilliland R. C. , Ausman A. ( 1994 ) Effect of persuasive appeals in AIDS PSAs and condom commercials on intentions to use condoms 1, Journal of Applied Social Psychology , 24 , 2223 – 2244 .

Sun H. J. ( 2015 ) A study on the development of public campaign messages for organ donation promotion in Korea . Health Promotion International , 30 , 903 – 918 .

Tannenbaum M. B. , Hepler J. , Zimmerman R. S. , Saul L. , Jacobs S. , Wilson K. et al.  ( 2015 ) Appealing to fear: a meta-analysis of fear appeal effectiveness and theories . Psychological Bulletin , 141 , 1178 – 1204 .

Taute H. A. , McQuitty S. , Sautter E. P. ( 2011 ) Emotional information management and responses to emotional appeals . Journal of Advertising , 40 , 31 – 43 .

Tay R. ( 2005 ) The effectiveness of enforcement and publicity campaigns on serious crashes involving young male drivers: are drink driving and speeding similar? Accident; Analysis and Prevention , 37 , 922 – 929 .

Tay R. ( 2011 ) Drivers’ perception of two seatbelt wearing advertisements with different emotional appeals and cultural settings . Journal of the Australasian College of Road Safety , 22 , 82 .

Thainiyom P. , Elder K. ( 2017 ) Emotional Appeals in HIV Prevention Campaigns: unintended stigma effects . American Journal of Health Behavior , 41 , 390 – 400 .

Thomas B. H. , Ciliska D. , Dobbins M. , Micucci S. ( 2004 ) A process for systematically reviewing the literature: providing the research evidence for public health nursing interventions . Worldviews on Evidence-Based Nursing , 1 , 176 – 184 .

Vaala S. E. , Bleakley A. , Hennessy M. , Jordan A. B. ( 2016 ) Weight stigmatization moderates the effects of sugar-sweetened beverage-related PSAs among US parents . Media Psychology , 19 , 534 – 560 .

Wadsworth D. D. , Hallam J. S. ( 2010 ) Effect of a web site intervention on physical activity of college females . American Journal of Health Behavior , 34 , 60 – 69 .

Wang J. M. , Bao J. , Wang C. C. , Wu L. C. ( 2017 ) The impact of different emotional appeals on the purchase intention for green products: The moderating effects of green involvement and Confucian cultures . Sustainable Cities and Society , 34 , 32 – 42 .

Williams C. L. , Grechanaia T. , Romanova O. , Komro K. A. , Perry C. L. , Farbakhsh K. ( 2001 ) Russian-American partners for prevention: adaptation of a school-based parent-child programme for alcohol use prevention . European Journal of Public Health , 11 , 314 – 321 .

Williams J. D. , Lee W.-N. , Haugtvedt C. P. ( 2004 ) Diversity in Advertising: Broadening the Scope of Research Directions . Psychology Press, Hove, East Sussex, United Kingdom .

Willmott T. , Pang B. , Rundle-Thiele S. , Badejo A. ( 2019 ) Reported theory use in electronic health weight management interventions targeting young adults: a systematic review . Health Psychology Review , 13 , 295 – 317 .

Witte K. , Allen M. ( 2000 ) A meta-analysis of fear appeals: implications for effective public health campaigns . Health Education & Behavior , 27 , 591 – 615 .

Wu C. H. , Sundiman D. , Kao S. C. , Chen C. H. ( 2018 ) Emotion induction in click intention of picture advertisement: a field examination . Journal of Internet Commerce , 17 , 356 – 382 .

Yoon H. J. ( 2018 ) Using humour to increase effectiveness of shameful health issue advertising: testing the effects of health worry level . International Journal of Advertising , 37 , 914 – 936 .

Yoon H. J. , Tinkham S. F. ( 2013 ) Humorous threat persuasion in advertising: The effects of humor, threat intensity, and issue involvement . Journal of Advertising , 42 , 30 – 41 .

Yousef M. , Dietrich T. , Rundle-Thiele S. ( 2021 ) Social advertising effectiveness in driving action: a study of positive, negative and coactive appeals on social media . International Journal of Environmental Research and Public Health , 18 , 5954 .

Yousef M. , Dietrich T. , Torrisi G. ( 2021 ) Positive, negative or both? Assessing emotional appeals effectiveness in anti-drink driving advertisements . Social Marketing Quarterly , 27 , 195 – 212 .

Zemack-Rugar Y. , Klucarova-Travani S. ( 2018 ) Should donation ads include happy victim images? The moderating role of regulatory focus . Marketing Letters , 29 , 421 – 434 .

Supplementary data

Email alerts, citing articles via.

• Recommend to Your Librarian
• Journals Career Network

Affiliations

• Online ISSN 1460-2245
• Print ISSN 0957-4824
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Type your tag names separated by a space and hit enter

Respiratory Syncytial Virus Sequelae Among Adults in High-Income Countries: A Systematic Literature Review and Meta-analysis. Infect Dis Ther . 2024 May 06 [Online ahead of print] ID

Introduction.

Respiratory syncytial virus (RSV) can cause severe respiratory infections in adults; however, information on associated sequelae is limited. This systematic literature review aimed to identify sequelae in adults within 1 year following RSV-related hospitalization or resolution of acute infection.

Studies were identified from Embase, MEDLINE, LILACS, SciELO, and grey literature. Random-effects meta-analyses using restricted maximum likelihood were used to calculate the proportions and relative risks of sequelae in patients with RSV compared with controls (patients with RSV-negative influenza-like illness, influenza, and parainfluenza) per follow-up period, population, and treatment setting, where possible.

Twenty-one relevant studies covering the period from 1990 to 2019 were included. Among the general population, the most frequent clinical sequela was sustained function loss (33.5% [95% CI 27.6-39.9]). Decline in lung function and cardiovascular event or congestive heart failure were also identified. Utilization sequelae were readmission (highest at > 6 months after discharge) and placement in a skilled nursing facility. The only subpopulation with data regarding sequelae was transplant patients. Among lung transplant patients, the most frequently reported clinical sequelae were decline in lung function, followed by graft dysfunction and bronchiolitis obliterans syndrome. Pooled relative risks were calculated for the following sequela with controls (primarily influenza-positive patients): cardiovascular event (general population) and pulmonary impairment (hematogenic-transplant patients) both 1.4 (95% CI 1.0-2.0) and for readmission (general population) 1.2 (95% CI 1.1-1.3).

CONCLUSIONS

Although less data are available for RSV than for influenza or other lower respiratory tract infections, RSV infection among adults is associated with medically important sequelae, with a prevalence similar to other respiratory pathogens. RSV sequelae should be included in disease burden estimates.

Authors +Show Affiliations

Pub type(s).

• Publisher Full Text (DOI)
• Rozenbaum M