tools for systematic literature review

Start your free trial

Arrange a trial for your organisation and discover why FSTA is the leading database for reliable research on the sciences of food and health.

REQUEST A FREE TRIAL

• Research Skills Blog

5 software tools to support your systematic review processes

By Dr. Mina Kalantar on 19-Jan-2021 13:01:01

Systematic reviews are a reassessment of scholarly literature to facilitate decision making. This methodical approach of re-evaluating evidence was initially applied in healthcare, to set policies, create guidelines and answer medical questions.

Systematic reviews are large, complex projects and, depending on the purpose, they can be quite expensive to conduct. A team of researchers, data analysts and experts from various fields may collaborate to review and examine incredibly large numbers of research articles for evidence synthesis. Depending on the spectrum, systematic reviews often take at least 6 months, and sometimes upwards of 18 months to complete.

The main principles of transparency and reproducibility require a pragmatic approach in the organisation of the required research activities and detailed documentation of the outcomes. As a result, many software tools have been developed to help researchers with some of the tedious tasks required as part of the systematic review process.

hbspt.cta._relativeUrls=true;hbspt.cta.load(97439, 'ccc20645-09e2-4098-838f-091ed1bf1f4e', {"useNewLoader":"true","region":"na1"});

The first generation of these software tools were produced to accommodate and manage collaborations, but gradually developed to help with screening literature and reporting outcomes. Some of these software packages were initially designed for medical and healthcare studies and have specific protocols and customised steps integrated for various types of systematic reviews. However, some are designed for general processing, and by extending the application of the systematic review approach to other fields, they are being increasingly adopted and used in software engineering, health-related nutrition, agriculture, environmental science, social sciences and education.

Software tools

There are various free and subscription-based tools to help with conducting a systematic review. Many of these tools are designed to assist with the key stages of the process, including title and abstract screening, data synthesis, and critical appraisal. Some are designed to facilitate the entire process of review, including protocol development, reporting of the outcomes and help with fast project completion.

As time goes on, more functions are being integrated into such software tools. Technological advancement has allowed for more sophisticated and user-friendly features, including visual graphics for pattern recognition and linking multiple concepts. The idea is to digitalise the cumbersome parts of the process to increase efficiency, thus allowing researchers to focus their time and efforts on assessing the rigorousness and robustness of the research articles.

This article introduces commonly used systematic review tools that are relevant to food research and related disciplines, which can be used in a similar context to the process in healthcare disciplines.

These reviews are based on IFIS' internal research, thus are unbiased and not affiliated with the companies.

This online platform is a core component of the Cochrane toolkit, supporting parts of the systematic review process, including title/abstract and full-text screening, documentation, and reporting.

The Covidence platform enables collaboration of the entire systematic reviews team and is suitable for researchers and students at all levels of experience.

From a user perspective, the interface is intuitive, and the citation screening is directed step-by-step through a well-defined workflow. Imports and exports are straightforward, with easy export options to Excel and CVS.

Access is free for Cochrane authors (a single reviewer), and Cochrane provides a free trial to other researchers in healthcare. Universities can also subscribe on an institutional basis.

Rayyan is a free and open access web-based platform funded by the Qatar Foundation, a non-profit organisation supporting education and community development initiative . Rayyan is used to screen and code literature through a systematic review process.

Unlike Covidence, Rayyan does not follow a standard SR workflow and simply helps with citation screening. It is accessible through a mobile application with compatibility for offline screening. The web-based platform is known for its accessible user interface, with easy and clear export options.

Function comparison of 5 software tools to support the systematic review process

EPPI-Reviewer

EPPI-Reviewer is a web-based software programme developed by the Evidence for Policy and Practice Information and Co-ordinating Centre  (EPPI) at the UCL Institute for Education, London .

It provides comprehensive functionalities for coding and screening. Users can create different levels of coding in a code set tool for clustering, screening, and administration of documents. EPPI-Reviewer allows direct search and import from PubMed. The import of search results from other databases is feasible in different formats. It stores, references, identifies and removes duplicates automatically. EPPI-Reviewer allows full-text screening, text mining, meta-analysis and the export of data into different types of reports.

There is no limit for concurrent use of the software and the number of articles being reviewed. Cochrane reviewers can access EPPI reviews using their Cochrane subscription details.

EPPI-Centre has other tools for facilitating the systematic review process, including coding guidelines and data management tools.

CADIMA is a free, online, open access review management tool, developed to facilitate research synthesis and structure documentation of the outcomes.

The Julius Institute and the Collaboration for Environmental Evidence established the software programme to support and guide users through the entire systematic review process, including protocol development, literature searching, study selection, critical appraisal, and documentation of the outcomes. The flexibility in choosing the steps also makes CADIMA suitable for conducting systematic mapping and rapid reviews.

CADIMA was initially developed for research questions in agriculture and environment but it is not limited to these, and as such, can be used for managing review processes in other disciplines. It enables users to export files and work offline.

The software allows for statistical analysis of the collated data using the R statistical software. Unlike EPPI-Reviewer, CADIMA does not have a built-in search engine to allow for searching in literature databases like PubMed.

DistillerSR

DistillerSR is an online software maintained by the Canadian company, Evidence Partners which specialises in literature review automation. DistillerSR provides a collaborative platform for every stage of literature review management. The framework is flexible and can accommodate literature reviews of different sizes. It is configurable to different data curation procedures, workflows and reporting standards. The platform integrates necessary features for screening, quality assessment, data extraction and reporting. The software uses Artificial Learning (AL)-enabled technologies in priority screening. It is to cut the screening process short by reranking the most relevant references nearer to the top. It can also use AL, as a second reviewer, in quality control checks of screened studies by human reviewers. DistillerSR is used to manage systematic reviews in various medical disciplines, surveillance, pharmacovigilance and public health reviews including food and nutrition topics. The software does not support statistical analyses. It provides configurable forms in standard formats for data extraction.

DistillerSR allows direct search and import of references from PubMed. It provides an add on feature called LitConnect which can be set to automatically import newly published references from data providers to keep reviews up to date during their progress.

The systematic review Toolbox is a web-based catalogue of various tools, including software packages which can assist with single or multiple tasks within the evidence synthesis process. Researchers can run a quick search or tailor a more sophisticated search by choosing their approach, budget, discipline, and preferred support features, to find the right tools for their research.

If you enjoyed this blog post, you may also be interested in our recently published blog post addressing the difference between a systematic review and a systematic literature review.

• FSTA - Food Science & Technology Abstracts
• IFIS Collections
• Resources Hub
• Diversity Statement
• Sustainability Commitment
• Company news
• Frequently Asked Questions
• Privacy Policy
• Terms of Use for IFIS Collections

Ground Floor, 115 Wharfedale Road,  Winnersh Triangle, Wokingham, Berkshire RG41 5RB

Get in touch with IFIS

© International Food Information Service (IFIS Publishing) operating as IFIS – All Rights Reserved     |     Charity Reg. No. 1068176     |     Limited Company No. 3507902     |     Designed by Blend

• Help Center

GET STARTED

COLLABORATE ON YOUR REVIEWS WITH ANYONE, ANYWHERE, ANYTIME

Save precious time and maximize your productivity with a Rayyan membership. Receive training, priority support, and access features to complete your systematic reviews efficiently.

Rayyan Teams+ makes your job easier. It includes VIP Support, AI-powered in-app help, and powerful tools to create, share and organize systematic reviews, review teams, searches, and full-texts.

RESEARCHERS

Rayyan makes collaborative systematic reviews faster, easier, and more convenient. Training, VIP support, and access to new features maximize your productivity. Get started now!

Over 1 billion reference articles reviewed by research teams, and counting...

Intelligent, scalable and intuitive.

Rayyan understands language, learns from your decisions and helps you work quickly through even your largest systematic literature reviews.

WATCH A TUTORIAL NOW

Solutions for Organizations and Businesses

Rayyan Enterprise and Rayyan Teams+ make it faster, easier and more convenient for you to manage your research process across your organization.

• Accelerate your research across your team or organization and save valuable researcher time.
• Build and preserve institutional assets, including literature searches, systematic reviews, and full-text articles.
• Onboard team members quickly with access to group trainings for beginners and experts.
• Receive priority support to stay productive when questions arise.
• SCHEDULE A DEMO
• LEARN MORE ABOUT RAYYAN TEAMS+

RAYYAN SYSTEMATIC LITERATURE REVIEW OVERVIEW

LEARN ABOUT RAYYAN’S PICO HIGHLIGHTS AND FILTERS

Join now to learn why Rayyan is trusted by already more than 500,000 researchers

Individual plans, teams plans.

For early career researchers just getting started with research.

Free forever

• 3 Active Reviews
• Invite Unlimited Reviewers
• Import Directly from Mendeley
• Industry Leading De-Duplication
• 5-Star Relevance Ranking
• Advanced Filtration Facets
• Mobile App Access
• 100 Decisions on Mobile App
• Standard Support
• Revoke Reviewer
• Online Training
• PICO Highlights & Filters
• PRISMA (Beta)
• Auto-Resolver 
• Multiple Teams & Management Roles
• Monitor & Manage Users, Searches, Reviews, Full Texts
• Onboarding and Regular Training

Professional

For researchers who want more tools for research acceleration.

per month, billed annually

• Unlimited Active Reviews
• Unlimited Decisions on Mobile App
• Priority Support
• Auto-Resolver

For currently enrolled students with valid student ID.

per month, billed quarterly

For a team that wants professional licenses for all members.

per month, per user, billed annually

• Single Team
• High Priority Support

For teams that want support and advanced tools for members.

• Multiple Teams
• Management Roles

For organizations who want access to all of their members.

Annual Subscription

Contact Sales

• Organizational Ownership
• For an organization or a company
• Access to all the premium features such as PICO Filters, Auto-Resolver, PRISMA and Mobile App
• Store and Reuse Searches and Full Texts
• A management console to view, organize and manage users, teams, review projects, searches and full texts
• Highest tier of support – Support via email, chat and AI-powered in-app help
• GDPR Compliant
• Single Sign-On
• API Integration
• Training for Experts
• Training Sessions Students Each Semester
• More options for secure access control

———————–

ANNUAL ONLY

Rayyan Subscription

membership starts with 2 users. You can select the number of additional members that you’d like to add to your membership.

Total amount:

Click Proceed to get started.

Great usability and functionality. Rayyan has saved me countless hours. I even received timely feedback from staff when I did not understand the capabilities of the system, and was pleasantly surprised with the time they dedicated to my problem. Thanks again!

This is a great piece of software. It has made the independent viewing process so much quicker. The whole thing is very intuitive.

Rayyan makes ordering articles and extracting data very easy. A great tool for undertaking literature and systematic reviews!

Excellent interface to do title and abstract screening. Also helps to keep a track on the the reasons for exclusion from the review. That too in a blinded manner.

Rayyan is a fantastic tool to save time and improve systematic reviews!!! It has changed my life as a researcher!!! thanks

Easy to use, friendly, has everything you need for cooperative work on the systematic review.

Rayyan makes life easy in every way when conducting a systematic review and it is easy to use.

Accelerate your research with the best systematic literature review tools

The ideal literature review tool helps you make sense of the most important insights in your research field. ATLAS.ti empowers researchers to perform powerful and collaborative analysis using the leading software for literature review.

Finalize your literature review faster with comfort

ATLAS.ti makes it easy to manage, organize, and analyze articles, PDFs, excerpts, and more for your projects. Conduct a deep systematic literature review and get the insights you need with a comprehensive toolset built specifically for your research projects.

Figure out the "why" behind your participant's motivations

Understand the behaviors and emotions that are driving your focus group participants. With ATLAS.ti, you can transform your raw data and turn it into qualitative insights you can learn from. Easily determine user intent in the same spot you're deciphering your overall focus group data.

Visualize your research findings like never before

We make it simple to present your analysis results with meaningful charts, networks, and diagrams. Instead of figuring out how to communicate the insights you just unlocked, we enable you to leverage easy-to-use visualizations that support your goals.

Paper Search – Access 200+ Million Papers with AI-Powered Insights

Unlock access to over 200 million scientific papers and streamline your research with our cutting-edge AI-powered Paper Search 2.0. Easily find, summarize, and integrate relevant papers directly into your ATLAS.ti Web workspace.

Everything you need to elevate your literature review

Import and organize literature data.

Import and analyze any type of text content – ATLAS.ti supports all standard text and transcription files such as Word and PDF.

Analyze with ease and speed

Utilize easy-to-learn workflows that save valuable time, such as auto coding, sentiment analysis, team collaboration, and more.

Leverage AI-driven tools

Make efficiency a priority and let ATLAS.ti do your work with AI-powered research tools and features for faster results.

Visualize and present findings

With just a few clicks, you can create meaningful visualizations like charts, word clouds, tables, networks, among others for your literature data.

The faster way to make sense of your literature review. Try it for free, today.

A literature review analyzes the most current research within a research area. A literature review consists of published studies from many sources:

• Peer-reviewed academic publications
• Full-length books
• University bulletins
• Conference proceedings
• Dissertations and theses

Literature reviews allow researchers to:

• Summarize the state of the research
• Identify unexplored research inquiries
• Recommend practical applications
• Critique currently published research

Literature reviews are either standalone publications or part of a paper as background for an original research project. A literature review, as a section of a more extensive research article, summarizes the current state of the research to justify the primary research described in the paper.

For example, a researcher may have reviewed the literature on a new supplement's health benefits and concluded that more research needs to be conducted on those with a particular condition. This research gap warrants a study examining how this understudied population reacted to the supplement. Researchers need to establish this research gap through a literature review to persuade journal editors and reviewers of the value of their research.

Consider a literature review as a typical research publication presenting a study, its results, and the salient points scholars can infer from the study. The only significant difference with a literature review treats existing literature as the research data to collect and analyze. From that analysis, a literature review can suggest new inquiries to pursue.

Identify a focus

Similar to a typical study, a literature review should have a research question or questions that analysis can answer. This sort of inquiry typically targets a particular phenomenon, population, or even research method to examine how different studies have looked at the same thing differently. A literature review, then, should center the literature collection around that focus.

Collect and analyze the literature

With a focus in mind, a researcher can collect studies that provide relevant information for that focus. They can then analyze the collected studies by finding and identifying patterns or themes that occur frequently. This analysis allows the researcher to point out what the field has frequently explored or, on the other hand, overlooked.

Suggest implications

The literature review allows the researcher to argue a particular point through the evidence provided by the analysis. For example, suppose the analysis makes it apparent that the published research on people's sleep patterns has not adequately explored the connection between sleep and a particular factor (e.g., television-watching habits, indoor air quality). In that case, the researcher can argue that further study can address this research gap.

External requirements aside (e.g., many academic journals have a word limit of 6,000-8,000 words), a literature review as a standalone publication is as long as necessary to allow readers to understand the current state of the field. Even if it is just a section in a larger paper, a literature review is long enough to allow the researcher to justify the study that is the paper's focus.

Note that a literature review needs only to incorporate a representative number of studies relevant to the research inquiry. For term papers in university courses, 10 to 20 references might be appropriate for demonstrating analytical skills. Published literature reviews in peer-reviewed journals might have 40 to 50 references. One of the essential goals of a literature review is to persuade readers that you have analyzed a representative segment of the research you are reviewing.

Researchers can find published research from various online sources:

• Journal websites
• Research databases
• Search engines (Google Scholar, Semantic Scholar)
• Research repositories
• Social networking sites (Academia, ResearchGate)

Many journals make articles freely available under the term "open access," meaning that there are no restrictions to viewing and downloading such articles. Otherwise, collecting research articles from restricted journals usually requires access from an institution such as a university or a library.

Evidence of a rigorous literature review is more important than the word count or the number of articles that undergo data analysis. Especially when writing for a peer-reviewed journal, it is essential to consider how to demonstrate research rigor in your literature review to persuade reviewers of its scholarly value.

Select field-specific journals

The most significant research relevant to your field focuses on a narrow set of journals similar in aims and scope. Consider who the most prominent scholars in your field are and determine which journals publish their research or have them as editors or reviewers. Journals tend to look favorably on systematic reviews that include articles they have published.

Incorporate recent research

Recently published studies have greater value in determining the gaps in the current state of research. Older research is likely to have encountered challenges and critiques that may render their findings outdated or refuted. What counts as recent differs by field; start by looking for research published within the last three years and gradually expand to older research when you need to collect more articles for your review.

Consider the quality of the research

Literature reviews are only as strong as the quality of the studies that the researcher collects. You can judge any particular study by many factors, including:

• the quality of the article's journal
• the article's research rigor
• the timeliness of the research

The critical point here is that you should consider more than just a study's findings or research outputs when including research in your literature review.

Narrow your research focus

Ideally, the articles you collect for your literature review have something in common, such as a research method or research context. For example, if you are conducting a literature review about teaching practices in high school contexts, it is best to narrow your literature search to studies focusing on high school. You should consider expanding your search to junior high school and university contexts only when there are not enough studies that match your focus.

You can create a project in ATLAS.ti for keeping track of your collected literature. ATLAS.ti allows you to view and analyze full text articles and PDF files in a single project. Within projects, you can use document groups to separate studies into different categories for easier and faster analysis.

For example, a researcher with a literature review that examines studies across different countries can create document groups labeled "United Kingdom," "Germany," and "United States," among others. A researcher can also use ATLAS.ti's global filters to narrow analysis to a particular set of studies and gain insights about a smaller set of literature.

ATLAS.ti allows you to search, code, and analyze text documents and PDF files. You can treat a set of research articles like other forms of qualitative data. The codes you apply to your literature collection allow for analysis through many powerful tools in ATLAS.ti:

• Code Co-Occurrence Explorer
• Code Co-Occurrence Table
• Code-Document Table

Other tools in ATLAS.ti employ machine learning to facilitate parts of the coding process for you. Some of our software tools that are effective for analyzing literature include:

• Named Entity Recognition
• Opinion Mining
• Sentiment Analysis

As long as your documents are text documents or text-enable PDF files, ATLAS.ti's automated tools can provide essential assistance in the data analysis process.

University of Tasmania, Australia

Systematic reviews for health: tools for systematic review.

• Handbooks / Guidelines for Systematic Reviews
• Standards for Reporting
• Registering a Protocol
• Tools for Systematic Review
• Online Tutorials & Courses
• Books and Articles about Systematic Reviews
• Finding Systematic Reviews
• Critical Appraisal
• Library Help
• Bibliographic Databases
• Grey Literature
• Handsearching
• Citation Searching
• 1. Formulate the Research Question
• 2. Identify the Key Concepts
• 3. Develop Search Terms - Free-Text
• 4. Develop Search Terms - Controlled Vocabulary
• 5. Search Fields
• 6. Phrase Searching, Wildcards and Proximity Operators
• 7. Boolean Operators
• 8. Search Limits
• 9. Pilot Search Strategy & Monitor Its Development
• 10. Final Search Strategy
• 11. Adapt Search Syntax
• Documenting Search Strategies
• Handling Results & Storing Papers

Tools for Systematic Reviews

Managing the selection process can be challenging, particularly in a large-scale systematic review that involves multiple reviewers. There are various free and subscription-based tools available that support the study selection process ( Cochrane Handbook, 4.6.6.1 ).

This page describes various tools available to help conduct a systematic review. The University of Tasmania has access to EndNote, Covidence and JBI SUMARI.

Covidence   is an online systematic review program developed by, and for, systematic reviewers. It can import citations from reference managers like EndNote, facilitate the screening of abstracts and full-text, populate risk of bias tables, assist with data extraction, and export to all common formats.

Covidence Demo video [3:24]

Covidence is a core component of Cochrane's review production toolkit and has also been endorsed by JBI.

Access to UTAS Covidence account

If you are the project leader, follow these steps to create a UTAS Covidence account:

• Create a UTAS Covidence account Step-by-step instructions on how to create at UTAS Covidence account

Once you have created your UTAS Covidence account, you can create a review and invite others to join the review.

If you are not the project leader, please wait for your invitation from your project leader to join the review (you don't need to create a UTAS Covidence account).  

Covidence Training & Help

• Need some help getting started? Covidence offers regular  training webinars . You also have the option to listen to the recording of a recent session.
• Work through the content of the Covidence Academy .
• Subscribe to the Covidence YouTube channel to find video tutorials and recorded webinars. The Step-by-step webinars playlist is particularly useful for in-depth guidance.
• Head to the Covidence knowledge base to get answers to FAQs.
• Contact [email protected] anytime to answer any of your questions.

Abstrackr   is a software for semi-automated abstract screening for systematic reviews. At present, Abstrackr is a free, open-source tool for facilitating the citation screening process. Upload your abstracts, invite reviewers, and get to screening!

Rayyan  is a free online tool that can be used for screening and coding of studies in a systematic review. It uses tagging and filtering to code and organise references.

The System for the Unified Management, Assessment and Review of Information ( SUMARI ) is  JBI 's software for the systematic review of literature.

I t is designed to assist researchers to conduct systematic reviews and facilitates the entire review process. SUMARI supports 10 review types. It is especially useful for new review types and qualitative reviews.

University of Tasmania researchers have access to SUMARI via the JBI EBP Database  under EBP Tools .

• JBI EBP Database via Ovid This link opens in a new window

SUMARI support:

• JBI SUMARI Video Tutorials
• JBI SUMARI Knowledge Base

Systematic Review Accelerator

The Systematic Review Accelerator (SRA) is a suite of automation tools developed by the Institute for Evidence-Based Healthcare at  Bond University. The SRA tools aim to make literature review and synthesis processes faster while maintaining and enhancing quality. The suite includes tools that can help with designing search strategies, title and abstract screening, citation tracking, and writing drafts for  methods and result sections.

The SRA tools are free and include extensive help pages .  

RevMan 5 is the software used for preparing and maintaining Cochrane Reviews. RevMan facilitates preparation of protocols and full reviews, including text, characteristics of studies, comparison tables, and study data. It can perform meta-analysis of the data entered, and present the results graphically.

RevMan 5 is no longer being developed, but they continue to support Cochrane authors.

RevMan Web   is the next generation of Cochrane's software for preparing and maintaining systematic reviews.  This web-based version of RevMan works across all platforms, is installation-free, and automatically updated. 

DistillerSR

DistillerSR is a systematic review software. It was designed from the ground up to provide a better review experience, faster project completion and transparent, audit-ready results.

What can you do in DistillerSR? Upload your references from any reference management software, create screening and data extraction forms, lay out workflow and assign reviewers, monitor study progress and review process, export results (incl PRISMA flowchart automation).

This software is more sophisticated and a bit harder to learn. DistillerSR attracts a fee .

The Systematic Review Toolbox is a community-driven, searchable, web-based catalogue of tools that support the systematic review process across multiple domains. The resource aims to help reviewers find appropriate tools based on how they provide support for the systematic review process. Users can perform a simple keyword search (i.e. Quick Search) to locate tools, a more detailed search (i.e. Advanced Search) allowing users to select various criteria to find specific types of tools and submit new tools to the database.

Need More Help? Book a consultation with a  Learning and Research Librarian  or contact  [email protected] .

• << Previous: Registering a Protocol
• Next: Online Tutorials & Courses >>
• Last Updated: Sep 12, 2024 9:53 AM
• URL: https://utas.libguides.com/SystematicReviews

Literature Review Tips & Tools

• Tips & Examples

Organizational Tools

Tools for systematic reviews.

• Bubbl.us Free online brainstorming/mindmapping tool that also has a free iPad app.
• Coggle Another free online mindmapping tool.
• Organization & Structure tips from Purdue University Online Writing Lab
• Literature Reviews from The Writing Center at University of North Carolina at Chapel Hill Gives several suggestions and descriptions of ways to organize your lit review.
• Cochrane Handbook for Systematic Reviews of Interventions "The Cochrane Handbook for Systematic Reviews of Interventions is the official guide that describes in detail the process of preparing and maintaining Cochrane systematic reviews on the effects of healthcare interventions. "
• Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) website "PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. PRISMA focuses on the reporting of reviews evaluating randomized trials, but can also be used as a basis for reporting systematic reviews of other types of research, particularly evaluations of interventions."
• PRISMA Flow Diagram Generator Free tool that will generate a PRISMA flow diagram from a CSV file (sample CSV template provided) more... less... Please cite as: Haddaway, N. R., Page, M. J., Pritchard, C. C., & McGuinness, L. A. (2022). PRISMA2020: An R package and Shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and Open Synthesis Campbell Systematic Reviews, 18, e1230. https://doi.org/10.1002/cl2.1230
• Rayyan "Rayyan is a 100% FREE web application to help systematic review authors perform their job in a quick, easy and enjoyable fashion. Authors create systematic reviews, collaborate on them, maintain them over time and get suggestions for article inclusion."
• Covidence Covidence is a tool to help manage systematic reviews (and create PRISMA flow diagrams). **UMass Amherst doesn't subscribe, but Covidence offers a free trial for 1 review of no more than 500 records. It is also set up for researchers to pay for each review.
• PROSPERO - Systematic Review Protocol Registry "PROSPERO accepts registrations for systematic reviews, rapid reviews and umbrella reviews. PROSPERO does not accept scoping reviews or literature scans. Sibling PROSPERO sites registers systematic reviews of human studies and systematic reviews of animal studies."
• Critical Appraisal Tools from JBI Joanna Briggs Institute at the University of Adelaide provides these checklists to help evaluate different types of publications that could be included in a review.
• Systematic Review Toolbox "The Systematic Review Toolbox is a community-driven, searchable, web-based catalogue of tools that support the systematic review process across multiple domains. The resource aims to help reviewers find appropriate tools based on how they provide support for the systematic review process. Users can perform a simple keyword search (i.e. Quick Search) to locate tools, a more detailed search (i.e. Advanced Search) allowing users to select various criteria to find specific types of tools and submit new tools to the database. Although the focus of the Toolbox is on identifying software tools to support systematic reviews, other tools or support mechanisms (such as checklists, guidelines and reporting standards) can also be found."
• Abstrackr Free, open-source tool that "helps you upload and organize the results of a literature search for a systematic review. It also makes it possible for your team to screen, organize, and manipulate all of your abstracts in one place." -From Center for Evidence Synthesis in Health
• SRDR Plus (Systematic Review Data Repository: Plus) An open-source tool for extracting, managing,, and archiving data developed by the Center for Evidence Synthesis in Health at Brown University
• RoB 2 Tool (Risk of Bias for Randomized Trials) A revised Cochrane risk of bias tool for randomized trials
• << Previous: Tips & Examples
• Next: Writing & Citing Help >>
• Last Updated: Sep 9, 2024 10:52 AM
• URL: https://guides.library.umass.edu/litreviews

© 2022 University of Massachusetts Amherst • Site Policies • Accessibility

10 Best Literature Review Tools for Researchers

Boost your research game with these Best Literature Review Tools for Researchers! Uncover hidden gems, organize your findings, and ace your next research paper!

Researchers struggle to identify key sources, extract relevant information, and maintain accuracy while manually conducting literature reviews. This leads to inefficiency, errors, and difficulty in identifying gaps or trends in existing literature.

Table of Contents

Top 10 Literature Review Tools for Researchers: In A Nutshell (2023)

#1. Semantic Scholar – A free, AI-powered research tool for scientific literature

Not all scholarly content may be indexed, and occasional false positives or inaccurate associations can occur. Furthermore, the tool primarily focuses on computer science and related fields, potentially limiting coverage in other disciplines. 

#2. Elicit – Research assistant using language models like GPT-3

However, users should be cautious when using Elicit. It is important to verify the credibility and accuracy of the sources found through the tool, as the database encompasses a wide range of publications. 

Additionally, occasional glitches in the search function have been reported, leading to incomplete or inaccurate results. While Elicit offers tremendous benefits, researchers should remain vigilant and cross-reference information to ensure a comprehensive literature review.

#3. Scite.Ai – Your personal research assistant

Scite.Ai is a popular literature review tool that revolutionizes the research process for scholars. With its innovative citation analysis feature, researchers can evaluate the credibility and impact of scientific articles, making informed decisions about their inclusion in their own work. 

However, while Scite.Ai offers numerous advantages, there are a few aspects to be cautious about. As with any data-driven tool, occasional errors or inaccuracies may arise, necessitating researchers to cross-reference and verify results with other reputable sources. 

Rayyan offers the following paid plans:

#4. DistillerSR – Literature Review Software

Despite occasional technical glitches reported by some users, the developers actively address these issues through updates and improvements, ensuring a better user experience. 

#5. Rayyan – AI Powered Tool for Systematic Literature Reviews

However, it’s important to be aware of a few aspects. The free version of Rayyan has limitations, and upgrading to a premium subscription may be necessary for additional functionalities. 

#6. Consensus – Use AI to find you answers in scientific research

With Consensus, researchers can save significant time by efficiently organizing and accessing relevant research material.People consider Consensus for several reasons. 

Consensus offers both free and paid plans:

#7. RAx – AI-powered reading assistant

#8. lateral – advance your research with ai.

Additionally, researchers must be mindful of potential biases introduced by the tool’s algorithms and should critically evaluate and interpret the results. 

#9. Iris AI – Introducing the researcher workspace

Researchers are drawn to this tool because it saves valuable time by automating the tedious task of literature review and provides comprehensive coverage across multiple disciplines. 

#10. Scholarcy – Summarize your literature through AI

Scholarcy’s ability to extract key information and generate concise summaries makes it an attractive option for scholars looking to quickly grasp the main concepts and findings of multiple papers.

Scholarcy’s automated summarization may not capture the nuanced interpretations or contextual information presented in the full text. 

Final Thoughts

In conclusion, conducting a comprehensive literature review is a crucial aspect of any research project, and the availability of reliable and efficient tools can greatly facilitate this process for researchers. This article has explored the top 10 literature review tools that have gained popularity among researchers.

Q1. What are literature review tools for researchers?

Q2. what criteria should researchers consider when choosing literature review tools.

When choosing literature review tools, researchers should consider factors such as the tool’s search capabilities, database coverage, user interface, collaboration features, citation management, annotation and highlighting options, integration with reference management software, and data extraction capabilities. 

Q3. Are there any literature review tools specifically designed for systematic reviews or meta-analyses?

Meta-analysis support: Some literature review tools include statistical analysis features that assist in conducting meta-analyses. These features can help calculate effect sizes, perform statistical tests, and generate forest plots or other visual representations of the meta-analytic results.

Q4. Can literature review tools help with organizing and annotating collected references?

Integration with citation managers: Some literature review tools integrate with popular citation managers like Zotero, Mendeley, or EndNote, allowing seamless transfer of references and annotations between platforms.

By leveraging these features, researchers can streamline the organization and annotation of their collected references, making it easier to retrieve relevant information during the literature review process.

Leave a Comment Cancel reply

Distiller SR: Literature Review Software

Smarter reviews: trusted evidence.

Securely automate every stage of your literature review to produce evidence-based research faster, more accurately, and more

transparently at scale.

Software Built for Every Stage of a Literature Review

DistillerSR automates the management of literature collection, screening, and assessment using AI and intelligent workflows. From a systematic literature review to a rapid review to a living review, DistillerSR makes any project simpler to manage and configure to produce transparent, audit-ready, and compliant results.

Broader, Automated Literature Searches

Search more efficiently with DistillerSR’s integrations with data providers, such as PubMed, automatic review updates, and AI-powered duplicate detection and removal.

PubMed Integration

Automatic review updates.

Automatically import newly published references, always keeping literature reviews up-to-date with DistillerSR LitConnect .

Duplicate Detection

Detect and remove duplicate citations preventing skew and bias caused by studies included more than once.

Faster, More Effective Reference Screening

Reduce your screening burden by 60% with DistillerSR. Start working on later stages of your review sooner by finding relevant references faster and addressing conflicts more easily.

AI-Powered Screening

Conflict resolution.

Automatically identifies conflicts and disagreements between literature reviewers for easy resolution.

AI Quality Check

Increase the thoroughness of your literature review by having AI double-check your exclusion decisions and validate your categorization of records with the help of DistillerSR AI Classifiers software module.

Cost-Effective Access  to Full-Text Documents

Ensure your literature review is always up-to-date with DistillerSR’s direct connections to full-text data sources, all the while lowering overall subscription costs.

Ensure your review is always up-to-date with DistillerSR’s direct connections to full-text data sources, all the while lowering overall subscription costs.

Open Access Integrations

Automatically search for and upload full-text documents from PMC , and link directly to source material through DOI.org .

Copyright Compliant Bulk Search

Retrieve full-text articles for the lowest possible cost through Article Galaxy .

Ad-Hoc Document Retrieval

Leverage existing RightFind and Article Galaxy subscriptions, the open access Unpaywall plugin, and internal libraries to access copyright compliant documents.

Simple Yet Powerful Data-Extraction

Simplify data extraction through templates and configurable forms. Extract data easily with in-form validations and calculations, and easily capture repeating, complex data sets.

Cross-Review, Data Reuse

Prevent duplication of effort across your organization and reduce data extraction times with DistillerSR CuratorCR by easily reusing data across literature reviews.

Capturing Complex Output

Easily capture complex data, such as a variable number of time points across multiple studies in an easy-to-understand and ready-to-analyze way.

Smart Forms

Cut down on literature review data cleaning, data conversions, and effective measure calculations with input validation and built-in form calculations.

Automatic and Configurable Reporting

Customizable Reporting Engine

Build reports and schedule automated email updates to stakeholders. Integrate your data with third-party reporting applications and databases with DistillerSR API .

Auto-Generated Reports

Comprehensive audit-trail.

Automatically keeps track of every entry and decision providing transparency and reproducibility in your literature review.

Easy-to-use Literature Review Project Management

Facilitate project management throughout the literature review process with real-time user and project metric monitoring, reusable configurations, and granular user permissions.

Facilitate project management throughout the review process with real-time user and project metric monitoring, reusable configurations, and granular user permissions.

Real-Time User and Project Metrics

Monitor teams and literature review progress in real-time, improving management and quality oversight into projects.

Repeatable, Configurable Processes

Secure literature reviews.

Single sign-on (SSO) and fully configurable user roles and permissions simplify the literature reviewer experience while also ensuring data integrity and security .

I can’t think of a way to do reviews faster than with DistillerSR. Being able to monitor progress and collaborate with team members, no matter where they are located makes my life a lot easier.

DistillerSR Case Studies

Maple Health Group

University of Florida

Distillersr frequently asked questions, what types of reviews can be done with distillersr systematic reviews, living reviews, rapid reviews, or clinical evaluation report (cer) literature reviews.

Literature reviews can be a very simple or highly complex process, and literature reviews can use a variety of methods for finding, assessing, and presenting evidence. We describe DistillerSR as a literature review software because it supports all types of reviews , from systematic reviews to rapid reviews, and from living reviews to CER literature reviews.

DistillerSR software is used by over 300 customers in many different industries to support their evidence generation initiatives, from guideline development to HEOR analysis to CERs to post-market surveillance (PMS) and pharmacovigilance.

What are some of DistillerSR’s capabilities that support conducting systematic reviews?

Systematic reviews are the gold standard of literature reviews that aim to identify and screen all evidence relating to a specific research question. DistillerSR facilitates systematic reviews through a configurable, transparent, reproducible process that makes it easy to view the provenance of every cell of data.

DistillerSR was originally designed to support systematic reviews. The software handles dual reviewer screening, conflict resolution, capturing exclusion reasons while you work, risk of bias assessments, duplicate detection, multiple database searches, and reporting templates such as PRISMA . DistillerSR can readily scale for systematic reviews of all sizes, supporting more than 700,000 references per project through a robust enterprise-grade technical architecture . Using software like DistillerSR makes conducting systematic reviews easier to manage and configure to produce transparent evidence-based research faster and more accurately.

How does DistillerSR support clinical evaluation reports (CERs) and performance evaluation reports (PERs) program management?

The new European Union Medical Device Regulation (EU-MDR) and In-Vitro Device Regulation (EU-IVDR) require medical device manufacturers to increase the frequency, traceability, and overall documentation for CERs in the MDR program or PERs in the IVDR counterpart. Literature review software is an ideal tool to help you comply with these regulations.

DistillerSR automates literature reviews to enable a more transparent, repeatable, and auditable process , enabling manufacturers to create and implement a standard framework for literature reviews. This framework for conducting literature reviews can then be incorporated into all CER and PER program management plans consistently across every product, division, and research group.

How can DistillerSR help rapid reviews?

DistillerSR AI is ideal to speed up the rapid review process without compromising on quality. The AI-powered screening enables you to find references faster by continuously reordering relevant references, resulting in accelerated screening. The AI can also double-check your exclusion decisions to ensure relevant references are not left out of the rapid review.

DistillerSR title screening functionality enables you to quickly perform title screening on large numbers of references.

Does DistillerSR support living reviews?

The short answer is yes. DistillerSR has multiple capabilities that automate living systematic reviews , such as automatically importing newly published references into your projects and notifying reviewers that there’s screening to do. You can also put reports on an automated schedule so you’re never caught off guard when important new data is collected.   These capabilities help ensure the latest research is included in your living systematic review and that your review is up-to-date. 

How can DistillerSR help ensure the accuracy of Literature and Systematic reviews?

The quality of systematic reviews is foundational to evidence-based research. However, quality may be compromised because systematic reviews – by their very nature – are often tedious and repetitive, and prone to human error. Tracking all review activity in systematic review software, like DistillerSR, and making it easy to trace the provenance of every cell of data, delivers total transparency and auditability into the systematic review process. DistillerSR enables reviewers to work on the same project simultaneously without the risk of duplicating work or overwriting each other’s results. Configurable workflow filters ensure that the right references are automatically assigned to the right reviewers, and DistillerSR’s cross-project dashboard allows reviewers to monitor to-do lists for all projects from one place.

Why should I add DistillerSR to my Literature and Systematic Review Toolbox and retire my current spreadsheet solution?

It’s estimated that 90% of spreadsheets contain formula errors and approximately 50% have material defects. These errors, coupled with the time and resources necessary to fix them, adversely impact the management of the systematic review process. DistillerSR software was specifically designed to address the challenges faced by systematic review authors, namely the ever-increasing volume of research to screen and extract, review bottlenecks, and regulatory requirements for auditability and transparency, as well as a tool for managing a remote global workforce. Efficiency, consistency, better collaboration, and quality control are just a few of the benefits you’ll get when you choose DistillerSR’s systematic review process over a manual spreadsheet tool for your reviews.

What is the role of AI in your systematic review process?

DistillerSR AI enables the automation of the logistic-heavy tasks involved in conducting a systematic literature review, such as finding references faster using AI to continuously reorder references based on relevance. Continuous AI Reprioritization uses machine learning to learn from the references you are including and excluding and automatically reorder the ones you have left to screen, putting the most pertinent references in front of you first. This means that you find included references much more quickly during the screening process. DistillerSR also uses classifiers , which use NLP to classify and process information in the systematic review.  DistillerSR can also increase the thoroughness of your systematic review by having AI double-check your exclusion decisions.

What about the security and scalability of systematic literature reviews done on DistillerSR?

DistillerSR builds security, scalability, and availability into everything we do, so you can focus on producing evidence-based research faster, more accurately, and more securely with our  systematic review software. We undergo an annual independent third-party audit and certify our products using the American Institute of Certified Public Accountants SOC 2 framework. In terms of scalability, systematic review projects in DistillerSR can easily handle a large number of references; some of our customers have over 700,000 references in their projects.

Do you offer any commitments on the frequency of new product and capability launches?

We pride ourselves on listening to and working with our customers to regularly introduce new capabilities that improve DistillerSR and the systematic review process. We plan on offering two major releases a year in addition to two minor feature enhancements. We notify customers in advance about upcoming releases, host webinars, develop tools and training to introduce the new capabilities, and provide extensive release notes for our reviewers.

I have a unique literature review protocol. Is your software configurable with my literature review data and process?

Configurability is one of the key foundations of DistillerSR software. In fact, with over 300 customers in many different industries, we have yet to see a literature review protocol that our software couldn’t handle. DistillerSR is a professional B2B SaaS company with an exceptional customer success team that will work with you to understand your unique requirements and systematic review process to get you started quickly. Our global support team is available 24/7 to help you.

Still unsure if DistillerSR will meet your systematic literature review requirements?

Adopting a new software is about more than just money. New software is also about commitment and trusting that the new platform will match your systematic review and scalability needs. We have resources to help you in your analysis and decision: check out the systematic review software checklist or the literature review software checklist .

Learn More About DistillerSR

• Open access
• Published: 08 June 2023

Guidance to best tools and practices for systematic reviews

• Kat Kolaski 1 ,
• Lynne Romeiser Logan 2 &
• John P. A. Ioannidis 3  

Systematic Reviews volume  12 , Article number:  96 ( 2023 ) Cite this article

24k Accesses

26 Citations

76 Altmetric

Metrics details

Data continue to accumulate indicating that many systematic reviews are methodologically flawed, biased, redundant, or uninformative. Some improvements have occurred in recent years based on empirical methods research and standardization of appraisal tools; however, many authors do not routinely or consistently apply these updated methods. In addition, guideline developers, peer reviewers, and journal editors often disregard current methodological standards. Although extensively acknowledged and explored in the methodological literature, most clinicians seem unaware of these issues and may automatically accept evidence syntheses (and clinical practice guidelines based on their conclusions) as trustworthy.

A plethora of methods and tools are recommended for the development and evaluation of evidence syntheses. It is important to understand what these are intended to do (and cannot do) and how they can be utilized. Our objective is to distill this sprawling information into a format that is understandable and readily accessible to authors, peer reviewers, and editors. In doing so, we aim to promote appreciation and understanding of the demanding science of evidence synthesis among stakeholders. We focus on well-documented deficiencies in key components of evidence syntheses to elucidate the rationale for current standards. The constructs underlying the tools developed to assess reporting, risk of bias, and methodological quality of evidence syntheses are distinguished from those involved in determining overall certainty of a body of evidence. Another important distinction is made between those tools used by authors to develop their syntheses as opposed to those used to ultimately judge their work.

Exemplar methods and research practices are described, complemented by novel pragmatic strategies to improve evidence syntheses. The latter include preferred terminology and a scheme to characterize types of research evidence. We organize best practice resources in a Concise Guide that can be widely adopted and adapted for routine implementation by authors and journals. Appropriate, informed use of these is encouraged, but we caution against their superficial application and emphasize their endorsement does not substitute for in-depth methodological training. By highlighting best practices with their rationale, we hope this guidance will inspire further evolution of methods and tools that can advance the field.

Part 1. The state of evidence synthesis

Evidence syntheses are commonly regarded as the foundation of evidence-based medicine (EBM). They are widely accredited for providing reliable evidence and, as such, they have significantly influenced medical research and clinical practice. Despite their uptake throughout health care and ubiquity in contemporary medical literature, some important aspects of evidence syntheses are generally overlooked or not well recognized. Evidence syntheses are mostly retrospective exercises, they often depend on weak or irreparably flawed data, and they may use tools that have acknowledged or yet unrecognized limitations. They are complicated and time-consuming undertakings prone to bias and errors. Production of a good evidence synthesis requires careful preparation and high levels of organization in order to limit potential pitfalls [ 1 ]. Many authors do not recognize the complexity of such an endeavor and the many methodological challenges they may encounter. Failure to do so is likely to result in research and resource waste.

Given their potential impact on people’s lives, it is crucial for evidence syntheses to correctly report on the current knowledge base. In order to be perceived as trustworthy, reliable demonstration of the accuracy of evidence syntheses is equally imperative [ 2 ]. Concerns about the trustworthiness of evidence syntheses are not recent developments. From the early years when EBM first began to gain traction until recent times when thousands of systematic reviews are published monthly [ 3 ] the rigor of evidence syntheses has always varied. Many systematic reviews and meta-analyses had obvious deficiencies because original methods and processes had gaps, lacked precision, and/or were not widely known. The situation has improved with empirical research concerning which methods to use and standardization of appraisal tools. However, given the geometrical increase in the number of evidence syntheses being published, a relatively larger pool of unreliable evidence syntheses is being published today.

Publication of methodological studies that critically appraise the methods used in evidence syntheses is increasing at a fast pace. This reflects the availability of tools specifically developed for this purpose [ 4 , 5 , 6 ]. Yet many clinical specialties report that alarming numbers of evidence syntheses fail on these assessments. The syntheses identified report on a broad range of common conditions including, but not limited to, cancer, [ 7 ] chronic obstructive pulmonary disease, [ 8 ] osteoporosis, [ 9 ] stroke, [ 10 ] cerebral palsy, [ 11 ] chronic low back pain, [ 12 ] refractive error, [ 13 ] major depression, [ 14 ] pain, [ 15 ] and obesity [ 16 , 17 ]. The situation is even more concerning with regard to evidence syntheses included in clinical practice guidelines (CPGs) [ 18 , 19 , 20 ]. Astonishingly, in a sample of CPGs published in 2017–18, more than half did not apply even basic systematic methods in the evidence syntheses used to inform their recommendations [ 21 ].

These reports, while not widely acknowledged, suggest there are pervasive problems not limited to evidence syntheses that evaluate specific kinds of interventions or include primary research of a particular study design (eg, randomized versus non-randomized) [ 22 ]. Similar concerns about the reliability of evidence syntheses have been expressed by proponents of EBM in highly circulated medical journals [ 23 , 24 , 25 , 26 ]. These publications have also raised awareness about redundancy, inadequate input of statistical expertise, and deficient reporting. These issues plague primary research as well; however, there is heightened concern for the impact of these deficiencies given the critical role of evidence syntheses in policy and clinical decision-making.

Methods and guidance to produce a reliable evidence synthesis

Several international consortiums of EBM experts and national health care organizations currently provide detailed guidance (Table 1 ). They draw criteria from the reporting and methodological standards of currently recommended appraisal tools, and regularly review and update their methods to reflect new information and changing needs. In addition, they endorse the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system for rating the overall quality of a body of evidence [ 27 ]. These groups typically certify or commission systematic reviews that are published in exclusive databases (eg, Cochrane, JBI) or are used to develop government or agency sponsored guidelines or health technology assessments (eg, National Institute for Health and Care Excellence [NICE], Scottish Intercollegiate Guidelines Network [SIGN], Agency for Healthcare Research and Quality [AHRQ]). They offer developers of evidence syntheses various levels of methodological advice, technical and administrative support, and editorial assistance. Use of specific protocols and checklists are required for development teams within these groups, but their online methodological resources are accessible to any potential author.

Notably, Cochrane is the largest single producer of evidence syntheses in biomedical research; however, these only account for 15% of the total [ 28 ]. The World Health Organization requires Cochrane standards be used to develop evidence syntheses that inform their CPGs [ 29 ]. Authors investigating questions of intervention effectiveness in syntheses developed for Cochrane follow the Methodological Expectations of Cochrane Intervention Reviews [ 30 ] and undergo multi-tiered peer review [ 31 , 32 ]. Several empirical evaluations have shown that Cochrane systematic reviews are of higher methodological quality compared with non-Cochrane reviews [ 4 , 7 , 9 , 11 , 14 , 32 , 33 , 34 , 35 ]. However, some of these assessments have biases: they may be conducted by Cochrane-affiliated authors, and they sometimes use scales and tools developed and used in the Cochrane environment and by its partners. In addition, evidence syntheses published in the Cochrane database are not subject to space or word restrictions, while non-Cochrane syntheses are often limited. As a result, information that may be relevant to the critical appraisal of non-Cochrane reviews is often removed or is relegated to online-only supplements that may not be readily or fully accessible [ 28 ].

Influences on the state of evidence synthesis

Many authors are familiar with the evidence syntheses produced by the leading EBM organizations but can be intimidated by the time and effort necessary to apply their standards. Instead of following their guidance, authors may employ methods that are discouraged or outdated 28]. Suboptimal methods described in in the literature may then be taken up by others. For example, the Newcastle–Ottawa Scale (NOS) is a commonly used tool for appraising non-randomized studies [ 36 ]. Many authors justify their selection of this tool with reference to a publication that describes the unreliability of the NOS and recommends against its use [ 37 ]. Obviously, the authors who cite this report for that purpose have not read it. Authors and peer reviewers have a responsibility to use reliable and accurate methods and not copycat previous citations or substandard work [ 38 , 39 ]. Similar cautions may potentially extend to automation tools. These have concentrated on evidence searching [ 40 ] and selection given how demanding it is for humans to maintain truly up-to-date evidence [ 2 , 41 ]. Cochrane has deployed machine learning to identify randomized controlled trials (RCTs) and studies related to COVID-19, [ 2 , 42 ] but such tools are not yet commonly used [ 43 ]. The routine integration of automation tools in the development of future evidence syntheses should not displace the interpretive part of the process.

Editorials about unreliable or misleading systematic reviews highlight several of the intertwining factors that may contribute to continued publication of unreliable evidence syntheses: shortcomings and inconsistencies of the peer review process, lack of endorsement of current standards on the part of journal editors, the incentive structure of academia, industry influences, publication bias, and the lure of “predatory” journals [ 44 , 45 , 46 , 47 , 48 ]. At this juncture, clarification of the extent to which each of these factors contribute remains speculative, but their impact is likely to be synergistic.

Over time, the generalized acceptance of the conclusions of systematic reviews as incontrovertible has affected trends in the dissemination and uptake of evidence. Reporting of the results of evidence syntheses and recommendations of CPGs has shifted beyond medical journals to press releases and news headlines and, more recently, to the realm of social media and influencers. The lay public and policy makers may depend on these outlets for interpreting evidence syntheses and CPGs. Unfortunately, communication to the general public often reflects intentional or non-intentional misrepresentation or “spin” of the research findings [ 49 , 50 , 51 , 52 ] News and social media outlets also tend to reduce conclusions on a body of evidence and recommendations for treatment to binary choices (eg, “do it” versus “don’t do it”) that may be assigned an actionable symbol (eg, red/green traffic lights, smiley/frowning face emoji).

Strategies for improvement

Many authors and peer reviewers are volunteer health care professionals or trainees who lack formal training in evidence synthesis [ 46 , 53 ]. Informing them about research methodology could increase the likelihood they will apply rigorous methods [ 25 , 33 , 45 ]. We tackle this challenge, from both a theoretical and a practical perspective, by offering guidance applicable to any specialty. It is based on recent methodological research that is extensively referenced to promote self-study. However, the information presented is not intended to be substitute for committed training in evidence synthesis methodology; instead, we hope to inspire our target audience to seek such training. We also hope to inform a broader audience of clinicians and guideline developers influenced by evidence syntheses. Notably, these communities often include the same members who serve in different capacities.

In the following sections, we highlight methodological concepts and practices that may be unfamiliar, problematic, confusing, or controversial. In Part 2, we consider various types of evidence syntheses and the types of research evidence summarized by them. In Part 3, we examine some widely used (and misused) tools for the critical appraisal of systematic reviews and reporting guidelines for evidence syntheses. In Part 4, we discuss how to meet methodological conduct standards applicable to key components of systematic reviews. In Part 5, we describe the merits and caveats of rating the overall certainty of a body of evidence. Finally, in Part 6, we summarize suggested terminology, methods, and tools for development and evaluation of evidence syntheses that reflect current best practices.

Part 2. Types of syntheses and research evidence

A good foundation for the development of evidence syntheses requires an appreciation of their various methodologies and the ability to correctly identify the types of research potentially available for inclusion in the synthesis.

Types of evidence syntheses

Systematic reviews have historically focused on the benefits and harms of interventions; over time, various types of systematic reviews have emerged to address the diverse information needs of clinicians, patients, and policy makers [ 54 ] Systematic reviews with traditional components have become defined by the different topics they assess (Table 2.1 ). In addition, other distinctive types of evidence syntheses have evolved, including overviews or umbrella reviews, scoping reviews, rapid reviews, and living reviews. The popularity of these has been increasing in recent years [ 55 , 56 , 57 , 58 ]. A summary of the development, methods, available guidance, and indications for these unique types of evidence syntheses is available in Additional File 2 A.

Both Cochrane [ 30 , 59 ] and JBI [ 60 ] provide methodologies for many types of evidence syntheses; they describe these with different terminology, but there is obvious overlap (Table 2.2 ). The majority of evidence syntheses published by Cochrane (96%) and JBI (62%) are categorized as intervention reviews. This reflects the earlier development and dissemination of their intervention review methodologies; these remain well-established [ 30 , 59 , 61 ] as both organizations continue to focus on topics related to treatment efficacy and harms. In contrast, intervention reviews represent only about half of the total published in the general medical literature, and several non-intervention review types contribute to a significant proportion of the other half.

Types of research evidence

There is consensus on the importance of using multiple study designs in evidence syntheses; at the same time, there is a lack of agreement on methods to identify included study designs. Authors of evidence syntheses may use various taxonomies and associated algorithms to guide selection and/or classification of study designs. These tools differentiate categories of research and apply labels to individual study designs (eg, RCT, cross-sectional). A familiar example is the Design Tree endorsed by the Centre for Evidence-Based Medicine [ 70 ]. Such tools may not be helpful to authors of evidence syntheses for multiple reasons.

Suboptimal levels of agreement and accuracy even among trained methodologists reflect challenges with the application of such tools [ 71 , 72 ]. Problematic distinctions or decision points (eg, experimental or observational, controlled or uncontrolled, prospective or retrospective) and design labels (eg, cohort, case control, uncontrolled trial) have been reported [ 71 ]. The variable application of ambiguous study design labels to non-randomized studies is common, making them especially prone to misclassification [ 73 ]. In addition, study labels do not denote the unique design features that make different types of non-randomized studies susceptible to different biases, including those related to how the data are obtained (eg, clinical trials, disease registries, wearable devices). Given this limitation, it is important to be aware that design labels preclude the accurate assignment of non-randomized studies to a “level of evidence” in traditional hierarchies [ 74 ].

These concerns suggest that available tools and nomenclature used to distinguish types of research evidence may not uniformly apply to biomedical research and non-health fields that utilize evidence syntheses (eg, education, economics) [ 75 , 76 ]. Moreover, primary research reports often do not describe study design or do so incompletely or inaccurately; thus, indexing in PubMed and other databases does not address the potential for misclassification [ 77 ]. Yet proper identification of research evidence has implications for several key components of evidence syntheses. For example, search strategies limited by index terms using design labels or study selection based on labels applied by the authors of primary studies may cause inconsistent or unjustified study inclusions and/or exclusions [ 77 ]. In addition, because risk of bias (RoB) tools consider attributes specific to certain types of studies and study design features, results of these assessments may be invalidated if an inappropriate tool is used. Appropriate classification of studies is also relevant for the selection of a suitable method of synthesis and interpretation of those results.

An alternative to these tools and nomenclature involves application of a few fundamental distinctions that encompass a wide range of research designs and contexts. While these distinctions are not novel, we integrate them into a practical scheme (see Fig. 1 ) designed to guide authors of evidence syntheses in the basic identification of research evidence. The initial distinction is between primary and secondary studies. Primary studies are then further distinguished by: 1) the type of data reported (qualitative or quantitative); and 2) two defining design features (group or single-case and randomized or non-randomized). The different types of studies and study designs represented in the scheme are described in detail in Additional File 2 B. It is important to conceptualize their methods as complementary as opposed to contrasting or hierarchical [ 78 ]; each offers advantages and disadvantages that determine their appropriateness for answering different kinds of research questions in an evidence synthesis.

Distinguishing types of research evidence

Application of these basic distinctions may avoid some of the potential difficulties associated with study design labels and taxonomies. Nevertheless, debatable methodological issues are raised when certain types of research identified in this scheme are included in an evidence synthesis. We briefly highlight those associated with inclusion of non-randomized studies, case reports and series, and a combination of primary and secondary studies.

Non-randomized studies

When investigating an intervention’s effectiveness, it is important for authors to recognize the uncertainty of observed effects reported by studies with high RoB. Results of statistical analyses that include such studies need to be interpreted with caution in order to avoid misleading conclusions [ 74 ]. Review authors may consider excluding randomized studies with high RoB from meta-analyses. Non-randomized studies of intervention (NRSI) are affected by a greater potential range of biases and thus vary more than RCTs in their ability to estimate a causal effect [ 79 ]. If data from NRSI are synthesized in meta-analyses, it is helpful to separately report their summary estimates [ 6 , 74 ].

Nonetheless, certain design features of NRSI (eg, which parts of the study were prospectively designed) may help to distinguish stronger from weaker ones. Cochrane recommends that authors of a review including NRSI focus on relevant study design features when determining eligibility criteria instead of relying on non-informative study design labels [ 79 , 80 ] This process is facilitated by a study design feature checklist; guidance on using the checklist is included with developers’ description of the tool [ 73 , 74 ]. Authors collect information about these design features during data extraction and then consider it when making final study selection decisions and when performing RoB assessments of the included NRSI.

Case reports and case series

Correctly identified case reports and case series can contribute evidence not well captured by other designs [ 81 ]; in addition, some topics may be limited to a body of evidence that consists primarily of uncontrolled clinical observations. Murad and colleagues offer a framework for how to include case reports and series in an evidence synthesis [ 82 ]. Distinguishing between cohort studies and case series in these syntheses is important, especially for those that rely on evidence from NRSI. Additional data obtained from studies misclassified as case series can potentially increase the confidence in effect estimates. Mathes and Pieper provide authors of evidence syntheses with specific guidance on distinguishing between cohort studies and case series, but emphasize the increased workload involved [ 77 ].

Primary and secondary studies

Synthesis of combined evidence from primary and secondary studies may provide a broad perspective on the entirety of available literature on a topic. This is, in fact, the recommended strategy for scoping reviews that may include a variety of sources of evidence (eg, CPGs, popular media). However, except for scoping reviews, the synthesis of data from primary and secondary studies is discouraged unless there are strong reasons to justify doing so.

Combining primary and secondary sources of evidence is challenging for authors of other types of evidence syntheses for several reasons [ 83 ]. Assessments of RoB for primary and secondary studies are derived from conceptually different tools, thus obfuscating the ability to make an overall RoB assessment of a combination of these study types. In addition, authors who include primary and secondary studies must devise non-standardized methods for synthesis. Note this contrasts with well-established methods available for updating existing evidence syntheses with additional data from new primary studies [ 84 , 85 , 86 ]. However, a new review that synthesizes data from primary and secondary studies raises questions of validity and may unintentionally support a biased conclusion because no existing methodological guidance is currently available [ 87 ].

Recommendations

We suggest that journal editors require authors to identify which type of evidence synthesis they are submitting and reference the specific methodology used for its development. This will clarify the research question and methods for peer reviewers and potentially simplify the editorial process. Editors should announce this practice and include it in the instructions to authors. To decrease bias and apply correct methods, authors must also accurately identify the types of research evidence included in their syntheses.

Part 3. Conduct and reporting

The need to develop criteria to assess the rigor of systematic reviews was recognized soon after the EBM movement began to gain international traction [ 88 , 89 ]. Systematic reviews rapidly became popular, but many were very poorly conceived, conducted, and reported. These problems remain highly prevalent [ 23 ] despite development of guidelines and tools to standardize and improve the performance and reporting of evidence syntheses [ 22 , 28 ]. Table 3.1  provides some historical perspective on the evolution of tools developed specifically for the evaluation of systematic reviews, with or without meta-analysis.

These tools are often interchangeably invoked when referring to the “quality” of an evidence synthesis. However, quality is a vague term that is frequently misused and misunderstood; more precisely, these tools specify different standards for evidence syntheses. Methodological standards address how well a systematic review was designed and performed [ 5 ]. RoB assessments refer to systematic flaws or limitations in the design, conduct, or analysis of research that distort the findings of the review [ 4 ]. Reporting standards help systematic review authors describe the methodology they used and the results of their synthesis in sufficient detail [ 92 ]. It is essential to distinguish between these evaluations: a systematic review may be biased, it may fail to report sufficient information on essential features, or it may exhibit both problems; a thoroughly reported systematic evidence synthesis review may still be biased and flawed while an otherwise unbiased one may suffer from deficient documentation.

We direct attention to the currently recommended tools listed in Table 3.1  but concentrate on AMSTAR-2 (update of AMSTAR [A Measurement Tool to Assess Systematic Reviews]) and ROBIS (Risk of Bias in Systematic Reviews), which evaluate methodological quality and RoB, respectively. For comparison and completeness, we include PRISMA 2020 (update of the 2009 Preferred Reporting Items for Systematic Reviews of Meta-Analyses statement), which offers guidance on reporting standards. The exclusive focus on these three tools is by design; it addresses concerns related to the considerable variability in tools used for the evaluation of systematic reviews [ 28 , 88 , 96 , 97 ]. We highlight the underlying constructs these tools were designed to assess, then describe their components and applications. Their known (or potential) uptake and impact and limitations are also discussed.

Evaluation of conduct

Development.

AMSTAR [ 5 ] was in use for a decade prior to the 2017 publication of AMSTAR-2; both provide a broad evaluation of methodological quality of intervention systematic reviews, including flaws arising through poor conduct of the review [ 6 ]. ROBIS, published in 2016, was developed to specifically assess RoB introduced by the conduct of the review; it is applicable to systematic reviews of interventions and several other types of reviews [ 4 ]. Both tools reflect a shift to a domain-based approach as opposed to generic quality checklists. There are a few items unique to each tool; however, similarities between items have been demonstrated [ 98 , 99 ]. AMSTAR-2 and ROBIS are recommended for use by: 1) authors of overviews or umbrella reviews and CPGs to evaluate systematic reviews considered as evidence; 2) authors of methodological research studies to appraise included systematic reviews; and 3) peer reviewers for appraisal of submitted systematic review manuscripts. For authors, these tools may function as teaching aids and inform conduct of their review during its development.

Description

Systematic reviews that include randomized and/or non-randomized studies as evidence can be appraised with AMSTAR-2 and ROBIS. Other characteristics of AMSTAR-2 and ROBIS are summarized in Table 3.2 . Both tools define categories for an overall rating; however, neither tool is intended to generate a total score by simply calculating the number of responses satisfying criteria for individual items [ 4 , 6 ]. AMSTAR-2 focuses on the rigor of a review’s methods irrespective of the specific subject matter. ROBIS places emphasis on a review’s results section— this suggests it may be optimally applied by appraisers with some knowledge of the review’s topic as they may be better equipped to determine if certain procedures (or lack thereof) would impact the validity of a review’s findings [ 98 , 100 ]. Reliability studies show AMSTAR-2 overall confidence ratings strongly correlate with the overall RoB ratings in ROBIS [ 100 , 101 ].

Interrater reliability has been shown to be acceptable for AMSTAR-2 [ 6 , 11 , 102 ] and ROBIS [ 4 , 98 , 103 ] but neither tool has been shown to be superior in this regard [ 100 , 101 , 104 , 105 ]. Overall, variability in reliability for both tools has been reported across items, between pairs of raters, and between centers [ 6 , 100 , 101 , 104 ]. The effects of appraiser experience on the results of AMSTAR-2 and ROBIS require further evaluation [ 101 , 105 ]. Updates to both tools should address items shown to be prone to individual appraisers’ subjective biases and opinions [ 11 , 100 ]; this may involve modifications of the current domains and signaling questions as well as incorporation of methods to make an appraiser’s judgments more explicit. Future revisions of these tools may also consider the addition of standards for aspects of systematic review development currently lacking (eg, rating overall certainty of evidence, [ 99 ] methods for synthesis without meta-analysis [ 105 ]) and removal of items that assess aspects of reporting that are thoroughly evaluated by PRISMA 2020.

Application

A good understanding of what is required to satisfy the standards of AMSTAR-2 and ROBIS involves study of the accompanying guidance documents written by the tools’ developers; these contain detailed descriptions of each item’s standards. In addition, accurate appraisal of a systematic review with either tool requires training. Most experts recommend independent assessment by at least two appraisers with a process for resolving discrepancies as well as procedures to establish interrater reliability, such as pilot testing, a calibration phase or exercise, and development of predefined decision rules [ 35 , 99 , 100 , 101 , 103 , 104 , 106 ]. These methods may, to some extent, address the challenges associated with the diversity in methodological training, subject matter expertise, and experience using the tools that are likely to exist among appraisers.

The standards of AMSTAR, AMSTAR-2, and ROBIS have been used in many methodological studies and epidemiological investigations. However, the increased publication of overviews or umbrella reviews and CPGs has likely been a greater influence on the widening acceptance of these tools. Critical appraisal of the secondary studies considered evidence is essential to the trustworthiness of both the recommendations of CPGs and the conclusions of overviews. Currently both Cochrane [ 55 ] and JBI [ 107 ] recommend AMSTAR-2 and ROBIS in their guidance for authors of overviews or umbrella reviews. However, ROBIS and AMSTAR-2 were released in 2016 and 2017, respectively; thus, to date, limited data have been reported about the uptake of these tools or which of the two may be preferred [ 21 , 106 ]. Currently, in relation to CPGs, AMSTAR-2 appears to be overwhelmingly popular compared to ROBIS. A Google Scholar search of this topic (search terms “AMSTAR 2 AND clinical practice guidelines,” “ROBIS AND clinical practice guidelines” 13 May 2022) found 12,700 hits for AMSTAR-2 and 1,280 for ROBIS. The apparent greater appeal of AMSTAR-2 may relate to its longer track record given the original version of the tool was in use for 10 years prior to its update in 2017.

Barriers to the uptake of AMSTAR-2 and ROBIS include the real or perceived time and resources necessary to complete the items they include and appraisers’ confidence in their own ratings [ 104 ]. Reports from comparative studies available to date indicate that appraisers find AMSTAR-2 questions, responses, and guidance to be clearer and simpler compared with ROBIS [ 11 , 101 , 104 , 105 ]. This suggests that for appraisal of intervention systematic reviews, AMSTAR-2 may be a more practical tool than ROBIS, especially for novice appraisers [ 101 , 103 , 104 , 105 ]. The unique characteristics of each tool, as well as their potential advantages and disadvantages, should be taken into consideration when deciding which tool should be used for an appraisal of a systematic review. In addition, the choice of one or the other may depend on how the results of an appraisal will be used; for example, a peer reviewer’s appraisal of a single manuscript versus an appraisal of multiple systematic reviews in an overview or umbrella review, CPG, or systematic methodological study.

Authors of overviews and CPGs report results of AMSTAR-2 and ROBIS appraisals for each of the systematic reviews they include as evidence. Ideally, an independent judgment of their appraisals can be made by the end users of overviews and CPGs; however, most stakeholders, including clinicians, are unlikely to have a sophisticated understanding of these tools. Nevertheless, they should at least be aware that AMSTAR-2 and ROBIS ratings reported in overviews and CPGs may be inaccurate because the tools are not applied as intended by their developers. This can result from inadequate training of the overview or CPG authors who perform the appraisals, or to modifications of the appraisal tools imposed by them. The potential variability in overall confidence and RoB ratings highlights why appraisers applying these tools need to support their judgments with explicit documentation; this allows readers to judge for themselves whether they agree with the criteria used by appraisers [ 4 , 108 ]. When these judgments are explicit, the underlying rationale used when applying these tools can be assessed [ 109 ].

Theoretically, we would expect an association of AMSTAR-2 with improved methodological rigor and an association of ROBIS with lower RoB in recent systematic reviews compared to those published before 2017. To our knowledge, this has not yet been demonstrated; however, like reports about the actual uptake of these tools, time will tell. Additional data on user experience is also needed to further elucidate the practical challenges and methodological nuances encountered with the application of these tools. This information could potentially inform the creation of unifying criteria to guide and standardize the appraisal of evidence syntheses [ 109 ].

Evaluation of reporting

Complete reporting is essential for users to establish the trustworthiness and applicability of a systematic review’s findings. Efforts to standardize and improve the reporting of systematic reviews resulted in the 2009 publication of the PRISMA statement [ 92 ] with its accompanying explanation and elaboration document [ 110 ]. This guideline was designed to help authors prepare a complete and transparent report of their systematic review. In addition, adherence to PRISMA is often used to evaluate the thoroughness of reporting of published systematic reviews [ 111 ]. The updated version, PRISMA 2020 [ 93 ], and its guidance document [ 112 ] were published in 2021. Items on the original and updated versions of PRISMA are organized by the six basic review components they address (title, abstract, introduction, methods, results, discussion). The PRISMA 2020 update is a considerably expanded version of the original; it includes standards and examples for the 27 original and 13 additional reporting items that capture methodological advances and may enhance the replicability of reviews [ 113 ].

The original PRISMA statement fostered the development of various PRISMA extensions (Table 3.3 ). These include reporting guidance for scoping reviews and reviews of diagnostic test accuracy and for intervention reviews that report on the following: harms outcomes, equity issues, the effects of acupuncture, the results of network meta-analyses and analyses of individual participant data. Detailed reporting guidance for specific systematic review components (abstracts, protocols, literature searches) is also available.

Uptake and impact

The 2009 PRISMA standards [ 92 ] for reporting have been widely endorsed by authors, journals, and EBM-related organizations. We anticipate the same for PRISMA 2020 [ 93 ] given its co-publication in multiple high-impact journals. However, to date, there is a lack of strong evidence for an association between improved systematic review reporting and endorsement of PRISMA 2009 standards [ 43 , 111 ]. Most journals require a PRISMA checklist accompany submissions of systematic review manuscripts. However, the accuracy of information presented on these self-reported checklists is not necessarily verified. It remains unclear which strategies (eg, authors’ self-report of checklists, peer reviewer checks) might improve adherence to the PRISMA reporting standards; in addition, the feasibility of any potentially effective strategies must be taken into consideration given the structure and limitations of current research and publication practices [ 124 ].

Pitfalls and limitations of PRISMA, AMSTAR-2, and ROBIS

Misunderstanding of the roles of these tools and their misapplication may be widespread problems. PRISMA 2020 is a reporting guideline that is most beneficial if consulted when developing a review as opposed to merely completing a checklist when submitting to a journal; at that point, the review is finished, with good or bad methodological choices. However, PRISMA checklists evaluate how completely an element of review conduct was reported, but do not evaluate the caliber of conduct or performance of a review. Thus, review authors and readers should not think that a rigorous systematic review can be produced by simply following the PRISMA 2020 guidelines. Similarly, it is important to recognize that AMSTAR-2 and ROBIS are tools to evaluate the conduct of a review but do not substitute for conceptual methodological guidance. In addition, they are not intended to be simple checklists. In fact, they have the potential for misuse or abuse if applied as such; for example, by calculating a total score to make a judgment about a review’s overall confidence or RoB. Proper selection of a response for the individual items on AMSTAR-2 and ROBIS requires training or at least reference to their accompanying guidance documents.

Not surprisingly, it has been shown that compliance with the PRISMA checklist is not necessarily associated with satisfying the standards of ROBIS [ 125 ]. AMSTAR-2 and ROBIS were not available when PRISMA 2009 was developed; however, they were considered in the development of PRISMA 2020 [ 113 ]. Therefore, future studies may show a positive relationship between fulfillment of PRISMA 2020 standards for reporting and meeting the standards of tools evaluating methodological quality and RoB.

Choice of an appropriate tool for the evaluation of a systematic review first involves identification of the underlying construct to be assessed. For systematic reviews of interventions, recommended tools include AMSTAR-2 and ROBIS for appraisal of conduct and PRISMA 2020 for completeness of reporting. All three tools were developed rigorously and provide easily accessible and detailed user guidance, which is necessary for their proper application and interpretation. When considering a manuscript for publication, training in these tools can sensitize peer reviewers and editors to major issues that may affect the review’s trustworthiness and completeness of reporting. Judgment of the overall certainty of a body of evidence and formulation of recommendations rely, in part, on AMSTAR-2 or ROBIS appraisals of systematic reviews. Therefore, training on the application of these tools is essential for authors of overviews and developers of CPGs. Peer reviewers and editors considering an overview or CPG for publication must hold their authors to a high standard of transparency regarding both the conduct and reporting of these appraisals.

Part 4. Meeting conduct standards

Many authors, peer reviewers, and editors erroneously equate fulfillment of the items on the PRISMA checklist with superior methodological rigor. For direction on methodology, we refer them to available resources that provide comprehensive conceptual guidance [ 59 , 60 ] as well as primers with basic step-by-step instructions [ 1 , 126 , 127 ]. This section is intended to complement study of such resources by facilitating use of AMSTAR-2 and ROBIS, tools specifically developed to evaluate methodological rigor of systematic reviews. These tools are widely accepted by methodologists; however, in the general medical literature, they are not uniformly selected for the critical appraisal of systematic reviews [ 88 , 96 ].

To enable their uptake, Table 4.1  links review components to the corresponding appraisal tool items. Expectations of AMSTAR-2 and ROBIS are concisely stated, and reasoning provided.

Issues involved in meeting the standards for seven review components (identified in bold in Table 4.1 ) are addressed in detail. These were chosen for elaboration for one (or both) of two reasons: 1) the component has been identified as potentially problematic for systematic review authors based on consistent reports of their frequent AMSTAR-2 or ROBIS deficiencies [ 9 , 11 , 15 , 88 , 128 , 129 ]; and/or 2) the review component is judged by standards of an AMSTAR-2 “critical” domain. These have the greatest implications for how a systematic review will be appraised: if standards for any one of these critical domains are not met, the review is rated as having “critically low confidence.”

Research question

Specific and unambiguous research questions may have more value for reviews that deal with hypothesis testing. Mnemonics for the various elements of research questions are suggested by JBI and Cochrane (Table 2.1 ). These prompt authors to consider the specialized methods involved for developing different types of systematic reviews; however, while inclusion of the suggested elements makes a review compliant with a particular review’s methods, it does not necessarily make a research question appropriate. Table 4.2  lists acronyms that may aid in developing the research question. They include overlapping concepts of importance in this time of proliferating reviews of uncertain value [ 130 ]. If these issues are not prospectively contemplated, systematic review authors may establish an overly broad scope, or develop runaway scope allowing them to stray from predefined choices relating to key comparisons and outcomes.

Once a research question is established, searching on registry sites and databases for existing systematic reviews addressing the same or a similar topic is necessary in order to avoid contributing to research waste [ 131 ]. Repeating an existing systematic review must be justified, for example, if previous reviews are out of date or methodologically flawed. A full discussion on replication of intervention systematic reviews, including a consensus checklist, can be found in the work of Tugwell and colleagues [ 84 ].

Protocol development is considered a core component of systematic reviews [ 125 , 126 , 132 ]. Review protocols may allow researchers to plan and anticipate potential issues, assess validity of methods, prevent arbitrary decision-making, and minimize bias that can be introduced by the conduct of the review. Registration of a protocol that allows public access promotes transparency of the systematic review’s methods and processes and reduces the potential for duplication [ 132 ]. Thinking early and carefully about all the steps of a systematic review is pragmatic and logical and may mitigate the influence of the authors’ prior knowledge of the evidence [ 133 ]. In addition, the protocol stage is when the scope of the review can be carefully considered by authors, reviewers, and editors; this may help to avoid production of overly ambitious reviews that include excessive numbers of comparisons and outcomes or are undisciplined in their study selection.

An association with attainment of AMSTAR standards in systematic reviews with published prospective protocols has been reported [ 134 ]. However, completeness of reporting does not seem to be different in reviews with a protocol compared to those without one [ 135 ]. PRISMA-P [ 116 ] and its accompanying elaboration and explanation document [ 136 ] can be used to guide and assess the reporting of protocols. A final version of the review should fully describe any protocol deviations. Peer reviewers may compare the submitted manuscript with any available pre-registered protocol; this is required if AMSTAR-2 or ROBIS are used for critical appraisal.

There are multiple options for the recording of protocols (Table 4.3 ). Some journals will peer review and publish protocols. In addition, many online sites offer date-stamped and publicly accessible protocol registration. Some of these are exclusively for protocols of evidence syntheses; others are less restrictive and offer researchers the capacity for data storage, sharing, and other workflow features. These sites document protocol details to varying extents and have different requirements [ 137 ]. The most popular site for systematic reviews, the International Prospective Register of Systematic Reviews (PROSPERO), for example, only registers reviews that report on an outcome with direct relevance to human health. The PROSPERO record documents protocols for all types of reviews except literature and scoping reviews. Of note, PROSPERO requires authors register their review protocols prior to any data extraction [ 133 , 138 ]. The electronic records of most of these registry sites allow authors to update their protocols and facilitate transparent tracking of protocol changes, which are not unexpected during the progress of the review [ 139 ].

Study design inclusion

For most systematic reviews, broad inclusion of study designs is recommended [ 126 ]. This may allow comparison of results between contrasting study design types [ 126 ]. Certain study designs may be considered preferable depending on the type of review and nature of the research question. However, prevailing stereotypes about what each study design does best may not be accurate. For example, in systematic reviews of interventions, randomized designs are typically thought to answer highly specific questions while non-randomized designs often are expected to reveal greater information about harms or real-word evidence [ 126 , 140 , 141 ]. This may be a false distinction; randomized trials may be pragmatic [ 142 ], they may offer important (and more unbiased) information on harms [ 143 ], and data from non-randomized trials may not necessarily be more real-world-oriented [ 144 ].

Moreover, there may not be any available evidence reported by RCTs for certain research questions; in some cases, there may not be any RCTs or NRSI. When the available evidence is limited to case reports and case series, it is not possible to test hypotheses nor provide descriptive estimates or associations; however, a systematic review of these studies can still offer important insights [ 81 , 145 ]. When authors anticipate that limited evidence of any kind may be available to inform their research questions, a scoping review can be considered. Alternatively, decisions regarding inclusion of indirect as opposed to direct evidence can be addressed during protocol development [ 146 ]. Including indirect evidence at an early stage of intervention systematic review development allows authors to decide if such studies offer any additional and/or different understanding of treatment effects for their population or comparison of interest. Issues of indirectness of included studies are accounted for later in the process, during determination of the overall certainty of evidence (see Part 5 for details).

Evidence search

Both AMSTAR-2 and ROBIS require systematic and comprehensive searches for evidence. This is essential for any systematic review. Both tools discourage search restrictions based on language and publication source. Given increasing globalism in health care, the practice of including English-only literature should be avoided [ 126 ]. There are many examples in which language bias (different results in studies published in different languages) has been documented [ 147 , 148 ]. This does not mean that all literature, in all languages, is equally trustworthy [ 148 ]; however, the only way to formally probe for the potential of such biases is to consider all languages in the initial search. The gray literature and a search of trials may also reveal important details about topics that would otherwise be missed [ 149 , 150 , 151 ]. Again, inclusiveness will allow review authors to investigate whether results differ in gray literature and trials [ 41 , 151 , 152 , 153 ].

Authors should make every attempt to complete their review within one year as that is the likely viable life of a search. (1) If that is not possible, the search should be updated close to the time of completion [ 154 ]. Different research topics may warrant less of a delay, for example, in rapidly changing fields (as in the case of the COVID-19 pandemic), even one month may radically change the available evidence.

Excluded studies

AMSTAR-2 requires authors to provide references for any studies excluded at the full text phase of study selection along with reasons for exclusion; this allows readers to feel confident that all relevant literature has been considered for inclusion and that exclusions are defensible.

Risk of bias assessment of included studies

The design of the studies included in a systematic review (eg, RCT, cohort, case series) should not be equated with appraisal of its RoB. To meet AMSTAR-2 and ROBIS standards, systematic review authors must examine RoB issues specific to the design of each primary study they include as evidence. It is unlikely that a single RoB appraisal tool will be suitable for all research designs. In addition to tools for randomized and non-randomized studies, specific tools are available for evaluation of RoB in case reports and case series [ 82 ] and single-case experimental designs [ 155 , 156 ]. Note the RoB tools selected must meet the standards of the appraisal tool used to judge the conduct of the review. For example, AMSTAR-2 identifies four sources of bias specific to RCTs and NRSI that must be addressed by the RoB tool(s) chosen by the review authors. The Cochrane RoB-2 [ 157 ] tool for RCTs and ROBINS-I [ 158 ] for NRSI for RoB assessment meet the AMSTAR-2 standards. Appraisers on the review team should not modify any RoB tool without complete transparency and acknowledgment that they have invalidated the interpretation of the tool as intended by its developers [ 159 ]. Conduct of RoB assessments is not addressed AMSTAR-2; to meet ROBIS standards, two independent reviewers should complete RoB assessments of included primary studies.

Implications of the RoB assessments must be explicitly discussed and considered in the conclusions of the review. Discussion of the overall RoB of included studies may consider the weight of the studies at high RoB, the importance of the sources of bias in the studies being summarized, and if their importance differs in relationship to the outcomes reported. If a meta-analysis is performed, serious concerns for RoB of individual studies should be accounted for in these results as well. If the results of the meta-analysis for a specific outcome change when studies at high RoB are excluded, readers will have a more accurate understanding of this body of evidence. However, while investigating the potential impact of specific biases is a useful exercise, it is important to avoid over-interpretation, especially when there are sparse data.

Synthesis methods for quantitative data

Syntheses of quantitative data reported by primary studies are broadly categorized as one of two types: meta-analysis, and synthesis without meta-analysis (Table 4.4 ). Before deciding on one of these methods, authors should seek methodological advice about whether reported data can be transformed or used in other ways to provide a consistent effect measure across studies [ 160 , 161 ].

Meta-analysis

Systematic reviews that employ meta-analysis should not be referred to simply as “meta-analyses.” The term meta-analysis strictly refers to a specific statistical technique used when study effect estimates and their variances are available, yielding a quantitative summary of results. In general, methods for meta-analysis involve use of a weighted average of effect estimates from two or more studies. If considered carefully, meta-analysis increases the precision of the estimated magnitude of effect and can offer useful insights about heterogeneity and estimates of effects. We refer to standard references for a thorough introduction and formal training [ 165 , 166 , 167 ].

There are three common approaches to meta-analysis in current health care–related systematic reviews (Table 4.4 ). Aggregate meta-analyses is the most familiar to authors of evidence syntheses and their end users. This standard meta-analysis combines data on effect estimates reported by studies that investigate similar research questions involving direct comparisons of an intervention and comparator. Results of these analyses provide a single summary intervention effect estimate. If the included studies in a systematic review measure an outcome differently, their reported results may be transformed to make them comparable [ 161 ]. Forest plots visually present essential information about the individual studies and the overall pooled analysis (see Additional File 4  for details).

Less familiar and more challenging meta-analytical approaches used in secondary research include individual participant data (IPD) and network meta-analyses (NMA); PRISMA extensions provide reporting guidelines for both [ 117 , 118 ]. In IPD, the raw data on each participant from each eligible study are re-analyzed as opposed to the study-level data analyzed in aggregate data meta-analyses [ 168 ]. This may offer advantages, including the potential for limiting concerns about bias and allowing more robust analyses [ 163 ]. As suggested by the description in Table 4.4 , NMA is a complex statistical approach. It combines aggregate data [ 169 ] or IPD [ 170 ] for effect estimates from direct and indirect comparisons reported in two or more studies of three or more interventions. This makes it a potentially powerful statistical tool; while multiple interventions are typically available to treat a condition, few have been evaluated in head-to-head trials [ 171 ]. Both IPD and NMA facilitate a broader scope, and potentially provide more reliable and/or detailed results; however, compared with standard aggregate data meta-analyses, their methods are more complicated, time-consuming, and resource-intensive, and they have their own biases, so one needs sufficient funding, technical expertise, and preparation to employ them successfully [ 41 , 172 , 173 ].

Several items in AMSTAR-2 and ROBIS address meta-analysis; thus, understanding the strengths, weaknesses, assumptions, and limitations of methods for meta-analyses is important. According to the standards of both tools, plans for a meta-analysis must be addressed in the review protocol, including reasoning, description of the type of quantitative data to be synthesized, and the methods planned for combining the data. This should not consist of stock statements describing conventional meta-analysis techniques; rather, authors are expected to anticipate issues specific to their research questions. Concern for the lack of training in meta-analysis methods among systematic review authors cannot be overstated. For those with training, the use of popular software (eg, RevMan [ 174 ], MetaXL [ 175 ], JBI SUMARI [ 176 ]) may facilitate exploration of these methods; however, such programs cannot substitute for the accurate interpretation of the results of meta-analyses, especially for more complex meta-analytical approaches.

Synthesis without meta-analysis

There are varied reasons a meta-analysis may not be appropriate or desirable [ 160 , 161 ]. Syntheses that informally use statistical methods other than meta-analysis are variably referred to as descriptive, narrative, or qualitative syntheses or summaries; these terms are also applied to syntheses that make no attempt to statistically combine data from individual studies. However, use of such imprecise terminology is discouraged; in order to fully explore the results of any type of synthesis, some narration or description is needed to supplement the data visually presented in tabular or graphic forms [ 63 , 177 ]. In addition, the term “qualitative synthesis” is easily confused with a synthesis of qualitative data in a qualitative or mixed methods review. “Synthesis without meta-analysis” is currently the preferred description of other ways to combine quantitative data from two or more studies. Use of this specific terminology when referring to these types of syntheses also implies the application of formal methods (Table 4.4 ).

Methods for syntheses without meta-analysis involve structured presentations of the data in any tables and plots. In comparison to narrative descriptions of each study, these are designed to more effectively and transparently show patterns and convey detailed information about the data; they also allow informal exploration of heterogeneity [ 178 ]. In addition, acceptable quantitative statistical methods (Table 4.4 ) are formally applied; however, it is important to recognize these methods have significant limitations for the interpretation of the effectiveness of an intervention [ 160 ]. Nevertheless, when meta-analysis is not possible, the application of these methods is less prone to bias compared with an unstructured narrative description of included studies [ 178 , 179 ].

Vote counting is commonly used in systematic reviews and involves a tally of studies reporting results that meet some threshold of importance applied by review authors. Until recently, it has not typically been identified as a method for synthesis without meta-analysis. Guidance on an acceptable vote counting method based on direction of effect is currently available [ 160 ] and should be used instead of narrative descriptions of such results (eg, “more than half the studies showed improvement”; “only a few studies reported adverse effects”; “7 out of 10 studies favored the intervention”). Unacceptable methods include vote counting by statistical significance or magnitude of effect or some subjective rule applied by the authors.

AMSTAR-2 and ROBIS standards do not explicitly address conduct of syntheses without meta-analysis, although AMSTAR-2 items 13 and 14 might be considered relevant. Guidance for the complete reporting of syntheses without meta-analysis for systematic reviews of interventions is available in the Synthesis without Meta-analysis (SWiM) guideline [ 180 ] and methodological guidance is available in the Cochrane Handbook [ 160 , 181 ].

Familiarity with AMSTAR-2 and ROBIS makes sense for authors of systematic reviews as these appraisal tools will be used to judge their work; however, training is necessary for authors to truly appreciate and apply methodological rigor. Moreover, judgment of the potential contribution of a systematic review to the current knowledge base goes beyond meeting the standards of AMSTAR-2 and ROBIS. These tools do not explicitly address some crucial concepts involved in the development of a systematic review; this further emphasizes the need for author training.

We recommend that systematic review authors incorporate specific practices or exercises when formulating a research question at the protocol stage, These should be designed to raise the review team’s awareness of how to prevent research and resource waste [ 84 , 130 ] and to stimulate careful contemplation of the scope of the review [ 30 ]. Authors’ training should also focus on justifiably choosing a formal method for the synthesis of quantitative and/or qualitative data from primary research; both types of data require specific expertise. For typical reviews that involve syntheses of quantitative data, statistical expertise is necessary, initially for decisions about appropriate methods, [ 160 , 161 ] and then to inform any meta-analyses [ 167 ] or other statistical methods applied [ 160 ].

Part 5. Rating overall certainty of evidence

Report of an overall certainty of evidence assessment in a systematic review is an important new reporting standard of the updated PRISMA 2020 guidelines [ 93 ]. Systematic review authors are well acquainted with assessing RoB in individual primary studies, but much less familiar with assessment of overall certainty across an entire body of evidence. Yet a reliable way to evaluate this broader concept is now recognized as a vital part of interpreting the evidence.

Historical systems for rating evidence are based on study design and usually involve hierarchical levels or classes of evidence that use numbers and/or letters to designate the level/class. These systems were endorsed by various EBM-related organizations. Professional societies and regulatory groups then widely adopted them, often with modifications for application to the available primary research base in specific clinical areas. In 2002, a report issued by the AHRQ identified 40 systems to rate quality of a body of evidence [ 182 ]. A critical appraisal of systems used by prominent health care organizations published in 2004 revealed limitations in sensibility, reproducibility, applicability to different questions, and usability to different end users [ 183 ]. Persistent use of hierarchical rating schemes to describe overall quality continues to complicate the interpretation of evidence. This is indicated by recent reports of poor interpretability of systematic review results by readers [ 184 , 185 , 186 ] and misleading interpretations of the evidence related to the “spin” systematic review authors may put on their conclusions [ 50 , 187 ].

Recognition of the shortcomings of hierarchical rating systems raised concerns that misleading clinical recommendations could result even if based on a rigorous systematic review. In addition, the number and variability of these systems were considered obstacles to quick and accurate interpretations of the evidence by clinicians, patients, and policymakers [ 183 ]. These issues contributed to the development of the GRADE approach. An international working group, that continues to actively evaluate and refine it, first introduced GRADE in 2004 [ 188 ]. Currently more than 110 organizations from 19 countries around the world have endorsed or are using GRADE [ 189 ].

GRADE approach to rating overall certainty

GRADE offers a consistent and sensible approach for two separate processes: rating the overall certainty of a body of evidence and the strength of recommendations. The former is the expected conclusion of a systematic review, while the latter is pertinent to the development of CPGs. As such, GRADE provides a mechanism to bridge the gap from evidence synthesis to application of the evidence for informed clinical decision-making [ 27 , 190 ]. We briefly examine the GRADE approach but only as it applies to rating overall certainty of evidence in systematic reviews.

In GRADE, use of “certainty” of a body of evidence is preferred over the term “quality.” [ 191 ] Certainty refers to the level of confidence systematic review authors have that, for each outcome, an effect estimate represents the true effect. The GRADE approach to rating confidence in estimates begins with identifying the study type (RCT or NRSI) and then systematically considers criteria to rate the certainty of evidence up or down (Table 5.1 ).

This process results in assignment of one of the four GRADE certainty ratings to each outcome; these are clearly conveyed with the use of basic interpretation symbols (Table 5.2 ) [ 192 ]. Notably, when multiple outcomes are reported in a systematic review, each outcome is assigned a unique certainty rating; thus different levels of certainty may exist in the body of evidence being examined.

GRADE’s developers acknowledge some subjectivity is involved in this process [ 193 ]. In addition, they emphasize that both the criteria for rating evidence up and down (Table 5.1 ) as well as the four overall certainty ratings (Table 5.2 ) reflect a continuum as opposed to discrete categories [ 194 ]. Consequently, deciding whether a study falls above or below the threshold for rating up or down may not be straightforward, and preliminary overall certainty ratings may be intermediate (eg, between low and moderate). Thus, the proper application of GRADE requires systematic review authors to take an overall view of the body of evidence and explicitly describe the rationale for their final ratings.

Advantages of GRADE

Outcomes important to the individuals who experience the problem of interest maintain a prominent role throughout the GRADE process [ 191 ]. These outcomes must inform the research questions (eg, PICO [population, intervention, comparator, outcome]) that are specified a priori in a systematic review protocol. Evidence for these outcomes is then investigated and each critical or important outcome is ultimately assigned a certainty of evidence as the end point of the review. Notably, limitations of the included studies have an impact at the outcome level. Ultimately, the certainty ratings for each outcome reported in a systematic review are considered by guideline panels. They use a different process to formulate recommendations that involves assessment of the evidence across outcomes [ 201 ]. It is beyond our scope to describe the GRADE process for formulating recommendations; however, it is critical to understand how these two outcome-centric concepts of certainty of evidence in the GRADE framework are related and distinguished. An in-depth illustration using examples from recently published evidence syntheses and CPGs is provided in Additional File 5 A (Table AF5A-1).

The GRADE approach is applicable irrespective of whether the certainty of the primary research evidence is high or very low; in some circumstances, indirect evidence of higher certainty may be considered if direct evidence is unavailable or of low certainty [ 27 ]. In fact, most interventions and outcomes in medicine have low or very low certainty of evidence based on GRADE and there seems to be no major improvement over time [ 202 , 203 ]. This is still a very important (even if sobering) realization for calibrating our understanding of medical evidence. A major appeal of the GRADE approach is that it offers a common framework that enables authors of evidence syntheses to make complex judgments about evidence certainty and to convey these with unambiguous terminology. This prevents some common mistakes made by review authors, including overstating results (or under-reporting harms) [ 187 ] and making recommendations for treatment. This is illustrated in Table AF5A-2 (Additional File 5 A), which compares the concluding statements made about overall certainty in a systematic review with and without application of the GRADE approach.

Theoretically, application of GRADE should improve consistency of judgments about certainty of evidence, both between authors and across systematic reviews. In one empirical evaluation conducted by the GRADE Working Group, interrater reliability of two individual raters assessing certainty of the evidence for a specific outcome increased from ~ 0.3 without using GRADE to ~ 0.7 by using GRADE [ 204 ]. However, others report variable agreement among those experienced in GRADE assessments of evidence certainty [ 190 ]. Like any other tool, GRADE requires training in order to be properly applied. The intricacies of the GRADE approach and the necessary subjectivity involved suggest that improving agreement may require strict rules for its application; alternatively, use of general guidance and consensus among review authors may result in less consistency but provide important information for the end user [ 190 ].

GRADE caveats

Simply invoking “the GRADE approach” does not automatically ensure GRADE methods were employed by authors of a systematic review (or developers of a CPG). Table 5.3 lists the criteria the GRADE working group has established for this purpose. These criteria highlight the specific terminology and methods that apply to rating the certainty of evidence for outcomes reported in a systematic review [ 191 ], which is different from rating overall certainty across outcomes considered in the formulation of recommendations [ 205 ]. Modifications of standard GRADE methods and terminology are discouraged as these may detract from GRADE’s objectives to minimize conceptual confusion and maximize clear communication [ 206 ].

Nevertheless, GRADE is prone to misapplications [ 207 , 208 ], which can distort a systematic review’s conclusions about the certainty of evidence. Systematic review authors without proper GRADE training are likely to misinterpret the terms “quality” and “grade” and to misunderstand the constructs assessed by GRADE versus other appraisal tools. For example, review authors may reference the standard GRADE certainty ratings (Table 5.2 ) to describe evidence for their outcome(s) of interest. However, these ratings are invalidated if authors omit or inadequately perform RoB evaluations of each included primary study. Such deficiencies in RoB assessments are unacceptable but not uncommon, as reported in methodological studies of systematic reviews and overviews [ 104 , 186 , 209 , 210 ]. GRADE ratings are also invalidated if review authors do not formally address and report on the other criteria (Table 5.1 ) necessary for a GRADE certainty rating.

Other caveats pertain to application of a GRADE certainty of evidence rating in various types of evidence syntheses. Current adaptations of GRADE are described in Additional File 5 B and included on Table 6.3 , which is introduced in the next section.

The expected culmination of a systematic review should be a rating of overall certainty of a body of evidence for each outcome reported. The GRADE approach is recommended for making these judgments for outcomes reported in systematic reviews of interventions and can be adapted for other types of reviews. This represents the initial step in the process of making recommendations based on evidence syntheses. Peer reviewers should ensure authors meet the minimal criteria for supporting the GRADE approach when reviewing any evidence synthesis that reports certainty ratings derived using GRADE. Authors and peer reviewers of evidence syntheses unfamiliar with GRADE are encouraged to seek formal training and take advantage of the resources available on the GRADE website [ 211 , 212 ].

Part 6. Concise Guide to best practices

Accumulating data in recent years suggest that many evidence syntheses (with or without meta-analysis) are not reliable. This relates in part to the fact that their authors, who are often clinicians, can be overwhelmed by the plethora of ways to evaluate evidence. They tend to resort to familiar but often inadequate, inappropriate, or obsolete methods and tools and, as a result, produce unreliable reviews. These manuscripts may not be recognized as such by peer reviewers and journal editors who may disregard current standards. When such a systematic review is published or included in a CPG, clinicians and stakeholders tend to believe that it is trustworthy. A vicious cycle in which inadequate methodology is rewarded and potentially misleading conclusions are accepted is thus supported. There is no quick or easy way to break this cycle; however, increasing awareness of best practices among all these stakeholder groups, who often have minimal (if any) training in methodology, may begin to mitigate it. This is the rationale for inclusion of Parts 2 through 5 in this guidance document. These sections present core concepts and important methodological developments that inform current standards and recommendations. We conclude by taking a direct and practical approach.

Inconsistent and imprecise terminology used in the context of development and evaluation of evidence syntheses is problematic for authors, peer reviewers and editors, and may lead to the application of inappropriate methods and tools. In response, we endorse use of the basic terms (Table 6.1 ) defined in the PRISMA 2020 statement [ 93 ]. In addition, we have identified several problematic expressions and nomenclature. In Table 6.2 , we compile suggestions for preferred terms less likely to be misinterpreted.

We also propose a Concise Guide (Table 6.3 ) that summarizes the methods and tools recommended for the development and evaluation of nine types of evidence syntheses. Suggestions for specific tools are based on the rigor of their development as well as the availability of detailed guidance from their developers to ensure their proper application. The formatting of the Concise Guide addresses a well-known source of confusion by clearly distinguishing the underlying methodological constructs that these tools were designed to assess. Important clarifications and explanations follow in the guide’s footnotes; associated websites, if available, are listed in Additional File 6 .

To encourage uptake of best practices, journal editors may consider adopting or adapting the Concise Guide in their instructions to authors and peer reviewers of evidence syntheses. Given the evolving nature of evidence synthesis methodology, the suggested methods and tools are likely to require regular updates. Authors of evidence syntheses should monitor the literature to ensure they are employing current methods and tools. Some types of evidence syntheses (eg, rapid, economic, methodological) are not included in the Concise Guide; for these, authors are advised to obtain recommendations for acceptable methods by consulting with their target journal.

We encourage the appropriate and informed use of the methods and tools discussed throughout this commentary and summarized in the Concise Guide (Table 6.3 ). However, we caution against their application in a perfunctory or superficial fashion. This is a common pitfall among authors of evidence syntheses, especially as the standards of such tools become associated with acceptance of a manuscript by a journal. Consequently, published evidence syntheses may show improved adherence to the requirements of these tools without necessarily making genuine improvements in their performance.

In line with our main objective, the suggested tools in the Concise Guide address the reliability of evidence syntheses; however, we recognize that the utility of systematic reviews is an equally important concern. An unbiased and thoroughly reported evidence synthesis may still not be highly informative if the evidence itself that is summarized is sparse, weak and/or biased [ 24 ]. Many intervention systematic reviews, including those developed by Cochrane [ 203 ] and those applying GRADE [ 202 ], ultimately find no evidence, or find the evidence to be inconclusive (eg, “weak,” “mixed,” or of “low certainty”). This often reflects the primary research base; however, it is important to know what is known (or not known) about a topic when considering an intervention for patients and discussing treatment options with them.

Alternatively, the frequency of “empty” and inconclusive reviews published in the medical literature may relate to limitations of conventional methods that focus on hypothesis testing; these have emphasized the importance of statistical significance in primary research and effect sizes from aggregate meta-analyses [ 183 ]. It is becoming increasingly apparent that this approach may not be appropriate for all topics [ 130 ]. Development of the GRADE approach has facilitated a better understanding of significant factors (beyond effect size) that contribute to the overall certainty of evidence. Other notable responses include the development of integrative synthesis methods for the evaluation of complex interventions [ 230 , 231 ], the incorporation of crowdsourcing and machine learning into systematic review workflows (eg the Cochrane Evidence Pipeline) [ 2 ], the shift in paradigm to living systemic review and NMA platforms [ 232 , 233 ] and the proposal of a new evidence ecosystem that fosters bidirectional collaborations and interactions among a global network of evidence synthesis stakeholders [ 234 ]. These evolutions in data sources and methods may ultimately make evidence syntheses more streamlined, less duplicative, and more importantly, they may be more useful for timely policy and clinical decision-making; however, that will only be the case if they are rigorously reported and conducted.

We look forward to others’ ideas and proposals for the advancement of methods for evidence syntheses. For now, we encourage dissemination and uptake of the currently accepted best tools and practices for their development and evaluation; at the same time, we stress that uptake of appraisal tools, checklists, and software programs cannot substitute for proper education in the methodology of evidence syntheses and meta-analysis. Authors, peer reviewers, and editors must strive to make accurate and reliable contributions to the present evidence knowledge base; online alerts, upcoming technology, and accessible education may make this more feasible than ever before. Our intention is to improve the trustworthiness of evidence syntheses across disciplines, topics, and types of evidence syntheses. All of us must continue to study, teach, and act cooperatively for that to happen.

Muka T, Glisic M, Milic J, Verhoog S, Bohlius J, Bramer W, et al. A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research. Eur J Epidemiol. 2020;35(1):49–60.

Article   PubMed   Google Scholar  

Thomas J, McDonald S, Noel-Storr A, Shemilt I, Elliott J, Mavergames C, et al. Machine learning reduced workload with minimal risk of missing studies: development and evaluation of a randomized controlled trial classifier for cochrane reviews. J Clin Epidemiol. 2021;133:140–51.

Article   PubMed   PubMed Central   Google Scholar  

Fontelo P, Liu F. A review of recent publication trends from top publishing countries. Syst Rev. 2018;7(1):147.

Whiting P, Savović J, Higgins JPT, Caldwell DM, Reeves BC, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–34.

Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:1–7.

Article   Google Scholar  

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358: j4008.

Goldkuhle M, Narayan VM, Weigl A, Dahm P, Skoetz N. A systematic assessment of Cochrane reviews and systematic reviews published in high-impact medical journals related to cancer. BMJ Open. 2018;8(3): e020869.

Ho RS, Wu X, Yuan J, Liu S, Lai X, Wong SY, et al. Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease: a cross-sectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. NPJ Prim Care Respir Med. 2015;25:14102.

Tsoi AKN, Ho LTF, Wu IXY, Wong CHL, Ho RST, Lim JYY, et al. Methodological quality of systematic reviews on treatments for osteoporosis: a cross-sectional study. Bone. 2020;139(June): 115541.

Arienti C, Lazzarini SG, Pollock A, Negrini S. Rehabilitation interventions for improving balance following stroke: an overview of systematic reviews. PLoS ONE. 2019;14(7):1–23.

Kolaski K, Romeiser Logan L, Goss KD, Butler C. Quality appraisal of systematic reviews of interventions for children with cerebral palsy reveals critically low confidence. Dev Med Child Neurol. 2021;63(11):1316–26.

Almeida MO, Yamato TP, Parreira PCS, do Costa LOP, Kamper S, Saragiotto BT. Overall confidence in the results of systematic reviews on exercise therapy for chronic low back pain: a cross-sectional analysis using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) 2 tool. Braz J Phys Ther. 2020;24(2):103–17.

Mayo-Wilson E, Ng SM, Chuck RS, Li T. The quality of systematic reviews about interventions for refractive error can be improved: a review of systematic reviews. BMC Ophthalmol. 2017;17(1):1–10.

Matthias K, Rissling O, Pieper D, Morche J, Nocon M, Jacobs A, et al. The methodological quality of systematic reviews on the treatment of adult major depression needs improvement according to AMSTAR 2: a cross-sectional study. Heliyon. 2020;6(9): e04776.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Riado Minguez D, Kowalski M, Vallve Odena M, Longin Pontzen D, Jelicic Kadic A, Jeric M, et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth Analg. 2017;125(4):1348–54.

Churuangsuk C, Kherouf M, Combet E, Lean M. Low-carbohydrate diets for overweight and obesity: a systematic review of the systematic reviews. Obes Rev. 2018;19(12):1700–18.

Article   CAS   PubMed   Google Scholar  

Storman M, Storman D, Jasinska KW, Swierz MJ, Bala MM. The quality of systematic reviews/meta-analyses published in the field of bariatrics: a cross-sectional systematic survey using AMSTAR 2 and ROBIS. Obes Rev. 2020;21(5):1–11.

Franco JVA, Arancibia M, Meza N, Madrid E, Kopitowski K. [Clinical practice guidelines: concepts, limitations and challenges]. Medwave. 2020;20(3):e7887 ([Spanish]).

Brito JP, Tsapas A, Griebeler ML, Wang Z, Prutsky GJ, Domecq JP, et al. Systematic reviews supporting practice guideline recommendations lack protection against bias. J Clin Epidemiol. 2013;66(6):633–8.

Zhou Q, Wang Z, Shi Q, Zhao S, Xun Y, Liu H, et al. Clinical epidemiology in China series. Paper 4: the reporting and methodological quality of Chinese clinical practice guidelines published between 2014 and 2018: a systematic review. J Clin Epidemiol. 2021;140:189–99.

Lunny C, Ramasubbu C, Puil L, Liu T, Gerrish S, Salzwedel DM, et al. Over half of clinical practice guidelines use non-systematic methods to inform recommendations: a methods study. PLoS ONE. 2021;16(4):1–21.

Faber T, Ravaud P, Riveros C, Perrodeau E, Dechartres A. Meta-analyses including non-randomized studies of therapeutic interventions: a methodological review. BMC Med Res Methodol. 2016;16(1):1–26.

Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. Milbank Q. 2016;94(3):485–514.

Møller MH, Ioannidis JPA, Darmon M. Are systematic reviews and meta-analyses still useful research? We are not sure. Intensive Care Med. 2018;44(4):518–20.

Moher D, Glasziou P, Chalmers I, Nasser M, Bossuyt PMM, Korevaar DA, et al. Increasing value and reducing waste in biomedical research: who’s listening? Lancet. 2016;387(10027):1573–86.

Barnard ND, Willet WC, Ding EL. The misuse of meta-analysis in nutrition research. JAMA. 2017;318(15):1435–6.

Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94.

Page MJ, Shamseer L, Altman DG, Tetzlaff J, Sampson M, Tricco AC, et al. Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study. PLoS Med. 2016;13(5):1–31.

World Health Organization. WHO handbook for guideline development, 2nd edn. WHO; 2014. Available from: https://www.who.int/publications/i/item/9789241548960 . Cited 2022 Jan 20

Higgins J, Lasserson T, Chandler J, Tovey D, Thomas J, Flemying E, et al. Methodological expectations of Cochrane intervention reviews. Cochrane; 2022. Available from: https://community.cochrane.org/mecir-manual/key-points-and-introduction . Cited 2022 Jul 19

Cumpston M, Chandler J. Chapter II: Planning a Cochrane review. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook . Cited 2022 Jan 30

Henderson LK, Craig JC, Willis NS, Tovey D, Webster AC. How to write a cochrane systematic review. Nephrology. 2010;15(6):617–24.

Page MJ, Altman DG, Shamseer L, McKenzie JE, Ahmadzai N, Wolfe D, et al. Reproducible research practices are underused in systematic reviews of biomedical interventions. J Clin Epidemiol. 2018;94:8–18.

Lorenz RC, Matthias K, Pieper D, Wegewitz U, Morche J, Nocon M, et al. AMSTAR 2 overall confidence rating: lacking discriminating capacity or requirement of high methodological quality? J Clin Epidemiol. 2020;119:142–4.

Posadzki P, Pieper D, Bajpai R, Makaruk H, Könsgen N, Neuhaus AL, et al. Exercise/physical activity and health outcomes: an overview of Cochrane systematic reviews. BMC Public Health. 2020;20(1):1–12.

Wells G, Shea B, O’Connell D, Peterson J, Welch V, Losos M. The Newcastile-Ottawa Scale (NOS) for assessing the quality of nonrandomized studies in meta-analyses. The Ottawa Hospital; 2009. Available from: https://www.ohri.ca/programs/clinical_epidemiology/oxford.asp . Cited 2022 Jul 19

Stang A. Critical evaluation of the Newcastle-Ottawa scale for the assessment of the quality of nonrandomized studies in meta-analyses. Eur J Epidemiol. 2010;25(9):603–5.

Stang A, Jonas S, Poole C. Case study in major quotation errors: a critical commentary on the Newcastle-Ottawa scale. Eur J Epidemiol. 2018;33(11):1025–31.

Ioannidis JPA. Massive citations to misleading methods and research tools: Matthew effect, quotation error and citation copying. Eur J Epidemiol. 2018;33(11):1021–3.

Khalil H, Ameen D, Zarnegar A. Tools to support the automation of systematic reviews: a scoping review. J Clin Epidemiol. 2022;144:22–42.

Crequit P, Boutron I, Meerpohl J, Williams H, Craig J, Ravaud P. Future of evidence ecosystem series: 2. Current opportunities and need for better tools and methods. J Clin Epidemiol. 2020;123:143–52.

Shemilt I, Noel-Storr A, Thomas J, Featherstone R, Mavergames C. Machine learning reduced workload for the cochrane COVID-19 study register: development and evaluation of the cochrane COVID-19 study classifier. Syst Rev. 2022;11(1):15.

Nguyen P-Y, Kanukula R, McKensie J, Alqaidoom Z, Brennan SE, Haddaway N, et al. Changing patterns in reporting and sharing of review data in systematic reviews with meta-analysis of the effects of interventions: a meta-research study. medRxiv; 2022 Available from: https://doi.org/10.1101/2022.04.11.22273688v3 . Cited 2022 Nov 18

Afshari A, Møller MH. Broken science and the failure of academics—resignation or reaction? Acta Anaesthesiol Scand. 2018;62(8):1038–40.

Butler E, Granholm A, Aneman A. Trustworthy systematic reviews–can journals do more? Acta Anaesthesiol Scand. 2019;63(4):558–9.

Negrini S, Côté P, Kiekens C. Methodological quality of systematic reviews on interventions for children with cerebral palsy: the evidence pyramid paradox. Dev Med Child Neurol. 2021;63(11):1244–5.

Page MJ, Moher D. Mass production of systematic reviews and meta-analyses: an exercise in mega-silliness? Milbank Q. 2016;94(3):515–9.

Clarke M, Chalmers I. Reflections on the history of systematic reviews. BMJ Evid Based Med. 2018;23(4):121–2.

Alnemer A, Khalid M, Alhuzaim W, Alnemer A, Ahmed B, Alharbi B, et al. Are health-related tweets evidence based? Review and analysis of health-related tweets on twitter. J Med Internet Res. 2015;17(10): e246.

PubMed   PubMed Central   Google Scholar  

Haber N, Smith ER, Moscoe E, Andrews K, Audy R, Bell W, et al. Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): a systematic review. PLoS ONE. 2018;13(5): e196346.

Swetland SB, Rothrock AN, Andris H, Davis B, Nguyen L, Davis P, et al. Accuracy of health-related information regarding COVID-19 on Twitter during a global pandemic. World Med Heal Policy. 2021;13(3):503–17.

Nascimento DP, Almeida MO, Scola LFC, Vanin AA, Oliveira LA, Costa LCM, et al. Letter to the editor – not even the top general medical journals are free of spin: a wake-up call based on an overview of reviews. J Clin Epidemiol. 2021;139:232–4.

Ioannidis JPA, Fanelli D, Dunne DD, Goodman SN. Meta-research: evaluation and improvement of research methods and practices. PLoS Biol. 2015;13(10):1–7.

Munn Z, Stern C, Aromataris E, Lockwood C, Jordan Z. What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences. BMC Med Res Methodol. 2018;18(1):1–9.

Pollock M, Fernandez R, Becker LA, Pieper D, Hartling L. Chapter V: overviews of reviews. Cochrane handbook for systematic reviews of interventions. In:  Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane; 2022. Available from: https://training.cochrane.org/handbook/current/chapter-v . Cited 2022 Mar 7

Tricco AC, Lillie E, Zarin W, O’Brien K, Colquhoun H, Kastner M, et al. A scoping review on the conduct and reporting of scoping reviews. BMC Med Res Methodol. 2016;16(1):1–10.

Garritty C, Gartlehner G, Nussbaumer-Streit B, King VJ, Hamel C, Kamel C, et al. Cochrane rapid reviews methods group offers evidence-informed guidance to conduct rapid reviews. J Clin Epidemiol. 2021;130:13–22.

Elliott JH, Synnot A, Turner T, Simmonds M, Akl EA, McDonald S, et al. Living systematic review: 1. Introduction—the why, what, when, and how. J Clin Epidemiol. 2017;91:23–30.

Higgins JPT, Thomas J, Chandler J. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook . Cited 2022 Jan 25

Aromataris E, Munn Z. JBI Manual for Evidence Synthesis [internet]. JBI; 2020 [cited 2022 Jan 15]. Available from: https://synthesismanual.jbi.global .

Tufanaru C, Munn Z, Aromartaris E, Campbell J, Hopp L. Chapter 3: Systematic reviews of effectiveness. In Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis [internet]. JBI; 2020 [cited 2022 Jan 25]. Available from: https://synthesismanual.jbi.global .

Leeflang MMG, Davenport C, Bossuyt PM. Defining the review question. In: Deeks JJ, Bossuyt PM, Leeflang MMG, Takwoingi Y, editors. Cochrane handbook for systematic reviews of diagnostic test accuracy [internet]. Cochrane; 2022 [cited 2022 Mar 30]. Available from: https://training.cochrane.org/6-defining-review-question .

Noyes J, Booth A, Cargo M, Flemming K, Harden A, Harris J, et al.Qualitative evidence. In: Higgins J, Tomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions [internet]. Cochrane; 2022 [cited 2022 Mar 30]. Available from: https://training.cochrane.org/handbook/current/chapter-21#section-21-5 .

Lockwood C, Porritt K, Munn Z, Rittenmeyer L, Salmond S, Bjerrum M, et al. Chapter 2: Systematic reviews of qualitative evidence. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis [internet]. JBI; 2020 [cited 2022 Jul 11]. Available from: https://synthesismanual.jbi.global .

Debray TPA, Damen JAAG, Snell KIE, Ensor J, Hooft L, Reitsma JB, et al. A guide to systematic review and meta-analysis of prediction model performance. BMJ. 2017;356:i6460.

Moola S, Munn Z, Tufanaru C, Aromartaris E, Sears K, Sfetcu R, et al. Systematic reviews of etiology and risk. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis [internet]. JBI; 2020 [cited 2022 Mar 30]. Available from: https://synthesismanual.jbi.global/ .

Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN checklist for assessing the methodological quality of studies on measurement properties of health status measurement instruments: an international Delphi study. Qual Life Res. 2010;19(4):539–49.

Prinsen CAC, Mokkink LB, Bouter LM, Alonso J, Patrick DL, de Vet HCW, et al. COSMIN guideline for systematic reviews of patient-reported outcome measures. Qual Life Res. 2018;27(5):1147–57.

Munn Z, Moola S, Lisy K, Riitano D, Tufanaru C. Chapter 5: Systematic reviews of prevalence and incidence. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis [internet]. JBI; 2020 [cited 2022 Mar 30]. Available from: https://synthesismanual.jbi.global/ .

Centre for Evidence-Based Medicine. Study designs. CEBM; 2016. Available from: https://www.cebm.ox.ac.uk/resources/ebm-tools/study-designs . Cited 2022 Aug 30

Hartling L, Bond K, Santaguida PL, Viswanathan M, Dryden DM. Testing a tool for the classification of study designs in systematic reviews of interventions and exposures showed moderate reliability and low accuracy. J Clin Epidemiol. 2011;64(8):861–71.

Crowe M, Sheppard L, Campbell A. Reliability analysis for a proposed critical appraisal tool demonstrated value for diverse research designs. J Clin Epidemiol. 2012;65(4):375–83.

Reeves BC, Wells GA, Waddington H. Quasi-experimental study designs series—paper 5: a checklist for classifying studies evaluating the effects on health interventions—a taxonomy without labels. J Clin Epidemiol. 2017;89:30–42.

Reeves BC, Deeks JJ, Higgins JPT, Shea B, Tugwell P, Wells GA. Chapter 24: including non-randomized studies on intervention effects.  In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook/current/chapter-24 . Cited 2022 Mar 1

Reeves B. A framework for classifying study designs to evaluate health care interventions. Forsch Komplementarmed Kl Naturheilkd. 2004;11(Suppl 1):13–7.

Google Scholar  

Rockers PC, Røttingen J, Shemilt I. Inclusion of quasi-experimental studies in systematic reviews of health systems research. Health Policy. 2015;119(4):511–21.

Mathes T, Pieper D. Clarifying the distinction between case series and cohort studies in systematic reviews of comparative studies: potential impact on body of evidence and workload. BMC Med Res Methodol. 2017;17(1):8–13.

Jhangiani R, Cuttler C, Leighton D. Single subject research. In: Jhangiani R, Cuttler C, Leighton D, editors. Research methods in psychology, 4th edn. Pressbooks KPU; 2019. Available from: https://kpu.pressbooks.pub/psychmethods4e/part/single-subject-research/ . Cited 2022 Aug 15

Higgins JP, Ramsay C, Reeves BC, Deeks JJ, Shea B, Valentine JC, et al. Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions. Res Synth Methods. 2013;4(1):12–25.

Cumpston M, Lasserson T, Chandler J, Page M. 3.4.1 Criteria for considering studies for this review, Chapter III: Reporting the review. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook/current/chapter-iii#section-iii-3-4-1 . Cited 2022 Oct 12

Kooistra B, Dijkman B, Einhorn TA, Bhandari M. How to design a good case series. J Bone Jt Surg. 2009;91(Suppl 3):21–6.

Murad MH, Sultan S, Haffar S, Bazerbachi F. Methodological quality and synthesis of case series and case reports. Evid Based Med. 2018;23(2):60–3.

Robinson K, Chou R, Berkman N, Newberry S, FU R, Hartling L, et al. Methods guide for comparative effectiveness reviews integrating bodies of evidence: existing systematic reviews and primary studies. AHRQ; 2015. Available from: https://archive.org/details/integrating-evidence-report-150226 . Cited 2022 Aug 7

Tugwell P, Welch VA, Karunananthan S, Maxwell LJ, Akl EA, Avey MT, et al. When to replicate systematic reviews of interventions: consensus checklist. BMJ. 2020;370: m2864.

Tsertsvadze A, Maglione M, Chou R, Garritty C, Coleman C, Lux L, et al. Updating comparative effectiveness reviews:current efforts in AHRQ’s effective health care program. J Clin Epidemiol. 2011;64(11):1208–15.

Cumpston M, Chandler J. Chapter IV: Updating a review. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook . Cited 2022 Aug 2

Pollock M, Fernandes RM, Newton AS, Scott SD, Hartling L. A decision tool to help researchers make decisions about including systematic reviews in overviews of reviews of healthcare interventions. Syst Rev. 2019;8(1):1–8.

Pussegoda K, Turner L, Garritty C, Mayhew A, Skidmore B, Stevens A, et al. Identifying approaches for assessing methodological and reporting quality of systematic reviews: a descriptive study. Syst Rev. 2017;6(1):1–12.

Bhaumik S. Use of evidence for clinical practice guideline development. Trop Parasitol. 2017;7(2):65–71.

Moher D, Eastwood S, Olkin I, Drummond R, Stroup D. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Lancet. 1999;354:1896–900.

Stroup D, Berlin J, Morton S, Olkin I, Williamson G, Rennie D, et al. Meta-analysis of observational studies in epidemiology A proposal for reporting. JAMA. 2000;238(15):2008–12.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol. 2009;62(10):1006–12.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372: n71.

Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44(11):1271–8.

Centre for Evidence-Based Medicine. Critical appraisal tools. CEBM; 2015. Available from: https://www.cebm.ox.ac.uk/resources/ebm-tools/critical-appraisal-tools . Cited 2022 Apr 10

Page MJ, McKenzie JE, Higgins JPT. Tools for assessing risk of reporting biases in studies and syntheses of studies: a systematic review. BMJ Open. 2018;8(3):1–16.

Article   CAS   Google Scholar  

Ma LL, Wang YY, Yang ZH, Huang D, Weng H, Zeng XT. Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better? Mil Med Res. 2020;7(1):1–11.

Banzi R, Cinquini M, Gonzalez-Lorenzo M, Pecoraro V, Capobussi M, Minozzi S. Quality assessment versus risk of bias in systematic reviews: AMSTAR and ROBIS had similar reliability but differed in their construct and applicability. J Clin Epidemiol. 2018;99:24–32.

Swierz MJ, Storman D, Zajac J, Koperny M, Weglarz P, Staskiewicz W, et al. Similarities, reliability and gaps in assessing the quality of conduct of systematic reviews using AMSTAR-2 and ROBIS: systematic survey of nutrition reviews. BMC Med Res Methodol. 2021;21(1):1–10.

Pieper D, Puljak L, González-Lorenzo M, Minozzi S. Minor differences were found between AMSTAR 2 and ROBIS in the assessment of systematic reviews including both randomized and nonrandomized studies. J Clin Epidemiol. 2019;108:26–33.

Lorenz RC, Matthias K, Pieper D, Wegewitz U, Morche J, Nocon M, et al. A psychometric study found AMSTAR 2 to be a valid and moderately reliable appraisal tool. J Clin Epidemiol. 2019;114:133–40.

Leclercq V, Hiligsmann M, Parisi G, Beaudart C, Tirelli E, Bruyère O. Best-worst scaling identified adequate statistical methods and literature search as the most important items of AMSTAR2 (A measurement tool to assess systematic reviews). J Clin Epidemiol. 2020;128:74–82.

Bühn S, Mathes T, Prengel P, Wegewitz U, Ostermann T, Robens S, et al. The risk of bias in systematic reviews tool showed fair reliability and good construct validity. J Clin Epidemiol. 2017;91:121–8.

Gates M, Gates A, Duarte G, Cary M, Becker M, Prediger B, et al. Quality and risk of bias appraisals of systematic reviews are inconsistent across reviewers and centers. J Clin Epidemiol. 2020;125:9–15.

Perry R, Whitmarsh A, Leach V, Davies P. A comparison of two assessment tools used in overviews of systematic reviews: ROBIS versus AMSTAR-2. Syst Rev. 2021;10(1):273.

Gates M, Gates A, Guitard S, Pollock M, Hartling L. Guidance for overviews of reviews continues to accumulate, but important challenges remain: a scoping review. Syst Rev. 2020;9(1):1–19.

Aromataris E, Fernandez R, Godfrey C, Holly C, Khalil H, Tungpunkom P. Chapter 10: umbrella reviews. In: Aromataris E, Munn Z, editors. JBI Manual for Evidence Synthesis. JBI; 2020. Available from: https://synthesismanual.jbi.global . Cited 2022 Jul 11

Pieper D, Lorenz RC, Rombey T, Jacobs A, Rissling O, Freitag S, et al. Authors should clearly report how they derived the overall rating when applying AMSTAR 2—a cross-sectional study. J Clin Epidemiol. 2021;129:97–103.

Franco JVA, Meza N. Authors should also report the support for judgment when applying AMSTAR 2. J Clin Epidemiol. 2021;138:240.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7): e1000100.

Page MJ, Moher D. Evaluations of the uptake and impact of the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement and extensions: a scoping review. Syst Rev. 2017;6(1):263.

Page MJ, Moher D, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ. 2021;372: n160.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. Updating guidance for reporting systematic reviews: development of the PRISMA 2020 statement. J Clin Epidemiol. 2021;134:103–12.

Welch V, Petticrew M, Petkovic J, Moher D, Waters E, White H, et al. Extending the PRISMA statement to equity-focused systematic reviews (PRISMA-E 2012): explanation and elaboration. J Clin Epidemiol. 2016;70:68–89.

Beller EM, Glasziou PP, Altman DG, Hopewell S, Bastian H, Chalmers I, et al. PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS Med. 2013;10(4): e1001419.

Moher D, Shamseer L, Clarke M. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1.

Hutton B, Salanti G, Caldwell DM, Chaimani A, Schmid CH, Cameron C, et al. The PRISMA extension statement for reporting of systematic reviews incorporating network meta-analyses of health care interventions: checklist and explanations. Ann Intern Med. 2015;162(11):777–84.

Stewart LA, Clarke M, Rovers M, Riley RD, Simmonds M, Stewart G, et al. Preferred reporting items for a systematic review and meta-analysis of individual participant data: The PRISMA-IPD statement. JAMA. 2015;313(16):1657–65.

Zorzela L, Loke YK, Ioannidis JP, Golder S, Santaguida P, Altman DG, et al. PRISMA harms checklist: Improving harms reporting in systematic reviews. BMJ. 2016;352: i157.

McInnes MDF, Moher D, Thombs BD, McGrath TA, Bossuyt PM, Clifford T, et al. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy studies The PRISMA-DTA statement. JAMA. 2018;319(4):388–96.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Wang X, Chen Y, Liu Y, Yao L, Estill J, Bian Z, et al. Reporting items for systematic reviews and meta-analyses of acupuncture: the PRISMA for acupuncture checklist. BMC Complement Altern Med. 2019;19(1):1–10.

Rethlefsen ML, Kirtley S, Waffenschmidt S, Ayala AP, Moher D, Page MJ, et al. PRISMA-S: An extension to the PRISMA statement for reporting literature searches in systematic reviews. J Med Libr Assoc. 2021;109(2):174–200.

Blanco D, Altman D, Moher D, Boutron I, Kirkham JJ, Cobo E. Scoping review on interventions to improve adherence to reporting guidelines in health research. BMJ Open. 2019;9(5): e26589.

Koster TM, Wetterslev J, Gluud C, Keus F, van der Horst ICC. Systematic overview and critical appraisal of meta-analyses of interventions in intensive care medicine. Acta Anaesthesiol Scand. 2018;62(8):1041–9.

Johnson BT, Hennessy EA. Systematic reviews and meta-analyses in the health sciences: best practice methods for research syntheses. Soc Sci Med. 2019;233:237–51.

Pollock A, Berge E. How to do a systematic review. Int J Stroke. 2018;13(2):138–56.

Gagnier JJ, Kellam PJ. Reporting and methodological quality of systematic reviews in the orthopaedic literature. J Bone Jt Surg. 2013;95(11):1–7.

Martinez-Monedero R, Danielian A, Angajala V, Dinalo JE, Kezirian EJ. Methodological quality of systematic reviews and meta-analyses published in high-impact otolaryngology journals. Otolaryngol Head Neck Surg. 2020;163(5):892–905.

Boutron I, Crequit P, Williams H, Meerpohl J, Craig J, Ravaud P. Future of evidence ecosystem series 1. Introduction-evidence synthesis ecosystem needs dramatic change. J Clin Epidemiol. 2020;123:135–42.

Ioannidis JPA, Bhattacharya S, Evers JLH, Der Veen F, Van SE, Barratt CLR, et al. Protect us from poor-quality medical research. Hum Reprod. 2018;33(5):770–6.

Lasserson T, Thomas J, Higgins J. Section 1.5 Protocol development, Chapter 1: Starting a review. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook/archive/v6/chapter-01#section-1-5 . Cited 2022 Mar 20

Stewart L, Moher D, Shekelle P. Why prospective registration of systematic reviews makes sense. Syst Rev. 2012;1(1):7–10.

Allers K, Hoffmann F, Mathes T, Pieper D. Systematic reviews with published protocols compared to those without: more effort, older search. J Clin Epidemiol. 2018;95:102–10.

Ge L, Tian J, Li Y, Pan J, Li G, Wei D, et al. Association between prospective registration and overall reporting and methodological quality of systematic reviews: a meta-epidemiological study. J Clin Epidemiol. 2018;93:45–55.

Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, et al. Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015: elaboration and explanation. BMJ. 2015;350: g7647.

Pieper D, Rombey T. Where to prospectively register a systematic review. Syst Rev. 2022;11(1):8.

PROSPERO. PROSPERO will require earlier registration. NIHR; 2022. Available from: https://www.crd.york.ac.uk/prospero/ . Cited 2022 Mar 20

Kirkham JJ, Altman DG, Williamson PR. Bias due to changes in specified outcomes during the systematic review process. PLoS ONE. 2010;5(3):3–7.

Victora CG, Habicht JP, Bryce J. Evidence-based public health: moving beyond randomized trials. Am J Public Health. 2004;94(3):400–5.

Peinemann F, Kleijnen J. Development of an algorithm to provide awareness in choosing study designs for inclusion in systematic reviews of healthcare interventions: a method study. BMJ Open. 2015;5(8): e007540.

Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350: h2147.

Junqueira DR, Phillips R, Zorzela L, Golder S, Loke Y, Moher D, et al. Time to improve the reporting of harms in randomized controlled trials. J Clin Epidemiol. 2021;136:216–20.

Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA. Routinely collected data and comparative effectiveness evidence: promises and limitations. CMAJ. 2016;188(8):E158–64.

Murad MH. Clinical practice guidelines: a primer on development and dissemination. Mayo Clin Proc. 2017;92(3):423–33.

Abdelhamid AS, Loke YK, Parekh-Bhurke S, Chen Y-F, Sutton A, Eastwood A, et al. Use of indirect comparison methods in systematic reviews: a survey of cochrane review authors. Res Synth Methods. 2012;3(2):71–9.

Jüni P, Holenstein F, Sterne J, Bartlett C, Egger M. Direction and impact of language bias in meta-analyses of controlled trials: empirical study. Int J Epidemiol. 2002;31(1):115–23.

Vickers A, Goyal N, Harland R, Rees R. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998;19(2):159–66.

Jones CW, Keil LG, Weaver MA, Platts-Mills TF. Clinical trials registries are under-utilized in the conduct of systematic reviews: a cross-sectional analysis. Syst Rev. 2014;3(1):1–7.

Baudard M, Yavchitz A, Ravaud P, Perrodeau E, Boutron I. Impact of searching clinical trial registries in systematic reviews of pharmaceutical treatments: methodological systematic review and reanalysis of meta-analyses. BMJ. 2017;356: j448.

Fanelli D, Costas R, Ioannidis JPA. Meta-assessment of bias in science. Proc Natl Acad Sci USA. 2017;114(14):3714–9.

Hartling L, Featherstone R, Nuspl M, Shave K, Dryden DM, Vandermeer B. Grey literature in systematic reviews: a cross-sectional study of the contribution of non-English reports, unpublished studies and dissertations to the results of meta-analyses in child-relevant reviews. BMC Med Res Methodol. 2017;17(1):64.

Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database Syst Rev. 2007;2:MR000010.

Shojania K, Sampson M, Ansari MT, Ji J, Garritty C, Radar T, et al. Updating systematic reviews. AHRQ Technical Reviews. 2007: Report 07–0087.

Tate RL, Perdices M, Rosenkoetter U, Wakim D, Godbee K, Togher L, et al. Revision of a method quality rating scale for single-case experimental designs and n-of-1 trials: The 15-item Risk of Bias in N-of-1 Trials (RoBiNT) Scale. Neuropsychol Rehabil. 2013;23(5):619–38.

Tate RL, Perdices M, McDonald S, Togher L, Rosenkoetter U. The design, conduct and report of single-case research: Resources to improve the quality of the neurorehabilitation literature. Neuropsychol Rehabil. 2014;24(3–4):315–31.

Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366: l4894.

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355: i4919.

Igelström E, Campbell M, Craig P, Katikireddi SV. Cochrane’s risk of bias tool for non-randomized studies (ROBINS-I) is frequently misapplied: a methodological systematic review. J Clin Epidemiol. 2021;140:22–32.

McKenzie JE, Brennan SE. Chapter 12: Synthesizing and presenting findings using other methods. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook/current/chapter-12 . Cited 2022 Apr 10

Ioannidis J, Patsopoulos N, Rothstein H. Reasons or excuses for avoiding meta-analysis in forest plots. BMJ. 2008;336(7658):1413–5.

Stewart LA, Tierney JF. To IPD or not to IPD? Eval Health Prof. 2002;25(1):76–97.

Tierney JF, Stewart LA, Clarke M. Chapter 26: Individual participant data. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook/current/chapter-26 . Cited 2022 Oct 12

Chaimani A, Caldwell D, Li T, Higgins J, Salanti G. Chapter 11: Undertaking network meta-analyses. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook . Cited 2022 Oct 12.

Cooper H, Hedges L, Valentine J. The handbook of research synthesis and meta-analysis. 3rd ed. Russell Sage Foundation; 2019.

Sutton AJ, Abrams KR, Jones DR, Sheldon T, Song F. Methods for meta-analysis in medical research. Methods for meta-analysis in medical research; 2000.

Deeks J, Higgins JPT, Altman DG. Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic review of interventions. Cochrane; 2022. Available from: http://www.training.cochrane.org/handbook . Cited 2022 Mar 20.

Clarke MJ. Individual patient data meta-analyses. Best Pract Res Clin Obstet Gynaecol. 2005;19(1):47–55.

Catalá-López F, Tobías A, Cameron C, Moher D, Hutton B. Network meta-analysis for comparing treatment effects of multiple interventions: an introduction. Rheumatol Int. 2014;34(11):1489–96.

Debray T, Schuit E, Efthimiou O, Reitsma J, Ioannidis J, Salanti G, et al. An overview of methods for network meta-analysis using individual participant data: when do benefits arise? Stat Methods Med Res. 2016;27(5):1351–64.

Tonin FS, Rotta I, Mendes AM, Pontarolo R. Network meta-analysis : a technique to gather evidence from direct and indirect comparisons. Pharm Pract (Granada). 2017;15(1):943.

Tierney JF, Vale C, Riley R, Smith CT, Stewart L, Clarke M, et al. Individual participant data (IPD) metaanalyses of randomised controlled trials: guidance on their use. PLoS Med. 2015;12(7): e1001855.

Rouse B, Chaimani A, Li T. Network meta-analysis: an introduction for clinicians. Intern Emerg Med. 2017;12(1):103–11.

Cochrane Training. Review Manager RevMan Web. Cochrane; 2022. Available from: https://training.cochrane.org/online-learning/core-software/revman . Cited 2022 Jun 24

MetaXL. MetalXL. Epi Gear; 2016. Available from: http://epigear.com/index_files/metaxl.html . Cited 2022 Jun 24.

JBI. JBI SUMARI. JBI; 2019. Available from: https://sumari.jbi.global/ . Cited 2022 Jun 24.

Ryan R. Cochrane Consumers and Communication Review Group: data synthesis and analysis. Cochrane Consumers and Communication Review Group; 2013. Available from: http://cccrg.cochrane.org . Cited 2022 Jun 24

McKenzie JE, Beller EM, Forbes AB. Introduction to systematic reviews and meta-analysis. Respirology. 2016;21(4):626–37.

Campbell M, Katikireddi SV, Sowden A, Thomson H. Lack of transparency in reporting narrative synthesis of quantitative data: a methodological assessment of systematic reviews. J Clin Epidemiol. 2019;105:1–9.

Campbell M, McKenzie JE, Sowden A, Katikireddi SV, Brennan SE, Ellis S, et al. Synthesis without meta-analysis (SWiM) in systematic reviews: reporting guideline. BMJ. 2020;368: l6890.

McKenzie JE, Brennan S, Ryan R. Summarizing study characteristics and preparing for synthesis. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook . Cited 2022 Oct 12

AHRQ. Systems to rate the strength of scientific evidence. Evidence report/technology assessment no. 47. AHRQ; 2002. Available from: https://archive.ahrq.gov/clinic/epcsums/strengthsum.htm . Cited 2022 Apr 10.

Atkins D, Eccles M, Flottorp S, Guyatt GH, Henry D, Hill S, et al. Systems for grading the quality of evidence and the strength of recommendations I: critical appraisal of existing approaches. BMC Health Serv Res. 2004;4(1):38.

Ioannidis JPA. Meta-research: the art of getting it wrong.  Res Synth Methods. 2010;1(3–4):169–84.

Lai NM, Teng CL, Lee ML. Interpreting systematic reviews:  are we ready to make our own conclusions? A cross sectional study. BMC Med. 2011;9(1):30.

Glenton C, Santesso N, Rosenbaum S, Nilsen ES, Rader T, Ciapponi A, et al. Presenting the results of Cochrane systematic reviews to a consumer audience: a qualitative study. Med Decis Making. 2010;30(5):566–77.

Yavchitz A, Ravaud P, Altman DG, Moher D, HrobjartssonA, Lasserson T, et al. A new classification of spin in systematic reviews and meta-analyses was developed and ranked according to the severity. J Clin Epidemiol. 2016;75:56–65.

Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, et al. GRADE Working Group. Grading quality of evidence and strength of recommendations. BMJ. 2004;328:7454.

GRADE Working Group. Organizations. GRADE; 2022 [cited 2023 May 2].  Available from: www.gradeworkinggroup.org .

Hartling L, Fernandes RM, Seida J, Vandermeer B, Dryden DM. From the trenches: a cross-sectional study applying the grade tool in systematic reviews of healthcare interventions.  PLoS One. 2012;7(4):e34697.

Hultcrantz M, Rind D, Akl EA, Treweek S, Mustafa RA, Iorio A, et al. The GRADE working group clarifies the construct of certainty of evidence. J Clin Epidemiol. 2017;87:4–13.

Schünemann H, Brozek J, Guyatt G, Oxman AD, Editors. Section 6.3.2. Symbolic representation. GRADE Handbook [internet].  GRADE; 2013 [cited 2022 Jan 27]. Available from: https://gdt.gradepro.org/app/handbook/handbook.html#h.lr8e9vq954 .

Siemieniuk R, Guyatt G What is GRADE? [internet] BMJ Best Practice; 2017 [cited 2022 Jul 20]. Available from: https://bestpractice.bmj.com/info/toolkit/learn-ebm/what-is-grade/ .

Guyatt G, Oxman AD, Sultan S, Brozek J, Glasziou P, Alonso-Coello P, et al. GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J Clin Epidemiol. 2013;66(2):151–7.

Guyatt GH, Oxman AD, Sultan S, Glasziou P, Akl EA, Alonso-Coello P, et al. GRADE guidelines: 9. Rating up the quality of evidence. J Clin Epidemiol. 2011;64(12):1311–6.

Guyatt GH, Oxman AD, Vist G, Kunz R, Brozek J, Alonso-Coello P, et al. GRADE guidelines: 4. Rating the quality of evidence - Study limitations (risk of bias). J Clin Epidemiol. 2011;64(4):407–15.

Guyatt GH, Oxman AD, Kunz R, Brozek J, Alonso-Coello P, Rind D, et al. GRADE guidelines 6. Rating the quality of evidence - Imprecision. J Clin Epidemiol. 2011;64(12):1283–93.

Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 7. Rating the quality of evidence - Inconsistency. J Clin Epidemiol. 2011;64(12):1294–302.

Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, et al. GRADE guidelines: 8. Rating the quality of evidence - Indirectness. J Clin Epidemiol. 2011;64(12):1303–10.

Guyatt GH, Oxman AD, Montori V, Vist G, Kunz R, Brozek J, et al. GRADE guidelines: 5. Rating the quality of evidence - Publication bias. J Clin Epidemiol. 2011;64(12):1277–82.

Andrews JC, Schünemann HJ, Oxman AD, Pottie K, Meerpohl JJ, Coello PA, et al. GRADE guidelines: 15. Going from evidence to recommendation - Determinants of a recommendation’s direction and strength. J Clin Epidemiol. 2013;66(7):726–35.

Fleming PS, Koletsi D, Ioannidis JPA, Pandis N. High quality of the evidence for medical and other health-related interventions was uncommon in Cochrane systematic reviews. J Clin Epidemiol. 2016;78:34–42.

Howick J, Koletsi D, Pandis N, Fleming PS, Loef M, Walach H, et al. The quality of evidence for medical interventions does not improve or worsen: a metaepidemiological study of Cochrane reviews. J Clin Epidemiol. 2020;126:154–9.

Mustafa RA, Santesso N, Brozek J, Akl EA, Walter SD, Norman G, et al. The GRADE approach is reproducible in assessing the quality of evidence of quantitative evidence syntheses. J Clin Epidemiol. 2013;66(7):736-742.e5.

Schünemann H, Brozek J, Guyatt G, Oxman A, editors. Section 5.4: Overall quality of evidence. GRADE Handbook. GRADE; 2013. Available from: https://gdt.gradepro.org/app/handbook/handbook.html#h.lr8e9vq954a . Cited 2022 Mar 25.

GRADE Working Group. Criteria for using GRADE. GRADE; 2016. Available from: https://www.gradeworkinggroup.org/docs/Criteria_for_using_GRADE_2016-04-05.pdf . Cited 2022 Jan 26

Werner SS, Binder N, Toews I, Schünemann HJ, Meerpohl JJ, Schwingshackl L. Use of GRADE in evidence syntheses published in high-impact-factor nutrition journals: a methodological survey. J Clin Epidemiol. 2021;135:54–69.

Zhang S, Wu QJ, Liu SX. A methodologic survey on use of the GRADE approach in evidence syntheses published in high-impact factor urology and nephrology journals. BMC Med Res Methodol. 2022;22(1):220.

Li L, Tian J, Tian H, Sun R, Liu Y, Yang K. Quality and transparency of overviews of systematic reviews. J Evid Based Med. 2012;5(3):166–73.

Pieper D, Buechter R, Jerinic P, Eikermann M. Overviews of reviews often have limited rigor: a systematic review. J Clin Epidemiol. 2012;65(12):1267–73.

Cochrane Editorial Unit. Appendix 1: Checklist for auditing GRADE and SoF tables in protocols of intervention reviews. Cochrane Training; 2022. Available from: https://training.cochrane.org/gomo/modules/522/resources/8307/Checklist for GRADE and SoF methods in Protocols for Gomo.pdf. Cited 2022 Mar 12

Ryan R, Hill S. How to GRADE the quality of the evidence. Cochrane Consumers and Communication Group. Cochrane; 2016. Available from: https://cccrg.cochrane.org/author-resources .

Cunningham M, France EF, Ring N, Uny I, Duncan EA, Roberts RJ, et al. Developing a reporting guideline to improve meta-ethnography in health research: the eMERGe mixed-methods study. Heal Serv Deliv Res. 2019;7(4):1–116.

Tong A, Flemming K, McInnes E, Oliver S, Craig J. Enhancing transparency in reporting the synthesis of qualitative research: ENTREQ. BMC Med Res Methodol. 2012;12:181.

Gates M, Gates G, Pieper D, Fernandes R, Tricco A, Moher D, et al. Reporting guideline for overviews of reviews of healthcare interventions: development of the PRIOR statement. BMJ. 2022;378:e070849.

Whiting PF, Reitsma JB, Leeflang MMG, Sterne JAC, Bossuyt PMM, Rutjes AWSS, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(4):529–36.

Hayden JA, van der Windt DA, Cartwright JL, Co P. Research and reporting methods assessing bias in studies of prognostic factors. Ann Intern Med. 2013;158(4):280–6.

Critical Appraisal Skills Programme. CASP qualitative checklist. CASP; 2018. Available from: https://casp-uk.net/images/checklist/documents/CASP-Qualitative-Studies-Checklist/CASP-Qualitative-Checklist-2018_fillable_form.pdf . Cited 2022 Apr 26

Hannes K, Lockwood C, Pearson A. A comparative analysis of three online appraisal instruments’ ability to assess validity in qualitative research. Qual Health Res. 2010;20(12):1736–43.

Munn Z, Moola S, Riitano D, Lisy K. The development of a critical appraisal tool for use in systematic reviews addressing questions of prevalence. Int J Heal Policy Manag. 2014;3(3):123–8.

Lewin S, Bohren M, Rashidian A, Munthe-Kaas H, Glenton C, Colvin CJ, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table. Implement Sci. 2018;13(suppl 1):10.

Munn Z, Porritt K, Lockwood C, Aromataris E, Pearson A.  Establishing confidence in the output of qualitative research synthesis: the ConQual approach. BMC Med Res Methodol. 2014;14(1):108.

Flemming K, Booth A, Hannes K, Cargo M, Noyes J. Cochrane Qualitative and Implementation Methods Group guidance series—paper 6: reporting guidelines for qualitative, implementation, and process evaluation evidence syntheses. J Clin Epidemiol. 2018;97:79–85.

Lockwood C, Munn Z, Porritt K. Qualitative research synthesis:  methodological guidance for systematic reviewers utilizing meta-aggregation. Int J Evid Based Health. 2015;13(3):179–87.

Schünemann HJ, Mustafa RA, Brozek J, Steingart KR, Leeflang M, Murad MH, et al. GRADE guidelines: 21 part 1.  Study design, risk of bias, and indirectness in rating the certainty across a body of evidence for test accuracy. J Clin Epidemiol. 2020;122:129–41.

Schünemann HJ, Mustafa RA, Brozek J, Steingart KR, Leeflang M, Murad MH, et al. GRADE guidelines: 21 part 2. Test accuracy: inconsistency, imprecision, publication bias, and other domains for rating the certainty of evidence and presenting it in evidence profiles and summary of findings tables. J Clin Epidemiol. 2020;122:142–52.

Foroutan F, Guyatt G, Zuk V, Vandvik PO, Alba AC, Mustafa R, et al. GRADE Guidelines 28: use of GRADE for the assessment of evidence about prognostic factors:  rating certainty in identification of groups of patients with different absolute risks. J Clin Epidemiol. 2020;121:62–70.

Janiaud P, Agarwal A, Belbasis L, Tzoulaki I. An umbrella review of umbrella reviews for non-randomized observational evidence on putative risk and protective factors [internet]. OSF protocol; 2021 [cited 2022 May 28]. Available from: https://osf.io/xj5cf/ .

Mokkink LB, Prinsen CA, Patrick DL, Alonso J, Bouter LM, et al. COSMIN methodology for systematic reviews of Patient-Reported Outcome Measures (PROMs) - user manual. COSMIN; 2018 [cited 2022 Feb 15]. Available from:  http://www.cosmin.nl/ .

Thomas J, M P, Noyes J, Chandler J, Rehfuess E, Tugwell P, et al. Chapter 17: Intervention complexity. In: Higgins J, Thomas J, Chandler J, Cumpston M, Li T, Page M, et al., editors. Cochrane handbook for systematic reviews of interventions. Cochrane; 2022. Available from: https://training.cochrane.org/handbook/current/chapter-17 . Cited 2022 Oct 12

Guise JM, Chang C, Butler M, Viswanathan M, Tugwell P. AHRQ series on complex intervention systematic reviews—paper 1: an introduction to a series of articles that provide guidance and tools for reviews of complex interventions. J Clin Epidemiol. 2017;90:6–10.

Riaz IB, He H, Ryu AJ, Siddiqi R, Naqvi SAA, Yao Y, et al. A living, interactive systematic review and network meta-analysis of first-line treatment of metastatic renal cell carcinoma [formula presented]. Eur Urol. 2021;80(6):712–23.

Créquit P, Trinquart L, Ravaud P. Live cumulative network meta-analysis: protocol for second-line treatments in advanced non-small-cell lung cancer with wild-type or unknown status for epidermal growth factor receptor. BMJ Open. 2016;6(8):e011841.

Ravaud P, Créquit P, Williams HC, Meerpohl J, Craig JC, Boutron I. Future of evidence ecosystem series: 3. From an evidence synthesis ecosystem to an evidence ecosystem. J Clin Epidemiol. 2020;123:153–61.

Download references

Acknowledgements

Michelle Oakman Hayes for her assistance with the graphics, Mike Clarke for his willingness to answer our seemingly arbitrary questions, and Bernard Dan for his encouragement of this project.

The work of John Ioannidis has been supported by an unrestricted gift from Sue and Bob O’Donnell to Stanford University.

Author information

Authors and affiliations.

Departments of Orthopaedic Surgery, Pediatrics, and Neurology, Wake Forest School of Medicine, Winston-Salem, NC, USA

Kat Kolaski

Department of Physical Medicine and Rehabilitation, SUNY Upstate Medical University, Syracuse, NY, USA

Lynne Romeiser Logan

Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, and Meta-Research Innovation Center at Stanford (METRICS), Stanford University School of Medicine, Stanford, CA, USA

John P. A. Ioannidis

You can also search for this author in PubMed   Google Scholar

Contributions

All authors participated in the development of the ideas, writing, and review of this manuscript. The author(s) read and approved the final manuscript.

Corresponding author

Correspondence to Kat Kolaski .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’ s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article has been published simultaneously in BMC Systematic Reviews, Acta Anaesthesiologica Scandinavica, BMC Infectious Diseases, British Journal of Pharmacology, JBI Evidence Synthesis, the Journal of Bone and Joint Surgery Reviews , and the Journal of Pediatric Rehabilitation Medicine .

Supplementary Information

Additional file 2a..

Overviews, scoping reviews, rapid reviews and living reviews.

Additional file 2B.

Practical scheme for distinguishing types of research evidence.

Additional file 4.

Presentation of forest plots.

Additional file 5A.

Illustrations of the GRADE approach.

Additional file 5B.

 Adaptations of GRADE for evidence syntheses.

Additional file 6.

 Links to Concise Guide online resources.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kolaski, K., Logan, L.R. & Ioannidis, J.P.A. Guidance to best tools and practices for systematic reviews. Syst Rev 12 , 96 (2023). https://doi.org/10.1186/s13643-023-02255-9

Download citation

Received : 03 October 2022

Accepted : 19 February 2023

Published : 08 June 2023

DOI : https://doi.org/10.1186/s13643-023-02255-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Certainty of evidence
• Critical appraisal
• Methodological quality
• Risk of bias
• Systematic review

Systematic Reviews

ISSN: 2046-4053

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Writing in the Health and Social Sciences

• Journal Publishing
• Style and Writing Guides
• Readings about Writing
• Resources for Dissertation Authors
• Citation Management and Formatting Tools

Systematic Literature Reviews: Steps & Resources

Literature review & systematic review steps.

• What are Literature Reviews?
• Conducting & Reporting Systematic Reviews
• Finding Systematic Reviews
• Tutorials & Tools for Literature Reviews

What are Systematic Reviews? (3 minutes, 24 second YouTube Video)

These steps for conducting a systematic literature review are listed below . 

Also see subpages for more information about:

• The different types of literature reviews, including systematic reviews and other evidence synthesis methods
• Tools & Tutorials
• Develop a Focused Question
• Scope the Literature  (Initial Search)
• Refine & Expand the Search
• Limit the Results
• Download Citations
• Abstract & Analyze
• Create Flow Diagram
• Synthesize & Report Results

1. Develop a Focused   Question 

Consider the PICO Format: Population/Problem, Intervention, Comparison, Outcome

Focus on defining the Population or Problem and Intervention (don't narrow by Comparison or Outcome just yet!)

"What are the effects of the Pilates method for patients with low back pain?"

Tools & Additional Resources:

• PICO Question Help
• Stillwell, Susan B., DNP, RN, CNE; Fineout-Overholt, Ellen, PhD, RN, FNAP, FAAN; Melnyk, Bernadette Mazurek, PhD, RN, CPNP/PMHNP, FNAP, FAAN; Williamson, Kathleen M., PhD, RN Evidence-Based Practice, Step by Step: Asking the Clinical Question, AJN The American Journal of Nursing : March 2010 - Volume 110 - Issue 3 - p 58-61 doi: 10.1097/01.NAJ.0000368959.11129.79

2. Scope the Literature

A "scoping search" investigates the breadth and/or depth of the initial question or may identify a gap in the literature. 

Eligible studies may be located by searching in:

• Background sources (books, point-of-care tools)
• Article databases
• Trial registries
• Grey literature
• Cited references
• Reference lists

When searching, if possible, translate terms to controlled vocabulary of the database. Use text word searching when necessary.

Use Boolean operators to connect search terms:

• Combine separate concepts with AND  (resulting in a narrower search)
• Connecting synonyms with OR  (resulting in an expanded search)

Search:  pilates AND ("low back pain"  OR  backache )

Video Tutorials - Translating PICO Questions into Search Queries

• Translate Your PICO Into a Search in PubMed (YouTube, Carrie Price, 5:11) 
• Translate Your PICO Into a Search in CINAHL (YouTube, Carrie Price, 4:56)

3. Refine & Expand Your Search

Expand your search strategy with synonymous search terms harvested from:

• database thesauri
• reference lists
• relevant studies

Example: 

(pilates OR exercise movement techniques) AND ("low back pain" OR backache* OR sciatica OR lumbago OR spondylosis)

As you develop a final, reproducible strategy for each database, save your strategies in a:

• a personal database account (e.g., MyNCBI for PubMed)
• Log in with your NYU credentials
• Open and "Make a Copy" to create your own tracker for your literature search strategies

4. Limit Your Results

Use database filters to limit your results based on your defined inclusion/exclusion criteria.  In addition to relying on the databases' categorical filters, you may also need to manually screen results.  

• Limit to Article type, e.g.,:  "randomized controlled trial" OR multicenter study
• Limit by publication years, age groups, language, etc.

NOTE: Many databases allow you to filter to "Full Text Only".  This filter is  not recommended . It excludes articles if their full text is not available in that particular database (CINAHL, PubMed, etc), but if the article is relevant, it is important that you are able to read its title and abstract, regardless of 'full text' status. The full text is likely to be accessible through another source (a different database, or Interlibrary Loan).  

• Filters in PubMed
• CINAHL Advanced Searching Tutorial

5. Download Citations

Selected citations and/or entire sets of search results can be downloaded from the database into a citation management tool. If you are conducting a systematic review that will require reporting according to PRISMA standards, a citation manager can help you keep track of the number of articles that came from each database, as well as the number of duplicate records.

In Zotero, you can create a Collection for the combined results set, and sub-collections for the results from each database you search.  You can then use Zotero's 'Duplicate Items" function to find and merge duplicate records.

• Citation Managers - General Guide

6. Abstract and Analyze

• Migrate citations to data collection/extraction tool
• Screen Title/Abstracts for inclusion/exclusion
• Screen and appraise full text for relevance, methods, 
• Resolve disagreements by consensus

Covidence is a web-based tool that enables you to work with a team to screen titles/abstracts and full text for inclusion in your review, as well as extract data from the included studies.

• Covidence Support
• Critical Appraisal Tools
• Data Extraction Tools

7. Create Flow Diagram

The PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) flow diagram is a visual representation of the flow of records through different phases of a systematic review.  It depicts the number of records identified, included and excluded.  It is best used in conjunction with the PRISMA checklist .

Example from: Stotz, S. A., McNealy, K., Begay, R. L., DeSanto, K., Manson, S. M., & Moore, K. R. (2021). Multi-level diabetes prevention and treatment interventions for Native people in the USA and Canada: A scoping review. Current Diabetes Reports, 2 (11), 46. https://doi.org/10.1007/s11892-021-01414-3

• PRISMA Flow Diagram Generator (ShinyApp.io, Haddaway et al. )
• PRISMA Diagram Templates  (Word and PDF)
• Make a copy of the file to fill out the template
• Image can be downloaded as PDF, PNG, JPG, or SVG
• Covidence generates a PRISMA diagram that is automatically updated as records move through the review phases

8. Synthesize & Report Results

There are a number of reporting guideline available to guide the synthesis and reporting of results in systematic literature reviews.

It is common to organize findings in a matrix, also known as a Table of Evidence (ToE).

• Reporting Guidelines for Systematic Reviews
• Download a sample template of a health sciences review matrix  (GoogleSheets)

Steps modified from: 

Cook, D. A., & West, C. P. (2012). Conducting systematic reviews in medical education: a stepwise approach.   Medical Education , 46 (10), 943–952.

• << Previous: Citation Management and Formatting Tools
• Next: What are Literature Reviews? >>
• Last Updated: Sep 10, 2024 2:49 PM
• URL: https://guides.nyu.edu/healthwriting

Systematic Reviews and Meta Analysis

• Getting Started
• Guides and Standards
• Review Protocols
• Databases and Sources
• Randomized Controlled Trials
• Controlled Clinical Trials
• Observational Designs
• Tests of Diagnostic Accuracy
• Software and Tools
• Where do I get all those articles?
• Collaborations
• EPI 233/528
• Countway Mediated Search
• Risk of Bias (RoB)

Systematic review Q & A

What is a systematic review.

A systematic review is guided filtering and synthesis of all available evidence addressing a specific, focused research question, generally about a specific intervention or exposure. The use of standardized, systematic methods and pre-selected eligibility criteria reduce the risk of bias in identifying, selecting and analyzing relevant studies. A well-designed systematic review includes clear objectives, pre-selected criteria for identifying eligible studies, an explicit methodology, a thorough and reproducible search of the literature, an assessment of the validity or risk of bias of each included study, and a systematic synthesis, analysis and presentation of the findings of the included studies. A systematic review may include a meta-analysis.

For details about carrying out systematic reviews, see the Guides and Standards section of this guide.

Is my research topic appropriate for systematic review methods?

A systematic review is best deployed to test a specific hypothesis about a healthcare or public health intervention or exposure. By focusing on a single intervention or a few specific interventions for a particular condition, the investigator can ensure a manageable results set. Moreover, examining a single or small set of related interventions, exposures, or outcomes, will simplify the assessment of studies and the synthesis of the findings.

Systematic reviews are poor tools for hypothesis generation: for instance, to determine what interventions have been used to increase the awareness and acceptability of a vaccine or to investigate the ways that predictive analytics have been used in health care management. In the first case, we don't know what interventions to search for and so have to screen all the articles about awareness and acceptability. In the second, there is no agreed on set of methods that make up predictive analytics, and health care management is far too broad. The search will necessarily be incomplete, vague and very large all at the same time. In most cases, reviews without clearly and exactly specified populations, interventions, exposures, and outcomes will produce results sets that quickly outstrip the resources of a small team and offer no consistent way to assess and synthesize findings from the studies that are identified.

If not a systematic review, then what?

You might consider performing a scoping review . This framework allows iterative searching over a reduced number of data sources and no requirement to assess individual studies for risk of bias. The framework includes built-in mechanisms to adjust the analysis as the work progresses and more is learned about the topic. A scoping review won't help you limit the number of records you'll need to screen (broad questions lead to large results sets) but may give you means of dealing with a large set of results.

This tool can help you decide what kind of review is right for your question.

Can my student complete a systematic review during her summer project?

Probably not. Systematic reviews are a lot of work. Including creating the protocol, building and running a quality search, collecting all the papers, evaluating the studies that meet the inclusion criteria and extracting and analyzing the summary data, a well done review can require dozens to hundreds of hours of work that can span several months. Moreover, a systematic review requires subject expertise, statistical support and a librarian to help design and run the search. Be aware that librarians sometimes have queues for their search time. It may take several weeks to complete and run a search. Moreover, all guidelines for carrying out systematic reviews recommend that at least two subject experts screen the studies identified in the search. The first round of screening can consume 1 hour per screener for every 100-200 records. A systematic review is a labor-intensive team effort.

How can I know if my topic has been been reviewed already?

Before starting out on a systematic review, check to see if someone has done it already. In PubMed you can use the systematic review subset to limit to a broad group of papers that is enriched for systematic reviews. You can invoke the subset by selecting if from the Article Types filters to the left of your PubMed results, or you can append AND systematic[sb] to your search. For example:

"neoadjuvant chemotherapy" AND systematic[sb]

The systematic review subset is very noisy, however. To quickly focus on systematic reviews (knowing that you may be missing some), simply search for the word systematic in the title:

"neoadjuvant chemotherapy" AND systematic[ti]

Any PRISMA-compliant systematic review will be captured by this method since including the words "systematic review" in the title is a requirement of the PRISMA checklist. Cochrane systematic reviews do not include 'systematic' in the title, however. It's worth checking the Cochrane Database of Systematic Reviews independently.

You can also search for protocols that will indicate that another group has set out on a similar project. Many investigators will register their protocols in PROSPERO , a registry of review protocols. Other published protocols as well as Cochrane Review protocols appear in the Cochrane Methodology Register, a part of the Cochrane Library .

• Next: Guides and Standards >>
• Last Updated: Sep 4, 2024 4:04 PM
• URL: https://guides.library.harvard.edu/meta-analysis

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Systematic Review | Definition, Example, & Guide

Systematic Review | Definition, Example & Guide

Published on June 15, 2022 by Shaun Turney . Revised on November 20, 2023.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesize all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question “What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?”

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs. meta-analysis, systematic review vs. literature review, systematic review vs. scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, other interesting articles, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce bias . The methods are repeatable, and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesize the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesizing all available evidence and evaluating the quality of the evidence. Synthesizing means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism. Run a free check.

Systematic reviews often quantitatively synthesize the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesize results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarize and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimize bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis ), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimize research bias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinized by others.
• They’re thorough : they summarize all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fifth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomized control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective (s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesize the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Gray literature: Gray literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of gray literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of gray literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Gray literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarize what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgment of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomized into the control and treatment groups.

Step 6: Synthesize the data

Synthesizing the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesizing the data:

• Narrative ( qualitative ): Summarize the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarize and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analyzed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

In their report, Boyle and colleagues concluded that probiotics cannot be recommended for reducing eczema symptoms or improving quality of life in patients with eczema. Note Generative AI tools like ChatGPT can be useful at various stages of the writing and research process and can help you to write your systematic review. However, we strongly advise against trying to pass AI-generated text off as your own work.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a thesis, dissertation , or research paper , in order to situate your work in relation to existing knowledge.

A literature review is a survey of credible sources on a topic, often used in dissertations , theses, and research papers . Literature reviews give an overview of knowledge on a subject, helping you identify relevant theories and methods, as well as gaps in existing research. Literature reviews are set up similarly to other  academic texts , with an introduction , a main body, and a conclusion .

An  annotated bibliography is a list of  source references that has a short description (called an annotation ) for each of the sources. It is often assigned as part of the research process for a  paper .  

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2023, November 20). Systematic Review | Definition, Example & Guide. Scribbr. Retrieved September 10, 2024, from https://www.scribbr.com/methodology/systematic-review/

Is this article helpful?

Shaun Turney

Other students also liked, how to write a literature review | guide, examples, & templates, how to write a research proposal | examples & templates, what is critical thinking | definition & examples, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

Grand Valley State University

• University Libraries

Search the Library for Articles, Books, and More Find It!

More research tools:

• Subject Guides
• Find, Borrow, Request
• Teaching & Scholarship
• Visit, Study, Learn
• Systematic and Scoping Reviews

Systematic and Scoping Reviews: Home

• Types of Reviews
• Review Guidelines
• 0. Plan your Review
• 1. Define the Question
• 2. Check for Recent Reviews and Protocols
• 3. Write and register your protocol
• Developing your Search Terms
• Database Search Tips and Filters
• Grey Literature
• Record and Report your Search Strategy
• 5. Organize & Screen Search Results
• 6. Appraise the Studies
• 7. Extract Data
• 8. Analyze / Synthesize Data
• 9. Write the Review
• Rapid Reviews
• Scoping Reviews
• Librarian Support

What is Evidence Synthesis?

'Evidence synthesis' is a collective term for types of literature research that bring together all relevant information on a well-formulated research question using a consistent, reproducible methodology. Most forms of evidence synthesis have one or more sets of guidelines for conducting a high-quality review. Systematic reviews and scoping reviews are two of the more common types.

Evidence syntheses should be conducted in an unbiased, reproducible way to provide evidence for practice and policy-making, as well as to identify gaps in the research. Some types include a meta-analysis, a more quantitative process of synthesizing and visualizing data retrieved from various studies.

Although systematic reviews are one of the most well-known review types, there are a variety of different types of reviews that vary in terms of scope, comprehensiveness, time constraints, and types of studies included. For more information about different review types, visit the Types of Reviews section.

About this guide

This guide presents practical tools and advice for conducting Systematic and Scoping Reviews and other evidence syntheses and comprehensive literature search projects:

• Review types
• Outline of the SR process
• Formulating an effective search strategy
• Selecting & searching databases
• Managing search results
• Reporting search methods
• How GVSU librarians can help

This guide does NOT replace the understanding of research design and methodology you will gain from reading sources such as the Cochrane Collaboration Handbook or the JBI Manual for Evidence Synthesis . Researchers new to systematic reviews, scoping reviews, and other forms of comprehensive evidence synthesis are strongly encouraged to read a guide appropriate to their review type and research question. If you're not sure which would be the most useful, your librarian can make recommendations.

A free, asynchronous training course on conducting systematic reviews and meta-analyses is available from Johns Hopkins University through Coursera at https://www.coursera.org/learn/systematic-review

Guide Credit

This guide was created with permission from the Evidence Synthesis and Systematic Review guides of the University of Washington , University of Michigan , and Cornell University .

• Next: Types of Reviews >>
• Last Updated: Sep 11, 2024 8:22 AM
• URL: https://libguides.gvsu.edu/SR

Login to LibApps

Email us! [email protected]

Text Us! (616) 818-0219

Mary Idema Pew Library (616) 331-3500

Steelcase Library (616) 331-6799

Frey Foundation Learning Commons (616) 331-5933

Social Media

Committed to Equality

Federal Depository Library Program

• GVSU is an AA/EO Institution
• Privacy Policy
• Disclosures
• Copyright © 2018 GVSU

Top 10 Best AI Tools for Literature Review (Free + Paid)

Ayush Chaturvedi

16 min read

Best AI Tools for Literature Review at a Glance 

Elephas pricing , elephas reviews, research rabbit key features:, research rabbit pricing , research rabbit , semantic scholar key features:, semantic scholar pricing , semantic scholar reviews , r discovery key features , r discovery pricing , r discovery reviews , mendeley key features:, mendeley pricing , mendeley reviews , scholarcy key features:, scholarcy pricing , scholarcy reviews , rayyan key features:, rayaan pricing , rayaan reviews , consensus key features:, consensus pricing , consensus reviews , unpaywall key features:, unpaywall pricing , unpaywall reviews , lateral key features:, lateral pricing , lateral reviews, what is a literature ai tool , benefits of using ai tools for literature review , how did we pick the best ai tools for literature review, conclusion , 1. what is the best ai tool for literature review, 2. are ai tools for literature review suitable for all types of research , 3. are there any limitations to using ai tools for literature review.

AI tools are revolutionizing the literature review process, offering researchers a powerful alternative to manual searches. These tools can rapidly analyze vast amounts of data, identifying relevant studies and key information with precision and efficiency. 

By streamlining the research process, AI-powered literature review tools save time and reduce frustration, allowing researchers to focus on analysis and interpretation.

This article examines the top AI tools for literature review, evaluating both free and paid options. 

We'll explore how these tools can enhance your research workflow and help you conduct more comprehensive literature reviews.

So let's get started.

Elephas: Best for comprehensive AI-powered literature reviews and writing.

Research Rabbit: Best for organizing and discovering academic papers.

Semantic Scholar: Best for personalised, context-aware academic searches.

R Discovery: Best for personalized research feeds and multilingual access.

Mendeley: Best for reference management and collaborative research.

Scholarcy: Best for generating concise academic summaries.

Rayyan: Best for systematic literature reviews with collaboration.

Consensus: Best for finding evidence-based answers quickly.

Unpaywall: Best for accessing open-access scholarly articles.

Lateral: Best for organizing and analyzing research documents

Top 10 Best AI Tools for Literature Review

​ 1. Elephas 

​ Elephas is the best AI tool for literature review, designed to revolutionize your writing and research experience. With its robust suite of features, Elephas ensures that every aspect of your writing process is covered. From its offline capabilities, which keep your data secure, to the ability to integrate multiple AI models like OpenAI, Claude, and Gemini, Elephas offers unparalleled versatility. 

The Super Brain feature takes it a step further by indexing YouTube videos and web pages, allowing you to store and access valuable research material easily. Whether you need to generate content, fix grammar, or create engaging replies, Elephas has the tools to enhance your productivity and creativity.

Key Features:

Multiple AI Providers: Experiment with various writing styles and voices from OpenAI, Claude, Gemini, and Groq.

Offline Functionality: Write with confidence using local LLms that ensure your data is never shared or used for external training.

Web Search: Seamlessly search the web and incorporate relevant information into your writing.

Super Brain: Index YouTube videos and web pages, store them for future use, and retrieve content easily for in-depth research.

Rewrite Modes: Choose from Zinsser, Friendly, Professional, and Viral modes to tailor your writing style to any need.

Smart Write: Generate high-quality content quickly with just a few prompts or keywords.

Continue Writing: Overcome writer's block by letting Elephas continue your text based on the context you provide.

Personalized Tones: Train Elephas to match your unique writing voice and style for a more personalized touch.

Many users have shared how Elephas has transformed their daily workflow, making it an essential tool they can’t live without. One user mentioned that Elephas is incredibly addictive, boosting productivity by 10x and ensuring their emails always look great. ​

Another long-time user praised the app for lasting through the years and highlighted the "brains" feature, which speeds up content creation, programming, and editing. 

With Elephas, users experience unmatched efficiency and quality, making it the best tool for anyone looking to enhance their productivity and content creation.

2. Research Rabbit  

Research Rabbit is a versatile AI-powered tool designed to streamline the process of finding, managing, and analyzing research papers. As one of the best AI tools for literature review, it offers a user-friendly platform for anyone to access academic publications. 

After registering, users can search for research articles by author, topic, or keyword, and organize their findings in a personalized library. This tool is dedicated to enhancing scholarly work by supporting every stage of research, from discovery to collaboration. 

AI-driven search engine: It has AI search engines that can find and index relevant academic papers from across the web.

Customizable collections: It has already built in collections for organizing and managing research articles in a way that suits your specific needs and preferences.

User-friendly interface: It is designed for seamless navigation and an intuitive research management experience.

Broad search criteria: It includes detailed filters for author, topic, and keyword to refine your research findings.

Free access: It has all features for free, providing a cost-effective solution for research management.

Free to use 

We could not find any public reviews on the tool, so we advise users to be cautious while using the tool.

3. Semantic Scholar

Semantic Scholar is an advanced AI tool designed to enhance your literature review process by providing in-depth, context-aware search results. Ideal for researchers across various disciplines, it simplifies the search for academic papers, helping users navigate through over 200 million publications efficiently. 

By understanding the content and context of scientific articles, Semantic Scholar delivers personalized search outcomes, making it an invaluable resource for accelerating your research efforts. As one of the Best AI Tools for Literature Review, it stands out for its ability to filter and present the most relevant literature based on your specific needs.

Speeds up literature searches: Delivers context-rich results that save time and streamline the research process.

Customized search outcomes: Provides personalized results by deeply understanding the content and context of academic articles.

Versatile academic support: Accommodates a wide range of disciplines, enhancing its utility across different research areas.

Extensive database access: Offers a comprehensive database of over 200 million papers, ensuring broad coverage of research topics.

Enhanced research efficiency: Utilizes advanced AI to drive personalized search capabilities, improving overall research productivity.

4. R Discovery 

R Discovery is a powerful tool designed to enhance the research discovery process for students and researchers. With access to over 250 million research papers, it provides personalized reading feeds customized to your specific interests, ensuring you stay updated with the latest research in your field. 

The platform allows you to create and manage multiple reading lists, offers multilingual and full-text audio features for enhanced accessibility, and sends smart research alerts to keep your research organized. 

Personalized Research Feeds: R Discovery curates a customized reading list based on your interests, ensuring you stay up-to-date with the latest research.

Multiple Reading Lists: Organize your research with separate reading lists for different projects.

Multilingual & Full-Text Audio: Access research in over 30 languages, including audio versions for enhanced comprehension.

Smart Research Alerts: Receive targeted notifications about relevant research without being overwhelmed.

Integration with Reference Managers: Seamlessly sync your library with tools other research tools. 

Paid Plan stats from 2.29$/month 

5. Mendeley

Mendeley is a versatile reference management software, ideal for researchers, academics, and students involved in literature reviews. As one of the best AI tools for literature review, it helps users organize and manage their references efficiently, making research more streamlined. Mendeley also enables users to annotate PDFs, collaborate with others, and discover relevant literature, ensuring a comprehensive research experience.

Reference Management: Easily organize, store, and search through all your references from a single, centralized library, simplifying literature management.

PDF Viewing and Annotation: Open PDFs directly within Mendeley’s viewer, where you can add highlights and detailed notes, all stored for easy access.

Collaboration: Share references and annotated documents with research teams by creating private groups, enhancing collaboration and teamwork.

Literature Discovery: Import references from external sources and use Mendeley’s network to find and share key research papers with ease.

Citation Generation: Effortlessly generate accurate citations and bibliographies in multiple styles using the Mendeley Cite add-in for Microsoft Word.

Paid Plan starts from $4.99/month

Several users have expressed disappointment with Mendeley, noting that it has become increasingly frustrating to use. One user mentioned that the tool has too many flaws, requiring constant log-ins and failing to save passwords, making it unbearable. 

Another user shared that Mendeley is now a pain to use, with issues like the Word plug-in needing constant reinstallation, corrupted passwords, and disappearing or duplicated references.

6. Scholarcy

Scholarcy is a powerful AI-driven tool that simplifies the literature review process by generating concise summaries from academic papers. Designed to assist researchers, students, and academics, it quickly extracts key information, making it easier to evaluate and understand complex research. Scholarcy stands out as one of the Best AI Tools for Literature Review, ensuring efficient management of vast academic content.

Flashcard Summaries: Quickly grasp the main points of research papers with interactive flashcards that provide a concise, easy-to-read overview of the content.

Smart Highlighting: Easily identify factual statements and research findings with color-coded highlights that guide you to the most critical sections of the text.

Full-Text Access: Directly access full-text articles and cited papers through convenient links, streamlining your literature review process.

Literature Discovery: Efficiently discover and screen relevant literature with detailed synopses and highlights, helping you absorb key points in minutes.

Reference Management Integration: Seamlessly export flashcard summaries and key highlights to reference management tools like Zotero for organized and efficient citation management.

A user expressed dissatisfaction with Scholarcy, describing it as offering "no value added." The review highlighted concerns that Scholarcy essentially copies and pastes sections of articles or chapters and misleadingly labels it as "AI summarizing." 

The user also noted that the quality of the service dropped significantly after their free subscription expired, and they experienced issues with the interface being glitchy. The review strongly advises against paying for this service.

Rayyan is a powerful AI-driven app designed to streamline the systematic literature review process. It helps researchers quickly sift through vast amounts of research by enabling efficient reference management, de-duplication, screening, and organization. 

With Rayyan, users can import references from diverse sources, apply inclusion and exclusion criteria, assign labels, and export data for detailed analysis. The tool also supports collaboration among remote teams, making it an excellent choice for students, librarians, and researchers globally.

Collaborative Reviews: Seamlessly collaborate with distributed teams from anywhere using Rayyan’s intuitive mobile app.

Efficient Reference Management: Quickly import, de-duplicate, and organize your research references to save time and reduce errors.

Customizable Criteria: Easily apply and adjust inclusion and exclusion criteria to fit your specific review needs.

Advanced Analytics: Export your data for in-depth analysis and generate comprehensive reports to support your findings.

Priority Support: Benefit from dedicated training and VIP support to enhance your productivity and overcome challenges efficiently.

Paid Plan starts from $8.33/month

8. Consensus 

Consensus AI is a cutting-edge search engine designed to help you quickly find evidence-based answers from scientific research. It uses artificial intelligence to extract and summarize findings from peer-reviewed studies , providing a fast and efficient way to access reliable information. 

Consensus allows users to refine their searches, explore various research topics, and save time by delivering concise answers and full-text access to relevant papers. For academic research, Consensus AI is among the Best AI Tools for Literature Review due to its ability to synthesize and present information clearly and accurately.

AI-Powered Insights: Extracts and synthesizes findings from over 200 million scholarly documents.

Advanced Search Capabilities: Answers direct questions and explores relationships between concepts.

Consensus Meter: Provides a summary of agreement levels among multiple studies.

ChatGPT Integration: Access scientific research directly within the ChatGPT interface. ​

Customizable Searches: Offers tools to refine searches and explore more options based on research needs.

Paid Plan starts from $8.99/month

We couldn’t find any trustworthy reviews available on the internet for the Consensus. We advise users to use the tool with caution.

9. Unpaywall

Unpaywall is a free tool that aims to make scholarly research more accessible by providing open access to a vast collection of academic articles. It is integrated with major databases like Scopus and Web of Science, searching over 50,000 publishers and repositories globally. 

Users can find free, full-text versions of articles using Digital Object Identifiers (DOIs), making Unpaywall a vital resource for researchers seeking literature without barriers. This makes it one of the Best AI Tools for Literature Review.

Simple Query Tool: Allows users to quickly determine if an open access version of a specific list of articles, identified by DOIs, is available in the Unpaywall database.

Browser Extension: Automatically searches for and highlights legally available, free versions of scholarly articles as you browse , providing instant access to full texts.

Extensive Database: Offers access to a comprehensive index of over 20 million free, legal full-text PDFs, ensuring that users can find a wide range of open access literature.

Global Integration: Seamlessly integrates with major academic databases like Dimensions, Scopus, and Web of Science, enhancing the reach and effectiveness of your literature search.

API Access: Provides flexible data retrieval options, including REST API, R API Wrapper, or full dataset download, catering to various research and data management needs.

10. Lateral

Lateral is one of the Best AI Tools for Literature Review, designed to enhance your academic research process. This AI-powered app helps streamline your workflow by organizing, searching, and saving information from various research papers. 

With Lateral, you can efficiently analyze key concepts, relationships, and trends across your documents. The tool supports literature reviews by enabling you to manage sources and citations effortlessly, making research and paper writing much faster and easier.

Auto-Generated Table: Keeps an organized overview of all your research findings and references.

AI-Powered Concepts: Suggests relevant text across all your papers based on named concepts.

Super Search: Allows searching across all papers at once with highlighted similar results.

Smart PDF Reader: Facilitates reading and highlighting directly in the browser for better connection discovery.

Powerful OCR: Converts text from scanned PDFs into searchable and highlightable formats.

Paid Plans starts at $11.02/month

Literature AI tools are designed to significantly speed up the process of conducting literature research, helping researchers, students, and professionals save valuable time. These tools use advanced algorithms to automate various tasks, making literature research more efficient. Here’s an overview of the different types of literature AI tools available:

Literature Summary Tools: Quickly condense lengthy texts into concise summaries, making it easier to grasp key points.

Literature Research Tools: Assist in finding and organizing relevant research papers and articles.

Literature Review Tools: Provide detailed analyses and critiques of existing literature to support comprehensive reviews.

Writing Assistance Tools: Aid in drafting and editing texts, improving writing quality and coherence.

However, there are some tools such as Elephas which have all the features combined and it is perfect for researchers. It can summarize, review, write, assist, and many. ​

Using AI tools for literature review brings significant advantages, making the entire process smoother and more effective. These tools are particularly valuable for researchers, students, and anyone engaged in extensive literature work.

Here are some key benefits of these tools:

Time Efficiency: AI tools cut down the time needed to gather and summarize information. This can make research tasks much faster, almost halving the time you spend on literature review.

Accuracy: With AI handling data analysis, you can trust that the summaries and insights are precise, reducing the likelihood of mistakes.

Better Organization: AI tools help keep research materials neatly organized. This makes it easier to track and retrieve relevant information when needed.

Deep Insights: These tools dive deep into texts, offering detailed analysis and extracting essential points that might be missed otherwise.

Boosted Productivity: By automating repetitive tasks, AI tools let you focus on more critical parts of your work, increasing overall productivity.

To select the best AI tools for literature review, we carefully evaluated several key factors to ensure that  each  tool provides significant value to researchers, students, and academics. Here’s how we picked the top tools:

Functionality: We looked at the core features each tool offers, such as summarization, reference management, and advanced search capabilities. Tools that provide comprehensive and unique features stood out.

User Experience: The ease of use and intuitive interface were essential. We favored tools that are user-friendly and require minimal training, making them accessible for everyone.

Pricing: We assessed the cost-effectiveness of each tool, considering both free and paid options. Tools that offer a good balance between features and affordability were given priority.

Performance and Accuracy: We tested how well each tool performs its tasks, such as summarizing research papers or managing references. Tools that deliver accurate and reliable results were preferred.

Customer Reviews: User feedback and reviews helped us gauge the real-world effectiveness of each tool. We considered both positive and critical reviews to ensure a well-rounded selection.

By focusing on these criteria, we identified the best AI tools for literature review that provide robust features, ease of use, and excellent value, making them ideal choices for anyone involved in academic research. 

To wrap things up, the right AI tool can make a huge difference in your literature review process, turning hours of work into a streamlined, efficient task. Each tool on the list has its strengths—like Research Rabbit’s intuitive organization or Semantic Scholar’s smart search options. However, Elephas really shines when it comes to an all-in-one solution.

With its blend of multiple AI models, offline support, and features like Super Brain indexing, Elephas isn't just another tool—it's a game-changer for anyone serious about research. 

It simplifies complex tasks and adapts to your workflow, making it an indispensable part of your toolkit. If you want to elevate your literature review experience and work smarter, Elephas is the choice to make.

However, test out each tool according to your requirements and choose the one that fits best to your needs. All the best AI tools for literature are the best; you need to choose the one that can exactly fit your research requirements.

Elephas is the best AI tool for literature review, offering a comprehensive suite of features including offline capabilities, multiple AI models, and advanced indexing options like Super Brain for YouTube and web pages.

AI tools for literature review are versatile and can be adapted to various research fields. However, their effectiveness may vary depending on the complexity of the research topic and the specific needs of the researcher. Research and choose the tool that aligns with your research objectives. 

Some limitations of AI tools for literature review include potential biases in AI algorithms, the need for human oversight to ensure accuracy, and the possibility of missing nuanced information that requires expert interpretation. It’s important to use AI tools as a supplement to, rather than a replacement for, thorough research

AI assistant

Sign up now

Get a deep dive into the most important AI story of the week. Deliverd to your inbox for free!

Elephas helps you write faster and smarter on Mac - It's the best AI powered writing assistant for your Apple devices - Mac, iPhone and iPad.

You may also want to read

What’s New from Apple? iPhone 16 with AI, Apple Watch Series 10, and More!

iPhone Productivity

Jenni AI Review HONEST Review (2024): Is it the Best AI Writing Assistant for Research Papers

How to Use AI for Literature Review (2024): Complete 7 Step Guide for Researchers

Library Services Menu

• Open Access & Publishing Resources
• AI Tools & Responsible Use
• Library Orientation
• Literature Searching

Systematic Reviews

• Course Support, Reserves, and Linking

Research Guide for Faculty

• Information for Faculty by Adorée Hatton Makusztak Last Updated Aug 26, 2024 914 views this year

Systematic Review Process with a Librarian

The librarian plays an integral role in systematic reviews at Loma Linda University. 

What is a systematic review?

Cochrane Reviews provides the following definition for a systematic review: "A systematic review attempts to identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question. Researchers conducting systematic reviews use explicit, systematic methods that are selected with a view aimed at minimizing bias, to produce more reliable findings to inform decision making."

A systematic review is a rigorous and comprehensive approach to reviewing and synthesizing existing research literature on a specific topic. It goes beyond a traditional literature review by using a systematic and transparent process to identify, select, appraise, and analyze relevant studies.

The purpose of a systematic review is to provide a reliable and unbiased summary of the available evidence on a particular research question or topic. By systematically searching for and critically evaluating all relevant studies, systematic reviews aim to minimize bias and provide a more objective assessment of the existing evidence.

Systematic reviews are essential in research for several reasons:

Evidence-based decision making

Summarizing complex bodies of evidence

Identifying research gaps and priorities

Resolving conflicting findings

Improving research efficiency

Systematic Review Service Staff:

To request a systematic review service, contact the jbi certified librarians below: .

 Research & Instruction Librarian

 liaison to the school of allied health professions,    and the school of public health.

 office  (909) 558-1000 ext. 47564  ·   e-mail   [email protected]

  Make an appointment with Adorée

 Liaison to the School of Pharmacy, the School of Dentistry, 

   and the school of nursing (undergraduate).

 office: (909) 558-1000 ext. 47561 e-mail:  [email protected]

 Shan Tamares

 library director.

 office:  (909) 558-1000 ext. 47501 

 e-mail:  [email protected]

• << Previous: Literature Searching
• Next: EndNote >>
• Last Updated: Sep 12, 2024 10:00 AM
• URL: https://libguides.llu.edu/library-menu

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• For authors
• Browse by collection
• BMJ Journals

You are here

• Volume 14, Issue 9
• Guideline concordant screening and monitoring of extrapyramidal symptoms in patients prescribed antipsychotic medication: a protocol for a systematic literature review and narrative synthesis
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0009-0008-8045-1704 Rebekah Aubry 1 ,
• http://orcid.org/0000-0002-7777-0981 Thomas Hastings 2 , 3 ,
• http://orcid.org/0009-0001-5954-4702 Micheal Morgan 4 ,
• Jacqueline Hastings 5 ,
• Marie Bolton 6 ,
• Maura Grummell 7 ,
• Sinead Killeen 1 ,
• Cathal Coyne 8 ,
• Risa Shorr 9 ,
• http://orcid.org/0000-0003-4877-7233 Marco Solmi 10
• 1 Department of Psychiatry , Lucena Clinic Services , Dublin , Ireland
• 2 Department of Psychiatry , McMaster University , Hamilton , Ontario , Canada
• 3 Department of Psychiatry , University of Toronto , Toronto , Ontario , Canada
• 4 Department of Psychiatry , South Louth CAMHS , Drogheda , Ireland
• 5 School of Medicine , UCD , Dublin , Ireland
• 6 Department of Child and Adolescent Psychiatry , St Vincent's Hospital Fairview , Dublin , Ireland
• 7 Department of Psychiatry , Mater Misericordiae University Hospital , Dublin , Ireland
• 8 Department of Child and Adolescent Psychiatry , West Kildare CAMHS Linn Dara , Abbeylands Clane , Ireland
• 9 Learning Services , Ottawa Hospital , Ottawa , Ontario , Canada
• 10 Ottawa Hospital Research Institute , Ottawa , Ontario , Canada
• Correspondence to Dr Rebekah Aubry; rebekah.aubry{at}sjog.ie

Introduction Given the increasing rates of antipsychotic use in multiple psychiatric conditions, greater attention to the assessment, monitoring and documentation of their side effects is warranted. While a significant degree of attention has been provided to metabolic side effect monitoring, comparatively little is known about how clinicians screen for, document and monitor the motor side effects of antipsychotics (ie, parkinsonism, akathisia, dystonia and dyskinesias, collectively ‘extrapyramidal side effects’, EPS). This review aims to systematically assess the literature for insights into current trends in EPS monitoring practices within various mental health settings globally.

Methods and analysis An electronic search will be performed using the OVID Medline, PubMed, Embase, CINAHL and APA PsycINFO databases for studies published in the last quarter century (1998 to present day). Two independent reviewers will conduct the initial title and abstract screenings, using predetermined criteria for inclusion and exclusion. A third reviewer will resolve disagreements if consensus cannot be reached. If selected for inclusion, full-text data extraction will then be conducted using a pilot-tested data extraction form. Quality assessment will be conducted for all included studies using a modified version of the Quality Improvement Minimum Quality Criteria Set. A narrative synthesis and summary of the data will be provided. All stages of the review process will be reported in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines.

Ethics and dissemination Ethical approval is not required. Findings will be peer reviewed, published and shared verbally, electronically and in print with interested clinicians and will also be presented as posters or talks at relevant medical conferences and meetings.

PROSPERO registration number CRD42023482372.

• Systematic Review
• Schizophrenia & psychotic disorders
• Protocols & guidelines

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See:  http://creativecommons.org/licenses/by-nc/4.0/ .

https://doi.org/10.1136/bmjopen-2024-087632

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

STRENGTHS AND LIMITATIONS OF THIS STUDY

The search strategy was developed a priori in collaboration with an experienced health sciences librarian and involves a comprehensive search across five large databases and platforms.

The protocol follows the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols guidelines enhancing replicability and transparency.

Included studies will be rated based on their methodological quality using a modified version of the Quality Improvement Minimum Quality Criteria Set quality assessment tool developed by Hempel et al , which is suitable for the quality assessment of various types of service evaluation studies.

Due to resource constraints, the literature search will be restricted to English-only, peer-reviewed publications, possibly increasing the risk of selection bias and limiting the generalisability of review findings.

Introduction

Second generation antipsychotics (SGAs) are broadly used in clinical practice, not only for the treatment of psychotic and bipolar disorders but also for a variety of other conditions. 1–3 While SGAs are associated with a lower risk of motor side effects (ie, parkinsonism, akathisia, dystonia and dyskinesias, collectively ‘extrapyramidal side effects’, EPSs) than first-generation antipsychotics the rates of EPS remain significant. 4–8 Furthermore, EPSs are associated with impaired quality of life, medication non-adherence, increased morbidity, mortality, caregiver burden, utilisation of healthcare resources and higher medical costs. 8–16 This has resulted in some advocating for ‘better monitoring … to assess their true effect on patients’ quality of life and functioning and to prevent underascertainment’, 17 something especially important in higher risk populations, for instance, children, adolescents and the elderly. 18–20 The most recent American Psychiatric Association’s guidelines (2020) for the treatment of patients with schizophrenia calls for clinical assessment of EPS at baseline or initial assessment, at each subsequent visit as well as an assessment using a ‘structured instrument’ every 6 months in patients at increased risk of tardive dyskinesia and every 12 months for all other patients. 21 In the UK, the National Institute for Health and Care Excellence guidelines recommend assessment of any movement disorders before starting antipsychotic medication as part of baseline investigations and to monitor and record side effects of treatment and their impact on functioning, and the emergence of movement disorders, regularly and systematically throughout treatment and especially during titration. 22 Unfortunately, evidence demonstrates that actual monitoring rates fall far below these standards. 23–25

Rationale for the review

While a significant degree of attention has been provided to metabolic side effect monitoring, with several systematic reviews conducted on the subject, 26 27 comparatively little is known about EPS monitoring practices.

When it comes to EPS, its incidence and prevalence in research and naturalistic settings have been thoroughly investigated in numerous studies and reviews. 4–6 28 However, there seems to be a paucity of data about current practices relating to how clinicians screen for, monitor and document EPS in patients prescribed antipsychotics. Gaining a better understanding of current practice may allow for the introduction of effective interventions that help address the existing discrepancy between current practice and best practice.

Aim and objectives

The aim of this review is to systematically assess the literature, seeking insights into current EPS monitoring practices within various mental health settings globally.

Our three main objectives are as follows: (1) to identify the extent to which patients prescribed antipsychotic medication receive guideline concordant monitoring, (2) to gather data on interventions that have been proposed to improve this aspect of care and (3) to identify any existing barriers.

Research questions

In accordance with the aim and objectives outlined above, this review will seek to answer the following questions as regards EPS monitoring for patients who are prescribed antipsychotic medication:

Which guidelines if any are being used to guide current practice and arerecommended standards being met? What screening tools are being used?

What is the frequency of monitoring? Has it improved or worsened over the years?

What interventions have been proposed to improve monitoring standards?

What are some of the possible barriers to adequate monitoring?

Methods and design

All stages of the review process including literature searching, screening, applying inclusion and exclusion criteria and data extraction will be reported and documented in accordance with the Preferred Reporting Items for Systematic Review and Met-Analysis Protocol (PRISMA-P) statement. 29 The PRISMA-P was used to guide the development of the review protocol (see online supplemental file 1 for PRISMA-P checklist). 30 In accordance with the guidelines, this systematic review protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO) under the reference number CRD42023482372. Any amendments to the protocol will be reported when publishing the results.

Supplemental material

Inclusion and exclusion criteria (eligibility of studies).

These are grouped under the following seven subsections:

Study design

Study designs aimed at gathering data on current practices relating to EPS documentation and monitoring as well as studies describing interventions developed to improve clinical performance in the area of documentation and monitoring of EPS will be included in the review. Examples of study designs that will be included are as follows:

Clinical audits without intervention.

Clinical audits with completed audit cycles after intervention.

Service evaluations without a quality improvement intervention.

Service evaluations following a quality improvement intervention.

However, the following study design types will be excluded:

Case reports.

Any trial design, including randomized controlled trials(RCTs).

Literature reviews.

Discussion and viewpoint studies.

Grey literature.

Abstract-only publications.

Epidemiological studies of incidence/prevalence of EPS.

Survey designs.

Types of intervention

All types of interventions concerned with the assessment, screening and monitoring of EPS will be included. This will involve gathering data on the types of processes currently used to carry out EPS monitoring and documentation as well as on any proposed interventions aimed at improving EPS documentation and monitoring such as educational interventions, adoption of novel screening instruments, etc.

Study language

This systematic review will be restricted to English language studies only.

Publication dates

Studies published from 1998 to the present will be included, spanning the last 25 years of clinical practice. We consider this sufficiently representative of contemporary trends in practice.

Study population/demographics

The first population of interest includes patients of all ages and genders receiving treatment for one or more mental health conditions and prescribed one or more antipsychotic medications. While it is true that EPS can manifest spontaneously in patients who were never exposed to antipsychotic agents 31 32 or can be caused by substances other than antipsychotics, 33–35 a substantial proportion of reported EPS is attributed to antipsychotic medication. 6 36 37 Moreover, even within cohorts of previously neuroleptic naïve patients, research suggests that dopamine D2 receptor antagonist antipsychotics interact with the disease process in such a way that ‘precipitates’ and ‘accentuates’ movement disorders intrinsic to schizophrenia’. 38 This review will, therefore, focus on patients prescribed antipsychotic medication, as they may be at higher risk of developing severe EPS. In addition, most available guidelines on EPS monitoring specifically refer to patients prescribed antipsychotic medications.

The second population of interest includes the healthcare professionals involved in the care of the patients (eg, nurses, residents, clinicians and pharmacists) and tasked with carrying out EPS monitoring.

Study settings

Studies reporting on EPS monitoring practices in any naturalistic, real-world clinical setting, including inpatient hospitals, day hospitals, outpatient clinics, community settings, etc will be included.

Other phenomena of interest

Where available, data on the views, experiences and behaviours of healthcare professionals and patients involved in the assessment, screening and monitoring of EPS will also be collected.

Patient and public involvement

Patients and/or the public were not involved in the design, conduct, reporting or dissemination plans of this protocol.

Information sources

Electronic sources.

The literature search was conducted using the following five databases and search platforms: OVID Medline, PubMed, EMBASE, PsycINFO and CINAHL. The initial search covers 25 years and includes studies published between April 1998 and April 2023. These searches will be re-run immediately prior to the final analysis (projected to take place in September 2024) and potential further studies will be retrieved for inclusion, ensuring that the most up-to-date information is presented in the review. The reference lists of all eligible articles will be manually searched to identify any additional relevant citations to ensure a comprehensive search.

Search strategy

Review authors RA and RS (librarian and information specialist with expertise in electronic searching) developed and ran a comprehensive search strategy. A scoping search was undertaken against each database to inform how the search terms were being translated and hence to identify the corresponding text words in each database. Following this, the complete search strategy was tested for its sensitivity to locate the key papers that the researchers are already aware of, along with relevant articles which are consistent with the inclusion criteria just before running the search through all the selected search engines.

The search strategy used variations in text words found in the title, abstract or keyword fields, and relevant focused subject headings to retrieve articles combining the following three search concepts, linked by the Boolean operator ‘AND’:

(1) One or more medication terms: antipsychotic* OR psychotropic* OR haloperidol OR olanzapine OR quetiapine OR risperidone OR cariprazine OR amisulpride OR aripiprazole OR lurasidone etc… (to include full list of antipsychotic medication listed as per the WHO Collaboration Centre for Drug Statistics Methodology ATC classification).

(2) One or more EPS terms: “Extrapyramidal symptom*” OR “Extrapyramidal side effect*” OR “drug-induced movement disorder*” OR ‘Drug-Related Side Effects and Adverse Reactions’ OR ‘movement side effects’ OR Dystonia OR ‘acute dystonia’ OR parkinsonism OR ‘drug-induced parkinsonism’ OR akathisia OR “tardive dyskinesia” OR tremor

(3) One or more terms relating to monitoring, screening, documenting or auditing clinical practice (including screening instruments): ‘Monitoring’ OR ‘Screening’ OR ‘Documenting’ OR ‘Documentation’ OR ‘Assessing’ OR ‘Assessment’ OR ‘Abnormal Involuntary Movement Scale’ OR ‘Extrapyramidal Symptom Rating Scale’ OR ‘Simpson-Angus Scale’ OR ‘Barnes Akathisia Scale’.

The search included all relevant synonyms, truncations and Mesh terms. Full details of search terms used for the OVID Medline search are shown in online supplemental file 2 . A similar search was conducted using the other databases and search platforms. The full search strategy is available on request from the corresponding author.

Study records

Data management.

The search results will be uploaded into web-based, systematic review management software (Covidence). Duplicates will be removed automatically by Covidence software. Authors RA and MM will scan through the results to remove any remaining duplicate records manually. Using Covidence, the initial title and abstract screening, and the full-text review will be logged. All standardised forms will be piloted and revised as needed by the reviewers before starting the review.

Screening and selection process

After identification of articles from searching the electronic databases, titles and abstracts will be screened independently by two review authors according to the predefined eligibility criteria. Disagreements will be resolved by consensus and the opinion of a third reviewer will be sought if necessary. The full-text copies of each potentially relevant study will then be retrieved and screened independently by at least two reviewers including the first author (RA). Consensus will be reached through discussion, and in the event that no consensus can be reached for a study, a third reviewer will arbitrate. All studies not meeting the eligibility criteria will be excluded. The results will be reported using the PRISMA flow diagram.

Data extraction and reporting of results

A standardised data extraction form will be developed to extract all relevant data from included studies. Information to be extracted will be as follows:

Study characteristics: authors, date, settings, country of origin, study design and sample size.

Patient characteristics: demographic data (age, gender, diagnosis, type of antipsychotic prescribed, etc.).

Monitoring characteristics: frequency, use of a structured tool, healthcare professionals involved in monitoring, guidelines followed, etc.

Intervention characteristics: (if study incorporated a preintervention/postintervention design): educational intervention, adoption of a new instrument, etc.

The data extraction form will be piloted on a small random sample (n=3) of the illegible studies to assess its reliability in extracting the targeted study data. Review authors TH, MB and SK will each independently conduct data extraction on the three studies. Review authors RA and MM will then review this extracted data, checking against the full text of the three studies for any discrepancies (eg, errors, omissions or failure to have consensus in any area) and will decide on how to resolve any that may arise. If the above pilot data extraction process is deemed reliable then the review authors TH, MB and SK will each independently conduct data extraction on the remaining studies in the systematic review. Review authors RA and MM will then cross-check the extracted data against the full-text articles in a similar process to that highlighted above.

Additionally, study authors will be contacted if necessary to gain information for clarification purposes and access to raw material when needed.

Critical appraisal of study quality

Authors RA and MM will use the Quality Improvement Minimum Quality Criteria Set (QI-MQCS) developed by Hempel et al to conduct the quality assessment of included studies. 39 Disagreements will be resolved by consensus; the opinion of a third reviewer (MG) will be sought if necessary. The QI-MQCS is a 16-domain, validated, reliable critical appraisal tool that assesses expert-endorsed QI domains for studies that include a QI intervention component. The QI-MQCS will be modified to be suitable for the body of studies included in our review, and in particular, to be able to assess studies with no intervention component, that is, clinical audits and service evaluations with no intervention. This will involve accepting a broader definition of several domains of the appraisal instrument to include studies evaluating existing services or standards in addition to QI intervention. This approach was chosen in the absence of a suitable tool for critical appraisal of service evaluation studies with no intervention component.

The QI-MQCS tool is designed to provide a score for each domain as well as a total score, which is expressed as a percentage of the maximum possible score.

Data synthesis

In this review, the search is expected to reveal heterogeneous studies and meta-analysis of study findings is therefore not a study objective. Therefore, data synthesis will take the form of a structured narrative synthesis of the included studies. The defining characteristic of a narrative synthesis is that it adopts a textual approach to the process of synthesis in order to provide answers to the identified research questions in a structured manner. Study findings pertaining to the following three themes will be examined and synthesised: (1) Data concerning the extent and quality of EPS monitoring being carried out in various mental health settings will be summarised. (2) Following this, details about any potential interventions employed to improve monitoring practices will be synthesised. And finally, (3) Information about any identifiable barriers or facilitators to guideline concordant EPS monitoring will be synthesised and discussed.

Study status

The study is ongoing and is expected to be completed by September 2024.

Proposed value of the systematic review and use of the findings

This systematic review seeks to shed light on the existing patterns of EPS monitoring occurring within various mental health settings. The findings of this systematic review may be of interest to mental health organisations and services as they are expected to provide insights into the potential barriers or facilitators (including possible quality improvement interventions) influencing whether EPS monitoring is carried out in a guideline concordant manner. This may in turn encourage organisations and services to assess their existing EPS monitoring practice and/or lead them to consider the adoption or development of interventions to improve monitoring standards.

Ethics statements

Patient consent for publication.

Not applicable.

• Radojčić MR ,
• Hope H , et al
• McGrath J ,
• Teeling M , et al
• Kelleher JE ,
• Hsieh C-H ,
• Kane JM , et al
• Misdrahi D ,
• Tessier A ,
• Daubigney A , et al
• Martino D ,
• Osland S , et al
• Baba Shettima F ,
• Lateef Sheikh T ,
• Abba Wakil M , et al
• Kadakia A ,
• Dembek C , et al
• Schouten HJ ,
• Egberts TCG , et al
• Gandhi SK ,
• Rizio AA , et al
• Millier A ,
• Boyer L , et al
• V G , et al
• Cutler AJ ,
• Caroff SN ,
• Tanner CM , et al
• Yeomans K ,
• Lenderking WR , et al
• Monteleone P ,
• Cascino G ,
• Monteleone AM , et al
• Vitiello B ,
• Correll C ,
• van Zwieten-Boot B , et al
• Estevez-Fraga C ,
• López-Sendón Moreno JL
• McInerney BE ,
• Alderman CP , et al
• Keepers GA ,
• Fochtmann LJ ,
• Anzia JM , et al
• NICE clinical guideline
• Cortese L ,
• McAuley TJ , et al
• Butler MI ,
• Chandrakanth J
• Mitchell AJ ,
• Delaffon V ,
• Vancampfort D , et al
• Blake JA , et al
• Tariku M , et al
• Liberati A ,
• Shamseer L ,
• Clarke M , et al
• Perju-Dumbrava L ,
• Procyshyn RM ,
• Jones AA , et al
• Seltenreich D ,
• Letmaier M , et al
• Blanchet P ,
• Wang D , et al
• Waddington JL
• Shekelle PG ,
• Liu JL , et al

Contributors RA is the author acting as guarantor. The study was conceived by RA, MS, MM and TH. RA and MM developed the eligibility criteria, search strategy, quality assessment strategy and data extraction plan with guidance from MS and RS. RA, TH and MM wrote the manuscript. MS, MB, MM, MG, JH, SK and CC read all drafts of the manuscript, provided feedback and approved the final manuscript. All contributors meet the ICMJE criteria for authorship.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests MS has received honoraria/has been a consultant for AbbVie, Angelini, Lundbeck, Otsuka.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF:

This week: the arXiv Accessibility Forum

Help | Advanced Search

Computer Science > Software Engineering

Title: generative ai for requirements engineering: a systematic literature review.

Abstract: Context: Generative AI (GenAI) has emerged as a transformative tool in software engineering, with requirements engineering (RE) actively exploring its potential to revolutionize processes and outcomes. The integration of GenAI into RE presents both promising opportunities and significant challenges that necessitate systematic analysis and evaluation. Objective: This paper presents a comprehensive systematic literature review (SLR) analyzing state-of-the-art applications and innovative proposals leveraging GenAI in RE. It surveys studies focusing on the utilization of GenAI to enhance RE processes while identifying key challenges and opportunities in this rapidly evolving field. Method: A rigorous SLR methodology was used to analyze 27 carefully selected primary studies in-depth. The review examined research questions pertaining to the application of GenAI across various RE phases, the models and techniques used, and the challenges encountered in implementation and adoption. Results: The most salient findings include i) a predominant focus on the early stages of RE, particularly the elicitation and analysis of requirements, indicating potential for expansion into later phases; ii) the dominance of large language models, especially the GPT series, highlighting the need for diverse AI approaches; and iii) persistent challenges in domain-specific applications and the interpretability of AI-generated outputs, underscoring areas requiring further research and development. Conclusions: The results highlight the critical need for comprehensive evaluation frameworks, improved human-AI collaboration models, and thorough consideration of ethical implications in GenAI-assisted RE. Future research should prioritize extending GenAI applications across the entire RE lifecycle, enhancing domain-specific capabilities, and developing strategies for responsible AI integration in RE practices.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

How-to conduct a systematic literature review: A quick guide for computer science research

Angela carrera-rivera.

a Faculty of Engineering, Mondragon University

William Ochoa

Felix larrinaga.

b Design Innovation Center(DBZ), Mondragon University

Associated Data

• No data was used for the research described in the article.

Performing a literature review is a critical first step in research to understanding the state-of-the-art and identifying gaps and challenges in the field. A systematic literature review is a method which sets out a series of steps to methodically organize the review. In this paper, we present a guide designed for researchers and in particular early-stage researchers in the computer-science field. The contribution of the article is the following:

• • Clearly defined strategies to follow for a systematic literature review in computer science research, and
• • Algorithmic method to tackle a systematic literature review.

Graphical abstract

Specifications table

Method details

A Systematic Literature Review (SLR) is a research methodology to collect, identify, and critically analyze the available research studies (e.g., articles, conference proceedings, books, dissertations) through a systematic procedure [12] . An SLR updates the reader with current literature about a subject [6] . The goal is to review critical points of current knowledge on a topic about research questions to suggest areas for further examination [5] . Defining an “Initial Idea” or interest in a subject to be studied is the first step before starting the SLR. An early search of the relevant literature can help determine whether the topic is too broad to adequately cover in the time frame and whether it is necessary to narrow the focus. Reading some articles can assist in setting the direction for a formal review., and formulating a potential research question (e.g., how is semantics involved in Industry 4.0?) can further facilitate this process. Once the focus has been established, an SLR can be undertaken to find more specific studies related to the variables in this question. Although there are multiple approaches for performing an SLR ( [5] , [26] , [27] ), this work aims to provide a step-by-step and practical guide while citing useful examples for computer-science research. The methodology presented in this paper comprises two main phases: “Planning” described in section 2, and “Conducting” described in section 3, following the depiction of the graphical abstract.

Defining the protocol is the first step of an SLR since it describes the procedures involved in the review and acts as a log of the activities to be performed. Obtaining opinions from peers while developing the protocol, is encouraged to ensure the review's consistency and validity, and helps identify when modifications are necessary [20] . One final goal of the protocol is to ensure the replicability of the review.

Define PICOC and synonyms

The PICOC (Population, Intervention, Comparison, Outcome, and Context) criteria break down the SLR's objectives into searchable keywords and help formulate research questions [ 27 ]. PICOC is widely used in the medical and social sciences fields to encourage researchers to consider the components of the research questions [14] . Kitchenham & Charters [6] compiled the list of PICOC elements and their corresponding terms in computer science, as presented in Table 1 , which includes keywords derived from the PICOC elements. From that point on, it is essential to think of synonyms or “alike” terms that later can be used for building queries in the selected digital libraries. For instance, the keyword “context awareness” can also be linked to “context-aware”.

Planning Step 1 “Defining PICOC keywords and synonyms”.

Formulate research questions

Clearly defined research question(s) are the key elements which set the focus for study identification and data extraction [21] . These questions are formulated based on the PICOC criteria as presented in the example in Table 2 (PICOC keywords are underlined).

Research questions examples.

Select digital library sources

The validity of a study will depend on the proper selection of a database since it must adequately cover the area under investigation [19] . The Web of Science (WoS) is an international and multidisciplinary tool for accessing literature in science, technology, biomedicine, and other disciplines. Scopus is a database that today indexes 40,562 peer-reviewed journals, compared to 24,831 for WoS. Thus, Scopus is currently the largest existing multidisciplinary database. However, it may also be necessary to include sources relevant to computer science, such as EI Compendex, IEEE Xplore, and ACM. Table 3 compares the area of expertise of a selection of databases.

Planning Step 3 “Select digital libraries”. Description of digital libraries in computer science and software engineering.

Define inclusion and exclusion criteria

Authors should define the inclusion and exclusion criteria before conducting the review to prevent bias, although these can be adjusted later, if necessary. The selection of primary studies will depend on these criteria. Articles are included or excluded in this first selection based on abstract and primary bibliographic data. When unsure, the article is skimmed to further decide the relevance for the review. Table 4 sets out some criteria types with descriptions and examples.

Planning Step 4 “Define inclusion and exclusion criteria”. Examples of criteria type.

Define the Quality Assessment (QA) checklist

Assessing the quality of an article requires an artifact which describes how to perform a detailed assessment. A typical quality assessment is a checklist that contains multiple factors to evaluate. A numerical scale is used to assess the criteria and quantify the QA [22] . Zhou et al. [25] presented a detailed description of assessment criteria in software engineering, classified into four main aspects of study quality: Reporting, Rigor, Credibility, and Relevance. Each of these criteria can be evaluated using, for instance, a Likert-type scale [17] , as shown in Table 5 . It is essential to select the same scale for all criteria established on the quality assessment.

Planning Step 5 “Define QA assessment checklist”. Examples of QA scales and questions.

Define the “Data Extraction” form

The data extraction form represents the information necessary to answer the research questions established for the review. Synthesizing the articles is a crucial step when conducting research. Ramesh et al. [15] presented a classification scheme for computer science research, based on topics, research methods, and levels of analysis that can be used to categorize the articles selected. Classification methods and fields to consider when conducting a review are presented in Table 6 .

Planning Step 6 “Define data extraction form”. Examples of fields.

The data extraction must be relevant to the research questions, and the relationship to each of the questions should be included in the form. Kitchenham & Charters [6] presented more pertinent data that can be captured, such as conclusions, recommendations, strengths, and weaknesses. Although the data extraction form can be updated if more information is needed, this should be treated with caution since it can be time-consuming. It can therefore be helpful to first have a general background in the research topic to determine better data extraction criteria.

After defining the protocol, conducting the review requires following each of the steps previously described. Using tools can help simplify the performance of this task. Standard tools such as Excel or Google sheets allow multiple researchers to work collaboratively. Another online tool specifically designed for performing SLRs is Parsif.al 1 . This tool allows researchers, especially in the context of software engineering, to define goals and objectives, import articles using BibTeX files, eliminate duplicates, define selection criteria, and generate reports.

Build digital library search strings

Search strings are built considering the PICOC elements and synonyms to execute the search in each database library. A search string should separate the synonyms with the boolean operator OR. In comparison, the PICOC elements are separated with parentheses and the boolean operator AND. An example is presented next:

(“Smart Manufacturing” OR “Digital Manufacturing” OR “Smart Factory”) AND (“Business Process Management” OR “BPEL” OR “BPM” OR “BPMN”) AND (“Semantic Web” OR “Ontology” OR “Semantic” OR “Semantic Web Service”) AND (“Framework” OR “Extension” OR “Plugin” OR “Tool”

Gather studies

Databases that feature advanced searches enable researchers to perform search queries based on titles, abstracts, and keywords, as well as for years or areas of research. Fig. 1 presents the example of an advanced search in Scopus, using titles, abstracts, and keywords (TITLE-ABS-KEY). Most of the databases allow the use of logical operators (i.e., AND, OR). In the example, the search is for “BIG DATA” and “USER EXPERIENCE” or “UX” as a synonym.

Example of Advanced search on Scopus.

In general, bibliometric data of articles can be exported from the databases as a comma-separated-value file (CSV) or BibTeX file, which is helpful for data extraction and quantitative and qualitative analysis. In addition, researchers should take advantage of reference-management software such as Zotero, Mendeley, Endnote, or Jabref, which import bibliographic information onto the software easily.

Study Selection and Refinement

The first step in this stage is to identify any duplicates that appear in the different searches in the selected databases. Some automatic procedures, tools like Excel formulas, or programming languages (i.e., Python) can be convenient here.

In the second step, articles are included or excluded according to the selection criteria, mainly by reading titles and abstracts. Finally, the quality is assessed using the predefined scale. Fig. 2 shows an example of an article QA evaluation in Parsif.al, using a simple scale. In this scenario, the scoring procedure is the following YES= 1, PARTIALLY= 0.5, and NO or UNKNOWN = 0 . A cut-off score should be defined to filter those articles that do not pass the QA. The QA will require a light review of the full text of the article.

Performing quality assessment (QA) in Parsif.al.

Data extraction

Those articles that pass the study selection are then thoroughly and critically read. Next, the researcher completes the information required using the “data extraction” form, as illustrated in Fig. 3 , in this scenario using Parsif.al tool.

Example of data extraction form using Parsif.al.

The information required (study characteristics and findings) from each included study must be acquired and documented through careful reading. Data extraction is valuable, especially if the data requires manipulation or assumptions and inferences. Thus, information can be synthesized from the extracted data for qualitative or quantitative analysis [16] . This documentation supports clarity, precise reporting, and the ability to scrutinize and replicate the examination.

Analysis and Report

The analysis phase examines the synthesized data and extracts meaningful information from the selected articles [10] . There are two main goals in this phase.

The first goal is to analyze the literature in terms of leading authors, journals, countries, and organizations. Furthermore, it helps identify correlations among topic s . Even when not mandatory, this activity can be constructive for researchers to position their work, find trends, and find collaboration opportunities. Next, data from the selected articles can be analyzed using bibliometric analysis (BA). BA summarizes large amounts of bibliometric data to present the state of intellectual structure and emerging trends in a topic or field of research [4] . Table 7 sets out some of the most common bibliometric analysis representations.

Techniques for bibliometric analysis and examples.

Several tools can perform this type of analysis, such as Excel and Google Sheets for statistical graphs or using programming languages such as Python that has available multiple  data visualization libraries (i.e. Matplotlib, Seaborn). Cluster maps based on bibliographic data(i.e keywords, authors) can be developed in VosViewer which makes it easy to identify clusters of related items [18] . In Fig. 4 , node size is representative of the number of papers related to the keyword, and lines represent the links among keyword terms.

[1] Keyword co-relationship analysis using clusterization in vos viewer.

This second and most important goal is to answer the formulated research questions, which should include a quantitative and qualitative analysis. The quantitative analysis can make use of data categorized, labelled, or coded in the extraction form (see Section 1.6). This data can be transformed into numerical values to perform statistical analysis. One of the most widely employed method is frequency analysis, which shows the recurrence of an event, and can also represent the percental distribution of the population (i.e., percentage by technology type, frequency of use of different frameworks, etc.). Q ualitative analysis includes the narration of the results, the discussion indicating the way forward in future research work, and inferring a conclusion.

Finally, the literature review report should state the protocol to ensure others researchers can replicate the process and understand how the analysis was performed. In the protocol, it is essential to present the inclusion and exclusion criteria, quality assessment, and rationality beyond these aspects.

The presentation and reporting of results will depend on the structure of the review given by the researchers conducting the SLR, there is no one answer. This structure should tie the studies together into key themes, characteristics, or subgroups [ 28 ].

SLR can be an extensive and demanding task, however the results are beneficial in providing a comprehensive overview of the available evidence on a given topic. For this reason, researchers should keep in mind that the entire process of the SLR is tailored to answer the research question(s). This article has detailed a practical guide with the essential steps to conducting an SLR in the context of computer science and software engineering while citing multiple helpful examples and tools. It is envisaged that this method will assist researchers, and particularly early-stage researchers, in following an algorithmic approach to fulfill this task. Finally, a quick checklist is presented in Appendix A as a companion of this article.

CRediT author statement

Angela Carrera-Rivera: Conceptualization, Methodology, Writing-Original. William Ochoa-Agurto : Methodology, Writing-Original. Felix Larrinaga : Reviewing and Supervision Ganix Lasa: Reviewing and Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

Funding : This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie Grant No. 814078.

Carrera-Rivera, A., Larrinaga, F., & Lasa, G. (2022). Context-awareness for the design of Smart-product service systems: Literature review. Computers in Industry, 142, 103730.

1 https://parsif.al/

Data Availability

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Co-creation with ai in b2b markets: a systematic literature review.

1. Introduction

2. artificial intelligence in b2b marketing, 3. co-creation and artificial intelligence, 4. materials and methods, 5.1. descriptive analysis, 5.2. thematic analysis, 5.2.1. co-creation with ai in b2b: actors, motives, and characteristics, 5.2.2. co-creation with ai in b2b: processes, 5.2.3. co-creation with ai in b2b: content, 6. discussion and conclusions, 7. limitations, supplementary materials, author contributions, conflicts of interest.

• Statista. Ways in Which Marketers Used Artificial Intelligence (AI) Worldwide in 2021 and 2022. 2024. Available online: https://www.statista.com/statistics/1365066/ai-usage-marketing/ (accessed on 12 June 2024).
• Dwivedi, Y.K.; Kshetri, N.; Hughes, L.; Slade, E.L.; Jeyaraj, A.; Kar, A.K.; Baabdullah, A.M.; Koohang, A.; Raghavan, V.; Ahuja, M.; et al. “So what if ChatGPT wrote it?” Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. Int. J. Inf. Manag. 2023 , 71 , 102642. [ Google Scholar ] [ CrossRef ]
• Peres, R.; Schreier, M.; Schweidel, D.; Sorescu, A. On ChatGPT and beyond: How generative artificial intelligence may affect research, teaching, and practice. Int. J. Res. Mark. 2023 , 40 , 269–275. [ Google Scholar ] [ CrossRef ]
• Spanjol, J.; Noble, C.H. From the Editors: Engaging with generative artificial intelligence technologies in innovation management research—Some answers and more questions. J. Prod. Innov. Manag. 2023 , 40 , 383–390. [ Google Scholar ] [ CrossRef ]
• Bouschery, S.G.; Blazevic, V.; Piller, F.T. Augmenting human innovation teams with artificial intelligence: Exploring transformer-based language models. J. Prod. Innov. Manag. 2023 , 40 , 139–153. [ Google Scholar ] [ CrossRef ]
• Mariani, M.M.; Hashemi, N.; Wirtz, J. Artificial intelligence empowered conversational agents: A systematic literature review and research agenda. J. Bus. Res. 2023 , 161 , 113838. [ Google Scholar ] [ CrossRef ]
• OpenAI. GPT-4. 2023. Available online: https://openai.com/research/gpt-4 (accessed on 20 May 2024).
• McKinsey and Company. The State of AI in 2023: Generative AI’s Breakout Year. 2023. Available online: https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year (accessed on 20 May 2024).
• Deloitte Center for Integrated Research. Generating Value from Generative AI. 2023. Available online: https://www2.deloitte.com/us/en/insights/topics/digital-transformation/companies-investing-in-ai-to-generate-value.html (accessed on 20 May 2024).
• Leone, D.; Schiavone, F.; Appio, F.P.; Chiao, B. How does artificial intelligence enable and enhance value co-creation in industrial markets? An exploratory case study in the healthcare ecosystem. J. Bus. Res. 2021 , 129 , 849–859. [ Google Scholar ] [ CrossRef ]
• Kohtamäki, M.; Rajala, R. Theory and practice of value co-creation in B2B systems. Ind. Mark. Manag. 2016 , 56 , 4–13. [ Google Scholar ] [ CrossRef ]
• Prahalad, C.K.; Ramaswamy, V. Co-opting customer competence. Harv. Bus. Rev. 2000 , 78 , 79–90. [ Google Scholar ]
• Prahalad, C.K.; Ramaswamy, V. Co-creating unique value with customers. Strategy Leadersh. 2004 , 32 , 4–9. [ Google Scholar ] [ CrossRef ]
• Vargo, S.L.; Lusch, R.F. Evolving to a new dominant logic for marketing. J. Mark. 2004 , 68 , 1–17. [ Google Scholar ] [ CrossRef ]
• Marcos-Cuevas, J.; Nätti, S.; Palo, T.; Baumann, J. Value co-creation practices and capabilities: Sustained purposeful engagement across B2B systems. Ind. Mark. Manag. 2016 , 56 , 97–107. [ Google Scholar ] [ CrossRef ]
• Rusthollkarhu, S.; Hautamaki, P.; Aarikka-Stenroos, L. Value (co-) creation in B2B sales ecosystems. J. Bus. Ind. Mark. 2020 , 36 , 590–598. [ Google Scholar ] [ CrossRef ]
• Paschen, J.; Paschen, U.; Pala, E.; Kietzmann, J. Artificial intelligence (AI) and value co-creation in B2B sales: Activities, actors and resources. Australas. Mark. J. 2021 , 29 , 243–251. [ Google Scholar ] [ CrossRef ]
• Chandra, B.; Rahman, Z. Artificial intelligence and value co-creation: A review, conceptual framework and directions for future research. J. Serv. Theory Pract. 2024 , 34 , 7–32. [ Google Scholar ] [ CrossRef ]
• Palmatier, R.W.; Houston, M.B.; Hulland, J. Review articles: Purpose, process, and structure. J. Acad. Mark. Sci. 2018 , 46 , 1–5. [ Google Scholar ] [ CrossRef ]
• Tranfield, D.; Denyer, D.; Smart, P. Towards a methodology for developing evidence-informed management knowledge by means of systematic review. Br. J. Manag. 2003 , 14 , 207–222. [ Google Scholar ] [ CrossRef ]
• Haenlein, M.; Kaplan, A. A brief history of artificial intelligence: On the past, present, and future of artificial intelligence. Calif. Manag. Rev. 2019 , 61 , 5–14. [ Google Scholar ] [ CrossRef ]
• Steinberg, M.; Plank, R.E. Expert systems: The integrative sales management tool of the future. J. Acad. Mark. Sci. 1987 , 15 , 55–62. [ Google Scholar ] [ CrossRef ]
• Alexander, V.; Blinder, C.; Zak, P.J. Why trust an algorithm? Performance, cognition, and neurophysiology. Comput. Hum. Behav. 2018 , 89 , 279–288. [ Google Scholar ] [ CrossRef ]
• Murray, A.; Rhymer, J.E.N.; Sirmon, D.G. Humans and technology: Forms of conjoined agency in organizations. Acad. Manag. Rev. 2021 , 46 , 552–571. [ Google Scholar ] [ CrossRef ]
• Kaplan, A.; Haenlein, M. Siri, Siri, in my hand: Who’s the fairest in the land? On the interpretations, illustrations, and implications of artificial intelligence. Bus. Horiz. 2019 , 62 , 15–25. [ Google Scholar ] [ CrossRef ]
• Gupta, S.; Leszkiewicz, A.; Kumar, V.; Bijmolt, T.; Potapov, D. Digital analytics: Modeling for insights and new methods. J. Interact. Mark. 2020 , 51 , 26–43. [ Google Scholar ] [ CrossRef ]
• Ayodele, T.O. Types of Machine Learning Algorithms. In New Advances in Machine Learning ; Intech Open: Rijeka, Croatia, 2010; Volume 3, pp. 19–48. [ Google Scholar ]
• Ma, L.; Sun, B. Machine learning and AI in marketing—Connecting computing power to human insights. Int. J. Res. Mark. 2020 , 37 , 481–504. [ Google Scholar ] [ CrossRef ]
• Littman, M.L.; Ajunwa, I.; Berger, G.; Boutilier, C.; Currie, M.; Doshi-Velez, F.; Hadfield, G.; Horowitz, M.C.; Isbell, C.; Kitano, H.; et al. Gathering strength, gathering storms: The one hundred year study on artificial intelligence (AI100) 2021 study panel report. arXiv 2022 , arXiv:2210.15767. [ Google Scholar ]
• Kshetri, N.; Dwivedi, Y.K.; Davenport, T.H.; Panteli, N. Generative artificial intelligence in marketing: Applications, opportunities, challenges, and research agenda. Int. J. Inf. Manag. 2024 , 75 , 102716. [ Google Scholar ] [ CrossRef ]
• Sohrabpour, V.; Oghazi, P.; Toorajipour, R.; Nazarpour, A. Export sales forecasting using artificial intelligence. Technol. Forecast. Soc. Chang. 2021 , 163 , 120480. [ Google Scholar ] [ CrossRef ]
• Stadlmann, C.; Zehetner, A. Human Intelligence Versus Artificial Intelligence: A Comparison of Traditional and AI-Based Methods for Prospect Generation. In Marketing and Smart Technologies: Proceedings of ICMarkTech 2020 ; Springer: Singapore, 2021; pp. 11–22. [ Google Scholar ]
• Colias, J.V.; Park, S.; Horn, E. Optimizing B2B product offers with machine learning, mixed logit, and nonlinear programming. J. Mark. Anal. 2021 , 9 , 157–172. [ Google Scholar ] [ CrossRef ]
• Singh, S.K.; Marinova, D.; Singh, J. Business-to-business e-negotiations and influence tactics. J. Mark. 2020 , 84 , 47–68. [ Google Scholar ] [ CrossRef ]
• Leyer, M.; Schneider, S. Decision augmentation and automation with artificial intelligence: Threat or opportunity for managers? Bus. Horiz. 2021 , 64 , 711–724. [ Google Scholar ] [ CrossRef ]
• Singh, J.; Flaherty, K.; Sohi, R.S.; Deeter-Schmelz, D.; Habel, J.; Le Meunier-FitzHugh, K.; Malshe, A.; Mullins, R.; Onyemah, V. Sales profession and professionals in the age of digitization and artificial intelligence technologies: Concepts, priorities, and questions. J. Pers. Sell. Sales Manag. 2019 , 39 , 2–22. [ Google Scholar ] [ CrossRef ]
• Grönroos, C.; Voima, P. Critical service logic: Making sense of value creation and co-creation. J. Acad. Mark. Sci. 2013 , 41 , 133–150. [ Google Scholar ] [ CrossRef ]
• Russo-Spena, T.; Cristina, M.; Marzullo, M. Practising value innovation through artificial intelligence: The IBM Watson case. J. Creat. Value 2019 , 5 , 11–24. [ Google Scholar ] [ CrossRef ]
• Grönroos, C. Service-informed marketing reform. J. Serv. Mark. 2024 , 38 , 32–43. [ Google Scholar ] [ CrossRef ]
• Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Moher, D. Updating guidance for reporting systematic reviews: Development of the PRISMA 2020 statement. J. Clin. Epidemiol. 2021 , 134 , 103–112. [ Google Scholar ] [ CrossRef ]
• Aquilani, B.; Piccarozzi, M.; Abbate, T.; Codini, A. The role of open innovation and value co-creation in the challenging transition from industry 4.0 to society 5.0: Toward a theoretical framework. Sustainability 2020 , 12 , 8943. [ Google Scholar ] [ CrossRef ]
• Barile, S.; Bassano, C.; Piciocchi, P.; Saviano, M.; Spohrer, J.C. Empowering value co-creation in the digital age. J. Bus. Ind. Mark. 2024 , 39 , 1130–1143. [ Google Scholar ] [ CrossRef ]
• Kot, M.; Leszczyński, G. AI-activated value co-creation. An exploratory study of conversational agents. Ind. Mark. Manag. 2022 , 107 , 287–299. [ Google Scholar ] [ CrossRef ]
• Li, S.; Peng, G.; Xing, F.; Zhang, J.; Zhang, B. Value co-creation in industrial AI: The interactive role of B2B supplier, customer and technology provider. Ind. Mark. Manag. 2021 , 98 , 105–114. [ Google Scholar ] [ CrossRef ]
• Petrescu, M.; Krishen, A.S.; Kachen, S.; Gironda, J.T. AI-based innovation in B2B marketing: An interdisciplinary framework incorporating academic and practitioner perspectives. Ind. Mark. Manag. 2022 , 103 , 61–72. [ Google Scholar ] [ CrossRef ]
• Raghupathi, H.; Choudhury, D.; Sfeir, C.J. Application of artificial intelligence in market knowledge and B2B marketing Co-creation. In Machine Learning for Business Analytics ; Productivity Press: New York, NY, USA, 2022; pp. 107–115. [ Google Scholar ]
• Sjödin, D.; Parida, V.; Palmié, M.; Wincent, J. How AI capabilities enable business model innovation: Scaling AI through co-evolutionary processes and feedback loops. J. Bus. Res. 2021 , 134 , 574–587. [ Google Scholar ] [ CrossRef ]
• Wei, R.; Pardo, C. Looking for resource density in the platform supply network. J. Purch. Supply Manag. 2024 , 30 , 100938. [ Google Scholar ] [ CrossRef ]
• Füller, J. Refining virtual co-creation from a consumer perspective. Calif. Manag. Rev. 2010 , 52 , 98–122. [ Google Scholar ] [ CrossRef ]
• Čaić, M.; Odekerken-Schröder, G.; Mahr, D. Service robots: Value co-creation and co-destruction in elderly care networks. J. Serv. Manag. 2018 , 29 , 178–205. [ Google Scholar ] [ CrossRef ]
• Page, M.J.; McKenzie, J.E.; Bossuyt, P.M.; Boutron, I.; Hoffmann, T.C.; Mulrow, C.D.; Shamseer, L.; Tetzlaff, J.M.; Akl, E.A.; Brennan, S.E.; et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ 2021 , 372 , n71. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Fehrenbach, D.; Herrando, C.; Martín-De Hoyos, M.J. Co-Creation with AI in B2B Markets: A Systematic Literature Review. Sustainability 2024 , 16 , 8009. https://doi.org/10.3390/su16188009

Fehrenbach D, Herrando C, Martín-De Hoyos MJ. Co-Creation with AI in B2B Markets: A Systematic Literature Review. Sustainability . 2024; 16(18):8009. https://doi.org/10.3390/su16188009

Fehrenbach, David, Carolina Herrando, and María José Martín-De Hoyos. 2024. "Co-Creation with AI in B2B Markets: A Systematic Literature Review" Sustainability 16, no. 18: 8009. https://doi.org/10.3390/su16188009

Article Metrics

Supplementary material.

ZIP-Document (ZIP, 129 KiB)

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

A Systematic Literature Review on Flexible Strategies and Performance Indicators for Supply Chain Resilience

• REVIEW PAPERS
• Open access
• Published: 10 September 2024

Cite this article

You have full access to this open access article

• Ananna Paul 1 &
• Suvash C. Saha   ORCID: orcid.org/0000-0002-9962-8919 1  

Supply chain resilience is a widely useful concept for managing risk and disruption. Designing strategies for preparedness, response, and recovery can help businesses to mitigate risks and disruptions. Among them, flexible strategies can effectively improve supply chain resilience. In the literature, several studies have considered different types of flexible strategies and investigated their impacts on supply chain resilience. However, a systematic literature review (SLR) paper on this topic can further help to understand the scientific progress, research gaps, and avenues for future research. Hence, this study aims to explore how the literature has contributed to the area of flexible strategies and the impact on supply chain resilience performance. To achieve our objective, we apply an SLR methodology to identify themes such as research areas and key findings, contexts and industry sectors, methodologies, and key strategies and performance indicators in the connection between flexible strategies and supply chain resilience. The findings show that many studies connect flexible strategies to supply chain resilience. However, research gaps exist in analysing relationships between flexible strategies and performance, conducting comparative studies, developing dynamic resilience plans, applying flexible strategies, conducting theoretically grounded empirical studies, and applying multiple analytical tools to develop decision-making models for supply chain resilience. Finally, this study suggests several future research opportunities to advance the research on the topic. The findings can be a benchmark for researchers who are interested in conducting research in the area of flexible strategies and supply chain resilience.

Similar content being viewed by others

Flexibility in Enhancing Supply Chain Resilience: Developing a Resilience Capability Portfolio in the Event of Severe Disruption

Analysis of Theoretical Aspects of Supply Chain Resilience Determinants and Strategies

Supply Chain Resilience: A Literature Review and Gap Analysis

Explore related subjects.

• Artificial Intelligence

Avoid common mistakes on your manuscript.

Introduction

Supply chain management is critical in supplying, producing, and distributing goods and services to consumers and communities. However, any risks, disruptions, and uncertainties at any supply chain stage could make the whole operation vulnerable (Paul et al., 2017 ). The ultimate consequences could include delivery and supply delays, demand unfulfilment, and loss of revenue and business goodwill (Rahman et al., 2022 ). Hence, developing a resilient supply chain to absorb disruptions and keep operations going is important.

Supply chain resilience is defined by the preparedness and ability to respond to recover from and deal with disruptions (Ponis & Koronis, 2012 ; Ribeiro & Barbosa-Povoa, 2018 ; Tukamuhabwa et al., 2015 ). Preparedness means taking proactive actions, such as assessing risk and disruption factors and planning for strategies and resources (Paul & Chowdhury, 2020 ; Rahman et al., 2022 ). Meanwhile, response and recovery are reactive actions. Response includes the ability to quickly and accurately sense the impacts of a disruption and respond to mitigate such impacts (Scholten et al., 2020 ). For example, swiftly accessing alternative suppliers and emergency sources in case of a supply disruption can help mitigate the consequences. Recovery includes the planning and replanning for a future period after the occurrence of a disruption to bring the plan to the normal stage (Paul et al., 2017 ). For example, utilising alternative suppliers and resources to revise the supply chain plan for a certain period after the occurrence of supply disruption mitigates the impacts and helps restore the original plan. Recovery requires a sophisticated plan that utilises appropriate mitigation strategies. Preparedness, response, and recovery are well connected, as response and recovery can be difficult without good preparedness.

The flexible supply chain is a popular concept for managing variability in supply chains (Dhillon et al., 2023 ; Varma et al., 2024 ; Wadhwa et al., 2008 ). Variability includes changes in demand, processing time, lead time, and so on. Supply chain flexible strategies include flexibility in design, supply, manufacturing, transportation, and logistics. It also connects the flexibility of supply chain partners, such as flexible suppliers, manufacturing plants, logistics, and transportation.

Supply chain variabilities are well connected to risks and uncertainties. Flexible strategies can help manage supply chain uncertainties, risks, and variabilities (Tang & Tomlin, 2008 ; Yi et al., 2011 ). For example, utilising multiple suppliers and safety inventory can be useful to mitigate supply risks and uncertainties. The literature shows that flexible strategies effectively build resilient supply chains and can help manage risk and uncertainty and improve supply chain resilience by preparing well and/or enhancing capabilities to respond and recover (Chowdhury et al., 2024 ; Chunsheng et al., 2020 ; Dwivedi et al., 2023 ; Kamalahmadi et al., 2022 ; Kazancoglu et al., 2022 ; Mackay et al., 2020 ; Piprani et al., 2022 ; Rajesh, 2021 ; Sharma et al., 2023 ; Tang & Tomlin, 2008 ).

In the literature, several studies explore the usefulness of flexible strategies to improve supply chain resilience. Moreover, a few review papers exist in the literature which analysed supply chain resilience with drivers, vulnerabilities, risks and impacts, and robustness (Shishodia et al., 2023 ), supply chain resilience strategies (Rahman et al., 2022 ), framework, barriers, and strategies for supply chain resilience (Shashi et al., 2020 ), and recovery ability for supply chain resilience (Mandal, 2014 ). However, a systematic literature review (SLR) and content analysis of previously published papers on flexible strategies and supply chain resilience are non-existent. An SLR and content analysis are very helpful for researchers to understand the progress and development and plan for future research. Accordingly, this review article develops the following research questions (RQs).

RQ1: What contributions have been made in the connection between flexible strategies and supply chain resilience?

RQ2: What are the emerging research opportunities in the area of flexible strategies and supply chain resilience?

To answer the above RQs, this paper investigates flexible strategies and performance indicators for supply chain resilience by conducting an SLR and analysing articles under different themes, such as research area and key findings, context and industry sectors, methodologies, key dimensions, strategies, and performance indicators. Finally, this study also analyses the research gaps and suggests a number of meaningful future research opportunities.

The rest of the paper is organised as follows. Section “ Review Methodologies ” describes the review methodologies. Section “ Analysing Reviewed Articles ” analyses previous articles on flexible strategies for supply chain resilience. Research gaps and future research directions are provided in Sect. “ Research gaps and Future Research Opportunities ”. Finally, Sect. “ Conclusions ” provides conclusions and limitations of the study.

Review Methodologies

In this paper, an SLR process is utilised to analyse the content of the reviewed articles (Tranfield et al., 2003 ). An SLR provides a more accurate literature search and in-depth content analysis than other methods, such as generic and bibliometric reviews. It also helps in the systematic and critical analysis of the content of previously published articles.

In this paper, Scopus was the primary database to identify articles on flexible strategies and performance indicators for supply chain resilience. The following search criteria were used:

Keywords: flexible strategy, supply chain, resilience, performance.

Language: English.

Source type: Journal.

Search timeline: up to 2023.

The initial search using keywords identified a total of 138 articles. After filtering for language and source type, 46 articles were removed and 92 articles remained.

Next, we read the article’s title, abstract, and content and applied inclusion and exclusion criteria to finalise the articles. The inclusion criteria were: (i) articles focused on flexible strategies for different aspects of supply chain resilience, and (ii) both the keywords “flexible” or “flexibility” and “resilience” appeared in the main text. The exclusion criteria were if one or more keywords mentioned in the implications and/or in the reference list were available, but the article did not focus on the flexible strategies in supply chain resilience. After applying inclusion and exclusion criteria, 30 articles were removed and 62 articles remained.

Finally, other databases, such as Google Scholar and Web of Science, were used to search the articles. The reference check was also conducted to ensure that all relevant articles were included in the analysis. These checks did not include any new articles. A total of 62 articles were finalised for the analysis in this review. The review methodology is presented in Fig.  1 .

Review methodology

Analysing Reviewed Articles

This section analyses the finalised articles in key different dimensions, including subject areas, key contributions and findings, contexts of the studies, methodologies used, key sectors (manufacturing or service), different flexible strategies for supply chain resilience, and performance indicators for supply chain resilience.

Key Subject Areas

We analysed the subject areas for the 62 articles. As flexibility and supply chain resilience is a multidisciplinary research area, the articles were expected to contribute to several subject areas. Thus, we observed the common subject areas to be business, management and accounting, engineering, decision sciences, computer science, and social sciences. The key subject areas for the reviewed articles are presented in Fig.  2 .

Key subject areas of the reviewed articles

Key Contributions and Findings of Previous Studies

Over the last few years, many studies have contributed in the area of flexible strategies and supply chain resilience. We observed that eight articles used a literature review approach, while the remaining 54 were technical studies. This section delves into the details of previous contributions and findings.

Previously Published Review Articles

From the systematic review, we identified eight review articles in the area of supply chain resilience. The main contributions and findings of those review articles are summarised in Table  1 . The previous review articles analysed the literature in different supply chain resilience dimensions, including drivers, vulnerabilities, risks and impacts, and robustness (Shishodia et al., 2023 ), resilience strategies (Rahman et al., 2022 ), framework, barriers, and strategies (Shashi et al., 2020 ), and recovery (Mandal, 2014 ). Significant research gaps exist in reviewing the literature on how different flexible strategies are applied to improve supply chain resilience and the potential future research directions. This paper fills these gaps.

Table 1 shows that five articles used a systematic literature review approach, while others used bibliometric analysis and literature review along with expert opinions and conceptual modelling/framework.

Contributions and Findings of Technical Studies

We analysed the contributions and main findings of 54 technical studies and observed the following main areas of study.

Analysing resilience strategies using varieties of methodologies (Kummer et al., 2022 ; Nagariya et al., 2023 ; Purvis et al., 2016 ; Wang et al., 2016 ),

Analysing impacts of strategies on performance (Alvarenga et al., 2023 ; Hamidu et al., 2024 ; Isti’anah et al., 2021 ; Lin et al., 2023 ; Nguyen et al., 2022 ; Xu et al., 2023 ),

Exploring capabilities for supply chain resilience (Faruquee et al., 2023 ; Shweta et al., 2023 ; Um & Han, 2021 ; Zhou et al., 2022 ),

Evaluating critical factors, enablers, and antecedents for supply chain resilience (Das et al., 2022 ; Pu et al., 2023a , 2023b ; Sangari & Dashtpeyma, 2019 ),

Analysing impacts of disruption on supply chains (Ivanov, 2022 ),

Designing/re-designing supply chain networks to improve resilience (Alikhani et al., 2021 ; Carvalho et al., 2012 ; Fattahi et al., 2020 ), and

Selecting suppliers for supply chain resilience (Suryadi & Rau, 2023 ).

The main contributions and findings are summarised in Table  2 .

This section analyses different contexts used in the literature. The contexts include both industry sectors and regions of data collection and applications. We observed that 38 studies used a specific industry context, while 41 papers used a country/regional context in their studies.

Industry Context

Our analysis of the articles shows that both single and multiple sectors have been considered in previous studies. Fourteen studies considered multiple industry sectors, and 24 studies considered a single industry sector. The single industry sectors include maritime (Isti’anah et al., 2021 ; Praharsi et al., 2021 ; Zavitsas et al., 2018 ), food (Li et al., 2022 ; Purvis et al., 2016 ), healthcare (Vimal 2022a ; Shweta et al., 2023 ), and textile and apparel sectors (Fahimnia et al., 2018 ; Nagariya et al., 2023 ). The other single industry sectors are container handling, delivery services, e-commerce of clothing and grocery, industrialised construction, copper industry, retail, ICT industry, automotive, sportswear, and electronic sectors.

Previous studies also considered multiple industry sectors. For example, Alvarenga et al. ( 2023 ) considered multiple sectors, including chemical and petroleum, food and beverage, and machinery sectors. Maharjan and Kato ( 2023 ) considered multiple sectors, including manufacturing, assembly, agricultural machinery parts, apparel business, and trading companies. Zhou et al. ( 2022 ) considered multiple sectors, including electronics and appliances, metals, machinery and engineering, construction materials, textiles, and clothing. Gölgeci and Kuivalainen ( 2020 ) considered multiple sectors, including chemical and pharmaceutical, food and beverage, construction equipment, retail, textile, clothing, and apparel.

Country/Regional Context

Forty-one studies considered a specific country/regional context. Several studies considered global or multiple regions. For example, Alvarenga et al. ( 2023 ) considered a global context, including North America, Europe, Asia, Africa, South America, and Oceania countries. Faruquee et al. ( 2023 ) collected data from the USA and the UK. Das et al. ( 2022 ) collected data from countries in Asia, Europe, and the Americas.

The majority of the studies considered a single country/regional context. Among them, seven studies considered India (Altay et al., 2018 ; Vimal et al., 2022a , 2022b ; Nagariya et al., 2023 ; Rajesh, 2016 ; Shweta et al., 2023 ; Suryawanshi et al., 2021 ), four studies considered Iran (Alikhani et al., 2021 ; Fattahi et al., 2020 ; Moosavi & Hosseini, 2021 ; Suryadi & Rau, 2023 ), three studies considered China (Pu et al., 2023a , 2023b ; Zhu & Wu, 2022 ) and three studies considered Ghana (Hamidu et al., 2023a , 2023b , 2024 ) in the country context.

The details of industry sectors and country/regional contexts are presented in Table  3 .

Methodologies Used

Both qualitative and quantitative methods have been applied to analyse strategies and performance indicators in supply chain resilience. Qualitative methods include literature reviews (see Table  1 ), interviews (Chen et al., 2019 ; Lin et al., 2023 ; Maharjan & Kato, 2023 ; Purvis et al., 2016 ; Silva et al., 2023 ), conceptual modelling (Mackay et al., 2020 ), DMAIC framework (Praharsi et al., 2021 ), and FEWSION for the community resilience process (Ryan et al., 2021 ).

Quantitative methods include structural equation modelling (Alvarenga et al., 2023 ; Gölgeci & Kuivalainen, 2020 ; Pu et al., 2023a , 2023b ; Purvis et al., 2016 ; Um & Han, 2021 ), mathematical programming (Alikhani et al., 2021 ; Mao et al., 2020 ; Mikhail et al., 2019 ; Suryawanshi et al., 2021 ; Zavitsas et al., 2018 ), MCDM methods (Das et al., 2022 ; Shweta et al., 2023 ), simulation (Ivanov, 2022 ; Kummer et al., 2022 ; Moosavi & Hosseini, 2021 ; Tan et al., 2020 ), partial least squares (Altay et al., 2018 ), and regression analysis (Donadoni et al., 2018 ; Trabucco & De Giovanni, 2021 ).

Table 4 provides a summary of the methods used.

Several studies integrated multiple methods such as PLS-SEM (Ekanayake et al., 2021 ; Hamidu et al., 2023a , 2023b ; Nguyen et al., 2022 ), Fuzzy DEMATEL and best–worst method (Shweta et al., 2023 ), analytic hierarchy process and linear programming (Suryadi & Rau, 2023 ), analysis of variance and polynomial regression (Faruquee et al., 2023 ), best–worst method and fuzzy TOPSIS (Vima et al., 2022b ), Delphi method and best–worst method (Nagariya et al., 2023 ), AHP and DEMATEL (Das et al., 2022 ), mixed-integer linear programming and Monte Carlo simulation (Suryawanshi et al., 2021 ), interpretive structural modelling and fuzzy analytical network process (Sangari & Dashtpeyma, 2019 ), and discrete-event simulation and regression analysis (Macdonald et al., 2018 ).

Case studies were combined with other methods in several studies. For example, Purvis et al. ( 2016 ) conducted a case study in the UK’s food and drink sector to analyse supply chain resilience strategies. Maharjan and Kato ( 2023 ) included a case study from Japan’s manufacturing, agricultural, apparel, and trading companies to identify the current resilience status. Lin et al. ( 2023 ) provided a case study from delivery services in the UK to investigate supply chain resilience in responding to disruptions. Silva et al. ( 2023 ) discussed the findings from coffee-producing firms in Brazil to explore the relationship between sustainability and resilience. Carvalho et al. ( 2012 ) explained a case study from the automotive sector in Portugal to analyse the scenario-based design for supply chain resilience.

Key Sectors (Manufacturing or Service)

The reviewed articles show that previous studies considered both the manufacturing and service sectors as the key application areas. Figure  3 provides a summary of key sectors. Figure  3 shows that 49 out of 62 articles considered a sector, with most (35 articles) focusing on the manufacturing sector. Nine studies considered both manufacturing and service sectors, and only five considered the service sector. Sect. “ Contexts ” shows the specific contexts previous studies considered.

Summary of key sectors

Different Flexible Strategies for Supply Chain Resilience

We observed that numerous strategies have been used for supply chain resilience. We have categorised them as supply, manufacturing/operational strategies, transportation and distribution strategies, and supply chain levels.

The most common supply strategies were multiple suppliers/sourcing, improving collaboration with suppliers/partners, backup/alternative suppliers, supplier development, and building trust with suppliers. These strategies help to improve supply chain flexibility and supply chain resilience. For example, multiple suppliers/sourcing includes having multiple suppliers or sources of materials for mitigating risks and disruptions (Ekanayake et al., 2021 ; Mikhail et al., 2019 ; Praharsi et al., 2021 ; Rahman et al., 2022 ). It improves supply flexibility, further allowing for the diversification of the supply base. Similarly, another popular strategy in supply chain resilience is improving collaboration with suppliers/partners. It enhances communication processes, information, and resource sharing and working together to deal with risks and uncertainties in their supply chains (Chen et al., 2019 ; Faruquee et al., 2023 ; Sangari & Dashtpeyma, 2019 ; Silva et al., 2023 ).

Flexible transportation/distribution channels were the most widely applied transportation and distribution strategy. This includes flexible routes, flexible transportation capacities, and multiple distribution channels, spanning online, and physical distributions (Faruquee et al., 2023 ; Hohenstein et al., 2015 ; Massari & Giannoccaro, 2021 ; Suryadi & Rau, 2023 ). This strategy is very effective in improving resilience in transportation and distribution, particularly, and the supply chain, in general. The other flexible strategies included alternative shipment/transportation modes and backup distribution centres.

Strategies such as utilising extra capacity, resource allocation/reallocation, managing the quality of products, and using safety stock were widely applied in manufacturing/operations. Extra capacities in manufacturing plants improve production flexibilities and help mitigate supply and demand uncertainties (Altay et al., 2018 ; Fattahi et al., 2020 ; Rahman et al., 2022 ). Other strategies, such as resource allocation/reallocation, managing the quality of products, and using safety stock, are also effective in dealing with risk and disruption in supply chains and improving business reputation.

In supply chain-level strategies, the common strategies were adopting digital technologies, knowledge/information sharing, business continuity/contingency planning, and multi-skilled labour. The recent studies highlighted that adopting digital technologies at the supply chain level could improve communication, tracking, data analysis, and information processing (Alvarenga et al., 2023 ; Nagariya et al., 2023 ; Nguyen et al., 2022 ; Trabucco & De Giovanni, 2021 ). All these contribute to improving supply chain performance and resilience. Similarly, the literature proved that supply chain-level strategies help improve operational, financial, and reputational performance by enhancing supply chain resilience.

The full list of flexible strategies for supply chain resilience and their categories are presented in Table  5 .

Performance Indicators for Supply Chain Resilience

Supply chain resilience studies have used several performance indicators to measure performance, including financial, operational, reputational, and supply chain performance.

In supply chain resilience, financial performance indicators include cost efficiency, return on investment, market share, sales growth, profit, and return on sales and assets. Cost efficiency is the most significant performance indicator (Alikhani et al., 2021 ; Donadoni et al., 2018 ; Fattahi et al., 2020 ; Nagariya et al., 2023 ). Organisations set their desired price while maintaining the quality of products or services and improving customer satisfaction. Another significant performance indicator is profit (Hohenstein et al., 2015 ; Mikhail et al., 2019 ; Moosavi & Hosseini, 2021 ; Shashi et al., 2020 ). Profit is a goal for organisations to enhance overall performance. Return on investment (Gölgeci & Kuivalainen, 2020 ; Juan & Li, 2023 ; Trabucco & De Giovanni, 2021 ) and market share (Hohenstein et al., 2015 ; Juan & Li, 2023 ; Pu et al., 2023a , 2023b ; Zhou et al., 2022 ) are also used to evaluate organisational performance.

The most common operational performance indicators in supply chain resilience are on-time delivery, demand fulfilment, and enhanced operational efficiency and delivery time. On-time delivery (Rajesh, 2021 ; Shweta et al., 2023 ; Trabucco & De Giovanni, 2021 ) improves the efficiency of business processes and fulfils customer commitment. Customer order processing depends on demand fulfilment. Demand fulfilment (Moosavi & Hosseini, 2021 ; Rajesh, 2021 ; Tan et al., 2020 ) positively impacts the firm’s performance in the competitive market. Enhanced operational efficiency (Praharsi et al., 2021 ) and delivery time (Mao et al., 2020 ) increases customer satisfaction and improves business performance.

In supply chain resilience, reputational performance indicators include customer satisfaction, service-level improvement, customer loyalty, meeting customer satisfaction/request, quality performance, and corporate image. Service-level improvement (Hohenstein et al., 2015 ; Isti’anah et al., 2021 ; Praharsi et al., 2021 ) is one of the most important performance indicators. Maximising service level increases the overall performance of organisations. Customer satisfaction is the second most crucial reputational performance indicator (Gölgeci & Kuivalainen, 2020 ; Zhu & Wu, 2022 ). Customer satisfaction with a product/service enhances organisational reputation.

Resilience performance also depends on supply chain performance indicators such as restoring material flow, quickly moving to a desirable state, lead time reduction, supply chain visibility, recovery time, and response time. Among these indicators, lead time reduction (Donadoni et al., 2018 ; Ivanov, 2022 ; Nagariya et al., 2023 ), recovery time (Altay et al., 2018 ; Singh & Singh, 2019 ), and response time (Altay et al., 2018 ; Faruquee et al., 2023 ) are the significant performance indicators. Lead time reduction minimises the time duration of the product or service process. Reduction of recovery time and response time enhances the efficiency of organisational performance.

Table 6 summarises the list of performance indicators in supply chain resilience.

Mapping of Strategies and Performance Indicators

The literature review shows that flexible strategies are useful in improving supply chain performance. This section explains the mapping between different flexible strategies and performance indications and discusses the strategies that effectively improve or influence performance.

From the literature analysis, we have observed that “improving collaboration with suppliers/partners” influences all major resilience performances, including cost efficiency, return on investment, market share, profit, customer satisfaction, service-level improvement, on-time delivery, demand fulfilment, lead time reduction, recovery time, and response time (Chen et al., 2019 ; Donadoni et al., 2018 ; Faruquee et al., 2023 ; Hohenstein et al., 2015 ; Juan & Li, 2023 ; Ladeira et al., 2021 ; Moosavi & Hosseini, 2021 ; Praharsi et al., 2021 ; Shashi et al., 2020 ; Shweta et al., 2023 ; Suryadi & Rau, 2023 ; Zhou et al., 2022 ; Zhu & Wu, 2022 ).

Similarly, multiple suppliers/sourcing, backup/alternative suppliers, flexible transportation/distribution channels, utilising extra capacity, adopting digital technologies, knowledge/information sharing, and multi-skilled labour are effective in improving resilience performance in supply chain management.

Table 7 provides the mapping between different strategies and their influence on resilience performance indicators.

Research Gaps and Future Research Opportunities

We have observed the following research gaps from the literature review and have suggested future research opportunities.

Relationship Between Strategies and Performance In Supply Chain Resilience

Very few studies analysed the relationship between strategies and performance in supply chain resilience. While a few studies did, they only considered a limited number of strategies and performance indicators (Donadoni et al., 2018 ; Faruquee et al., 2023 ; Gölgeci & Kuivalainen, 2020 ; Isti’anah et al., 2021 ; Juan & Li, 2023 ; Mikhail et al., 2019 ; Nagariya et al., 2023 ; Praharsi et al., 2021 ; Pu et al., 2023a , 2023b ; Shishodia et al., 2023 ; Suryadi & Rau, 2023 ; Trabucco & De Giovanni, 2021 ; Wang et al., 2016 ; Zhou et al., 2022 ). For example, Shishodia et al. ( 2023 ) considered managing product quality, multiple sourcing, demand aggregation, flexible transportation systems, backup suppliers, fortification of partners, and risk sharing as strategies and cost efficiency and lead time reduction as performance indicators. Similar analyses were found in other studies. This makes the literature less comprehensive in analysing the thorough impacts of different strategies, individually and combined, on supply chain resilience performance.

To close this gap and improve the literature, we propose studies to consider the holistic list of strategies and performance indicators (as shown in Sects. “ Different Flexible Strategies for Supply Chain Resilience ” and “ Performance Indicators for Supply Chain Resilience ”) and analyse how major strategies influence major performance indicators in supply chain resilience.

Comparative Studies

There is a significant research gap in the literature regarding comparative studies. Very few studies considered both the manufacturing and service sectors and multiple industry sectors (Alikhani et al., 2021 ; Alvarenga et al., 2023 ; Nguyen et al., 2022 ; Singh & Singh, 2019 ; Zhu & Wu, 2022 ). However, the literature has research gaps for comparative studies between developed and developing economies, large and small and medium enterprises, and their longitudinal analyses. Hence, there is a gap in generalising the findings.

To contribute to this area, we suggest conducting the following studies.

Comparative studies of flexible strategies and/or performance indicators for developed and developing economies.

Comparative studies of flexible strategies and/or performance indicators between large, small, and medium enterprises.

Analysis of findings over time for different economies and enterprises.

Developing models for generalising the findings for different economies and enterprises.

Service Sectors

Service sectors get less attention in the literature even though they are dominant in many countries. Only a few studies considered service sectors (Fattahi et al., 2020 ; Isti’anah et al., 2021 ; Lin et al., 2023 ; Suryawanshi et al., 2021 ). Hence, the literature provided few findings on supply chain resilience and their strategies and performance indicators in service sectors.

We suggest conducting more studies for service sectors, including the analysis of different flexible strategies used by different service sectors and how they influence service performance to improve supply chain resilience.

Dynamic Plans for Supply Chain Resilience

Many studies have developed models and frameworks for analysis strategies and performance indicators in supply chain resilience (Juan & Li, 2023 ; Shishodia et al., 2023 ; Suryadi & Rau, 2023 ). Still, there is a gap in the literature on developing dynamic resilience plans for the changed environment. As risks and disruptions change over time, it is important to change the plan and its flexible strategies to ensure supply chains can deal with the impacts of the changing environment and improve resilience. These types of studies on flexible strategies and supply chain resilience are non-existent in the current literature.

To contribute to this area, we suggest developing the following studies.

Developing dynamic and flexible strategies for supply chain resilience for different disruption scenarios.

Analysing the impacts of dynamic strategies on resilience performance over time.

Developing dynamic supply chain resilience models for preparedness, response, and recovery considering different flexible strategies.

Comparing the findings for different flexible strategies to obtain the most suitable plans for dynamic supply chain resilience plans.

Theoretically Grounded Studies

Few studies developed theoretically grounded empirical models (Alvarenga et al., 2023 ; Gölgeci & Kuivalainen, 2020 ; Juan & Li, 2023 ; Ladeira et al., 2021 ; Pu et al., 2023a , 2023b ; Singh & Singh, 2019 ; Um & Han, 2021 ; Zhou et al., 2022 ; Zhu & Wu, 2022 ). However, there is a gap in the literature in relation to applying emergent theories such as the awareness–motivation–capability framework.

In the future, we propose considering theories from multiple disciplines to develop and test models to analyse the impacts of flexible strategies on supply chain resilience, including in dynamic and changed environments.

Analytical Studies

According to the literature review, different studies applied different analytical tools, such as mathematical programming and simulation approaches (Alikhani et al., 2021 ; Fattahi et al., 2020 ; Ivanov, 2022 ; Kummer et al., 2022 ; Mikhail et al., 2019 ; Pu et al., 2023a , 2023b ; Zavitsas et al., 2018 ). Integrating multiple analytical tools improves the quality of findings and the decision-making process in supply chain management. The flexible strategies and supply chain resilience literature has a gap in relation to integrating multiple analytical tools for analysing strategies and performance indicators.

In future, we propose applying multiple analytical tools to develop decision-making models for practitioners. We also suggest dividing the studies into different sections, applying analytical tools and connecting them again to improve the quality of findings.

Conclusions

The main objective of this study was to critically review the existing studies that considered flexible strategies for supply chain resilience. To fulfil this objective, we applied an SLR technique and analysed 62 related studies in the domain of contributions and findings, research contexts and business sectors, methodologies, different flexible strategies and performance indicators, and relationship mapping between flexible strategies and performance indicators.

The main contributions of this study are: (i) conducting an SLR in flexible strategies for supply chain resilience, which has not yet been explored in the literature, (ii) critically analysing the existing studies and presenting the findings, and (iii) proposing future research directions based on the identified research gaps.

The main findings indicated that more research is needed to analyse holistic relationships between flexible strategies and supply chain performance. Moreover, the service sector should be studied more, as it has been widely ignored in the literature thus far. Future research should also consider developing dynamic resilience plans using flexible strategies. Finally, more theoretically grounded and analytical studies should be conducted in the area of flexible strategies and supply chain resilience.

However, this review article has some limitations. First, we consider only journal articles published until 2023 and written in English. Second, the scope of the study was limited to flexible strategies and performance indicators used in the area of supply chain resilience. In the future, the timeline of published articles and the scope of the study can be further broadened. As this SLR paper provided a critical review, a summary of existing studies, and significant future research directions, the findings of the study can be used as a benchmark for future research in flexible strategies for supply chain resilience.

Key Questions

What contributions have been made in the connection between flexible strategies and supply chain resilience?

What are the emerging research opportunities in the area of flexible strategies and supply chain resilience?

There is no funding for this article.

Alikhani, R., Torabi, S. A., & Altay, N. (2021). Retail supply chain network design with concurrent resilience capabilities. International Journal of Production Economics, 234 , 108042. https://doi.org/10.1016/j.ijpe.2021.108042

Article   Google Scholar  

Altay, N., Gunasekaran, A., Dubey, R., & Childe, S. J. (2018). Agility and resilience as antecedents of supply chain performance under moderating effects of organizational culture within the humanitarian setting: A dynamic capability view. Production Planning and Control, 29 (14), 1158–1174. https://doi.org/10.1080/09537287.2018.1542174

Alvarenga, M. Z., de Oliveira, M. P. V., & de Oliveira, T. A. G. F. (2023). The impact of using digital technologies on supply chain resilience and robustness: The role of memory under the covid-19 outbreak. Supply Chain Management, 28 (5), 825–842. https://doi.org/10.1108/SCM-06-2022-0217

Carvalho, H., Barroso, A. P., MacHado, V. H., Azevedo, S., & Cruz-Machado, V. (2012). Supply chain redesign for resilience using simulation. Computers and Industrial Engineering, 62 (1), 329–341. https://doi.org/10.1016/j.cie.2011.10.003

Chen, H. Y., Das, A., & Ivanov, D. (2019). Building resilience and managing post-disruption supply chain recovery: Lessons from the information and communication technology industry. International Journal of Information Management, 49 , 330–342. https://doi.org/10.1016/j.ijinfomgt.2019.06.002

Chowdhury, M. M. H., Chowdhury, P., Quaddus, M., Rahman, K. W., & Shahriar, S. (2024). Flexibility in enhancing supply chain resilience: developing a resilience capability portfolio in the event of severe disruption. Global Journal of Flexible Systems Management , 25 (2), 395–417.

Chunsheng, L., Wong, C. W. Y., Yang, C. C., Shang, K. C., & Lirn, T. (2020). Value of supply chain resilience: Roles of culture, flexibility, and integration. International Journal of Physical Distribution & Logistics Management, 50 (1), 80–100. https://doi.org/10.1108/IJPDLM-02-2019-0041

Das, D., Datta, A., Kumar, P., Kazancoglu, Y., & Ram, M. (2022). Building supply chain resilience in the era of COVID-19: An AHP-DEMATEL approach. Operations Management Research, 15 (1–2), 249–267. https://doi.org/10.1007/s12063-021-00200-4

Dhillon, M. K., Rafi-ul-Shan, P. M., Amar, H., Sher, F., & Ahmed, S. (2023). Flexible green supply chain management in emerging economies: a systematic literature review. Global Journal of Flexible Systems Management, 24 (1), 1–28

Donadoni, M., Caniato, F., & Cagliano, R. (2018). Linking product complexity, disruption and performance: The moderating role of supply chain resilience. Supply Chain Forum, 19 (4), 300–310. https://doi.org/10.1080/16258312.2018.1551039

Dwivedi, A., Srivastava, S., Agrawal, D., Jha, A., & Paul, S. K. (2023). Analyzing the inter-relationships of business recovery challenges in the manufacturing industry: implications for post-pandemic supply chain resilience. Global Journal of Flexible Systems Management, 24 (1), 31–48. https://doi.org/10.1007/s40171-023-00365-w

Ekanayake, E. M. A. C., Shen, G. Q. P., Kumaraswamy, M. M., Owusu, E. K., & Saka, A. B. (2021). Modeling supply chain resilience in industrialized construction: A Hong Kong case. Journal of Construction Engineering and Management, 147 (11), 1–16. https://doi.org/10.1061/(asce)co.1943-7862.0002188

Fahimnia, B., & Jabbarzadeh, A. (2016). Marrying supply chain sustainability and resilience: A match made in heaven. Transportation Research Part e: Logistics and Transportation Review, 91 , 306–324. https://doi.org/10.1016/j.tre.2016.02.007

Fahimnia, B., Jabbarzadeh, A., & Sarkis, J. (2018). Greening versus resilience: A supply chain design perspective. Transportation Research Part e: Logistics and Transportation Review, 119 , 129–148. https://doi.org/10.1016/j.tre.2018.09.005

Faruquee, M., Paulraj, A., & Irawan, C. A. (2023). A typology of supply chain resilience: recognising the multi-capability nature of proactive and reactive contexts. Production Planning and Control , in press, 1–21. https://doi.org/10.1080/09537287.2023.2202151

Fattahi, M., Govindan, K., & Maihami, R. (2020). Stochastic optimization of disruption-driven supply chain network design with a new resilience metric. International Journal of Production Economics, 230 , 107755. https://doi.org/10.1016/j.ijpe.2020.107755

Gölgeci, I., & Kuivalainen, O. (2020). Does social capital matter for supply chain resilience? The role of absorptive capacity and marketing-supply chain management alignment. Industrial Marketing Management, 84 , 63–74. https://doi.org/10.1016/j.indmarman.2019.05.006

Grzybowska, K., & Stachowiak, A. (2022). Global changes and disruptions in supply chains—preliminary research to sustainable resilience of supply chains. Energies, 15 , 4579. https://doi.org/10.3390/en15134579

Hamidu, Z., Boachie-Mensah, F. O., & Issau, K. (2023a). Supply chain resilience and performance of manufacturing firms: Role of supply chain disruption. Journal of Manufacturing Technology Management, 34 (3), 361–382. https://doi.org/10.1108/JMTM-08-2022-0307

Hamidu, Z., Mensah, B. D., Issau, K., & Asafo-Adjei, E. (2023b). Does technological innovation matter in the nexus between supply chain resilience and performance of manufacturing firms in a developing economy? Journal of Manufacturing Technology Management, 34 (6), 981–1003. https://doi.org/10.1108/JMTM-11-2022-0384

Hamidu, Z., Issau, K., Boachie-Mensah, F. O., & Asafo-Adjei, E. (2024). On the interplay of supply chain network complexity on the nexus between supply chain resilience and performance. Benchmarking, 31 (5), 1590–1610. https://doi.org/10.1108/BIJ-09-2022-0551

Hohenstein, N. O., Feise, E., Hartmann, E., & Giunipero, L. (2015). Research on the phenomenon of supply chain resilience: A systematic review and paths for further investigation. International Journal of Physical Distribution and Logistics Management, 45 , 90–117. https://doi.org/10.1108/IJPDLM-05-2013-0128

Isti’anah, P. R., Praharsi, Y., Maharani, A., & Wee, H. M. (2021). Supply chain resilience analysis using the quality function deployment (QFD) approach in a freight forwarding company. Reliability: Theory and Applications, 16 (2), 15–26. https://doi.org/10.24412/1932-2321-2021-264-15-26

Ivanov, D. (2022). Blackout and supply chains: Cross-structural ripple effect, performance, resilience and viability impact analysis. Annals of Operations Research, in Press. https://doi.org/10.1007/s10479-022-04754-9

Juan, S. J., & Li, E. Y. (2023). Financial performance of firms with supply chains during the COVID-19 pandemic: The roles of dynamic capability and supply chain resilience. International Journal of Operations and Production Management, 43 (5), 712–737. https://doi.org/10.1108/IJOPM-04-2022-0249

Kamalahmadi, M., Shekarian, M., & Mellat Parast, M. (2022). The impact of flexibility and redundancy on improving supply chain resilience to disruptions. International Journal of Production Research, 60 (6), 1992–2020. https://doi.org/10.1080/00207543.2021.1883759

Kazancoglu, I., Ozbiltekin-Pala, M., Mangla, S. K., Kazancoglu, Y., & Jabeen, F. (2022). Role of flexibility, agility and responsiveness for sustainable supply chain resilience during COVID-19. Journal of Cleaner Production, 362 , 132431. https://doi.org/10.1016/j.jclepro.2022.132431

Kummer, Y., Fikar, C., Burtscher, J., Strobl, M., Fuchs, R., Domig, K. J., & Hirsch, P. (2022). Facilitating resilience during an african swine fever outbreak in the austrian pork supply chain through hybrid simulation modelling. Agriculture (Switzerland), 12 (3), 1–17. https://doi.org/10.3390/agriculture12030352

Ladeira, M. B., de Oliveira, M. P. V., de Sousa, P. R., & Barbosa, M. W. (2021). Firm’s supply chain agility enabling resilience and performance in turmoil times. International Journal of Agile Systems and Management, 14 (2), 224–253. https://doi.org/10.1504/IJASM.2021.118068

Li, Z., Liu, Q., Ye, C., Dong, M., & Zheng, Y. (2022). Achieving resilience: Resilient price and quality strategies of fresh food dual-channel supply chain considering the disruption. Sustainability (Switzerland), 14 (11), 6645. https://doi.org/10.3390/su14116645

Lin, Y., Chen, A., Zhong, S., Giannikas, V., Lomas, C., & Worth, T. (2023). Service supply chain resilience: A social-ecological perspective on last-mile delivery operations. International Journal of Operations and Production Management, 43 (1), 140–165. https://doi.org/10.1108/IJOPM-03-2022-0180

Macdonald, J. R., Zobel, C. W., Melnyk, S. A., & Griffis, S. E. (2018). Supply chain risk and resilience: Theory building through structured experiments and simulation. International Journal of Production Research, 56 (12), 4337–4355. https://doi.org/10.1080/00207543.2017.1421787

Mackay, J., Munoz, A., & Pepper, M. (2020). Conceptualising redundancy and flexibility towards supply chain robustness and resilience. Journal of Risk Research, 23 (12), 1541–1561. https://doi.org/10.1080/13669877.2019.1694964

Maharjan, R., & Kato, H. (2023). Logistics and supply chain resilience of Japanese companies: Perspectives from Impacts of the COVID-19 pandemic. Logistics, 7 (2), 27. https://doi.org/10.3390/logistics7020027

Mandal, S. (2014). Supply chain resilience: A state-of-the-art review and research directions. International Journal of Disaster Resilience in the Built Environment, 5 (4), 427–453. https://doi.org/10.1108/IJDRBE-03-2013-0003

Mao, X., Lou, X., Yuan, C., & Zhou, J. (2020). Resilience-based restoration model for supply chain networks. Mathematics, 8 (2), 163. https://doi.org/10.3390/math8020163

Massari, G. F., & Giannoccaro, I. (2021). Investigating the effect of horizontal coopetition on supply chain resilience in complex and turbulent environments. International Journal of Production Economics, 237 , 108150. https://doi.org/10.1016/j.ijpe.2021.108150

Mikhail, M., El-Beheiry, M., & Afia, N. (2019). Incorporating resilience determinants in supply chain network design model. Journal of Modelling in Management, 14 (3), 738–753. https://doi.org/10.1108/JM2-05-2018-0057

Moosavi, J., & Hosseini, S. (2021). Simulation-based assessment of supply chain resilience with consideration of recovery strategies in the COVID-19 pandemic context. Computers and Industrial Engineering, 160 , 107593. https://doi.org/10.1016/j.cie.2021.107593

Nagariya, R., Mukherjee, S., Baral, M. M., & Chittipaka, V. (2023). Analyzing blockchain-based supply chain resilience strategies: Resource-based perspective. International Journal of Productivity and Performance Management, in Press. https://doi.org/10.1108/IJPPM-07-2022-0330

Nguyen, D. N., Nguyen, T. T. H., Nguyen, T. T., Nguyen, X. H., Do, T. K. T., & Ngo, H. N. (2022). The effect of supply chain finance on supply chain risk, supply chain risk resilience, and performance of vietnam smes in global supply chain. Uncertain Supply Chain Management, 10 (1), 225–238. https://doi.org/10.5267/j.uscm.2021.9.005

Olivares-Aguila, J., & Vital-Soto, A. (2021). Supply chain resilience roadmaps for major disruptions. Logistics, 5 (4), 78. https://doi.org/10.3390/logistics5040078

Paul, S. K., & Chowdhury, P. (2020). Strategies for managing the impacts of disruptions during COVID-19: An example of toilet paper. Global Journal of Flexible Systems Management, 21 , 283–293. https://doi.org/10.1007/s40171-020-00248-4

Paul, S. K., Sarker, R., & Essam, D. (2017). A quantitative model for disruption mitigation in a supply chain. European Journal of Operational Research, 257 (3), 881–895. https://doi.org/10.1016/j.ejor.2016.08.035

Piprani, A. Z., Jaafar, N. I., Ali, S. M., Mubarik, M. S., & Shahbaz, M. (2022). Multi-dimensional supply chain flexibility and supply chain resilience: The role of supply chain risks exposure. Operations Management Research, 15 (1–2), 307–325. https://doi.org/10.1007/s12063-021-00232-w

Ponis, S. T., & Koronis, E. (2012). Supply Chain Resilience? Definition of concept and its formative elements. The Journal of Applied Business Research, 28 (5), 921–935. https://doi.org/10.19030/jabr.v28i5.7234

Praharsi, Y., Jamiin, M. A., Suhardjito, G., & Wee, H. M. (2021). The application of Lean Six Sigma and supply chain resilience in maritime industry during the era of COVID-19. International Journal of Lean Six Sigma, 12 (4), 800–834. https://doi.org/10.1108/IJLSS-11-2020-0196

Pu, G., Qiao, W., & Feng, Z. (2023a). Antecedents and outcomes of supply chain resilience: Integrating dynamic capabilities and relational perspective. Journal of Contingencies and Crisis Management, 31 (4), 706–726. https://doi.org/10.1111/1468-5973.12473

Pu, W., Ma, S., & Yan, X. (2023b). Geographical relevance-based multi-period optimization for e-commerce supply chain resilience strategies under disruption risks. International Journal of Production Research, in Press. https://doi.org/10.1080/00207543.2023.2217937

Purvis, L., Spall, S., Naim, M., & Spiegler, V. (2016). Developing a resilient supply chain strategy during ‘boom’ and ‘bust.’ Production Planning and Control, 27 (7–8), 579–590. https://doi.org/10.1080/09537287.2016.1165306

Rahman, T., Paul, S. K., Shukla, N., Agarwal, R., & Taghikhah, F. (2022). Supply chain resilience initiatives and strategies: A systematic review. Computers and Industrial Engineering, 170 , 108317. https://doi.org/10.1016/j.cie.2022.108317

Rajesh, R. (2016). Forecasting supply chain resilience performance using grey prediction. Electronic Commerce Research and Applications, 20 , 42–58. https://doi.org/10.1016/j.elerap.2016.09.006

Rajesh, R. (2021). Flexible business strategies to enhance resilience in manufacturing supply chains: An empirical study. Journal of Manufacturing Systems, 60 , 903–919. https://doi.org/10.1016/j.jmsy.2020.10.010

Ribeiro, J. P., & Barbosa-Povoa, A. (2018). Supply chain resilience: Definitions and quantitative modelling approaches–A literature review. Computers & Industrial Engineering, 115 , 109–122. https://doi.org/10.1016/j.cie.2017.11.006

Ryan, S. M., Roberts, E., Hibbett, E., Bloom, N., Haden, C., Rushforth, R. R., Pfeiffer, K., & Ruddell, B. L. (2021). The FEWSION for community resilience (F4R) process: Building local technical and social capacity for critical supply chain resilience. Frontiers in Environmental Science, 9 , 1–14. https://doi.org/10.3389/fenvs.2021.601220

Sangari, M. S., & Dashtpeyma, M. (2019). An integrated framework of supply chain resilience enablers: A hybrid ISM-FANP approach. International Journal of Business Excellence, 18 (2), 242–268. https://doi.org/10.1504/IJBEX.2019.099558

Scholten, K., Stevenson, M., & van Donk, D. P. (2020). Dealing with the unpredictable: Supply chain resilience. International Journal of Operations & Production Management, 40 (1), 1–10. https://doi.org/10.1108/IJOPM-01-2020-789

Shashi, Centobelli, P., Cerchione, R., & Ertz, M. (2020). Managing supply chain resilience to pursue business and environmental strategies. Business Strategy and the Environment, 29 (3), 1215–1246. https://doi.org/10.1002/bse.2428

Sharma, B., Mittal, M. L., Soni, G., & Ramtiyal, B. (2023). An implementation framework for resiliency assessment in a supply chain. Global Journal of Flexible Systems Management , 24 (4), 591–614

Shishodia, A., Sharma, R., Rajesh, R., & Munim, Z. H. (2023). Supply chain resilience: A review, conceptual framework and future research. International Journal of Logistics Management, 34 (4), 879–908. https://doi.org/10.1108/IJLM-03-2021-0169

Shweta, K., & D., & Chandra, D. (2023). A hybrid framework to model resilience in the generic medicine supply chain of MSMEs. Benchmarking, 30 (6), 2189–2224. https://doi.org/10.1108/BIJ-11-2021-0697

Silva, M. E., Pereira, M. M. O., & Hendry, L. C. (2023). Embracing change in tandem: Resilience and sustainability together transforming supply chains. International Journal of Operations and Production Management, 43 (1), 166–196. https://doi.org/10.1108/IJOPM-09-2022-0625

Singh, N. P., & Singh, S. (2019). Building supply chain risk resilience: Role of big data analytics in supply chain disruption mitigation. Benchmarking, 26 (7), 2318–2342. https://doi.org/10.1108/BIJ-10-2018-0346

Suryadi, A., & Rau, H. (2023). Considering region risks and mitigation strategies in the supplier selection process for improving supply chain resilience. Computers and Industrial Engineering, 181 , 109288. https://doi.org/10.1016/j.cie.2023.109288

Suryawanshi, P., Dutta, P., Varun, L., & Deepak, G. (2021). Sustainable and resilience planning for the supply chain of online hyperlocal grocery services. Sustainable Production and Consumption, 28 , 496–518. https://doi.org/10.1016/j.spc.2021.05.001

Tan, W. J., Cai, W., & Zhang, A. N. (2020). Structural-aware simulation analysis of supply chain resilience. International Journal of Production Research, 58 (17), 5175–5195. https://doi.org/10.1080/00207543.2019.1705421

Tang, C., & Tomlin, B. (2008). The power of flexibility for mitigating supply chain risks. International Journal of Production Economics, 116 (1), 12–27. https://doi.org/10.1016/j.ijpe.2008.07.008

Trabucco, M., & De Giovanni, P. (2021). Achieving resilience and business sustainability during COVID-19: The role of lean supply chain practices and digitalization. Sustainability (Switzerland), 13 (22), 12369. https://doi.org/10.3390/su132212369

Tranfield, D., Denyer, D., & Smart, P. (2003). Towards a methodology for developing evidence-informed management knowledge by means of systematic review. British Journal of Management, 14 (3), 207–222. https://doi.org/10.1111/1467-8551.00375

Tukamuhabwa, B. R., Stevenson, M., Busby, J., & Zorzini, M. (2015). Supply chain resilience: Definition, review and theoretical foundations for further study. International Journal of Production Research, 53 (18), 5592–5623. https://doi.org/10.1080/00207543.2015.1037934

Um, J., & Han, N. (2021). Understanding the relationships between global supply chain risk and supply chain resilience: The role of mitigating strategies. Supply Chain Management, 26 (2), 240–255. https://doi.org/10.1108/SCM-06-2020-0248

Varma, S., Singh, N., & Patra, A. (2024). Supply chain flexibility: Unravelling the research trajectory through citation path analysis. Global Journal of Flexible Systems Management, 25 (2), 199–222

Vimal, K. E. K., Nadeem, S. P., Meledathu Sunil, S., Suresh, G., Sanjeev, N., & Kandasamy, J. (2022a). Modelling the strategies for improving maturity and resilience in medical oxygen supply chain through digital technologies. Journal of Global Operations and Strategic Sourcing, 15 (4), 566–595. https://doi.org/10.1108/JGOSS-10-2021-0088

Vimal, K. E. K., Nadeem, S. P., Ravichandran, M., Ethirajan, M., & Kandasamy, J. (2022b). Resilience strategies to recover from the cascading ripple effect in a copper supply chain through project management. Operations Management Research, 15 , 440–460. https://doi.org/10.1007/s12063-021-00231-x

Wadhwa, S., Saxena, A., & Chan, F. T. S. (2008). Framework for flexibility in dynamic supply chain management. International Journal of Production Research, 46 (6), 1373–1404. https://doi.org/10.1080/00207540600570432

Wang, X., Herty, M., & Zhao, L. (2016). Contingent rerouting for enhancing supply chain resilience from supplier behavior perspective. International Transactions in Operational Research, 23 (4), 775–796. https://doi.org/10.1111/itor.12151

Xu, B., Liu, W., Li, J., Yang, Y., Wen, F., & Song, H. (2023). Resilience measurement and dynamic optimization of container logistics supply chain under adverse events. Computers and Industrial Engineering, 180 , 109202. https://doi.org/10.1016/j.cie.2023.109202

Yi, C. Y., Ngai, E. W. T., & Moon, K. (2011). Supply chain flexibility in an uncertain environment: Exploratory findings from five case studies. Supply Chain Management: An International Journal, 16 (4), 271–283. https://doi.org/10.1108/13598541111139080

Zavala-Alcívar, A., Verdecho, M. J., & Alfaro-Saiz, J. J. (2020). A conceptual framework to manage resilience and increase sustainability in the supply chain. Sustainability (Switzerland), 12 (16), 6300. https://doi.org/10.3390/SU12166300

Zavitsas, K., Zis, T., & Bell, M. G. H. (2018). The impact of flexible environmental policy on maritime supply chain resilience. Transport Policy, 72 , 116–128. https://doi.org/10.1016/j.tranpol.2018.09.020

Zhou, J., Hu, L., Yu, Y., Zhang, J. Z., & Zheng, L. J. (2022). Impacts of IT capability and supply chain collaboration on supply chain resilience: Empirical evidence from China in COVID-19 pandemic. Journal of Enterprise Information Management, in Press. https://doi.org/10.1108/JEIM-03-2022-0091

Zhu, X., & Wu, Y. J. (2022). How does supply chain resilience affect supply chain performance? The Mediating Effect of Sustainability. Sustainability (Switzerland), 14 (21), 1–19. https://doi.org/10.3390/su142114626

Download references

Acknowledgements

This research is supported by an Australian Government Research Training Program Scholarship.

Open Access funding enabled and organized by CAUL and its Member Institutions.

Author information

Authors and affiliations.

Faculty of Engineering and IT, School of Mechanical and Mechatronic Engineering, University of Technology Sydney, Ultimo, NSW, 2007, Australia

Ananna Paul & Suvash C. Saha

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Suvash C. Saha .

Ethics declarations

Conflict of interest.

The authors declare that they have no known competing financial interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Paul, A., Saha, S.C. A Systematic Literature Review on Flexible Strategies and Performance Indicators for Supply Chain Resilience. Glob J Flex Syst Manag (2024). https://doi.org/10.1007/s40171-024-00415-x

Download citation

Received : 16 April 2024

Accepted : 20 August 2024

Published : 10 September 2024

DOI : https://doi.org/10.1007/s40171-024-00415-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Flexible strategies
• Performance indicators
• Supply chain resilience
• Systematic literature review
• Find a journal
• Publish with us
• Track your research