critical appraisal of a research paper tool

Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

Critical Appraisal tools

Critical appraisal worksheets to help you appraise the reliability, importance and applicability of clinical evidence.

Critical appraisal is the systematic evaluation of clinical research papers in order to establish:

• Does this study address a  clearly focused question ?
• Did the study use valid methods to address this question?
• Are the valid results of this study important?
• Are these valid, important results applicable to my patient or population?

If the answer to any of these questions is “no”, you can save yourself the trouble of reading the rest of it.

This section contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples.

Critical Appraisal Worksheets

• Systematic Reviews  Critical Appraisal Sheet
• Diagnostics  Critical Appraisal Sheet
• Prognosis  Critical Appraisal Sheet
• Randomised Controlled Trials  (RCT) Critical Appraisal Sheet
• Critical Appraisal of Qualitative Studies  Sheet
• IPD Review  Sheet

Chinese - translated by Chung-Han Yang and Shih-Chieh Shao

• Systematic Reviews  Critical Appraisal Sheet
• Diagnostic Study  Critical Appraisal Sheet
• Prognostic Critical Appraisal Sheet
• RCT  Critical Appraisal Sheet
• IPD reviews Critical Appraisal Sheet
• Qualitative Studies Critical Appraisal Sheet 

German - translated by Johannes Pohl and Martin Sadilek

• Systematic Review  Critical Appraisal Sheet
• Diagnosis Critical Appraisal Sheet
• Prognosis Critical Appraisal Sheet
• Therapy / RCT Critical Appraisal Sheet

Lithuanian - translated by Tumas Beinortas

• Systematic review appraisal Lithuanian (PDF)
• Diagnostic accuracy appraisal Lithuanian  (PDF)
• Prognostic study appraisal Lithuanian  (PDF)
• RCT appraisal sheets Lithuanian  (PDF)

Portugese - translated by Enderson Miranda, Rachel Riera and Luis Eduardo Fontes

• Portuguese – Systematic Review Study Appraisal Worksheet
• Portuguese – Diagnostic Study Appraisal Worksheet
• Portuguese – Prognostic Study Appraisal Worksheet
• Portuguese – RCT Study Appraisal Worksheet
• Portuguese – Systematic Review Evaluation of Individual Participant Data Worksheet
• Portuguese – Qualitative Studies Evaluation Worksheet

Spanish - translated by Ana Cristina Castro

• Systematic Review  (PDF)
• Diagnosis  (PDF)
• Prognosis  Spanish Translation (PDF)
• Therapy / RCT  Spanish Translation (PDF)

Persian - translated by Ahmad Sofi Mahmudi

• Prognosis  (PDF)
• PICO  Critical Appraisal Sheet (PDF)
• PICO Critical Appraisal Sheet (MS-Word)
• Educational Prescription  Critical Appraisal Sheet (PDF)

Explanations & Examples

• Pre-test probability
• SpPin and SnNout
• Likelihood Ratios

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Published: 20 January 2009

How to critically appraise an article

• Jane M Young 1 &
• Michael J Solomon 2  

Nature Clinical Practice Gastroenterology & Hepatology volume  6 ,  pages 82–91 ( 2009 ) Cite this article

52k Accesses

99 Citations

421 Altmetric

Metrics details

Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article in order to assess the usefulness and validity of research findings. The most important components of a critical appraisal are an evaluation of the appropriateness of the study design for the research question and a careful assessment of the key methodological features of this design. Other factors that also should be considered include the suitability of the statistical methods used and their subsequent interpretation, potential conflicts of interest and the relevance of the research to one's own practice. This Review presents a 10-step guide to critical appraisal that aims to assist clinicians to identify the most relevant high-quality studies available to guide their clinical practice.

Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article

Critical appraisal provides a basis for decisions on whether to use the results of a study in clinical practice

Different study designs are prone to various sources of systematic bias

Design-specific, critical-appraisal checklists are useful tools to help assess study quality

Assessments of other factors, including the importance of the research question, the appropriateness of statistical analysis, the legitimacy of conclusions and potential conflicts of interest are an important part of the critical appraisal process

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 12 print issues and online access

195,33 € per year

only 16,28 € per issue

Buy this article

• Purchase on SpringerLink
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Making sense of the literature: an introduction to critical appraisal for the primary care practitioner

How to appraise the literature: basic principles for the busy clinician - part 2: systematic reviews and meta-analyses

How to appraise the literature: basic principles for the busy clinician - part 1: randomised controlled trials

Druss BG and Marcus SC (2005) Growth and decentralisation of the medical literature: implications for evidence-based medicine. J Med Libr Assoc 93 : 499–501

PubMed   PubMed Central   Google Scholar  

Glasziou PP (2008) Information overload: what's behind it, what's beyond it? Med J Aust 189 : 84–85

PubMed   Google Scholar  

Last JE (Ed.; 2001) A Dictionary of Epidemiology (4th Edn). New York: Oxford University Press

Google Scholar  

Sackett DL et al . (2000). Evidence-based Medicine. How to Practice and Teach EBM . London: Churchill Livingstone

Guyatt G and Rennie D (Eds; 2002). Users' Guides to the Medical Literature: a Manual for Evidence-based Clinical Practice . Chicago: American Medical Association

Greenhalgh T (2000) How to Read a Paper: the Basics of Evidence-based Medicine . London: Blackwell Medicine Books

MacAuley D (1994) READER: an acronym to aid critical reading by general practitioners. Br J Gen Pract 44 : 83–85

CAS   PubMed   PubMed Central   Google Scholar  

Hill A and Spittlehouse C (2001) What is critical appraisal. Evidence-based Medicine 3 : 1–8 [ http://www.evidence-based-medicine.co.uk ] (accessed 25 November 2008)

Public Health Resource Unit (2008) Critical Appraisal Skills Programme (CASP) . [ http://www.phru.nhs.uk/Pages/PHD/CASP.htm ] (accessed 8 August 2008)

National Health and Medical Research Council (2000) How to Review the Evidence: Systematic Identification and Review of the Scientific Literature . Canberra: NHMRC

Elwood JM (1998) Critical Appraisal of Epidemiological Studies and Clinical Trials (2nd Edn). Oxford: Oxford University Press

Agency for Healthcare Research and Quality (2002) Systems to rate the strength of scientific evidence? Evidence Report/Technology Assessment No 47, Publication No 02-E019 Rockville: Agency for Healthcare Research and Quality

Crombie IK (1996) The Pocket Guide to Critical Appraisal: a Handbook for Health Care Professionals . London: Blackwell Medicine Publishing Group

Heller RF et al . (2008) Critical appraisal for public health: a new checklist. Public Health 122 : 92–98

Article   Google Scholar  

MacAuley D et al . (1998) Randomised controlled trial of the READER method of critical appraisal in general practice. BMJ 316 : 1134–37

Article   CAS   Google Scholar  

Parkes J et al . Teaching critical appraisal skills in health care settings (Review). Cochrane Database of Systematic Reviews 2005, Issue 3. Art. No.: cd001270. 10.1002/14651858.cd001270

Mays N and Pope C (2000) Assessing quality in qualitative research. BMJ 320 : 50–52

Hawking SW (2003) On the Shoulders of Giants: the Great Works of Physics and Astronomy . Philadelphia, PN: Penguin

National Health and Medical Research Council (1999) A Guide to the Development, Implementation and Evaluation of Clinical Practice Guidelines . Canberra: National Health and Medical Research Council

US Preventive Services Taskforce (1996) Guide to clinical preventive services (2nd Edn). Baltimore, MD: Williams & Wilkins

Solomon MJ and McLeod RS (1995) Should we be performing more randomized controlled trials evaluating surgical operations? Surgery 118 : 456–467

Rothman KJ (2002) Epidemiology: an Introduction . Oxford: Oxford University Press

Young JM and Solomon MJ (2003) Improving the evidence-base in surgery: sources of bias in surgical studies. ANZ J Surg 73 : 504–506

Margitic SE et al . (1995) Lessons learned from a prospective meta-analysis. J Am Geriatr Soc 43 : 435–439

Shea B et al . (2001) Assessing the quality of reports of systematic reviews: the QUORUM statement compared to other tools. In Systematic Reviews in Health Care: Meta-analysis in Context 2nd Edition, 122–139 (Eds Egger M. et al .) London: BMJ Books

Chapter   Google Scholar  

Easterbrook PH et al . (1991) Publication bias in clinical research. Lancet 337 : 867–872

Begg CB and Berlin JA (1989) Publication bias and dissemination of clinical research. J Natl Cancer Inst 81 : 107–115

Moher D et al . (2000) Improving the quality of reports of meta-analyses of randomised controlled trials: the QUORUM statement. Br J Surg 87 : 1448–1454

Shea BJ et al . (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology 7 : 10 [10.1186/1471-2288-7-10]

Stroup DF et al . (2000) Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 283 : 2008–2012

Young JM and Solomon MJ (2003) Improving the evidence-base in surgery: evaluating surgical effectiveness. ANZ J Surg 73 : 507–510

Schulz KF (1995) Subverting randomization in controlled trials. JAMA 274 : 1456–1458

Schulz KF et al . (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273 : 408–412

Moher D et al . (2001) The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Medical Research Methodology 1 : 2 [ http://www.biomedcentral.com/ 1471-2288/1/2 ] (accessed 25 November 2008)

Rochon PA et al . (2005) Reader's guide to critical appraisal of cohort studies: 1. Role and design. BMJ 330 : 895–897

Mamdani M et al . (2005) Reader's guide to critical appraisal of cohort studies: 2. Assessing potential for confounding. BMJ 330 : 960–962

Normand S et al . (2005) Reader's guide to critical appraisal of cohort studies: 3. Analytical strategies to reduce confounding. BMJ 330 : 1021–1023

von Elm E et al . (2007) Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 335 : 806–808

Sutton-Tyrrell K (1991) Assessing bias in case-control studies: proper selection of cases and controls. Stroke 22 : 938–942

Knottnerus J (2003) Assessment of the accuracy of diagnostic tests: the cross-sectional study. J Clin Epidemiol 56 : 1118–1128

Furukawa TA and Guyatt GH (2006) Sources of bias in diagnostic accuracy studies and the diagnostic process. CMAJ 174 : 481–482

Bossyut PM et al . (2003)The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 138 : W1–W12

STARD statement (Standards for the Reporting of Diagnostic Accuracy Studies). [ http://www.stard-statement.org/ ] (accessed 10 September 2008)

Raftery J (1998) Economic evaluation: an introduction. BMJ 316 : 1013–1014

Palmer S et al . (1999) Economics notes: types of economic evaluation. BMJ 318 : 1349

Russ S et al . (1999) Barriers to participation in randomized controlled trials: a systematic review. J Clin Epidemiol 52 : 1143–1156

Tinmouth JM et al . (2004) Are claims of equivalency in digestive diseases trials supported by the evidence? Gastroentrology 126 : 1700–1710

Kaul S and Diamond GA (2006) Good enough: a primer on the analysis and interpretation of noninferiority trials. Ann Intern Med 145 : 62–69

Piaggio G et al . (2006) Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA 295 : 1152–1160

Heritier SR et al . (2007) Inclusion of patients in clinical trial analysis: the intention to treat principle. In Interpreting and Reporting Clinical Trials: a Guide to the CONSORT Statement and the Principles of Randomized Controlled Trials , 92–98 (Eds Keech A. et al .) Strawberry Hills, NSW: Australian Medical Publishing Company

National Health and Medical Research Council (2007) National Statement on Ethical Conduct in Human Research 89–90 Canberra: NHMRC

Lo B et al . (2000) Conflict-of-interest policies for investigators in clinical trials. N Engl J Med 343 : 1616–1620

Kim SYH et al . (2004) Potential research participants' views regarding researcher and institutional financial conflicts of interests. J Med Ethics 30 : 73–79

Komesaroff PA and Kerridge IH (2002) Ethical issues concerning the relationships between medical practitioners and the pharmaceutical industry. Med J Aust 176 : 118–121

Little M (1999) Research, ethics and conflicts of interest. J Med Ethics 25 : 259–262

Lemmens T and Singer PA (1998) Bioethics for clinicians: 17. Conflict of interest in research, education and patient care. CMAJ 159 : 960–965

Download references

Author information

Authors and affiliations.

JM Young is an Associate Professor of Public Health and the Executive Director of the Surgical Outcomes Research Centre at the University of Sydney and Sydney South-West Area Health Service, Sydney,

Jane M Young

MJ Solomon is Head of the Surgical Outcomes Research Centre and Director of Colorectal Research at the University of Sydney and Sydney South-West Area Health Service, Sydney, Australia.,

Michael J Solomon

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jane M Young .

Ethics declarations

Competing interests.

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Young, J., Solomon, M. How to critically appraise an article. Nat Rev Gastroenterol Hepatol 6 , 82–91 (2009). https://doi.org/10.1038/ncpgasthep1331

Download citation

Received : 10 August 2008

Accepted : 03 November 2008

Published : 20 January 2009

Issue Date : February 2009

DOI : https://doi.org/10.1038/ncpgasthep1331

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Emergency physicians’ perceptions of critical appraisal skills: a qualitative study.

• Sumintra Wood
• Jacqueline Paulis
• Angela Chen

BMC Medical Education (2022)

An integrative review on individual determinants of enrolment in National Health Insurance Scheme among older adults in Ghana

• Anthony Kwame Morgan
• Anthony Acquah Mensah

BMC Primary Care (2022)

Autopsy findings of COVID-19 in children: a systematic review and meta-analysis

• Anju Khairwa
• Kana Ram Jat

Forensic Science, Medicine and Pathology (2022)

The use of a modified Delphi technique to develop a critical appraisal tool for clinical pharmacokinetic studies

• Alaa Bahaa Eldeen Soliman
• Shane Ashley Pawluk
• Ousama Rachid

International Journal of Clinical Pharmacy (2022)

Critical Appraisal: Analysis of a Prospective Comparative Study Published in IJS

• Ramakrishna Ramakrishna HK
• Swarnalatha MC

Indian Journal of Surgery (2021)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Critical Appraisal Tools
• Introduction
• Related Guides
• Getting Help

Critical Appraisal of Studies

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value/relevance in a particular context by providing a framework to evaluate the research. During the critical appraisal process, researchers can:

• Decide whether studies have been undertaken in a way that makes their findings reliable as well as valid and unbiased
• Make sense of the results
• Know what these results mean in the context of the decision they are making
• Determine if the results are relevant to their patients/schoolwork/research

Burls, A. (2009). What is critical appraisal? In What Is This Series: Evidence-based medicine. Available online at  What is Critical Appraisal?

Critical appraisal is included in the process of writing high quality reviews, like systematic and integrative reviews and for evaluating evidence from RCTs and other study designs. For more information on systematic reviews, check out our  Systematic Review  guide.

• Next: Critical Appraisal Tools >>
• Last Updated: Nov 16, 2023 1:27 PM
• URL: https://guides.library.duq.edu/critappraise

Critical Appraisal: Assessing the Quality of Studies

• First Online: 05 August 2020

Cite this chapter

• Edward Purssell   ORCID: orcid.org/0000-0003-3748-0864 3 &
• Niall McCrae   ORCID: orcid.org/0000-0001-9776-7694 4  

8748 Accesses

There is great variation in the type and quality of research evidence. Having completed your search and assembled your studies, the next step is to critically appraise the studies to ascertain their quality. Ultimately you will be making a judgement about the overall evidence, but that comes later. You will see throughout this chapter that we make a clear differentiation between the individual studies and what we call the body of evidence , which is all of the studies and anything else that we use to answer the question or to make a recommendation. This chapter deals with only the first of these—the individual studies. Critical appraisal, like everything else in systematic literature reviewing, is a scientific exercise that requires individual judgement, and we describe some tools to help you.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Oxford Centre for Evidence-Based Medicine (OCEBM) (2016) OCEBM levels of evidence. In: CEBM. https://www.cebm.net/2016/05/ocebm-levels-of-evidence/ . Accessed 17 Apr 2020

Aromataris E, Munn Z (eds) (2017) Joanna Briggs Institute reviewer’s manual. The Joanna Briggs Institute, Adelaide

Google Scholar  

Daly J, Willis K, Small R et al (2007) A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol 60:43–49. https://doi.org/10.1016/j.jclinepi.2006.03.014

Article   PubMed   Google Scholar  

EQUATOR Network (2020) What is a reporting guideline?—The EQUATOR Network. https://www.equator-network.org/about-us/what-is-a-reporting-guideline/ . Accessed 7 Mar 2020

Tong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 19:349–357. https://doi.org/10.1093/intqhc/mzm042

von Elm E, Altman DG, Egger M et al (2007) The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med 4:e296. https://doi.org/10.1371/journal.pmed.0040296

Article   Google Scholar  

Brouwers MC, Kerkvliet K, Spithoff K, AGREE Next Steps Consortium (2016) The AGREE reporting checklist: a tool to improve reporting of clinical practice guidelines. BMJ 352:i1152. https://doi.org/10.1136/bmj.i1152

Article   PubMed   PubMed Central   Google Scholar  

Moher D, Liberati A, Tetzlaff J et al (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6:e1000097. https://doi.org/10.1371/journal.pmed.1000097

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A (2019) Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019), Cochrane. https://www.training.cochrane.org/handbook

Critical Appraisal Skills Programme (2018) CASP checklists. In: CASP—critical appraisal skills programme. https://casp-uk.net/casp-tools-checklists/ . Accessed 7 Mar 2020

Higgins JPT, Savović J, Page MJ et al (2019) Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Chapter   Google Scholar  

Guyatt GH, Oxman AD, Kunz R et al (2011) GRADE guidelines 6. Rating the quality of evidence—imprecision. J Clin Epidemiol 64:1283–1293. https://doi.org/10.1016/j.jclinepi.2011.01.012

Sterne JAC, Savović J, Page MJ et al (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898. https://doi.org/10.1136/bmj.l4898

Sterne JA, Hernán MA, Reeves BC et al (2016) ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355:i4919. https://doi.org/10.1136/bmj.i4919

Wells GA, Shea B, O’Connell D et al (2019) The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute, Ottawa. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp . Accessed 7 Mar 2020

Cochrane Community (2020) Glossary—Cochrane community. https://community.cochrane.org/glossary#letter-R . Accessed 8 Mar 2020

Messick S (1989) Meaning and values in test validation: the science and ethics of assessment. Educ Res 18:5–11. https://doi.org/10.3102/0013189X018002005

Sparkes AC (2001) Myth 94: qualitative health researchers will agree about validity. Qual Health Res 11:538–552. https://doi.org/10.1177/104973230101100409

Article   CAS   PubMed   Google Scholar  

Aguinis H, Solarino AM (2019) Transparency and replicability in qualitative research: the case of interviews with elite informants. Strat Manag J 40:1291–1315. https://doi.org/10.1002/smj.3015

Lincoln YS, Guba EG (1985) Naturalistic inquiry. Sage Publications, Beverly Hills, CA

Book   Google Scholar  

Hannes K (2011) Chapter 4: Critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K et al (eds) Supplementary guidance for inclusion of qualitative research in Cochrane systematic reviews of interventions. Cochrane Collaboration Qualitative Methods Group, London

Munn Z, Porritt K, Lockwood C et al (2014) Establishing confidence in the output of qualitative research synthesis: the ConQual approach. BMC Med Res Methodol 14:108. https://doi.org/10.1186/1471-2288-14-108

Toye F, Seers K, Allcock N et al (2013) ‘Trying to pin down jelly’—exploring intuitive processes in quality assessment for meta-ethnography. BMC Med Res Methodol 13:46. https://doi.org/10.1186/1471-2288-13-46

Katikireddi SV, Egan M, Petticrew M (2015) How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study. J Epidemiol Community Health 69:189–195. https://doi.org/10.1136/jech-2014-204711

McKenzie JE, Brennan SE, Ryan RE et al (2019) Chapter 9: Summarizing study characteristics and preparing for synthesis. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Deeks JJ, Higgins JPT, Altman DG (2019) Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Download references

Author information

Authors and affiliations.

School of Health Sciences, City, University of London, London, UK

Edward Purssell

Florence Nightingale Faculty of Nursing, Midwifery & Palliative Care, King’s College London, London, UK

Niall McCrae

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Edward Purssell .

Rights and permissions

Reprints and permissions

Copyright information

© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Purssell, E., McCrae, N. (2020). Critical Appraisal: Assessing the Quality of Studies. In: How to Perform a Systematic Literature Review. Springer, Cham. https://doi.org/10.1007/978-3-030-49672-2_6

Download citation

DOI : https://doi.org/10.1007/978-3-030-49672-2_6

Published : 05 August 2020

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-49671-5

Online ISBN : 978-3-030-49672-2

eBook Packages : Medicine Medicine (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• Geoffrey R. Weller Library

View complete hours

• Subject Guides

Knowledge Synthesis Guide

• Critical Appraisal
• What is Knowledge Synthesis?
• Developing your question
• Consider Eligibility Criteria
• Grey Literature
• Create Search Terms for Each Concept
• Identify Controlled Vocabulary for Each Concept
• Building Your Search
• Translating a Search Strategy
• Run Your Searches
• Reporting Your Results with PRISMA
• Utilize a Screening Tool
• Data Extraction
• Information About Publishing This link opens in a new window

Tools for Critical Appraisal

Critical appraisal is the careful analysis of a study to assess trustworthiness, relevance and results of published research. Here are some tools to guide you. 

• JBI Critical Appraisal
• CASP Checklists
• The AACODS checklist

Appraisal Resources - Grey Literature

Appraising Grey Literature:

• Guide to Appraising Grey Literature ( Public Health Ontario)
• << Previous: Utilize a Screening Tool
• Next: Data Extraction >>
• Last Updated: Sep 20, 2024 9:48 AM
• URL: https://libguides.unbc.ca/KnowledgeSynthesis

Geoffrey R. Weller Library University of Northern British Columbia 3333 University Way Prince George, B.C. V2N 4Z9

Circulation: (250) 960-6613 Reference: (250) 960-6475 Regional Services: 1-888-440-3440 (toll free within 250 area code)

• Suggestions Form
• Planning & Policies
• Staff Directory
• Frequently Called Numbers
• Citation Management
• Course Reserves
• Faculty Services
• Interlibrary Loans
• Open Access
• Data & Statistics
• Maps & Photos
• Research Help

• Get new issue alerts Get alerts
• Submit a Manuscript

Secondary Logo

Journal logo.

Colleague's E-mail is Invalid

Your message has been successfully sent to your colleague.

Save my selection

Critical appraisal of a clinical research paper

What one needs to know.

Manjali, Jifmi Jose; Gupta, Tejpal

Department of Radiation Oncology, Tata Memorial Centre, Homi Bhabha National Institute, Mumbai, Maharashtra, India

Address for correspondence: Dr. Tejpal Gupta, ACTREC, Tata Memorial Centre, Homi Bhabha National Institute, Kharghar, Navi Mumbai - 410 210, Maharashtra, India. E-mail: [email protected]

Received May 25, 2020

Received in revised form June 11, 2020

Accepted June 19, 2020

This is an open access journal, and articles are distributed under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License, which allows others to remix, tweak, and build upon the work non-commercially, as long as appropriate credit is given and the new creations are licensed under the identical terms.

In the present era of evidence-based medicine (EBM), integrating best research evidence into the clinical practice necessitates developing skills to critically evaluate and analyze the scientific literature. Critical appraisal is the process of systematically examining research evidence to assess its validity, results, and relevance to inform clinical decision-making. All components of a clinical research article need to be appraised as per the study design and conduct. As research bias can be introduced at every step in the flow of a study leading to erroneous conclusions, it is essential that suitable measures are adopted to mitigate bias. Several tools have been developed for the critical appraisal of scientific literature, including grading of evidence to help clinicians in the pursuit of EBM in a systematic manner. In this review, we discuss the broad framework for the critical appraisal of a clinical research paper, along with some of the relevant guidelines and recommendations.

INTRODUCTION

Medical research information is ever growing and branching day by day. Despite the vastness of medical literature, it is necessary that as clinicians we offer the best treatment to our patients as per the current knowledge. Integrating best research evidence with clinical expertise and patient values has led to the concept of evidence-based medicine (EBM).[ 1 ] Although this philosophy originated in the middle of the 19 th century,[ 2 ] it first appeared in its current form in the modern medical literature in 1991.[ 3 ] EBM is defined as the conscientious, explicit, and judicious use of the current best evidence in making decisions about the care of an individual patient.[ 1 ] The essentials of EBM include generating a clinical question, tracking the best available evidence, critically evaluating the evidence for validity and clinical usefulness, further applying the results to clinical practice, and evaluating its performance. Appropriate application of EBM can result in cost-effectiveness and improve health-care efficiency.[ 4 ] Without continual accumulation of new knowledge, existing dogmas and paradigms quickly become outdated and may prove detrimental to the patients. The current growth of medical literature with 1.8 million scientific articles published in the year 2012,[ 5 ] often makes it difficult for the clinicians to keep pace with the vast amount of scientific data, thus making foraging (alerts to new information) and hunting (finding answers to clinical questions) essential skills to help navigate the so-called “jungle” of information.[ 6 ] Therefore, it is essential that health-care professionals read medical literature selectively to effectively utilize their limited time and assiduously imbibe new knowledge to improve decision-making for their patients. To practice EBM in its true sense, a clinician not only needs to devote time to develop the skill of effectively searching the literature, but also needs to learn to evaluate the significance, methodology, outcomes, and transparency of the study.[ 4 ] Along with the evaluation and interpretation of a study, a thorough understanding of its methodology is necessary. It is common knowledge that studies with positive results are relatively easy to publish.[ 7 8 ] However, it is the critical appraisal of any research study (even those with negative results) that helps us to understand the science better and ask relevant questions in future using an appropriate study design and endpoints. Therefore, this review is focused on the framework for the critical appraisal of a clinical research paper. In addition, we have also discussed some of the relevant guidelines and recommendations for the critical appraisal of clinical research papers.

CRITICAL APPRAISAL

Critical appraisal is the process of systematically examining the research evidence to assess its validity, results, and relevance before using it to inform a decision.[ 9 ] It entails the following:

• Balanced assessment of the benefits/strengths and flaws/weaknesses of a study
• Assessment of the research process and results
• Consideration of quantitative and qualitative aspects.

Critical appraisal is performed to assess the following

aspects of a study:

• Validity – Is the methodology robust?
• Reliability – Are the results credible?
• Applicability– Do the results have the potential to change the current practice?

Contrary to the common belief, a critical appraisal is not the negative dismissal of any piece of research or an assessment of the results alone; it is neither solely based on a statistical analysis nor a process undertaken by the experts only. When performing a critical appraisal of a scientific article, it is essential that we know its basic composition and assess every section meticulously.

Initial assessment

This involves taking a generalized look at the details of the article. The journal it was published in holds special value – a peer reviewed, indexed journal with a good impact factor adds robustness to the paper. The setting, timeline, and year of publication of the study also need to be noted, as they provide a better understanding of the evolution of thoughts in that particular subject. Declaration of the conflicts of interest by the authors, the role of the funding source if any, and any potential commercial bias should also be noted.[ 10 ]

COMPONENTS OF A CLINICAL RESEARCH PAPER

The components of any scientific article or clinical research paper remain largely the same. An article begins with a title, abstract, and keywords, which are followed by the main text, which includes the IMRAD – introduction, methods, results and discussion, and ends with the conclusion and references.

It is a brief summary of the research article which helps the readers understand the purpose, methods, and results of the study. Although an abstract may provide a brief overview of the study, the full text of the article needs to be read and evaluated for a thorough understanding. There are two types of abstracts, namely structured and unstructured. A structured abstract comprises different sections typically labelled as background/purpose, methods, results, and conclusion, whereas an unstructured abstract is not divided into these sections.

Introduction

The introduction of a research paper familiarizes the reader with the topic. It refers to the current evidence in the particular subject and the possible lacunae which necessitate the present study. In other words, the introduction puts the study in perspective. The findings of other related studies have to be quoted and referenced, especially their central statements. The introduction also needs to justify the appropriateness of the chosen study.[ 11 ]

This section highlights the procedure followed while conducting the study. It provides all the data necessary for the study's appraisal and lays out the study design which is paramount. For clinical research articles, this section should describe the participant or patient/population/problem (P), intervention (I), comparison (C), outcome (O), and study design (S) PICO(S), generally referred to as the PICO(S) framework [ Table 1 ].

Study designs and levels of evidence

Study designs are broadly divided into descriptive and interventional studies,[ 12 ] which can be further subdivided as shown in Figure 1 . Each study design has its own characteristics and should be used in the appropriate setting. The various study designs form the building blocks of evidence. This in turn justifies the need for a hierarchical classification of evidence, referred to as “Levels of Evidence,” as it forms the cornerstone of EBM [ Table 2 ]. Most medical journals now mandate that the submitted manuscript conform to and comply with the clinical research reporting statements and guidelines as applicable to the study design [ Table 3 ] to maintain clarity, transparency, and reproducibility and ensure comparability across different studies asking the same research question. As per the study design, the appropriate descriptive and inferential statistical analyses should be specified in the statistical plan. For prospective studies, a clear mention of sample size calculation (depending on the type of study, power, alpha error, meaningful difference, and variance) is mandatory, so as to identify whether the study was adequately powered.[ 13 ] The endpoints (primary, secondary, and exploratory, if any) should be mentioned clearly along with the exact methods used for the measurement of the variables.

Statistical testing

The statistical framework of any research study is commonly based on testing the null hypothesis, wherein the results are deemed significant by comparing P values obtained from an experimental dataset to a predefined significance level (0.05 being the most popular choice). By definition, P value is the probability under the specified statistical model to obtain a statistical summary equal to or more extreme than the one computed from the data and can range from 0 to 1. P < 0.05 indicates that results are unlikely to be due to chance alone. Unfortunately, P value does not indicate the magnitude of the observed difference, which may also be desirable. An alternative and complementary approach is the use of confidence intervals (CI), which is a range of values calculated from the observed data, that is likely to contain the true value at a specified probability. The probability is chosen by the investigator, and it is set customarily at 95% (1– alpha error of 0.05). CI provides information that may be used to test hypotheses; additionally, they provide information related to the precision, power, sample size, and effect size.

This section contains the findings of the study, presented clearly and objectively. The results obtained using the descriptive and inferential statistical analyses (as mentioned in the methods section) should be described. The use of tables and figures, including graphical representation [ Table 4 ], is encouraged to improve the clarity;[ 14 ] however, the duplication of these data in the text should be avoided.

The discussion section presents the authors' interpretations of the obtained results. This section includes:

• A comparison of the study results with what is currently known, drawing similarities and differences
• Novel findings of the study that have added to the existing body of knowledge
• Caveats and limitations.

It is imperative that the key relevant references are cited in any research paper in the appropriate format which allows the readers to access the original source of the specified statement or evidence. A brief look at the reference list gives an overview of how well the indexed medical literature was searched for the purpose of writing the manuscript.

Overall assessment

After a careful assessment of the various sections of a research article, it is necessary to assess the relevance of the study findings to the present scenario and weigh the potential benefits and drawbacks of its application to the population. In this context, it is necessary that the integrity of the intervention be noted. This can be verified by assessing the factors such as adherence to the specified program, the exposure needed, quality of delivery, participant responsiveness, and potential contamination. This relates to the feasibility of applying the intervention to the community.

BIAS IN CLINICAL RESEARCH

Research articles are the media through which science is communicated, and it is necessary that we adhere to the basic principles of transparency and accuracy when communicating our findings. Any such trend or deviation from the truth in data collection, analysis, interpretation, or publication is called bias.[ 15 ] This may lead to erroneous conclusions, and hence, all scientists and clinicians must be aware of the bias and employ all possible measures to mitigate it.

The extent to which a study is free from bias defines its internal validity. Internal validity is different from the external validity and precision. The external validity of a study is about its generalizability or applicability (depends on the purpose of the study), while precision is the extent to which a study is free from random errors (depends on the number of participants). A study is irrelevant without internal validity even if it is applicable and precise.[ 16 ] A bias can be introduced at every step in the flow of a study [ Figure 2 ].

The various types of biases in clinical research include:

• Selection bias: This happens while recruiting patients. This may lead to the differences in the way patients are accepted or rejected for a trial and the way in which interventions are assigned to the individuals. We need to assess whether the study population is a true representative of the target population. Furthermore, when there is no or an inadequate sequence generation, it can result in the over-estimation of treatment effects compared to randomized trials.[ 14 ] This can be mitigated by using a process called randomization. Randomization is the process of assigning clinical trial participants to treatment groups, such that each participant has an equal chance of being assigned to a particular group. This process should be completely random (e.g., tossing a coin, using a computer program, and throwing dice). When the process is not exactly random (e.g., randomization by date of birth, odd-even numbers, alternation, registration date, etc.), there is a significant potential for a selection bias
• Allocation bias: This is a bias that sets in when the person responsible for the study also allocates the treatment. It is known that inadequate or unclear concealment of allocation can lead to an overestimation of the treatment effects.[ 17 ] Adequate allocation concealment helps in mitigating this bias. This can be done by sequentially numbering identical drug containers or through central allocation by a person not involved in study enrollment
• Confounding bias: Having an effect on the dependent and independent variables through a spurious association, confounding factors can introduce a significant bias. Hence, the baseline characteristics need to be similar in the groups being compared. Known confounders can be managed during the selection process by stratified randomization (in randomized trials) and matching (in observational studies) or during analysis by meta-regression.[ 18 ] However, the unknown confounders can be minimized only through randomization
• Performance bias: This is a bias that is introduced because of the knowledge about the intervention allocation in the patient, investigator, or outcome assessor. This results in ascertainment or recall bias (patient), reporting bias (investigator), and detection bias (outcome assessor), all of which can lead to an overestimation of the treatment effects.[ 17 ] This can be mitigated by blinding – a process in which the treatment allocation is hidden from the patient, investigator, and/or outcome assessor. However, it has to be noted that blinding may not be practical or possible in all kinds of clinical trials
• Method bias: In clinical trials, it is necessary that the outcomes be assessed and recorded using valid and reliable tools, the lack of which can introduce a method bias[ 19 ]
• Attrition bias: This is a bias that is introduced because of the systematic differences between the groups in the loss of participants from the study. It is necessary to describe the completeness of the outcomes including the exclusions (along with the reasons), loss to follow-up, and drop-outs from the analysis
• Other bias: This includes any important concerns about biases not covered in the other domains.

Trial registration

In the recent times, it has become an ethical as well as a regulatory requirement in most countries to register the clinical trials prospectively before the enrollment of the first subject. Registration of a clinical trial is defined as the publication of an internationally agreed upon set of information about the design, conduct, and administration of any clinical trial on a publicly accessible website managed by a registry conforming to international standards. Apart from improving the awareness and visibility of the study, registration ensures transparency in the conduct and reduces publication bias and selective reporting. Some of the common sites are the ClinicalTrials. gov run by the National Library of Medicine of the National Institutes of Health (), Clinical Trials Registry-India () run by the Indian Council of Medical Research, and the International Clinical Trials Registry Platform () run by the World Health Organization.

Tools for critical appraisal

Several tools have been developed to assess the transparency of the scientific research papers and the degree of congruence of the research question with the study in the context of the various sections listed above [ Table 5 ].

Ethical considerations

Bad ethics cannot produce good science. Therefore, all scientific research must follow the ethical principles laid out in the declaration of Helsinki. For clinical research, it is mandatory that team members be trained in good clinical practice, familiarize themselves with clinical research methodology, and follow standard operating procedures as prescribed. Although the regulatory framework and landscape may vary to a certain extent depending upon the country where the research work is conducted, it is the responsibility of the Institutional Review Boards/Institutional Ethics Committees to provide study oversight such that the safety, well-being, and rights of the participants are adequately protected.

CONCLUSIONS

Critical appraisal is the systematic examination of the research evidence reported in the scientific articles to assess their validity, reliability, and applicability before using their findings to inform decision-making. It should be considered as the first step to grade the quality of evidence.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

• Cited Here |
• Google Scholar

Appraisal; bias; clinical study; evidence-based medicine; guidelines; tools

• + Favorites
• View in Gallery

Readers Of this Article Also Read

Epidemiological studies of risk factors could aid in designing risk....

• Research article
• Open access
• Published: 16 September 2004

A systematic review of the content of critical appraisal tools

• Persis Katrak 1 ,
• Andrea E Bialocerkowski 2 ,
• Nicola Massy-Westropp 1 ,
• VS Saravana Kumar 1 &
• Karen A Grimmer 1  

BMC Medical Research Methodology volume  4 , Article number:  22 ( 2004 ) Cite this article

160k Accesses

208 Citations

11 Altmetric

Metrics details

Consumers of research (researchers, administrators, educators and clinicians) frequently use standard critical appraisal tools to evaluate the quality of published research reports. However, there is no consensus regarding the most appropriate critical appraisal tool for allied health research. We summarized the content, intent, construction and psychometric properties of published, currently available critical appraisal tools to identify common elements and their relevance to allied health research.

A systematic review was undertaken of 121 published critical appraisal tools sourced from 108 papers located on electronic databases and the Internet. The tools were classified according to the study design for which they were intended. Their items were then classified into one of 12 criteria based on their intent. Commonly occurring items were identified. The empirical basis for construction of the tool, the method by which overall quality of the study was established, the psychometric properties of the critical appraisal tools and whether guidelines were provided for their use were also recorded.

Eighty-seven percent of critical appraisal tools were specific to a research design, with most tools having been developed for experimental studies. There was considerable variability in items contained in the critical appraisal tools. Twelve percent of available tools were developed using specified empirical research. Forty-nine percent of the critical appraisal tools summarized the quality appraisal into a numeric summary score. Few critical appraisal tools had documented evidence of validity of their items, or reliability of use. Guidelines regarding administration of the tools were provided in 43% of cases.

Conclusions

There was considerable variability in intent, components, construction and psychometric properties of published critical appraisal tools for research reports. There is no "gold standard' critical appraisal tool for any study design, nor is there any widely accepted generic tool that can be applied equally well across study types. No tool was specific to allied health research requirements. Thus interpretation of critical appraisal of research reports currently needs to be considered in light of the properties and intent of the critical appraisal tool chosen for the task.

Peer Review reports

Consumers of research (clinicians, researchers, educators, administrators) frequently use standard critical appraisal tools to evaluate the quality and utility of published research reports [ 1 ]. Critical appraisal tools provide analytical evaluations of the quality of the study, in particular the methods applied to minimise biases in a research project [ 2 ]. As these factors potentially influence study results, and the way that the study findings are interpreted, this information is vital for consumers of research to ascertain whether the results of the study can be believed, and transferred appropriately into other environments, such as policy, further research studies, education or clinical practice. Hence, choosing an appropriate critical appraisal tool is an important component of evidence-based practice.

Although the importance of critical appraisal tools has been acknowledged [ 1 , 3 – 5 ] there appears to be no consensus regarding the 'gold standard' tool for any medical evidence. In addition, it seems that consumers of research are faced with a large number of critical appraisal tools from which to choose. This is evidenced by the recent report by the Agency for Health Research Quality in which 93 critical appraisal tools for quantitative studies were identified [ 6 ]. Such choice may pose problems for research consumers, as dissimilar findings may well be the result when different critical appraisal tools are used to evaluate the same research report [ 6 ].

Critical appraisal tools can be broadly classified into those that are research design-specific and those that are generic. Design-specific tools contain items that address methodological issues that are unique to the research design [ 5 , 7 ]. This precludes comparison however of the quality of different study designs [ 8 ]. To attempt to overcome this limitation, generic critical appraisal tools have been developed, in an attempt to enhance the ability of research consumers to synthesise evidence from a range of quantitative and or qualitative study designs (for instance [ 9 ]). There is no evidence that generic critical appraisal tools and design-specific tools provide a comparative evaluation of research designs.

Moreover, there appears to be little consensus regarding the most appropriate items that should be contained within any critical appraisal tool. This paper is concerned primarily with critical appraisal tools that address the unique properties of allied health care and research [ 10 ]. This approach was taken because of the unique nature of allied health contacts with patients, and because evidence-based practice is an emerging area in allied health [ 10 ]. The availability of so many critical appraisal tools (for instance [ 6 ]) may well prove daunting for allied health practitioners who are learning to critically appraise research in their area of interest. For the purposes of this evaluation, allied health is defined as encompassing "...all occasions of service to non admitted patients where services are provided at units/clinics providing treatment/counseling to patients. These include units primarily concerned with physiotherapy, speech therapy, family panning, dietary advice, optometry occupational therapy..." [ 11 ].

The unique nature of allied health practice needs to be considered in allied health research. Allied health research thus differs from most medical research, with respect to:

• the paradigm underpinning comprehensive and clinically-reasoned descriptions of diagnosis (including validity and reliability). An example of this is in research into low back pain, where instead of diagnosis being made on location and chronicity of pain (as is common) [ 12 ], it would be made on the spinal structure and the nature of the dysfunction underpinning the symptoms, which is arrived at by a staged and replicable clinical reasoning process [ 10 , 13 ].

• the frequent use of multiple interventions within the one contact with the patient (an occasion of service), each of which requires appropriate description in terms of relationship to the diagnosis, nature, intensity, frequency, type of instruction provided to the patient, and the order in which the interventions were applied [ 13 ]

• the timeframe and frequency of contact with the patient (as many allied health disciplines treat patients in episodes of care that contain multiple occasions of service, and which can span many weeks, or even years in the case of chronic problems [ 14 ])

• measures of outcome, including appropriate methods and timeframes of measuring change in impairment, function, disability and handicap that address the needs of different stakeholders (patients, therapists, funders etc) [ 10 , 12 , 13 ].

Search strategy

In supplementary data [see additional file 1 ].

Data organization and extraction

Two independent researchers (PK, NMW) participated in all aspects of this review, and they compared and discussed their findings with respect to inclusion of critical appraisal tools, their intent, components, data extraction and item classification, construction and psychometric properties. Disagreements were resolved by discussion with a third member of the team (KG).

Data extraction consisted of a four-staged process. First, identical replica critical appraisal tools were identified and removed prior to analysis. The remaining critical appraisal tools were then classified according to the study design for which they were intended to be used [ 1 , 2 ]. The scientific manner in which the tools had been constructed was classified as whether an empirical research approach has been used, and if so, which type of research had been undertaken. Finally, the items contained in each critical appraisal tool were extracted and classified into one of eleven groups, which were based on the criteria described by Clarke and Oxman [ 4 ] as:

• Study aims and justification

• Methodology used , which encompassed method of identification of relevant studies and adherence to study protocol;

• Sample selection , which ranged from inclusion and exclusion criteria, to homogeneity of groups;

• Method of randomization and allocation blinding;

• Attrition : response and drop out rates;

• Blinding of the clinician, assessor, patient and statistician as well as the method of blinding;

• Outcome measure characteristics;

• Intervention or exposure details;

• Method of data analyses ;

• Potential sources of bias ; and

• Issues of external validity , which ranged from application of evidence to other settings to the relationship between benefits, cost and harm.

An additional group, " miscellaneous ", was used to describe items that could not be classified into any of the groups listed above.

Data synthesis

Data was synthesized using MS Excel spread sheets as well as narrative format by describing the number of critical appraisal tools per study design and the type of items they contained. Descriptions were made of the method by which the overall quality of the study was determined, evidence regarding the psychometric properties of the tools (validity and reliability) and whether guidelines were provided for use of the critical appraisal tool.

One hundred and ninety-three research reports that potentially provided a description of a critical appraisal tool (or process) were identified from the search strategy. Fifty-six of these papers were unavailable for review due to outdated Internet links, or inability to source the relevant journal through Australian university and Government library databases. Of the 127 papers retrieved, 19 were excluded from this review, as they did not provide a description of the critical appraisal tool used, or were published in languages other than English. As a result, 108 papers were reviewed, which yielded 121 different critical appraisal tools [ 1 – 5 , 7 , 9 , 15 – 102 , 116 ].

Empirical basis for tool construction

We identified 14 instruments (12% all tools) which were reported as having been constructed using a specified empirical approach [ 20 , 29 , 30 , 32 , 35 , 40 , 49 , 51 , 70 – 72 , 79 , 103 , 116 ]. The empirical research reflected descriptive and/or qualitative approaches, these being critical review of existing tools [ 40 , 72 ], Delphi techniques to identify then refine data items [ 32 , 51 , 71 ], questionnaires and other forms of written surveys to identify and refine data items [ 70 , 79 , 103 ], facilitated structured consensus meetings [ 20 , 29 , 30 , 35 , 40 , 49 , 70 , 72 , 79 , 116 ], and pilot validation testing [ 20 , 40 , 72 , 103 , 116 ]. In all the studies which reported developing critical appraisal tools using a consensus approach, a range of stakeholder input was sought, reflecting researchers and clinicians in a range of health disciplines, students, educators and consumers. There were a further 31 papers which cited other studies as the source of the tool used in the review, but which provided no information on why individual items had been chosen, or whether (or how) they had been modified. Moreover, for 21 of these tools, the cited sources of the critical appraisal tool did not report the empirical basis on which the tool had been constructed.

Critical appraisal tools per study design

Seventy-eight percent (N = 94) of the critical appraisal tools were developed for use on primary research [ 1 – 5 , 7 , 9 , 18 , 19 , 25 – 27 , 34 , 37 – 41 ], while the remainder (N = 26) were for secondary research (systematic reviews and meta-analyses) [ 2 – 5 , 15 – 36 , 116 ]. Eighty-seven percent (N = 104) of all critical appraisal tools were design-specific [ 2 – 5 , 7 , 9 , 15 – 90 ], with over one third (N = 45) developed for experimental studies (randomized controlled trials, clinical trials) [ 2 – 4 , 25 – 27 , 34 , 37 – 73 ]. Sixteen critical appraisal tools were generic. Of these, six were developed for use on both experimental and observational studies [ 9 , 91 – 95 ], whereas 11 were purported to be useful for any qualitative and quantitative research design [ 1 , 18 , 41 , 96 – 102 , 116 ] (see Figure 1 , Table 1 ).

Number of critical appraisal tools per study design [1,2]

Critical appraisal items

One thousand, four hundred and seventy five items were extracted from these critical appraisal tools. After grouping like items together, 173 different item types were identified, with the most frequently reported items being focused towards assessing the external validity of the study (N = 35) and method of data analyses (N = 28) (Table 2 ). The most frequently reported items across all critical appraisal tools were:

Eligibility criteria (inclusion/exclusion criteria) (N = 63)

Appropriate statistical analyses (N = 47)

Random allocation of subjects (N = 43)

Consideration of outcome measures used (N = 43)

Sample size justification/power calculations (N = 39)

Study design reported (N = 36)

Assessor blinding (N = 36)

Design-specific critical appraisal tools

Systematic reviews.

Eighty-seven different items were extracted from the 26 critical appraisal tools, which were designed to evaluate the quality of systematic reviews. These critical appraisal tools frequently contained items regarding data analyses and issues of external validity (Tables 2 and 3 ).

Items assessing data analyses were focused to the methods used to summarize the results, assessment of sensitivity of results and whether heterogeneity was considered, whereas the nature of reporting of the main results, interpretation of them and their generalizability were frequently used to assess the external validity of the study findings. Moreover, systematic review critical appraisal tools tended to contain items such as identification of relevant studies, search strategy used, number of studies included and protocol adherence, that would not be relevant for other study designs. Blinding and randomisation procedures were rarely included in these critical appraisal tools.

Experimental studies

One hundred and twenty thirteen different items were extracted from the 45 experimental critical appraisal tools. These items most frequently assessed aspects of data analyses and blinding (Tables 1 and 2 ). Data analyses items were focused on whether appropriate statistical analysis was performed, whether a sample size justification or power calculation was provided and whether side effects of the intervention were recorded and analysed. Blinding was focused on whether the participant, clinician and assessor were blinded to the intervention.

Diagnostic studies

Forty-seven different items were extracted from the seven diagnostic critical appraisal tools. These items frequently addressed issues involving data analyses, external validity of results and sample selection that were specific to diagnostic studies (whether the diagnostic criteria were defined, definition of the "gold" standard, the calculation of sensitivity and specificity) (Tables 1 and 2 ).

Observational studies

Seventy-four different items were extracted from the 19 critical appraisal tools for observational studies. These items primarily focused on aspects of data analyses (see Tables 1 and 2 , such as whether confounders were considered in the analysis, whether a sample size justification or power calculation was provided and whether appropriate statistical analyses were preformed.

Qualitative studies

Thirty-six different items were extracted from the seven qualitative study critical appraisal tools. The majority of these items assessed issues regarding external validity, methods of data analyses and the aims and justification of the study (Tables 1 and 2 ). Specifically, items were focused to whether the study question was clearly stated, whether data analyses were clearly described and appropriate, and application of the study findings to the clinical setting. Qualitative critical appraisal tools did not contain items regarding sample selection, randomization, blinding, intervention or bias, perhaps because these issues are not relevant to the qualitative paradigm.

Generic critical appraisal tools

Experimental and observational studies.

Forty-two different items were extracted from the six critical appraisal tools that could be used to evaluate experimental and observational studies. These tools most frequently contained items that addressed aspects of sample selection (such as inclusion/exclusion criteria of participants, homogeneity of participants at baseline) and data analyses (such as whether appropriate statistical analyses were performed, whether a justification of the sample size or power calculation were provided).

All study designs

Seventy-eight different items were contained in the ten critical appraisal tools that could be used for all study designs (quantitative and qualitative). The majority of these items focused on whether appropriate data analyses were undertaken (such as whether confounders were considered in the analysis, whether a sample size justification or power calculation was provided and whether appropriate statistical analyses were preformed) and external validity issues (generalization of results to the population, value of the research findings) (see Tables 1 and 2 ).

Allied health critical appraisal tools

We found no critical appraisal instrument specific to allied health research, despite finding at least seven critical appraisal instruments associated with allied health topics (mostly physiotherapy management of orthopedic conditions) [ 37 , 39 , 52 , 58 , 59 , 65 ]. One critical appraisal development group proposed two instruments [ 9 ], specific to quantitative and qualitative research respectively. The core elements of allied health research quality (specific diagnosis criteria, intervention descriptions, nature of patient contact and appropriate outcome measures) were not addressed in any one tool sourced for this evaluation. We identified 152 different ways of considering quality reporting of outcome measures in the 121 critical appraisal tools, and 81 ways of considering description of interventions. Very few tools which were not specifically targeted to diagnostic studies (less than 10% of the remaining tools) addressed diagnostic criteria. The critical appraisal instrument that seemed most related to allied health research quality [ 39 ] sought comprehensive evaluation of elements of intervention and outcome, however this instrument was relevant only to physiotherapeutic orthopedic experimental research.

Overall study quality

Forty-nine percent (N = 58) of critical appraisal tools summarised the results of the quality appraisal into a single numeric summary score [ 5 , 7 , 15 – 25 , 37 – 59 , 74 – 77 , 80 – 83 , 87 , 91 – 93 , 96 , 97 ] (Figure 2 ). This was achieved by one of two methods:

Number of critical appraisal tools with, and without, summary quality scores

An equal weighting system, where one point was allocated to each item fulfilled; or

A weighted system, where fulfilled items were allocated various points depending on their perceived importance.

However, there was no justification provided for any of the scoring systems used. In the remaining critical appraisal tools (N = 62), a single numerical summary score was not provided [ 1 – 4 , 9 , 25 – 36 , 60 – 73 , 78 , 79 , 84 – 90 , 94 , 95 , 98 – 102 ]. This left the research consumer to summarize the results of the appraisal in a narrative manner, without the assistance of a standard approach.

Psychometric properties of critical appraisal tools

Few critical appraisal tools had documented evidence of their validity and reliability. Face validity was established in nine critical appraisal tools, seven of which were developed for use on experimental studies [ 38 , 40 , 45 , 49 , 51 , 63 , 70 ] and two for systematic reviews [ 32 , 103 ]. Intra-rater reliability was established for only one critical appraisal tool as part of its empirical development process [ 40 ], whereas inter-rater reliability was reported for two systematic review tools [ 20 , 36 ] (for one of these as part of the developmental process [ 20 ]) and seven experimental critical appraisal tools [ 38 , 40 , 45 , 51 , 55 , 56 , 63 ] (for two of these as part of the developmental process [ 40 , 51 ]).

Critical appraisal tool guidelines

Forty-three percent (N = 52) of critical appraisal tools had guidelines that informed the user of the interpretation of each item contained within them (Table 2 ). These guidelines were most frequently in the form of a handbook or published paper (N = 31) [ 2 , 4 , 9 , 15 , 20 , 25 , 28 , 29 , 31 , 36 , 37 , 41 , 50 , 64 – 67 , 69 , 80 , 84 – 87 , 89 , 90 , 95 , 100 , 116 ], whereas in 14 critical appraisal tools explanations accompanied each item [ 16 , 26 , 27 , 40 , 49 , 51 , 57 , 59 , 79 , 83 , 91 , 102 ].

Our search strategy identified a large number of published critical appraisal tools that are currently available to critically appraise research reports. There was a distinct lack of information on tool development processes in most cases. Many of the tools were reported to be modifications of other published tools, or reflected specialty concerns in specific clinical or research areas, without attempts to justify inclusion criteria. Less than 10 of these tools were relevant to evaluation of the quality of allied health research, and none of these were based on an empirical research approach. We are concerned that although our search was systematic and extensive [ 104 , 105 ], our broad key words and our lack of ready access to 29% of potentially useful papers (N = 56) potentially constrained us from identifying all published critical appraisal tools. However, consumers of research seeking critical appraisal instruments are not likely to seek instruments from outdated Internet links and unobtainable journals, thus we believe that we identified the most readily available instruments. Thus, despite the limitations on sourcing all possible tools, we believe that this paper presents a useful synthesis of the readily available critical appraisal tools.

The majority of the critical appraisal tools were developed for a specific research design (87%), with most designed for use on experimental studies (38% of all critical appraisal tools sourced). This finding is not surprising as, according to the medical model, experimental studies sit at or near the top of the hierarchy of evidence [ 2 , 8 ]. In recent years, allied health researchers have strived to apply the medical model of research to their own discipline by conducting experimental research, often by using the randomized controlled trial design [ 106 ]. This trend may be the reason for the development of experimental critical appraisal tools reported in allied health-specific research topics [ 37 , 39 , 52 , 58 , 59 , 65 ].

We also found a considerable number of critical appraisal tools for systematic reviews (N = 26), which reflects the trend to synthesize research evidence to make it relevant for clinicians [ 105 , 107 ]. Systematic review critical appraisal tools contained unique items (such as identification of relevant studies, search strategy used, number of studies included, protocol adherence) compared with tools used for primary studies, a reflection of the secondary nature of data synthesis and analysis.

In contrast, we identified very few qualitative study critical appraisal tools, despite the presence of many journal-specific guidelines that outline important methodological aspects required in a manuscript submitted for publication [ 108 – 110 ]. This finding may reflect the more traditional, quantitative focus of allied health research [ 111 ]. Alternatively, qualitative researchers may view the robustness of their research findings in different terms compared with quantitative researchers [ 112 , 113 ]. Hence the use of critical appraisal tools may be less appropriate for the qualitative paradigm. This requires further consideration.

Of the small number of generic critical appraisal tools, we found few that could be usefully applied (to any health research, and specifically to the allied health literature), because of the generalist nature of their items, variable interpretation (and applicability) of items across research designs, and/or lack of summary scores. Whilst these types of tools potentially facilitate the synthesis of evidence across allied health research designs for clinicians, their lack of specificity in asking the 'hard' questions about research quality related to research design also potentially precludes their adoption for allied health evidence-based practice. At present, the gold standard study design when synthesizing evidence is the randomized controlled trial [ 4 ], which underpins our finding that experimental critical appraisal tools predominated in the allied health literature [ 37 , 39 , 52 , 58 , 59 , 65 ]. However, as more systematic literature reviews are undertaken on allied health topics, it may become more accepted that evidence in the form of other research design types requires acknowledgement, evaluation and synthesis. This may result in the development of more appropriate and clinically useful allied health critical appraisal tools.

A major finding of our study was the volume and variation in available critical appraisal tools. We found no gold standard critical appraisal tool for any type of study design. Therefore, consumers of research are faced with frustrating decisions when attempting to select the most appropriate tool for their needs. Variable quality evaluations may be produced when different critical appraisal tools are used on the same literature [ 6 ]. Thus, interpretation of critical analysis must be carefully considered in light of the critical appraisal tool used.

The variability in the content of critical appraisal tools could be accounted for by the lack of any empirical basis of tool construction, established validity of item construction, and the lack of a gold standard against which to compare new critical tools. As such, consumers of research cannot be certain that the content of published critical appraisal tools reflect the most important aspects of the quality of studies that they assess [ 114 ]. Moreover, there was little evidence of intra- or inter-rater reliability of the critical appraisal tools. Coupled with the lack of protocols for use, this may mean that critical appraisers could interpret instrument items in different ways over repeated occasions of use. This may produce variable results [123].

Based on the findings of this evaluation, we recommend that consumers of research should carefully select critical appraisal tools for their needs. The selected tools should have published evidence of the empirical basis for their construction, validity of items and reliability of interpretation, as well as guidelines for use, so that the tools can be applied and interpreted in a standardized manner. Our findings highlight the need for consensus to be reached regarding the important and core items for critical appraisal tools that will produce a more standardized environment for critical appraisal of research evidence. As a consequence, allied health research will specifically benefit from having critical appraisal tools that reflect best practice research approaches which embed specific research requirements of allied health disciplines.

National Health and Medical Research Council: How to Review the Evidence: Systematic Identification and Review of the Scientific Literature. Canberra. 2000

Google Scholar  

National Health and Medical Research Council: How to Use the Evidence: Assessment and Application of Scientific Evidence. Canberra. 2000

Joanna Briggs Institute. [ http://www.joannabriggs.edu.au ]

Clarke M, Oxman AD: Cochrane Reviewer's Handbook 4.2.0. 2003, Oxford: The Cochrane Collaboration

Crombie IK: The Pocket Guide to Critical Appraisal: A Handbook for Health Care Professionals. 1996, London: BMJ Publishing Group

Agency for Healthcare Research and Quality: Systems to Rate the Strength of Scientific Evidence. Evidence Report/Technology Assessment No. 47, Publication No. 02-E016. Rockville. 2002

Elwood JM: Critical Appraisal of Epidemiological Studies and Clinical Trials. 1998, Oxford: Oxford University Press, 2

Sackett DL, Richardson WS, Rosenberg W, Haynes RB: Evidence Based Medicine. How to Practice and Teach EBM. 2000, London: Churchill Livingstone

Critical literature reviews. [ http://www.cotfcanada.org/cotf_critical.htm ]

Bialocerkowski AE, Grimmer KA, Milanese SF, Kumar S: Application of current research evidence to clinical physiotherapy practice. J Allied Health Res Dec.

The National Health Data Dictionary – Version 10. http://www.aihw.gov.au/publications/hwi/nhdd12/nhdd12-v1.pdf and http://www.aihw.gov.au/publications/hwi/nhdd12/nhdd12-v2.pdf

Grimmer K, Bowman P, Roper J: Episodes of allied health outpatient care: an investigation of service delivery in acute public hospital settings. Disability and Rehabilitation. 2000, 22 (1/2): 80-87.

CAS   PubMed   Google Scholar  

Grimmer K, Milanese S, Bialocerkowski A: Clinical guidelines for low back pain: A physiotherapy perspective. Physiotherapy Canada. 2003, 55 (4): 1-9.

Grimmer KA, Milanese S, Bialocerkowski AE, Kumar S: Producing and implementing evidence in clinical practice: the therapies' dilemma. Physiotherapy. 2004,

Greenhalgh T: How to read a paper: papers that summarize other papers (systematic reviews and meta-analysis). BMJ. 1997, 315: 672-675.

CAS   PubMed   PubMed Central   Google Scholar  

Auperin A, Pignon J, Poynard T: Review article: critical review of meta-analysis of randomised clinical trials in hepatogastroenterology. Alimentary Pharmacol Therapeutics. 1997, 11: 215-225. 10.1046/j.1365-2036.1997.131302000.x.

CAS   Google Scholar  

Barnes DE, Bero LA: Why review articles on the health effects of passive smoking reach different conclusions. J Am Med Assoc. 1998, 279: 1566-1570. 10.1001/jama.279.19.1566.

Beck CT: Use of meta-analysis as a teaching strategy in nursing research courses. J Nurs Educat. 1997, 36: 87-90.

Carruthers SG, Larochelle P, Haynes RB, Petrasovits A, Schiffrin EL: Report of the Canadian Hypertension Society Consensus Conference: 1. Introduction. Can Med Assoc J. 1993, 149: 289-293.

Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchinson BG, Milner RA, Streiner DL: Agreement among reviewers of review articles. J Clin Epidemiol. 1991, 44: 91-98. 10.1016/0895-4356(91)90205-N.

Sacks HS, Reitman D, Pagano D, Kupelnick B: Meta-analysis: an update. Mount Sinai Journal of Medicine. 1996, 63: 216-224.

Smith AF: An analysis of review articles published in four anaesthesia journals. Can J Anaesth. 1997, 44: 405-409.

L'Abbe KA, Detsky AS, O'Rourke K: Meta-analysis in clinical research. Ann Intern Med. 1987, 107: 224-233.

PubMed   Google Scholar  

Mulrow CD, Antonio S: The medical review article: state of the science. Ann Intern Med. 1987, 106: 485-488.

Continuing Professional Development: A Manual for SIGN Guideline Developers. [ http://www.sign.ac.uk ]

Learning and Development Public Health Resources Unit. [ http://www.phru.nhs.uk/ ]

FOCUS Critical Appraisal Tool. [ http://www.focusproject.org.uk ]

Cook DJ, Sackett DL, Spitzer WO: Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on meta-analysis. J Clin Epidemiol. 1995, 48: 167-171. 10.1016/0895-4356(94)00172-M.

Cranney A, Tugwell P, Shea B, Wells G: Implications of OMERACT outcomes in arthritis and osteoporosis for Cochrane metaanalysis. J Rheumatol. 1997, 24: 1206-1207.

Guyatt GH, Sackett DL, Sinclair JC, Hoyward R, Cook DJ, Cook RJ: User's guide to the medical literature. IX. A method for grading health care recommendations. J Am Med Assoc. 1995, 274: 1800-1804. 10.1001/jama.274.22.1800.

Gyorkos TW, Tannenbaum TN, Abrahamowicz M, Oxman AD, Scott EAF, Milson ME, Rasooli Iris, Frank JW, Riben PD, Mathias RG: An approach to the development of practice guidelines for community health interventions. Can J Public Health. 1994, 85: S8-13.

Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF: Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet. 1999, 354: 1896-1900. 10.1016/S0140-6736(99)04149-5.

Oxman AD, Cook DJ, Guyatt GH: Users' guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group. J Am Med Assoc. 1994, 272: 1367-1371. 10.1001/jama.272.17.1367.

Pogue J, Yusuf S: Overcoming the limitations of current meta-analysis of randomised controlled trials. Lancet. 1998, 351: 47-52. 10.1016/S0140-6736(97)08461-4.

Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ, Sipe TA, Thacker SB: Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of observational studies in epidemiology (MOOSE) group. J Am Med Assoc. 2000, 283: 2008-2012. 10.1001/jama.283.15.2008.

Irwig L, Tosteson AN, Gatsonis C, Lau J, Colditz G, Chalmers TC, Mostellar F: Guidelines for meta-analyses evaluating diagnostic tests. Ann Intern Med. 1994, 120: 667-676.

Moseley AM, Herbert RD, Sherrington C, Maher CG: Evidence for physiotherapy practice: A survey of the Physiotherapy Evidence Database. Physiotherapy Evidence Database (PEDro). Australian Journal of Physiotherapy. 2002, 48: 43-50.

Cho MK, Bero LA: Instruments for assessing the quality of drug studies published in the medical literature. J Am Med Assoc. 1994, 272: 101-104. 10.1001/jama.272.2.101.

De Vet HCW, De Bie RA, Van der Heijden GJ, Verhagen AP, Sijpkes P, Kipschild PG: Systematic reviews on the basis of methodological criteria. Physiotherapy. 1997, 83: 284-289.

Downs SH, Black N: The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998, 52: 377-384.

Evans M, Pollock AV: A score system for evaluating random control clinical trials of prophylaxis of abdominal surgical wound infection. Br J Surg. 1985, 72: 256-260.

Fahey T, Hyde C, Milne R, Thorogood M: The type and quality of randomized controlled trials (RCTs) published in UK public health journals. J Public Health Med. 1995, 17: 469-474.

Gotzsche PC: Methodology and overt and hidden bias in reports of 196 double-blind trials of nonsteroidal antiinflammatory drugs in rheumatoid arthritis. Control Clin Trials. 1989, 10: 31-56. 10.1016/0197-2456(89)90017-2.

Imperiale TF, McCullough AJ: Do corticosteroids reduce mortality from alcoholic hepatitis? A meta-analysis of the randomized trials. Ann Int Med. 1990, 113: 299-307.

Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Control Clin Trials. 1996, 17: 1-12. 10.1016/0197-2456(95)00134-4.

Khan KS, Daya S, Collins JA, Walter SD: Empirical evidence of bias in infertility research: overestimation of treatment effect in crossover trials using pregnancy as the outcome measure. Fertil Steril. 1996, 65: 939-945.

Kleijnen J, Knipschild P, ter Riet G: Clinical trials of homoeopathy. BMJ. 1991, 302: 316-323.

Liberati A, Himel HN, Chalmers TC: A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol. 1986, 4: 942-951.

Moher D, Schulz KF, Altman DG, for the CONSORT Group: The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. J Am Med Assoc. 2001, 285: 1987-1991. 10.1001/jama.285.15.1987.

Reisch JS, Tyson JE, Mize SG: Aid to the evaluation of therapeutic studies. Pediatrics. 1989, 84: 815-827.

Sindhu F, Carpenter L, Seers K: Development of a tool to rate the quality assessment of randomized controlled trials using a Delphi technique. J Advanced Nurs. 1997, 25: 1262-1268. 10.1046/j.1365-2648.1997.19970251262.x.

Van der Heijden GJ, Van der Windt DA, Kleijnen J, Koes BW, Bouter LM: Steroid injections for shoulder disorders: a systematic review of randomized clinical trials. Br J Gen Pract. 1996, 46: 309-316.

Van Tulder MW, Koes BW, Bouter LM: Conservative treatment of acute and chronic nonspecific low back pain. A systematic review of randomized controlled trials of the most common interventions. Spine. 1997, 22: 2128-2156. 10.1097/00007632-199709150-00012.

Garbutt JC, West SL, Carey TS, Lohr KN, Crews FT: Pharmacotherapy for Alcohol Dependence. Evidence Report/Technology Assessment No. 3, AHCPR Publication No. 99-E004. Rockville. 1999

Oremus M, Wolfson C, Perrault A, Demers L, Momoli F, Moride Y: Interarter reliability of the modified Jadad quality scale for systematic reviews of Alzheimer's disease drug trials. Dement Geriatr Cognit Disord. 2001, 12: 232-236. 10.1159/000051263.

Clark O, Castro AA, Filho JV, Djubelgovic B: Interrater agreement of Jadad's scale. Annual Cochrane Colloqium Abstracts. 2001, [ http://www.biomedcentral.com/abstracts/COCHRANE/1/op031 ]October Lyon

Jonas W, Anderson RL, Crawford CC, Lyons JS: A systematic review of the quality of homeopathic clinical trials. BMC Alternative Medicine. 2001, 1: 12-10.1186/1472-6882-1-12.

Van Tulder M, Malmivaara A, Esmail R, Koes B: Exercises therapy for low back pain: a systematic review within the framework of the Cochrane Collaboration back review group. Spine. 2000, 25: 2784-2796. 10.1097/00007632-200011010-00011.

Van Tulder MW, Ostelo R, Vlaeyen JWS, Linton SJ, Morley SJ, Assendelft WJJ: Behavioral treatment for chronic low back pain: a systematic review within the framework of the cochrane back. Spine. 2000, 25: 2688-2699. 10.1097/00007632-200010150-00024.

Aronson N, Seidenfeld J, Samson DJ, Aronson N, Albertson PC, Bayoumi AM, Bennett C, Brown A, Garber ABA, Gere M, Hasselblad V, Wilt T, Ziegler MPHK, Pharm D: Relative Effectiveness and Cost Effectiveness of Methods of Androgen Suppression in the Treatment of Advanced Prostate Cancer. Evidence Report/Technology Assessment No. 4, AHCPR Publication No.99-E0012. Rockville. 1999

Chalmers TC, Smith H, Blackburn B, Silverman B, Schroeder B, Reitman D, Ambroz A: A method for assessing the quality of a randomized control trial. Control Clin Trials. 1981, 2: 31-49. 10.1016/0197-2456(81)90056-8.

der Simonian R, Charette LJ, McPeek B, Mosteller F: Reporting on methods in clinical trials. New Eng J Med. 1982, 306: 1332-1337.

Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbe KA: Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol. 1992, 45: 255-265. 10.1016/0895-4356(92)90085-2.

Goudas L, Carr DB, Bloch R, Balk E, Ioannidis JPA, Terrin MN: Management of Cancer Pain. Evidence Report/Technology Assessment No. 35 (Contract 290-97-0019 to the New England Medical Center), AHCPR Publication No. 99-E004. Rockville. 2000

Guyatt GH, Sackett DL, Cook DJ: Users' guides to the medical literature. II. How to use an article about therapy or prevention. A. Are the results of the study valid? Evidence-Based Medicine Working Group. J Am Med Assoc. 1993, 270: 2598-2601. 10.1001/jama.270.21.2598.

Khan KS, Ter Riet G, Glanville J, Sowden AJ, Kleijnen J: Undertaking Systematic Reviews of Research on Effectiveness: Centre of Reviews and Dissemination's Guidance for Carrying Out or Commissioning Reviews: York. 2000

McNamara R, Bass EB, Marlene R, Miller J: Management of New Onset Atrial Fibrillation. Evidence Report/Technology Assessment No.12, AHRQ Publication No. 01-E026. Rockville. 2001

Prendiville W, Elbourne D, Chalmers I: The effects of routine oxytocic administration in the management of the third stage of labour: an overview of the evidence from controlled trials. Br J Obstet Gynae Col. 1988, 95: 3-16.

Schulz KF, Chalmers I, Hayes RJ, Altman DG: Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. J Am Med Assoc. 1995, 273: 408-412. 10.1001/jama.273.5.408.

The Standards of Reporting Trials Group: A proposal for structured reporting of randomized controlled trials. J Am Med Assoc. 1994, 272: 1926-1931. 10.1001/jama.272.24.1926.

Verhagen AP, de Vet HC, de Bie RA, Kessels AGH, Boers M, Bouter LM, Knipschild PG: The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol. 1998, 51: 1235-1241. 10.1016/S0895-4356(98)00131-0.

Zaza S, Wright-De Aguero LK, Briss PA, Truman BI, Hopkins DP, Hennessy MH, Sosin DM, Anderson L, Carande-Kullis VG, Teutsch SM, Pappaioanou M: Data collection instrument and procedure for systematic reviews in the guide to community preventive services. Task force on community preventive services. Am J Prevent Med. 2000, 18: 44-74. 10.1016/S0749-3797(99)00122-1.

Haynes BB, Wilczynski N, McKibbon A, Walker CJ, Sinclair J: Developing optimal search strategies for detecting clinically sound studies in MEDLINE. J Am Informatics Assoc. 1994, 1: 447-458.

Greenhalgh T: How to read a paper: papers that report diagnostic or screening tests. BMJ. 1997, 315: 540-543.

Arroll B, Schechter MT, Sheps SB: The assessment of diagnostic tests: a comparison of medical literature in 1982 and 1985. J Gen Int Med. 1988, 3: 443-447.

Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, Bossuyt PM: Empirical evidence of design-related bias in studies of diagnostic tests. J Am Med Assoc. 1999, 282: 1061-1066. 10.1001/jama.282.11.1061.

Sheps SB, Schechter MT: The assessment of diagnostic tests. A survey of current medical research. J Am Med Assoc. 1984, 252: 2418-2422. 10.1001/jama.252.17.2418.

McCrory DC, Matchar DB, Bastian L, Dutta S, Hasselblad V, Hickey J, Myers MSE, Nanda K: Evaluation of Cervical Cytology. Evidence Report/Technology Assessment No. 5, AHCPR Publication No.99-E010. Rockville. 1999

Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig LM, Lijmer JG, Moher D, Rennie D, DeVet HCW: Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative. Clin Chem. 2003, 49: 1-6. 10.1373/49.1.1.

Greenhalgh T: How to Read a Paper: Assessing the methodological quality of published papers. BMJ. 1997, 315: 305-308.

Angelillo I, Villari P: Residential exposure to electromagnetic fields and childhood leukaemia: a meta-analysis. Bull World Health Org. 1999, 77: 906-915.

Ariens G, Mechelen W, Bongers P, Bouter L, Van der Wal G: Physical risk factors for neck pain. Scand J Work Environ Health. 2000, 26: 7-19.

Hoogendoorn WE, van Poppel MN, Bongers PM, Koes BW, Bouter LM: Physical load during work and leisure time as risk factors for back pain. Scand J Work Environ Health. 1999, 25: 387-403.

Laupacis A, Wells G, Richardson WS, Tugwell P: Users' guides to the medical literature. V. How to use an article about prognosis. Evidence-Based Medicine Working Group. J Am Med Assoc. 1994, 272: 234-237. 10.1001/jama.272.3.234.

Levine M, Walter S, Lee H, Haines T, Holbrook A, Moyer V: Users' guides to the medical literature. IV. How to use an article about harm. Evidence-Based Medicine Working Group. J Am Med Assoc. 1994, 271: 1615-1619. 10.1001/jama.271.20.1615.

Carey TS, Boden SD: A critical guide to case series reports. Spine. 2003, 28: 1631-1634. 10.1097/00007632-200308010-00001.

Greenhalgh T, Taylor R: How to read a paper: papers that go beyond numbers (qualitative research). BMJ. 1997, 315: 740-743.

Hoddinott P, Pill R: A review of recently published qualitative research in general practice. More methodological questions than answers?. Fam Pract. 1997, 14: 313-319. 10.1093/fampra/14.4.313.

Mays N, Pope C: Quality research in health care: Assessing quality in qualitative research. BMJ. 2000, 320: 50-52. 10.1136/bmj.320.7226.50.

Mays N, Pope C: Rigour and qualitative research. BMJ. 1995, 311: 109-112.

Colditz GA, Miller JN, Mosteller F: How study design affects outcomes in comparisons of therapy. I: Medical. Stats Med. 1989, 8: 441-454.

Turlik MA, Kushner D: Levels of evidence of articles in podiatric medical journals. J Am Pod Med Assoc. 2000, 90: 300-302.

Borghouts JAJ, Koes BW, Bouter LM: The clinical course and prognostic factors of non-specific neck pain: a systematic review. Pain. 1998, 77: 1-13. 10.1016/S0304-3959(98)00058-X.

Spitzer WO, Lawrence V, Dales R, Hill G, Archer MC, Clark P, Abenhaim L, Hardy J, Sampalis J, Pinfold SP, Morgan PP: Links between passive smoking and disease: a best-evidence synthesis. A report of the working group on passive smoking. Clin Invest Med. 1990, 13: 17-46.

Sutton AJ, Abrams KR, Jones DR, Sheldon TA, Song F: Systematic reviews of trials and other studies. Health Tech Assess. 1998, 2: 1-276.

Chestnut RM, Carney N, Maynard H, Patterson P, Mann NC, Helfand M: Rehabilitation for Traumatic Brain Injury. Evidence Report/Technology Assessment No. 2, Agency for Health Care Research and Quality Publication No. 99-E006. Rockville. 1999

Lohr KN, Carey TS: Assessing best evidence: issues in grading the quality of studies for systematic reviews. Joint Commission J Qual Improvement. 1999, 25: 470-479.

Greer N, Mosser G, Logan G, Halaas GW: A practical approach to evidence grading. Joint Commission J Qual Improvement. 2000, 26: 700-712.

Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, Atkins D: Current methods of the U.S. Preventive Services Task Force: a review of the process. Am J Prevent Med. 2001, 20: 21-35. 10.1016/S0749-3797(01)00261-6.

Anonymous: How to read clinical journals: IV. To determine etiology or causation. Can Med Assoc J. 1981, 124: 985-990.

Whitten PS, Mair FS, Haycox A, May CR, Williams TL, Hellmich S: Systematic review of cost effectiveness studies of telemedicine interventions. BMJ. 2002, 324: 1434-1437. 10.1136/bmj.324.7351.1434.

PubMed   PubMed Central   Google Scholar  

Forrest JL, Miller SA: Evidence-based decision making in action: Part 2-evaluating and applying the clinical evidence. J Contemp Dental Pract. 2002, 4: 42-52.

Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol. 1991, 44: 1271-1278. 10.1016/0895-4356(91)90160-B.

Jones T, Evans D: Conducting a systematic review. Aust Crit Care. 2000, 13: 66-71.

Papadopoulos M, Rheeder P: How to do a systematic literature review. South African J Physiother. 2000, 56: 3-6.

Selker LG: Clinical research in Allied Health. J Allied Health. 1994, 23: 201-228.

Stevens KR: Systematic reviews: the heart of evidence-based practice. AACN Clin Issues. 2001, 12: 529-538.

Devers KJ, Frankel RM: Getting qualitative research published. Ed Health. 2001, 14: 109-117. 10.1080/13576280010021888.

Canadian Journal of Public Health: Review guidelines for qualitative research papers submitted for consideration to the Canadian Journal of Public Health. Can J Pub Health. 2000, 91: I2-

Malterud K: Shared understanding of the qualitative research process: guidelines for the medical researcher. Fam Pract. 1993, 10: 201-206.

Higgs J, Titchen A: Research and knowledge. Physiotherapy. 1998, 84: 72-80.

Maggs-Rapport F: Best research practice: in pursuit of methodological rigour. J Advan Nurs. 2001, 35: 373-383. 10.1046/j.1365-2648.2001.01853.x.

Cutcliffe JR, McKenna HP: Establishing the credibility of qualitative research findings: the plot thickens. J Advan Nurs. 1999, 30: 374-380. 10.1046/j.1365-2648.1999.01090.x.

Andresen EM: Criteria for assessing the tools of disability outcomes research. Arch Phys Med Rehab. 2000, 81: S15-S20. 10.1053/apmr.2000.20619.

Beatie P: Measurement of health outcomes in the clinical setting: applications to physiotherapy. Phys Theory Pract. 2001, 17: 173-185. 10.1080/095939801317077632.

Charnock DF, (Ed): The DISCERN Handbook: Quality criteria for consumer health information on treatment choices. 1998, Radcliffe Medical Press

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/4/22/prepub

Download references

Author information

Authors and affiliations.

Centre for Allied Health Evidence: A Collaborating Centre of the Joanna Briggs Institute, City East Campus, University of South Australia, North Terrace, Adelaide, 5000, Australia

Persis Katrak, Nicola Massy-Westropp, VS Saravana Kumar & Karen A Grimmer

School of Physiotherapy, The University of Melbourne, Melbourne, 3010, Australia

Andrea E Bialocerkowski

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Karen A Grimmer .

Additional information

Competing interests.

No competing interests.

Authors' contributions

PK Sourced critical appraisal tools

Categorized the content and psychometric properties of critical appraisal tools

AEB Synthesis of findings

Drafted manuscript

NMW Sourced critical appraisal tools

VSK Sourced critical appraisal tools

KAG Study conception and design

Assisted with critiquing critical appraisal tools and categorization of the content and psychometric properties of critical appraisal tools

Drafted and reviewed manuscript

Addressed reviewer's comments and re-submitted the article

Electronic supplementary material

Additional file 1: search strategy. (doc 30 kb), authors’ original submitted files for images.

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, authors’ original file for figure 3, authors’ original file for figure 4, authors’ original file for figure 5, rights and permissions.

Reprints and permissions

About this article

Cite this article.

Katrak, P., Bialocerkowski, A.E., Massy-Westropp, N. et al. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol 4 , 22 (2004). https://doi.org/10.1186/1471-2288-4-22

Download citation

Received : 10 May 2004

Accepted : 16 September 2004

Published : 16 September 2004

DOI : https://doi.org/10.1186/1471-2288-4-22

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Research Consumer
• Empirical Basis
• Ally Health Literature
• Critical Appraisal Tool
• Sample Size Justification

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Medicine: A Brief Guide to Critical Appraisal

• About this guide
• First Year Library Essentials
• Literature Reviews and Data Management
• Systematic Search for Health This link opens in a new window
• Guide to Using EndNote This link opens in a new window
• A Brief Guide to Critical Appraisal
• Manage Research Data This link opens in a new window
• Articles & Databases
• Point of Care Tools
• Anatomy & Radiology
• Drugs and Medicines
• Diagnostic Tests & Calculators
• Health Statistics
• Multimedia Sources
• News & Public Opinion

Have you ever seen a news piece about a scientific breakthrough and wondered how accurate the reporting is? Or wondered about the research behind the headlines? This is the beginning of critical appraisal: thinking critically about what you see and hear, and asking questions to determine how much of a 'breakthrough' something really is.

The article " Is this study legit? 5 questions to ask when reading news stories of medical research " is a succinct introduction to the sorts of questions you should ask in these situations, but there's more than that when it comes to critical appraisal. Read on to learn more about this practical and crucial aspect of evidence-based practice.

What is Critical Appraisal?

Critical appraisal forms part of the process of evidence-based practice. “ Evidence-based practice across the health professions ” outlines the fives steps of this process. Critical appraisal is step three:

• Ask a question
• Access the information
• Appraise the articles found
• Apply the information

Critical appraisal is the examination of evidence to determine applicability to clinical practice. It considers (1) :

• Are the results of the study believable?
• Was the study methodologically sound?  
• What is the clinical importance of the study’s results?
• Are the findings sufficiently important? That is, are they practice-changing?  
• Are the results of the study applicable to your patient?
• Is your patient comparable to the population in the study?

Why Critically Appraise?

If practitioners hope to ‘stand on the shoulders of giants’, practicing in a manner that is responsive to the discoveries of the research community, then it makes sense for the responsible, critically thinking practitioner to consider the reliability, influence, and relevance of the evidence presented to them.

While critical thinking is valuable, it is also important to avoid treading too much into cynicism; in the words of Hoffman et al. (1):

… keep in mind that no research is perfect and that it is important not to be overly critical of research articles. An article just needs to be good enough to assist you to make a clinical decision.

How do I Critically Appraise?

Evidence-based practice is intended to be practical . To enable this, critical appraisal checklists have been developed to guide practitioners through the process in an efficient yet comprehensive manner.

Critical appraisal checklists guide the reader through the appraisal process by prompting the reader to ask certain questions of the paper they are appraising. There are many different critical appraisal checklists but the best apply certain questions based on what type of study the paper is describing. This allows for a more nuanced and appropriate appraisal. Wherever possible, choose the appraisal tool that best fits the study you are appraising.

Like many things in life, repetition builds confidence and the more you apply critical appraisal tools (like checklists) to the literature the more the process will become second nature for you and the more effective you will be.

How do I Identify Study Types?

Identifying the study type described in the paper is sometimes harder than it should be. Helpful papers spell out the study type in the title or abstract, but not all papers are helpful in this way. As such, the critical appraiser may need to do a little work to identify what type of study they are about to critique. Again, experience builds confidence, but understanding the typical features of common study types certainly helps.

To assist with this, the Library has produced a guide to study designs in health research .

The following selected references will help also with understanding study types but there are also other resources in the Library’s collection and freely available online:

• The “ How to read a paper ” article series from The BMJ is a well-known source for establishing an understanding of the features of different study types; this series was subsequently adapted into a book (“ How to read a paper: the basics of evidence-based medicine ”) which offers more depth and currency than that found in the articles. (2)  
• Chapter two of “ Evidence-based practice across the health professions ” briefly outlines some study types and their application; subsequent chapters go into more detail about different study types depending on what type of question they are exploring (intervention, diagnosis, prognosis, qualitative) along with systematic reviews.  
• “ Translational research and clinical practice: basic tools for medical decision making and self-learning ” unpacks the components of a paper, explaining their purpose along with key features of different study designs. (3)  
• The BMJ website contains the contents of the fourth edition of the book “ Epidemiology for the uninitiated ”. This eBook contains chapters exploring ecological studies, longitudinal studies, case-control and cross-sectional studies, and experimental studies.

Reporting Guidelines

In order to encourage consistency and quality, authors of reports on research should follow reporting guidelines when writing their papers. The EQUATOR Network is a good source of reporting guidelines for the main study types.

While these guidelines aren't critical appraisal tools as such, they can assist by prompting you to consider whether the reporting of the research is missing important elements.

Once you've identified the study type at hand, visit EQUATOR to find the associated reporting guidelines and ask yourself: does this paper meet the guideline for its study type?

Which Checklist Should I Use?

Determining which checklist to use ultimately comes down to finding an appraisal tool that:

• Fits best with the study you are appraising
• Is reliable, well-known or otherwise validated
• You understand and are comfortable using

Below are some sources of critical appraisal tools. These have been selected as they are known to be widely accepted, easily applicable, and relevant to appraisal of a typical journal article. You may find another tool that you prefer, which is acceptable as long as it is defensible:

• CASP (Critical Appraisal Skills Programme)
• JBI (Joanna Briggs Institute)
• CEBM (Centre for Evidence-Based Medicine)
• SIGN (Scottish Intercollegiate Guidelines Network)
• STROBE (Strengthing the Reporting of Observational Studies in Epidemiology)
• BMJ Best Practice

The information on this page has been compiled by the Medical Librarian. Please contact the Library's Health Team ( [email protected] ) for further assistance.

Reference list

1. Hoffmann T, Bennett S, Del Mar C. Evidence-based practice across the health professions. 2nd ed. Chatswood, N.S.W., Australia: Elsevier Churchill Livingston; 2013.

2. Greenhalgh T. How to read a paper: the basics of evidence-based medicine. 5th ed. Chichester, West Sussex: Wiley; 2014.

3.  Aronoff SC. Translational research and clinical practice: basic tools for medical decision making and self-learning. New York: Oxford University Press; 2011.

• << Previous: Guide to Using EndNote
• Next: Manage Research Data >>
• Last Updated: Sep 16, 2024 1:27 PM
• URL: https://deakin.libguides.com/medicine

CASP Checklists

How to use our CASP Checklists

Referencing and Creative Commons

• Online Training Courses
• CASP Workshops
• What is Critical Appraisal
• Study Designs
• Useful Links
• Bibliography
• View all Tools and Resources
• Testimonials

Critical Appraisal Checklists

We offer a number of free downloadable checklists to help you more easily and accurately perform critical appraisal across a number of different study types.

The CASP checklists are easy to understand but in case you need any further guidance on how they are structured, take a look at our guide on how to use our CASP checklists .

Systematic Reviews with Meta-Analysis of Observational Studies (BETA)

Systematic Reviews with Meta-Analysis of Randomised Controlled Trials (RCTs) (BETA)

Randomised Controlled Trial (RCT) Checklist

Systematic Review Checklist

Qualitative Studies Checklist

Cohort Study Checklist

Diagnostic Study Checklist

Case Control Study Checklist

Economic Evaluation Checklist

Clinical Prediction Rule Checklist

Need more information?

• Online Learning
• Privacy Policy

Critical Appraisal Skills Programme

Critical Appraisal Skills Programme (CASP) will use the information you provide on this form to be in touch with you and to provide updates and marketing. Please let us know all the ways you would like to hear from us:

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

Copyright 2024 CASP UK - OAP Ltd. All rights reserved Website by Beyond Your Brand

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Indian J Anaesth
• v.60(9); 2016 Sep

Critical appraisal of published literature

Goneppanavar umesh.

Department of Anaesthesia, Dharwad Institute of Mental Health and Neuro Sciences, Dharwad, Karnataka, India

John George Karippacheril

1 Department of Anaesthesiology, Universal Hospital, Abu Dhabi, UAE

Rahul Magazine

2 Department of Pulmonary Medicine, Kasturba Medical College, Manipal University, Manipal, Karnataka, India

With a large output of medical literature coming out every year, it is impossible for readers to read every article. Critical appraisal of scientific literature is an important skill to be mastered not only by academic medical professionals but also by those involved in clinical practice. Before incorporating changes into the management of their patients, a thorough evaluation of the current or published literature is an important step in clinical practice. It is necessary for assessing the published literature for its scientific validity and generalizability to the specific patient community and reader's work environment. Simple steps have been provided by Consolidated Standard for Reporting Trial statements, Scottish Intercollegiate Guidelines Network and several other resources which if implemented may help the reader to avoid reading flawed literature and prevent the incorporation of biased or untrustworthy information into our practice.

INTRODUCTION

Critical appraisal

‘ The process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context ’ -Burls A[ 1 ]

The objective of medical literature is to provide unbiased, accurate medical information, backed by robust scientific evidence that could aid and enhance patient care. With the ever increasing load of scientific literature (more than 12,000 new articles added every week to the MEDLINE database),[ 2 ] keeping abreast of the current literature can be arduous. Critical appraisal of literature may help distinguish between useful and flawed studies. Although substantial resources of peer-reviewed literature are available, flawed studies may abound in unreliable sources. Flawed studies if used to guide clinical decisions may end up with no benefit or at worse result in significant harm. Readers can, thus, make informed decisions by critically evaluating medical literature.

STEPS TO CRITICALLY EVALUATE AN ARTICLE

Initial evaluation of an article published in literature should be based on certain core questions. These may include querying what could be the key learning points in the article, about its clinical relevance, if the study has a robust methodology, if the results are reproducible and could there be any bias or conflict of interest [ Table 1 ]. If there are serious doubts regarding any of these steps, the reader could skip the article at this stage itself.

Core questions for initial evaluation of a scientific article

Introduction, methods, results and discussion pattern of scientific literature

Introduction.

Evaluate if the need (as dearth of studies on the topic in scientific literature) and the purpose of the study (attempting to find answers to one of the important unanswered queries of clinical relevance) are properly explained with scientific rationale. If the research objective and hypothesis were not clearly defined or the findings of the study are different from the objectives (chance findings), the study outcomes become questionable.

A good working scientific hypothesis backed by a strong methodology is the stepping stone for carrying out a meaningful research. The groups to be involved and the study end points should be determined prior to starting the study. Strong methodology depends on several aspects that must be properly addressed and evaluated [ Table 2 ]. The methodology for statistical analysis including tests for distribution pattern of study data, level of significance and sample size calculation should be clearly defined in the methods section. Data that violate the assumption of normal distribution pattern must be analysed with non-parametric statistical tests. Inadequate sample size can lead to false-negative results or beta error (aide-memoire: beta error is blindness). Setting a higher level of significance, especially when performing multiple comparisons, can lead to false-positive results or alpha error (aide-memoire: alpha error is hallucination). A confidence interval when used in the study methodology provides information on the direction and strength of the effect, in contrast to P values alone, from which the magnitude, direction or comparison of relative risk between groups cannot be inferred. P value simply accepts or rejects the null hypothesis, therefore it must be reported in conjunction with confidence intervals.[ 6 ] An important guideline for evaluating and reporting randomised controlled trials, mandatory for publication in several international medical journals, is the Consolidated Standard for Reporting Trial (CONSORT) statement.[ 7 ] Other scientific societies such as the Scottish Intercollegiate Guidelines Network have devised checklists that may aid in the critical evaluation of articles depending on the type of study methodology.[ 8 , 9 ]

Appraisal elements for robust evaluation of study methodology

The results section should only report the findings of the study, without attempting to reason them out. The total number of participants with the number of those excluded, dropped out or withdrawn from the study should be analysed. Failure to do so may lead to underestimation or overestimation of results.[ 10 ] A summary flowchart containing enrolment data could be created as per the CONSORT statement.[ 5 ] Actual values including the mean with standard deviation/error or median with interquartile range should be reported. Evaluate for completeness – all the variables in the study methodology should be analysed using appropriate statistical tests. Ensure that findings stated in the results are the same in other areas of the article – abstract, tables and figures. Appropriate tables and graphs should be used to provide the results of the study. Assess if the results of the study can be generalised and are useful to our workplace or patient population.

Although significant positive results from the study are more likely to be accepted for publication (publication bias), remember that high rates of falsely inflated results are demonstrated in studies that had flawed methodology (due to improper or lack of appropriate randomisation, allocation concealment, blinding and/or assessing outcomes through statistical modeling). Further, the publication bias towards studies with positive outcome leads to scientific distortions in the body of scientific knowledge.[ 11 , 12 ]

Discussion and conclusion

The discussion is used to account for or reason out the outcomes of the study, including dropouts and any change in methodology, to comment on the external validity of the study and to discuss its limitations. The authors should report their findings in comparison with that previously published in literature, if the study results added new information to the current literature, if it could alter patient management and if the findings need larger studies for further evaluation or confirmation. When concluding, the interpretation should be consistent with the actual findings. Evaluate if the questions in the study hypothesis were adequately addressed and if the conclusions were justified by the actual data. Authors should also provide limitations of their study and constructive suggestions for future research.

Readers may find useful resources on how to constructively read the published literature at the following resources:

• Consolidated Standard for Reporting Trial 2010 for randomised trials – http://www.consort-statement.org
• Sign checklists – http://www.sign.ac.uk/methodology/checklists.html
• BMJ series of articles – http://www.bmj.com/about-bmj/resources-readers/publications/how-read-paper
• Equator network for health research – http://www.equator-network.org
• Strobe statement for observational studies – http://www.strobe-statement.org/index.php?id=strobe-home
• Care for case reports – http://www.care-statement.org
• PRISMA statement for meta-analytical studies and systematic reviews: http://www.prisma-statement.org
• Agree – http://www.agreetrust.org
• CASP – http://www.casp-uk.net
• http://www.delfini.org .

Critical appraisal of scientific literature is an important skill to be mastered not just by academic medical professionals but also by those involved in clinical practice. Before incorporating changes in the management of their patients, a thorough evaluation of the current or published literature is a necessary step in practicing evidence-based medicine.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

Critical Appraisal : Critical appraisal full list of checklists and tools

• What is critical appraisal?
• Where to start
• Education and childhood studies
• Occupational Therapy
• Physiotherapy
• Interpreting statistics
• Further reading and resources

Which checklist or tool should I use?

There are hundreds of critical appraisal checklists and tools you can choose from, which can be very overwhelming. There are so many because there are many kinds of research, knowledge can be communicated in a wide range of ways, and whether something is appropriate to meet your information needs depends on your specific context. 

We have asked for recommendations from lecturers in different academic departments, to give you an idea about which checklists and tools may be the most relevant for you. Please hover over the drop-down menu at the top of the page, underneath 'Critical appraisal checklists and tools' to view the individual subject pages.

Below are lists of as many critical appraisal tools and checklists as we have been able to find. These are split into health sciences and social sciences because the two areas tend to take different approaches to evaluation, for various reasons!

To see a selection of checklists more suitable for your subject, hover over the top tab of this page.  

Critical appraisal checklists and tools for Health Sciences

• AACODS  Checklist for appraising grey literature
• AMSTAR 2  critical appraisal tool for systematic reviews that include randomised and non-randomised studies of healthcare interventions or both
• AOTA Critically Appraised Papers  American Occupational Therapy Association 
• Bandolier - "Evidence based thinking about healthcare"
• BestBETS critical appraisal worksheet
• BMJ critical appraisal checklists
• CASP  Critical Appraisal Skills Programme includes checklists for case control studies, clinical prediction rule, cohort studies, diagnostic studies, economic evaluation, qualitative studies, RCTs and systematic reviews
• Centre for Evidence Based Medicine (Oxford) Critical Appraisal Tools  CEBM's worksheets to assess systematic reviews, diagnostic, prognosis, and RCTs
• Centre for Evidence Based Medicine (Oxford) CATmaker and EBM calculator  CEBM's computer assisted critical appraisal tool CATmaker 
• CEMB critical appraisal sheets  (Centre for Evidence Based Medicine)
• Cochrane Assessing Risk of Bias in a Randomized Trial
• Critical appraisal: a checklist from Students for Best Evidence S4BE (student network with simple explanations of difficult concepts)
• Critical appraisal and statistical skills (Knowledge for Healthcare)
• Critical appraisal of clinical trials  from Testing Treatments International
• Critical appraisal of clinical trials (Medicines Learning Portal)
• Critical appraisal of quantitative research  
• Critical appraisal of a quantitative paper  from Teeside University
• Critical appraisal of a qualitative paper  from Teeside University
• Critical appraisal tools  from the Centre for Evidence-Based Medicine
• Critical Evaluation of Research Papers – Qualitative Studies from Teeside University
• Critical Evaluation of Research Papers – RCTs/Experimental Studies from Teeside University
• Evaluation tool for mixed methods study designs 
• GRADE - The Grading of Recommendations Assessment, Development and Evaluation working group  guidelines and publications for grading the quality of evidence in healthcare research and policy
• HCPRDU Evaluation Tool for Mixed Methods Studies  - University of Salford Health Care Practice R&D Unit 
• HCPRDU Evaluation Tool for Qualitative Studies  - University of Salford Health Care Practice R&D Unit 
• HCPRDU Evaluation Tool for Quantitative Studies  - University of Salford Health Care Practice R&D Unit 
• JBI Joanna Briggs Institute critical appraisal tools  checklists for Analytical cross sectional studies, case control studies, case reports, case series, cohort studies, diagnostic test accuracy, economic evaluations, prevalence studies, qualitative research, quasi-experimental (non-randomised) studies, RCTs, systematic reviews and for text and opinion  
• Knowledge Translation Program  - Toronto based KTP critical appraisal worksheets for systematic reviews, prognosis, diagnosis, harm and therapy
• MMAT Mixed Methods Appraisal Tool 
• McMaster University Evidence Based Practice Research Group quantitative and qualitative review forms
• NHLBI (National Heart, Blood and lung Institute) study quality assessment tools for case control studies, case series, controlled intervention, observational cohort and cross sectional studies, before-after (pre-post) studies with no control group, systematic reviews and meta analyses 
• NICE Guidelines, The Manual Appendix H. pp9-24
• QUADAS-2  tool for evaluating risk of bias in systematic reviews from the University of Bristol
• PEDro  PEDro (Physiotherapy Evidence Database) Scale - appraisal resources including a tutorial and appraisal tool
• RoB 2   A revised Cochrane risk-of-bias tool for randomized trials
• ROBINS-I Risk Of Bias In Non-Randomized Studies of Interventions 
• ROBIS  Risk of Bias in Systematic Reviews
• ROB-ME   A tool for assessing Risk Of Bias due to Missing Evidence in a synthesis
• SIGN  - Critical appraisal notes and checklists for case control studies, cohort studies, diagnostic studies, economic studies, RCTs, meta-analyses and systematic reviews
• Strength of Recommendation Taxonomy  - the SORT scale for quality, quantity and consistency of evidence in individual studies or bodies of evidence
• STROBE (Strengthening the Reporting of Observational studies in Epidemiology)  for cohort, case-control, and cross-sectional studies (combined),  cohort, case-control, cross-sectional studies and conference abstracts
• SURE Case Controlled Studies Critical Appraisal checklist
• SURE Case Series Studies Critical Appraisal checklist
• SURE Cohort Studies Critical Appraisal checklist
• SURE Cross-sectional Studies Critical Appraisal checklist
• SURE Experimental Studies Critical Appraisal checklist
• SURE Qualitative Studies Critical Appraisal checklist
• SURE Systematic Review Critical Appraisal checklist

Critical appraisal checklists and tools for Social Sciences

• AACODS   Checklist for appraising grey literature
• CRAAP test to evaluate sources of information 
• Critical Appraisal of an Article on an Educational Intervention  (variable study design) from the University of Glasgow
• Educational Interventions Critical Appraisal worksheet  from BestBETs
• PROMPT  from Open University
• PROVEN  - tool to evaluate any source of information 

SIFT (The Four Moves)  to help students distinguish between truth and fake news 

Some Guidelines for the Critical Reviewing of Conceptual Papers

• << Previous: Where to start
• Next: Business >>
• Last Updated: Aug 16, 2024 9:02 AM
• URL: https://libguides.qmu.ac.uk/critical-appraisal

Critical appraisal

• International Review of Sport and Exercise Psychology 15(1):1-21
• CC BY-NC-ND 4.0
• This person is not on ResearchGate, or hasn't claimed this research yet.

• Durham University

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations
• Eunice Garces
• John Marco Recio
• Viviana Patricia Rios
• Anna Barbara Sanchez

• Eileen Furlong
• Tina Bedenik

• Noor Al-Chalabi
• EUR J ANAESTH

• Diane Dixon

• HEALTH SOC CARE COMM

• Rachel Flynn

• Boris Zevin
• Isabelle Raiche
• ADV PHYSIOL EDUC
• Heather MacDonald
• Veronic Bezaire

• Douglas G. Altman
• Patricia Aluko
• Camilla Young

• Barbara Gastel
• Robert A. Day
• Mark B. Andersen
• QUAL HEALTH RES
• Janice M. Morse
• Maresa McGettigan
• Chris R Cardwell
• Marie M Cantwell

• Judy Van Raalte

• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Critical appraisal of published research papers - A reinforcing tool for research methodology: Questionnaire-based study

Affiliations.

• 1 Department of Pharmacology and Therapeutics, Seth GS Medical College and KEM Hospital, Mumbai, Maharashtra, India.
• 2 Department of Clinical Trials, Serum Institute of India, Pune, Maharashtra, India.
• PMID: 34012907
• PMCID: PMC8112331
• DOI: 10.4103/picr.PICR_107_18

Background and objectives: Critical appraisal of published research papers is routinely conducted as a journal club (JC) activity in pharmacology departments of various medical colleges across Maharashtra, and it forms an important part of their postgraduate curriculum. The objective of this study was to evaluate the perception of pharmacology postgraduate students and teachers toward use of critical appraisal as a reinforcing tool for research methodology. Evaluation of performance of the in-house pharmacology postgraduate students in the critical appraisal activity constituted secondary objective of the study.

Materials and methods: The study was conducted in two parts. In Part I, a cross-sectional questionnaire-based evaluation on perception toward critical appraisal activity was carried out among pharmacology postgraduate students and teachers. In Part II of the study, JC score sheets of 2 nd - and 3 rd -year pharmacology students over the past 4 years were evaluated.

Results: One hundred and twenty-seven postgraduate students and 32 teachers participated in Part I of the study. About 118 (92.9%) students and 28 (87.5%) faculties considered the critical appraisal activity to be beneficial for the students. JC score sheet assessments suggested that there was a statistically significant improvement in overall scores obtained by postgraduate students ( n = 25) in their last JC as compared to the first JC.

Conclusion: Journal article criticism is a crucial tool to develop a research attitude among postgraduate students. Participation in the JC activity led to the improvement in the skill of critical appraisal of published research articles, but this improvement was not educationally relevant.

Keywords: Journal club; perception; performance; pharmacology; postgraduate.

Copyright: © 2019 Perspectives in Clinical Research.

PubMed Disclaimer

Conflict of interest statement

There are no conflicts of interest.

Graphical representation of the percentage…

Graphical representation of the percentage of students/teachers who agreed that critical appraisal of…

Similar articles

• Comparison of two formats of journal club for postgraduate students at two centers in developing critical appraisal skills. Kaur M, Sharma HB, Kaur S, Sharma R, Sharma R, Kapoor R, Deepak KK. Kaur M, et al. Adv Physiol Educ. 2020 Dec 1;44(4):592-601. doi: 10.1152/advan.00111.2019. Adv Physiol Educ. 2020. PMID: 32990464
• Do community medicine residency trainees learn through journal club? An experience from a developing country. Akhund S, Kadir MM. Akhund S, et al. BMC Med Educ. 2006 Aug 22;6:43. doi: 10.1186/1472-6920-6-43. BMC Med Educ. 2006. PMID: 16925800 Free PMC article.
• Refining the Journal Club Presentations of Postgraduate Students in Seven Clinical Departments for Better Evidence-based Practice. Herur A, Kolagi S, Ramadurg U, Hiremath CS, Hadimani CP, Goudar SS. Herur A, et al. Ann Med Health Sci Res. 2016 May-Jun;6(3):185-9. doi: 10.4103/2141-9248.183939. Ann Med Health Sci Res. 2016. PMID: 27398252 Free PMC article.
• Promoting the Conduct of Medical Education Journal Clubs in Teaching Medical Institutions. Shrivastava SR, Shrivastava PS. Shrivastava SR, et al. Avicenna J Med. 2021 Aug 31;11(3):156-159. doi: 10.1055/s-0041-1735126. eCollection 2021 Jul. Avicenna J Med. 2021. PMID: 34646793 Free PMC article. Review.
• Assessment methods and tools to evaluate postgraduate critical care nursing students' competence in clinical placement. An integrative review. Øvrebø LJ, Dyrstad DN, Hansen BS. Øvrebø LJ, et al. Nurse Educ Pract. 2022 Jan;58:103258. doi: 10.1016/j.nepr.2021.103258. Epub 2021 Nov 18. Nurse Educ Pract. 2022. PMID: 34847502 Review.
• Curriculum, competency development, and assessment methods of MSc and PhD pharmacy programs: a scoping review. ElKhalifa D, Hussein O, Hamid A, Al-Ziftawi N, Al-Hashimi I, Ibrahim MIM. ElKhalifa D, et al. BMC Med Educ. 2024 Sep 11;24(1):989. doi: 10.1186/s12909-024-05820-5. BMC Med Educ. 2024. PMID: 39261860 Free PMC article. Review.
• Burls A. What is Critical Appraisal? 2nd ed. London: Hayward Medical Communications; 2009.
• Glasziou PP. Information overload: What's behind it, what's beyond it? Med J Aust. 2008;189:84–5. - PubMed
• Haranath PS. Medical curriculum and pharmacology: An appraisal. Indian J Pharmacol. 2016;48:S10–S13. - PMC - PubMed
• Medical Council of India. Postgraduate Medical Education Regulations, 2000. 2013. May, [Last accessed on 2018 Mar 13]. Available from: http://www.mciindia.org .
• Maharashtra University of Health Sciences. Nashik. 2006. [Last accessed on 2018 May 25]. Available from: https://www.muhs.ac.in/upload/syllabus/MD_Pharmacology.pdf .

Related information

Linkout - more resources, full text sources.

• Europe PubMed Central
• Ovid Technologies, Inc.
• PubMed Central

Other Literature Sources

• scite Smart Citations

Miscellaneous

• NCI CPTAC Assay Portal
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

To read this content please select one of the options below:

Please note you do not have access to teaching notes, a critical appraisal tool for library and information research.

Library Hi Tech

ISSN : 0737-8831

Article publication date: 1 July 2006

As the interest in evidence‐based librarianship increases, so does the need for a standardized practice methodology. One of the most essential components of EBL, critical appraisal, has not been fully established within the library literature. The purpose of this paper is to outline and describe a thorough critical appraisal tool and process that can be applied to library and information research in an evidence based setting.

Design/methodology/approach

To create a critical appraisal tool for EBL, it was essential to look at other models. Exhaustive searches were carried out in several databases. Numerous articles were retrieved which provided “evidence” or “best practice” based on a critical appraisal. The initial tool, when created, was distributed to several librarians who provided comments to the author regarding its exhaustiveness, ease of use and applicability and was subsequently revised to reflect their suggestions and comments.

The critical appraisal tool provides a thorough, generic list of questions that one would ask when attempting to determine the validity, applicability and appropriateness of a study.

Originality/value

More rigorous research and publishing will be encouraged as more librarians and information professionals adopt the practice of EBL and utilize this critical appraisal model

• Evidence‐based practice
• Librarianship

Glynn, L. (2006), "A critical appraisal tool for library and information research", Library Hi Tech , Vol. 24 No. 3, pp. 387-399. https://doi.org/10.1108/07378830610692154

Emerald Group Publishing Limited

Copyright © 2006, Emerald Group Publishing Limited

Related articles

All feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

• Conferences
• Editorial Process
• Recent Advances
• Sustainability
• Academic Resources

New Tools for Advancing Research Integrity and Peer Review

The theme of Peer Review Week 2024 is the Intersection of Innovation and Technology . This reflects the rapid advances in technology, particularly artificial intelligence (AI), that have shifted the academic landscape in recent years. Whilst there are challenges around advancing research integrity, recent innovations have led to automating repetitive tasks, improving ethical and grammar checks, and streamlining multi-step processes, all of which can benefit overworked reviewers.

In response to this year’s theme, we will highlight two tools that MDPI has developed to support research integrity, easing reviewers’ workloads and improving their efficiency.

Eureka – Reviewer Recommender

Eureka is an artificial intelligence tool designed to assist in the reviewer selection process.

Because of the high demand for reviewers, and the importance of the process, finding the right reviewers for a paper is essential for maintaining research integrity.

Eureka utilises natural language processing and machine learning to extract the core concept and themes from a manuscript based on its title and abstract. It then uses a similar process to search for comparable, previously published articles and reviews. From these, Eureka extracts potential reviewers from our internal databases with extensive publication records within the field of the manuscript and presents information in an easy-to-read dashboard.

In a recent interview, Difan Lin, Project Manager of Eureka – Reviewer Recommender, described the aims behind developing the tool:

We want the tool to be able to increase the matchmaking ability, streamline the reviewer finding part of the editorial process, and be able to send more targeted reviewer requests. In addition, the tool should assist us in upgrading the quality of our reviewer reports and be able to assess the needs and requirements of the whole editorial process.

MDPI strives to improve the experience of both authors and reviewers. Eureka ensures that authors get the best reviewers possible, whilst also ensuring that reviewers receive requests to review the right articles for them based on their area of expertise.

Online proofreading

Online Proofreading is a platform that allows external authors and MDPI internal employees to work together on finalising articles during the proofreading process. So far, the platform has been used successfully on several MDPI journals.

Tracy (Yue) Wu, Department Manager of IT/Senior IT Project Manager, described the impulse behind the platform as “simplifying the proofreading process”. He goes on to explain

During traditional offline proofreading, emails are generally sent back and forth with authors to figure out which part in the article needs to be edited, and sometimes it takes days for a reply. For Online Proofreading, we can invite authors to access the system along with MDPI employees to edit the articles together, which saves time, is more interesting, and allows for more communication.

Online Proofreading is accessed through SuSy, MDPI’s in-house submission system. The platform can be seen as an expansion of SuSy, which already streamlines various parts of the publication process, like the peer review. The Online Proofreading platform expands SuSy to also include:

• English editing, Academic Editor proof reading, and author proof reading.
• Editing tables and uploading figures.
• Adding, editing, deleting, and swapping references in real time.
• Editing and viewing equations.
• Previewing papers as PDFs.

All changes are tracked, with previous versions being viewable throughout the process. Further, authors can directly respond to comments made on their manuscripts.

MDPI is interested in serving the scientific community and advancing science in a sustainable and transparent manner. Online Proofreading reflects this mission by further simplifying the editorial process and ensuring authors are involved every step of the way.

If you want to learn more about Online Proofreader, visit the announcement or the manual for the tool .

Advancing research integrity

Peer Review Week 2024 is centred around Innovation and Technology. As a pioneer in open science for over 25 years, MDPI has led the way with innovations to advance science and support research integrity.

Both Eureka – Reviewer Recommender and Online Proofreading highlight MDPI’s commitment to providing a platform for researchers to be as efficient as possible whilst advancing research integrity. This is vital at a time when the peer review process is under strain. Reviewers must be provided with everything they need to conduct their work.

MDPI is committed to supporting reviewers and preserving the highly valuable peer review process.

For Peer Review 2024, we are publishing a range of content to celebrate the peer review process and everyone involved. If you want to learn more, please click here .

Related posts

Open Science

Vaccines for Alzheimer’s Disease

Insights from MDPI Top Picks: August 2024

Academic Resources , Open Access , Open Science

Open Access in Saudi Arabia

Link Found Between Nighttime Phone Use, Cyberbullying and Sleep in Teens

Author Services Guide To Writing An Effective Abstract

What are Open Access Mandates?

Infant Food in Top US Stores Fails to Meet EU Standards

MDPI Papers Cited in the News: August 2024

How Open Access Supports Women in Education

Exploring Genetic Traits of Domestication in Chickens and Pigs

Add comment cancel reply.

Save my name, email, and website in this browser for the next time I comment.

Privacy Preference Center

Privacy preferences.

From sharing the latest MDPI blog news, to showcasing our most popular articles, the newsletters will keep you in the loop with everything good going on in the science world.

• Open access
• Published: 20 September 2024

Dementia Friendly communities (DFCs) to improve quality of life for people with dementia: a realist review

• Stephanie Craig   ORCID: orcid.org/0000-0003-0783-4975 1 ,
• Peter O’ Halloran   ORCID: orcid.org/0000-0002-0022-7331 1 ,
• Gary Mitchell   ORCID: orcid.org/0000-0003-2133-2998 1 ,
• Patrick Stark   ORCID: orcid.org/0000-0003-2659-0865 1 &
• Christine Brown Wilson   ORCID: orcid.org/0000-0002-7861-9538 1  

BMC Geriatrics volume  24 , Article number:  776 ( 2024 ) Cite this article

8 Altmetric

Metrics details

Currently, there are more than 55 million people living with dementia worldwide. Supporting people with dementia to live as independently as possible in their communities is a global public health objective. There is limited research exploring the implementation of such interventions in the community context. The aim of the review was to create and refine programme theory – in the form of context mechanism-outcome configurations – on how the characteristics of dementia-friendly communities (DFCs) as geographical locations interact with their social and organisational contexts to understand what works for whom and why.

This realist review sourced literature from 5 electronic databases: Cochrane Library, CINAHL, Medline, Scopus, PsychINFO and Google Scholar, as well as relevant websites such as Alzheimer’s Society to identify grey literature. Methodological rigour was assessed using the Joanna Briggs Institute critical appraisal tool.

Seven papers were included in this realist review that focused on DFCs in a geographical context The implementation of DFC interventions emerged as a process characterised by two pivotal implementation phases, intricately linked with sub-interventions. The first intervention, termed Hierarchy Commitment (I1a/b), involves the formalisation of agreements by businesses and organizations, along with the implementation of dementia-friendly action plans. Additionally, Educational Resources (I1c) play a significant role in this phase, engaging individuals with dementia and their caregivers in educational initiatives. The second phase, Geographical/Environmental Requirements (I2), encompasses the establishment of effective dementia-friendly signage, accessible meeting places, and community support.

Conclusions

This realist review highlighted a theoretical framework that might guide the development of dementia-friendly communities to enhance the experiences of individuals with dementia and their caregivers within DFCs. Emphasising the need for a theoretical framework in developing geographical DFCs, the review outlines contextual elements, mechanisms, and outcomes, providing a foundation for future studies. The ultimate goal is to establish a robust body of evidence for the sustainable implementation of dementia-friendly communities, thereby improving the quality of life for those with dementia.

Study registration

This study is registered as PROSPERO 2022 CRD42022317784.

Peer Review reports

Introduction

Currently, there are more than 55 million people living with dementia worldwide [ 1 ]. It is estimated that this number will rise to 139 million by 2050. Dementia is the seventh leading cause of death and one of the major causes of disability and dependence among older people globally, resulting in reduced quality of life for people with dementia and their care partners, with associated social and financial consequences [ 1 ].

Neurological changes that occur with dementia cause the individual to experience impairments; however, it is increasingly recognised that it is the intersection of these impairments with the physical and social environments encountered that creates the experience of disability for the person with dementia [ 2 ]. Since most people who have dementia live in communities, the structure and culture of those communities are likely to have an impact on how dementia is perceived [ 3 ]. In response to this, the World Health Organisation, Dementia Alliance International, and Alzheimer’s Disease International have created programmes that promote a community model of social participation [ 4 ].

People with dementia, as well as their families and carers, value meaningful connections [ 5 , 6 ] and need to be active participants in their social networks to maintain meaningful social connections [ 7 ]. Supporting people with dementia and their carers to live as independently as possible in their communities by providing social and emotional support is a global public health objective [ 8 ]. The worldwide action plan on the public health response to dementia was adopted by the World Health Organisation (WHO) in May 2017 [ 8 , 9 ]. The plan suggests that increasing public awareness and understanding of dementia and making the environment dementia-friendly will enable people with dementia to maximise their autonomy through improved social participation [ 10 ].

ADI [ 3 ] define a dementia-friendly community (DFC) as a place or culture in which people with dementia and their care partners can feel empowered, supported, and included in society- Table  1 identifies the main elements of a DFC.

While a community is typically characterised by its geographic location, communities can also be made up of people who have similar hobbies, religious affiliations, or ethnic backgrounds e.g., organisations with a specific focus of dementia- friendliness [ 3 ]. According to Lin and Lewis [ 11 ], the idea of dementia-friendly communities focuses on the lived experiences of individuals with dementia and is most pertinent to addressing both their needs and the needs of those who live with and support them. According to Mitchell, Burton, and Raman [ 12 ], dementia-friendly communities are likely to be all-inclusive and promote community engagement for everyone, not only those who have dementia.

Several models and frameworks have been developed to operationalise DFCs. The Dementia Friends USA Framework [ 13 ] focuses on raising awareness and understanding of dementia across various sectors. The Alzheimer’s Society in the UK [ 14 ] has a model emphasising awareness, participation, and stakeholder involvement. The Community Engagement Model prioritises the involvement of people with dementia and their caregivers in developing DFC initiatives. Social Inclusion Strategies aim to improve social inclusion through supportive environments and community education [ 15 ]. The Multi-Sector Collaboration Model promotes cooperation among local governments, healthcare providers, businesses, and other organisations to support people with dementia comprehensively.

The DFC concept is inspired by the World Health Organisation’s Age-Friendly Cities initiative [ 15 , 16 ], which aims to create inclusive environments supporting active and healthy aging [ 17 , 18 ]. Both dementia-friendly and age-friendly approaches emphasise empowering local stakeholders to enhance social inclusion, reduce stigma, and remove barriers in physical and social environments [ 19 ].

Despite its potential, the DFC concept faces challenges and criticisms. Swaffer [ 20 ] highlights that the language around dementia often perpetuates stigma, negatively impacting those affected. Swaffer [ 20 ] and Rahman & Swaffer [ 21 ] criticise many DFC initiatives as tokenistic, often failing to genuinely include people with dementia in decision-making. They advocate for an assets-based approach, recognising and leveraging the strengths of individuals with dementia. Shakespeare et al. [ 22 ] emphasise the need for a human rights framework to ensure dignity, respect, and full inclusion for people with dementia. Effective DFCs should go beyond superficial friendliness to ensure authentic inclusion, empowerment, and adherence to a rights-based approach.

Person-centered care is a foundational approach that emphasises treating individuals with dementia with respect, valuing their uniqueness, and understanding their behaviours as meaningful communication [ 23 ]. The bio-psychosocial approach provides a holistic framework [ 24 ], recognising dementia as influenced by biological, psychological, and social factors, guiding comprehensive care strategies. Attachment theory [ 25 ] offers insights into the behaviours and relationships of individuals with dementia based on their attachment histories. The need-driven dementia-compromised behaviour model [ 26 ] shifts focus to addressing underlying needs behind behavioural symptoms rather than merely managing them. Thijssen and colleagues’ work on social health and dementia-friendly communities [ 27 ] aligns well with these person-centered and psychosocial approaches, emphasising social participation, autonomy, and environmental adaptation. Key principles for dementia-friendly communities derived from these theories include recognising individuality, fostering supportive environments, promoting autonomy and meaningful engagement, interpreting behaviours as expressions of needs, and prioritising holistic health and positive relationships. Implementing these principles can enhance inclusivity and support for people with dementia, with ongoing evaluation and adaptation crucial for sustained effectiveness of dementia-friendly initiatives [ 28 , 29 ].

The existing body of evidence offers support for the effectiveness of DFCs, with previous research exploring various dimensions of their establishment. One perspective underscores the significance of a robust policy framework and an enhanced support infrastructure [ 30 , 31 ]. Alternatively, other studies delve into the priorities of individuals with dementia and their caregivers, emphasising factors such as fostering social connections and promoting acceptance of dementia within the community [ 4 , 15 , 32 , 33 ]. Additionally, investigations into the experiences of people with dementia residing in DFCs, including their awareness of living in such a community, have been conducted [ 34 ].

Despite extensive efforts to evaluate DFCs, their effectiveness remains challenging to ascertain due to the multifaceted and complex nature of the intervention. The evaluation process is further complicated by the diverse needs and preferences of individuals with dementia, variations in resources and support across different communities, and the dynamic nature of dementia care and research. A recent rapid-realist review by Thijssen et al. [ 27 ] comprehensively examined how dementia-friendly initiatives (DFIs) function for people with dementia and their caregivers. While some studies have reviewed dementia-friendly hospital settings, such as Lin [ 35 ] and a realist review by Handley [ 36 ] Thijssen et al.‘s [ 27 ] rapid realist review primarily focused on initiatives often serving as building blocks in DFC development. These initiatives are typically activity-based and on a smaller scale compared to larger communities. Despite these valuable insights, there remains a limited understanding of how geographical DFCs specifically contribute to improving the quality of life for individuals living with dementia.

Dementia-friendly communities are complex interventions. Understanding what works, why and what factors help or hinder their effectiveness can optimise the design and implementation of DFCs for the benefit of individuals with dementia and their caregivers [ 37 ], thus contributing to the development of robust and impactful DFC interventions [ 38 ].

DFCs are often understood primarily as geographical communities, which has several important implications [ 30 ]. Defining DFCs geographically allows for a localised approach tailored to specific towns, cities, or regions, enabling initiatives to address the unique needs and characteristics of particular areas [ 39 ]. Geographical DFCs aim to transform entire villages, towns, cities, or regions to become more inclusive and supportive of people with dementia, potentially impacting all aspects of community life [ 2 ]. This approach emphasises the importance of adapting the physical and built environment to be more accessible and navigable for people with dementia, including clear signage, rest areas, and dementia-friendly urban design. A geographical focus also encourages involvement from various local stakeholders, such as businesses, public services, and residents, fostering a collective effort to support people with dementia. Countries like England have incorporated geographically defined DFCs into national policy [ 30 ], setting targets for their creation and establishing recognition systems, allowing for more structured implementation and evaluation. Different geographical areas may adopt diverse strategies based on their specific demographics, resources, and needs, allowing for innovation and context-specific solutions. Additionally, geographical DFCs can facilitate increased social and cultural engagement for people with dementia within their local area, helping them remain active and valued community members [ 34 ]. Defining DFCs geographically enables more straightforward evaluation of their impact on the lives of people affected by dementia within a specific area [ 40 ]. While some DFCs are also defined as communities of interest, focusing on specific groups or shared experiences rather than physical location, the geographical approach remains significant due to its comprehensive nature and ability to create tangible changes in the everyday environments where people with dementia live and interact.

This realist review will therefore offer a novel and unique contribution to the existing literature enabling a greater understanding of geographical DFCs and enable the identification of relevant interventions related to outcomes.

Aim and objectives

The aim of this review is to create and refine a programme theory – in the form of context-mechanism-outcome (CMO) configurations – that explains how the characteristics of geographical Dementia-Friendly Communities (DFCs) interact with their social and The aim of this review is to create and refine a programme The aim of this review is to create and refine a programme The aim of this review is to create and refine a programme.

To identify the dominant programme theories on how geographical DFCs can be successful in improving the quality of life for people with dementia.

To determine the characteristics of geographical DFCs, and the social and organisational contexts that may aid or hinder their effectiveness in providing individual benefits for people with dementia.

Study design

A project protocol was registered with PROSPERO in March 2022 [ 41 ] with the review conducted between April 2022- February 2024. This review followed RAMESES (Realist and Meta-narrative Evidence Syntheses Evolving Standards) guidelines [ 42 ], aiming to create and refine programme theory in the form of context-mechanism-outcome (CMO) configurations.

Step 1: scoping the literature

The first step in the review process was to define the scope of the review. This phase offered the framework and structure for examining and synthesising a variety of study findings [ 43 ]. To understand broad implementation strategies, an initial exploratory literature search was conducted. This included combining worldwide research literature to ensure a comprehensive view, grey literature such as reports and theses for practical insights, and pertinent policy papers to understand real-world applications and guidelines. Implementation strategies aim to identify and understand various methods used to implement changes effectively.

Step 2: search methods for the review

The search strategy was developed in consultation with a subject librarian at Queen’s University Belfast. The databases searched included Cochrane Library, CINAHL, Medline, Scopus, PsychINFO and Google Scholar, as well as relevant websites such as Alzheimer’s Society to identify grey literature. The reference lists of all articles included in this review were also searched. An example of the search strategy used is shown in table 2 .

Step 3: Selection and appraisal of articles

Covidence software [ 44 ] was utilised for the selection of articles, which automatically removed duplicate papers. All articles were reviewed by SC. PS/GM reviewed 50% of each of the articles. This ensured that two people independently and blindly reviewed each script. Any conflicts were resolved as a three-way discussion between all reviewers. The selection of articles was based on inclusion/exclusion criteria (Table  3 ) alongside how well they informed the programme theory. No temporal limits were applied to initial searches, however, we only searched for papers written in English language. Traditionally, realist reviews do not assess methodological quality. However, this aspect was included in this review to provide the reader with an understanding of the strength of the evidence underpinning the conclusions. The methodological quality of all included studies was assessed using JBI appraisal tools [ 45 ].

Step 4: data extraction

A data extraction form based on the RAMESES recommendations for realist synthesis and previously used in realist reviews [ 46 , 47 , 48 ] was used to extract data from the included full-text papers [ 42 ] in the following areas: theoretical foundation of the intervention, participant characteristics, type of DFC intervention, how the intervention was intended to function, implementation characteristics, and contextual issues that facilitated or hindered implementation of the DFC intervention.

The review focused on theoretical foundations such as community social capital, social contagion, empowerment of PLWD, lessons from global best practices, culturally competent approaches, economic and social benefits, stakeholder involvement, and flexible adaptation of DFC models were integral. The review was also guided by strategic policies supporting DFC development and sustainability. Context-Mechanism-Outcome (CMO ) configurations were utilised to identify contexts that enabled or hindered DFC initiatives, the processes or resources activated by DFCs ( mechanisms ), and the outcomes for people with dementia and their caregivers. Key aspects of DFCs, including physical environment adaptations, social and cultural initiatives and education and awareness programs, were systematically analysed. Implementation strategies, stakeholder engagement processes, barriers, and facilitators were also explored. The review further examined the experiences and perspectives of people living with dementia and caregivers, the impact of DFCs on caregivers, policies supporting DFCs, cultural adaptations of DFC concepts, and evaluation frameworks used to assess DFC effectiveness.

Step 5: synthesising the evidence and drawing conclusions

Identification of candidate theories.

A realist review focuses on the discovery, articulation, and analysis of underlying programme theories to determine if these theories are supported by the evidence [ 49 ]. Following data extraction, candidate theories were formulated, debated and reviewed with the study team. Few papers explain their programme theory; therefore, implicit theories were presumed from components of the interventions. Identifying contextual factors that aided or impeded implementation further developed each candidate theory. Candidate theories from each paper were written in the C-M-O configurations by identifying contextual factors that aid or hinder implementation.

Synthesis of candidate theories

The initial candidate theories were synthesised and grouped into themes relating to the context (C), mechanism (M), outcome (O), and intervention (I). All members of the research team and the study’s expert reference group discussed the relevance of the synthesised candidate theories as the programme theory was developed. The synthesised theories were combined into an overarching programme theory to indicate how geographically bounded DFC interventions may be successfully implemented in the community for people with dementia and their carers (Fig.  2 ).

Study selection

The search identified 2,861 records in total (Fig.  1 ). After duplicates were removed a total of 2,516 papers were left. Titles and abstracts were reviewed together by S.C, P.S and G.M. Following this stage S.C. reviewed all full-text articles while P.S and G.M reviewed 50% of full-text papers. Full-text screening resulted in 68 articles for full-text review, 61 papers were excluded This was resulting in 7 papers for data extraction. Reasons for exclusion are documented in Fig.  1 .

PRISMA flow diagram

Study characteristics (table  4 )

The seven studies employed a range of methodological designs. Three studies used cross-sectional study designs [ 50 , 51 , 52 ]. Three articles used qualitative methodology [ 53 , 54 , 55 ] and one study was a mixed-methods design [ 56 ].

Methodological quality

The methodological quality of the empirical evidence in each of the seven papers included in this review was critically appraised using Joanna Briggs Institute critical appraisal tools [ 45 ]. Using the JBI tool, Goodman et al. [ 56 ] was assessed as strong, two articles were accessed as moderate [ 51 , 52 ] and four were accessed as weak [ 50 , 53 , 54 , 55 ].

Main objectives of the studies

The included studies had three main sets of objectives: to explore the experiences of living/ working within a DFC [ 51 , 56 ] and to understand how a community can become dementia-friendly [ 50 , 52 , 53 , 55 ]. The third objective focused on the perception of residents on building a DFC in a minority area [ 54 ].

Study populations

The studies described different types of DFCs across four continents; Asia [ 51 ] Oceania [ 52 ] North America [ 50 , 54 ] and Europe [ 53 , 55 , 56 ]. Two studies collected data from people with dementia ( n  = 35) [ 54 , 56 ]. Three studies from caregivers/ family care partners ( n  = 152) [ 50 , 54 , 55 ]. Four of the studies collected data from additional participants ( n  = 454). For example, community workers [ 52 , 53 , 54 , 55 ]. Tsuda et al. [ 51 ] categorised their participants ( n  = 2633) as older adults living in an apartment block with a mean age of 77.4, 45.7% living alone and 7.7% reported living with impaired cognitive function. Participants with a diagnosis of dementia did not disclose the clinical stage of their diagnosis.

Characteristics of DFC interventions

All studies explored the use of dementia-friendly programmes within the community. DFC programmes involve the implementation of various person-centred approaches to the community environment to support people with dementia. The programmes identified in this realist review are not standardised interventions and do not involve a single intervention but rather a collective of different community activities interventions aided by members of the public/policymakers with ongoing input from dementia charities e.g., Alzheimer’s Society or Alzheimer’s Disease International. These programmes focus on improving the places in which people with dementia interact and live in their daily lives.

Characteristics of DFC outcomes

DFC interventions have been shown to yield a variety of positive outcomes. These interventions have led to increased social interaction [ 51 ] among individuals living with dementia, fostering a sense of belonging and reducing social isolation [ 52 ]. Moreover, interventions promoting the involvement of people with dementia within the community have resulted in improved quality of life for people with dementia [ 52 , 54 ]. DFC intervention results in improved community capacity to deliver dementia-friendly services, such as support groups and workshops, these interventions have also positively impacted caregivers by reducing depression and promoting healthy outcomes for carers [ 50 ]. Additionally, DFC interventions support people with dementia’s independence and ability to continue living in their own homes [ 55 ]. Small-scale initiatives developed by PWD and their caregivers, such as the EndAge Day and Memory Bank projects, have further enriched community engagement and encouraged participation in meaningful activities [ 53 ]. The interventions have also led to greater access to public amenities, which promotes a greater quality of life which contributes to active participation in the community and people with dementia living longer in their own homes [ 56 ].

Candidate theories

The preliminary scoping of the literature did not identify any explicit theory underlying the implementation of DFCs for people living with dementia or their caregivers. However, common sense implicit theories were identified. It was evident that providing dementia awareness information in the community is a key component of a DFC [ 51 , 52 , 53 , 54 , 56 ]. If dementia awareness is raised within the community, further support can be provided for people living with dementia and their caregivers which can contribute to positive changes within the environment [ 51 , 54 , 55 ]  and government policies [ 51 ]. This will likely encourage people with dementia and their caregivers to engage in DFCs as they will feel supported and confident in the community [ 51 , 52 , 53 , 54 , 56 ]. In addition, this will improve the quality of life for people with dementia [ 56 ]. However, one study identified how hierarchy commitment is necessary for a business/ organisation to become dementia friendly [ 55 ]. This indicates a strong organisational commitment from the top-down of a business/ organisation. This commitment involves leaders and decision-makers at varying levels endorsing and actively participating in efforts to make the organisations more supportive of people living with dementia. This involves the business/ organisation formalising agreements to become dementia-friendly and implementing dementia-friendly action plans. This is reinforced by another study which states that communities need to prioritise an action plan when implementing a dementia friendly community [ 54 ].

Contextual factors that help or hinder the implementation of DFC interventions

Several contextual factors were identified that help or hinder the implementation of DFC interventions for people living with dementia. The issue of having a recognisable geographical boundary for a DFC remains one of the most significant contextual factors that help the implementation of DFC interventions [ 51 , 52 , 54 , 56 ]. However, one study states that dementia-friendly communities are not defined by a geographical boundary, they are locations where people with dementia can find their way around and feel safe in their locality/ community/ city where they can maintain their social networks, so they feel they still belong in the community [ 53 ].

Dementia-friendly communities thrive in rural areas where there is often a smaller population and a strong sense of community [ 52 , 54 ] and it may be easier to engage local stakeholders [ 55 ]. Close-knit communities where people know each other well can foster greater understanding and support for people living with dementia and their caregivers, and also allow a greater opportunity for tailored and personalised interventions [ 54 , 55 ].

Existing resources e.g., advisory groups, awareness activities, diagnostic and treatment centres, community and family caregiver education and care services and political support are crucial facilitators in the successful implementation of dementia-friendly communities [ 50 , 52 , 53 , 54 , 55 , 56 ]. The presence of ample resources [ 50 , 51 , 52 , 54 ] coupled with robust political endorsement [ 56 ], constitutes a pivotal framework for the success of such initiatives. Governmental bodies, as exemplified, play a crucial role by furnishing financial support for community projects and endorsing policies, thereby enabling a comprehensive approach to assist individuals with dementia and their caregivers (However, a range of factors that both facilitated or hindered these DFCs was also identified – for example, DFCs exhibit notable success in rural settings, as evidenced by their thriving presence in such areas [ 50 , 52 , 53 , 54 , 55 , 56 ]. Sufficient funding is imperative for sustaining programs and services, and financial backing from governmental entities, philanthropic organisations, and local authorities becomes instrumental in meeting the expenses associated with the implementation of DFC interventions [ 53 , 54 ]. Financial constraints can limit the availability of resources, services and infrastructure needed to create and sustain dementia-friendly communities [ 53 , 54 ]. Political support extends beyond mere financial contributions; it catalyses the development and implementation of policies conducive to dementia-friendly practices, addressing issues like anti-discrimination measures and caregiver support. This, in turn, fosters collaboration among stakeholders [ 54 , 55 ]. The establishment of policies also catalyses public awareness campaigns, aimed at mitigating associated stigmas [ 52 , 54 , 56 ]. By leveraging existing resources and garnering political support, communities can cultivate an environment where individuals with dementia are comprehended, esteemed, and supported. This concerted effort leads to the achievement of dementia-friendly communities, ultimately enhancing the overall quality of life for both individuals living with dementia and their caregivers.

Factors identified as hindering implementation can include the younger population’s involvement due to lack of awareness, or lack of involvement or understanding, and can indeed present some challenges in the implementation of a DFC [ 52 , 55 ]. While typically younger individuals may not directly experience dementia first-hand themselves, their attitudes, understanding, and engagement in the community play a significant role in shaping the overall dementia-friendly environment. The gender of people living with dementia can also influence the implementation of dementia-friendly interventions through the concept of social contagion and the existing differences in social networks between men and women. The existing gender differences in social networks can impact the effectiveness of a DFC intervention because typically women already have stronger social networks than men [ 51 ]. Negative cultural stereotypes can also hinder implementation due to the lack of culturally appropriate services, and a lack of understanding of dementia [ 50 ]. Disparities in Alzheimer’s disease and Alzheimer’s Disease-related dementia’s create significant obstacles to the adoption of dementia-friendly communities across all communities, particularly those of colour [ 54 ].

This section explains the intervention (I), mechanism (M), and contexts (C) that are thought to produce the outcome (O) of improved quality of life (QOL) for people living with dementia, increased social interactions, support and inclusivity for people with dementia and their carers. The aim of this synthesis was to create and refine programme theory on how DFCs’ characteristics interact with their social and organisational contexts to produce desired outcomes. Figure  2  depicts the theoretical paradigm for how DFC interventions are expected to work.

A theoretical model of how DFC interventions for people with dementia are thought to work. Legend: Theoretical model of the Context +Mechanism = Outcome (CMO) configuration. Context is shown as either helping (C+) or hindering (C-) implementation. The intervention is divided into two phases, facilitation (I1) and display (I2), activating underlying mechanisms (M) that result in improved outcomes (O)

The implementation of DFC interventions appeared to involve two crucial implementation phases: Hierarchy commitment (I 1a/b )interlinked with educational resources (I 1c ) and Geographical/ environmental requirements (I 2 ). Hierarchy commitment involves two sub-interventions, which are seen in existing public-facing businesses and organisations within a community (C). Organisations and businesses demonstrate a commitment to fostering dementia-friendly communities by formalising agreements and implementing dementia-friendly action plans (I 1a ). This is driven by the sense of obligation experienced by management, primarily driven by concerns about their reputation (M); leading to a change in behaviour among the business/ organisation as they allocate resources such as time and staff training to enhance their public image (O). This leads to businesses and organisations implementing mandatory training for all public-facing staff (I 1b ), which increases staff awareness about dementia friendliness (M), giving staff confidence in their ability to support PWD (M) and staff will feel prepared and supported by their employers/ organisations (M); Staffs preparedness will strengthen social interactions between the staff and PWD, improving public perceptions of the business/ organisation (O). By the same intervention, PWD will feel supported in using the business and organisations within the community (M), increasing the sense of security and confidence felt by PWD in their community (M); leading to increased social interactions, and likelihood to contribute and interact within the community improving the overall quality of life for PWD (O).

Mandatory training provided to businesses and organisations should include co-designed dementia awareness training integrating personal experiences shared by PWD and their caregivers, public awareness events and educational resources (I 1C ), staff will gain confidence in their knowledge and ability to support PWD (M), staff awareness about dementia will develop (M), staff will feel equipped in their role (M); Staffs preparedness will strengthen the social interactions between the staff and PWD (O). By the same intervention, PWD will feel supported in the community (M) increasing their sense of security and confidence knowing the general public will be more aware and have a greater knowledge of dementia (M), promoting self-efficacy for PWD (M); such educational resources will contribute to enhanced support for PWD and enhance caregiver support, improving QOL for PWD (O). These outcomes are likely to be seen when PWD are actively involved in the implementation of training and resources in the community (C).

Secondly, dementia-friendly signage creates inclusive community environments within a communal accessible location (I2) to increase the sense of security and confidence in the community for PWD and their carers (M). Further, PWD will feel at ease to navigate the environment (M), increasing social networks for PWD (M); therefore, implementing a dementia- friendly environment will increase PWD involvement in the community, increase their independence and social interaction within the community (O). These outcomes are more likely seen in a small area with a recognised geographical boundary where there is access to funding to support DFCs (C).

This realist review elucidates the underlying mechanisms that drive the success of DFC interventions in diverse community settings. The realist approach, rooted in understanding the interactions between contexts, mechanisms, and outcomes, allowed this review to identify the complexities of DFC interventions. The initial candidate theory emerging from the synthesis of the literature emphasised the importance of creating dementia-friendly communities to support those affected by dementia [ 50 , 51 , 52 , 53 , 54 , 55 , 56 ]. This theoretical model builds upon this by explicitly identifying the context and mechanisms involved in successful DFC implementation in geographical locations.

The theoretical model posits that hierarchical commitment, educational resources, and geographical/environmental requirements [ 50 , 51 , 52 , 54 , 56 ] are pivotal interventions leading to positive outcomes for individuals living with dementia. These findings extend those of the DEMCOM study’s logic model [ 56 ] by highlighting the critical role of cultural appropriateness and community structures in the success of DFCs. For instance, DFCs thrive in rural settings due to strong community ties and the utilisation of existing resources, which stimulate localised services for people with dementia [ 57 ]. However, these supports may be weakened when younger family members move away [ 7 ]. Moreover, governmental support and utilisation of existing resources significantly contribute to the facilitation of DFCs [ 4 ]. This suggests that while the DEMCOM logic model provides a robust framework, it may benefit from a more explicit integration of cultural and geographical factors. These findings challenge some conclusions of the DEMCOM study by showing that political support and financial backing, while necessary, are not sufficient on their own. The presence of culturally appropriate services and strong community engagement are equally vital. For example, the use of culturally sensitive language and involvement of community leaders were found to be critical in the API community, which was not a primary focus in the DEMCOM logic model.

The combination of context and mechanisms in this review provides an explanation as to why DFC interventions were successfully implemented. For example, recognisable geographical boundaries and rural areas [ 51 , 52 , 54 , 56 ] facilitate the accessibility of dementia-friendly communities for people living with dementia and their carers. Government support is critical in providing resources in such areas that enable appropriate signage and environmental changes that enable engagement within this geographical boundary [ 50 , 52 , 53 , 55 , 56 ]. Effective signage tailored to individuals can create a positive environment for people living with dementia, overall improving the environment [ 58 ]. Training for the public and businesses to generate awareness with their staff supports the sustainability of dementia-friendly communities as it facilitates a widespread understanding of the disease and fosters inclusivity. Staff will also feel an increase in confidence in supporting people living with dementia in businesses within DFCs, which fosters an inclusive community that empowers people living with dementia to maintain their independence and improve their quality of life. It is acknowledged that people with dementia need to be appropriately supported and empowered to remain part of their community [ 59 ].

There are notable gaps in the evidence regarding the long-term impacts of DFCs on different demographic groups. While this study identified several immediate benefits, such as increased social engagement and reduced stigma, more longitudinal research is needed to understand the sustained impact on the quality of life and mental health outcomes for people with dementia. Additionally, there is limited evidence on the specific mechanisms through which DFCs benefit caregivers. Furthermore, factors such as the outmigration of younger individuals to larger urban areas and gender dynamics can hinder the implementation of DFCs, as evidenced by Wiersma and Denton [ 7 ] and Herron and Rosenberg [ 60 ], respectively.

This study indicates that DFCs primarily benefit people with dementia and their caregivers by enhancing social inclusion, reducing stigma, and providing culturally relevant support. In rural settings, the entire community benefits from increased awareness and support structures, contributing to a more inclusive and supportive environment for all residents. However, these findings also suggest that not all groups benefit equally. For example, in urban areas with diverse populations, the lack of culturally tailored services can limit the effectiveness of DFCs. Therefore, for DFCs to be truly effective, they must be designed with the specific needs and characteristics of the target communities in mind. According to Phillipson et al. [ 52 ], creating a model that satisfies everyone’s needs is challenging. According to Turner and Cannon [ 61 ], given their commonalities and the possibility that certain groups will have overlapping interests, it might be beneficial if projects were collaborative rather than parallel. According to research on age-friendliness in rural areas, there is variation both within and between rural communities. While younger people may leave some communities, others may see an influx of relatively wealthy retirees, which may marginalise older residents who have lived in poverty for a longer period of time [ 62 ].

The WHO [ 63 ] toolkit for dementia-friendly initiatives (DFIs) provides a valuable framework for understanding the foundational components necessary for the successful implementation of DFCs. Although our review primarily focuses on geographical DFCs, the toolkit’s recommendations can be relevant as they highlight the importance of establishing strong partnerships, engaging key stakeholders, and creating structured, well-planned initiatives that serve as the building blocks for DFCs [ 17 , 27 , 64 ]. DFIs and DFCs are closely related since DFIs are a part of DFCs and their results are essential to DFC support. The toolkit offers detailed guidance on how to set up DFIs, which can be seen as essential precursors to the broader goal of developing inclusive and supportive communities for individuals living with dementia.

While the research available offers significant insights into the theoretical aspects of DFC interventions, it is important to acknowledge the current lack of concrete evidence on their efficacy. Nonetheless, the realist review methodology enables us to consider the diverse perspectives of participants and stakeholders, leading to a more comprehensive understanding of the complex interplay between interventions, mechanisms, and outcomes. To ensure the sustainability of DFCs, future research should focus on the long-term impacts of existing interventions and the perspectives of decision-makers and programme creators, such as the Alzheimer’s Society. By applying the realist lens to these investigations, we can further refine our theoretical framework and identify the critical elements needed for the continued success of DFC initiatives. The realist review methodology has been instrumental in shaping a theoretical framework for the implementation of dementia-friendly communities. By acknowledging the specific contexts, identifying underlying mechanisms, and exploring outcomes, this approach moves beyond conventional systematic reviews and offers a more nuanced understanding of how DFC interventions work. While evidence on their effectiveness may still be evolving, the insights gained from this realist review contribute significantly to the growing body of knowledge, guiding the development of sustainable and effective dementia-friendly communities that truly enhance the quality of life for individuals living with dementia and their caregivers.

Strengths and limitations

This realist review has contributed to an ever-growing evidence- base on the creation of a theoretical framework for the implementation of dementia-friendly communities, and it includes both the elements required for implementation and the underlying mechanisms that might affect outcomes. However, there was no advice on how to carry out these interventions. There is also little understanding of how the interplay between the intervention, mechanism, and setting affects people with dementia or their caregivers because DFCs were developed in various contexts and ways.

Further research looking into the sustainability of existing dementia-friendly communities is urgently needed. Future studies should also consider the lessons learned from the implementation of complex DFC interventions from people living with dementia in/and people working/volunteering within dementia-friendly communities. In acknowledging the limitations of this study, it is important to note that the existing body of literature is limited. The scarcity of relevant studies in this area may impact the generalisability of our findings and the overall programme theory. Due to the nature of the review, we could only screen English papers and therefore there may have been key literature missed. Additionally, another limitation to this study is that this review focuses solely on geographical DFCs. However, this helped to narrow the focus of this review amongst the literature.

This realist review has illuminated a theoretical framework that might guide the development of geographical dementia-friendly communities for those with dementia and their caregivers. However, it has highlighted a gap in the existing literature, specifically the lack of a realist approach that explicitly theorises the specific contexts, intervention components, and resulting mechanisms. The review’s aim is to create and refine a programme theory on how to improve the experiences of living in dementia-friendly communities, which is significant for both individuals living with dementia and their caregivers. Moreover, there is a need to apply this theoretical framework to the development of geographical dementia-friendly communities, enhancing the quality of life for people living with dementia. This realist review outlines significant contextual elements, mechanisms, and outcomes in relation to geographical dementia-friendly communities which can guide future studies (Fig.  2 ). Future research should concentrate on building a robust body of evidence to support the sustainable implementation of dementia-friendly communities, further improving the quality of life for those diagnosed with dementia.

Availability of data and materials

No datasets were generated or analysed during the current study.

Reference list

World Health Organization. Dementia. 2022. https://www.who.int/news-room/fact-sheets/detail/dementia .

Dementia Alliance International. Human rights for people living with dementia: From rhetoric to reality. Dementiaallianceinternational.org. 2016. Available from: https://dementiaallianceinternational.org/assets/2016/05/Human-Rights-for-People-Living-with-Dementia-Rhetoric-to-Reality.pdf .

Alzheimer’s Disease International. Dementia Friendly Communities. 2022. (Online). Available at: https://www.alzint.org/what-we-do/policy/dementia-friendly-communities/ ].

Novak LS, Horne E, Brackett JR, Meyer K, Ajtai RM. Dementia-friendly communities: A review of current literature and reflections onimplementation. Curr Geriatr Rep. 2020;9:176–82.

Article   Google Scholar  

Phinney A, Chaudhury H, O’Connor DL. Doing as much as I can do: the meaning of activity for people with dementia. Aging Ment Health. 2007;11:384–93.

Article   PubMed   Google Scholar  

Phinney A, Kelson E, Baumbusch J, et al. Walking in the neighbourhood: performing social citizenship in dementia. Dementia. 2016;15:381–94.

Wiersma EC, Denton A. From social network to safety net: dementia-friendly communities in rural Northern Ontario. Dementia. 2016;15:51–68.

World Health Organization. Global action plan on the public health response to dementia 2017–2025. 2017. https://www.who.int/publications/i/item/global-action-plan-on-the-public-health-response-to-dementia-2017---2025

Alzheimer’s Disease International. WHO Global action plan on dementia. 2022. Available at: https://www.alzint.org/what-we-do/partnerships/world-health-organization/who-global-plan-on-dementia/ .

Alzheimer’s Disease International.  Dementia-Friendly Communities: Global Developments 2nd Edition. https://www.alzint.org/u/dfc-developments.pdf . (2017). Accessed 28 June 2024.

Lin SY, Lewis FM. Dementia friendly, dementia capable, and dementia positive: concepts to prepare for the future. Gerontologist. 2015;55(2):237–44.

Article   PubMed   PubMed Central   Google Scholar  

Mitchell L, Burton E, Raman S. Dementia-friendly cities: designing intelligible neighbourhoods for life. J Urban Des. 2004;9(1):89–101.

Dementia Friendly America. Dementia Friends USA [Online] Dementia Friendly America. 2024. https://dfamerica.org/overview-and-5-key-messages/

Alzheimer’s Society. What is a dementia friendly community? Alzheimer’s Society. 2024.  https://www.alzheimers.org.uk/get-involved/dementia-friendly-resources/what-dementia-friendly-community .

Hung L, Hudson A, Gregorio M, Jackson L, Mann J, Horne N, Berndt A, Wallsworth C, Wong L, Phinney A. Creating dementia-friendly communities for social inclusion: a scoping review. Gerontol Geriatric Med. 2021;7:p23337214211013596.

Ogilvie K, And Eggleton A. Standing Senate Committee on Social Affairs, Science and Technology. Dementia in Canada: a national strategy for dementia-friendly communities; 2016.

Google Scholar  

Hebert CA, Scales K. Dementia friendly initiatives: a state of the science review. Dementia. 2019;18(5):1858–95. https://doi.org/10.1177/1471301217731433 .

Webster D. Dementia-friendly communities Ontario: a Multi-sector collaboration to improve quality of life for people living with dementia and Care Partners Ontario. The Alzheimer Society of Ontario; 2016.

Grogan C. Doing dementia friendly communities locally: Tensions in committee practices and micro-processes. Diss. Queensland University of Technology; 2022.

Swaffer K. Dementia: stigma, language, and dementia-friendly. Dementia (London). 2014;13(6):709–16. https://doi.org/10.1177/1471301214548143 .

Rahman S, Swaffer K. Assets-based approaches and dementia-friendly communities. Dementia (London). 2018;17(2):131–7. https://doi.org/10.1177/1471301217751533 .

Shakespeare T, Zeilig H, Mittler P. Rights in mind: thinking differently about dementia and disability. Dement (London). 2019;18(3):1075–88. https://doi.org/10.1177/1471301217701506 .

Fazio S, Pace D, Flinner J, Kallmyer B. The fundamentals of person-centered care for individuals with dementia. The Gerontologist. 2018;58(Issue suppl_1):S10–9. https://doi.org/10.1093/geront/gnx122 .

Spector A, Orrell M. Using a biopsychosocial model of dementia as a tool to guide clinical practice. Int Psychogeriatr. 2010;22(6):957–65.

Miesen B. Attachment theory and dementia. Care-giving in Dementia. Routledge; 2014. pp. 38–56.

Kovach CR, Noonan PE, Schlidt AM, Wells T. A model of consequences of need-driven, dementia‐compromised behavior. J Nurs Scholarsh. 2005;37(2):134–40.

Thijssen M, Daniels R, Lexis M, Jansens R, Peeters J, Chadborn N, Nijhuis‐van der Sanden MW, Kuijer‐Siebelink W, Graff M. How do community based dementia friendly initiatives work for people with dementia and their caregivers, and why? A rapid realist review. Int J Geriatr Psychiatry. 2022;37(2).

Thijssen M, Graff MJ, Lexis MA, Nijhuis-van der Sanden MW, Radford K, Logan PA, Daniels R, Kuijer-Siebelink W. Collaboration for developing and sustaining community dementia-friendly initiatives: a realist evaluation. Int J Environ Res Public Health. 2023;20(5):4006.

Thijssen M, Kuijer-Siebelink W, Lexis MA, Nijhuis-Van Der Sanden MW, Daniels R, Graff M. What matters in development and sustainment of community dementia friendly initiatives and why? A realist multiple case study. BMC Public Health 2023 23(1):296.

Buckner S, Mattocks C, Rimmer M, Lafortune L. An evaluation tool for age-friendly and dementia friendly communities. Work Older People. 2018;22(1):48–58. https://doi.org/10.1108/WWOP-11-2017-0032 .

Shannon K, Bail K, Neville S. Dementia-friendly community initiatives: an integrative review. J Clin Nurs. 2019;28(11–12):2035–45.

Smith K, Gee S, Sharrock T, Croucher M. Developing a dementia-friendly Christchurch: Perspectives of people with dementia. Australas J Ageing. 2016;35(3):188–92.

Wu SM, Huang HL, Chiu YC, Tang LY, Yang PS, Hsu JL, Liu CL, Wang WS, Shyu YIL. Dementia-friendly community indicators from the perspectives of people living with dementia and dementia-family caregivers. J Adv Nurs. 2019;75(11):2878–9.

Darlington N, Arthur A, Woodward M, et al. A survey of the experience of living with dementia in a dementia-friendly community. Dementia. 2021;20(5):1711–22.

Lin SY. ‘Dementia-friendly communities’ and being dementia friendly in healthcare settings. Curr Opin Psychiatry. 2017;30(2):145.

Handley M, Bunn F, Goodman C. Dementia-friendly interventions to improve the care of people living with dementia admitted to hospitals: a realist review. BMJ open. 2017;7(7):e015257.

Singh NS, Kovacs RJ, Cassidy R, Kristensen SR, Borghi J, Brown GW. A realist review to assess for whom, under what conditions and how pay for performance programmes work in low-and middle-income countries. Soc Sci Med. 2021;270:113624.

Haynes A, Gilchrist H, Oliveira JS, Tiedemann A. Using realist evaluation to understand process outcomes in a COVID-19-impacted yoga intervention trial: a worked example. Int J Environ Res Public Health. 2021;18(17):p9065.

European Foundations on Dementia. Mapping Dementia- Friendly Communities Across Europe. 2016. [PDF] https://www.dataplan.info/img_upload/5c84ed46aa0abfec4ac40610dde11285/mapping_dfcs_across_europe_final_v2.pdf .

Craig S, Mitchell G, Halloran PO, et al. Exploring the experiences of people living with dementia in Dementia Friendly communities (dfcs) in Northern Ireland: a realist evaluation protocol. BMC Geriatr. 2023;23:361. https://doi.org/10.1186/s12877-023-04090-y .

Craig S, Wilson B, O’Halloran C. P., Mitchell G, Stark P. How do people with dementia and their caregivers experience Dementia Friendly communities (dfcs) and how are these sustained over time: a realist review. 2022.

Wong G, Greenhalgh T, Westhorp G, Et al. RAMESES publication standards: realist syntheses. BMC Med. 2013;11:21. https://doi.org/10.1186/1741-7015-11-21 .

Pawson R, Greenhalgh T, Harvey G, Walshe K. Realist review-a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy. 2005;10(1suppl):21–34.

Covidence systematic review software. Veritas Health Innovation, Melbourne, Australia. 2022. Available at www.covidence.org .

The Joanna Briggs Institute. Critical appraisal tools. The Joanna Briggs Institute. 2017. Retrieved June 16, 2022, from https://jbi.global/critical-appraisal-tools .

Morton T, Wong G, Atkinson T, et al. Sustaining community-based interventions for people affected by dementia long term: the SCI-Dem realist review. BMJ Open. 2021;11:e047789. https://doi.org/10.1136/bmjopen-2020-047789 .

O’Halloran P, Scott D, Reid J, Porter S. Multimedia psychoeducational interventions to support patient self-care in degenerative conditions: a realist review. Palliat Support Care. 2015;13(5):1473–86. https://doi.org/10.1017/S1478951514001229 .

McGrath D, O’Halloran P, Prue G, Brown M, Millar J, O’Donnell A, McWilliams L, Murphy C, Hinds G, Reid J. Exercise interventions for women with ovarian cancer: a realist review. Healthcare. 2022;10(4):720.

Rycroft-Malone J, mccormack B, Hutchinson AM, et al. Realist synthesis: illustrating the method for implementation research. Implement Sci. 2012;7:33. https://doi.org/10.1186/1748-5908-7-33 .

Kally Z, Cherry DL, Howland S, And Villarruel M. Asian Pacific Islander dementia care network: a model of care for underserved communities. J Gerontol Soc Work. 2014;57(6–7):710–27.

Tsuda S, Inagaki H, Okamura T, Sugiyama M, Ogawa M, Miyamae F, Edahiro A, Ura C, Sakuma N, Awata S. Promoting cultural change towards dementia friendly communities: a multi-level intervention in Japan. BMC Geriatr. 2022;22(1):1–13.

Phillipson L, Hall D, Cridland E, Fleming R, Brennan-Horley C, Guggisberg N, Frost D, Hasan H. Involvement of people with dementia in raising awareness and changing attitudes in a dementia friendly community pilot project. Dementia. 2019;18(7–8):2679–94.

Dean J, Silversides K, Crampton JA, Wrigley J. Evaluation of the York Dementia Friendly Communities Programme. York: Joseph Rowntree Foundation; 2015a.

Bergeron CD, Robinson MT, Willis FB, Albertie ML, Wainwright JD, Fudge MR, Parfitt FC, Lucas JA. Creating a dementia friendly community in an African American neighborhood: perspectives of people living with dementia, care partners, stakeholders, and community residents. J Appl Gerontol. 2023;42(2):280–9.

Heward M, Innes A, Cutler C, Hambidge S. Dementia-friendly communities: challenges and strategies for achieving stakeholder involvement. Health Soc Care Commun. 2017;25(3):858–67.

Goodman C, Arthur A, Buckner S, Buswell M, Darlington N, Dickinson A, Killett A, Lafortune L, Mathie E, Mayrhofer A, Reilly P, Skedgel C, Thurman J, Woodward M. The DEMCOM study: a national evaluation of Dementia Friendly communities. National Institute for Health Research Policy Research (NIHR); 2019. https://uhra.herts.ac.uk/handle/2299/23477 .

Marshall F, Basiri A, Riley M, et al. Scaling the Peaks Research Protocol: understanding the barriers and drivers to providing and using dementia-friendly community services in rural areas—a mixed methods study. BMJ Open. 2018;8:e020374. https://doi.org/10.1136/bmjopen-2017-020374 .

Gresham M, Taylor L, Keyes S, Wilkinson H, McIntosh D, Cunningham C. Developing evaluation of signage for people with dementia. Hous Care Support. 2019;22(3):153–61.

Quinn C, Hart N, Henderson C, Litherland R, Pickett J, Clare L. Developing supportive local communities: perspectives from people with dementia and caregivers participating in the IDEAL programme. J Aging Soc Policy. 2022;34(6):839–59.

Herron RV, Rosenberg MW. “Not there yet”: Examining community support from the perspective of people with dementia and their partners in care. Soc Sci Med. 2017;173:81–7.

Article   CAS   PubMed   Google Scholar  

Turner N, Cannon S. Aligning age-friendly and dementia‐ friendly communities in the UK. Working Older People. 2018;22(1):9–19. https://doi.org/10.1108/WWOP-12-2017-0036 .

Keating N, Eales J, Phillips JE. Age-friendly rural com‐ munities: conceptualizing ‘best‐fit’. Can J Aging/La Revue Canadienne Du Vieillissement. 2013;32(4):319–32. https://doi.org/10.1017/S0714980813000408 .

World Health Organization. WHO launches new toolkit to promote dementia inclusive societies [Internet]. 2021. Available from: https://www.who.int/news/item/06-08-2021-who-launches-new-toolkit-to-promote-dementia-inclusive-societies.

Williamson T. Mapping Dementia-Friendly Communities Across Europe. European Foundations’ Initiative on Dementia, Brussels. 2016. https://ec.europa.eu/eip/ageing/sites/eipaha/files/results_attachments/mapping_dfcs_across_europe_final.pdf .

Download references

Acknowledgements

Not applicable.

This research is fully funded by the Department for Education (DfE) in Northern Ireland.

Author information

Authors and affiliations.

School of Nursing and Midwifery, Queen’s University Belfast, Belfast, Northern, Ireland

Stephanie Craig, Peter O’ Halloran, Gary Mitchell, Patrick Stark & Christine Brown Wilson

You can also search for this author in PubMed   Google Scholar

Contributions

CBW, GM, POH, and PS are co-investigators and led on the design of the study. SC led the write-up of this manuscript, and all authors made comments and approved the final manuscript. 

Corresponding author

Correspondence to Stephanie Craig .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Craig, S., Halloran, P.O., Mitchell, G. et al. Dementia Friendly communities (DFCs) to improve quality of life for people with dementia: a realist review. BMC Geriatr 24 , 776 (2024). https://doi.org/10.1186/s12877-024-05343-0

Download citation

Received : 19 February 2024

Accepted : 29 August 2024

Published : 20 September 2024

DOI : https://doi.org/10.1186/s12877-024-05343-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Realist review
• Realist synthesis
• Dementia-friendly communities
• Dementia friends
• Quality of Life

BMC Geriatrics

ISSN: 1471-2318

• General enquiries: [email protected]