critical appraisal of research paper

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Published: 20 January 2009

How to critically appraise an article

• Jane M Young 1 &
• Michael J Solomon 2  

Nature Clinical Practice Gastroenterology & Hepatology volume  6 ,  pages 82–91 ( 2009 ) Cite this article

53k Accesses

426 Altmetric

Metrics details

Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article in order to assess the usefulness and validity of research findings. The most important components of a critical appraisal are an evaluation of the appropriateness of the study design for the research question and a careful assessment of the key methodological features of this design. Other factors that also should be considered include the suitability of the statistical methods used and their subsequent interpretation, potential conflicts of interest and the relevance of the research to one's own practice. This Review presents a 10-step guide to critical appraisal that aims to assist clinicians to identify the most relevant high-quality studies available to guide their clinical practice.

Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article

Critical appraisal provides a basis for decisions on whether to use the results of a study in clinical practice

Different study designs are prone to various sources of systematic bias

Design-specific, critical-appraisal checklists are useful tools to help assess study quality

Assessments of other factors, including the importance of the research question, the appropriateness of statistical analysis, the legitimacy of conclusions and potential conflicts of interest are an important part of the critical appraisal process

This is a preview of subscription content, access via your institution

Access options

Subscribe to this journal

Receive 12 print issues and online access

195,33 € per year

only 16,28 € per issue

Buy this article

• Purchase on SpringerLink
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

Making sense of the literature: an introduction to critical appraisal for the primary care practitioner

How to appraise the literature: basic principles for the busy clinician - part 2: systematic reviews and meta-analyses

How to appraise the literature: basic principles for the busy clinician - part 1: randomised controlled trials

Druss BG and Marcus SC (2005) Growth and decentralisation of the medical literature: implications for evidence-based medicine. J Med Libr Assoc 93 : 499–501

PubMed   PubMed Central   Google Scholar  

Glasziou PP (2008) Information overload: what's behind it, what's beyond it? Med J Aust 189 : 84–85

PubMed   Google Scholar  

Last JE (Ed.; 2001) A Dictionary of Epidemiology (4th Edn). New York: Oxford University Press

Google Scholar  

Sackett DL et al . (2000). Evidence-based Medicine. How to Practice and Teach EBM . London: Churchill Livingstone

Guyatt G and Rennie D (Eds; 2002). Users' Guides to the Medical Literature: a Manual for Evidence-based Clinical Practice . Chicago: American Medical Association

Greenhalgh T (2000) How to Read a Paper: the Basics of Evidence-based Medicine . London: Blackwell Medicine Books

MacAuley D (1994) READER: an acronym to aid critical reading by general practitioners. Br J Gen Pract 44 : 83–85

CAS   PubMed   PubMed Central   Google Scholar  

Hill A and Spittlehouse C (2001) What is critical appraisal. Evidence-based Medicine 3 : 1–8 [ http://www.evidence-based-medicine.co.uk ] (accessed 25 November 2008)

Public Health Resource Unit (2008) Critical Appraisal Skills Programme (CASP) . [ http://www.phru.nhs.uk/Pages/PHD/CASP.htm ] (accessed 8 August 2008)

National Health and Medical Research Council (2000) How to Review the Evidence: Systematic Identification and Review of the Scientific Literature . Canberra: NHMRC

Elwood JM (1998) Critical Appraisal of Epidemiological Studies and Clinical Trials (2nd Edn). Oxford: Oxford University Press

Agency for Healthcare Research and Quality (2002) Systems to rate the strength of scientific evidence? Evidence Report/Technology Assessment No 47, Publication No 02-E019 Rockville: Agency for Healthcare Research and Quality

Crombie IK (1996) The Pocket Guide to Critical Appraisal: a Handbook for Health Care Professionals . London: Blackwell Medicine Publishing Group

Heller RF et al . (2008) Critical appraisal for public health: a new checklist. Public Health 122 : 92–98

Article   Google Scholar  

MacAuley D et al . (1998) Randomised controlled trial of the READER method of critical appraisal in general practice. BMJ 316 : 1134–37

Article   CAS   Google Scholar  

Parkes J et al . Teaching critical appraisal skills in health care settings (Review). Cochrane Database of Systematic Reviews 2005, Issue 3. Art. No.: cd001270. 10.1002/14651858.cd001270

Mays N and Pope C (2000) Assessing quality in qualitative research. BMJ 320 : 50–52

Hawking SW (2003) On the Shoulders of Giants: the Great Works of Physics and Astronomy . Philadelphia, PN: Penguin

National Health and Medical Research Council (1999) A Guide to the Development, Implementation and Evaluation of Clinical Practice Guidelines . Canberra: National Health and Medical Research Council

US Preventive Services Taskforce (1996) Guide to clinical preventive services (2nd Edn). Baltimore, MD: Williams & Wilkins

Solomon MJ and McLeod RS (1995) Should we be performing more randomized controlled trials evaluating surgical operations? Surgery 118 : 456–467

Rothman KJ (2002) Epidemiology: an Introduction . Oxford: Oxford University Press

Young JM and Solomon MJ (2003) Improving the evidence-base in surgery: sources of bias in surgical studies. ANZ J Surg 73 : 504–506

Margitic SE et al . (1995) Lessons learned from a prospective meta-analysis. J Am Geriatr Soc 43 : 435–439

Shea B et al . (2001) Assessing the quality of reports of systematic reviews: the QUORUM statement compared to other tools. In Systematic Reviews in Health Care: Meta-analysis in Context 2nd Edition, 122–139 (Eds Egger M. et al .) London: BMJ Books

Chapter   Google Scholar  

Easterbrook PH et al . (1991) Publication bias in clinical research. Lancet 337 : 867–872

Begg CB and Berlin JA (1989) Publication bias and dissemination of clinical research. J Natl Cancer Inst 81 : 107–115

Moher D et al . (2000) Improving the quality of reports of meta-analyses of randomised controlled trials: the QUORUM statement. Br J Surg 87 : 1448–1454

Shea BJ et al . (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology 7 : 10 [10.1186/1471-2288-7-10]

Stroup DF et al . (2000) Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 283 : 2008–2012

Young JM and Solomon MJ (2003) Improving the evidence-base in surgery: evaluating surgical effectiveness. ANZ J Surg 73 : 507–510

Schulz KF (1995) Subverting randomization in controlled trials. JAMA 274 : 1456–1458

Schulz KF et al . (1995) Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 273 : 408–412

Moher D et al . (2001) The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials. BMC Medical Research Methodology 1 : 2 [ http://www.biomedcentral.com/ 1471-2288/1/2 ] (accessed 25 November 2008)

Rochon PA et al . (2005) Reader's guide to critical appraisal of cohort studies: 1. Role and design. BMJ 330 : 895–897

Mamdani M et al . (2005) Reader's guide to critical appraisal of cohort studies: 2. Assessing potential for confounding. BMJ 330 : 960–962

Normand S et al . (2005) Reader's guide to critical appraisal of cohort studies: 3. Analytical strategies to reduce confounding. BMJ 330 : 1021–1023

von Elm E et al . (2007) Strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 335 : 806–808

Sutton-Tyrrell K (1991) Assessing bias in case-control studies: proper selection of cases and controls. Stroke 22 : 938–942

Knottnerus J (2003) Assessment of the accuracy of diagnostic tests: the cross-sectional study. J Clin Epidemiol 56 : 1118–1128

Furukawa TA and Guyatt GH (2006) Sources of bias in diagnostic accuracy studies and the diagnostic process. CMAJ 174 : 481–482

Bossyut PM et al . (2003)The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 138 : W1–W12

STARD statement (Standards for the Reporting of Diagnostic Accuracy Studies). [ http://www.stard-statement.org/ ] (accessed 10 September 2008)

Raftery J (1998) Economic evaluation: an introduction. BMJ 316 : 1013–1014

Palmer S et al . (1999) Economics notes: types of economic evaluation. BMJ 318 : 1349

Russ S et al . (1999) Barriers to participation in randomized controlled trials: a systematic review. J Clin Epidemiol 52 : 1143–1156

Tinmouth JM et al . (2004) Are claims of equivalency in digestive diseases trials supported by the evidence? Gastroentrology 126 : 1700–1710

Kaul S and Diamond GA (2006) Good enough: a primer on the analysis and interpretation of noninferiority trials. Ann Intern Med 145 : 62–69

Piaggio G et al . (2006) Reporting of noninferiority and equivalence randomized trials: an extension of the CONSORT statement. JAMA 295 : 1152–1160

Heritier SR et al . (2007) Inclusion of patients in clinical trial analysis: the intention to treat principle. In Interpreting and Reporting Clinical Trials: a Guide to the CONSORT Statement and the Principles of Randomized Controlled Trials , 92–98 (Eds Keech A. et al .) Strawberry Hills, NSW: Australian Medical Publishing Company

National Health and Medical Research Council (2007) National Statement on Ethical Conduct in Human Research 89–90 Canberra: NHMRC

Lo B et al . (2000) Conflict-of-interest policies for investigators in clinical trials. N Engl J Med 343 : 1616–1620

Kim SYH et al . (2004) Potential research participants' views regarding researcher and institutional financial conflicts of interests. J Med Ethics 30 : 73–79

Komesaroff PA and Kerridge IH (2002) Ethical issues concerning the relationships between medical practitioners and the pharmaceutical industry. Med J Aust 176 : 118–121

Little M (1999) Research, ethics and conflicts of interest. J Med Ethics 25 : 259–262

Lemmens T and Singer PA (1998) Bioethics for clinicians: 17. Conflict of interest in research, education and patient care. CMAJ 159 : 960–965

Download references

Author information

Authors and affiliations.

JM Young is an Associate Professor of Public Health and the Executive Director of the Surgical Outcomes Research Centre at the University of Sydney and Sydney South-West Area Health Service, Sydney,

Jane M Young

MJ Solomon is Head of the Surgical Outcomes Research Centre and Director of Colorectal Research at the University of Sydney and Sydney South-West Area Health Service, Sydney, Australia.,

Michael J Solomon

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Jane M Young .

Ethics declarations

Competing interests.

The authors declare no competing financial interests.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Young, J., Solomon, M. How to critically appraise an article. Nat Rev Gastroenterol Hepatol 6 , 82–91 (2009). https://doi.org/10.1038/ncpgasthep1331

Download citation

Received : 10 August 2008

Accepted : 03 November 2008

Published : 20 January 2009

Issue Date : February 2009

DOI : https://doi.org/10.1038/ncpgasthep1331

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

How to critically appraise an article

Affiliation.

• 1 Surgical Outcomes Research Centre, Royal Prince Alfred Hospital, Missenden Road, Sydney, NSW 2050, Australia. [email protected]
• PMID: 19153565
• DOI: 10.1038/ncpgasthep1331

Critical appraisal is a systematic process used to identify the strengths and weaknesses of a research article in order to assess the usefulness and validity of research findings. The most important components of a critical appraisal are an evaluation of the appropriateness of the study design for the research question and a careful assessment of the key methodological features of this design. Other factors that also should be considered include the suitability of the statistical methods used and their subsequent interpretation, potential conflicts of interest and the relevance of the research to one's own practice. This Review presents a 10-step guide to critical appraisal that aims to assist clinicians to identify the most relevant high-quality studies available to guide their clinical practice.

Publication types

• Decision Making*
• Evidence-Based Medicine*

Critical Appraisal: Assessing the Quality of Studies

• First Online: 05 August 2020

Cite this chapter

• Edward Purssell   ORCID: orcid.org/0000-0003-3748-0864 3 &
• Niall McCrae   ORCID: orcid.org/0000-0001-9776-7694 4  

9137 Accesses

There is great variation in the type and quality of research evidence. Having completed your search and assembled your studies, the next step is to critically appraise the studies to ascertain their quality. Ultimately you will be making a judgement about the overall evidence, but that comes later. You will see throughout this chapter that we make a clear differentiation between the individual studies and what we call the body of evidence , which is all of the studies and anything else that we use to answer the question or to make a recommendation. This chapter deals with only the first of these—the individual studies. Critical appraisal, like everything else in systematic literature reviewing, is a scientific exercise that requires individual judgement, and we describe some tools to help you.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Oxford Centre for Evidence-Based Medicine (OCEBM) (2016) OCEBM levels of evidence. In: CEBM. https://www.cebm.net/2016/05/ocebm-levels-of-evidence/ . Accessed 17 Apr 2020

Aromataris E, Munn Z (eds) (2017) Joanna Briggs Institute reviewer’s manual. The Joanna Briggs Institute, Adelaide

Google Scholar  

Daly J, Willis K, Small R et al (2007) A hierarchy of evidence for assessing qualitative health research. J Clin Epidemiol 60:43–49. https://doi.org/10.1016/j.jclinepi.2006.03.014

Article   PubMed   Google Scholar  

EQUATOR Network (2020) What is a reporting guideline?—The EQUATOR Network. https://www.equator-network.org/about-us/what-is-a-reporting-guideline/ . Accessed 7 Mar 2020

Tong A, Sainsbury P, Craig J (2007) Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 19:349–357. https://doi.org/10.1093/intqhc/mzm042

von Elm E, Altman DG, Egger M et al (2007) The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies. PLoS Med 4:e296. https://doi.org/10.1371/journal.pmed.0040296

Article   Google Scholar  

Brouwers MC, Kerkvliet K, Spithoff K, AGREE Next Steps Consortium (2016) The AGREE reporting checklist: a tool to improve reporting of clinical practice guidelines. BMJ 352:i1152. https://doi.org/10.1136/bmj.i1152

Article   PubMed   PubMed Central   Google Scholar  

Moher D, Liberati A, Tetzlaff J et al (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 6:e1000097. https://doi.org/10.1371/journal.pmed.1000097

Boutron I, Page MJ, Higgins JPT, Altman DG, Lundh A, Hróbjartsson A (2019) Chapter 7: Considering bias and conflicts of interest among the included studies. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (eds). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019), Cochrane. https://www.training.cochrane.org/handbook

Critical Appraisal Skills Programme (2018) CASP checklists. In: CASP—critical appraisal skills programme. https://casp-uk.net/casp-tools-checklists/ . Accessed 7 Mar 2020

Higgins JPT, Savović J, Page MJ et al (2019) Chapter 8: Assessing risk of bias in a randomized trial. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Chapter   Google Scholar  

Guyatt GH, Oxman AD, Kunz R et al (2011) GRADE guidelines 6. Rating the quality of evidence—imprecision. J Clin Epidemiol 64:1283–1293. https://doi.org/10.1016/j.jclinepi.2011.01.012

Sterne JAC, Savović J, Page MJ et al (2019) RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 366:l4898. https://doi.org/10.1136/bmj.l4898

Sterne JA, Hernán MA, Reeves BC et al (2016) ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355:i4919. https://doi.org/10.1136/bmj.i4919

Wells GA, Shea B, O’Connell D et al (2019) The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Ottawa Hospital Research Institute, Ottawa. http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp . Accessed 7 Mar 2020

Cochrane Community (2020) Glossary—Cochrane community. https://community.cochrane.org/glossary#letter-R . Accessed 8 Mar 2020

Messick S (1989) Meaning and values in test validation: the science and ethics of assessment. Educ Res 18:5–11. https://doi.org/10.3102/0013189X018002005

Sparkes AC (2001) Myth 94: qualitative health researchers will agree about validity. Qual Health Res 11:538–552. https://doi.org/10.1177/104973230101100409

Article   CAS   PubMed   Google Scholar  

Aguinis H, Solarino AM (2019) Transparency and replicability in qualitative research: the case of interviews with elite informants. Strat Manag J 40:1291–1315. https://doi.org/10.1002/smj.3015

Lincoln YS, Guba EG (1985) Naturalistic inquiry. Sage Publications, Beverly Hills, CA

Book   Google Scholar  

Hannes K (2011) Chapter 4: Critical appraisal of qualitative research. In: Noyes J, Booth A, Hannes K et al (eds) Supplementary guidance for inclusion of qualitative research in Cochrane systematic reviews of interventions. Cochrane Collaboration Qualitative Methods Group, London

Munn Z, Porritt K, Lockwood C et al (2014) Establishing confidence in the output of qualitative research synthesis: the ConQual approach. BMC Med Res Methodol 14:108. https://doi.org/10.1186/1471-2288-14-108

Toye F, Seers K, Allcock N et al (2013) ‘Trying to pin down jelly’—exploring intuitive processes in quality assessment for meta-ethnography. BMC Med Res Methodol 13:46. https://doi.org/10.1186/1471-2288-13-46

Katikireddi SV, Egan M, Petticrew M (2015) How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study. J Epidemiol Community Health 69:189–195. https://doi.org/10.1136/jech-2014-204711

McKenzie JE, Brennan SE, Ryan RE et al (2019) Chapter 9: Summarizing study characteristics and preparing for synthesis. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Deeks JJ, Higgins JPT, Altman DG (2019) Chapter 10: Analysing data and undertaking meta-analyses. In: Higgins JPT, Thomas J, Chandler J et al (eds) Cochrane handbook for systematic reviews of interventions version 6.0 (updated July 2019). Cochrane, London

Download references

Author information

Authors and affiliations.

School of Health Sciences, City, University of London, London, UK

Edward Purssell

Florence Nightingale Faculty of Nursing, Midwifery & Palliative Care, King’s College London, London, UK

Niall McCrae

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Edward Purssell .

Rights and permissions

Reprints and permissions

Copyright information

© 2020 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Purssell, E., McCrae, N. (2020). Critical Appraisal: Assessing the Quality of Studies. In: How to Perform a Systematic Literature Review. Springer, Cham. https://doi.org/10.1007/978-3-030-49672-2_6

Download citation

DOI : https://doi.org/10.1007/978-3-030-49672-2_6

Published : 05 August 2020

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-49671-5

Online ISBN : 978-3-030-49672-2

eBook Packages : Medicine Medicine (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

• En español – ExME
• Em português – EME

Critical Appraisal: A Checklist

Posted on 6th September 2016 by Robert Will

Critical appraisal of scientific literature is a necessary skill for healthcare students. Students can be overwhelmed by the vastness of search results. Database searching is a skill in itself, but will not be covered in this blog. This blog assumes that you have found a relevant journal article to answer a clinical question. After selecting an article, you must be able to sit with the article and critically appraise it. Critical appraisal of a journal article is a literary and scientific systematic dissection in an attempt to assign merit to the conclusions of an article. Ideally, an article will be able to undergo scrutiny and retain its findings as valid.

The specific questions used to assess validity change slightly with different study designs and article types. However, in an attempt to provide a generalized checklist, no specific subtype of article has been chosen. Rather, the 20 questions below should be used as a quick reference to appraise any journal article. The first four checklist questions should be answered “Yes.” If any of the four questions are answered “no,” then you should return to your search and attempt to find an article that will meet these criteria.

Critical appraisal of…the Introduction

• Does the article attempt to answer the same question as your clinical question?
• Is the article recently published (within 5 years) or is it seminal (i.e. an earlier article but which has strongly influenced later developments)?
• Is the journal peer-reviewed?
• Do the authors present a hypothesis?

Critical appraisal of…the Methods

• Is the study design valid for your question?
• Are both inclusion and exclusion criteria described?
• Is there an attempt to limit bias in the selection of participant groups?
• Are there methodological protocols (i.e. blinding) used to limit other possible bias?
• Do the research methods limit the influence of confounding variables?
• Are the outcome measures valid for the health condition you are researching?

Critical appraisal of…the Results

• Is there a table that describes the subjects’ demographics?
• Are the baseline demographics between groups similar?
• Are the subjects generalizable to your patient?
• Are the statistical tests appropriate for the study design and clinical question?
• Are the results presented within the paper?
• Are the results statistically significant and how large is the difference between groups?
• Is there evidence of significance fishing (i.e. changing statistical tests to ensure significance)?

Critical appraisal of…the Discussion/Conclusion

• Do the authors attempt to contextualise non-significant data in an attempt to portray significance? (e.g. talking about findings which had a  trend  towards significance as if they were significant).
• Do the authors acknowledge limitations in the article?
• Are there any conflicts of interests noted?

This is by no means a comprehensive checklist of how to critically appraise a scientific journal article. However, by answering the previous 20 questions based on a detailed reading of an article, you can appraise most articles for their merit, and thus determine whether the results are valid. I have attempted to list the questions based on the sections most commonly present in a journal article, starting at the introduction and progressing to the conclusion. I believe some of these items are weighted heavier than others (i.e. methodological questions vs journal reputation). However, without taking this list through rigorous testing, I cannot assign a weight to them. Maybe one day, you will be able to critically appraise my future paper:  How Online Checklists Influence Healthcare Students’ Ability to Critically Appraise Journal Articles.

Feature Image by Arek Socha from Pixabay

Robert Will

Leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

No Comments on Critical Appraisal: A Checklist

Hi Ella, I have found a checklist here for before and after study design: https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools and you may also find a checklist from this blog, which has a huge number of tools listed: https://s4be.cochrane.org/blog/2018/01/12/appraising-the-appraisal/

What kind of critical appraisal tool can be used for before and after study design article? Thanks

Hello, I am currently writing a book chapter on critical appraisal skills. This chapter is limited to 1000 words so your simple 20 questions framework would be the perfect format to cite within this text. May I please have your permission to use your checklist with full acknowledgement given to you as author? Many thanks

Thank you Robert, I came across your checklist via the Royal College of Surgeons of England website; https://www.rcseng.ac.uk/library-and-publications/library/blog/dissecting-the-literature-the-importance-of-critical-appraisal/ . I really liked it and I have made reference to it for our students. I really appreciate your checklist and it is still current, thank you.

Hi Kirsten. Thank you so much for letting us know that Robert’s checklist has been used in that article – that’s so good to see. If any of your students have any comments about the blog, then do let us know. If you also note any topics that you would like to see on the website, then we can add this to the list of suggested blogs for students to write about. Thank you again. Emma.

i am really happy with it. thank you very much

A really useful guide for helping you ask questions about the studies you are reviewing BRAVO

Dr.Suryanujella,

Thank you for the comment. I’m glad you find it helpful.

Feel free to use the checklist. S4BE asks that you cite the page when you use it.

I have read your article and found it very useful , crisp with all relevant information.I would like to use it in my presentation with your permission

That’s great thank you very much. I will definitely give that a go.

I find the MEAL writing approach very versatile. You can use it to plan the entire paper and each paragraph within the paper. There are a lot of helpful MEAL resources online. But understanding the acronym can get you started.

M-Main Idea (What are you arguing?) E-Evidence (What does the literature say?) A-Analysis (Why does the literature matter to your argument?) L-Link (Transition to next paragraph or section)

I hope that is somewhat helpful. -Robert

Hi, I am a university student at Portsmouth University, UK. I understand the premise of a critical appraisal however I am unsure how to structure an essay critically appraising a paper. Do you have any pointers to help me get started?

Thank you. I’m glad that you find this helpful.

Very informative & to the point for all medical students

How can I know what is the name of this checklist or tool?

This is a checklist that the author, Robert Will, has designed himself.

Thank you for asking. I am glad you found it helpful. As Emma said, please cite the source when you use it.

Greetings Robert, I am a postgraduate student at QMUL in the UK and I have just read this comprehensive critical appraisal checklist of your. I really appreciate you. if I may ask, can I have it downloaded?

Please feel free to use the information from this blog – if you could please cite the source then that would be much appreciated.

Robert Thank you for your comptrehensive account of critical appraisal. I have just completed a teaching module on critical appraisal as part of a four module Evidence Based Medicine programme for undergraduate Meducal students at RCSI Perdana medical school in Malaysia. If you are agreeable I would like to cite it as a reference in our module.

Anthony, Please feel free to cite my checklist. Thank you for asking. I hope that your students find it helpful. They should also browse around S4BE. There are numerous other helpful articles on this site.

Subscribe to our newsletter

You will receive our monthly newsletter and free access to Trip Premium.

Related Articles

Risk Communication in Public Health

Learn why effective risk communication in public health matters and where you can get started in learning how to better communicate research evidence.

Why was the CONSORT Statement introduced?

The CONSORT statement aims at comprehensive and complete reporting of randomized controlled trials. This blog introduces you to the statement and why it is an important tool in the research world.

Measures of central tendency in clinical research papers: what we should know whilst analysing them

Learn more about the measures of central tendency (mean, mode, median) and how these need to be critically appraised when reading a paper.

• CASP Checklists
• How to use our CASP Checklists
• Referencing and Creative Commons
• Online Training Courses
• CASP Workshops
• Subscriptions
• What is Critical Appraisal
• Study Designs
• Useful Links
• Bibliography
• View all Tools and Resources
• Testimonials

How to Critically Appraise a Research Paper

Research papers are a powerful means through which millions of researchers around the globe pass on knowledge about our world.

However, the quality of research can be highly variable. To avoid being misled, it is vital to perform critical appraisals of research studies to assess the validity, results and relevance of the published research. Critical appraisal skills are essential to be able to identify whether published research provides results that can be used as evidence to help improve your practice.

What is a critical appraisal?

Most of us know not to believe everything we read in the newspaper or on various media channels. But when it comes to research literature and journals, they must be critically appraised due to the nature of the context. In order for us to trust research papers, we want to be safe in the knowledge that they have been efficiently and professionally checked to confirm what they are saying. This is where a critical appraisal comes in.

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, value and relevance in a particular context. We have put together a more detailed page to explain what critical appraisal is to give you more information.

Why is a critical appraisal of research required?

Critical appraisal skills are important as they enable you to systematically and objectively assess the trustworthiness, relevance and results of published papers. When a research article is published, who wrote it should not be an indication of its trustworthiness and relevance.

What are the benefits of performing critical appraisals for research papers?

Performing a critical appraisal helps to:

• Reduce information overload by eliminating irrelevant or weak studies
• Identify the most relevant papers
• Distinguish evidence from opinion, assumptions, misreporting, and belief
• Assess the validity of the study
• Check the usefulness and clinical applicability of the study

How to critically appraise a research paper

There are some key questions to consider when critically appraising a paper. These include:

• Is the study relevant to my field of practice?
• What research question is being asked?
• Was the study design appropriate for the research question?

CASP has several checklists to help with performing a critical appraisal which we believe are crucial because:

• They help the user to undertake a complex task involving many steps
• They support the user in being systematic by ensuring that all important factors or considerations are taken into account
• They increase consistency in decision-making by providing a framework

In addition to our free checklists, CASP has developed a number of valuable online e-learning modules designed to increase your knowledge and confidence in conducting a critical appraisal.

Introduction To Critical Appraisal & CASP

This Module covers the following:

• Challenges using evidence to change practice
• 5 steps of evidence-based practice
• Developing critical appraisal skills
• Integrating and acting on the evidence
• The Critical Appraisal Skills Programme (CASP)
• Online Learning
• Quasi-Experimental Design
• The Role of Homogeneity in Research
• PICO Search Strategy Tips & Examples
• What Is a Subgroup Analysis?
• What Is Evidence-Based Practice?
• What Is A Cross-Sectional Study?
• What Is A PICO Tool?
• What Is A Pilot Study?
• Different Types of Bias in Research
• What is Qualitative Research?
• Privacy Policy

Critical Appraisal Skills Programme

Critical Appraisal Skills Programme (CASP) will use the information you provide on this form to be in touch with you and to provide updates and marketing. Please let us know all the ways you would like to hear from us:

We use Mailchimp as our marketing platform. By clicking below to subscribe, you acknowledge that your information will be transferred to Mailchimp for processing. Learn more about Mailchimp's privacy practices here.

Copyright 2024 CASP UK - OAP Ltd. All rights reserved Website by Beyond Your Brand

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

Critical Appraisal Toolkit (CAT) for assessing multiple types of evidence

• Author information
• Article notes
• Copyright and License information

Correspondence: [email protected]

Contributor: Jennifer Kruse, Public Health Agency of Canada – Conceptualization and project administration

Series information

Scientific writing

Collection date 2017 Sep 7.

Healthcare professionals are often expected to critically appraise research evidence in order to make recommendations for practice and policy development. Here we describe the Critical Appraisal Toolkit (CAT) currently used by the Public Health Agency of Canada. The CAT consists of: algorithms to identify the type of study design, three separate tools (for appraisal of analytic studies, descriptive studies and literature reviews), additional tools to support the appraisal process, and guidance for summarizing evidence and drawing conclusions about a body of evidence. Although the toolkit was created to assist in the development of national guidelines related to infection prevention and control, clinicians, policy makers and students can use it to guide appraisal of any health-related quantitative research. Participants in a pilot test completed a total of 101 critical appraisals and found that the CAT was user-friendly and helpful in the process of critical appraisal. Feedback from participants of the pilot test of the CAT informed further revisions prior to its release. The CAT adds to the arsenal of available tools and can be especially useful when the best available evidence comes from non-clinical trials and/or studies with weak designs, where other tools may not be easily applied.

Introduction

Healthcare professionals, researchers and policy makers are often involved in the development of public health policies or guidelines. The most valuable guidelines provide a basis for evidence-based practice with recommendations informed by current, high quality, peer-reviewed scientific evidence. To develop such guidelines, the available evidence needs to be critically appraised so that recommendations are based on the "best" evidence. The ability to critically appraise research is, therefore, an essential skill for health professionals serving on policy or guideline development working groups.

Our experience with working groups developing infection prevention and control guidelines was that the review of relevant evidence went smoothly while the critical appraisal of the evidence posed multiple challenges. Three main issues were identified. First, although working group members had strong expertise in infection prevention and control or other areas relevant to the guideline topic, they had varying levels of expertise in research methods and critical appraisal. Second, the critical appraisal tools in use at that time focused largely on analytic studies (such as clinical trials), and lacked definitions of key terms and explanations of the criteria used in the studies. As a result, the use of these tools by working group members did not result in a consistent way of appraising analytic studies nor did the tools provide a means of assessing descriptive studies and literature reviews. Third, working group members wanted guidance on how to progress from assessing individual studies to summarizing and assessing a body of evidence.

To address these issues, a review of existing critical appraisal tools was conducted. We found that the majority of existing tools were design-specific, with considerable variability in intent, criteria appraised and construction of the tools. A systematic review reported that fewer than half of existing tools had guidelines for use of the tool and interpretation of the items ( 1 ). The well-known Grading of Recommendations Assessment, Development and Evaluation (GRADE) rating-of-evidence system and the Cochrane tools for assessing risk of bias were considered for use ( 2 ), ( 3 ). At that time, the guidelines for using these tools were limited, and the tools were focused primarily on randomized controlled trials (RCTs) and non-randomized controlled trials. For feasibility and ethical reasons, clinical trials are rarely available for many common infection prevention and control issues ( 4 ), ( 5 ). For example, there are no intervention studies assessing which practice restrictions, if any, should be placed on healthcare workers who are infected with a blood-borne pathogen. Working group members were concerned that if they used GRADE, all evidence would be rated as very low or as low quality or certainty, and recommendations based on this evidence may be interpreted as unconvincing, even if they were based on the best or only available evidence.

The team decided to develop its own critical appraisal toolkit. So a small working group was convened, led by an epidemiologist with expertise in research, methodology and critical appraisal, with the goal of developing tools to critically appraise studies informing infection prevention and control recommendations. This article provides an overview of the Critical Appraisal Toolkit (CAT). The full document, entitled Infection Prevention and Control Guidelines Critical Appraisal Tool Kit is available online ( 6 ).

Following a review of existing critical appraisal tools, studies informing infection prevention and control guidelines that were in development were reviewed to identify the types of studies that would need to be appraised using the CAT. A preliminary draft of the CAT was used by various guideline development working groups and iterative revisions were made over a two year period. A pilot test of the CAT was then conducted which led to the final version ( 6 ).

The toolkit is set up to guide reviewers through three major phases in the critical appraisal of a body of evidence: appraisal of individual studies; summarizing the results of the appraisals; and appraisal of the body of evidence.

Tools for critically appraising individual studies

The first step in the critical appraisal of an individual study is to identify the study design; this can be surprisingly problematic, since many published research studies are complex. An algorithm was developed to help identify whether a study was an analytic study, a descriptive study or a literature review (see text box for definitions). It is critical to establish the design of the study first, as the criteria for assessment differs depending on the type of study.

Definitions of the types of studies that can be analyzed with the Critical Appraisal Toolkit*

Analytic study: A study designed to identify or measure effects of specific exposures, interventions or risk factors. This design employs the use of an appropriate comparison group to test epidemiologic hypotheses, thus attempting to identify associations or causal relationships.

Descriptive study: A study that describes characteristics of a condition in relation to particular factors or exposure of interest. This design often provides the first important clues about possible determinants of disease and is useful for the formulation of hypotheses that can be subsequently tested using an analytic design.

Literature review: A study that analyzes critical points of a published body of knowledge. This is done through summary, classification and comparison of prior studies. With the exception of meta-analyses, which statistically re-analyze pooled data from several studies, these studies are secondary sources and do not report any new or experimental work.

* Public Health Agency of Canada. Infection Prevention and Control Guidelines Critical Appraisal Tool Kit ( 6 )

Separate algorithms were developed for analytic studies, descriptive studies and literature reviews to help reviewers identify specific designs within those categories. The algorithm below, for example, helps reviewers determine which study design was used within the analytic study category ( Figure 1 ). It is based on key decision points such as number of groups or allocation to group. The legends for the algorithms and supportive tools such as the glossary provide additional detail to further differentiate study designs, such as whether a cohort study was retrospective or prospective.

Figure 1. Algorithm for identifying the type of analytic study.

Abbreviations: CBA, controlled before-after; ITS, interrupted time series; NRCT, non-randomized controlled trial; RCT, randomized controlled trial; UCBA, uncontrolled before-after

Separate critical appraisal tools were developed for analytic studies, for descriptive studies and for literature reviews, with relevant criteria in each tool. For example, a summary of the items covered in the analytic study critical appraisal tool is shown in Table 1 . This tool is used to appraise trials, observational studies and laboratory-based experiments. A supportive tool for assessing statistical analysis was also provided that describes common statistical tests used in epidemiologic studies.

Table 1. Aspects appraised in analytic study critical appraisal tool.

The descriptive study critical appraisal tool assesses different aspects of sampling, data collection, statistical analysis, and ethical conduct. It is used to appraise cross-sectional studies, outbreak investigations, case series and case reports.

The literature review critical appraisal tool assesses the methodology, results and applicability of narrative reviews, systematic reviews and meta-analyses.

After appraisal of individual items in each type of study, each critical appraisal tool also contains instructions for drawing a conclusion about the overall quality of the evidence from a study, based on the per-item appraisal. Quality is rated as high, medium or low. While a RCT is a strong study design and a survey is a weak design, it is possible to have a poor quality RCT or a high quality survey. As a result, the quality of evidence from a study is distinguished from the strength of a study design when assessing the quality of the overall body of evidence. A definition of some terms used to evaluate evidence in the CAT is shown in Table 2 .

Table 2. Definition of terms used to evaluate evidence.

* Considered strong design if there are at least two control groups and two intervention groups. Considered moderate design if there is only one control and one intervention group

Tools for summarizing the evidence

The second phase in the critical appraisal process involves summarizing the results of the critical appraisal of individual studies. Reviewers are instructed to complete a template evidence summary table, with key details about each study and its ratings. Studies are listed in descending order of strength in the table. The table simplifies looking across all studies that make up the body of evidence informing a recommendation and allows for easy comparison of participants, sample size, methods, interventions, magnitude and consistency of results, outcome measures and individual study quality as determined by the critical appraisal. These evidence summary tables are reviewed by the working group to determine the rating for the quality of the overall body of evidence and to facilitate development of recommendations based on evidence.

Rating the quality of the overall body of evidence

The third phase in the critical appraisal process is rating the quality of the overall body of evidence. The overall rating depends on the five items summarized in Table 2 : strength of study designs, quality of studies, number of studies, consistency of results and directness of the evidence. The various combinations of these factors lead to an overall rating of the strength of the body of evidence as strong, moderate or weak as summarized in Table 3 .

Table 3. Criteria for rating evidence on which recommendations are based.

A unique aspect of this toolkit is that recommendations are not graded but are formulated based on the graded body of evidence. Actions are either recommended or not recommended; it is the strength of the available evidence that varies, not the strength of the recommendation. The toolkit does highlight, however, the need to re-evaluate new evidence as it becomes available especially when recommendations are based on weak evidence.

Pilot test of the CAT

Of 34 individuals who indicated an interest in completing the pilot test, 17 completed it. Multiple peer-reviewed studies were selected representing analytic studies, descriptive studies and literature reviews. The same studies were assigned to participants with similar content expertise. Each participant was asked to appraise three analytic studies, two descriptive studies and one literature review, using the appropriate critical appraisal tool as identified by the participant. For each study appraised, one critical appraisal tool and the associated tool-specific feedback form were completed. Each participant also completed a single general feedback form. A total of 101 of 102 critical appraisals were conducted and returned, with 81 tool-specific feedback forms and 14 general feedback forms returned.

The majority of participants (>85%) found the flow of each tool was logical and the length acceptable but noted they still had difficulty identifying the study designs ( Table 4 ).

Table 4. Pilot test feedback on user friendliness.

* Number of tool-specific forms returned for total number of critical appraisals conducted

The vast majority of the feedback forms (86–93%) indicated that the different tools facilitated the critical appraisal process. In the assessment of consistency, however, only four of ten analytic studies appraised (40%), had complete agreement on the rating of overall study quality by participants, the other six studies had differences noted as mismatches. Four of the six studies with mismatches were observational studies. The differences were minor. None of the mismatches included a study that was rated as both high and low quality by different participants. Based on the comments provided by participants, most mismatches could likely have been resolved through discussion with peers. Mismatched ratings were not an issue for the descriptive studies and literature reviews. In summary, the pilot test provided useful feedback on different aspects of the toolkit. Revisions were made to address the issues identified from the pilot test and thus strengthen the CAT.

The Infection Prevention and Control Guidelines Critical Appraisal Tool Kit was developed in response to the needs of infection control professionals reviewing literature that generally did not include clinical trial evidence. The toolkit was designed to meet the identified needs for training in critical appraisal with extensive instructions and dictionaries, and tools applicable to all three types of studies (analytic studies, descriptive studies and literature reviews). The toolkit provided a method to progress from assessing individual studies to summarizing and assessing the strength of a body of evidence and assigning a grade. Recommendations are then developed based on the graded body of evidence. This grading system has been used by the Public Health Agency of Canada in the development of recent infection prevention and control guidelines ( 5 ), ( 7 ). The toolkit has also been used for conducting critical appraisal for other purposes, such as addressing a practice problem and serving as an educational tool ( 8 ), ( 9 ).

The CAT has a number of strengths. It is applicable to a wide variety of study designs. The criteria that are assessed allow for a comprehensive appraisal of individual studies and facilitates critical appraisal of a body of evidence. The dictionaries provide reviewers with a common language and criteria for discussion and decision making.

The CAT also has a number of limitations. The tools do not address all study designs (e.g., modelling studies) and the toolkit provides limited information on types of bias. Like the majority of critical appraisal tools ( 10 ), ( 11 ), these tools have not been tested for validity and reliability. Nonetheless, the criteria assessed are those indicated as important in textbooks and in the literature ( 12 ), ( 13 ). The grading scale used in this toolkit does not allow for comparison of evidence grading across organizations or internationally, but most reviewers do not need such comparability. It is more important that strong evidence be rated higher than weak evidence, and that reviewers provide rationales for their conclusions; the toolkit enables them to do so.

Overall, the pilot test reinforced that the CAT can help with critical appraisal training and can increase comfort levels for those with limited experience. Further evaluation of the toolkit could assess the effectiveness of revisions made and test its validity and reliability.

A frequent question regarding this toolkit is how it differs from GRADE as both distinguish stronger evidence from weaker evidence and use similar concepts and terminology. The main differences between GRADE and the CAT are presented in Table 5 . Key differences include the focus of the CAT on rating the quality of individual studies, and the detailed instructions and supporting tools that assist those with limited experience in critical appraisal. When clinical trials and well controlled intervention studies are or become available, GRADE and related tools from Cochrane would be more appropriate ( 2 ), ( 3 ). When descriptive studies are all that is available, the CAT is very useful.

Table 5. Comparison of features of the Critical Appraisal Toolkit (CAT) and GRADE.

Abbreviation: GRADE, Grading of Recommendations Assessment, Development and Evaluation

The Infection Prevention and Control Guidelines Critical Appraisal Tool Kit was developed in response to needs for training in critical appraisal, assessing evidence from a wide variety of research designs, and a method for going from assessing individual studies to characterizing the strength of a body of evidence. Clinician researchers, policy makers and students can use these tools for critical appraisal of studies whether they are trying to develop policies, find a potential solution to a practice problem or critique an article for a journal club. The toolkit adds to the arsenal of critical appraisal tools currently available and is especially useful in assessing evidence from a wide variety of research designs.

Authors’ Statement

DM – Conceptualization, methodology, investigation, data collection and curation and writing – original draft, review and editing

TO – Conceptualization, methodology, investigation, data collection and curation and writing – original draft, review and editing

KD – Conceptualization, review and editing, supervision and project administration

Acknowledgements

We thank the Infection Prevention and Control Expert Working Group of the Public Health Agency of Canada for feedback on the development of the toolkit, Lisa Marie Wasmund for data entry of the pilot test results, Katherine Defalco for review of data and cross-editing of content and technical terminology for the French version of the toolkit, Laurie O’Neil for review and feedback on early versions of the toolkit, Frédéric Bergeron for technical support with the algorithms in the toolkit and the Centre for Communicable Diseases and Infection Control of the Public Health Agency of Canada for review, feedback and ongoing use of the toolkit. We thank Dr. Patricia Huston, Canada Communicable Disease Report Editor-in-Chief, for a thorough review and constructive feedback on the draft manuscript.

Conflict of interest: None.

Funding: This work was supported by the Public Health Agency of Canada.

• 1. Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol 2004. Sep;4:22. 10.1186/1471-2288-4-22 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 2. GRADE Working Group. Criteria for applying or using GRADE. www.gradeworkinggroup.org [Accessed July 25, 2017].
• 3. Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration, 2011. http://handbook.cochrane.org
• 4. Khan AR, Khan S, Zimmerman V, Baddour LM, Tleyjeh IM. Quality and strength of evidence of the Infectious Diseases Society of America clinical practice guidelines. Clin Infect Dis 2010. Nov;51(10):1147–56. 10.1086/656735 [ DOI ] [ PubMed ] [ Google Scholar ]
• 5. Public Health Agency of Canada. Routine practices and additional precautions for preventing the transmission of infection in healthcare settings. http://www.phac-aspc.gc.ca/nois-sinp/guide/summary-sommaire/tihs-tims-eng.php [Accessed December 4, 2015].
• 6. Public Health Agency of Canada. Infection Prevention and Control Guidelines Critical Appraisal Tool Kit. http://publications.gc.ca/collections/collection_2014/aspc-phac/HP40-119-2014-eng.pdf [Accessed December 4, 2015].
• 7. Public Health Agency of Canada. Hand hygiene practices in healthcare settings. https://www.canada.ca/en/public-health/services/infectious-diseases/nosocomial-occupational-infections/hand-hygiene-practices-healthcare-settings.html [Accessed December 4, 2015].
• 8. Ha S, Paquette D, Tarasuk J, Dodds J, Gale-Rowe M, Brooks JI et al. A systematic review of HIV testing among Canadian populations. Can J Public Health 2014. Jan;105(1):e53–62. 10.17269/cjph.105.4128 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 9. Stevens LK, Ricketts ED, Bruneau JE. Critical appraisal through a new lens. Nurs Leadersh (Tor Ont) 2014. Jun;27(2):10–3. 10.12927/cjnl.2014.23843 [ DOI ] [ PubMed ] [ Google Scholar ]
• 10. Lohr KN. Rating the strength of scientific evidence: relevance for quality improvement programs. Int J Qual Health Care 2004. Feb;16(1):9–18. 10.1093/intqhc/mzh005 [ DOI ] [ PubMed ] [ Google Scholar ]
• 11. Crowe M, Sheppard L. A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed. J Clin Epidemiol 2011. Jan;64(1):79–89. 10.1016/j.jclinepi.2010.02.008 [ DOI ] [ PubMed ] [ Google Scholar ]
• 12. Young JM, Solomon MJ. How to critically appraise an article. Nat Clin Pract Gastroenterol Hepatol 2009. Feb;6(2):82–91. 10.1038/ncpgasthep1331 [ DOI ] [ PubMed ] [ Google Scholar ]
• 13. Polit DF, Beck CT. Nursing Research: Generating and Assessing Evidence for Nursing Practice. 9 th ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2008. Chapter XX, Literature reviews: Finding and critiquing the evidence p. 94-125. [ Google Scholar ]
• View on publisher site
• PDF (348.4 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections

How To Write a Critical Appraisal

A critical appraisal is an academic approach that refers to the systematic identification of strengths and weakness of a research article with the intent of evaluating the usefulness and validity of the work’s research findings. As with all essays, you need to be clear, concise, and logical in your presentation of arguments, analysis, and evaluation. However, in a critical appraisal there are some specific sections which need to be considered which will form the main basis of your work.

Structure of a Critical Appraisal

Introduction.

Your introduction should introduce the work to be appraised, and how you intend to proceed. In other words, you set out how you will be assessing the article and the criteria you will use. Focusing your introduction on these areas will ensure that your readers understand your purpose and are interested to read on. It needs to be clear that you are undertaking a scientific and literary dissection and examination of the indicated work to assess its validity and credibility, expressed in an interesting and motivational way.

Body of the Work

The body of the work should be separated into clear paragraphs that cover each section of the work and sub-sections for each point that is being covered. In all paragraphs your perspectives should be backed up with hard evidence from credible sources (fully cited and referenced at the end), and not be expressed as an opinion or your own personal point of view. Remember this is a critical appraisal and not a presentation of negative parts of the work.

When appraising the introduction of the article, you should ask yourself whether the article answers the main question it poses. Alongside this look at the date of publication, generally you want works to be within the past 5 years, unless they are seminal works which have strongly influenced subsequent developments in the field. Identify whether the journal in which the article was published is peer reviewed and importantly whether a hypothesis has been presented. Be objective, concise, and coherent in your presentation of this information.

Once you have appraised the introduction you can move onto the methods (or the body of the text if the work is not of a scientific or experimental nature). To effectively appraise the methods, you need to examine whether the approaches used to draw conclusions (i.e., the methodology) is appropriate for the research question, or overall topic. If not, indicate why not, in your appraisal, with evidence to back up your reasoning. Examine the sample population (if there is one), or the data gathered and evaluate whether it is appropriate, sufficient, and viable, before considering the data collection methods and survey instruments used. Are they fit for purpose? Do they meet the needs of the paper? Again, your arguments should be backed up by strong, viable sources that have credible foundations and origins.

One of the most significant areas of appraisal is the results and conclusions presented by the authors of the work. In the case of the results, you need to identify whether there are facts and figures presented to confirm findings, assess whether any statistical tests used are viable, reliable, and appropriate to the work conducted. In addition, whether they have been clearly explained and introduced during the work. In regard to the results presented by the authors you need to present evidence that they have been unbiased and objective, and if not, present evidence of how they have been biased. In this section you should also dissect the results and identify whether any statistical significance reported is accurate and whether the results presented and discussed align with any tables or figures presented.

The final element of the body text is the appraisal of the discussion and conclusion sections. In this case there is a need to identify whether the authors have drawn realistic conclusions from their available data, whether they have identified any clear limitations to their work and whether the conclusions they have drawn are the same as those you would have done had you been presented with the findings.

The conclusion of the appraisal should not introduce any new information but should be a concise summing up of the key points identified in the body text. The conclusion should be a condensation (or precis) of all that you have already written. The aim is bringing together the whole paper and state an opinion (based on evaluated evidence) of how valid and reliable the paper being appraised can be considered to be in the subject area. In all cases, you should reference and cite all sources used. To help you achieve a first class critical appraisal we have put together some key phrases that can help lift you work above that of others.

Key Phrases for a Critical Appraisal

• Whilst the title might suggest
• The focus of the work appears to be…
• The author challenges the notion that…
• The author makes the claim that…
• The article makes a strong contribution through…
• The approach provides the opportunity to…
• The authors consider…
• The argument is not entirely convincing because…
• However, whilst it can be agreed that… it should also be noted that…
• Several crucial questions are left unanswered…
• It would have been more appropriate to have stated that…
• This framework extends and increases…
• The authors correctly conclude that…
• The authors efforts can be considered as…
• Less convincing is the generalisation that…
• This appears to mislead readers indicating that…
• This research proves to be timely and particularly significant in the light of…

Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

Critical Appraisal tools

Critical appraisal worksheets to help you appraise the reliability, importance and applicability of clinical evidence.

Critical appraisal is the systematic evaluation of clinical research papers in order to establish:

• Does this study address a  clearly focused question ?
• Did the study use valid methods to address this question?
• Are the valid results of this study important?
• Are these valid, important results applicable to my patient or population?

If the answer to any of these questions is “no”, you can save yourself the trouble of reading the rest of it.

This section contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples.

Critical Appraisal Worksheets

• Systematic Reviews  Critical Appraisal Sheet
• Diagnostics  Critical Appraisal Sheet
• Prognosis  Critical Appraisal Sheet
• Randomised Controlled Trials  (RCT) Critical Appraisal Sheet
• Critical Appraisal of Qualitative Studies  Sheet
• IPD Review  Sheet

Chinese - translated by Chung-Han Yang and Shih-Chieh Shao

• Systematic Reviews  Critical Appraisal Sheet
• Diagnostic Study  Critical Appraisal Sheet
• Prognostic Critical Appraisal Sheet
• RCT  Critical Appraisal Sheet
• IPD reviews Critical Appraisal Sheet
• Qualitative Studies Critical Appraisal Sheet 

German - translated by Johannes Pohl and Martin Sadilek

• Systematic Review  Critical Appraisal Sheet
• Diagnosis Critical Appraisal Sheet
• Prognosis Critical Appraisal Sheet
• Therapy / RCT Critical Appraisal Sheet

Lithuanian - translated by Tumas Beinortas

• Systematic review appraisal Lithuanian (PDF)
• Diagnostic accuracy appraisal Lithuanian  (PDF)
• Prognostic study appraisal Lithuanian  (PDF)
• RCT appraisal sheets Lithuanian  (PDF)

Portugese - translated by Enderson Miranda, Rachel Riera and Luis Eduardo Fontes

• Portuguese – Systematic Review Study Appraisal Worksheet
• Portuguese – Diagnostic Study Appraisal Worksheet
• Portuguese – Prognostic Study Appraisal Worksheet
• Portuguese – RCT Study Appraisal Worksheet
• Portuguese – Systematic Review Evaluation of Individual Participant Data Worksheet
• Portuguese – Qualitative Studies Evaluation Worksheet

Spanish - translated by Ana Cristina Castro

• Systematic Review  (PDF)
• Diagnosis  (PDF)
• Prognosis  Spanish Translation (PDF)
• Therapy / RCT  Spanish Translation (PDF)

Persian - translated by Ahmad Sofi Mahmudi

• Prognosis  (PDF)
• PICO  Critical Appraisal Sheet (PDF)
• PICO Critical Appraisal Sheet (MS-Word)
• Educational Prescription  Critical Appraisal Sheet (PDF)

Explanations & Examples

• Pre-test probability
• SpPin and SnNout
• Likelihood Ratios

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

Critical appraisal of the literature. Why do we care?

Avaliação crítica da literatura. por que nos importamos, juliana carvalho ferreira, cecilia maria patino.

• Author information
• Article notes
• Copyright and License information

Issue date 2018 Nov-Dec.

This is an open-access article distributed under the terms of the Creative Commons Attribution License

PRACTICAL SCENARIO

Investigators conducted a noninferiority, double-blind clinical trial involving 4,215 patients with mild asthma, randomly assigned to receive twice-daily placebo plus budesonide-formoterol used as needed vs. maintenance therapy with twice-daily budesonide plus terbutaline as needed. They found that budesonide-formoterol used as needed was noninferior to twice-daily budesonide concerning the rate of severe asthma exacerbations but was inferior in controlling symptoms. 1

HOW TO CRITICALLY APPRAISE THE MEDICAL LITERATURE

As clinicians, when we read a paper reporting the benefit of a given intervention, we make a judgment regarding whether we should use those results to inform how we care for our patients. In our example, after reading the paper, we ask ourselves: should a clinician working in a public hospital in Brazil start prescribing budesonide-formoterol as needed rather than maintenance budesonide for her patients with mild asthma? What criteria should guide her decision to adopt a new intervention? One may think that if a study is published in a high-impact, peer-reviewed journal, it is of high quality and should therefore be used to guide clinical decision making. However, if the population included in the study or the context is different from her population, that may not be the case. Therefore, examining the external validity of a study is critical to informing local practice.

Other commonly used criteria are related to evaluating the quality of the evidence by evaluating the type of study design used. The pyramid of evidence puts meta-analyses at the top (as providing the highest quality of evidence), followed by systematic reviews and randomized controlled trials; then come observational studies (cohort, case-control, and cross-sectional studies); whereas case reports and case series are categorized as offering the lowest quality of evidence. Although those criteria may be helpful, making a detailed appraisal of a paper, taking into account aspects other than the study design, is a skill that researchers and clinicians can learn and apply when reading the literature.

Critical appraisal is the systematic evaluation of clinical research papers that helps us establish if the results are valid and if they could be used to inform medical decision in a given local population and context. There are several published guidelines for critically appraising the scientific literature, most of which are structured as checklists and address specific study designs. 2 Although different appraisal tools may vary, the general structure is shown in Table 1 .

Table 1. How to appraise medical literature.

The items in Table 1 are a guide to appraising the content of a research article. There are also guidelines for appraising the quality of reporting of health research which focus on the reporting accuracy and completeness of research studies. 3 These two types of appraisal (content and reporting) are complementary and should both be used, because it is possible that a research paper has high reporting quality but is not relevant to the context in question.

KEY MESSAGE

Critical appraisal of the literature is an essential skill for researchers and clinicians, and there are easy-to-use guidelines. Clinicians have the responsibility to help patients make health-related decisions, which should be based on high-quality, valid research that is applicable in their context.

• 1. Bateman ED, Reddel HK, O’Byrne PM, Barnes PJ, Zhong N, Keen C, et al. As-Needed Budesonide-Formoterol versus Maintenance Budesonide in Mild Asthma. N Engl J Med. 2018;378(20):1877–1887. doi: 10.1056/NEJMoa1715275. [ DOI ] [ PubMed ] [ Google Scholar ]
• 2. Critical Appraisal Skills Programme . Oxford (UK): CASP; 2018. https://casp-uk.net/casp-tools-checklists [ Google Scholar ]
• 3. Equator network . Oxford (UK): Centre for Statistics in Medicine, University of Oxford; https://www.equator-network.org [ Google Scholar ]
• View on publisher site
• PDF (498.9 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections

Loading metrics

Open Access

Peer-reviewed

Research Article

Harnessing nature-based solutions for economic recovery: A systematic review

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Software, Validation, Visualization, Writing – original draft, Writing – review & editing

Affiliation Nature-based Solutions Initiative, Department of Biology, University of Oxford, Oxford, United Kingdom

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Software, Validation, Visualization, Writing – original draft, Writing – review & editing

Affiliations Nature-based Solutions Initiative, Department of Biology, University of Oxford, Oxford, United Kingdom, Environmental Change Institute, University of Oxford, Oxford, United Kingdom

Roles Data curation, Formal analysis, Writing – review & editing

Affiliation Department of Economics, Stanford University, Stanford, California, United States of America

Roles Conceptualization, Investigation, Methodology, Validation, Writing – original draft, Writing – review & editing

Affiliations Institute for New Economic Thinking, Oxford Martin School, University of Oxford, Oxford, United Kingdom, Smith School of Enterprise and the Environment, School of Geography and the Environment, University of Oxford, Oxford, United Kingdom

Roles Investigation, Validation, Writing – review & editing

Affiliation Instituto de Montaña, Lima, Peru

Roles Conceptualization, Funding acquisition, Investigation, Resources, Supervision, Validation, Visualization, Writing – review & editing

* E-mail: [email protected]

Affiliations Nature-based Solutions Initiative, Department of Biology, University of Oxford, Oxford, United Kingdom, Smith School of Enterprise and the Environment, School of Geography and the Environment, University of Oxford, Oxford, United Kingdom

• Alexandre Chausson, 
• Alison Smith, 
• Ryne Zen-Zhi Reger, 
• Brian O’Callaghan, 
• Yadira Mori Clement, 
• Florencia Zapata, 
• Nathalie Seddon

• Published: October 28, 2024
• https://doi.org/10.1371/journal.pclm.0000281
• Reader Comments

Nature-based solutions (NbS) involve working with nature to address societal challenges in ways that benefit communities and biodiversity locally. However, their role supporting economic recovery from crises, such as those arising from conflicts or pandemics remains underexplored. To address this knowledge gap, we conducted a systematic review of 66 reviews on the economic impact of nature-based interventions. Most demonstrated positive outcomes for income and employment, though those with critical appraisal of underlying studies reported more mixed outcomes. These varied results were influenced by factors such as the balance between short-term and long-term gains, market conditions, regional effects, reliance on subsidies, and discrepancies between expected and actual economic benefits. National-scale economic growth assessments were scarce. Half of the cases featured nature-based food production investments, with much evidence from sub-Saharan Africa, East Asia and the Pacific. The few reviews comparing NbS with alternatives found that NbS delivered equal or better economic outcomes. NbS also provided broader benefits like food and water security, flood protection and community empowerment. We identified key factors influencing the delivery of benefits and trade-offs, finding that NbS must adhere to best practice standards, with community involvement being critical for equitable outcomes. Well-designed NbS can create diverse job opportunities at different skill levels, diversify income, and improve resilience, offering a rapid, flexible response to economic shocks that can be targeted at deprived communities. By integrating traditional, local and scientific knowledge, NbS can enable eco-innovation, and drive the transition to a clean and efficient circular economy, with high economic multipliers spreading benefits throughout economies. The evidence underscores the need to incorporate NbS in investment programs to concurrently address economic, environmental, and societal challenges. However, improved monitoring of economic, social and ecological outcomes and the development of comprehensive accounting systems are needed to better track public and private investments in NbS.

Citation: Chausson A, Smith A, Reger RZ-Z, O’Callaghan B, Mori Clement Y, Zapata F, et al. (2024) Harnessing nature-based solutions for economic recovery: A systematic review. PLOS Clim 3(10): e0000281. https://doi.org/10.1371/journal.pclm.0000281

Editor: Jennifer Lee Wilkening, US Fish and Wildlife Service, UNITED STATES OF AMERICA

Received: February 15, 2024; Accepted: August 19, 2024; Published: October 28, 2024

Copyright: © 2024 Chausson et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: A dataset including the list of included and excluded reviews. The dataset is now uploaded.

Funding: This work was supported with funding from the Oxford Martin School’s Biodiversity and Society Programme (University of Oxford) (to NS) and the Waterloo Foundation (to NS). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests: The authors have declared that no competing interests exist.

Introduction

The vital role of nature-based solutions (NbS) for reducing vulnerability to climate change [ 1 , 2 ] whilst also increasing carbon sequestration and reducing greenhouse gas emissions [ 3 , 4 ] is now widely recognized. There is also growing awareness that NbS could play a key role in recovery from economic shocks, including those related to conflicts or pandemics. Indeed, the COVID pandemic raised awareness of the importance of nature in addressing root causes of zoonotic disease emergence (human encroachment in wildlife habitat) and improving human wellbeing (e.g. [ 5 ]). However, despite the focus on ‘building back better’, there has been limited attention to how investments in nature can also drive economic recovery. By 2020, only 3% of COVID-19 recovery spending appeared likely to support investment in nature, while up to 17% risked negatively impacting it through new infrastructure, defense spending, and other measures [ 6 ]. Several barriers hinder the mainstreaming of NbS investments, including path dependency [ 7 ], siloed government decision-making [ 8 , 9 ], the pervasive misconception that environmental protection harms business [ 10 ], limited awareness [ 11 ], lack of skills, and uncertainty over the economic benefits of NbS compared to alternatives [ 36 ].

Fiscal policy (i.e. government spending and taxation) can be a powerful lever for influencing total demand for goods and services, particularly during economic downturn (see [ 12 , 13 ]), thereby promoting recovery. Faced with the need to act rapidly, economists have advised that policy makers should respond with measures that are “timely, targeted, and temporary”. However, this implies little consideration for the long-term impacts of policy, meaning that the relative benefits of more socially useful or long-term activities might not be appropriately considered. Keynes suggested that priority investments during the latter stages of the US depression should be in “durable goods such as housing, public utilities, and transport”, noting that “the necessities for such developments were unexampled” [ 14 ]. Given limited funds and capacity to secure finance, it is important that policymakers consider how short-term fiscal measures might influence long-term outcomes [ 6 , 15 ]. This is particularly important in emerging market and developing economies (EMDEs), where fiscal space is often tightly constrained and new debt is expensive. Put differently, policies that bring long-term debt servicing costs should deliver long-term assets that support well-being ([ 6 ]; see S3 Text for a glossary of terms). Biodiversity and long-term resilience are just some of the factors that might be harmed when recovery investments do not consider long-term needs [ 16 ].

In the context of post-pandemic economic recovery, it has been proposed that investments in measures reducing greenhouse gas emissions might offer economic benefits equivalent to, or perhaps greater than, traditional investments [ 6 , 17 – 20 ]. Building on investigations into low-carbon energy and energy efficiency during the Global Financial Crisis (GFC), [ 21 – 25 ], it was suggested that investing in nature could be an attractive option for rapid implementation [ 6 , 19 , 26 ].

NbS—formally defined by the United Nations Environment Assembly as “actions to protect, conserve, restore, sustainably use and manage natural or modified terrestrial, freshwater, coastal and marine ecosystems, which address social, economic and environmental challenges effectively and adaptively, while supporting human well-being, ecosystem services, resilience, and biodiversity benefits [ 27 ]” —have several characteristics that make them well-suited to support economic recovery. First, once designed, some NbS can be deployed relatively quickly [ 6 , 28 ]. Second, they can create demand for both skilled labor (e.g., for mapping, design, monitoring and evaluation) and low-skilled labor, making them particularly useful in addressing high unemployment among unskilled workers [ 6 , 29 ]. Third, many NbS are viable in rural areas where populations are vulnerable [ 19 , 30 , 31 ], unlike with other low-carbon initiatives such as public transport investment that require high population density to be cost-effective. Fourth, NbS can support climate change mitigation and adaptation and can be integrated with built infrastructure [ 1 , 3 , 4 , 32 ]. They also support many other sustainable development goals by helping to address pollution, food, and water security, while protecting and restoring biodiversity and human well-being [ 33 – 36 ]. However, realizing these benefits requires conscientious design and implementation, informed by a robust understanding of potential trade-offs and equitable distribution of costs and benefits. Therefore, alongside their economic potential, it is crucial to understand when, where, and how NbS can deliver biodiversity, climate, and social benefits, and ensure that these gains are distributed fairly across different groups.

Previous research suggests that investments in nature (e.g., restoration) deliver high gross value added and higher returns per unit of investment than other sectors [ 10 , 29 , 37 ]. However, existing research is mainly limited to project-specific or sector-specific outcomes with a lack of evidence synthesis across the full range of NbS. Existing reviews typically cover specific sub-types of NbS, specific geographical locations, or a subset of economic outcomes. The highly dispersed nature of the evidence challenges the uptake of NbS research to inform fiscal policy measures. Furthermore, recent assessments [ 38 ] have not investigated economic recovery potential at a high enough granularity to compare short versus long-term economic characteristics of NbS, and their risks and opportunities. There is also a need to frame the evidence to support systemic policy change, requiring comparison of NbS to other economic recovery options. Finally, there has been a lack of focus on how different benefits are delivered, and how these socially disaggregate. In the absence of such information, nature can be sidelined in economic recovery policies, locking in the continued destruction of nature, with severe impacts for climate, biodiversity, and livelihoods. Economic stimulus packages therefore require robust evidence-based guidelines around what good investments in NbS look like and the benefits they can bring.

To address these issues, we conducted a systematic review of reviews [ 39 ] on the economic outcomes of investments in nature, and the pathways by which these benefits are delivered, focusing on jobs and labor demand, household income and business revenue generation, and economic growth. Reviews of reviews, or “umbrella reviews”, predominantly carried out in health and medical fields, allow rapid assessment of the evidence across a broad range of outcomes, interventions, and contexts amidst a rapidly increasing number of primary research studies [ 40 ].

Our focus was guided by the recognition that decision-makers involved in fiscal policy —our prime target audience— focus on economic criteria such as fiscal multipliers (leading to GDP growth) and job creation. We recognize that GDP growth is an inadequate measure of human progress and well-being [ 41 , 42 ], and that perpetual growth in a finite world severely jeopardizes progress towards addressing the climate and biodiversity crises [ 16 , 43 , 44 ]. A vast array of social and environmental factors shape well-being. These include material circumstances (e.g., income, livelihoods, health, the environment), social dynamics (e.g., community relations), and subjective wellbeing (e.g., psychological health) [ 45 ], many of which are closely tied to our relationship with nature, its ecosystems, landscapes, and nonhuman species [ 6 , 46 ]. Therefore, although we focus on conventional economic outcomes for jobs, incomes and growth, we also discuss the vital role of NbS in supporting many of these wider societal benefits.

Our overarching questions are:

• What is the distribution of the evidence on the economic impact of NbS between different regions, types of NbS, ecosystems and economic outcomes?

What are the reported economic impacts of nature-based solutions?

• How do nature-based solutions contribute to economic impact?
• What are the reported trade-offs and win-wins between economic impact outcomes, and biodiversity or climate outcomes?

How are costs and benefits distributed across social groups?

We address those questions by a) exploring the scope of NbS outcomes reported under the umbrella of ‘economic impact’ in the peer-reviewed literature; b) synthesizing this evidence with respect to geography, ecosystem, and type of intervention; c) highlighting trade-offs and win-wins in relation to biodiversity, climate, and social equity; and d) identifying how NbS deliver economic impacts (pathways and mediating factors). Unpacking when and where NbS deliver benefits, and for whom, is crucial to tailor and target NbS in fiscal policy measures to support broader climate and biodiversity objectives, including addressing potential trade-offs and win-wins for resilient development. Our primary method is a systematic review of the literature on the economic outcomes of NbS, but we supplement this with a review of the wider outcomes of NbS for sustainable development, and a detailed case study to add depth and nuance to our understanding. We also highlight knowledge gaps and biases in the literature, with recommendations for practitioners and researchers to support future evidence collection. In addressing these questions, our goal is to enable well-targeted scientific research on NbS to play a stronger role in informing fiscal policy. We conclude with a set of recommendations for policy makers.

Systematic review protocol

We drafted a systematic review protocol, including a conceptual framework ( S1 Text ), to catalogue evidence in a transparent and objective manner [ 47 ]. We revised the question scope ( Table 1 ), search string, review selection criteria, and coding framework (see S1 and S5 Texts) in early 2022 through meetings and workshops with an interdisciplinary group of experts in academia, civil society, and government, covering expertise on NbS and economic impact (see Acknowledgements and S1 Text ). We designed the coding framework to ensure relevance for policymakers focusing on economic policy, including economic recovery, while also noting any reported outcomes for climate and biodiversity.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pclm.0000281.t001

Searches and screening process

We ran the search string for English publications in SCOPUS and Web of Science CORE index collections incorporating indexed up to February 15, 2023, restricting the search to title, abstract content, and author keywords, and refining the search to articles tagged as review. We removed duplicates in EndNote (v8.2) and exported search results into Rayyan [ 48 ] for screening using a stepwise procedure, screening first reference titles, then abstracts. We progressively refined selection criteria for clarity and inter-reviewer consistency, and further refined these criteria after abstract screening to produce a manageable number of studies, based on time and team capacity constraints (see Table 1 and Table F in S5 Text ). We included only those studies where the methodology for the review was clearly described.

Decisions at each stage of screening were conservative; we assessed studies for which inclusion eligibility was unclear at the next stage. We randomly selected at least 10% of references to check for inter-reviewer coding consistency with a Kappa test. If the Kappa coefficient was below 0.6 (the threshold at which inter-reviewer coding consistency is deemed sufficient; [ 49 ], we reviewed any emerging inconsistencies and revised the screening strategy and selection criteria for clarity. We carried out single reviewer screening cautiously, i.e. checking screening consistency throughout the process. Approximately 15% of all screening decisions at the abstract and full-text stages were made by at least two reviewers. Studies excluded during full text screening, and reasons for their exclusion, are available in the supporting information ( S1 Table ). Inclusion decisions were guided by whether the review reported one or more economic impact(s) stemming from nature-based interventions, regardless of the aim of the intervention. We did not narrow our scope to studies explicitly using the terminology of NbS or interventions meeting all NbS criteria [ 50 , 51 ], because this would have excluded many relevant studies. Hence, hereafter we refer to nature-based interventions instead of NbS. In some reviews, the extent to which interventions supported biodiversity or local communities was heavily context dependent (depending on how the intervention was implemented). We did not exclude these reviews unless the information reported indicated that the interventions did not support (or were harmful) to biodiversity or local communities. In other words, if it was not clear whether an intervention fully met the criteria to be an NbS (with benefits for both biodiversity and local communities), we gave interventions the benefit of the doubt, but if it was clear that the intervention was not an NbS then it was excluded.

Coding strategy

The extraction of evidence from studies was guided by a coding framework developed from the conceptual framework ( S2 Text ) and entered in Excel by 3 coders (AC, AS, and RZR), with approximately 30% of the studies checked by at least 2 coders to ensure consistency. The coding framework captured data at three levels: for each review, for each intervention covered by a review, and for each outcome type recorded for an intervention.

For each review, we recorded bibliographic details and quality criteria such as whether the review was systematic and whether it excluded studies with no comparator. To map the distribution of evidence across geographies, we recorded which world regions or specific nations were associated with the evidence reported, following the World Bank regional classification scheme (2020) [ 52 ].

For each intervention, we recorded the broad category: (i) protection, (ii) restoration, (iii) other forms of management (hereafter management), (iv) creation of novel ecosystems, and (v) nature-based food production (see S2 Text for definitions). Ecosystems in which interventions took place were grouped into 28 categories, drawing from the typology devised for a systematic map of nature-based interventions to adaptation [ 1 ] to which we added categories for working landscapes (cropland, pastures, agroforestry, plantations, aquaculture) and urban green infrastructure.

For each outcome, we recorded the outcome type, description and direction of effect (positive, negative, mixed, no effect, or unclear). Outcomes were classed as mixed if a mix of positive and negative outcomes were recorded by the component papers of the review, or unclear where component papers found that evidence for outcome direction was inconclusive. Outcome types included i) income, revenue and profitability (thereafter income/revenue), ii) employment and labor demand (thereafter labor demand/job creation), iii) job security, iv) skills and training, v) economic growth and multiplier effects (thereafter economic growth). These are all interconnected, as economic growth is a function of income, income is related to employment levels, and job security, skills and training all affect income and employment. For labor demand, we coded increased labor demand as a positive outcome on the macro level, noting that in some micro studies (e.g., for nature-based food production) increased labor was viewed as a negative outcome because it led to increased production costs.

Reported outcomes did not need to be associated with a comparator (for example, if a review reported overall revenue generated, it was coded as positive, unless a baseline assessment was provided indicating that income generation was insufficient to overcome opportunity costs). To characterize the extent of evidence for each outcome category, we also captured the number of underlying studies associated with each outcome statement (where the information was provided by the review). We did not explore whether there was any overlap in the primary studies covered by different reviews due to time limitation, but significant overlaps seem unlikely given that most reviews covered quite different combinations of intervention types and geographical regions.

In addition to recording the economic outcomes, we also recorded whether wider outcomes for ecology, climate change or social equity were considered by the assessments. Ecological outcomes included those associated with species conservation, habitat quality, diversity (e.g., species richness), or resilience of natural ecosystems. Climate change mitigation outcomes included avoided greenhouse gas emissions, or changes in below or above ground carbon storage. For climate change adaptation, we coded outcomes for addressing vulnerability (exposure, sensitivity, or adaptive capacity) to climate change impacts or other hydrometeorological hazards, including climate hazards which may or may not be explicitly linked to climate change. Equity effects were identified as any reported distribution of outcomes across social groups, either within communities embedded in the intervention landscapes, or between local communities and external stakeholders (government, private sector and investors, or civil society organizations). Outcomes were deemed to be positive for equity if they resulted in benefits for low income or marginalized groups, and negative if benefits flowed primarily to high income beneficiaries or those with political power and influence.

Data analysis and mapping

The evidence base was characterized through descriptive statistics, mapping the number and percentage of studies with respect to methodology, geographical region, intervention type, type of ecosystem, type of outcomes, and associations between economic outcomes and intervention type. We then analyzed the direction of reported economic outcomes (positive, negative, mixed, or neutral), any comparisons with alternative approaches, any reported effects on climate change (adaptation and mitigation), and trade-offs and win-wins. For each review, reported evidence disaggregated by intervention (by the review authors) was recorded as a distinct case. Where absolute numbers are shown in figures, we only report percentages in the text. When proportions or counts are provided without an explicit sample size, it should be assumed that the calculation includes the entire set of studies, interventions, or outcomes.

We summarize reported effectiveness of interventions to characterize the evidence base and guide future analyses. Meta-analysis was not possible given the heterogeneity of the evidence and the underpinning review methodologies. This also precludes weighing reported categorical outcomes by strength of evidence, although we recorded the number of underlying papers supporting each outcome within each review. Because of the heterogeneity and context-dependence of the evidence base (meaning that there were a relatively low number of reviews covering each specific combination of intervention type, outcome and context), the results should not be used to generalize the effectiveness of a particular intervention type. To test the impact of evidence quality on the likelihood of reporting a positive economic impact, we considered whether the review was categorized as systematic or not, whether critical appraisal was undertaken, and whether the sample size (the number of evidence points underpinning the reported effect), was associated with the likelihood of reporting a positive effect. We employed mixed effects logistic regression models using R version 4.4.1, accounting for the nested structure of the data (multiple observations within the same article). The lme4 package was used to fit these models, with articleID specified as the random effect to account for within-article correlations. The dependent variable was binary (positive effect, or not), and the independent variables included appraisal (yes or no), article type (systematic or not), sample size, intervention type, and outcome category. To maintain simplicity and address reduced sample sizes for sub-categories, separate models were run for each predictor variable (see S5 Text for full models). Confidence intervals for the model coefficients were calculated using Wald confidence intervals.

Pathways and mediating factors

Within each review, we inductively extracted the pathways and mechanisms through which nature-based interventions were reported to shape economic outcomes. Relevant passages were extracted into Excel, and progressively refined to identify emergent categories (see S2 Text , Pathway definitions). Interventions and outcomes described within a review can be associated with one or more pathway categories. For example, a nature-based food production intervention such as agroforestry may boost yield (and hence income) by improved ecosystem services (such as pollination and erosion protection) and could also be associated with increased income via payment for ecosystem service (PES) schemes designed to promote adoption or offset opportunity costs.

We also conducted an analysis of mediating factors, i.e. any factors reported to modify the outcome of the intervention (see mediating factors in S2 Text ). First, we grouped mediating factors according to seven categories following categories of ecosystem-based adaptation constraints identified by Nalau et al. (2018) [ 53 ], in which most mediating factors fit. These are economic and financial, governance and institutional, social and cultural, biological, physical, or human resources. We added the category ‘technical factors’ to capture intervention design elements under the deliberate control of implementers (whether physical or biological). We then extracted and coded relevant passages by the relevant category. We coded mediating factors for each review, as disaggregating mediating factors for each intervention was not always possible. We counted the number of times each mediating factor category was represented across reviews (if more than one factor was identified in a review for a given category, we only counted that category once). The analysis of mediating factors and pathways is not exhaustive and is limited by the extent to which they were reported by review authors but provides an important window into the diversity of factors (internal or external) which shape the economic impact of nature-based solutions.

Trade-off and win-win analysis

We extracted all passages in the reviews explicitly mentioning trade-offs and win-wins and categorized them according to whether they specified trade-offs or win-wins between outcomes, between stakeholders, across time (e.g., short-term costs vs long term benefits), or spatially (e.g., costs in one area, benefits in another). Social trade-offs and win-wins were extracted from the previously coded material describing distributional effects and equity. We then identified emerging themes and summarized these narratively within each category along with descriptive statistics (number and percentage of studies reporting each category). We also explored associations between reported outcomes for climate (adaptation and mitigation) and economic impact, even if not explicitly reported as a trade-off or win-win by the underlying reviews.

As well as incomes and employment, NbS can deliver a wide range of societal and environmental benefits, many of which are crucial to support economic prosperity. To illustrate this, we conducted a supplementary analysis of a previous systematic review dataset, drawn from both academic and grey literature, which coded the outcomes of nature-based interventions for development in the Global South, focusing on interventions that delivered climate change adaptation outcomes [ 54 ].

Studies identified and methodological approaches adopted

The number of articles retained or excluded at each stage of the searching and screening process is shown schematically in Fig 1 . The search of literature reviews on the economic impact of nature-based interventions identified a total of 2,405 studies in Web of Science, and 1,261 in Scopus, resulting in 3,121 references after duplicate removal. After title, abstract and full text screening, 219 of these met initial selection criteria (Table F in S5 Text ). These were published across 99 academic journals, from 1996 to 2023. Only 66 of these specified a methodology, and therefore were included in our review. Of these, half (36) were categorized by the journal or labeled by the authors as systematic reviews, although not all conformed fully to established systematic review standards [ 47 ]. Only 21% (14) conducted some level of quality appraisal of the underlying studies, and only 29% (19) restricted the review to primary studies that used comparators (such as counterfactuals, baselines, or controls).

https://doi.org/10.1371/journal.pclm.0000281.g001

What is the distribution of the evidence on the economic impact of nature-based solutions?

Across the 66 reviews, we identified 95 intervention cases (as a review can have more than one intervention), reporting 168 distinct economic outcomes. The reviews reported between 1 and 9 intervention cases each (mean ± SD = 1.5 ± 1.4), and each intervention case was associated with between 1 and 4 reported outcomes (mean ± SD = 1.8 ± 0.8). Most outcome assessments were based on quantitative data (47%) or both qualitative and quantitative data (14%); 21% were qualitative, and for 18% the type of data was unclear.

Variation in numbers of reviews by region.

The most frequently represented region (noting that reviews often cover more than one region) was sub-Saharan Africa (covered in 44% of reviews), followed by South Asia (35%), East Asia & Pacific (30%), Latin America & Caribbean (18%), and Europe & Central Asia (15%) ( Fig 2A ). For most reviews, the geographical scope of the data synthesized was global (27, 41% of studies), followed by national (21, 32%), regional (13, 20%), and sub-national (3, 5%). Only one review was local.

Number of reviews covering (a) world region (World Bank, 2020), and number of interventions by (b) the broad type of NbS (c) ecosystem category, and (d) economic outcome type. A review or intervention can cover more than one of each category; note that only the most represented (top 6) ecosystem types are indicated.

https://doi.org/10.1371/journal.pclm.0000281.g002

Type of nature-based interventions.

Intervention cases were associated with up to five different broad intervention types (i.e. protection, restoration, management, creation of novel ecosystems or nature-based food production; see S1 Text ) (mean = 1.43, S.D. = 0.78). The most frequently represented type of intervention was nature-based food production (56% of cases) followed by management (33%), protection (27%), restoration (16%), and creation of novel ecosystems (12%) ( Fig 2B ). However, many interventions (31%) used a combination of these approaches (e.g., community-based natural management with natural resource use restrictions was coded as both protection and management). While 48% involved only nature-based food production, just 13% involved only management, 4% involved only creation of novel ecosystems, 4% involved only protection, and none involved only restoration.

Table 2 provides examples of the types of actions within each intervention category. Nature-based food production interventions involved a range of measures in rural working landscapes, plus one case of urban agriculture in South Africa. Of these, 45% involved measures targeting soil health (e.g., conservation tillage, cover crops, mulching), while 62% involved measures for above ground diversification (e.g., agroforestry (including silvopasture), intercropping, farmer-managed natural regeneration). Interventions involving elements of ecosystem protection included marine and terrestrial protected areas, resource use and access restrictions, and forest-based ecotourism. Interventions categorized as management involved community-based forest or fisheries management, forest management certification, grassland management, or indigenous practices to harvest NTFPs. Restoration measures included forest or rangeland restoration, or invasive species removal. Finally, interventions creating novel ecosystems involved urban nature-based solutions (e.g., green roofs or walls), or afforestation (i.e. planting trees on naturally treeless habitats or creating plantations of non-native species). Note that afforestation typically does not provide benefits for biodiversity, so it is not considered to be an NbS unless it is part of a process aimed at supporting landscape regeneration (e.g., by rehabilitating degraded land).

Interventions may not meet all guidelines for nature-based solutions (NbS) in practice, but we include evidence from all interventions because it is generally not possible to evaluate which are NbS with the information provided in each review, and it is also needed to build an understanding of what makes for effective NbS. A sample of references for each intervention is provided.

https://doi.org/10.1371/journal.pclm.0000281.t002

Ecosystem type.

Most intervention cases (79%) were associated with working landscapes (croplands, grazing lands and agroforestry), followed by forests (39%), (primarily tropical and subtropical forests), grasslands (16%), plantations (13%), and coastal ecosystems (11%) ( Fig 2C ). Of these, 52 (55%) intervention cases only involved created ecosystems or working landscapes, 27 (28%) only involved natural or semi-natural ecosystems, and 11 (12%) involved a mix of semi-natural/natural and working landscapes or novel ecosystems. Few studies reported on freshwater habitats (6, 6%), urban green infrastructure (5, 5%), oceans and seas (5, 5%), or desert and xeric shrublands (5, 5%), and none reported evidence from interventions involving aquaculture, mangroves, or peatlands.

Economic outcomes.

Overall, 96% of intervention cases reported outcomes for income/revenue, 46% for labor demand/job creation, 19% for skills and training, 11% for economic growth, and 7% for job security ( Fig 2D ). We also recorded the number of studies within each review that provided evidence to support each outcome assessment to understand the relative size of the evidence base. We found that 66% (1214) of the underlying studies provided evidence on income/revenue, followed by labor demand/job creation (21%, 391 studies), job security (6%, 109), economic growth (4%, 78), and skills and training (3%, 46).

Only 9 reviews reported evidence of indirect labor demand/job creation, such as where revenue from ecotourism provided indirect employment for transport and local food production to supply eco-lodges in Sri Lanka [ 55 ]. Of reviews reporting changes in labor demand/job creation only four reported on the length of employment, and only one quantified the proportion of short-term and long-term jobs [ 56 ]. Most outcome assessments were reported at the farm level or household level (35%), followed by community-level (14%), and sub-national scale (11%). Only 13 (8%) were national scale.

Associations between economic outcome and type of nature-based intervention.

We mapped associations between intervention category and outcome type, treating combined interventions as a separate category ( Fig 3 ). This revealed clusters of evidence for the income/revenue outcomes of nature-based food production (45 cases, 98% of all interventions involving nature-based food production) and combined interventions (27 cases, 93%), with smaller clusters for the labor outcomes of combined interventions (16 cases, 55%) and nature-based food production (20 cases, 43%), the income/revenue outcomes of management interventions (11 cases, 92%) and the skills or training outcomes of combined interventions (9 cases, 31%). Most of the limited evidence on economic growth and job security was associated with combined interventions (5 cases, 17%; and 4 cases, 14%, respectively).

https://doi.org/10.1371/journal.pclm.0000281.g003

Most reported outcome effects were positive (65%), with 25% mixed and only a few unclear (5%), negative (3%), or neutral (2%) ( Fig 4 ). The pattern for income/revenue outcomes matched the overall pattern, with most effects positive (67%), 25% mixed, and few unclear, negative, or neutral (3%, 2%, and 2% respectively). Two thirds (8, 67%) of the interventions framing increasing labor as negative (i.e. a cost) were associated with mixed positive and negative effects on labor demand. In contrast, where labor was framed as positive (for job creation; primarily for interventions other than nature-based food production) most reported outcomes (21, 75%) were positive.

https://doi.org/10.1371/journal.pclm.0000281.g004

However, the reviews that conducted critical appraisal reported a higher proportion of mixed effects (16, 53%) and a lower proportion of positive effects (12, 40%) compared to those that did not (26, 18% mixed and 97, 70% positive). Critical appraisal was found to be significantly associated with a decrease in the likelihood of reporting positive outcomes (Coefficient = -1.789, SE = 0.6815, z = -2.625, p = 0.009, 95% CI [-3.124, -0.453]; Table A in S5 Text ). Outcome type did not affect the relationship, except for job security (Coefficient = -2.673, SE = 1.3478, z = -1.983, p = 0.047, 95% CI [-5.315, -0.032]; Table A in S5 Text ) where there was a lower likelihood of a positive effect (see job security pathways below). In a separate model, intervention category was not significantly associated with the reported effect, whereas critical appraisal remained significantly associated with the likelihood of reporting a positive result (Coefficient = -2.072, SE = 0.7237, z = -2.863, p = 0.004, 95% CI [-3.490, -0.654]; Table B in S5 Text ).

The review category (systematic or not) was not associated with effect. However, in this model, there was, again, a decreased likelihood of a positive effect reported for job security (Coefficient = -2.571, SE = 1.3089, z = -1.964, p = 0.050, 95% CI [-5.137, -0.006]; Table C in S5 Text ). In a separate model examining the association with intervention category, there was a significant increase in the likelihood of reporting positive effects for nature-based food production (Coefficient = 1.267, SE = 0.613, z = 2.066, p = 0.039, 95% CI [0.065, 2.469]; Table D in S5 Text ). This association may be explained by the higher proportion of ’nature-based food production’ studies reporting positive effects, across economic impact categories, within the subset of systematic reviews compared to other intervention types. None of the other intervention types or outcome categories were associated with reported effect. Finally, we found no significant association between sample size and the reported effect (Table E in S5 Text ).

In the subset of reviews which had conducted critical appraisal, mixed effects arose for different reasons. First, variability in underlying studies contributed to the overall mixed categorization, as different studies report varying results for the same intervention type. In some cases, short-term income gains are observed, but the sustainability of these gains over the long term is uncertain, or vice-versa, some interventions may not be immediately profitable but could offer benefits over a longer period. The effect on income/revenue generation was also affected by external factors, including market conditions, and region-specific effects, with some areas showing significant benefits while others did not. Many interventions rely on external subsidies for financial sustainability, and without these subsidies, they might not be viable in the short term, such as in the case of certification and community-forest management. Additionally, some studies reported a gap between expected economic benefits (e.g., from price premiums) and the actual realized benefits, leading to mixed outcomes.

Few outcomes were reported for job security, 50% of which were mixed, or for economic growth, of which most (90%) were positive. For example, revenues from the sale of NTFPs (e.g., aromatic resins in Ethiopia) can contribute substantially to national economies [ 57 ], nature-based ecotourism stimulates local business development [ 58 ], and restoration investments in the US were found to yield as many as 33 jobs per $1 million invested, with an economic output multiplier between 1.6 and 2.59 [ 10 ].

Proportionally more reported effects on income/revenue were positive for nature-based food production, while there were proportionally more mixed outcomes for interventions involving protection, management, or restoration. There were no clear differences between intervention types for employment outcomes, apart for interventions involving nature-based food production where a greater proportion of reported outcomes were mixed (for the reason mentioned above).

Overall, few cases (12) reported positive contributions to skills and training, with two cases reporting mixed effects, and two reporting neutral outcomes. Investments in capacity strengthening either targeted technical skill building for the intervention itself (e.g., extension and training programs for agroforestry [ 59 ], crop-livestock integration [ 60 ], to meet certification requirements [ 61 ], or for alien species management [ 62 ]), or were complementary (e.g., business skills to establish agri-businesses and micro-enterprises [ 58 , 63 ]). Neutral effects reflected a lack of investment in capacity building (e.g. [ 64 ]), or where interventions did not require specialized skills (in turn providing low entry barriers to the labor market; [ 65 ]). Two reviews reported mixed effects, where the capacity building did not train workers with transferable skills, thereby limiting their opportunities to integrate into labor markets subsequently [ 56 ], or where the training prioritized quick environmental results over deep, enduring community benefits [ 62 ].

Viewing the number of underlying studies within each review reveals that although the overall patterns are similar, the evidence on skills and training and economic growth comes from a small number of studies ( Fig 5 ).

https://doi.org/10.1371/journal.pclm.0000281.g005

Effectiveness of nature-based interventions compared to alternative approaches.

Overall, 24 (36%) of the studies compared interventions involving Nature-based Solutions (NbS) with either non-NbS alternatives (21, 32%) or other NbS (10, 15%). Of the 26 non-NbS comparisons, the majority (17, 65%) showed positive outcomes, 19% (5) were negative, and the rest (15%, 4) had mixed or no significant effects. These comparisons mainly focused on nature-based agricultural practices like conservation agriculture or agroforestry versus conventional methods, highlighting benefits such as improved soil health, water retention, increased yields over time, and reduced production costs [ 66 – 68 ]. Several reviews found agroforestry offered higher productivity and more stable yields than crop monocultures [ 59 , 69 ]. Non-agricultural NbS comparisons (5 in total) explored revenue generation or profit margins. Interventions included forest management, where FSC certified management was found less profitable due to high costs outweighing price premiums [ 70 ], and decentralized forest management showing advantages for local communities over centralized approaches [ 57 , 70 ]. Green urban infrastructure, like green roofs, was noted for not being cost-effective for building owners despite broader societal benefits [ 71 ]. Additionally, the restoration industry was reported to have employment multiplier effects comparable to traditional sectors like oil and gas or construction [ 10 ].

Through what pathways do nature-based solutions contribute to economic impact?

All but two of the 66 reviews contained evidence on the pathways by which economic outcomes were delivered. We identified 12 distinct pathways by which NbS contributed to income/revenue (across 61 reviews), 8 pathways for effects on labor demand/job creation (across 31 reviews), 8 for economic growth (out of 10 reviews), and 5 pathways for job security (across 5 reviews).

Outcome pathways.

Income , revenue , or profitability pathways . These pathways fell into five overarching categories: 1) higher or new revenue generation (e.g., from the sale of goods (e.g., fish, NTFP, crops), services (e.g., offset credits), or property taxes), 2) avoided costs (e.g., energy savings from green roofs and walls, or reduced input costs for agriculture), 3) household income from employment generation, 4) labor shifts to off-farm jobs, which can be higher paid, and 5) household, business or community revenue from subsidies or payments for ecosystem services.

The most common pathway was where investment in nature-based food production influenced income (30 reviews, 50% of all income/revenue generation pathways), followed by revenue from payments for ecosystem service schemes (10 reviews, 17%), and revenue generation through ecotourism (8, 13%) ( Fig 5 ). The least commonly cited pathways included revenue generation through offset credit sales (for carbon storage [ 72 ] or wetland restoration [ 10 ]), where green infrastructure generated employment or ecosystem services reducing costs (e.g., reduced energy consumption through the installation of green roofs [ 71 ]), marine protected areas increasing or sustaining fishery catch [ 73 ], and conservation easements or green infrastructure increasing property values and generating tax revenue [ 71 , 74 ].

For eight out of the 12 pathways for income/revenue, most reviews reported positive effects ( Fig 6 ). For nature-based food production, benefits occurred through reduced input and labor costs [ 66 ], reduced exposure to income volatility (such as from diversified income streams or resilience to extreme weather [ 75 ]), and increased yield or output [ 76 – 78 ]. Key to these pathways is the positive effect of nature-based food production on ecosystem services (e.g., pollination, pest control, soil health), thereby also improving job security [ 76 ] and climate change adaptation.

https://doi.org/10.1371/journal.pclm.0000281.g006

For the other four pathways, at least half of the outcomes were mixed. This included cases where price premiums for certified goods were insufficient to overcome implementation costs [ 79 ], where producers became over-specialized in the certified commodity, thereby becoming more exposed to price downturns [ 80 ], where offset credit revenues were less than opportunity costs of land-use restrictions [ 81 , 82 ], where there was a lack of market access [ 83 ], or where yield fell after transitioning to agroforestry from monoculture [ 70 , 84 ]. Other factors potentially negatively impacting income included choice of crops [ 78 ], costs of human-wildlife conflict [ 64 ], or lack of available off-farm employment following restrictions in land-use. The one review reporting a purely negative impact was where the equipment and labor costs of conservation tillage were generally not offset by increased yield, especially where herbicides were used [ 85 ].

Labor demand/job creation pathways . The most common employment pathways involved nature-based food production (10, 32% of the reviews reporting labor pathways), ecotourism (6, 19%), green infrastructure or restoration investments (5, 16%), all of which generally increased labor demand ( Fig 7 ). Positive employment outcomes also occurred through revenue generated by community forest management, and through increased ecosystem services including the sale of NTFPs or increased fishing revenue adjacent to MPAs [ 86 ].

https://doi.org/10.1371/journal.pclm.0000281.g007

Mixed or negative impacts on employment occurred where there was a lack of ecotourism (e.g., due to low wildlife densities or lack of investment in in tourism operation; [ 64 ]), from shifts to off-farm labor following land-use restrictions for landscape regeneration [ 83 ], or where nature-based food production led to increases and decreases in labor demand, such as through reductions in labor demands for agrochemical application and increasing labor demand for hedge maintenance [ 75 ].

Job security pathways . Job security was reported to increase where agricultural diversification stabilized revenue streams [ 76 ], or where community-forestry strengthened ownership, use and access rights [ 72 ]. However, a lack of focus on transferable skill development can lead to job insecurity once the intervention ends due to challenges in integrating other sectors [ 56 ], or due to a lack of formal employment opportunities (such as where urban green infrastructure is established and maintained by informal workers) [ 65 ]. Furthermore, although nature-based tourism can create jobs, the unpredictable nature of tourist demand, like during the COVID-19 pandemic, can result in revenue and job losses [ 58 ].

Economic growth pathways . Impacts on economic growth were reported to emerge through business creation and revenues generated by ecotourism, [ 58 , 87 ], the sale of NTFPs [ 57 , 85 , 88 , 89 ], and investments in restoration which generated labor demand, business-to-business expenditures, and household spending with high economic multipliers [ 10 ]. Mixed (though mainly positive) effects on household expenditure were found under PES schemes (although a lack of data was noted), with revenue from PES also contributing to infrastructure construction (e.g., schools, clinics, power grids) [ 83 ]. Practices like agroecology, permaculture, and organic farming, along with investments in value chains, can improve economic prosperity by increasing market access, regional trade, and product quality [ 90 ].

Mediating factors . Across outcome pathways, we identified up to 18 distinct mediating factors per review (avg = 5.8; S.D. = 3.9) across 63 (95% of) included reviews. Mediating factors often influenced more than one outcome pathway, either positively or negatively. They included factors internal to the intervention (e.g., the density of trees in agroforestry, or the degree of stakeholder engagement), or external (e.g., legislative and regulatory frameworks, or the level of public and private finance). The most frequently identified category was economic and financial, reported in 70% of reviews, followed by technical factors (65%), governance and institutional factors (55%), and social and cultural factors (47%) ( Fig 8 ). Given heterogeneity in review methodology, quality, and scope of analysis, we advise caution in associating these proportions with overall prevalence. Mediating factors within each category are detailed in S4 Text and Table H in S5 Text .

For each category, the number of reviews specifying one or more mediating factors was summed up. See S4 Text for category definitions.

https://doi.org/10.1371/journal.pclm.0000281.g008

What trade-offs and win-wins are reported?

Overall, 51 (77%) of the reviews explicitly reported evidence of trade-offs or win-wins, but 11 noted a lack of data. Trade-offs and win-wins were either between outcomes (37, 73%), between stakeholders (distributional effects and equity) (32, 63%), over space (7, 14%), or over time (7, 14%).

Among reviews reporting trade-offs or win-wins between outcomes, 24 (65%) reported trade-offs between economic impact and biodiversity or ecosystem health, and 20 (54%) reported win-wins with biodiversity or ecosystem health. The most frequently reported trade-offs or win-wins were between biodiversity and provisioning ecosystem services, e.g., production of food or timber. Only 12 reviews explicitly reported win-wins and no-trade offs. For the reviews reporting distributional effects (i.e. how costs and benefits disaggregate across social groups), most (28, 88%) highlighted mixed or negative effects on equity (e.g., where income inequality increased between social groups). Six studies found positive economic and equity impacts, such as more equitable land holdings and social stability [ 76 ], improved gender equity [ 91 ], or increased employment for marginalized groups [ 92 ]. However, three of these also reported negative equity effects, such improved income equity within group (herders) but not between groups (between herders and other rural land users) [ 93 ], or where labor burden disproportionately fell on women [ 92 ]. All reviews explicitly reporting on spatial or temporal dimensions focused on trade-offs rather than win-wins. For example, short-term trade-offs occurred where high implementation costs or slow system maturity in nature-based food production led to a period of reduced profit subsequently offset by longer term increased yield or more resilient production over time [ 59 , 79 ]. Spatial trade-offs resulted from leakage, with displacement of ecosystem loss and degradation to neighboring areas [ 81 , 94 , 95 ].

Trade-offs between outcomes.

The most frequently reported trade-offs were between biodiversity and income or profitability, which can arise due to several mechanisms. First, restricting the use of natural resources in areas that are being protected or restored can reduce incomes, e.g., when pastoralists lost their livelihoods due grazing bans aimed at restoring degraded grassland in China [ 81 ]. Second, some reviews noted cases where nature-based production methods were less profitable than conventional methods, e.g., if the shade cast by agroforestry trees reduces yield, or where agroforestry or organic cropping systems optimized for cash crops provide higher returns but lower biodiversity [ 59 , 76 , 80 ]. Third, high implementation or labor costs can reduce profits, e.g., for agroforestry [ 75 ] or conservation agriculture where manual weeding is necessary (the alternative being the use of herbicides, which involves a further trade-off with biodiversity) [ 96 ]. Fourth, poor intervention design or management focused on short term profits can lead to adverse biodiversity outcomes, e.g., where ecotourism geared at maximizing tourism leads to environmental damage in protected areas [ 55 , 87 ], or in low biodiversity systems, such as tree monocultures (which are not NbS) [ 97 ]. Finally, ecosystem protection can be associated with increasing human-wildlife conflicts, reducing crop yield [ 70 ]. According to the sampled reviews, the extent of profitability trade-offs for nature-based food production depended on whether farmers received price premiums for nature-friendly products (e.g., through certification schemes) or whether compensation or subsidies offset opportunity and implementation costs (e.g., through PES for agroforestry) [ 59 ].

Win-wins between outcomes.

Several win-wins were reported in the literature. Agro-diversification was reported to drive increased profits, either from greater yield (e.g., integrated crop-livestock farming [ 75 ]), access to premium prices in markets (e.g., agroforestry [ 59 , 75 ]), the generation of multiple income streams [ 98 ], or reduced dependence on expensive inputs [ 92 ]. It was also found to reduce the risk of economic loss by promoting food production resilience, such as through crop rotation [ 75 ], intercropping [ 75 ], agroforestry [ 75 , 76 ], or integrated crop-livestock farming [ 75 ]) (see outcome pathways for more detail). Other nature-based food production measures reported to enhance ecosystem services and boost yield included climate-smart agriculture which reduced soil salinity, sustaining soil health and soil ecosystem services [ 73 ], crop residue retention and increased weed herbivory rate under conservation agriculture [ 85 ], or mulching and zero tillage [ 99 ]. Agroecological approaches boosted productivity and food security by improving soil health and biodiversity, which in turn promoted diversified and stable livelihoods [ 92 ]. Finally, win-wins were observed for conservancy schemes adjacent to protected areas in Namibia which harmonized biodiversity conservation with local livelihoods [ 100 ], or where payment for ecosystem service programs boosted income while reducing grazing pressures on grasslands in China [ 93 ].

Relationship between economic impact and climate change effect.

Most reviews did not directly compare economic impacts with effects on climate change adaptation or mitigation); therefore, we report associations between them instead. For adaptation, 23 (46% of those reporting on adaptation) found positive outcomes for both adaptation and economic impact, mainly in nature-based food production (see outcome pathways for more detail). Positive effects on both mitigation and economic outcomes were found in 11 (44%) studies reporting on mitigation, often through strategies like improved yields, reduced costs, or land regeneration while reducing emissions or enhancing sequestration. Trade-offs, where outcomes were positive for one and negative or mixed for the other, were noted in 44% of studies reporting effects on adaptation or mitigation. Trade-offs were commonly due to mixed labor effects in nature-based food production [e.g., 66 , 73 , 75 , 101 ], with most of these studies also showing win-wins for income/revenue. Negative or mixed income effects were primarily linked to opportunity costs [ 88 ], equipment and labor costs [ 85 , 102 ], or crop specific profitability [ 78 ]. Seven reviews highlighted positive effects on adaptation, mitigation, and income or profitability, focusing on soil health [ 66 , 75 ], or above-ground diversification in nature-based food production [ 90 , 98 , 103 ].

Wider benefits.

Our supplementary analysis of the previous systematic review dataset on the outcomes of nature-based interventions for development in the Global South [ 54 ] shows a wide range of development outcomes of which most (87%) are positive, 4% are mixed and 5% negative (the other 4% being unclear or having no effect). Direct impacts on local economies are the most frequently reported outcome, followed by food security and then rights / empowerment / equality ( Fig 9 ).

https://doi.org/10.1371/journal.pclm.0000281.g009

Although conventional direct economic outcomes for jobs, incomes and revenues are reported in the aggregated category of ‘Local economies’, all development outcomes can have indirect economic impacts. For example, improving household food security or livelihoods, or improving access to urban green spaces, can also improve physical and mental health (e.g. [ 104 ]), leading to lower healthcare costs [ 105 , 106 ] and higher workforce productivity [ 107 ]. Similarly, benefits for climate change mitigation, adaptation and disaster risk reduction translate to lower economic costs of damage to infrastructure or crop production from storms, floods, droughts, or fires. For example, coral reefs offer coastal flood protection worth US$272 billion globally [ 108 ]. Economic benefits also arise when NbS reduce local conflicts and geopolitical instability through better management of natural resources. NbS can also encourage the empowerment of women, and their contribution to the formal economy, such as by starting new businesses (e.g. [ 109 ]). Finally, NbS can improve food and livelihood security and provide resilience to economic shocks when other sources of income are lost [ 54 ]. This is particularly important as calls for greater emphasis on resilience in economic policy grow stronger [ 110 ].

Interventions not tailored to the needs of different social groups led to trade-offs for employment and income. Inequitable benefit distribution was attributed to 1) different opportunity costs, 2) elite capture, 3) conflict over ecosystem service use or benefit-sharing, or 4) and sociocultural and governance inequities. For example, gender inequity was exacerbated by engrained gender hierarchies subjecting women to unpaid labor burdens (e.g., PES schemes [ 83 ], agroforestry [ 96 ], conservation agriculture [ 96 , 111 ], agroecological practices [ 92 ]), women having unequal access to land [ 59 , 69 ], support from agricultural and extension services [ 59 , 69 ], information, technology, or capital and markets [ 59 , 70 ], or limited decision-making power [ 70 ].

Opportunity costs from NbS differ among social groups due to varying reliance on natural resources, such as where community forest management negatively impacted the most forest-dependent people [ 72 , 79 ]. In some cases, interventions increased transaction costs for poorer, under-resourced households, such as where certification schemes and grazing bans pose risks of market concentration and benefit disparities, favoring wealthier stakeholders [ 80 , 81 , 93 ]. Market-oriented rangeland policies in China were criticized for undermining traditional pastoralism, disrupting the social-cultural fabric [ 81 , 93 ]. Social trade-offs also occurred due to conflicts in ecosystem service use, such as where forest protection creates spatial trade-offs affecting water distribution [ 67 , 70 ].

Elite capture in environmental interventions exacerbates inequality, noted in 12 reviews across various interventions (e.g., sustainability certifications, ecotourism, community-based natural management, protected areas). This disadvantages the poor and enhances disparities between participants and non-participants, especially in PES schemes [ 72 , 83 , 93 ]. Addressing these social trade-offs and mitigating inequalities requires targeted support for marginalized groups, such as helping them meet certification standards [ 112 ].

A few reviews noted social trade-offs in revenue sharing from ecotourism or community resource management between local communities and government agencies [ 63 , 64 ]. Discussions included the imbalance in green roof investments, where private costs do not align with public benefits, suggesting a role for government subsidies to reconcile these differences and enhance societal gains [ 71 ].

Case study: Protected areas in Peru

Our systematic review was enhanced by a case study on Peru’s protected area system (SINANPE), demonstrating how participatory governance leads to beneficial outcomes where NbS support local livelihoods ( Box 1 ). SINANPE and local communities enter into landscape use contracts which facilitate local jobs and income from eco-tourism and selling sustainably harvested products at higher prices. Additionally, selling carbon offset credits helps fund the restoration, upkeep, and surveillance of these areas, creating jobs such as park ranger positions. Eco-tourism further stimulates the local economy by increasing demand for additional services, like handicraft sales, boosting income and job opportunities.

Box 1. The job creation and income generation potential of Peru’s national system of protected areas

SINANPE, Peru’s national system of protected natural areas (PNAs), includes 76 areas supporting ecosystem services vital for local livelihoods. Participatory governance, sustainable resource use contracts, "Aliados por la Conservación" certification, and eco-tourism promote income generation and subsistence livelihoods. The certification connects local producers to green markets, providing opportunities for people in or near protected areas. These programs supported communities during the pandemic, facilitated by the state’s ability to leverage public, private, and international cooperation funds.

To boost climate change adaptation, the protected area system emphasizes ancestral knowledge and sustainable resource management. It promotes ecological resilience through preventative actions, control measures, and ecosystem restoration. SINANPE monitors climate change impacts on forest ecosystems, effectively reducing deforestation rates. National deforestation spiked to 203,272 Ha during the COVID-19 lockdowns in 2020 but decreased to 137,976 Ha in 2021, down from 148,426 Ha in 2019 [ 144 ].

Economic impact

Jobs: SINANPE employment grew by 35%, from 942 people in 2011 to 1,273 people in 2021 [ 145 ]. Park rangers accounted for 55% of the workforce in 2021, with 26% being women. A volunteer program trained and supported 3,750 community members in 2019 and 2,366 in 2020 with food and stipend [ 146 , 147 ]. Tourist activities created 36,741 local jobs [ 148 ].

Income : Sustainable use contracts helped 4,587 families (21,100 people) in 2020, rising to 6,334 families in 2021 [ 149 , 150 ]. They sell local products (e.g., vicuña fiber, chestnut and aguaje fruits), generating USD 1,332,293 income and USD 39,906 for SINANPE [ 149 ]. "Aliados por la Conservacion" certification benefits 1,788 families in 18 PNAs, selling diverse products in Lima and international markets. These value-added products from protected areas (e.g, aguaje beverage, chocolate and coffee products, handicrafts, textiles) are sold in Lima or in Europe and USA. Also, 388 eco-tourism contracts were renewed, benefiting 2,621 families [ 150 ].

Tourism revenue: Pre-pandemic, there were 2,736,650 visitors in 2019. Visitor numbers dropped to 722,593 in 2020 but increased to 1,422,335 in 2021 due to domestic tourism [ 149 ]. Entry ticket sales generated USD 6,839,250 in 2019, USD 2,408,424 in 2020, and 2,721,519 in 2021 [ 149 ]. In 2017 economic impact of tourism was approximately USD 723 million, with USD 165 million directly benefiting households and salaries, not considering multiplier effects [ 148 ].

Other benefits

Subsistence livelihoods: An additional 69 agreements for sustainable NTFP harvesting (bushmeat, aguaje fruits, various tree and shrubby species, non-viable taricaya eggs) were renewed, benefiting 829 families over 98,199 Ha in 15 PNAs [ 149 ].

Greenhouse gas mitigation : SINANPE has 3 REDD+ projects in 4 PNAs, covering 2 million Ha. These projects avoided deforestation of 95,000 Ha from 2008-2020, resulting in 36.6 million tCO2e of verified emissions reductions [ 145 ]. Over 33 million carbon credits were sold, certified by the Verified Carbon Standard and Climate, Community, and Biodiversity standards [ 145 ]. Carbon finance funded training, park ranger employment, equipment, education, and livelihood support for local communities.

To our knowledge, this is the first systematic review assessing the economic recovery potential of nature-based solutions across a wide range of intervention types and geographical contexts. Our goal was to provide a comprehensive overview to help integrate evidence on NbS into fiscal policy, particularly for addressing economic downturns.

We conducted a "review of reviews" to synthesise fragmented evidence from multiple interventions and diverse outcome measures, supplementing this with additional data from grey literature, primary studies, and a detailed country-level case study from Peru (see Box 1 ). Due to the variability in reported variables and review methodologies, a quantitative meta-analysis was not feasible, so results should be interpreted with caution. The distribution of evidence on economic impacts, pathways, and mediating factors varied according to the scope and focus of the underlying studies, and some recent evidence may not have been captured by existing reviews.

Despite these limitations, our approach offers valuable insights into the evidence base, allowing us to explore pathways and mediating factors in different intervention contexts. Here, we discuss the key findings, limitations of the review, gaps in the evidence, and opportunities for future research and synthesis.

Synopsis of key findings

Our mapping revealed evidence on a range of nature-based interventions but with significant gaps. We found 66 reviews reporting economic outcomes from these interventions, although few explicitly categorized them as NbS. The evidence was biased towards nature-based food production which accounted for 50% of cases, while only 19% covered ecosystem restoration and 15% focused on novel ecosystems, such as urban NbS. Geographically, most studies concentrated on sub-Saharan Africa (44% of studies), South Asia (35%), East Asia and the Pacific (30%), Latin America & the Caribbean (18%), with more limited coverage in North America, Europe and central Asia, and the Middle East and North Africa. This distribution contrasts with the evidence base on ecosystem services and their valuation, which is concentrated in higher income countries [ 113 , 114 ], as is evidence on NbS for climate change adaptation [ 1 ]. Some gaps may be due to our exclusion of non-English language studies, although some reviews included primary non-English literature, which helped capture additional evidence.

Most evidence on outcomes focused on income/revenue generation, predominantly at the household level, followed by changes in labor demand, including employment generation. Research on broader impacts on economic growth is limited, although available evidence indicates that nature-based interventions often deliver high gross value added and deliver returns per unit of investment that are comparable to or better than those from other sectors [ 10 , 29 , 37 ]. Overall, most reported effects were positive, indicating that investments in nature contribute to income generation and employment across various skill levels. A more nuanced picture emerged from reviews that critically appraise the underlying studies. These reviews report a significantly higher proportion of mixed effects (53%) and a lower proportion of positive effects (40%) compared to those that did not (18% and 70%, respectively). The mixed effects observed are attributed to variability in study results, differences between short-term and long-term gains, market conditions, regional effects, reliance on external subsidies, and discrepancies between expected and actual economic benefits. This variability aligns with the growing understanding that the effectiveness of NbS is mediated by a range of internal and external factors shaping the enabling environment. Among the few studies that compared the impact of investments in nature with alternative approaches, most found that NbS are more effective, particularly in terms of income/revenue generation.

Most reviews (76%) reported trade-offs or win-wins, especially trade-offs between biodiversity and livelihoods due to transaction or opportunity costs when interventions reduce agricultural output or limit natural resource use. However, these short-term opportunity costs can be managed through strategies such as securing price premiums, offering compensation or providing subsidies, which can ultimately benefit ecosystem health, biodiversity, and economic outcomes. Agro-diversification builds resilience, reducing economic risks associated with crop loss. We found positive associations with adaptation resulting from livelihood or crop diversification, which can boost profits through reduced costs, increasing outputs, or providing additional revenue sources such as non-timber forest products (NTFPs) [ 57 , 66 , 68 , 75 , 76 ]. Furthermore, positive associations with climate change mitigation were observed, mainly through nature-based food production practices that increased carbon sequestration (above or below ground) or reduced emissions, while simultaneously improving farming profitability and employment opportunities.

How do nature-based solutions deliver economic impact?

We identified several pathways by which NbS can impact income/revenue, revenue generation, and employment. Income/revenue arises from the sale of ecosystem goods or services, cost savings, subsidies or payments for ecosystem services. Direct effects on labor are linked to transitions to nature-based food production, green infrastructure implementation, and investments in ecotourism.

While evidence of indirect and induced job creation, and economic multiplier effects through business-to-business spending is limited, some studies found positive impacts for economic growth. However, they also highlight many mediating factors, including the type of ecosystem or restoration project (which affects the size of investment required), the causes and extent of ecosystem degradation, labor cost, government legislation (shaping regulatory requirements to invest in NbS), and regulatory standards (e.g., procurement rules or requirements to source local labor) [ 10 ]. For nature-based food production, mediating factors can reduce revenue, in turn affecting economic growth through reduced expenditure and investment in supply chains. These include low market prices, lack of market regulation, constraints in marketing channels or limited lobbying capacity, lack of access to credit, or elite capture [ 57 , 106 ].

The importance of mediating factors makes it difficult to predict whether a specific NbS intervention will lead to positive or negative economic outcomes, or if trade-offs or win-wins will occur with other objectives, emphasizing the context dependency of NbS outcomes. A pathway can result in win-wins in one context and trade-offs in another, depending on mediating factors like market access, input costs, the ability to attain price premiums, or adequacy of subsidies or PES to offset opportunity costs. Outcomes are shaped by technical factors relating to intervention design, implementation, and management, but also by other internal and external economic, financial, governance, institutional, social, cultural, and to a lesser extent, biological factors. This highlights the importance of the broader social, economic, and bio-physical character of NbS, corroborating the evidence on how NbS reduce vulnerability [ 2 ], or how Ecosystem-based Adaptation (EbA) is effective [ 53 ]. This also reinforces the notion that NbS are actions which support biodiversity and human well-being [ 35 ] through enhanced and harmonious human-nature relations [ 46 ].

Is labor demand a cost or a benefit?

This review shows that NbS are often more labor intensive than other potential investment options, thus providing significant potential for job creation. For NbS food production, however, effects on labor varied with the mode of implementation [ 75 ]. For example, intercropping, agroforestry, and organic agriculture are generally found to increase labor demand [ 115 ], but conservation agriculture can either increase or decrease it for different cultivation stages; crop residue retention reduces the need for pre-tilling, but reduced tillage potentially increases the need for weeding unless herbicides are used [ 96 ]. Although most reviews treated labor as a cost, scaling-up nature-based food production can translate into employment opportunities for low-income households [ 96 , 116 ]. These measures also provide job security through diversified income streams and reduced income volatility [ 76 ]. The perception of increased labor demand as either beneficial or negative depends largely on the economic context. From a fiscal policy perspective, job creation is prioritized during economic downturns and periods of high unemployment [ 117 ]. Governments typically view job creation positively because it helps reduce unemployment and can garner political support. In contrast, businesses may view increased labor demand negatively, as higher employment can lead to decreased profits if output per employee is reduced.

Promoting equity in economic impact

Social equity is a core dimension of sustainable development and foundational property of NbS [ 35 , 50 ]. How effects (and costs, benefits) disaggregate across social groups has important material implications for achieving human well-being, notably by mediating the overall effectiveness of NbS [ 2 , 118 ]. Positive impacts on jobs and incomes can mask trade-offs between social groups, highlighting the importance of considering equity, which remains under-reported in the literature [ 59 , 83 ]. We found that social inequity occurred when interventions were not tailored to the needs of different groups, including consideration of vulnerabilities embedded in the sociocultural and governance context. This aligns with the scholarship on NbS (notably EbA) which calls for exploring how benefits disaggregate across groups, how this affects vulnerability, and in turn, how interventions can more effectively support adaptation [ 2 , 119 , 120 ]. A range of mediating factors shaped distributional effects, notably elite capture, differential opportunity costs per group (due to different types of livelihoods and dependencies on nature) or inequities embedded in the sociocultural or governance context, such as gender hierarchies. Many reviews across a range of intervention types highlighted elite capture as a major issue, and a crucial barrier in achieving equity in economic impact. This is a cross-cutting issue in natural resource management and development, whereby the powerful co-opt finance and benefits, thereby reinforcing unequal power relations [ 120 ] and jeopardizing progress towards the SDGs. Although the impacts of NbS on social equity are highly variable and context-specific, the articles collectively underscore the need for NbS to include mechanisms specifically addressing the needs of marginalized group and ensuring equitable benefit distribution. Addressing this requires ensuring local communities and disadvantaged groups, including women, children, disabled, and minorities, actively participate in intervention design and implementation to avoid skewed distribution of benefits (ibid). For example, SINANPE in Peru (see Box 1 ) seeks to engage vulnerable groups (e.g., women, Indigenous communities) in training to strengthen local capacities, organization skills and empowerment in resource management and conservation. Moreover, SINANPE operates a volunteer program for local people that provides training and a small stipend to support forest monitoring activities, involving 2,366 local community members in 2020.

Wider economic outcomes

Our supplementary analysis of the dataset from [ 54 ] demonstrated that NbS, if carefully implemented, bring substantial societal and ecological benefits that support economic prosperity, including climate change adaptation [ 1 ], climate mitigation (e.g. [ 3 , 4 ]) and improved ecosystem health [ 121 ]. Well-governed NbS support food and water security, provide green space for recreation, help protect against floods, droughts and heatwaves, and support social empowerment, all of which improve community health, well-being and economic resilience [ 1 ]. This was also demonstrated by the case study of protected areas in Peru, where there was emphasis on supporting local livelihoods through agreements allowing sustainable NTFP harvesting for subsistence, along with capacity building through training. Because these public benefits have limited direct market value, and are difficult to quantify in monetary terms, it is crucial to consider plural market and non-market values to stimulate policies that are inclusive and respond to human well-being [ 114 ]. This will require new methods to account for the diverse values of nature [ 122 ]. Policy and project evaluations and appraisals should also look beyond short-term economic objectives, to ensure long-term resilience and avoid maladaptation [ 123 ]. Ultimately, this requires transitioning towards a new economic paradigm, where well-being is the core objective rather than GDP growth and capital accumulation [ 41 , 44 ]. Such a transition would focus on regenerative human-nature relations, and thus enable a shift to circular economies that sustain both human well-being and the biosphere [ 42 ].

Comparison with other studies and evidence gaps for future research

In this section we compare the findings of our academic review with evidence from wider academic and grey literature and consider evidence gaps and priorities for further research.

Temporal dimensions of job creation.

Although impacts on labor demand were commonly reported, we found a lack of evidence in the academic literature on the temporal dimensions of job creation (short-term vs long-term), despite growing evidence in the grey literature that NbS stimulates short-and long-term job creation [ 124 , 125 ] ( S4 Text ).

Skills, training needs and job quality.

The evidence in our review suggests that nature-based interventions can stimulate both low- and high-skilled jobs. This is supported by additional evidence from grey literature ( S4 Text ). For instance, In South Africa, establishing green infrastructure creates jobs that do not require specialized skills, allowing for easy entry into the labor market for low-skilled individuals [ 65 ]. On the other hand, technical extension and training programs build specialized skills and knowledge [ 59 ] and leverage local traditional knowledge [ 77 ] to scale NbS. However, there is still a gap in understanding job quality, despite the recommendation of the IUCN Global NbS Standard [ 50 ] to prioritize “decent work” in NbS as defined by the International Labor Organization [ 126 ]. These could build on the work of Vardon et al., 2023, who detail the role of natural capital accounting in driving greener recovery [ 127 ].

Economic impact at regional or national scales.

Our analysis corroborates evidence from large-scale investments in nature in the grey literature ( S4 Text ), demonstrating strong job creation and protection to sustain crucial ecosystem services. Most employment outcomes were reported as positive effects (except for studies at the farm-scale that framed labor as a cost). Two studies from our review demonstrate high potential for job creation at national scale, in developing country contexts: [ 87 ] estimate that the forest tourism industry in China has employed half a million farmers, reducing poverty across 4,654 villages, and [ 116 ] report that 16,000 rural people in Kyrgyzstan were directly employed in the walnut value chain. Similarly, our case study in Peru showed creation of over 36,000 eco-tourism jobs ( Box 1 ).

Direct impacts on growth and multipliers.

Although there is compelling evidence that NbS can stimulate growth across a wide array of industries (e.g., via gross value added, economic multiplier effects) [ 10 , 37 ] ( S4 Text ), this comes from relatively few studies. Most studies reported economic outcomes at the household or community level, reflecting a lack of mechanisms to track fiscal policy measures and government spending at broader scales, such as through national inventories [ 10 ], as well as general lack of systematic data collection and reporting on NbS implementation. This is challenging because NbS cut across traditional sectors (e.g., water, agriculture, infrastructure, environmental protection), implicating many public and private sector actors. There is no standard industrial classification, and public and private funding sources are diverse, making investment and outcome tracking difficult [ 37 , 119 ]. To scale up the evidence base, we need comprehensive accounting systems that track both public and private investments in NbS, enabling the integration of this data into economic models for estimating the broader economic impacts of NbS activities, including indirect and induced effects [ 10 ].

Under-represented ecosystems.

Although the available evidence shows that NbS in grassland, dryland, freshwater, coastal and marine ecosystems hold important potential for both job creation and income generation ( S4 Text ), we found a lack of evidence across these ecosystems, in contrast to forest ecosystems and working landscapes (43% and 72% of intervention cases, respectively). This aligns with known biases in the evidence base on NbS towards forest ecosystems [ 1 , 128 ]. This is concerning, given the critical role of these ecosystems in supporting livelihoods (grasslands – [ 129 , 130 ]; coastal ecosystems – [ 131 ]), climate change adaptation [ 1 , 2 , 108 , 132 ] and mitigation [ 133 , 134 ]. Understanding how NbS in these ecosystems can support economic impact, as well as biodiversity and climate benefits, is critical to increase ambition and guide their scaling-up.

Urban nature-based solutions.

Surprisingly, we found little evidence on the direct economic impact of investments in urban NbS, although evidence from the grey literature helps to bridge the gap (see S4 Text ). The extensive literature on urban green infrastructure focuses mainly on benefits for climate change adaptation [ 135 ], water treatment [ 136 ], and human health and well-being [ 137 , 138 ], sometimes with economic valuation of the indirect outcomes. However, the few reviews that we found report important benefits for employment and income generation [ 65 ] and increased profits through reduced energy expenditure [ 71 ], with both also noting the potential for increased tax revenues. With the global urban population set to double by 2050 [ 139 ], NbS could provide a significant source of jobs and income for urban residents, in addition to benefits for health, human well-being, and climate change adaptation.

Comparison with alternative interventions.

We found a lack of comparisons of economic outcomes of NbS investments versus alternatives, particularly outside the context of food production. Evidence is however growing, showing high economic multipliers for nature restoration compared to other sectors [ 37 ], with greater benefits for jobs and incomes than conventional alternatives across both high- and low-income countries [ 140 ]. Although natural capital investment policies have high potential economic multipliers [ 19 ], lack of comparisons makes it more challenging to mainstream NbS in fiscal policy [ 7 , 9 – 11 ]. Unless this evidence-base is expanded significantly, economic stimulus policy may continue to focus primarily on traditional investments such as road construction or fossil fuel energy, despite the increasing emphasis on building back better and green economic recoveries [ 140 ]. On a regional or national scale, poor data collection on the economic outcomes of NbS investments limits cross-sectoral comparisons on the effects of stimulus measures.

Trade-offs and win-wins.

Assessing trade-offs to optimize the design of NbS for equitable delivery of multiple benefits is crucial but challenging due to limited evidence. There were few holistic assessments covering multiple outcomes, except for the interactions between biodiversity and livelihoods, jobs, or income [ 59 , 72 , 83 ], and few studies considered temporal or spatial trade-offs. Better monitoring of outcomes across social, economic, ecological, and climate dimensions is crucial to capture the broader array of material and non-material benefits NbS can bring and manage potential trade-offs [ 1 ]. This includes disaggregated social assessments of costs and benefits, which is currently lacking [ 83 ]. Assessing NbS exclusively through a narrow lens, economic or other, can result in undervaluing NbS and thereby undermining human well-being [ 141 ].

Protocol for gathering evidence on economic outcomes.

To expand the evidence base, we recommend that researchers and economists work with practitioners to develop guidelines to scale robust assessments of the economic outcomes of NbS. For example, this could learn from the guidance on well-being impact evaluation for conservation interventions developed by de Lange et al. (2017) [ 142 ]. Guidance on the use of standardized economic indicators is needed, such as full time equivalent (FTE) job years per unit investment or per Ha of land, while recognizing that the wide range of NbS sectors, contexts and study aims will inevitably require diverse indicators. It is also important to go beyond direct effects and account for indirect and induced impacts on jobs and revenue. Additionally, there is a lack of studies with comparators (e.g., suitable baselines, or counterfactuals such as controls). Although controls can have shortcomings (e.g., where the control and intervention sites evolve in different ways between sampling periods), comparators are crucial to infer impact. Randomized control trials could be explored for investments in some intervention types, if spillovers between control and treatment groups can be minimized, control and treatment groups are truly comparable, and measured indicators are of significance to the individuals and communities that are impacted. There is also a need to better track the social distribution of costs and benefits, as well as potential displacement of negative social and environmental impacts over space (e.g., leakage or potential displacement of jobs or incomes in other sectors), and time (e.g., short-term job creation of tree planting vs long term impacts on biodiversity and ecosystem services under natural regeneration).

Conclusion and recommendations for policy makers

This systematic review demonstrates that NbS can significantly contribute to economic recovery by stimulating economic output and creating employment. NbS can generate direct jobs and incomes, offering a high return on investment compared to other sectors. This leads to cascading benefits throughout the economy. Well-designed and carefully implemented NbS can respond flexibly to economic shocks, by providing diverse employment opportunities across different skill levels and targeting underserved communities and disadvantaged groups. NbS can also diversify income sources and enhance resilience to future shocks. By combining traditional, local, and scientific knowledge, NbS can be both socially and ecologically effective with potential to support green sector growth and eco-innovation, aiding the transition to a clean and efficient circular economy.

NbS can support additional benefits beyond those included in conventional economic assessments. They can restore biodiversity, help to address climate change, reduce reliance on costly resources, improve human health, and enhance resilience. By preventing climate-related damage, lowering healthcare costs, and bolstering economic stability, NbS support prosperity and resilience—outcomes crucial for human well-being but often overlooked in GDP measurements. It is crucial however to carefully design for equitable delivery of multiple benefits to all stakeholders, prioritizing vulnerable groups. To minimize trade-offs, interventions should be co-designed with Indigenous people and local communities and prioritize livelihoods. Enhancing the evidence base and monitoring of economic outcomes is also crucial.

Governments and investors should consider societal benefits and long-term resilience when investing in NbS, extending beyond traditional economic measures, short-term impacts, and market-based mechanisms [ 143 ]. A holistic policy framework is essential to support well-designed NbS that deliver multiple benefits, manage trade-offs, explicitly support biodiversity, are led by Indigenous people and local communities, and are not treated as a substitute for fossil fuel phaseout [ 35 ]. This transition can contribute to sustainable circular economies that sustain human well-being and biodiverse ecosystems.

Recommendations for policymakers

Based on our review, we recommend that:

• NbS suited to the local context form a central component of national and regional investment programs for economic recovery, development and climate action, as they tackle multiple economic, environmental, and social problems.
• National monitoring and evaluation frameworks are created by governments to track impact of fiscal policy measures and government spending on NbS, and their economic outcomes.
• Economic assessments incorporate wider outcomes, beyond jobs, incomes, and revenues, gross value added and multipliers, to understand the full benefits and trade-offs of NbS compared to alternatives.
• NbS are led by or designed and implemented in partnership with local communities, farmers, businesses, and/or Indigenous groups, in accordance with the four NbS guidelines [ 51 ] and the detailed IUCN global standard [ 50 ], to ensure social and ecological effectiveness and delivery of equitable benefits.
• Government agencies are provided with adequate resources to support the implementation and design of high quality NbS, with or as part of sustainable livelihood-focused interventions, and to monitor environmental, social, and economic outcomes.
• Governments and businesses invest in education and training programs to develop skills for design, implementation, and maintenance of NbS projects, creating high quality jobs and boosting innovation.
• Funding is generated for researchers to work with practitioners, economic experts, and local communities, including Indigenous Peoples, to support robust assessment of the socio-economic outcomes of NbS interventions, ensuring attention to the correct use of counterfactuals and a comprehensive indicator set. Research is also needed to address evidence gaps on outcomes for job security, skills, and economic growth; for under-represented ecosystems (coastal, grassland, montane, mangroves, peatlands and urban); holistic assessments of synergies and trade-offs; and comparisons of NbS to alternative non-NbS interventions.

Supporting information

S1 checklist..

https://doi.org/10.1371/journal.pclm.0000281.s001

S1 Data. Systematic review dataset.

https://doi.org/10.1371/journal.pclm.0000281.s002

S1 Table. Included and excluded reviews.

https://doi.org/10.1371/journal.pclm.0000281.s003

S1 Text. Systematic review protocol and conceptual framework.

https://doi.org/10.1371/journal.pclm.0000281.s004

S2 Text. Coding framework and definitions.

https://doi.org/10.1371/journal.pclm.0000281.s005

S3 Text. Glossary of terms.

https://doi.org/10.1371/journal.pclm.0000281.s006

S4 Text. Supplementary results.

https://doi.org/10.1371/journal.pclm.0000281.s007

S5 Text. Supplementary tables.

https://doi.org/10.1371/journal.pclm.0000281.s008

Acknowledgments

We extend our sincere gratitude to Erin Gray and Swati Utkarshini for their valuable comments and revisions to the systematic review protocol. We also wish to thank the two anonymous reviewers and the associate editor for their thorough and insightful feedback, which greatly enhanced the quality of this manuscript. Lastly, we are grateful to Dan Seddon for his excellent work in producing the graphics.

• View Article
• PubMed/NCBI
• Google Scholar
• 18. Hasna Z. The grass is actually greener on the other side: Evidence on green multipliers from the united states. Cambridge, UK: Univ. Cambridge; 2021.
• 21. Council of Economic Advisers (U.S.). A retrospective assessment of clean energy investments in the Recovery Act. In: Executive Office of the President, editor. Washington (DC): U.S. Government Publishing Office; 2016.
• 22. Mundaca L, Damen B, editors. Assessing the effectiveness of the ‘Green Economic Stimulus’ in South Korea: Evidence from the energy sector. 38th International Association for Energy Economics (IAEE) International Conference; 2015: International Association for Energy Economics.
• 24. Tienhaara K. Green Keynesianism and the global financial crisis: Routledge; 2018.
• 27. UNEA. UNEP/EA.5/Res.5 Nature-based solutions for supporting sustainable development. Nairobi: UNEP; 2022.
• 30. Ankarfjard R, Subsol S. and Harvey Williams F. Nature-based Solutions: Key results and lessons learned from IFAD’s Adaptation for Smallholder Agriculture Programme. Rome: IFAD, 2021.
• 38. Lieuw-Kie-Song M, Perez-Cirera V. Nature Hires: How nature-based solutions can power a green jobs recovery. Gland, Switzerland: ILO/WWF, 2020.
• 41. Jackson T. Prosperity without growth: Foundations for the economy of tomorrow: Routledge; 2016.
• 47. Collaboration for Environmental Evidence. Guidelines and Standards for Evidence Synthesis in Environmental Management. Version 5.0. 2022 Oct [cited 2024 Apr 22]. Available from: https://environmentalevidence.org/wp-content/uploads/2022/10/CEE-Guidelines-Version-5.0-051022.pdf .
• 50. IUCN. Global Standard for Nature-based Solutions. A user-friendly framework for the verification, design and scaling up of NbS. First edition. Gland, Switzerland: 2020.
• 51. NBSI. Nature-based solutions to climate change. Key messages for decision makers in 2020 and beyond. NBSI; 2020.
• 52. World bank country and lending groups [Internet]. 2020 [cited 2022 Feb 2]. Available from: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519-world-bank-country-and-lending-groups .
• 54. Roe D, Turner B., Chausson A., Hemmerle E., Seddon N. Investing in nature for development: do nature-based interventions deliver local development outcomes? London: IIED, 2021.
• 114. IPBES. Summary for Policymakers of the Methodological Assessment of the Diverse Values and Valuation of Nature of the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services. IPBES secretariat Bonn, Germany; 2022.
• 122. Dicks J, Dellacio O, Stenning J. Economic costs and benefits of nature-based solutions to mitigate climate change. Cambridge, UK: Cambridge Econometrics & RSPB, 2020.
• 124. Mutafoglu K. tB P., Schweitzer J-P., Underwood E., Tucker G., Russi D., Howe M., et al. Natura 2000 and Jobs: Scoping Study. Brussels: IEEP, 2017.
• 125. Raes L, Mittempergher D., Piaggio M. and Siikamäki J. Nature-based Recovery can create jobs, deliver growth and provide value for nature. Gland, Switzerland: IUCN, 2021.
• 126. Payen J, Lieuw-Kie-Song M. Desk review study on Employment Impact Assessment (EmpIA) potential of Natural Resource Management (NRM) investments on employment creation. International Labor Organization (ILO) Working Paper 24, 2020.
• 139. World Bank. Urban Development Overview: World Bank; 2022 [November 4, 2022]. Available from: https://www.worldbank.org/en/topic/urbandevelopment/overview#1 .
• 144. Ministerio del Ambiente del Perú - MINAM. Bosque y pérdida de bosques. Geobosques, Programa Nacional de Conservación de Bosques para la Mitigación del Cambio Climático. 2022. Available from: https://geobosques.minam.gob.pe/geobosque/view/perdida.php
• 145. SERNANP. Informe de Logros SINANPE (Diciembre 2011 - octubre 2021). Lima: SERNANP; 2021.
• 146. SERNANP. Informe de Transferencia de Gestión. Julio 2020 - noviembre 2020. Lima: SERNANP; 2020.
• 147. SERNANP. Memoria Anual 2020. Lima: SERNANP; 2021.
• 149. SERNANP. Memoria Anual 2021. Lima: SERNANP; 2022.
• 150. SERNANP. Información del Seguimiento de los Programas Presupuestales 2021. Lima: SERNANP; 2022.