critical research appraisal assignment

• Mayo Clinic Libraries
• Systematic Reviews
• Critical Appraisal by Study Design

Systematic Reviews: Critical Appraisal by Study Design

• Knowledge Synthesis Comparison
• Knowledge Synthesis Decision Tree
• Standards & Reporting Results
• Materials in the Mayo Clinic Libraries
• Training Resources
• Review Teams
• Develop & Refine Your Research Question
• Develop a Timeline
• Project Management
• Communication
• PRISMA-P Checklist
• Eligibility Criteria
• Register your Protocol
• Other Resources
• Other Screening Tools
• Grey Literature Searching
• Citation Searching
• Data Extraction Tools
• Minimize Bias
• Synthesis & Meta-Analysis
• Publishing your Systematic Review

Tools for Critical Appraisal of Studies

“The purpose of critical appraisal is to determine the scientific merit of a research report and its applicability to clinical decision making.” 1 Conducting a critical appraisal of a study is imperative to any well executed evidence review, but the process can be time consuming and difficult. 2 The critical appraisal process requires “a methodological approach coupled with the right tools and skills to match these methods is essential for finding meaningful results.” 3 In short, it is a method of differentiating good research from bad research.

Critical Appraisal by Study Design (featured tools)

• Non-RCTs or Observational Studies
• Diagnostic Accuracy
• Animal Studies
• Qualitative Research
• Tool Repository
• AMSTAR 2 The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to “identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions.” 4 more... less... AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• ROBIS ROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. “The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review.” 5 more... less... ROBIS (Risk of Bias in Systematic Reviews)
• BMJ Framework for Assessing Systematic Reviews This framework provides a checklist that is used to evaluate the quality of a systematic review.
• CASP Checklist for Systematic Reviews This CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering: 1. Are the results of the study valid? 2. What are the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Systematic Reviews Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• JBI Critical Appraisal Tools, Checklist for Systematic Reviews JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• NHLBI Study Quality Assessment of Systematic Reviews and Meta-Analyses The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• RoB 2 RoB 2 “provides a framework for assessing the risk of bias in a single estimate of an intervention effect reported from a randomized trial,” rather than the entire trial. 6 more... less... RoB 2 (revised tool to assess Risk of Bias in randomized trials)
• CASP Randomised Controlled Trials Checklist This CASP checklist considers various aspects of an RCT that require critical appraisal: 1. Is the basic study design valid for a randomized controlled trial? 2. Was the study methodologically sound? 3. What are the results? 4. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CONSORT Statement The CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. “Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report.” 7 more... less... CONSORT (Consolidated Standards of Reporting Trials)
• NHLBI Study Quality Assessment of Controlled Intervention Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• JBI Critical Appraisal Tools Checklist for Randomized Controlled Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• ROBINS-I ROBINS-I is a “tool for evaluating risk of bias in estimates of the comparative effectiveness… of interventions from studies that did not use randomization to allocate units… to comparison groups.” 8 more... less... ROBINS-I (Risk Of Bias in Non-randomized Studies – of Interventions)
• NOS This tool is used primarily to evaluate and appraise case-control or cohort studies. more... less... NOS (Newcastle-Ottawa Scale)
• AXIS Cross-sectional studies are frequently used as an evidence base for diagnostic testing, risk factors for disease, and prevalence studies. “The AXIS tool focuses mainly on the presented [study] methods and results.” 9 more... less... AXIS (Appraisal tool for Cross-Sectional Studies)
• NHLBI Study Quality Assessment Tools for Non-Randomized Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. • Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies • Quality Assessment of Case-Control Studies • Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group • Quality Assessment Tool for Case Series Studies more... less... NHLBI (National Heart, Lung, and Blood Institute)
• Case Series Studies Quality Appraisal Checklist Developed by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess “the robustness of the evidence of uncontrolled, [case series] studies.” 10
• Methodological Quality and Synthesis of Case Series and Case Reports In this paper, Dr. Murad and colleagues “present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.” 11
• MINORS The MINORS instrument contains 12 items and was developed for evaluating the quality of observational or non-randomized studies. 12 This tool may be of particular interest to researchers who would like to critically appraise surgical studies. more... less... MINORS (Methodological Index for Non-Randomized Studies)
• JBI Critical Appraisal Tools for Non-Randomized Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis. • Checklist for Analytical Cross Sectional Studies • Checklist for Case Control Studies • Checklist for Case Reports • Checklist for Case Series • Checklist for Cohort Studies
• QUADAS-2 The QUADAS-2 tool “is designed to assess the quality of primary diagnostic accuracy studies… [it] consists of 4 key domains that discuss patient selection, index test, reference standard, and flow of patients through the study and timing of the index tests and reference standard.” 13 more... less... QUADAS-2 (a revised tool for the Quality Assessment of Diagnostic Accuracy Studies)
• JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy Studies JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• STARD 2015 The authors of the standards note that “[e]ssential elements of [diagnostic accuracy] study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible.”10 The Standards for the Reporting of Diagnostic Accuracy Studies was developed “to help… improve completeness and transparency in reporting of diagnostic accuracy studies.” 14 more... less... STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies)
• CASP Diagnostic Study Checklist This CASP checklist considers various aspects of diagnostic test studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Diagnostic Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• SYRCLE’s RoB “[I]mplementation of [SYRCLE’s RoB tool] will facilitate and improve critical appraisal of evidence from animal studies. This may… enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies.” 15 more... less... SYRCLE’s RoB (SYstematic Review Center for Laboratory animal Experimentation’s Risk of Bias)
• ARRIVE 2.0 “The [ARRIVE 2.0] guidelines are a checklist of information to include in a manuscript to ensure that publications [on in vivo animal studies] contain enough information to add to the knowledge base.” 16 more... less... ARRIVE 2.0 (Animal Research: Reporting of In Vivo Experiments)
• Critical Appraisal of Studies Using Laboratory Animal Models This article provides “an approach to critically appraising papers based on the results of laboratory animal experiments,” and discusses various “bias domains” in the literature that critical appraisal can identify. 17
• CEBM Critical Appraisal of Qualitative Studies Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• CASP Qualitative Studies Checklist This CASP checklist considers various aspects of qualitative research studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• Quality Assessment and Risk of Bias Tool Repository Created by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
• Latitudes Network A library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.

References & Recommended Reading

1.     Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews .  British Journal of Pharmacology ,  181 (1), 180-210

2.    Portney LG.  Foundations of clinical research : applications to evidence-based practice.  Fourth edition. ed. Philadelphia: F A Davis; 2020.

3.     Fowkes FG, Fulton PM.  Critical appraisal of published research: introductory guidelines.   BMJ (Clinical research ed).  1991;302(6785):1136-1140.

4.     Singh S.  Critical appraisal skills programme.   Journal of Pharmacology and Pharmacotherapeutics.  2013;4(1):76-77.

5.     Shea BJ, Reeves BC, Wells G, et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.   BMJ (Clinical research ed).  2017;358:j4008.

6.     Whiting P, Savovic J, Higgins JPT, et al.  ROBIS: A new tool to assess risk of bias in systematic reviews was developed.   Journal of clinical epidemiology.  2016;69:225-234.

7.     Sterne JAC, Savovic J, Page MJ, et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials.  BMJ (Clinical research ed).  2019;366:l4898.

8.     Moher D, Hopewell S, Schulz KF, et al.  CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.  Journal of clinical epidemiology.  2010;63(8):e1-37.

9.     Sterne JA, Hernan MA, Reeves BC, et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.  BMJ (Clinical research ed).  2016;355:i4919.

10.     Downes MJ, Brennan ML, Williams HC, Dean RS.  Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS).   BMJ open.  2016;6(12):e011458.

11.   Guo B, Moga C, Harstall C, Schopflocher D.  A principal component analysis is conducted for a case series quality appraisal checklist.   Journal of clinical epidemiology.  2016;69:199-207.e192.

12.   Murad MH, Sultan S, Haffar S, Bazerbachi F.  Methodological quality and synthesis of case series and case reports.  BMJ evidence-based medicine.  2018;23(2):60-63.

13.   Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J.  Methodological index for non-randomized studies (MINORS): development and validation of a new instrument.   ANZ journal of surgery.  2003;73(9):712-716.

14.   Whiting PF, Rutjes AWS, Westwood ME, et al.  QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.   Annals of internal medicine.  2011;155(8):529-536.

15.   Bossuyt PM, Reitsma JB, Bruns DE, et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.   BMJ (Clinical research ed).  2015;351:h5527.

16.   Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW.  SYRCLE's risk of bias tool for animal studies.   BMC medical research methodology.  2014;14:43.

17.   Percie du Sert N, Ahluwalia A, Alam S, et al.  Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.  PLoS biology.  2020;18(7):e3000411.

18.   O'Connor AM, Sargeant JM.  Critical appraisal of studies using laboratory animal models.   ILAR journal.  2014;55(3):405-417.

• << Previous: Minimize Bias
• Next: GRADE >>
• Last Updated: May 10, 2024 7:59 AM
• URL: https://libraryguides.mayo.edu/systematicreviewprocess

How To Write a Critical Appraisal

A critical appraisal is an academic approach that refers to the systematic identification of strengths and weakness of a research article with the intent of evaluating the usefulness and validity of the work’s research findings. As with all essays, you need to be clear, concise, and logical in your presentation of arguments, analysis, and evaluation. However, in a critical appraisal there are some specific sections which need to be considered which will form the main basis of your work.

Structure of a Critical Appraisal

Introduction.

Your introduction should introduce the work to be appraised, and how you intend to proceed. In other words, you set out how you will be assessing the article and the criteria you will use. Focusing your introduction on these areas will ensure that your readers understand your purpose and are interested to read on. It needs to be clear that you are undertaking a scientific and literary dissection and examination of the indicated work to assess its validity and credibility, expressed in an interesting and motivational way.

Body of the Work

The body of the work should be separated into clear paragraphs that cover each section of the work and sub-sections for each point that is being covered. In all paragraphs your perspectives should be backed up with hard evidence from credible sources (fully cited and referenced at the end), and not be expressed as an opinion or your own personal point of view. Remember this is a critical appraisal and not a presentation of negative parts of the work.

When appraising the introduction of the article, you should ask yourself whether the article answers the main question it poses. Alongside this look at the date of publication, generally you want works to be within the past 5 years, unless they are seminal works which have strongly influenced subsequent developments in the field. Identify whether the journal in which the article was published is peer reviewed and importantly whether a hypothesis has been presented. Be objective, concise, and coherent in your presentation of this information.

Once you have appraised the introduction you can move onto the methods (or the body of the text if the work is not of a scientific or experimental nature). To effectively appraise the methods, you need to examine whether the approaches used to draw conclusions (i.e., the methodology) is appropriate for the research question, or overall topic. If not, indicate why not, in your appraisal, with evidence to back up your reasoning. Examine the sample population (if there is one), or the data gathered and evaluate whether it is appropriate, sufficient, and viable, before considering the data collection methods and survey instruments used. Are they fit for purpose? Do they meet the needs of the paper? Again, your arguments should be backed up by strong, viable sources that have credible foundations and origins.

One of the most significant areas of appraisal is the results and conclusions presented by the authors of the work. In the case of the results, you need to identify whether there are facts and figures presented to confirm findings, assess whether any statistical tests used are viable, reliable, and appropriate to the work conducted. In addition, whether they have been clearly explained and introduced during the work. In regard to the results presented by the authors you need to present evidence that they have been unbiased and objective, and if not, present evidence of how they have been biased. In this section you should also dissect the results and identify whether any statistical significance reported is accurate and whether the results presented and discussed align with any tables or figures presented.

The final element of the body text is the appraisal of the discussion and conclusion sections. In this case there is a need to identify whether the authors have drawn realistic conclusions from their available data, whether they have identified any clear limitations to their work and whether the conclusions they have drawn are the same as those you would have done had you been presented with the findings.

The conclusion of the appraisal should not introduce any new information but should be a concise summing up of the key points identified in the body text. The conclusion should be a condensation (or precis) of all that you have already written. The aim is bringing together the whole paper and state an opinion (based on evaluated evidence) of how valid and reliable the paper being appraised can be considered to be in the subject area. In all cases, you should reference and cite all sources used. To help you achieve a first class critical appraisal we have put together some key phrases that can help lift you work above that of others.

Key Phrases for a Critical Appraisal

• Whilst the title might suggest
• The focus of the work appears to be…
• The author challenges the notion that…
• The author makes the claim that…
• The article makes a strong contribution through…
• The approach provides the opportunity to…
• The authors consider…
• The argument is not entirely convincing because…
• However, whilst it can be agreed that… it should also be noted that…
• Several crucial questions are left unanswered…
• It would have been more appropriate to have stated that…
• This framework extends and increases…
• The authors correctly conclude that…
• The authors efforts can be considered as…
• Less convincing is the generalisation that…
• This appears to mislead readers indicating that…
• This research proves to be timely and particularly significant in the light of…

You may also like

• Critical Appraisal of Studies

Critical Appraisal

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context (Burls, 2009). Critical appraisal of studies involves checking the quality, reliability and relevance of the studies you've selected to help answer your review question. Depending on the type of study you are evaluating you may use different evaluation tools. When evaluating studies, some questions to consider are:

• ​​​​Has the study's aim been clearly stated?
• Does the sample accurately reflect the population?
• Has the sampling method and size been described and justified?
• Have exclusions been stated?
• Is the control group easily identified?
• Is the loss to follow-up detailed?
• Can the results be replicated?
• Are there confounding factors?
• Are the conclusions logical?
• Can the results be extrapolated to other populations?

Adapted from:  University of Illinois, Chicago Library

More on critical appraisal:

• Critical Appraisal This article helps to define critical appraisal, identify its benefits, discuss conceptual issues influencing the adequacy of a critical appraisal, and detail procedures to help reviewers undertake critical appraisals (Tod, et al., 2021).
• Critical Appraisal Tools and Reporting Guidelines for Evidence‐Based Practice The primary purpose of this paper is to help nurses understand the difference between critical appraisal tools and reporting guidelines (Buccheri et al., 2017).
• What is Critical Appraisal? An overview of how to critically appraise studies from Amanda Burls, Director of the Critical Appraisal Skills Programme, University of Oxford (Burls, 2009). From https://whatisseries.co.uk/product/what-is-critical-appraisal/
• Critical Appraisal Tools
• AMSTAR 2 - A Critical Appraisal Tool for Systematic Reviews (Shea et al., 2017).
• Best Bests Critical Appraisal Worksheets Critical appraisal checklists for various study types.
• CASP Checklists Critical Assessmentl Skills Programme (CASP) has appraisal checklists designed for use with Randomized Controlled Trials and other study types.
• Critical Appraisal Tools Critical appraisal questions to ask and worksheets from the Centre for Evidence-Based Medicine at Oxford University.
• Downs & Black- Checklist for Measuring Study Quality See Appendix (Downs & Black, 1998).
• Downs & Black Checklist for Clinical Trial Quality Assessment (Downs and Black Checklist, 2013)
• Joanna Briggs Institute (JBI) Critical Appraisal Tools JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers.
• Johns Hopkins Evidence-Based Practice Model The Johns Hopkins Evidence-Based Practice model for Nurses and Healthcare Professionals is a powerful problem-solving approach to clinical decision-making and is accompanied by user-friendly tools to guide individuals or groups through the EBP process. Must fill out online form to request permission to download tools.
• Mixed Methods Appraisal Tool (MMAT) The MMAT is intended to be used as a checklist for concomitantly appraising and/or describing studies included in systematic mixed studies reviews (reviews including original qualitative, quantitative and mixed methods studies) (Hong et al., 2018).
• Repository of Quality Assessment and Risk of Bias Tools A handy resource from Duke University' Medical Center Library for finding and selecting a risk of bias or quality assessment tool for evidence synthesis projects. Download spreadsheet for full functionality.

Tools for Specific Study Types

Integrative Reviews

• Critical Appraisal Skills Programme (CASP) Checklists  Appraisal checklists designed for use with Systematic Reviews, Randomized Controlled Trials, Cohort Studies, Case Control Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.
• Mixed Methods Appraisal Tool (MMAT)  The MMAT is a critical appraisal tool that is designed for the appraisal stage of systematic mixed studies reviews, i.e., reviews that include qualitative, quantitative and mixed methods studies. It permits to appraise the methodological quality of five categories to studies: qualitative research, randomized controlled trials, non-randomized studies, quantitative descriptive studies, and mixed methods studies. (Hong et al., 2018).

Randomized Controlled Trials

• CASP checklist for RCT 
• CASP Checklists  Critical Assessment Skills Programme (CASP) has appraisal checklists designed for use with Systematic Reviews, Randomized Controlled Trials, Cohort Studies, Case Contro l  Studies, Economic Evaluations, Diagnostic Studies, Qualitative studies and Clinical Prediction Rule.
• JBI Critical Appraisal Tools  Joanna Briggs Institute (JBI) is an independent, international, not-for-profit researching and development organization based at the University of Adelaide, South Australia. Contains a number of critical appraisal tools including Checklist for Randomized Controlled Trials
• RoB 2.0   A revised Cochrane risk of bias tool for randomized trials. Is suitable for individually-randomized, parallel-group, and cluster- randomized trials

Qualitative Studies

• CASP Qualitative Studies Checklist  Most frequently recommended tool for qualitative study assessment
• JBI Critical Appraisal Checklist for Qualitative Research

Systematic Reviews

• AMSTAR 2 - A Critical Appraisal Tool for Systematic Reviews  (Shea et al., 2017)
• BMJ Framework for Assessing Systematic Reviews
• CASP Systematic Review Checklist
• JBI Checklist for Systematic Reviews
• ROBIS A new tool for assessing the risk of bias in systematic reviews (rather than in primary studies). Here you can find the tool itself, information to help you complete a ROBIS assessment, and resources to help you present the results of your ROBIS assessment.

Scoping and Other Review Types

• CAT HPPR: A critical appraisal tool to assess the quality of systematic, rapid, and scoping reviews investigating interventions in health promotion and prevention  (Heise et al., 2022).
• CAT HPPR Critical Appraisal Tool for Health Promotion and Prevention Reviews
• CAT HPPR Manual and Instructions Manual and instructions to reviewers for using the Critical Appraisal Tool for Health Promotion and Prevention Reviews (CAT HPPR). 2020.

References:

Buccheri, R. K., & Sharifi, C. (2017).  Critical appraisal tools and reporting guidelines for evidence‐basedpPractice.   Worldviews on Evidence-Based Nursing, 14( 6), 463–472. https://doi.org/10.1111/wvn.12258

Burls, A. (2009).  What is critical appraisal?  Retrieved April 21, 2022, from www.whatisseries.co.uk

Downs, S. H., & Black, N. (1998). The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions.  Journal of Epidemiology and Community Health (1979) ,  52 (6), 377–384. https://doi.org/10.1136/jech.52.6.37 7

Downs and Black Checklist for Clinical Trial Quality Assessment.(2013). In Point-of-Care Testing of International Normalized Ratio for Patients on Oral Anticoagulant Therapy – Project Protocol [Internet] . Canadian Agency for Drugs and Technologies in Health. https://www.ncbi.nlm.nih.gov/books/NBK361373/

Heise, T. L., Seidler, A., Girbig, M., Freiberg, A., Alayli, A., Fischer, M., Haß, W., & Zeeb, H. (2022). CAT HPPR: A critical appraisal tool to assess the quality of systematic, rapid, and scoping reviews investigating interventions in health promotion and prevention.  BMC Medical Research Methodology ,  22 (1), 334–334. https://doi.org/10.1186/s12874-022-01821-4

Hong, Q.N., Fàbregues, S., Bartlett, G., Boardman, F.K., Cargo, M., Dagenais, P., Gagnon, M., Griffiths, F.E., Nicolau, B., O’Cathain, A., Rousseau, M.C., Vedel, I., & Pluye, P. (2018). The Mixed Methods Appraisal Tool (MMAT) version 2018 for information professionals and researchers. Education for Information, 34 (4), 285-291.DOI 10.3233/EFI-180221

Ma, Wang, Y., Yang, Z., Huang, D., Weng, H., & Zeng, X. (2020). Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: what are they and which is better?  Military Medical Research,  7 (1), 7. 

Motheral, B., Brooks, J., Clark, M. A., Crown, W. H., Davey, P., Hutchins, D., Martin, B. C., & Stang, P. (2003). A checklist for retrospective database studies—Report of the ISPOR task force on retrospective databases.  Value in Health ,  6 (2), 90–97. https://doi.org/10.1046/j.1524-4733.2003.00242.x

Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., & Henry, D. A. (2017). AMSTAR 2: A critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.  BMJ (Clinical research ed.) ,  358 , j4008. https://doi.org/10.1136/bmj.j4008

Tod, D., Booth, A., & Smith, B. (2021). Critical appraisal.  International Review of Sport and Exercise Psychology , 15 (1), 52-72.

• << Previous: Quantitative vs. Qualitative Research
• Next: SR Programs & Tools >>
• Types of Questions
• Key Features and Limitations
• Is a Systematic Review Right for Your Research?
• Integrative Review
• Scoping Review
• Rapid Review
• Meta-Analysis/Meta-Synthesis
• Selecting a Review Type
• Reducing Bias
• Guidelines for Student Researchers
• Training Resources
• Register Your Protocol
• Handbooks & Manuals
• Reporting Guidelines
• PRESS 2015 Guidelines
• Search Strategies
• Selected Databases
• Grey Literature
• Handsearching
• Citation Searching
• Study Types & Terminology
• Quantitative vs. Qualitative Research
• Broad Functionality Programs & Tools
• Search Strategy Tools
• Deduplication Tools
• CItation Screening
• Quality Assessment/Risk of Bias Tools
• Data Collection/Extraction
• Meta Analysis Tools
• Books on Systematic Reviews
• Finding Systematic Review Articles in the Databases
• Systematic Review Journals
• More Resources
• Evidence-Based Practice Research in Nursing
• Citation Management Programs
• Last Updated: May 3, 2024 5:52 PM
• URL: https://libguides.adelphi.edu/Systematic_Reviews

• Northeastern University Library
• Research Subject Guides
• Guides for Library Services
• Systematic Reviews and Evidence Syntheses
• Critical Appraisal
• Evidence Synthesis Service
• Types of Systematic Reviews in the Health Sciences
• Beginning Your Project
• Standards & Guidance
• Evidence-Based Assignments
• Tips for a Successful Review Team
• Training and Tutorials

Systematic Reviews and Evidence Syntheses : Critical Appraisal

Introduction.

“Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context,” ( Burls, 2009 ). -- Amanda Burls, Director of Postgraduate Programmes in Evidence-Based Health Care, University of Oxford

Critical appraisal, or risk of bias assessment, is an integral part of the systematic review methodology.

Bias can be introduced at any point in the research process, from study design to publication, and as such, there are many different forms of bias. (For descriptions and examples of forms of bias, see the University of Oxford’s Biases Archive ). Bias or systemic error can lead to inaccurate or incomplete conclusions. Therefore it is imperative to assess possible sources of bias in the research included in your review.

Hundreds of critical appraisal tools (CATs) have been developed to help you do so. Rather than providing a comprehensive list, this page provides a short list of CATs recommended by expert review groups and health technology assessment organizations. The list is organized by study design.

Defining terms

Critical appraisal includes the assessment of several related features: risk of bias, quality of reporting, precision, and external validity .

Critical appraisal has always been a defining feature of the systematic review methodology. However, early critical appraisal tools were structured as 'scales' which rolled many of the previously-mentioned features into one combined score. More recently, consensus has emerged within the health sciences that, in the case of systematic reviews of interventions, critical appraisal should focus on risk of bias alone. Cochrane's risk of bias tools, RoB-2 and ROBINS-I, were developed for this purpose.

Due to the evolution of critical appraisal within the systematic review methodology, you may hear folks use the terms "critical appraisal" and "risk of bias" interchangeably. It is useful to recall the differences between these terms and other related terms.

Critical appraisal (also called: critical assessment or quality assessment) includes the assessment of several related features: risk of bias, quality of reporting, precision, and external validity.

Risk of bias is equivalent to internal validity.

Internal validity can be defined as "the extent to which the observed results represent the truth in the population we are studying and, thus, are not due to methodological errors," ( Patino & Ferreira, 2018 ).

Quality of reporting refers to how accurately and thoroughly the study's methodology was reported.

Precision refers to random error. "Precision depends on the number of participants and (for dichotomous outcomes) the number of events in a study, and is reflected in the confidence interval around the intervention effect estimate from each study," ( Cochrane Handbook ).

External validity refers to generalizability; "the extent to which the results of a study can be generalized to other populations and settings," ( Cochrane Handbook ).

Critical Appraisal Tools

Case control.

• Creator: Critical Appraisal Skills Programme (CASP)
• Creator: Joanna Briggs Institute

Clinical Prediction Rule

Cross-sectional.

• Creator: Downes, Brennan, et al.
• Creator: Wirsching, et al.
• Creator: National Collaborating Centre for Environmental Health
• Creator: Oxford University’s Centre for Evidence-Based Medicine

Economic Evaluations

Mixed methods.

• Creator: National Collaborating Centre for Methods and Tools, McMaster University

Other Quantitative

• Creator: Cochrane Methods

Qualitative

Randomized controlled trials, systematic reviews.

• Creator: Bruyere Research Institute, Ottowa Hospital Research Institute, et al.

Reviews of Critical Appraisal Tools

This section lists articles which have reviewed or inventoried CATs. These articles can serve more comprehensive catalogs of previously developed CATs.

Buccheri RK, Sharifi C. Critical Appraisal Tools and Reporting Guidelines for Evidence-Based Practice. Worldviews Evid Based Nurs. 2017;14(6):463-472. doi: 10.1111/wvn.12258

Munthe-Kaas HM, Glenton C, Booth A, Noyes J, Lewin S. Systematic mapping of existing tools to appraise methodological strengths and limitations of qualitative research: first stage in the development of the CAMELOT tool. BMC Med Res Methodol. 2019;19(1):113. doi: 10.1186/s12874-019-0728-6

• << Previous: Tools
• Next: Evidence-Based Assignments >>
• Ask a Librarian
• Last Updated: May 16, 2024 7:56 AM
• URL: https://subjectguides.lib.neu.edu/systematicreview

Occupational Therapy and Rehabilitation Sciences

• Defining the Research Question(s)
• Reference Resources
• Evidence Summaries & Clinical Guidelines
• Health Data & Statistics
• Patient & Consumer Facing Materials
• Images/Streaming Video
• Database Tutorials
• Crafting a Search
• Narrowing / Filtering a Search
• Expanding a Search
• Cited Reference Searching
• Find Grey Literature
• Save Your Searches
• Cite Sources
• Critical Appraisal
• Different Types of Literature Reviews
• Conducting & Reporting Systematic Reviews
• Finding Systematic Reviews
• Tutorials & Tools for Literature Reviews
• Mobile Apps for Health

PRISMA  or Preferred Reporting Items for Systematic Reviews and Meta-Analyses is an evidence-based protocol for reporting information in systematic reviews and meta-analyses.

• The PRISMA STATEMENT , a 27-item checklist and a four-phase flow diagram to help authors improve the reporting of systematic reviews and meta-analyses.
• PRISMA also offers editable templates for the flow diagram as PDF and Word documents 

Appraising the Evidence: Getting Started

To appraise the quality of evidence, it is essential understand the nature of the evidence source. Begin the appraisal process by considering these general characteristics:

• Is the source primary, secondary or tertiary? (See University of Minnesota Library -  Primary, Secondary, and Tertiary Sources in the Health Sciences )
• If the source is a journal article, what kind of article is it? (A report of original research? A review article? An opinion or commentary?)
• If the source is reporting original research, what was the purpose of the research?
• What is the date of publication?
• Would the evidence presented in the source still be applicable today? (Consider: has technology changed? Have recommended best clinical practices changed? Has consensus understanding of a disease, condition, or treatment changed?)

Authority/Accuracy

• Who is the author? What are the author's credentials and qualifications and to write on the topic?
• Was the source published by a credible entity? (a scholarly journal? a popular periodical, e.g, newspaper or magazine?  an association? an organization?)
• Did the source go through a peer review or editorial process before being published? (See this section of the guide for more information about locating peer reviewed articles)

Determining Study Methodology

Understanding how a study was conducted (the methodology) is fundamental for determining the level of evidence that was generated by the study, as well as assessing the quality of the evidence it generated.  While some papers state explicitly in the title what kind of method was used, it is often not so straightforward.  When looking at report of a study, there are a few techniques you can use to help classify the study design.

1. Notice Metadata in Database Records

In some bibliographic databases, there is information found in the Subject field, or the Publication Type field of the record that can provide information about a study's methodology.  Try to locate the record for the article of interest in CINAHL, PubMed or PsycINFO and look for information describing the study (e.g., is it tagged as a "randomized controlled trial,"  a "case report," and "observational study", a "review" article, etc).

• A word of caution : A  "review" article is not necessarily a "systematic review."  Even if the title or abstract says "systematic review," carefully evaluate what type of review it is (a systematic review of interventions? a mixed methods SR? a scoping review? a narrative review?).

2. Read the Methods Section

While there may be some information in the abstract that indicates a study's design, it is often necessary to read the full methods section in order to truly understand how the study was conducted.  For help understanding the major types of research methodologies within the health sciences, see:

• Understanding Research Study Designs  (University of Minnesota Library)
• Study Designs  (Centre for Evidence Based Medicine)
• Jeremey Howick's  Introduction to Study Designs  (Flow Chart) [PDF]
• Quantitative Study Designs  (Deakin University Library)
• Grimes, D. A., & Schulz, K. F. (2002). An overview of clinical research: the lay of the land .  Lancet (London, England) ,  359 (9300), 57–61. https://doi.org/10.1016/S0140-6736(02)07283-5
• Deconstructing the Research Article (May/Jun2022; 42(3): 138-140)
• Background, Significance, and Literature Review (Jul-Aug2022; 42(4): 203-205)
• Purpose Statement, Research Questions, and Hypotheses (Sep/Oct2022; 42(5): 249-257)
• Quantitative Research Designs (Nov/Dec2022; 42(6): 303-311)
• Qualitative Research Designs (Jan/Feb2023; 43(1): 41-45)
• Non-Experimental Research Designs (Mar/Apr2023; 43(2): 99-102)

Once the study methodology is understood, a tool or checklist can be selected to appraise the quality of the evidence that was generated by that study.  

Critical Appraisal Resources

In order to select a tool for critical appraisal (also known as quality assessment or "risk of bias" assessment), it is necessary to understand what methodology was used in the study.  (For help understanding study design, see this section of the guide .)

The list below sets of contains critical appraisal tools and checklists, with information about what types of studies those tools are meant for.  Additionally, there are links to reporting guidelines for different types of students, which can also be useful for quality assessment.  

If you're new to critical appraisal, check out this helpful video overview of some of the common tools:

Checklists & Tools

The AGREE II an instrument is valid and reliable tool that can be applied to any practice guideline in any disease area and can be used by health care providers, guideline developers, researchers, decision/policy makers, and educators.

For help using the AGREE II instrument, see the AGREE II Training Tools

• AMSTAR 2 AMSTAR 2 is the revised version of the popular AMSTAR tool (a tool for critically appraising systematic reviews of RCTs). AMSTAR 2 can be used to critically appraise systematic reviews that include randomized or non-randomized studies of healthcare interventions, or both.

A collection of checklists for a number of purposes related to EBM, including finding, interpreting, and evaluating research evidence.

Found in Appendix 1 of Greenhalgh, Trisha. (2010). How to Read a Paper : The Basics of Evidence Based Medicine, 4th edition .

Systematic reviews Randomised controlled trials Qualitative research studies Economic evaluation studies Cohort studies Case control studies Diagnostic test studies

CEBM offers Critical Appraisal Sheets for:

• GRADE The GRADE working group has developed a common, sensible and transparent approach to grading quality of a body of evidence and strength of recommendations that can be drawn from randomized and non-randomized trials . GRADE is meant for use in systematic reviews and other evidence syntheses (e.g., clinical guidelines) where a recommendation impacting practice will be made.

JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers. There are checklists available for:

• The Patient Education Materials Assessment Tool (PEMAT) and User’s Guide The Patient Education Materials Assessment Tool (PEMAT) is a systematic method to evaluate and compare the understandability and actionability of patient education materials . It is designed as a guide to help determine whether patients will be able to understand and act on information. Separate tools are available for use with print and audiovisual materials.
• MMAT (Mixed Methods Appraisal Tool) 2018 "The MMAT is a critical appraisal tool that is designed for the appraisal stage of systematic mixed studies reviews, i.e., reviews that include qualitative, quantitative and mixed methods studies. It permits to appraise the methodological quality of five categories to studies: qualitative research, randomized controlled trials, non randomized studies, quantitative descriptive studies, and mixed methods studies."
• PEDro Scale (Physiotherapy Evidence Database) The PEDro scale was developed to help users rapidly identify trials that are likely to be internally valid and have sufficient statistical information to guide clinical decision-making.
• Risk of Bias (RoB) Tools The RoB 2 tool is designed for assessing risk of bias in randomized trials , while the ROBINS-I tool is meant for assessing non-randomized studies of interventions .
• CanChild / McMaster EBP Research Group - Evidence Review Forms Evidence review forms from the McMaster University Occupational Therapy Evidence-Based Practice for appraising quantitative and qualitative evidence.

Reporting Guidelines

• CONSORT (CONsolidated Standards Of Reporting Trials) The CONSORT Statement is an evidence-based, minimum set of standards for reporting of randomized trials . It offers a standard way for authors to prepare reports of trial findings, facilitating their complete and transparent reporting, and aiding their critical appraisal and interpretation.
• TREND (Transparent Reporting of Evaluations with Nonrandomized Designs) The TREND statement has a 22-item checklist specifically developed to guide standardized reporting of non-randomized controlled trials .

PRISMA is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses . PRISMA primarily focuses on the reporting of reviews evaluating the effects of interventions, but can also be used as a basis for reporting systematic reviews with objectives other than evaluating interventions.

There are also extensions available for scoping reviews , as well as other aspects or types of systematic reviews.

• SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence) The SQUIRE guidelines provide a framework for reporting new knowledge about how to improve healthcare (i.e., quality improvement ). These guidelines are intended for reports that describe system level work to improve the quality, safety, and value of healthcare, and used methods to establish that observed outcomes were due to the intervention(s).

Searchable Registries of Appraisal Tools & Reporting Guidelines

• Equator Network: Enhancing the QUAlity and Transparency Of health Research Comprehensive searchable database of reporting guidelines for main study types and also links to other resources relevant to research reporting.
• The Registry of Methods and Tools for Evidence-Informed Decision Making The Registry of Methods and Tools for Evidence-Informed Decision Making ("the Registry") is a collection of resources to support evidence-informed decision making in practice, programs and policy. This curated, searchable resource offers a selection of methods and tools for each step in the evidence-informed decision-making process. Includes tools related to implementation science , assessing the applicability and transferability of evidence.

For a list of additional tools, as well as some commentary on their use, see:

Ma, L.-L., Wang, Y.-Y., Yang, Z.-H., Huang, D., Weng, H., & Zeng, X.-T. (2020). Methodological quality (risk of bias) assessment tools for primary and secondary medical studies: What are they and which is better ? Military Medical Research, 7 (1), 7. https://doi.org/10.1186/s40779-020-00238-8

Determining Level of Evidence

Determining the level of evidence for a particular study or information source depends on understanding, the nature of the research question that is being investigated and the  methodology  that was used to collect the evidence.  See these these resources for help understanding study methodologies .  

There are a number of evidence hierarchies that could be used to 'rank' evidence. Which hierarchy is applied often depends on disciplinary norms - students should refer to materials and guidance from their professors about which hierarchy is appropriate to use.

• Oxford Centre for Evidence Based Medicine - Levels of Evidence The CEBM has put together a suite of documents to enable ranking of evidence into levels. Where a study falls in the ranking depends on the methodology of the study, and what kind of question (e.g., therapy, prognosis, diagnosis) is being addressed.
• Joanna Briggs Levels of Evidence [PDF] The JBI Levels of Evidence and Grades of Recommendation are meant to be used alongside the supporting document (PDF) outlining their use.
• << Previous: Evidence-Based Practice
• Next: Literature Reviews >>
• Last Updated: May 16, 2024 9:53 AM
• URL: https://guides.nyu.edu/ot

• Our services

Critical Appraisal Questionnaires

Critical appraisal is the process of carefully and systematically assessing the outcome of scientific research (evidence) to judge its trustworthiness, value and relevance in a particular context. Critical appraisal looks at the way a study is conducted and examines factors such as internal validity, generalizability and relevance.

Some initial appraisal questions you could ask are:

• Is the evidence from a known, reputable source?
• Has the evidence been evaluated in any way? If so, how and by whom?
• How up-to-date is the evidence?

Second, you could look at the study itself and ask the following general appraisal questions:

• How was the outcome measured?
• Is that a reliable way to measure?
• How large was the effect size?
• What implications does the study have for your practice? Is it relevant?
• Can the results be applied to your organization?

Questionnaires

If you would like to critically appraise a study, we strongly recommend using the app we have developed for iOS and Android:  CAT Manager App

You could also consider using the following appraisal questionnaires (checklists) for specific study designs, but we do not recommend this. 

Appraisal of a meta-analysis or systematic review

Appraisal of a controlled study, appraisal of a cohort or panel study, appraisal of a case control study, appraisal of a cross-sectional study (survey), appraisal of a qualitative study, appraisal of a case study.

• Navigate To
• Members area
• Bargelaan 200
• 2333 CW Leiden
• The Netherlands
• Want to stay up to date?

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.11(2); Spring 2007

Critical Appraisal of Clinical Studies: An Example from Computed Tomography Screening for Lung Cancer

Introduction.

Every physician is familiar with the impact that findings from studies published in scientific journals can have on medical practice, especially when the findings are amplified by popular press coverage and direct-to-consumer advertising. New studies are continually published in prominent journals, often proposing significant and costly changes in clinical practice. This situation has the potential to adversely affect the quality, delivery, and cost of care, especially if the proposed changes are not supported by the study's data. Reports about the results of a single study do not portray the many considerations inherent in a decision to recommend or not recommend an intervention in the context of a large health care organization like Kaiser Permanente (KP).

… in many cases, published articles do not discuss or acknowledge the weaknesses of the research …

Moreover, in many cases, published articles do not discuss or acknowledge the weaknesses of the research, and the reader must devote a considerable amount of time to identifying them. This creates a problem for the busy physician, who often lacks the time for systematic evaluation of the methodologic rigor and reliability of a study's findings. The Southern California Permanente Medical Group's Technology Assessment and Guidelines (TAG) Unit critically appraises studies published in peer-reviewed medical journals and provides evidence summaries to assist senior leaders and physicians in applying study findings to clinical practice. In the following sections, we provide a recent example of the TAG Unit's critical appraisal of a highly publicized study, highlighting key steps involved in the critical appraisal process.

Critical Appraisal: The I-ELCAP Study

In its October 26, 2006, issue, the New England Journal of Medicine published the results of the International Early Lung Cancer Action Program (I-ELCAP) study, a large clinical research study examining annual computed tomography (CT) screening for lung cancer in asymptomatic persons. Though the authors concluded that the screening program could save lives, and suggested that this justified screening asymptomatic populations, they offered no discussion of the shortcomings of the study. This report was accompanied by a favorable commentary containing no critique of the study's limitations, 1 and it garnered positive popular media coverage in outlets including the New York Times , CNN, and the CBS Evening News . Nevertheless, closer examination shows that the I-ELCAP study had significant limitations. Important harms of the study intervention were ignored. A careful review did not support the contention that screening for lung cancer with helical CT is clinically beneficial or that the benefits outweigh its potential harms and costs.

Critical appraisals of published studies address three questions:

• Are the study's results valid?
• What are the results?
• Will the results help in caring for my patient?

We discuss here the steps of critical appraisal in more detail and use the I-ELCAP study as an example of the way in which this process can identify important flaws in a given report.

Are the Study's Results Valid?

Assessing the validity of a study's results involves addressing three issues. First, does the study ask a clearly focused clinical question ? That is, does the paper clearly define the population of interest, the nature of the intervention, the standard of care to which the intervention is being compared, and the clinical outcomes of interest? If these are not obvious, it can be difficult to determine which patients the results apply to, the nature of the change in practice that the article proposes, and whether the intervention produces effects that both physician and patient consider important.

The clinical question researched in the I-ELCAP study 2 of CT screening for lung cancer is only partly defined. Although the outcomes of interest—early detection of lung carcinomas and lung cancer mortality—are obvious and the intervention is clearly described, the article is less clear with regard to the population of interest and the standard of care. The study population was not recruited through a standardized protocol. Rather, it included anyone deemed by physicians at the participating sites to be at above-average risk for lung cancer. Nearly 12% of the sample were individuals who had never smoked nor been exposed to lung carcinogens in the workplace; these persons were included on the basis of an unspecified level of secondhand smoke exposure. It is impossible to know whether they were subjected to enough secondhand smoke to give them a lung cancer risk profile similar to that of a smoker. It is also not obvious what was considered the standard of care in the I-ELCAP study. Although it is common for screening studies to compare intervention programs with “no screening,” the lack of a comparison group in this study leaves the standard entirely implicit.

Second, is the study's design appropriate to the clinical question ? Depending on the nature of the treatment or test, some study designs may be more appropriate to the question than others. The randomized controlled trial, in which a study subject sample is randomly divided into treatment and control groups and the clinical outcomes for each group are evaluated prospectively, is the gold standard for studies of screening programs and medical therapies. 3, 4 Cohort studies, in which a single group of study subjects is studied either prospectively or at a single point in time, are better suited to assessments of diagnostic or prognostic tools 3 and are less valid when applied to screening or treatment interventions. 5 Screening evaluations conducted without a control group may overestimate the effectiveness of the program relative to standard care by ignoring the benefits of standard care. Other designs, such as nonrandomized comparative studies, retrospective studies, case series, or case reports, are rarely appropriate for studying any clinical question. 5 However, a detailed discussion of threats to validity arising within particular study designs is beyond the scope of this article.

The I-ELCAP study illustrates the importance of this point. The nature of the intervention (a population screening program) called for a randomized controlled trial design, but the study was in fact a case series. Study subjects were recruited over time; however, because the intervention was an ongoing annual screening program, the number of CT examinations they received clearly varied, and it is impossible to tell from the data presented how the number of examinations per study subject is distributed within the sample. With different study subjects receiving different “doses” of the intervention, it thus becomes impossible to interpret the average effect of screening in the study. In particular, it is unclear how to interpret the ten-year survival curves the report presents; if the proportion of study subjects with ten years of data was relatively small, the survival rates would be very sensitive to the statistical model chosen to estimate them.

The lack of a control group also poses problems. Without a comparison group drawn from the same population, it is impossible to determine whether early detection through CT screening is superior to any other practice, including no screening. Survival data in a control group of unscreened persons would allow us to determine the lead time, or the interval of time between early detection of the disease and its clinical presentation. If individuals in whom stage I lung cancer was diagnosed would have survived for any length of time in the absence of screening, the mortality benefit of CT screening would have been overstated. Interpreting this interval as life saved because of screening is known as lead-time bias. The lack of a comparable control group also raises the question of overdiagnosis; without survival data from control subjects, it cannot be known how many of the lung cancers detected in I-ELCAP would have progressed to an advanced stage.

… does the paper clearly define the population of interest, the nature of the intervention, the standard of care to which the intervention is being compared, and the clinical outcomes of interest?

The types of cancers detected in the baseline and annual screening components of the I-ELCAP study only underscore this concern. Of the cancers diagnosed at baseline, only 9 cancers (3%) were small cell cancer, 263 (70%) were adenocarcinoma, and 45 (22%) were squamous cell cancer. Small cell and squamous cell cancers are almost always due to smoking. Data from nationally representative samples of lung cancer cases generally show that 20% of lung cancers are small cell, 40% are adenocarcinoma, and 30% are squamous cell. The prognosis for adenocarcinoma is better even at stage I than the prognoses for other cell types, especially small cell. 6 The I-ELCAP study data suggest that baseline screening might have detected the slow-growing tumors that would have presented much later.

A third question is whether the study was conducted in a methodologically sound way . This point concerns the conduct of the study and whether additional biases apart from those introduced by the design might have emerged. A discussion of the numerous sources of bias, including sample selection and measurement biases, is beyond the scope of this article. In randomized controlled trials of screening programs or therapies, it is important to know whether the randomization was done properly, whether the study groups were comparable at baseline, whether investigators were blinded to group assignments, whether contamination occurred (ie, intervention or control subjects not complying with study assignment), and whether intent-to-treat analyses were performed. In any prospective study, it is important to check whether significant attrition occurred, as a high dropout rate can greatly skew results.

In the case of the I-ELCAP study, 2 these concerns are somewhat overshadowed by those raised by the lack of a randomized design. It does not appear that the study suffered from substantial attrition over time. Diagnostic workups in the study were not defined by a strict protocol (protocols were recommended to participating physicians, but the decisions were left to the physician and the patient). This might have led to variation in how a true-positive case was determined.

What Are the Results?

Apart from simply describing the study's findings, the results component of critical appraisal requires the reader to address the size of the treatment effect and the precision of the treatment-effect estimate in the case of screening or therapy evaluations. The treatment effect is often expressed as the average difference between groups on some objective outcome measure (eg, SF-36 Health Survey score) or as a relative risk or odds ratio when the outcome is dichotomous (eg, mortality). In cohort studies without a comparison group, the treatment effect is frequently estimated by the difference between baseline and follow-up measures of the outcome, though such estimates are vulnerable to bias. The standard errors or confidence intervals around these estimates are the most common measures of precision.

The results of the I-ELCAP study 2 were as follows. At the baseline screening, 4186 of 31,567 study subjects (13%) were found by CT to have nodules qualifying as positive test results; of these, 405 (10%) were found to have lung cancer. An additional five study subjects (0.015%) with negative results at the baseline CT were given a diagnosis of lung cancer at the first annual CT screening, diagnoses that were thus classified as “interim.” At the subsequent annual CT screenings (delivered 27,456 times), 1460 study subjects showed new noncalcified nodules that qualified as significant results; of these, 74 study subjects (5%) were given a diagnosis of lung cancer. Of the 484 diagnoses of lung cancer, 412 involved clinical stage I disease. Among all patients with lung cancer, the estimated ten-year survival rate was 88%; among those who underwent resection within one month of diagnosis, estimated ten-year survival was 92%. Implied by these figures (but not stated by the study authors) is that the false-positive rate at the baseline screening was 90%—and 95% during the annual screens. Most importantly, without a control group, it is impossible to estimate the size or precision of the effect of screening for lung cancer. The design of the I-ELCAP study makes it impossible to estimate lead time in the sample, which was likely substantial, and again, the different “doses” of CT screening received by different study subjects make it impossible to determine how much screening actually produces the estimated benefit.

… would my patient have met the study's inclusion criteria, and if not, is the treatment likely to be similarly effective in my patient?

Will the Results Help in Caring for My Patient?

Answering the question of whether study results help in caring for one's patients requires careful consideration of three points. First, were the study's patients similar to my patient ? That is, would my patient have met the study's inclusion criteria, and if not, is the treatment likely to be similarly effective in my patient? This question is especially salient when we are contemplating new indications for a medical therapy. In the I-ELCAP study, 2 it is unclear whether the sample was representative of high-risk patients generally; insofar as nonsmokers exposed to secondhand smoke were recruited into the trial, it is likely that the risk profiles of the study's subjects were heterogeneous. The I-ELCAP study found a lower proportion of noncalcified nodules (13%) than did four other chest CT studies evaluated by our group (range, 23% to 51%), suggesting that it recruited a lower-risk population than these similar studies did. Thus, the progression of disease in the presence of CT screening in the I-ELCAP study might not be comparable to disease progression in any other at-risk population, including a population of smokers.

The second point for consideration is whether all clinically important outcomes were considered . That is, did the study evaluate all outcomes that both the physician and the patient are likely to view as important? Although the I-ELCAP study did provide data on rates of early lung cancers detected and lung cancer mortality, it did not address the question of morbidity or mortality related to diagnostic workup or cancer treatment, which are of interest in this population.

Finally, physicians should consider whether the likely treatment benefits are worth the potential harms and costs . Frequently, these considerations are blunted by the enthusiasm that new technologies engender. Investigators in studies such as I-ELCAP are often reluctant to acknowledge or discuss these concerns in the context of interventions that they strongly believe to be beneficial. The I-ELCAP investigators did not report any data on or discuss morbidity related to diagnostic procedures or treatment, and they explicitly considered treatment-related deaths to have been caused by lung cancer. Insofar as prior research has demonstrated that few pulmonary nodules prove to be cancerous, and because few positive test results in the trial led to diagnoses of lung cancer, it is reasonable to wonder whether the expected benefit to patients is offset by the difficulties and risks of procedures such as thoracotomy. The study report also did not discuss the carcinogenic risk associated with diagnostic imaging procedures. Data from the National Academy of Sciences' Seventh report on health risks from exposure to low levels of ionizing radiation 7 suggest that radiation would cause 11 to 22 cases of cancer in 10,000 persons undergoing one spiral CT. This risk would be greatly increased by a strategy of annual screening via CT, which would include many additional CT and positron-emission tomography examinations performed in diagnostic follow-ups of positive screening results. Were patients given annual CT screening for all 13 years of the I-ELCAP study, they would have absorbed an estimated total effective dose of 130 to 260 mSv, which would be associated with approximately 150 to 300 cases of cancer for every 10,000 persons screened. This is particularly critical for the nonsmoking study subjects in the I-ELCAP sample, who might have been at minimal risk for lung cancer; for them, radiation from screening CTs might have posed a significant and unnecessary health hazard.

In addition to direct harms, Eddy 5 and other advocates of evidence-based critical appraisal have argued that there are indirect harms to patients when resources are spent on unnecessary or ineffective forms of care at the expense of other services. In light of such indirect harms, the balance of benefits to costs is an important consideration. The authors of I-ELCAP 2 argued that the utility and cost-effectiveness of population mammography supported lung cancer screening in asymptomatic persons. A more appropriate comparison would involve other health care interventions aimed at reducing lung cancer mortality, including patient counseling and behavioral or pharmacologic interventions aimed at smoking cessation. Moreover, the authors cite an upper-bound cost of $200 for low-dose CT as suggestive of the intervention's cost-effectiveness. Although the I-ELCAP study data do not provide enough information for a valid cost-effectiveness analysis, the data imply that the study spent nearly $13 million on screening and diagnostic CTs. The costs of biopsies, positron-emission tomography scans, surgeries, and early-stage treatments were also not considered.

… did the study evaluate all outcomes that both the physician and the patient are likely to view as important?

Using the example of a recent, high-profile study of population CT screening for lung cancer, we discussed the various considerations that constitute a critical appraisal of a clinical trial. These steps include assessments of the study's validity, the magnitude and implications of its results, and its relevance for patient care. The appraisal process may appear long or tedious, but it is important to remember that the interpretation of emerging research can have enormous clinical and operational implications. In other words, in light of the stakes, we need to be sure that we understand what a given piece of research is telling us. As our critique of the I-ELCAP study report makes clear, even high-profile studies reported in prominent journals can have important weaknesses that may not be obvious on a cursory read of an article. Clearly, few physicians have time to critically evaluate all the research coming out in their field. The Technology Assessment and Guidelines Unit located in Southern California is available to assist KP physicians in reviewing the evidence for existing and emerging medical technologies.

Acknowledgments

Katharine O'Moore-Klopf of KOK Edit provided editorial assistance.

• Unger M. A pause, progress, and reassessment in lung cancer screening. N Engl J Med. 2006 Oct 26; 355 (17):1822–4. [ PubMed ] [ Google Scholar ]
• The International Early Lung Cancer Action Program Investigators. Survival of patients with stage I lung cancer detected on CT screening. N Engl J Med. 2006 Oct 26; 355 (17):1763–71. [ PubMed ] [ Google Scholar ]
• Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Chicago: Rand McNally; 1963. [ Google Scholar ]
• Holland P. Statistics and causal inference. J Am Stat Assoc. 1986; 81 :945–60. [ Google Scholar ]
• Eddy DM. A manual for assessing health practices and designing practice policies: the explicit approach. Philadelphia: American College of Physicians; 1992. [ Google Scholar ]
• Kufe DW, Pollock RE, Weichselbaum RR, et al., editors. Cancer Medicine. 6th ed. Hamilton, Ontario, Canada: BC Decker; 2003. (editors) [ Google Scholar ]
• National Academy of Sciences. Health risks from exposure to low levels of ionizing radiation: BEIR VII. Washington, DC: National Academies Press; 2005. [ PubMed ] [ Google Scholar ]