critical appraisal methodology

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Eldredge J. Evidence Based Practice: A Decision-Making Guide for Health Information Professionals [Internet]. Albuquerque (NM): University of New Mexico Health Sciences Library and Informatics Center; 2024.

Evidence Based Practice: A Decision-Making Guide for Health Information Professionals [Internet].

Critical appraisal.

Critical Appraisal: Wall Street. Bryce Canyon National Park. Utah

The goal of researchers should be to create accurate and unbiased representations of selected parts of reality. The goal of HIPs should be to critically examine how well these researchers achieve those representations of reality. Simultaneously, HIPs should gauge how relevant these studies are to answering their own EBP questions. In short, HIPs need to be skeptical consumers of the evidence produced by researchers.

This skepticism needs to be governed by critical thinking while recognizing that there are no perfect research studies . The expression, “The search for perfectionism is the enemy of progress,” 1 certainly applies when critically appraising evidence. HIPs should engage in “Proportional Skepticism,” meaning that finding minor flaws in a specific study does not automatically disqualify it from consideration in the EBP process. Research studies also vary in quality of implementation, regardless of whether they are systematic reviews or randomized controlled trials (RCTs). Proportional Skepticism acknowledges that some study designs are better than others at accurately representing parts of reality and answering different types of EBP questions. 2

This chapter provides tips for readers in their roles as consumers of evidence, particularly evidence produced by research studies. First, this chapter explores the various forms of bias that can cloud the representation of reality in research studies. It then presents a brief overview of common pitfalls in research studies. Many of these biases and pitfalls can easily be identified by how they might manifest in various forms of evidence, including local evidence, regional or national comparative data, and the grey literature. The emphasis in this chapter, however, is on finding significant weaknesses in research-generated evidence, as these can be more challenging to detect than in the other forms of evidence. Next, the chapter further outlines the characteristics of various research studies that might produce relevant evidence. Tables summarizing the major advantages and disadvantages of specific research designs appear throughout the chapter. Finally, critical appraisal sheets serve as appendices to the chapter. These sheets provide key questions to consider when evaluating evidence produced by the most common EBP research study designs.

• 4.1 Forms of Bias

Researchers and their research studies can be susceptible to many forms of bias, including implicit, structural, and systemic biases. These biases are deeply embedded in society and can inadvertently appear within research studies. 3 , 4 Bias in research studies can be defined as the “error in collecting or analyzing data that systematically over- or underestimates what the researcher is interested in studying.” 5 In simpler terms, bias results from an error in the design or conduct of a study. 6 As George Orwell reminds us, “..we are all capable of believing things we know to be untrue….” 7 Most experienced researchers are vigilant to avoid these types of biases, but biases can unintentionally permeate study designs or protocols irrespective of the researcher’s active vigilance. If researchers recognize these biases, they can mitigate them during their analyses or, at the very least, they should account for them in the limitations section of their research studies.

Expectancy Effect

For at least 60 years, behavioral scientists have observed a consistent phenomenon: when a researcher anticipates a particular response from someone, the likelihood of that person responding in the expected manner significantly increases. 8 This phenomenon, known as the Expectancy Effect, can be observed across research studies, ranging from interviews to experiments. In everyday life, we can observe the Expectancy Effect in action, such as when instructors selectively call on specific students in a classroom 9 , 10 or when the President or Press Secretary at a White House press conference chooses certain reporters while ignoring others. In both instances, individuals encouraged to interact with either the teacher or those conducting the press conference are more likely to seek future interactions. Robert Rosenthal, the psychologist most closely associated with uncovering and examining instances of the Expectancy Effect, has extensively documented these self-fulfilling prophecies in various everyday situations and within controlled research contexts. 11 , 12 , 13 , 14 It is important to recognize that HIP research studies have the potential to be influenced by the Expectancy Effect.

Hawthorne Effect

Participants in a research study, when aware of being observed by researchers, tend to behave differently than they would in other circumstances. 15 , 16 This phenomenon, known as the Hawthorne Effect, was initially discovered in an obscure management study conducted at the Hawthorne Electric Plant in Chicago. 17 , 18

The Hawthorne Effect might be seen in HIP participant-observer research studies where researchers monitor interactions such as help calls, chat sessions, and visits to a reference desk. In such studies, the employees responding to these requests might display heightened levels of patience, friendliness, or accommodation due to their awareness of being observed, which aligns with the Hawthorne Effect. It is important to note that the Hawthorne Effect is not inevitable, 19 and there are ways for researchers to mitigate its impact. 20

Historical events have the power to significantly shape research results. A compelling illustration of this can be found in the context of the COVID-19 pandemic, which left a profound mark on society. The impact of this crisis was so significant that a research study on attitudes toward infectious respiratory disease conducted in 2018 would likely yield different results compared to an identical study conducted in 2021, when society was cautiously emerging from the COVID-19 pandemic. Another instance demonstrating the impact of historical events is the banking crisis and recession of 2008-2009. Notably, two separate research studies on the most important research questions facing HIPs, conducted in 2008 and 2011, respectively, for the Medical Library Association Research Agenda, had unexpected differences in results. 21 The 2011 study identified HIPs as far more concerned about issues of economic and job security than the 2008 study. 22 The authors of the 2011 study specifically attributed the increased apprehension regarding financial insecurity to these historical events related to the economy. Additionally, historical events also can affect research results over the course of a long-term study. 23

During the course of longitudinal research studies, participants might experience changes in their attitudes or their knowledge. These changes, known as Maturation Effects, can sometimes be mistaken as outcomes resulting from specific events or interventions within the research study. 24 For instance, this might happen when a HIP provides instruction to first-year students aimed at fostering a positive attitude toward conducting literature searches. Following a second session on literature searching for second-year students, an attitudinal survey might indicate a higher opinion of this newly-learned skill among medical students. At this point, one might consider whether this attitude change resulted from the two instructional sessions or if there were other factors at play, such as a separate required course on research methods or other experiences of the students between the two searching sessions. In such a case, one would have to consider attributing the change to the Maturation Effect. 25

Misclassification

Misclassification can occur at multiple junctures in a research study, including when enrolling the participants, collecting data from participants, measuring exposures or interventions, or recording outcomes. 26 Minor misclassifications can introduce outsized distortions in any one of these junctures due to the multiple instances involved. The simple task of defining a research population can introduce the risk of misclassification, even among conscientious HIP researchers. 27

HIPs work with emerging information technologies far more than any other health sciences profession. Part of this work involves assessing the performance of these new information technologies as well as working on making adaptations to these technologies. Given the frequency that HIPs work with these new technologies, there remains the potential for novelty bias to arise, which refers to the initial fascination and perhaps even an enthusiasm towards innovations during their early phases of introduction and initial use. 28 This bias has been observed in publications from various health professions, spanning a wide range of innovations, such as dentistry tooth implants 29 or presentation software. 30

Many HIPs engage in partnerships with corporate information technology firms or other external organizations to pilot new platforms. These partnerships often result in reports on these technologies, typically case reports or new product reviews. Given the nature of these relationships, it is important for all authors to provide clear conflict of interest statements. For many HIPs, however, novelty bias is still an occupational risk due to their close involvement with information technologies. To mitigate this bias, HIPs can implement two strategies: increasing the number of participants in studies and continuing to try (often unsuccessfully) to replicate any initial rosy reports. 31

Recall Bias

Recall Bias poses a risk to study designs such as surveys and interviews, which heavily rely on the participants’ ability to recollect past events accurately. The wording of surveys and questions posed by interviewers can inadvertently direct participants’ attention to certain memories, thereby distorting the information provided to researchers. 32

Scalability Bias

Scalability Bias fails to consider the applicability of a study carried out in one specific context when transferred to another context. Shadish et al 33 identify two forms: Narrow-to-Broad Bias and Broad-to-Narrow Bias.

Narrow-to-Broad Bias applies findings in one setting and suggests that these findings apply to many other settings. For example, a researcher might attempt to depict the attitudes of all students on a large campus based on the interview of a single student or by surveying only five to 15 students who belong to a student interest group. Broad-to-Narrow Bias makes the inverse mistake by assuming that what generally applies to a large population should apply to an individual or a subset of that population. In this case, a researcher might conduct a survey on a campus to gauge attitudes toward a subject and assume that the general findings apply to every individual student. Readers familiar with classical training in logic or rhetoric will recognize these two biases as the Fallacy of Composition and the Fallacy of Hasty Generalization, respectively. 34

Selection Bias

Selection Bias happens when information or data collected in a research study does not accurately or fully represent the population of interest. It emerges when a sample distorts the realities of a larger population. For example, if a survey or a series of interviews with users only include friends of the researcher, it would be susceptible to Selection Bias, as it fails to encompass the broader range of attitudes present in the entire population. Recruitment into a study might occur only through media that are followed by a subset of the larger demographic profile needed. Engagement with an online patient portal, originally designed to mitigate Selection Bias in a particular study, unexpectedly gave rise to racial disparities instead. 35

Selection Bias can originate from within the study population itself, leading to potential distortions in the findings. For example, only those who feel strongly, either negatively or positively, towards a technology might volunteer to offer opinions on it. Selection Bias also might occur when an interviewer, for instance, either encourages interviewees or discourages interviewees from speaking on a subject. In all these cases, someone exerts control over the focus of the study that then misrepresents the actual experiences of the population. Rubin 36 reminds us that systemic and structural power structures in society exert control over what perspectives are heard in a research study.

While there are many other types of bias, the descriptions explained thus far should equip the vigilant consumer of research evidence with the ability to detect potential weaknesses across a wide range of HIP research articles.

• 4.2 Other Research Pitfalls

A cause is “an antecedent event, condition, or characteristic” that precedes an outcome. A sufficient cause provides the prerequisites for an outcome to occur, while a necessary cause must exist before the outcome can occur. 37 These definitions rely on the event, condition, or characteristic to precede the outcome temporally. At the same time, the cause and its outcome must comply with biological and physical laws. There must also be a plausible strength of the association, and the link between the putative cause and the outcome must be replicable across varied instances. 38 , 39 In the past century, philosophers and physicists have examined the concept of causality exhaustively, while 40 the concept of causality has been articulated over the past 70 years in the health sciences. 41 HIPs should keep these guidelines in mind when critically appraising any claims that an identified factor “caused” a specific outcome.

Confounding

Confounding relates to the inaccurate linkage of a possible cause to an identified outcome. It means that another concurrent event, condition, or characteristic actually caused the outcome. One instance of confounding might be an advertised noontime training on a new information platform that also features a highly desirable lunch buffet. The event planners might mistakenly assume that the high attendance rate stemmed from the perceived need for the training, while the primary motivation actually was the lunch buffet. In this case, the lunch served as a confounder.

Confounding presents the most significant alternative explanation to biases when trying to determine causation. 42 One recurring EBP question that arises among HIPs and academic librarians pertains to whether student engagement with HIPs and the use of information resources leads to student success and higher graduation rates. A research team investigated this issue and found that student engagement with HIPs and information resources did indeed predict student success. During the process, they were able to identify and eliminate potential confounders that might explain student success, such as high school grade point average, standardized exams, or socioeconomic status. 43 It turns out that even artificial intelligence can be susceptible to confounding, although it can “learn” to overcome those confounders. 44 Identifying and controlling for potential confounders can even resolve seemingly intractable questions. 45 RCTs are considered far superior in controlling for known or unknown confounders than other study designs, so they are often considered to be the highest form of evidence for a single intervention study. 46

Study Population

When considering a research study as evidence for making an EBP decision, it is crucial to evaluate whether the study population closely and credibly resembles your own user population. Each study design has specific features that can help answer this question. There are some general questions to consider that will sharpen one’s critical appraisal skills.

One thing to consider is whether the study population accurately represents the population it was drawn from. Are there any concerns regarding the sample size, which might make it insufficient to represent the larger population? Alternatively, were there issues with how the researchers publicized or recruited participants? 47 Selection Bias, mentioned earlier in this chapter, might contribute to the misrepresentation of a population if the researchers improperly included or excluded potential participants. It is also important to evaluate the response rate—was it too low, which could potentially introduce a nonresponse bias? Furthermore, consider whether the researchers’ specific incentives to enroll in the study attracted nonrepresentative participants.

HIPs should carefully analyze how research study populations align with their own user populations. In other words, what are the essential relevant traits that a research population might share or not share with a user population?

Validity refers to the use of an appropriate study design with measurements that are suitable for studying the subject. 48 , 49 It also applies to the appropriateness of the conclusions drawn from the research results. 50 Researchers devote considerable energy to examining the validity of their own studies as well as those conducted by others, so validity generally resides outside the scope of this guide intended for consumers of the research evidence.

Two brief examples might convey the concept of validity. In the first example, instructors conducting a training program on a new electronic health record system might claim success based on the number of providers they train. A more valid study, however, would include clear learning objectives that lead to demonstratable skills, which can be assessed after the training. Researchers could further extend the validity by querying trainees about their satisfaction or evaluating these trainees’ skills two weeks later to gauge the retention of the training. As a second example, a research study on a new platform might increase its validity by merely not reporting the number of visits to the platform. Instead, the study could gauge the level of user engagement through factors such as downloads, time spent on the platform, or the diversity of users.

• 4.3 Study Designs

Study designs, often referred to as “research methods” in journal articles or presentations, serve as the means to test researchers’ hypotheses. Study designs can be compared to tools such as hammers, screwdrivers, saws, or utensils used in a kitchen. Both analogies emphasize the importance of using the appropriate tool or utensil for the task at hand. One would not use a hammer when a saw would be a better choice, nor would one use a spatula to ladle a cup of soup from a pot. Similarly, researchers need to take care to use study designs best suited for the research question. Likewise, HIPs engaged in EBP should recognize the suitability of study designs in answering their EBP questions. The following section provides a review of the most common study designs 51 employed to answer EBP questions, which will be further discussed in the subsequent chapter on decision making.

Types of EBP Questions

There are three major types of EBP questions that repeatedly emerge from participants in continuing education courses: Exploration, Prediction, and Intervention. Coincidentally, these major types of questions also exist in research agendas for our profession. 52 Different study designs have optimal applicability in addressing the aforementioned issues of validity and in controlling biases for each type of question.

Exploration Questions

Exploration questions are frequently concerned with understanding the reasons behind certain phenomena and often begin with “Why.” For example, a central exploration question could be, “Why do health care providers seek health information?” Paradoxically, exploration “Why” research studies often do not ask participants direct “why” questions because this approach often leads to unproductive participant responses. 53 Other exploration questions might include:

• What are the specific Point-of-Care information needs of our front-line providers?
• Why do some potential users choose never to use the journals, books, and other electronic resources that we provide?
• Do our providers find the alerts that automatically populate their electronic health records useful?

Prediction Questions

Prediction questions aim to forecast future needs based on past patterns, and HIPs frequently pose such inquiries. These questions attempt to draw a causal connection between events, conditions, or characteristics in the present with outcomes in the future. Examples of prediction questions might include:

• To what extent do students retain their EBP question formulation and searching skills after two years?
• Do hospitals that employ HIPs produce better patient outcomes, as indicated by measures such as length of stay, mortality rates, or infection rates?
• Which archived committee meeting minutes within my organization are likely to be utilized within the next 30 years?

Intervention Questions

Intervention questions aim to distinguish between different potential courses of action to determine their effectiveness in achieving specific desirable outcomes. Examples of intervention questions might include:

• Does providing training on EBP question formulation and searching lead to an increase in information-seeking behavior among public health providers in rural practices?
• Which instructional approach yields better performance among medical students on standardized national licensure exams: didactic lecture or active learning with application exercises?
• Which Point-of-Care tool, DynaMed or UpToDate, generates more answers to patient care questions and higher provider satisfaction?

Case Reports

Case reports ( Table 1 ) are prevalent in the HIP literature and are often referred to interchangeably as “Case Studies.” 54 They are records of a single program, project, or experience, 55 with a particular focus on new developments or programs. These reports provide rich details on a single instance in a narrative format that is easy to understand. When done correctly, they can be far more challenging to develop than expected, contrary to the misconception that they are easy to assemble. 56 The most popular case reports revolve around innovation, which is broadly defined as “an idea, practice, or object” perceived to be new.” 57 A HIP innovation might be a new information technology, management initiative, or outreach program. A case report might also be the only available evidence due to the newness of the innovation, and in some rare instances, a case report might be the only usable evidence. For this reason, case reports hold the potential to point to new directions or emerging trends in the profession.

While case reports can serve an educational purpose and are generally interesting to read, they are not without controversy. Some researchers do not consider case reports as legitimate sources of research evidence, 58 and practitioners often approach them with skepticism. There are many opportunities for authors to unintentionally introduce biases into case reports. Skeptical practitioners criticize the unrealistic and overly positive accounts of innovations found in some case reports.

Case reports focus solely on a single instance of an innovation or noteworthy experience. 59 As a pragmatic matter, it can be difficult to justify adopting a program based on a case report carried out in one specific and different context. To illustrate the challenges of using case reports statistically, consider a hypothetical scenario: HIPs at 100 institutions might attempt to implement a highly publicized new information technology. HIPs at 96 of those institutions experience frustration at the poor performance of the new technology, and many eventually abandon it. Meanwhile, in this scenario, HIPs at four institutions present their highly positive case reports on their experiences with the new technology at an annual conference. These reports subsequently appear in the literature six months later. While these four case reports do not even reach the minimum standard for statistical significance, they become part of the only evidence base available for the new technology, thereby gaining prominence solely from “Survivor Bias” as they alone continued when most efforts to implement the new technology had failed. 60 , 61

Defenders of case reports argue that some of these issues can be mitigated when the reports include more rigorous elements. 62 , 63 For instance, a featured program in a case report might detail how carefully the authors evaluated the program using multiple meaningful measurements. 64 Another case report might document a well-conducted survey that provides plausible results in order to gauge peoples’ opinions. Practitioners tend to view case reports more favorably when they provide negative aspects of the program or innovation, framing them as “lessons learned.” Multiple authors representing different perspectives or institutions 65 seem to garner greater credibility. Full transparency, where authors make foundational documents and data available to readers, further bolsters potential applicability. Finally, a thorough literature review of other research studies, providing context and perhaps even external support for the case report’s results, can further increase credibility. 66 All of these elements increase the likelihood that the featured case report experience could potentially be transferred to another institution.

Case reports gain greater credibility, statistical probability, and transferability when combined with other case reports on the same innovation or similar experience, forming a related, although separate, study design known as a case series. Similar to case reports, case series gain greater credibility when their observations across cases are accompanied by literature reviews. 67

Interviews ( Table 2 ) are another common HIP research design. Interviews aim to understand the thoughts, preferences, or feelings of others. Interviews take different forms, including in-person or remote settings, as well as structured or unstructured questionnaires. They can involve one interviewer and one interviewee or a small team of interviewers who conduct group interviews, often referred to as a focus group. 68 , 69 , 70 While interviews technically fall under the category of surveys, they are discussed separately here due to their popularity in the HIP literature and the unique role of the interviewer in mediating and responding to interviewees.

Interviews can be highly exploratory, allowing researchers to discover unrecognized patterns or sentiments regardless of format. They might uniquely be able to answer “why?” research questions or probe interviewees’ motivations. 71 Interviews can sometimes lead to associations that can be further tested using other study designs. For instance, a set of interviews with non-hospital-affiliated practitioners about their information needs 72 can lead to an RCT comparing preferences for two different Point-of-Care tools. 73

Interviews have the potential to introduce various forms of biases. To minimize bias, researchers can employ strategies such as recruiting a representative sample of participants, using a neutral party to conduct the interviews, following a standardized protocol to ensure participants are interviewed equitably, and avoiding leading questions. The flexibility for interviewers to mediate and respond offers the strength to discover new insights. On balance, this flexibility also carries the risk of introducing bias if interviewers inject their own agendas into the interaction. Considering these potential biases, interviews are ranked below descriptive surveys in the Levels of Evidence discussed later in this chapter. To address concerns about bias, researchers should ensure that interviews thoroughly document and analyze all de-identified data collected in a transparent manner, allowing practitioners reviewing these studies to detect and mitigate any potential biases.

Descriptive Surveys

Surveys are an integral part of our society. Every ten years, Article 1, Section 2 of the United States Constitution requires everyone living in the United States to report demographic and other information about themselves as part of the Census. Governments have been taking censuses ever since ancient times in Babylon, Egypt, China, and India. 74 , 75 While censuses might offer highly accurate portrayals, they are time-consuming, complex, and expensive endeavors.

Most descriptive surveys involve polling a finite sample of the population, making sample surveys less time-consuming and less expensive compared to censuses. Surveys involving samples, however, are still complex. Surveys can be defined as a method for collecting information about a population of people. 76 They aim to describe, compare, or explain individual and societal knowledge, feelings, values, preferences, and behaviors.” 77 Surveys are accessed by respondents without a live intermediary administering them. For participants, surveys are almost always confidential and are often anonymous. There are three basic types of surveys: descriptive, change metric, and consensus.

Descriptive surveys ( Table 3 ) elicit respondents’ thoughts, emotions, or experiences regarding a subject or situation. Cross-sectional studies are one type of descriptive survey. 78 , 79 Change metric surveys are part of a cohort or experimental study where at least one survey takes place prior to an exposure or intervention. Later on during the study, the same participants are surveyed again to assess any changes or the extent of change. Sometimes, change metric surveys gauge the differences between user expectations and actual user experiments in surveys, known as Gap Analyses. Change metric surveys resemble descriptive surveys. They are discussed in subsequent study designs later in this chapter. The aim of consensus surveys is to facilitate agreement among groups regarding collective preferences or goals, even in situations where initial consensus might seem elusive. Consensus survey techniques on decision-making in EBP will be discussed in the next chapter.

Descriptive surveys are likely the most utilized research study design employed by HIPs. 80 The common use of surveys by HIP researchers and in society at large is one of the greatest weaknesses of descriptive surveys. While surveys are familiar, they have numerous pitfalls and limitations. 81 , 82 , 83 Even when large-scale public opinion surveys are conducted by experts, discrepancies often exist between survey results and actual population behavior, as evidenced by repeated erroneous election predictions by veteran pollsters. 84

Beyond the inherent limitations of survey designs, there are multiple points where researchers can unintentionally introduce bias or succumb to other pitfalls. Problems can arise at the outset when researchers design a survey without conducting an adequate literature review to consider the previous research on the subject. The survey instrument itself might contain confusing or misleading questions, including asking leading questions that elicit a “correct” answer rather than a truthful response. 85 , 86 For example, a question about alcohol consumption in a week might face validity issues due to social stigma. The recruitment process and the characteristics of participants can also introduce Selection Bias. 87 The introduction of the survey of the medium through which participants interact with the survey might underrepresent some demographic groups based on age, gender, class, or ethnicity. It is also important to consider the representativeness of the sample in relation to the target population. Is the sample large enough? 88 Interpreting survey results, particularly answers to open-ended questions, can also distort the study results. Regardless of how straightforward surveys might appear to participants or to the casual observer, they are oftentimes complex endeavors. 89 It is no wonder that the classic Survey Kit consists of 10 volumes to explain all the details that need to be attended to for a more successful survey. 90

Cohort Studies

Cohort studies ( Table 4 ) are one of several observational study designs that focus on observing possible causes (referred to as “exposures”) and their potential outcomes within a specific population. In cohort studies, investigators collect observations, usually in the form of data, without directly interjecting themselves into the situation. Cohort members are identified without the need for the explicit enrollment typically required in other designs. Figure 1 depicts the elements of a defined population, the exposure of interest, and the hypothesized outcome(s) in cohort studies. Cohort studies are fairly popular HIP research designs, although they are rarely labeled as such in the research literature. Cohort studies can be either prospective or retrospective. Retrospective cohort studies are conducted after the exposure has already occurred to some members of a cohort. These studies focus on examining past exposures and their impact on outcomes. 91

Cohort Study Design. Copyright Jonathan Eldredge. © 2023.

Many HIPs conducting studies on resource usage employ the retrospective cohort design. These studies link resource usage patterns to past exposures that might explain the observed patterns. That exposure might be a feature in the curriculum that requires learners to use that resource, or it could be an institutional expectation for employees to complete an online training module, which affects the volume of traffic on the module. On the other hand, prospective cohort studies begin in the present by identifying a cohort within the larger population and observing the exposure of interest and whether this exposure leads to an identified outcome. The researchers are collecting specific data as the cohort study progresses, including whether cohort members have been exposed and, if so, the duration or intensity of the exposure. These varied levels of exposure might be likened to different drug dosages.

Prospective cohort studies are generally regarded as less prone to bias or confounding than retrospective studies because researchers are intentional about collecting all measures throughout the study period. In contrast, retrospective studies are dependent on data collected in the past for other purposes. Those pre-existing compiled data sets might have missing elements necessary for the retrospective study. For example, in a retrospective cohort study, usage or traffic data on an online resource might have been originally collected by an institution to monitor the maintenance, increase, or decrease in the number of licensed simultaneous users. These data were originally gathered for administrative purposes, rather than for the primary research objectives of the study. Similarly, in a retrospective cohort study investigating the impact of providing tablets (exposure) to overcome barriers in using a portal (outcome), there might be a situation where the inventory system, initially created to track the distribution of tablets, is repurposed for a different objective, such as video conferencing. 92 Cohort studies regularly use change metric surveys, as discussed above. Prospective cohort studies are better at monitoring cohort members over the study duration, while retrospective cohort studies do not always have a clearly identified cohort membership due to the possible participant attrition not recorded in the outcomes. For this reason, plus the potentially higher integrity of the intentionally collected data, prospective cohort studies tend to be considered a higher form of evidence. The increased use of electronic inventories and the collection of greater amounts of data sometimes means that a data set created for one purpose can still be repurposed for a retrospective cohort study.

Quasi-Experiment

In contrast to observational studies like cohort studies, which involve researchers simply observing exposures and then measuring outcomes, quasi-experiments ( Table 5 ) include the active involvement of researchers in an intervention that is an intentional exposure. In quasi-experiments, researchers deliberately intervene and engage all members of a group of participants. 93 These interventions can take the form of training programs or work requirements that involve participants interacting with a new electronic resource, for example. Quasi-experiments are often employed by instructors who pre-test a group of learners, provide them with training on a specific skill or subject, and then conduct a post-test to measure the learners’ improved level of comprehension. 94 In this scenario, there is usually no explicit comparison with another untrained group of learners. The researchers’ active involvement tends to reduce some forms of bias and other pitfalls. Confounding, nevertheless, represents one looming potential weakness in quasi-experiments since a third unknown factor, a confounder, might be associated with the training and outcome but goes unrecognized by the researchers. 95 Quasi-experiments do not use randomization, which also can eliminate most confounders.

Quasi-Experiments

Randomized Controlled Trials (RCTs)

RCTs ( Table 6 ) are highly effective in resolving a choice between two seemingly reasonable courses of action. RCTs have helped answer some seemingly unresolvable HIP decisions in the past, including:

Randomized Controlled Trails (RCTs)

• Do embedded clinical librarians OR the availability of a reference service improve physician information-seeking behavior? 96 , 97
• Does weeding OR not weeding a collection lead to higher usage in a physical book collection? 98
• Does training in Evidence Based Public Health skills OR the lack of this training lead to an increase in public health practitioners’ formulated questions? 99

Paradoxically, RCTs are relatively uncommon in the HIP evidence base despite their powerful potential to resolve challenging decisions. 100 One explanation might be that, in the minds of the public and many health professionals, RCTs are often associated with pharmaceutical treatments. Researchers, however, have used RCTs far more broadly to resolve questions about devices, lifestyle modifications, or counseling in the realm of health care. Some HIP researchers believe that RCTs are too complex to implement, and some even consider RCTs to be unethical. These misapprehensions should be resolved by a closer reading about RCTs in this section and in the referenced sources.

Some HIPs, in their roles as consumers of research evidence, consider RCTs too difficult to interpret. This misconception might stem from the HIPs’ past involvement in systematic review teams that have evaluated pharmaceutical RCTs. Typically, these teams use risk-of-bias tools, which might appear overly complex to many HIPs. The most commonly used risk-of-bias tool 101 for critically appraising pharmaceutical RCTs appears to be published by Cochrane, particularly its checklist for identifying sources of bias. 102 Those HIPs involved with evaluating pharmaceutical RCTs should familiarize themselves with this resource. This section of Chapter 4 focuses instead on aspects of RCTs that HIPs might encounter in the HIP evidence base.

A few basic concepts and explanations of RCT protocols should alleviate any reluctance to use RCTs in the EBP Process. The first concept of equipoise means that researchers undertook the RCT because they were genuinely uncertain about which course of action among the two choices would lead to the more desired outcomes. Equipoise has practical and ethical dimensions. If prior studies have demonstrated a definitively superior course of action between the two choices, researchers would not invest their time and effort in pursuing an already answered question unless they needed to replicate the study. From an ethical perspective, why would researchers subject control group participants to a clearly inferior choice? 103 , 104 The background or introduction section of an article about RCTs should establish equipoise by drawing on evidence from past studies.

Figure 2 illustrates that an RCT begins with a representative sample of the larger population. Many consumers of RCTs should bear in mind that the number of participants in a study depends on more than a “magic number”; it also relies on the availability of eligible participants and the statistical significance of any differences measured. 105 , 106 , 107 , 108 Editors, peer reviewers, and statistical consultants at peer-reviewed journals play a key role in screening manuscripts for any major statistical problems.

Randomized Controlled Trial.

Recruiting a representative sample can be a challenge for researchers due to the various communication channels used by different people. Consumers of RCTs should be aware of these issues since they affect the applicability of any RCT to one’s local setting. Several studies have documented age, gender, socioeconomic, and ethnic underrepresentation in RCTs. 109 , 110 , 111 , 112 , 113 One approach to addressing this issue is to tailor the incentives for participation based on the appeals to different underrepresented groups. 114 Close collaboration with underrepresented groups through community outreach also can help increase participation. Many RCTs include a table that records the demographic representation of participants in the study, along with the demographic composition of those who dropped out. HIPs evaluating an RCT can scrutinize this table to assess how closely the study’s population aligns with their own user population. RCTs oftentimes screen out potential participants who are unlikely to adhere to the study protocol or who are likely to drop out. Participants who will be unavailable during key study dates might also be removed. HIP researchers might want to exclude potential participants who have prior knowledge of a platform under study or who might be repeating an academic course where they were previously exposed to the same content. These preliminary screening measures cannot anticipate all eventualities, which is why some articles include a CONSORT diagram to provide a comprehensive overview of the study design. 115

RCTs often control for most biases and confounding through randomization . Imagine you’re in the tenth car in a right-hand lane approaching a traffic signal at an intersection, and no one ahead of you uses their turn signal. You want to take a right turn immediately after the upcoming intersection. In this situation, you don’t know which cars will actually turn right and which ones will proceed straight. If you want to stay in the right-hand lane without turning right, you can’t predict who will slow you down by taking a right or who will continue at a faster pace straight ahead to your own turnoff. This scenario is similar to randomization in RCTs because, just like in the traffic situation, you don’t know in advance which participants will be assigned to each course of action. Randomization ensures that each participant has an equal chance of being assigned to either group regardless of the allocation of others before them, effectively eliminating the influence of bias, confounding, or any other unknown factors that could impact the study’s outcomes.

Contamination poses a threat to the effectiveness of the randomization. It occurs when members of the intervention or control groups interact and collaborate, thereby inadvertently altering the intended effects of the study. RCTs normally have an intervention group that receives a new experience, which will possibly lead to more desired outcomes. The intervention can involve accessing new technology, piloting a new teaching style, receiving specialized training content, or other deliberate actions by the researchers. On the other hand, control group members typically receive the established technology, experience the usual teaching techniques, receive standard training content, or have the usual set of experiences.

Contamination might arise when members of the intervention and control groups exchange information about their group’s experiences. Contamination interferes with the researchers deliberately treating the intervention and control groups differently. For example, in an academic setting, contamination might happen if the intervention group receives new training while the control group receives traditional training. In a contamination scenario, members of the two groups would exchange information. When their knowledge or skills are tested at the end of the study, the assessment might not accurately reflect their comparative progress since both groups have been exposed to each other’s training. A Delphi Study generated a list of common sources of contamination in RCTs, including participants’ physical proximity, frequent interaction, and high desirability of the intervention. 116 Information technology can assist researchers in avoiding, or at least countering, contamination by interacting with study participants virtually rather than in a physical environment. Additionally, electronic health records can similarly be employed in studies while minimizing contamination. 117

RCTs often employ concealment (“blinding”) techniques to ensure that participants, researchers, statisticians, and other analysts are unaware of which participants are enrolled in either the intervention or control groups. Concealment prevents participants from deviating from the protocols. Concealment also reduces the likelihood that the researchers, statisticians, or analysts interject their views into the study protocol, leading to unintended effects or biases. 118

Systematic Reviews

Systematic reviews ( Table 7 ) strive to offer a transparent and replicable synthesis of the best evidence to answer a narrowly focused question. They often involve exhaustive searches of the peer-reviewed literature. While many HIPs have participated in teams conducting systematic reviews, these efforts primarily serve health professions outside of HIP subject areas. 119 Systematic reviews can be time-consuming and labor-intensive endeavors. They rely on a number of the same critical appraisal skills covered in this chapter and its appendices to evaluate multiple studies.

Systematic reviews can include evidence produced by any study design except other reviews. Producers of systematic reviews often exclude study designs more prone to biases or confounding when a sufficient number of studies with fewer similar limitations are available. Systematic reviews are popular despite being relatively limited in number. If well-conducted, they can bypass the first three steps of the EBP Process and position the practitioner well to make an informed decision. The narrow scope of systematic reviews, however, does limit their applicability to a broad range of decisions.

Nearly all HIPs have used the findings of systematic reviews for their own EBP questions. 120 Since much of the HIP evidence base exists outside of the peer-reviewed literature, systematic reviews on HIP subjects can include grey literature, such as presented papers or posters from conferences or white papers from organizations. The MEDLINE database has a filter for selecting systematic reviews as an article type when searching the peer-reviewed literature. Unfortunately, this filter sometimes mistakenly includes meta-analyses and narrative review articles due to the likely confusion among indexers regarding the differences between these article types. It is important to note that meta-analyses are not even a design type; instead, they are a statistical method used to aggregate data sets from more than one study. They can be used for comparative study or systematic review study design types, but some people equate them solely to systematic reviews.

Narrative reviews, on the other hand, are an article type that provides a broad overview of a topic and often lacks the more rigorous features of a systematic review. Scoping reviews have increased in popularity in recent years but have a descriptive purpose that contrasts with systematic reviews. Sutton et al 121 have published an impressive inventory of the many types of reviews that might be confused with systematic reviews. The authors of systematic reviews themselves might contribute to the confusion by mislabeling these studies. The Library, Information Science, and Technology Abstracts database does not offer a filter for systematic reviews, so a keyword approach should be used when searching, followed by a manual screening of the resulting references.

Systematic reviews offer the potential to avoid many of the biases and pitfalls described at the beginning of this chapter.

In actuality, they can fall short of this potential to varying degrees, ranging from minor to monumental ways. The question being addressed needs to be narrowly focused to make the subsequent process manageable, which might disqualify some systematic reviews from application to HIPs’ actual EBP questions. The literature search might not be comprehensive, either due to limited sources searched or inadequately executed searches, leading to the possibility of missing important evidence. The searches might not be documented well enough to be reproduced by other researchers. The exclusion and inclusion criteria for identified studies might not calibrate with the needs of HIPs. The critical appraisal in some systematic reviews might exclude reviewed studies for trivial deficiencies or include studies with major flaws. The recommendations of some systematic reviews, therefore, might not be supported by the identified best available evidence.

Levels of Evidence

The Levels of Evidence, also known as “Hierarchies of Evidence,” are valuable sources of guidance in EBP. They serve as a reminder to busy practitioners that study designs at lower levels have difficulty in avoiding, controlling, or compensating for the many forms of bias or confounding that can affect research studies. Study designs at higher levels tend to be better at controlling biases. For example, higher-level study designs like RCTs can effectively control confounding. Table 8 organizes study designs according to EBP question type and arrays them into their approximate levels. In Table 8 , the “Intervention Question” column recognizes that a properly conducted systematic review incorporating multiple studies is generally more desirable for making an EBP decision compared to a case report. This is because the latter can be vulnerable to many forms of bias and confounding that rely on findings from a single instance. A systematic review, on the other hand, is ranked higher than even an RCT because it combines all available evidence from multiple studies and subjects them to a critical review, leading to a recommendation for making a decision.

Levels of Evidence: An Approximate Hierarchy Linked to Question Type

There are several important caveats to consider when using the Levels of Evidence. As noted earlier in this chapter, no perfect research study exists, and even higher-level of evidence research studies can have weaknesses. Hypothetically, an RCT could be so poorly executed that a well-conducted case report on the same topic could outshine it. While this is possible, it is highly unlikely due to the superior design of an RCT for controlling confounding or biases. Sometimes, a case report might be slightly more relevant than an RCT in answering an Intervention-type of EBP question. For these reasons, one cannot abandon their critical thinking skills even with a tacit acceptance of the Levels of Evidence.

The Levels of Evidence have been widely endorsed by HIP leaders for many years. 122 , 123 They undergo occasional adjustments, but their general organizing principles of controlling biases and other pitfalls remain intact. On balance, two of my otherwise respected colleagues have misinterpreted aspects of the early Levels of Evidence and made that the basis of their criticism. 124 , 125 A fair reading of the evolution of the Levels of Evidence over the years 126 , 127 , 128 , 129 should convince most readers that, when coupled with critical thinking, the underlying principles of the Levels of Evidence continue to provide HIPs with sound guidance.

Critical Appraisal Sheets

The critical appraisal sheets appended to this chapter are intended to serve as a guide for HIPs as they engage in critical appraisal of their potential evidence. The development of these sheets has been a culmination of over 20 years of effort. They draw upon my doctoral training in research methods, as well as my extensive experience conducting research using various study designs. Additionally, I have insights from multiple authorities. While it is impossible to credit all sources that have influenced the development of these sheets over the years, I have cited the readily recognized ones at the end of this sentence. 130 , 131 , 132 , 133 , 134 , 135 , 136 , 137 , 138 , 139 , 140 , 141

• Critical Appraisal Worksheets

Appendix 1: Case Reports

Instructions: Answer the following questions to critically appraise this piece of evidence.

Appendix 2: Interviews

Appendix 3: descriptive surveys.

Instructions: Answer the following questions to critically appraise this piece of evidence

Appendix 4: Cohort Studies

Appendix 5: quasi-experiments, appendix 6: randomized controlled trials, appendix 7: systematic reviews.

This is an open access publication. Except where otherwise noted, this work is distributed under the terms of a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International license (CC BY-NC-SA 4.0 DEED), a copy of which is available at https://creativecommons.org/licenses/by-nc-sa/4.0/ .

This open access peer-reviewed Book is brought to you at no cost to you by the Health Sciences Center at UNM Digital Repository. It has been accepted for inclusion in the Faculty Book Display Case by an authorized administrator of UNM Digital Repository. For more information, please contact [email protected].

• Cite this Page Eldredge J. Evidence Based Practice: A Decision-Making Guide for Health Information Professionals [Internet]. Albuquerque (NM): University of New Mexico Health Sciences Library and Informatics Center; 2024. Critical Appraisal.
• PDF version of this title (32M)

In this Page

Related information.

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• Critical Appraisal - Evidence Based Practice Critical Appraisal - Evidence Based Practice

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

• Mayo Clinic Libraries
• Evidence Synthesis Guide
• Risk of Bias by Study Design

Evidence Synthesis Guide : Risk of Bias by Study Design

• Review Types & Decision Tree
• Standards & Reporting Results
• Materials in the Mayo Clinic Libraries
• Training Resources
• Review Teams
• Develop & Refine Your Research Question
• Develop a Timeline
• Project Management
• Communication
• PRISMA-P Checklist
• Eligibility Criteria
• Register your Protocol
• Other Screening Tools
• Grey Literature Searching
• Citation Searching
• Minimize Bias
• GRADE & GRADE-CERQual
• Data Extraction Tools
• Synthesis & Meta-Analysis
• Publishing your Review
• Mayo Clinic Libraries Evidence Synthesis Service

Risk of Bias of Individual Studies

““Assessment of risk of bias is a key step that informs many other steps and decisions made in conducting systematic reviews. It plays an important role in the final assessment of the strength of the evidence.” 1  

Risk of Bias by Study Design (featured tools)

• Systematic Reviews
• Non-RCTs or Observational Studies
• Diagnostic Accuracy
• Animal Studies
• Qualitative Research
• Tool Repository
• AMSTAR 2 The original AMSTAR was developed to assess the risk of bias in systematic reviews that included only randomized controlled trials. AMSTAR 2 was published in 2017 and allows researchers to identify high quality systematic reviews, including those based on non-randomised studies of healthcare interventions. more... less... AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)
• ROBIS ROBIS is a tool designed specifically to assess the risk of bias in systematic reviews. The tool is completed in three phases: (1) assess relevance(optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help assess specific concerns about potential biases with the review. more... less... ROBIS (Risk of Bias in Systematic Reviews)
• BMJ Framework for Assessing Systematic Reviews This framework provides a checklist that is used to evaluate the quality of a systematic review.
• CASP Checklist for Systematic Reviews This CASP checklist is not a scoring system, but rather a method of appraising systematic reviews by considering: 1. Are the results of the study valid? 2. What are the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Systematic Reviews Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• JBI Critical Appraisal Tools, Checklist for Systematic Reviews JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• NHLBI Study Quality Assessment of Systematic Reviews and Meta-Analyses The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• RoB 2 RoB 2 provides a framework for assessing the risk of bias in a single estimate of an intervention effect reported from a randomized trial, rather than the entire trial. more... less... RoB 2 (revised tool to assess Risk of Bias in randomized trials)
• CASP Randomised Controlled Trials Checklist This CASP checklist considers various aspects of an RCT that require critical appraisal: 1. Is the basic study design valid for a randomized controlled trial? 2. Was the study methodologically sound? 3. What are the results? 4. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CONSORT Statement The CONSORT checklist includes 25 items to determine the quality of randomized controlled trials. Critical appraisal of the quality of clinical trials is possible only if the design, conduct, and analysis of RCTs are thoroughly and accurately described in the report. more... less... CONSORT (Consolidated Standards of Reporting Trials)
• NHLBI Study Quality Assessment of Controlled Intervention Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. more... less... NHLBI (National Heart, Lung, and Blood Institute)
• JBI Critical Appraisal Tools Checklist for Randomized Controlled Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• ROBINS-I ROBINS-I is a tool for evaluating risk of bias in estimates of the comparative effectiveness… of interventions from studies that did not use randomization to allocate units to comparison groups. more... less... ROBINS-I (Risk Of Bias in Non-randomized Studies – of Interventions)
• NOS This tool is used primarily to evaluate and appraise case-control or cohort studies. more... less... NOS (Newcastle-Ottawa Scale)
• AXIS Cross-sectional studies are frequently used as an evidence base for diagnostic testing, risk factors for disease, and prevalence studies. The AXIS tool focuses mainly on the presented study methods and results. more... less... AXIS (Appraisal tool for Cross-Sectional Studies)
• NHLBI Study Quality Assessment Tools for Non-Randomized Studies The NHLBI’s quality assessment tools were designed to assist reviewers in focusing on concepts that are key for critical appraisal of the internal validity of a study. • Quality Assessment Tool for Observational Cohort and Cross-Sectional Studies • Quality Assessment of Case-Control Studies • Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group • Quality Assessment Tool for Case Series Studies more... less... NHLBI (National Heart, Lung, and Blood Institute)
• Case Series Studies Quality Appraisal Checklist Developed by the Institute of Health Economics (Canada), the checklist is comprised of 20 questions to assess the robustness of the evidence of uncontrolled case series studies.
• Methodological Quality and Synthesis of Case Series and Case Reports In this paper, Dr. Murad and colleagues present a framework for appraisal, synthesis and application of evidence derived from case reports and case series.
• MINORS The MINORS instrument contains 12 items and was developed for evaluating the quality of observational or non-randomized studies. This tool may be of particular interest to researchers who would like to critically appraise surgical studies. more... less... MINORS (Methodological Index for Non-Randomized Studies)
• JBI Critical Appraisal Tools for Non-Randomized Trials JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis. • Checklist for Analytical Cross Sectional Studies • Checklist for Case Control Studies • Checklist for Case Reports • Checklist for Case Series • Checklist for Cohort Studies
• QUADAS-2 The QUADAS-2 tool is designed to assess the quality of primary diagnostic accuracy studies it consists of 4 key domains that discuss patient selection, index test, reference standard, and flow of patients through the study and timing of the index tests and reference standard. more... less... QUADAS-2 (a revised tool for the Quality Assessment of Diagnostic Accuracy Studies)
• JBI Critical Appraisal Tools Checklist for Diagnostic Test Accuracy Studies JBI Critical Appraisal Tools help you assess the methodological quality of a study and to determine the extent to which study has addressed the possibility of bias in its design, conduct and analysis.
• STARD 2015 The authors of the standards note that essential elements of diagnostic accuracy study methods are often poorly described and sometimes completely omitted, making both critical appraisal and replication difficult, if not impossible. The Standards for the Reporting of Diagnostic Accuracy Studies was developed to help improve completeness and transparency in reporting of diagnostic accuracy studies. more... less... STARD 2015 (Standards for the Reporting of Diagnostic Accuracy Studies)
• CASP Diagnostic Study Checklist This CASP checklist considers various aspects of diagnostic test studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• CEBM Diagnostic Critical Appraisal Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance, and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• SYRCLE’s RoB Implementation of SYRCLE’s RoB tool will facilitate and improve critical appraisal of evidence from animal studies. This may enhance the efficiency of translating animal research into clinical practice and increase awareness of the necessity of improving the methodological quality of animal studies. more... less... SYRCLE’s RoB (SYstematic Review Center for Laboratory animal Experimentation’s Risk of Bias)
• ARRIVE 2.0 The ARRIVE 2.0 guidelines are a checklist of information to include in a manuscript to ensure that publications on in vivo animal studies contain enough information to add to the knowledge base. more... less... ARRIVE 2.0 (Animal Research: Reporting of In Vivo Experiments)
• Critical Appraisal of Studies Using Laboratory Animal Models This article provides an approach to critically appraising papers based on the results of laboratory animal experiments, and discusses various bias domains in the literature that critical appraisal can identify.
• CEBM Critical Appraisal of Qualitative Studies Sheet The CEBM’s critical appraisal sheets are designed to help you appraise the reliability, importance and applicability of clinical evidence. more... less... CEBM (Centre for Evidence-Based Medicine)
• CASP Qualitative Studies Checklist This CASP checklist considers various aspects of qualitative research studies including: 1. Are the results of the study valid? 2. What were the results? 3. Will the results help locally? more... less... CASP (Critical Appraisal Skills Programme)
• Quality Assessment and Risk of Bias Tool Repository Created by librarians at Duke University, this extensive listing contains over 100 commonly used risk of bias tools that may be sorted by study type.
• Latitudes Network A library of risk of bias tools for use in evidence syntheses that provides selection help and training videos.

References & Recommended Reading

1.    Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., ... & Kane, R. L. (2018). Recommendations for assessing the risk of bias in systematic reviews of health-care interventions .  Journal of clinical epidemiology ,  97 , 26-34.

2.     Kolaski, K., Logan, L. R., & Ioannidis, J. P. (2024). Guidance to best tools and practices for systematic reviews .  British Journal of Pharmacology ,  181 (1), 180-210

3.     Fowkes FG, Fulton PM.  Critical appraisal of published research: introductory guidelines.   BMJ (Clinical research ed).  1991;302(6785):1136-1140.

4.     Shea BJ, Reeves BC, Wells G, et al.  AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both.   BMJ (Clinical research ed).  2017;358:j4008.

5..     Whiting P, Savovic J, Higgins JPT, et al.  ROBIS: A new tool to assess risk of bias in systematic reviews was developed.   Journal of clinical epidemiology.  2016;69:225-234.

6.     Sterne JAC, Savovic J, Page MJ, et al.  RoB 2: a revised tool for assessing risk of bias in randomised trials.  BMJ (Clinical research ed).  2019;366:l4898.

7.     Moher D, Hopewell S, Schulz KF, et al.  CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials.  Journal of clinical epidemiology.  2010;63(8):e1-37.

8..    Sterne JA, Hernan MA, Reeves BC, et al.  ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions.  BMJ (Clinical research ed).  2016;355:i4919.

9.    Downes MJ, Brennan ML, Williams HC, Dean RS.  Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS).   BMJ open.  2016;6(12):e011458.

10.   Guo B, Moga C, Harstall C, Schopflocher D.  A principal component analysis is conducted for a case series quality appraisal checklist.   Journal of clinical epidemiology.  2016;69:199-207.e192.

11.   Murad MH, Sultan S, Haffar S, Bazerbachi F.  Methodological quality and synthesis of case series and case reports.  BMJ evidence-based medicine.  2018;23(2):60-63.

12.   Slim K, Nini E, Forestier D, Kwiatkowski F, Panis Y, Chipponi J.  Methodological index for non-randomized studies (MINORS): development and validation of a new instrument.   ANZ journal of surgery.  2003;73(9):712-716.

13.   Whiting PF, Rutjes AWS, Westwood ME, et al.  QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies.   Annals of internal medicine.  2011;155(8):529-536.

14.   Bossuyt PM, Reitsma JB, Bruns DE, et al.  STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies.   BMJ (Clinical research ed).  2015;351:h5527.

15.   Hooijmans CR, Rovers MM, de Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW.  SYRCLE's risk of bias tool for animal studies.   BMC medical research methodology.  2014;14:43.

16.   Percie du Sert N, Ahluwalia A, Alam S, et al.  Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0.  PLoS biology.  2020;18(7):e3000411.

17.   O'Connor AM, Sargeant JM.  Critical appraisal of studies using laboratory animal models.   ILAR journal.  2014;55(3):405-417.

• << Previous: Minimize Bias
• Next: GRADE & GRADE-CERQual >>
• Last Updated: Oct 23, 2024 2:48 PM
• URL: https://libraryguides.mayo.edu/systematicreviewprocess

• University of Texas Libraries
• UT Libraries

Systematic Reviews & Evidence Synthesis Methods

Critical appraisal.

• Types of Reviews
• Formulate Question
• Find Existing Reviews & Protocols
• Register a Protocol
• Searching Systematically
• Search Hedges and Filters
• Supplementary Searching
• Managing Results
• Deduplication
• Glossary of Terms
• Librarian Support
• Systematic Review & Evidence Synthesis Boot Camp

Some reviews require a critical appraisal for each study that makes it through the screening process. This involves a risk of bias assessment and/or a quality assessment. The goal of these reviews is not just to find all of the studies, but to determine their methodological rigor, and therefore, their credibility.

"Critical appraisal is the balanced assessment of a piece of research, looking for its strengths and weaknesses and them coming to a balanced judgement about its trustworthiness and its suitability for use in a particular context." 1

It's important to consider the impact that poorly designed studies could have on your findings and to rule out inaccurate or biased work.

Selection of a valid critical appraisal tool, testing the tool with several of the selected studies, and involving two or more reviewers in the appraisal are good practices to follow.

1. Purssell E, McCrae N. How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students. 1st ed. Springer ;  2020.

Evaluation Tools

• The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) The Appraisal of Guidelines for Research & Evaluation Instrument (AGREE II) was developed to address the issue of variability in the quality of practice guidelines.
• Centre for Evidence-Based Medicine (CEBM). Critical Appraisal Tools "contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples."
• Critical Appraisal Skills Programme (CASP) Checklists Critical Appraisal checklists for many different study types
• Critical Review Form for Qualitative Studies Version 2, developed out of McMaster University
• Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS) Downes MJ, Brennan ML, Williams HC, et al. Development of a critical appraisal tool to assess the quality of cross-sectional studies (AXIS). BMJ Open 2016;6:e011458. doi:10.1136/bmjopen-2016-011458
• Downs & Black Checklist for Assessing Studies Downs, S. H., & Black, N. (1998). The Feasibility of Creating a Checklist for the Assessment of the Methodological Quality Both of Randomised and Non-Randomised Studies of Health Care Interventions. Journal of Epidemiology and Community Health (1979-), 52(6), 377–384.
• GRADE The Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group "has developed a common, sensible and transparent approach to grading quality (or certainty) of evidence and strength of recommendations."
• Grade Handbook Full handbook on the GRADE method for grading quality of evidence.
• MAGIC (Making GRADE the Irresistible choice) Clear succinct guidance in how to use GRADE
• Joanna Briggs Institute. Critical Appraisal Tools "JBI’s critical appraisal tools assist in assessing the trustworthiness, relevance and results of published papers." Includes checklists for 13 types of articles.
• Latitudes Network This is a searchable library of validity assessment tools for use in evidence syntheses. This website also provides access to training on the process of validity assessment.
• Mixed Methods Appraisal Tool A tool that can be used to appraise a mix of studies that are included in a systematic review - qualitative research, RCTs, non-randomized studies, quantitative studies, mixed methods studies.
• RoB 2 Tool Higgins JPT, Sterne JAC, Savović J, Page MJ, Hróbjartsson A, Boutron I, Reeves B, Eldridge S. A revised tool for assessing risk of bias in randomized trials In: Chandler J, McKenzie J, Boutron I, Welch V (editors). Cochrane Methods. Cochrane Database of Systematic Reviews 2016, Issue 10 (Suppl 1). dx.doi.org/10.1002/14651858.CD201601.
• ROBINS-I Risk of Bias for non-randomized (observational) studies or cohorts of interventions Sterne J A, Hernán M A, Reeves B C, Savović J, Berkman N D, Viswanathan M et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions BMJ 2016; 355 :i4919 doi:10.1136/bmj.i4919
• Scottish Intercollegiate Guidelines Network. Critical Appraisal Notes and Checklists "Methodological assessment of studies selected as potential sources of evidence is based on a number of criteria that focus on those aspects of the study design that research has shown to have a significant effect on the risk of bias in the results reported and conclusions drawn. These criteria differ between study types, and a range of checklists is used to bring a degree of consistency to the assessment process."
• The TREND Statement (CDC) Des Jarlais DC, Lyles C, Crepaz N, and the TREND Group. Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: The TREND statement. Am J Public Health. 2004;94:361-366.
• Assembling the Pieces of a Systematic Reviews, Chapter 8: Evaluating: Study Selection and Critical Appraisal.
• How to Perform a Systematic Literature Review, Chapter: Critical Appraisal: Assessing the Quality of Studies.

Other library guides

• Duke University Medical Center Library. Systematic Reviews: Assess for Quality and Bias
• UNC Health Sciences Library. Systematic Reviews: Assess Quality of Included Studies
• Last Updated: Oct 31, 2024 7:47 AM
• URL: https://guides.lib.utexas.edu/systematicreviews

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Volume 25, Issue 1
• Critical appraisal of qualitative research: necessity, partialities and the issue of bias
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0001-5660-8224 Veronika Williams ,
• Anne-Marie Boylan ,
• http://orcid.org/0000-0003-4597-1276 David Nunan
• Nuffield Department of Primary Care Health Sciences , University of Oxford, Radcliffe Observatory Quarter , Oxford , UK
• Correspondence to Dr Veronika Williams, Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford OX2 6GG, UK; veronika.williams{at}phc.ox.ac.uk

https://doi.org/10.1136/bmjebm-2018-111132

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

• qualitative research

Introduction

Qualitative evidence allows researchers to analyse human experience and provides useful exploratory insights into experiential matters and meaning, often explaining the ‘how’ and ‘why’. As we have argued previously 1 , qualitative research has an important place within evidence-based healthcare, contributing to among other things policy on patient safety, 2 prescribing, 3 4 and understanding chronic illness. 5 Equally, it offers additional insight into quantitative studies, explaining contextual factors surrounding a successful intervention or why an intervention might have ‘failed’ or ‘succeeded’ where effect sizes cannot. It is for these reasons that the MRC strongly recommends including qualitative evaluations when developing and evaluating complex interventions. 6

Critical appraisal of qualitative research

Is it necessary.

Although the importance of qualitative research to improve health services and care is now increasingly widely supported (discussed in paper 1), the role of appraising the quality of qualitative health research is still debated. 8 10 Despite a large body of literature focusing on appraisal and rigour, 9 11–15 often referred to as ‘trustworthiness’ 16 in qualitative research, there remains debate about how to —and even whether to—critically appraise qualitative research. 8–10 17–19 However, if we are to make a case for qualitative research as integral to evidence-based healthcare, then any argument to omit a crucial element of evidence-based practice is difficult to justify. That being said, simply applying the standards of rigour used to appraise studies based on the positivist paradigm (Positivism depends on quantifiable observations to test hypotheses and assumes that the researcher is independent of the study. Research situated within a positivist paradigm isbased purely on facts and consider the world to be external and objective and is concerned with validity, reliability and generalisability as measures of rigour.) would be misplaced given the different epistemological underpinnings of the two types of data.

Given its scope and its place within health research, the robust and systematic appraisal of qualitative research to assess its trustworthiness is as paramount to its implementation in clinical practice as any other type of research. It is important to appraise different qualitative studies in relation to the specific methodology used because the methodological approach is linked to the ‘outcome’ of the research (eg, theory development, phenomenological understandings and credibility of findings). Moreover, appraisal needs to go beyond merely describing the specific details of the methods used (eg, how data were collected and analysed), with additional focus needed on the overarching research design and its appropriateness in accordance with the study remit and objectives.

Poorly conducted qualitative research has been described as ‘worthless, becomes fiction and loses its utility’. 20 However, without a deep understanding of concepts of quality in qualitative research or at least an appropriate means to assess its quality, good qualitative research also risks being dismissed, particularly in the context of evidence-based healthcare where end users may not be well versed in this paradigm.

How is appraisal currently performed?

Appraising the quality of qualitative research is not a new concept—there are a number of published appraisal tools, frameworks and checklists in existence. 21–23  An important and often overlooked point is the confusion between tools designed for appraising methodological quality and reporting guidelines designed to assess the quality of methods reporting. An example is the Consolidate Criteria for Reporting Qualitative Research (COREQ) 24 checklist, which was designed to provide standards for authors when reporting qualitative research but is often mistaken for a methods appraisal tool. 10

Broadly speaking there are two types of critical appraisal approaches for qualitative research: checklists and frameworks. Checklists have often been criticised for confusing quality in qualitative research with ‘technical fixes’ 21 25 , resulting in the erroneous prioritisation of particular aspects of methodological processes over others (eg, multiple coding and triangulation). It could be argued that a checklist approach adopts the positivist paradigm, where the focus is on objectively assessing ‘quality’ where the assumptions is that the researcher is independent of the research conducted. This may result in the application of quantitative understandings of bias in order to judge aspects of recruitment, sampling, data collection and analysis in qualitative research papers. One of the most widely used appraisal tools is the Critical Appraisal Skills Programme (CASP) 26 and along with the JBI QARI (Joanna Briggs Institute Qualitative Assessment and Assessment Instrument) 27 presents examples which tend to mimic the quantitative approach to appraisal. The CASP qualitative tool follows that of other CASP appraisal tools for quantitative research designs developed in the 1990s. The similarities are therefore unsurprising given the status of qualitative research at that time.

Frameworks focus on the overarching concepts of quality in qualitative research, including transparency, reflexivity, dependability and transferability (see box 1 ). 11–13 15 16 20 28 However, unless the reader is familiar with these concepts—their meaning and impact, and how to interpret them—they will have difficulty applying them when critically appraising a paper.

The main issue concerning currently available checklist and framework appraisal methods is that they take a broad brush approach to ‘qualitative’ research as whole, with few, if any, sufficiently differentiating between the different methodological approaches (eg, Grounded Theory, Interpretative Phenomenology, Discourse Analysis) nor different methods of data collection (interviewing, focus groups and observations). In this sense, it is akin to taking the entire field of ‘quantitative’ study designs and applying a single method or tool for their quality appraisal. In the case of qualitative research, checklists, therefore, offer only a blunt and arguably ineffective tool and potentially promote an incomplete understanding of good ‘quality’ in qualitative research. Likewise, current framework methods do not take into account how concepts differ in their application across the variety of qualitative approaches and, like checklists, they also do not differentiate between different qualitative methodologies.

On the need for specific appraisal tools

Current approaches to the appraisal of the methodological rigour of the differing types of qualitative research converge towards checklists or frameworks. More importantly, the current tools do not explicitly acknowledge the prejudices that may be present in the different types of qualitative research.

Concepts of rigour or trustworthiness within qualitative research 31

Transferability: the extent to which the presented study allows readers to make connections between the study’s data and wider community settings, ie, transfer conceptual findings to other contexts.

Credibility: extent to which a research account is believable and appropriate, particularly in relation to the stories told by participants and the interpretations made by the researcher.

Reflexivity: refers to the researchers’ engagement of continuous examination and explanation of how they have influenced a research project from choosing a research question to sampling, data collection, analysis and interpretation of data.

Transparency: making explicit the whole research process from sampling strategies, data collection to analysis. The rationale for decisions made is as important as the decisions themselves.

However, we often talk about these concepts in general terms, and it might be helpful to give some explicit examples of how the ‘technical processes’ affect these, for example, partialities related to:

Selection: recruiting participants via gatekeepers, such as healthcare professionals or clinicians, who may select them based on whether they believe them to be ‘good’ participants for interviews/focus groups.

Data collection: poor interview guide with closed questions which encourage yes/no answers and/leading questions.

Reflexivity and transparency: where researchers may focus their analysis on preconceived ideas rather than ground their analysis in the data and do not reflect on the impact of this in a transparent way.

The lack of tailored, method-specific appraisal tools has potentially contributed to the poor uptake and use of qualitative research for informing evidence-based decision making. To improve this situation, we propose the need for more robust quality appraisal tools that explicitly encompass both the core design aspects of all qualitative research (sampling/data collection/analysis) but also considered the specific partialities that can be presented with different methodological approaches. Such tools might draw on the strengths of current frameworks and checklists while providing users with sufficient understanding of concepts of rigour in relation to the different types of qualitative methods. We provide an outline of such tools in the third and final paper in this series.

As qualitative research becomes ever more embedded in health science research, and in order for that research to have better impact on healthcare decisions, we need to rethink critical appraisal and develop tools that allow differentiated evaluations of the myriad of qualitative methodological approaches rather than continuing to treat qualitative research as a single unified approach.

• Williams V ,
• Boylan AM ,
• Lingard L ,
• Orser B , et al
• Brawn R , et al
• Van Royen P ,
• Vermeire E , et al
• Barker M , et al
• McGannon KR
• Dixon-Woods M ,
• Agarwal S , et al
• Greenhalgh T ,
• Dennison L ,
• Morrison L ,
• Conway G , et al
• Barrett M ,
• Mayan M , et al
• Lockwood C ,
• Santiago-Delefosse M ,
• Bruchez C , et al
• Sainsbury P ,
• ↵ CASP (Critical Appraisal Skills Programme). date unknown . http://www.phru.nhs.uk/Pages/PHD/CASP.htm .
• ↵ The Joanna Briggs Institute . JBI QARI Critical appraisal checklist for interpretive & critical research . Adelaide : The Joanna Briggs Institute , 2014 .
• Stephens J ,

Contributors VW and DN: conceived the idea for this article. VW: wrote the first draft. AMB and DN: contributed to the final draft. All authors approve the submitted article.

Competing interests None declared.

Provenance and peer review Not commissioned; externally peer reviewed.

Correction notice This article has been updated since its original publication to include a new reference (reference 1.)

Read the full text or download the PDF:

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

Critical Appraisal Toolkit (CAT) for assessing multiple types of evidence

• Author information
• Article notes
• Copyright and License information

Correspondence: [email protected]

Contributor: Jennifer Kruse, Public Health Agency of Canada – Conceptualization and project administration

Series information

Scientific writing

Collection date 2017 Sep 7.

Healthcare professionals are often expected to critically appraise research evidence in order to make recommendations for practice and policy development. Here we describe the Critical Appraisal Toolkit (CAT) currently used by the Public Health Agency of Canada. The CAT consists of: algorithms to identify the type of study design, three separate tools (for appraisal of analytic studies, descriptive studies and literature reviews), additional tools to support the appraisal process, and guidance for summarizing evidence and drawing conclusions about a body of evidence. Although the toolkit was created to assist in the development of national guidelines related to infection prevention and control, clinicians, policy makers and students can use it to guide appraisal of any health-related quantitative research. Participants in a pilot test completed a total of 101 critical appraisals and found that the CAT was user-friendly and helpful in the process of critical appraisal. Feedback from participants of the pilot test of the CAT informed further revisions prior to its release. The CAT adds to the arsenal of available tools and can be especially useful when the best available evidence comes from non-clinical trials and/or studies with weak designs, where other tools may not be easily applied.

Introduction

Healthcare professionals, researchers and policy makers are often involved in the development of public health policies or guidelines. The most valuable guidelines provide a basis for evidence-based practice with recommendations informed by current, high quality, peer-reviewed scientific evidence. To develop such guidelines, the available evidence needs to be critically appraised so that recommendations are based on the "best" evidence. The ability to critically appraise research is, therefore, an essential skill for health professionals serving on policy or guideline development working groups.

Our experience with working groups developing infection prevention and control guidelines was that the review of relevant evidence went smoothly while the critical appraisal of the evidence posed multiple challenges. Three main issues were identified. First, although working group members had strong expertise in infection prevention and control or other areas relevant to the guideline topic, they had varying levels of expertise in research methods and critical appraisal. Second, the critical appraisal tools in use at that time focused largely on analytic studies (such as clinical trials), and lacked definitions of key terms and explanations of the criteria used in the studies. As a result, the use of these tools by working group members did not result in a consistent way of appraising analytic studies nor did the tools provide a means of assessing descriptive studies and literature reviews. Third, working group members wanted guidance on how to progress from assessing individual studies to summarizing and assessing a body of evidence.

To address these issues, a review of existing critical appraisal tools was conducted. We found that the majority of existing tools were design-specific, with considerable variability in intent, criteria appraised and construction of the tools. A systematic review reported that fewer than half of existing tools had guidelines for use of the tool and interpretation of the items ( 1 ). The well-known Grading of Recommendations Assessment, Development and Evaluation (GRADE) rating-of-evidence system and the Cochrane tools for assessing risk of bias were considered for use ( 2 ), ( 3 ). At that time, the guidelines for using these tools were limited, and the tools were focused primarily on randomized controlled trials (RCTs) and non-randomized controlled trials. For feasibility and ethical reasons, clinical trials are rarely available for many common infection prevention and control issues ( 4 ), ( 5 ). For example, there are no intervention studies assessing which practice restrictions, if any, should be placed on healthcare workers who are infected with a blood-borne pathogen. Working group members were concerned that if they used GRADE, all evidence would be rated as very low or as low quality or certainty, and recommendations based on this evidence may be interpreted as unconvincing, even if they were based on the best or only available evidence.

The team decided to develop its own critical appraisal toolkit. So a small working group was convened, led by an epidemiologist with expertise in research, methodology and critical appraisal, with the goal of developing tools to critically appraise studies informing infection prevention and control recommendations. This article provides an overview of the Critical Appraisal Toolkit (CAT). The full document, entitled Infection Prevention and Control Guidelines Critical Appraisal Tool Kit is available online ( 6 ).

Following a review of existing critical appraisal tools, studies informing infection prevention and control guidelines that were in development were reviewed to identify the types of studies that would need to be appraised using the CAT. A preliminary draft of the CAT was used by various guideline development working groups and iterative revisions were made over a two year period. A pilot test of the CAT was then conducted which led to the final version ( 6 ).

The toolkit is set up to guide reviewers through three major phases in the critical appraisal of a body of evidence: appraisal of individual studies; summarizing the results of the appraisals; and appraisal of the body of evidence.

Tools for critically appraising individual studies

The first step in the critical appraisal of an individual study is to identify the study design; this can be surprisingly problematic, since many published research studies are complex. An algorithm was developed to help identify whether a study was an analytic study, a descriptive study or a literature review (see text box for definitions). It is critical to establish the design of the study first, as the criteria for assessment differs depending on the type of study.

Definitions of the types of studies that can be analyzed with the Critical Appraisal Toolkit*

Analytic study: A study designed to identify or measure effects of specific exposures, interventions or risk factors. This design employs the use of an appropriate comparison group to test epidemiologic hypotheses, thus attempting to identify associations or causal relationships.

Descriptive study: A study that describes characteristics of a condition in relation to particular factors or exposure of interest. This design often provides the first important clues about possible determinants of disease and is useful for the formulation of hypotheses that can be subsequently tested using an analytic design.

Literature review: A study that analyzes critical points of a published body of knowledge. This is done through summary, classification and comparison of prior studies. With the exception of meta-analyses, which statistically re-analyze pooled data from several studies, these studies are secondary sources and do not report any new or experimental work.

* Public Health Agency of Canada. Infection Prevention and Control Guidelines Critical Appraisal Tool Kit ( 6 )

Separate algorithms were developed for analytic studies, descriptive studies and literature reviews to help reviewers identify specific designs within those categories. The algorithm below, for example, helps reviewers determine which study design was used within the analytic study category ( Figure 1 ). It is based on key decision points such as number of groups or allocation to group. The legends for the algorithms and supportive tools such as the glossary provide additional detail to further differentiate study designs, such as whether a cohort study was retrospective or prospective.

Figure 1. Algorithm for identifying the type of analytic study.

Abbreviations: CBA, controlled before-after; ITS, interrupted time series; NRCT, non-randomized controlled trial; RCT, randomized controlled trial; UCBA, uncontrolled before-after

Separate critical appraisal tools were developed for analytic studies, for descriptive studies and for literature reviews, with relevant criteria in each tool. For example, a summary of the items covered in the analytic study critical appraisal tool is shown in Table 1 . This tool is used to appraise trials, observational studies and laboratory-based experiments. A supportive tool for assessing statistical analysis was also provided that describes common statistical tests used in epidemiologic studies.

Table 1. Aspects appraised in analytic study critical appraisal tool.

The descriptive study critical appraisal tool assesses different aspects of sampling, data collection, statistical analysis, and ethical conduct. It is used to appraise cross-sectional studies, outbreak investigations, case series and case reports.

The literature review critical appraisal tool assesses the methodology, results and applicability of narrative reviews, systematic reviews and meta-analyses.

After appraisal of individual items in each type of study, each critical appraisal tool also contains instructions for drawing a conclusion about the overall quality of the evidence from a study, based on the per-item appraisal. Quality is rated as high, medium or low. While a RCT is a strong study design and a survey is a weak design, it is possible to have a poor quality RCT or a high quality survey. As a result, the quality of evidence from a study is distinguished from the strength of a study design when assessing the quality of the overall body of evidence. A definition of some terms used to evaluate evidence in the CAT is shown in Table 2 .

Table 2. Definition of terms used to evaluate evidence.

* Considered strong design if there are at least two control groups and two intervention groups. Considered moderate design if there is only one control and one intervention group

Tools for summarizing the evidence

The second phase in the critical appraisal process involves summarizing the results of the critical appraisal of individual studies. Reviewers are instructed to complete a template evidence summary table, with key details about each study and its ratings. Studies are listed in descending order of strength in the table. The table simplifies looking across all studies that make up the body of evidence informing a recommendation and allows for easy comparison of participants, sample size, methods, interventions, magnitude and consistency of results, outcome measures and individual study quality as determined by the critical appraisal. These evidence summary tables are reviewed by the working group to determine the rating for the quality of the overall body of evidence and to facilitate development of recommendations based on evidence.

Rating the quality of the overall body of evidence

The third phase in the critical appraisal process is rating the quality of the overall body of evidence. The overall rating depends on the five items summarized in Table 2 : strength of study designs, quality of studies, number of studies, consistency of results and directness of the evidence. The various combinations of these factors lead to an overall rating of the strength of the body of evidence as strong, moderate or weak as summarized in Table 3 .

Table 3. Criteria for rating evidence on which recommendations are based.

A unique aspect of this toolkit is that recommendations are not graded but are formulated based on the graded body of evidence. Actions are either recommended or not recommended; it is the strength of the available evidence that varies, not the strength of the recommendation. The toolkit does highlight, however, the need to re-evaluate new evidence as it becomes available especially when recommendations are based on weak evidence.

Pilot test of the CAT

Of 34 individuals who indicated an interest in completing the pilot test, 17 completed it. Multiple peer-reviewed studies were selected representing analytic studies, descriptive studies and literature reviews. The same studies were assigned to participants with similar content expertise. Each participant was asked to appraise three analytic studies, two descriptive studies and one literature review, using the appropriate critical appraisal tool as identified by the participant. For each study appraised, one critical appraisal tool and the associated tool-specific feedback form were completed. Each participant also completed a single general feedback form. A total of 101 of 102 critical appraisals were conducted and returned, with 81 tool-specific feedback forms and 14 general feedback forms returned.

The majority of participants (>85%) found the flow of each tool was logical and the length acceptable but noted they still had difficulty identifying the study designs ( Table 4 ).

Table 4. Pilot test feedback on user friendliness.

* Number of tool-specific forms returned for total number of critical appraisals conducted

The vast majority of the feedback forms (86–93%) indicated that the different tools facilitated the critical appraisal process. In the assessment of consistency, however, only four of ten analytic studies appraised (40%), had complete agreement on the rating of overall study quality by participants, the other six studies had differences noted as mismatches. Four of the six studies with mismatches were observational studies. The differences were minor. None of the mismatches included a study that was rated as both high and low quality by different participants. Based on the comments provided by participants, most mismatches could likely have been resolved through discussion with peers. Mismatched ratings were not an issue for the descriptive studies and literature reviews. In summary, the pilot test provided useful feedback on different aspects of the toolkit. Revisions were made to address the issues identified from the pilot test and thus strengthen the CAT.

The Infection Prevention and Control Guidelines Critical Appraisal Tool Kit was developed in response to the needs of infection control professionals reviewing literature that generally did not include clinical trial evidence. The toolkit was designed to meet the identified needs for training in critical appraisal with extensive instructions and dictionaries, and tools applicable to all three types of studies (analytic studies, descriptive studies and literature reviews). The toolkit provided a method to progress from assessing individual studies to summarizing and assessing the strength of a body of evidence and assigning a grade. Recommendations are then developed based on the graded body of evidence. This grading system has been used by the Public Health Agency of Canada in the development of recent infection prevention and control guidelines ( 5 ), ( 7 ). The toolkit has also been used for conducting critical appraisal for other purposes, such as addressing a practice problem and serving as an educational tool ( 8 ), ( 9 ).

The CAT has a number of strengths. It is applicable to a wide variety of study designs. The criteria that are assessed allow for a comprehensive appraisal of individual studies and facilitates critical appraisal of a body of evidence. The dictionaries provide reviewers with a common language and criteria for discussion and decision making.

The CAT also has a number of limitations. The tools do not address all study designs (e.g., modelling studies) and the toolkit provides limited information on types of bias. Like the majority of critical appraisal tools ( 10 ), ( 11 ), these tools have not been tested for validity and reliability. Nonetheless, the criteria assessed are those indicated as important in textbooks and in the literature ( 12 ), ( 13 ). The grading scale used in this toolkit does not allow for comparison of evidence grading across organizations or internationally, but most reviewers do not need such comparability. It is more important that strong evidence be rated higher than weak evidence, and that reviewers provide rationales for their conclusions; the toolkit enables them to do so.

Overall, the pilot test reinforced that the CAT can help with critical appraisal training and can increase comfort levels for those with limited experience. Further evaluation of the toolkit could assess the effectiveness of revisions made and test its validity and reliability.

A frequent question regarding this toolkit is how it differs from GRADE as both distinguish stronger evidence from weaker evidence and use similar concepts and terminology. The main differences between GRADE and the CAT are presented in Table 5 . Key differences include the focus of the CAT on rating the quality of individual studies, and the detailed instructions and supporting tools that assist those with limited experience in critical appraisal. When clinical trials and well controlled intervention studies are or become available, GRADE and related tools from Cochrane would be more appropriate ( 2 ), ( 3 ). When descriptive studies are all that is available, the CAT is very useful.

Table 5. Comparison of features of the Critical Appraisal Toolkit (CAT) and GRADE.

Abbreviation: GRADE, Grading of Recommendations Assessment, Development and Evaluation

The Infection Prevention and Control Guidelines Critical Appraisal Tool Kit was developed in response to needs for training in critical appraisal, assessing evidence from a wide variety of research designs, and a method for going from assessing individual studies to characterizing the strength of a body of evidence. Clinician researchers, policy makers and students can use these tools for critical appraisal of studies whether they are trying to develop policies, find a potential solution to a practice problem or critique an article for a journal club. The toolkit adds to the arsenal of critical appraisal tools currently available and is especially useful in assessing evidence from a wide variety of research designs.

Authors’ Statement

DM – Conceptualization, methodology, investigation, data collection and curation and writing – original draft, review and editing

TO – Conceptualization, methodology, investigation, data collection and curation and writing – original draft, review and editing

KD – Conceptualization, review and editing, supervision and project administration

Acknowledgements

We thank the Infection Prevention and Control Expert Working Group of the Public Health Agency of Canada for feedback on the development of the toolkit, Lisa Marie Wasmund for data entry of the pilot test results, Katherine Defalco for review of data and cross-editing of content and technical terminology for the French version of the toolkit, Laurie O’Neil for review and feedback on early versions of the toolkit, Frédéric Bergeron for technical support with the algorithms in the toolkit and the Centre for Communicable Diseases and Infection Control of the Public Health Agency of Canada for review, feedback and ongoing use of the toolkit. We thank Dr. Patricia Huston, Canada Communicable Disease Report Editor-in-Chief, for a thorough review and constructive feedback on the draft manuscript.

Conflict of interest: None.

Funding: This work was supported by the Public Health Agency of Canada.

• 1. Katrak P, Bialocerkowski AE, Massy-Westropp N, Kumar S, Grimmer KA. A systematic review of the content of critical appraisal tools. BMC Med Res Methodol 2004. Sep;4:22. 10.1186/1471-2288-4-22 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 2. GRADE Working Group. Criteria for applying or using GRADE. www.gradeworkinggroup.org [Accessed July 25, 2017].
• 3. Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration, 2011. http://handbook.cochrane.org
• 4. Khan AR, Khan S, Zimmerman V, Baddour LM, Tleyjeh IM. Quality and strength of evidence of the Infectious Diseases Society of America clinical practice guidelines. Clin Infect Dis 2010. Nov;51(10):1147–56. 10.1086/656735 [ DOI ] [ PubMed ] [ Google Scholar ]
• 5. Public Health Agency of Canada. Routine practices and additional precautions for preventing the transmission of infection in healthcare settings. http://www.phac-aspc.gc.ca/nois-sinp/guide/summary-sommaire/tihs-tims-eng.php [Accessed December 4, 2015].
• 6. Public Health Agency of Canada. Infection Prevention and Control Guidelines Critical Appraisal Tool Kit. http://publications.gc.ca/collections/collection_2014/aspc-phac/HP40-119-2014-eng.pdf [Accessed December 4, 2015].
• 7. Public Health Agency of Canada. Hand hygiene practices in healthcare settings. https://www.canada.ca/en/public-health/services/infectious-diseases/nosocomial-occupational-infections/hand-hygiene-practices-healthcare-settings.html [Accessed December 4, 2015].
• 8. Ha S, Paquette D, Tarasuk J, Dodds J, Gale-Rowe M, Brooks JI et al. A systematic review of HIV testing among Canadian populations. Can J Public Health 2014. Jan;105(1):e53–62. 10.17269/cjph.105.4128 [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 9. Stevens LK, Ricketts ED, Bruneau JE. Critical appraisal through a new lens. Nurs Leadersh (Tor Ont) 2014. Jun;27(2):10–3. 10.12927/cjnl.2014.23843 [ DOI ] [ PubMed ] [ Google Scholar ]
• 10. Lohr KN. Rating the strength of scientific evidence: relevance for quality improvement programs. Int J Qual Health Care 2004. Feb;16(1):9–18. 10.1093/intqhc/mzh005 [ DOI ] [ PubMed ] [ Google Scholar ]
• 11. Crowe M, Sheppard L. A review of critical appraisal tools show they lack rigor: alternative tool structure is proposed. J Clin Epidemiol 2011. Jan;64(1):79–89. 10.1016/j.jclinepi.2010.02.008 [ DOI ] [ PubMed ] [ Google Scholar ]
• 12. Young JM, Solomon MJ. How to critically appraise an article. Nat Clin Pract Gastroenterol Hepatol 2009. Feb;6(2):82–91. 10.1038/ncpgasthep1331 [ DOI ] [ PubMed ] [ Google Scholar ]
• 13. Polit DF, Beck CT. Nursing Research: Generating and Assessing Evidence for Nursing Practice. 9 th ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2008. Chapter XX, Literature reviews: Finding and critiquing the evidence p. 94-125. [ Google Scholar ]
• View on publisher site
• PDF (348.4 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections

Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

Critical Appraisal tools

Critical appraisal worksheets to help you appraise the reliability, importance and applicability of clinical evidence.

Critical appraisal is the systematic evaluation of clinical research papers in order to establish:

• Does this study address a  clearly focused question ?
• Did the study use valid methods to address this question?
• Are the valid results of this study important?
• Are these valid, important results applicable to my patient or population?

If the answer to any of these questions is “no”, you can save yourself the trouble of reading the rest of it.

This section contains useful tools and downloads for the critical appraisal of different types of medical evidence. Example appraisal sheets are provided together with several helpful examples.

Critical Appraisal Worksheets

• Systematic Reviews  Critical Appraisal Sheet
• Diagnostics  Critical Appraisal Sheet
• Prognosis  Critical Appraisal Sheet
• Randomised Controlled Trials  (RCT) Critical Appraisal Sheet
• Critical Appraisal of Qualitative Studies  Sheet
• IPD Review  Sheet

Chinese - translated by Chung-Han Yang and Shih-Chieh Shao

• Systematic Reviews  Critical Appraisal Sheet
• Diagnostic Study  Critical Appraisal Sheet
• Prognostic Critical Appraisal Sheet
• RCT  Critical Appraisal Sheet
• IPD reviews Critical Appraisal Sheet
• Qualitative Studies Critical Appraisal Sheet 

German - translated by Johannes Pohl and Martin Sadilek

• Systematic Review  Critical Appraisal Sheet
• Diagnosis Critical Appraisal Sheet
• Prognosis Critical Appraisal Sheet
• Therapy / RCT Critical Appraisal Sheet

Lithuanian - translated by Tumas Beinortas

• Systematic review appraisal Lithuanian (PDF)
• Diagnostic accuracy appraisal Lithuanian  (PDF)
• Prognostic study appraisal Lithuanian  (PDF)
• RCT appraisal sheets Lithuanian  (PDF)

Portugese - translated by Enderson Miranda, Rachel Riera and Luis Eduardo Fontes

• Portuguese – Systematic Review Study Appraisal Worksheet
• Portuguese – Diagnostic Study Appraisal Worksheet
• Portuguese – Prognostic Study Appraisal Worksheet
• Portuguese – RCT Study Appraisal Worksheet
• Portuguese – Systematic Review Evaluation of Individual Participant Data Worksheet
• Portuguese – Qualitative Studies Evaluation Worksheet

Spanish - translated by Ana Cristina Castro

• Systematic Review  (PDF)
• Diagnosis  (PDF)
• Prognosis  Spanish Translation (PDF)
• Therapy / RCT  Spanish Translation (PDF)

Persian - translated by Ahmad Sofi Mahmudi

• Prognosis  (PDF)
• PICO  Critical Appraisal Sheet (PDF)
• PICO Critical Appraisal Sheet (MS-Word)
• Educational Prescription  Critical Appraisal Sheet (PDF)

Explanations & Examples

• Pre-test probability
• SpPin and SnNout
• Likelihood Ratios

Please enter both an email address and a password.

Account login

• Show/Hide Password Show password Hide password
• Reset Password

Need to reset your password?  Enter the email address which you used to register on this site (or your membership/contact number) and we'll email you a link to reset it. You must complete the process within 2hrs of receiving the link.

We've sent you an email.

An email has been sent to Simply follow the link provided in the email to reset your password. If you can't find the email please check your junk or spam folder and add [email protected] to your address book.

• About RCS England

• Dissecting the literature: the importance of critical appraisal

08 Dec 2017

Kirsty Morrison

This post was updated  in 2023.

Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context.

Amanda Burls, What is Critical Appraisal?

Why is critical appraisal needed?

Literature searches using databases like Medline or EMBASE often result in an overwhelming volume of results which can vary in quality. Similarly, those who browse medical literature for the purposes of CPD or in response to a clinical query will know that there are vast amounts of content available. Critical appraisal helps to reduce the burden and allow you to focus on articles that are relevant to the research question, and that can reliably support or refute its claims with high-quality evidence, or identify high-level research relevant to your practice.

Critical appraisal allows us to:

• reduce information overload by eliminating irrelevant or weak studies
• identify the most relevant papers
• distinguish evidence from opinion, assumptions, misreporting, and belief
• assess the validity of the study
• assess the usefulness and clinical applicability of the study
• recognise any potential for bias.

Critical appraisal helps to separate what is significant from what is not. One way we use critical appraisal in the Library is to prioritise the most clinically relevant content for our Current Awareness Updates .

How to critically appraise a paper

There are some general rules to help you, including a range of checklists highlighted at the end of this blog. Some key questions to consider when critically appraising a paper:

• Is the study question relevant to my field?
• Does the study add anything new to the evidence in my field?
• What type of research question is being asked? A well-developed research question usually identifies three components: the group or population of patients, the studied parameter (e.g. a therapy or clinical intervention) and outcomes of interest.
• Was the study design appropriate for the research question? You can learn more about different study types and the hierarchy of evidence here .
• Did the methodology address important potential sources of bias? Bias can be attributed to chance (e.g. random error) or to the study methods (systematic bias).
• Was the study performed according to the original protocol? Deviations from the planned protocol can affect the validity or relevance of a study, e.g. a decrease in the studied population over the course of a randomised controlled trial .
• Does the study test a stated hypothesis? Is there a clear statement of what the investigators expect the study to find which can be tested, and confirmed or refuted.
• Were the statistical analyses performed correctly? The approach to dealing with missing data, and the statistical techniques that have been applied should be specified. Original data should be presented clearly so that readers can check the statistical accuracy of the paper.
• Do the data justify the conclusions? Watch out for definite conclusions based on statistically insignificant results, generalised findings from a small sample size, and statistically significant associations being misinterpreted to imply a cause and effect.
• Are there any conflicts of interest? Who has funded the study and can we trust their objectivity? Do the authors have any potential conflicts of interest, and have these been declared?

And an important consideration for surgeons:

• Will the results help me manage my patients?

At the end of the appraisal process you should have a better appreciation of how strong the evidence is, and ultimately whether or not you should apply it to your patients.

Further resources:

• How to Read a Paper by Trisha Greenhalgh
• The Doctor’s Guide to Critical Appraisal by Narinder Kaur Gosall
• CASP checklists
• CEBM Critical Appraisal Tools
• Critical Appraisal: a checklist
• Critical Appraisal of a Journal Article (PDF)
• Introduction to...Critical appraisal of literature
• Reporting guidelines for the main study types

Kirsty Morrison, Information Specialist

Share this page:

• Library Blog

• Systematic review
• Open access
• Published: 29 October 2024

Facilitators and barriers to implementing patient-reported outcomes in clinical oncology practice: a systematic review based on the consolidated framework for implementation research

• Jianxia Lyu 1 ,
• Hao Zhang 2   na1 ,
• Hua Wang 1   na1 ,
• Xia Liu 1 ,
• Yunhua Jing 1 ,
• Li Yin 1 &
• Aiping Wang 3  

Implementation Science Communications volume  5 , Article number:  120 ( 2024 ) Cite this article

19 Accesses

Metrics details

In clinical oncology practice, patient-reported outcomes (PROs) are essential for assessing the symptom burden, quality of life, and psychological status of patients. However, there remains a gap between the use of PROs in an oncologic setting and its implementation. Furthermore, numerous reviews in PRO implementation are often based on one particular technology, setting, or health condition, making it difficult to obtain a comprehensive and coherent summary of available evidence to help plan and undertake implementation. This systematic review aims to identify and integrate enablers and barriers to PRO implementation through the comprehensive framework for implementation research (CFIR) to provide a reference for implementing patient-reported outcomes management in oncology settings.

This review strictly observed the preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines. PubMed, Web of Science, CINAHL, Embase, and PsycINFO databases were systematically searched using a three-step search strategy. The search was limited from the inception of each database to April 2024. Articles describing facilitators and barriers to implementing PROs in clinical oncology practice were included. Two researchers screened the literature independently, and the quality assessment of cross-sectional, qualitative, and mixed studies was completed using the critical appraisal tools recommended by Joanna Briggs Institute (JBI) and the mixed methods assessment tool, respectively. Basic information about the included studies and determinants affecting PRO implementation was extracted, and coding categorization of facilitators and barriers was completed based on the 48 constructs provided by the CFIR framework.

We included 30 studies from 5,649 search results, including 25 original and 5 review studies. The quality of the literature for qualitative studies was generally good, and the quality for quantitative and mixed studies was assessed as fair. We identified 52 facilitators and 50 barriers in the included literature, covering the domains used in the CFIR framework and 39 constructs, mainly including "Innovation Evidence-Base", "Innovation Complexity", "Innovation Design", "Structural Characteristics", "Compatibility", "Incentive Systems", "Access to Knowledge & Information", "Innovation Deliverers", "Innovation Recipients", and "Planning".

Conclusions

This systematic review integrated facilitators and barriers affecting PRO implementation in routine oncology clinical practice settings and categorized them through the CFIR framework. These influencing factors should be fully considered in future clinical practice to ensure the successful implementation of PROs.

Trial registration

It has been registered prospectively in PROSPERO under the registration number 42024532983.

Peer Review reports

Contribution to the literature

This study integrates and identifies specific facilitators and barriers to PRO implementation in oncology clinical practice settings based on the Consolidated Framework for Implementation Research.

This systemic review identified 52 facilitators and 50 barriers for PROs to be implemented in an oncology setting, covering all areas of the CFIR, and the main constructs involved included: "Innovation Evidence-Base," "Innovation Complexity", "Innovation Design", "Structural Characteristics", "Compatibility", "Incentive Systems", "Access to Knowledge & Information", "Innovation Deliverers", "Innovation Recipients", and "Planning".

The facilitators and barriers identified through the CFIR will inform stakeholders as they formulate intervention strategies to enhance the routine application of PRO tools in clinical oncology practice.

Introduction

Cancer is a common disease that affects public health globally. The symptoms and treatment of cancer often impose various physical and psychological burdens on patients, including anxiety, depression, fatigue, gastrointestinal reactions, pain, and financial constraint. An accurate assessment of these burdens can assist healthcare professionals (HCPs) in managing or resolving the problems experienced by patients [ 1 , 2 , 3 ]. Patient-reported outcomes (PROs) are commonly used to assess clinical outcomes. These outcomes are derived directly from patient health information, functional status, and treatment perception [ 4 , 5 ]. PROs are key indicators for improving the quality of treatment and care from the patient's perspective, thereby fully encapsulating the “patient-centered” healthcare concept [ 6 ]. Currently, PROs are widely used in Europe, the United States, Australia, and other regions for clinical effectiveness evaluation, drug review and approval, medical service quality assessment, and health technology assessment [ 7 , 8 ]. Incorporating PROs into oncology clinical practice can enhance the detection of patient clinical issues, doctor-patient communication, patient participation in clinical decision-making, patient experience, and satisfaction with medical services and lay a foundation for implementing new treatment options [ 9 , 10 ]. Additionally, previous studies have reported that effective implementation of PRO symptom monitoring and early warning in oncology survivors could significantly enhance patient physical functioning, health-related quality of life, hospital admission rates, overall survival, patient satisfaction, and cost-effectiveness [ 11 , 12 ].

However, although PRO implementation in clinical practice positively impacts the ability of HCPs to identify and manage patient health outcomes, there are still barriers to their translation into a clinical routine model at different levels [ 13 , 14 , 15 ]. At the HCP level, these include time constraints [ 16 ], lack of experience and training [ 17 , 18 ], and skepticism about PROs [ 19 ]. At the patient level, these include disease burden [ 20 , 21 ], unfamiliarity with the questionnaire [ 18 ], difficulty in accessing the questionnaire [ 22 ], and language barriers [ 16 , 23 ]. For the implementation environment, the barriers include a lack of appropriate workflow [ 24 ], embedded electronic health records (EHR) [ 25 ], and material resources and information technology support [ 16 , 26 ]. Previous studies have identified facilitators and barriers to PRO implementation. However, some of these studies lacked an appropriate framework, resulting in a limited understanding of the influencing factors. Therefore, the impact of various factors on PRO implementation in clinical settings may have been underestimated. Relying on a comprehensive and credible framework to systematically identify, analyze, and interpret the influencing factors in PRO implementation is essential to further improving the use of PROs in oncology clinical practice.

Consequently, this systematic review employed implementation science (IS) to integrate facilitators and barriers to PRO implementation. According to the National Institution of Health, IS is a discipline of research methods designed to facilitate the adoption and integration of evidence-based practices, interventions, and policies into routine healthcare and public health settings to improve the impact on population health [ 27 ]. IS identifies facilitators and barriers to implementing an intervention program and develops appropriate coping strategies based on these factors, which can be achieved to maximize assurance that an intervention program will be effectively implemented [ 28 , 29 ]. Different theoretical frameworks are used in implementation research. The consolidated framework for implementation research (CFIR) constructed by Damschroder in 2009 is the most commonly used framework in implementation research [ 30 , 31 ]. CFIR is a publicly available tool encompassing multiple areas and key concepts related to implementation. It can be adapted to different implementation contexts and phases [ 30 ], providing researchers with a structured approach to identify facilitators and barriers to implementation and progress, potential, and actual impact on effectiveness during implementation [ 32 , 33 ].

Accordingly, this systematic review aimed to identify, assess, and analyze facilitators and barriers to PRO implementation in oncology clinical practice by various stakeholders and integrate the available evidence regarding the implementation of PRO tools according to the CFIR to help future studies develop relevant responses to enhance the effectiveness of PRO tools in oncology clinical practice. The main questions to be answered are in the following areas:

What are the facilitators and barriers to implementing PROs in clinical oncology practice?

How do these facilitators and barriers, guided by the IS, affect PRO implementation in clinical oncology practice?

What areas of the CFIR framework can these facilitators and barriers be mapped to?

This systematic review followed the preferred reporting items for systematic reviews and meta-analyses (PRISMA) guidelines and was registered prospectively in PROSPERO (42024532983).

Search strategy

We searched five databases: PubMed, Web of Science, CINAHL, Embase, and PsycINFO. The search was limited from the database inception to April 2024. A three-step search strategy was strictly followed to retrieve relevant articles systematically. First, search terminologies were identified through an initial search of PubMed and Web of Science and an evaluation of the titles and abstracts of relevant articles. Second, each database was searched comprehensively using a search strategy that combined medical subject heading terms with free words to identify the search terminologies. Finally, the bibliography in the included studies was searched to screen for relevant articles. The search terms were as follows: Implement* AND (barrier* OR facilitat* OR determinant* OR factor* OR affect* OR influenc* OR enhanc* OR improv* OR support* OR gap) AND (‘Patient Reported Outcome Measure’ OR ‘Patient Reported Outcome*’ OR PROM OR PRO) AND (cancer* OR tumor* OR neoplasm* OR carcinoma* OR oncolog* OR malignan*). Supplementary File 1 presents detailed search information for each database.

Inclusion and exclusion criteria

We included original and secondary research published in peer-reviewed journals. These studies were designed to describe facilitators and barriers that affect the implementation of any PRO tools in routine oncology clinical practice settings by various stakeholders, including patients, HCPs, and other health workers. However, this review excluded ineligible researchers with the following themes: 1) reporting the development and feasibility validation of PRO tools; 2) PRO tools completed by an individual other than the patient; 3) unavailable full text or incomplete data; 4) duplicate publications; and 5) non-Chinese OR non-English literature.

Quality assessment

For qualitative and quantitative studies, the Joanna Briggs Institute Critical Appraisal tools were used to evaluate the quality of the literature. In the qualitative research section, the main consideration was whether the methodology used was consistent with the philosophical research questions, data collection, and analysis [ 34 ]. In the quantitative cross-sectional study section, the impact on the overall quality of the study was considered primarily based on the sample source, measurement credibility, and the effective control of confounding factors. For mixed-method research, the quality assessment tool used was the mixed methods appraisal tool (MMAT) [ 35 ]. The tool consists of a two-part evaluation question: 1) two screening questions to confirm whether an article is eligible for the MMAT; 2) scoring based on the type of study by selecting the evaluation criteria that correspond to it, with each category consisting of five evaluation criteria. For the included secondary studies, this study did not qualitatively assess the type of literature because of the lack of relevant quality assessment tools.

Two reviewers, LX and JYH, independently used MMAT to assess the quality of the included literature. Disagreements were resolved after discussion. A third reviewer, YL, performed a double assessment of the quality of the literature to ensure credibility. Studies that met the above inclusion criteria were included regardless of the quality of the study because the focus of the literature quality assessment was to identify the strengths and limitations of the included literature based on the study design and analysis process, which facilitated a rational interpretation of the results in this study.

Data extraction and analysis

The information needed for this systematic review was identified after LX and JYH reviewed the entire test. The research information was independently extracted into a pre-prepared table. Four main categories of information were involved: 1) basic information about the included studies: names of the authors, year of publication, country, purpose of the study, study design, participants, sample size, study setting, collection methods, and methods of analysis; 2) the form, characteristics, and stages of implementation of the PRO tools; 3) stakeholders implementing or receiving PROs; 4) definition and description of factors influencing PRO implementation in the original study.

The facilitators and barriers to PRO implementation reported in the study were imported into the NVivo® and categorized according to the CFIR during the data analysis phase. The CFIR framework is primarily used to predict or explain barriers and facilitators to implementation effectiveness. The revised version of the CFIR framework addresses the five domains of innovation: outer setting, inner setting, individuals, and implementation process, and comprises 48 constructs. ZH and WH independently coded the data based on the updated CFIR framework guidelines. Disagreements were resolved through discussion with a third researcher.

Study selection

We identified 6,549 articles from 5 electronic databases, and 6 additional studies were identified from the reference lists of these articles. After removing duplicates, 4,419 studies unrelated to the research topic were excluded, and 79 studies were reviewed in full text. We excluded 37 studies during the full-text review process because of the following: unoncological clinical practice ( n  = 6); protocol ( n  = 4); unreported determining factor of PROs ( n  = 7); inconsistency of research topics ( n  = 18); non-target language ( n  = 1); non-PRO instruments ( n  = 1). We included 25 primary studies [ 18 , 19 , 20 , 21 , 22 , 24 , 25 , 26 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 ] and 5 secondary studies [ 53 , 54 , 55 , 56 , 57 ]. Figure  1 depicts the flowchart of the literature screening process for this review.

PRISMA flowchart of the study selection process

Study characteristics

The 25 original studies comprised a qualitative study ( n  = 13), a quantitative cross-sectional study ( n  = 7), and a mixed-method study design ( n  = 5). The original studies were most frequently conducted in the United States ( n  = 8), Australia ( n  = 6), the Netherlands ( n  = 3), Germany ( n  = 3), and one study each in Canada, China, Italy, Norway, and the United Kingdom. Participants in the included studies were primarily cancer survivors and HCPs in routine cancer clinical practice settings. Some studies included hospital management, information technician professionals, and other stakeholders as participants. Based on cancer types, most studies focused on multiple cancer types, while some studies focused on breast cancer [ 20 , 21 , 37 , 42 , 52 ], head and neck cancer [ 26 , 36 ], and gastrointestinal tumors [ 25 , 38 ]. Two studies [ 48 , 49 ] were conducted on pediatric oncology. Based on study settings, most of the studies were conducted in a single-center setting, while some studies [ 22 , 23 , 24 , 25 , 36 , 37 , 40 , 42 , 43 , 45 ] involved multiple cancer and academic medical centers. Convenience and purposive sampling were often used as research methods in qualitative studies to select interviewees. Semi-structured interviews or focus groups were the primary interview methods used with a sample size of 11–56 interviewees, and thematic and content analyses were often used to analyze qualitative data. In the cross-sectional quantitative studies, perceptions of the stakeholders and determinants of PRO implementation were collected primarily through questionnaires, with sample sizes ranging from 129 to 727. In the mixed studies, mixed methods of qualitative studies combined with cross-sectional quantitative studies were used by most studies. Some studies limited the type of PRO tools used in the oncological clinical practice setting to single or multiple tools, while most studies did not strictly limit their choice. Standard tools included European Organization for Research and Treatment of Cancer Quality of Life Questionnaire—Core 30(EORTC QLQ-C30), Common Terminology Criteria for Adverse Events (CACTE), and BREAST-Questionnaire (BREAST-Q). Over half of the studies collected PROs in an electronic format. Table 1 presents basic information on the included studies.

Table 2 lists the basic information of the five included secondary studies [ 53 , 54 , 55 , 56 , 57 ], which included one review and two scoping and systematic reviews.

Quality assessment results

Supplementary File 2 presents the results of the quality assessment.

Qualitative studies

The assessment of the included 13 qualitative studies was good. All studies clarified the purpose of using qualitative research methods, adopted appropriate methods of data collection and analysis, and highlighted the typicality of the subjects and their perspectives in reporting the results. However, most studies did not elaborate on the situation of the researcher and interactions with the study, including their cultural background and values.

Analytical cross-sectional study

The quality of the seven quantitative studies was rated fair. These studies comprehensively described the participant populations, study settings, and analysis methods. However, unlike other quantitative cross-sectional studies, these studies primarily collected facilitators and barriers to implementing PROs, making it challenging to determine confounders, a significant factor that reduces the quality of the literature.

Mixed-method studies

The quality assessment of the five mixed studies was rated fair, primarily because of the lack of detailed descriptions of the research methodology, particularly the quantitative research methodology. Besides, the lack of detailed reporting on the reasons for using mixed studies was an essential factor in this result.

The determinants that affect the implementation of PROs

This systematic review synthesized the facilitators and barriers to implementing PROs from the included studies and categorized them according to the CFIR. Supplementary File 3 presents the detailed results. Overall, 52 facilitators and 50 barriers were identified, which encompassed the domains used in the CFIR framework and covered 39 constructs (39/48; 81.25%), primarily including "Innovation Evidence-Base", "Innovation Complexity", "Innovation Design", "Structural Characteristics", "Compatibility", "Incentive Systems", "Access to Knowledge & Information", "Innovation Deliverers", "Innovation Recipients", and "Planning". We reviewed the essential facilitators and barriers in all included studies according to the construct of the CFIR involved.

Innovation domain

Innovation evidence-base: In this construct, studies stated that a significant contributing factor is the potential for PRO tools to make a practical difference in clinical practice [ 47 ]. For instance, they could assist HCPs in monitoring and managing clinical problems [ 39 ] and improve patient care [ 18 , 42 ]. Conversely, if they are unsure about the utility and effectiveness of PRO tools, this will limit their routine use [ 22 , 26 ].

Innovation relative advantage: This construct indicated that applying PROs in routine oncology clinical practice can benefit various stakeholders, facilitating the implementation and dissemination of PRO tools. For instance, the PRO implementation can reflect and meet patient needs in oncology diagnosis, treatment, rehabilitation, and end-of-life care [ 18 , 42 ]. Furthermore, PROs could assist HCPs in accurately identifying existing or potential clinical problems in patients to improve clinical work efficiency [ 47 ]. However, the routine application of PRO tools is impeded by significant barriers, including the fear of associated legal liabilities and an increased workload, which can result in infringing on the rights of the stakeholders [ 39 , 51 ].

Innovation adaptability: Flexibility in the design of tools facilitates PRO implementation. The design of PRO tools needs to be flexible and changeable according to different clinical settings and patient characteristics [ 41 , 47 ], which can enhance the acceptability of these tools among HCPs and patients. Conversely, the diffusion of a tool would be diminished if it were designed without considering the individual variability of patients [ 42 ].

Innovation complexity and design: The simplicity of the design of the PRO tool and the appropriate difficulty of completion are facilitating factors that promote the routine adoption of the tool. Previous studies reported a high degree of individual variability in the ability of patients to complete the PRO tool [ 22 , 47 , 51 ], and assisting patients in achieving the clinical outcome report is a time-consuming process [ 19 ], which will limit the use of PROs. The complex design and lack of functionality of the PRO system was another significant barrier emphasized by the construct. This is primarily evident in the following: first, the complex design limits the convenience of HCPs and patients in accessing and using the PRO tools [ 42 , 43 ]; second, there is a lack of visualization tools to present the results of the PRO assessment in a quick and precise manner [ 38 , 41 , 49 ].

In the innovation domain, the feasibility of piloting PRO tools on a small scale and controlling PRO implementation costs within acceptable limits are contributing factors [ 46 ] that can effectively facilitate the application of PROs [ 41 ].

Outer setting domain

Partnerships and connections: This construct emphasizes establishing and strengthening partnerships with external organizations. Previous studies reported that collaborative relationships with other organizations, thereby facilitating the dissemination of lessons learned, are one of the facilitating factors for PRO implementation [ 41 , 42 ].

Policies and laws: PRO implementation in compliance with the guidelines and local legal norms is a significant contributing factor emphasized in this area [ 42 , 43 ]. The effective operation of PRO projects will be facilitated by acquiring appropriate resource allocation in this manner [ 42 ]. Conversely, in the absence of policy support, it will be challenging to implement PROs in the current environment [ 41 , 43 ].

Financing: Obtaining external financial support is essential to ensuring the stable operation of PRO programs. For instance, funding for PRO implementation can be increased by lobbying governments and converting PROs into commercialized models. [ 42 ]. However, a previous study reported that there is a lack of specialized funding models for PRO symptom monitoring [ 42 ].

In the outer setting domain, obtaining the support of external organizations [ 41 , 53 ] and the use of PROs as performance evaluation indicators [ 47 ] are factors contributing to the successful implementation of PROs. Barrier factors include the lack of support from local experts [ 24 , 26 ] and limited working hours for HCPs [ 21 ].

Inner setting domain

Structural characteristics: This construct emphasizes that physical, information technology, and work infrastructure are the foundational configuration requirements that impact PRO implementation in routine oncology practice. Configuring portable electronic devices, embedding PROs in the EHR, and planning a rational workflow facilitate PRO implementation in the clinical practice setting. For example, providing several tablets for PRO data collection can increase patient response rates [ 19 , 47 ]. Second, multiple studies have reported that integrating PROs into EHR could facilitate physicians to collect clinical data more conveniently during outpatient or follow-up visits and present these data in an easy-to-understand manner, thus increasing PRO implementation [ 16 , 25 , 41 ]. Furthermore, rational planning of work procedures and clear task assignments can avoid unnecessary work duplication [ 51 ]. However, the lack of equipment support for data collection [ 18 , 22 , 26 ], the absence of an information system for collecting [ 21 , 24 ] and managing PRO data, and the irrational arrangement of work procedures [ 16 , 17 , 49 ] will hinder the routine application of PROs.

Relational connections and communication: This construct focuses on enhancing coordination and information sharing among internal multidisciplinary departments [ 25 ]. Strengthening internal communication is essential in PRO implementation, particularly in the following two areas. First, communication and coordination among HCPs can improve clinical work efficiency and prevent patients from receiving multiple similar or identical surveys in a short duration [ 20 , 41 , 58 ]. Second, it is essential to improve communication between HCPs and patients so that timely feedback and opinions can be obtained from patients to facilitate the continuous improvement of the quality of the application of PRO tools [ 16 , 25 ].

Compatibility: This construct primarily emphasizes that PROs should be aligned with current clinical practice activities and workflows to avoid overlap in different systems [ 25 , 41 ]. The alignment of PRO collection with other clinical practice activities, workflows, and EHR is one of the key factors that facilitate the routine use of PRO [ 22 , 58 ]. However, previous studies reported that different clinical settings have unique work processes and a lack of coordination among departments, limiting the PRO integration into different contexts [ 26 , 43 , 49 ].

Incentive systems: Depending on the use of PRO tools, appropriate incentives, including the following two main approaches, can increase the motivation of HCPs to use PROs: First, material incentives can be provided in return for HCPs integrating PROs into their daily clinical work, including direct financial support and the use of PROs in employee performance evaluations [ 22 , 47 , 58 ]. In addition, intangible incentives are essential in the application of PRO tools. For example, the perceived benefits of the routine implementation of PRO tools by stakeholders [ 41 , 50 , 51 ] and recognition from management for using PROs [ 51 ] will boost the confidence of HCPs in using PROs.

Available resources: This construct emphasizes the integration of all available resources from the three aspects of funding, space, and materials and equipment, including ensuring adequate financial support [ 41 ], providing sufficient time and space [ 22 , 41 ], allocating required materials or equipment [ 19 , 42 ], and rationally distributing human resources, which can provide appropriate resource support for PRO implementation [ 26 ].

Access to knowledge and information: The lack of accessible training for HCPs limits the implementation of PROs. Providing sufficient time and resources to train HCPs can motivate them to use such tools in clinical practice before PRO implementation [ 20 , 50 ]. Besides, patients and their caregivers should be trained, and helping them understand the purpose and role of symptom monitoring will improve their use of PRO tools [ 41 ].

Several significant domain-influencing factors were less mentioned in the literature. These include reflecting people-centered values [ 46 ], considering the timing of PRO introduction into the system [ 41 ], ensuring that PRO projects have specific advantages over other projects [ 41 , 42 ], and setting strategic goals for coordinating PRO implementation [ 24 ]. All should be considered when designing PRO projects.

Individuals domain

This domain primarily emphasizes the crucial role of multi-stakeholders in PRO implementation.

Innovation deliverers: The structure focuses on the capacity and motivation of HCPs in PRO implementation. First, complete awareness, understanding, and familiarity with PRO tools may increase the ability and motivation of HCPs to utilize these tools [ 20 , 42 , 45 ]. However, insufficient implementation experience and limited working hours may hinder the effective use of PROs [ 18 ]. Furthermore, HCPs use PRO tools because of their perceived efficacy in clinical problem management. Therefore, using PRO tools may decrease if their efficacy is uncertain. [ 19 , 49 ].

Innovation recipients: PRO recipients will be limited in their ability to complete the questionnaire due to older age, low literacy, language barriers, severe disease conditions, and poor current physical condition [ 20 , 21 , 26 ]. Several studies have reported that educating patients on the value of PROs can increase their willingness to complete questionnaires [ 30 , 38 ]. However, due to limited clinic time [ 20 , 22 ], questions about the significance of PROs [ 21 ], and fear of replacing communication with HCPs [ 26 ], the willingness of patients to complete the questionnaire can be reduced. Stakeholders, including PRO project management, opinion leaders, and project facilitators who support and participate actively, significantly contribute to PRO implementation [ 41 , 46 ].

Implementation process domain

Teaming: This construct emphasizes the need to form multidisciplinary teams of HCPs, with members from each discipline working together to improve the clinical use of PROs [ 20 , 51 ].

Assess needs: PRO practitioners are looking for a tool to help them manage clinical issues and software support to visualize the PRO details of patients in charts and graphs [ 38 , 43 ]. However, some clinicians preferred using PROs during nurse consultations to increase clinical productivity [ 20 ]. Previous studies have reported that it is essential for PRO recipients to understand and consider the needs of patients and provide them with clear information [ 20 , 43 ]. In addition, patients want to be able to complete self-reporting from home via the Internet or telephone [ 22 ].

Planning: Developing workflows for the corresponding tasks, including the identification of reasonable points in time and frequency of assessment, and integrating the workflows into routine clinical work is necessary to ensure the smooth operation of the PROs [ 26 , 43 , 50 ].

Tailoring strategies: Small-scale system testing before the formal PRO implementation can facilitate long-term stable operations [ 22 , 39 ].

PROs in oncology clinical practice settings have received significant attention in recent years. Although previous review studies have investigated facilitators and barriers to implementing PROs, most have not systematically identified and integrated these determinants from the IS perspective [ 53 , 54 , 55 , 56 , 57 ]. Because the CFIR can be used to identify implementation influences, develop strategies, and evaluate implementation effectiveness [ 59 , 60 ], this systematic review will identify and weigh important factors influencing PRO implementation in oncology clinical practice settings. We found 52 facilitators and 50 barriers in the 30 included studies. Consistent with the findings of previous studies [ 53 , 54 , 55 , 56 , 57 ], which mentioned barriers such as time constraints, a lack of relevant training, doubts about the clinical value of PROs, a lack of financial support, and high patient symptom burden, and facilitators including well-functioning electronic tool availability, embedding the EHR, and understanding the clinical value of PROs.

In the innovation domain, the development and design of PRO tools should be based on clinical evidence to ensure that such tools are scientifically feasible to improve clinical effectiveness [ 11 , 61 , 62 ]. Evidence-based PRO practices can assist HCPs in identifying and addressing existing or potential health problems in patients and understanding how PRO tools can enhance clinical practice [ 63 ]. HCPs will reduce PRO use if they believe it is insufficient for current clinical decision-making or realize it will harm them and their patients. Second, the requirements and capabilities of multiple stakeholders need to be considered during the PRO design phase [ 64 , 65 ]. Patients vary in literacy, language communication skills, and medical conditions in clinical practice. Consequently, the ability of patients to complete those tools should be carefully considered when designing the PRO tool. The PRO tool design should be simplified, and strategies including limiting the number of entries, establishing an appropriate questionnaire level of difficulty, having a proper monitoring frequency and duration of follow-up, and being accessible to access and use may increase the acceptability of the PRO tools [ 61 ]. Besides, PRO data should be presented to HCPs in an understandable way, and delivering the collected information through trend graphs or visualizations may help HCPs understand and address the problems of their patients faster.The main factors affecting PRO implementation in the outer setting domain are policy support from government departments, communication between external organizations, and obtaining external financial support [ 66 ]. PRO implementation requires support from various departments. Implementing PROs to collect problems of patients in routine clinical practice should adhere to international and national organizations or government departments' guidelines and local policy norms. Conversely, matching sufficient resources to support the effective PRO implementation will be difficult [ 61 ]. Furthermore, external organizations should strengthen their partnerships and share best practices. Obtaining mutual support across sectors will facilitate better PRO implementation. However, the lack of support from local experts in low-income countries often impedes PRO implementation and promotion. Adequate funding is an essential facilitator in ensuring smooth PRO implementation. Lai-Kwon et al. [ 41 ] conducted a qualitative study to identify barriers and facilitators to implementing ePRO symptom monitoring in routine cancer care using the Consolidated Framework for Implementation Research (CFIR) and the barriers were matched to theory-informed implementation strategies using the CFIR-Expert Recommendations for Implementing Change (ERIC) matching tool. This research reported that appropriate financial support can be obtained by lobbying the government for long-term funding, incorporating oncology funding models, and commercialization. However, the current funding model does not provide dedicated financial support for PROs.

When implementing PROs in an internal clinical practice environment, several factors should be considered. There is a prevailing trend toward using electronic PRO tools to collect questions in clinical practice [ 67 ]. However, Lai-Kwon et al. [ 41 ] reported that it is too early to implement PROs in the early stages of digital transformation in hospitals. The infrastructure configuration of the clinical setting must be met by providing accessible computers, improving system functionality, and planning work processes and staffing [ 63 ]. Moreover, embedding PROs into EHRs should be considered, as it can assist HCPs in collecting and managing patient clinical data efficiently. Based on the internal environment, the ability and motivation of employees to use PRO tools should be considered. Employees can learn and use PRO tools better through systematic training and the promotion of multidisciplinary department exchanges. Furthermore, direct financial incentives and moderate encouragement and recognition from management are needed to motivate the routine use of the PRO tools. Adequate funding is essential in ensuring the successful implementation of PRO.

Multi-stakeholder involvement is essential in the individual domain, as innovation deliverers and recipients play an important role in PRO implementation [ 58 , 68 ,  69 ]. The innovation deliverers must be familiar with the content, clinical value, and implementation procedures of PRO tools and boost their confidence in implementing PROs through appropriate training as the prominent members who directly implement PROs. Additionally, the necessary conditions for innovation deliverers to implement PROs, including appropriate space, sufficient time, and available equipment, should be provided. Educational attainment, disease status, cultural differences, and language ability vary among innovative recipients. These factors can limit the application of PRO programs. As mentioned in the innovation domain, differences in patient ability must be considered during the design phase of an innovation program to ensure that the developed PRO tools can meet the needs of different patients. Besides, allowing patients to understand the clinical benefits and scientific value of PRO implementation is essential in increasing their willingness to use them. Other stakeholders, including leaders at all levels of innovation, opinion leaders, and executive leads, can increase the motivation of implementation team members by trusting and affirming the implementation of PRO programs.

Before implementing a PRO program, it is essential to form a multidisciplinary team, design a rational workflow, and test the program on a small scale. A sensible workflow for implementing PRO needs to be planned, including integrating it into the flow of routine clinical practice and determining reasonable nodes and frequency of evaluation practice [ 58 , 63 , 66 ]. As a result, the feasibility of the PRO program can be effectively verified through a small-scale trial, and potential problems in the process can be identified, allowing the workflow of PROs to be rationalized promptly. During implementation, it is necessary to meet the needs of various stakeholders [ 62 ]. Innovation deliverers must meet the hardware requirements of HCPs to implement PROs for information technology support, which can effectively improve implementation efficiency. However, because implementing PROs can be labor-, time-, and money-intensive, it is essential to consider a PRO tool with high sensitivity and specificity for screening patient populations for potential problems. Providing innovation recipients with the information to fill out the PROs, understanding and meeting their needs across the material, psychosocial, and other dimensions of the disease, and, significantly, timely treatment and feedback from HCPs when they experience early warning symptoms can increase the sense of benefit to the patient from filling out the PROM and improve better self-reporting.

The greatest strength of this study is the use of the updated CFIR framework, which provides a comprehensive and systematic categorization of the determinants that influence PRO implementation programs. The updated CFIR framework may provide researchers with a structured approach to analyzing and understanding the various factors influencing the successful implementation of a program, policy, or intervention. However, this study has several limitations. First, the dimensions and constructs of the updated CFIR exhibit instances of synonyms and polysemous terms, resulting in ambiguity during the coding and analysis process. The literature in this systematic review is almost exclusively from developed countries; thus, we have no idea what opportunities and challenges low- and middle-income countries face in implementing PRO programs. Second, cultural and healthcare system differences between countries mean that some of the influences identified in this study are unique to individual countries or regions, and the findings of this study should be applied with caution in clinical practice. Third, the diversity of the study types included the fact that fewer studies used the CFIR framework to identify influencing factors, resulting in variations in definitions and understandings of influencing factors across studies and the differences in how influencing factors were documented across studies, with the strength of the evidence for individual influences being further justified. Fourth, this study only systematically searched Chinese and English databases with no systematic search of literature in other languages. Unfortunately, the initial search of Chinese databases revealed no relevant studies describing the determinants of implementing PRO. This may be because the concept is newly introduced in the Chinese region, and no attention has been paid to implementing specific PROs.

This study systematically reviewed the barriers and facilitators to PRO implementation in oncology clinical practice settings based on the updated CFIR framework. Considering the design characteristics of PROs, available conditions in the outer and inner settings, and meeting the needs of all stakeholders in the implementation process are essential to ensuring the effective implementation of PRO programs. This systematic review provides a reference framework for departments to develop strategies to resolve current PRO implementation issues and ensure successful PRO implementation.

Data Availability

The datasets used and analyzed in this study are available upon request from the corresponding author.

Abbreviations

Patient Reported Outcome

Consolidated Framework for Implementation Research

Clinician-reported outcome

Observer-reported outcome

Performance outcome

Implementation Science

National Institution of Health

Reach, Effectiveness, Adoption, Implementation, Maintenance

Promoting Action on Research Implementation in Health Services Framework

Gordon BE, Chen RC. Patient-reported outcomes in cancer survivorship[J]. Acta Oncol. 2017;56(2):166–73. https://doi.org/10.1080/0284186x.2016.1268265 .

Article   PubMed   Google Scholar  

Milton L, Behroozian T, Coburn N, et al. Prediction of breast cancer-related outcomes with the Edmonton Symptom Assessment Scale: A literature review[J]. Support Care Cancer. 2021;29(2):595–603. https://doi.org/10.1007/s00520-020-05755-9 .

van den Hurk CJG, Mols F, Eicher M, et al. A Narrative Review on the Collection and Use of Electronic Patient-Reported Outcomes in Cancer Survivorship Care with Emphasis on Symptom Monitoring[J]. Curr Oncol. 2022;29(6):4370–85. https://doi.org/10.3390/curroncol29060349 .

Article   PubMed   PubMed Central   Google Scholar  

Basch E. Patient-Reported Outcomes - Harnessing Patients’ Voices to Improve Clinical Care[J]. N Engl J Med. 2017;376(2):105–8. https://doi.org/10.1056/NEJMp1611252 .

Shi Q, Mendoza TR, Wang XS, et al. Using a symptom-specific instrument to measure patient-reported daily functioning in patients with cancer[J]. Eur J Cancer. 2016;67:83–90. https://doi.org/10.1016/j.ejca.2016.07.027 .

Görlach MG, Schrage T, Bokemeyer C, et al. Implementation analysis of patient reported outcomes (PROs) in oncological routine care: an observational study protocol[J]. Health Qual Life Outcomes. 2020;18(1):3. https://doi.org/10.1186/s12955-019-1262-2 .

Black N. Patient reported outcome measures could help transform healthcare[J]. BMJ. 2013;346: f167. https://doi.org/10.1136/bmj.f167 .

Calvert M, Kyte D, Mercieca-Bebber R, et al. Guidelines for Inclusion of Patient-Reported Outcomes in Clinical Trial Protocols: The SPIRIT-PRO Extension[J]. JAMA. 2018;319(5):483–94. https://doi.org/10.1001/jama.2017.21903 .

Barbera L, Sutradhar R, Seow H, et al. Impact of Standardized Edmonton Symptom Assessment System Use on Emergency Department Visits and Hospitalization: Results of a Population-Based Retrospective Matched Cohort Analysis[J]. JCO Oncol Pract. 2020;16(9):e958–65. https://doi.org/10.1200/jop.19.00660 .

Warrington L, Absolom K, Velikova G. Integrated care pathways for cancer survivors - a role for patient-reported outcome measures and health informatics[J]. Acta Oncol. 2015;54(5):600–8. https://doi.org/10.3109/0284186x.2014.995778 .

Di Maio M, Basch E, Denis F, et al. The role of patient-reported outcome measures in the continuum of cancer clinical care: ESMO Clinical Practice Guideline[J]. Ann Oncol. 2022;33(9):878–92. https://doi.org/10.1016/j.annonc.2022.04.007 .

Basch E, Deal AM, Dueck AC, et al. Overall Survival Results of a Trial Assessing Patient-Reported Outcomes for Symptom Monitoring During Routine Cancer Treatment[J]. JAMA. 2017;318(2):197–8. https://doi.org/10.1001/jama.2017.7156 .

Collado-Borrell R, Escudero-Vilaplana V, Narrillos-Moraza Á, et al. Patient-reported outcomes and mobile applications. A review of their impact on patients’ health outcomes[J]. Farm Hosp. 2022;46(3):173–81.

PubMed   Google Scholar  

Terwee CB, Zuidgeest M, Vonkeman HE, et al. Common patient-reported outcomes across ICHOM Standard Sets: the potential contribution of PROMIS®[J]. BMC Med Inform Decis Mak. 2021;21(1):259. https://doi.org/10.1186/s12911-021-01624-5 .

Valderas JM, Kotzeva A, Espallargues M, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature[J]. Qual Life Res. 2008;17(2):179–93. https://doi.org/10.1007/s11136-007-9295-0 .

Article   PubMed   CAS   Google Scholar  

Amini M, Oemrawsingh A, Verweij LM, et al. Facilitators and barriers for implementing patient-reported outcome measures in clinical care: An academic center’s initial experience[J]. Health Policy. 2021;125(9):1247–55. https://doi.org/10.1016/j.healthpol.2021.07.001 .

Chen M, Jones CM, Bauer HE, et al. Barriers and Opportunities for Patient-Reported Outcome Implementation: A National Pediatrician Survey in the United States[J]. Children (Basel). 2022;9(2):185. https://doi.org/10.3390/children9020185 .

Consolo L, Colombo S, Basile I, et al. Barriers and facilitators of electronic patient-reported outcome measures (e-PROMs) for patients in home palliative cancer care: a qualitative study of healthcare professionals’ perceptions[J]. BMC Palliat Care. 2023;22(1):111. https://doi.org/10.1186/s12904-023-01234-0 .

Zhang R, Burgess ER, Reddy MC, et al. Provider perspectives on the integration of patient-reported outcomes in an electronic health record[J]. JAMIA Open. 2019;2(1):73–80. https://doi.org/10.1093/jamiaopen/ooz001 .

Graupner C, Breukink SO, Mul S, et al. Patient-reported outcome measures in oncology: a qualitative study of the healthcare professional’s perspective[J]. Support Care Cancer. 2021;29(9):5253–61. https://doi.org/10.1007/s00520-021-06052-9 .

Mott NM, Huynh V, Vemuru S, et al. Barriers and facilitators to measuring patient reported outcomes in an academic breast cancer clinic: An application of the RE-AIM framework[J]. Am J Surg. 2024;228:180–4. https://doi.org/10.1016/j.amjsurg.2023.09.022 .

Bruner DW, Hanisch LJ, Reeve BB, et al. Stakeholder perspectives on implementing the National Cancer Institute’s patient-reported outcomes version of the Common Terminology Criteria for Adverse Events (PRO-CTCAE)[J]. Transl Behav Med. 2011;1(1):110–22. https://doi.org/10.1007/s13142-011-0025-3 .

Allar BG, Eruchalu CN, Rahman S, et al. Lost in translation: A qualitative analysis of facilitators and barriers to collecting patient reported outcome measures for surgical patients with limited English proficiency[J]. Am J Surg. 2022;224(1 Pt B):514–21. https://doi.org/10.1016/j.amjsurg.2022.03.005 .

Cheung YT, Chan A, Charalambous A, et al. The use of patient-reported outcomes in routine cancer care: preliminary insights from a multinational scoping survey of oncology practitioners[J]. Support Care Cancer. 2022;30(2):1427–39. https://doi.org/10.1007/s00520-021-06545-7 .

Breidenbach C, Kowalski C, Wesselmann S, et al. Could existing infrastructure for using patient-reported outcomes as quality measures also be used for individual care in patients with colorectal cancer?[J]. BMC Health Serv Res. 2021;21(1):448. https://doi.org/10.1186/s12913-021-06457-6 .

Nguyen H, Butow P, Dhillon H, et al. Using patient-reported outcomes (PROs) and patient-reported outcome measures (PROMs) in routine head and neck cancer care: What do health professionals perceive as barriers and facilitators?[J]. J Med Imaging Radiat Oncol. 2020;64(5):704–10. https://doi.org/10.1111/1754-9485.13048 .

National Institutes of Health, National Cancer Institute. About Implementation Science. https://cancercontrol.cancer.gov/is/about . Accessed 17 May 2024.

Kuo GM, Trinkley KE, Rabin B. Research and Scholarly Methods: Implementation Science Studies[J]. J Am Coll Clin Pharm. 2022;5(9):995–1004. https://doi.org/10.1002/jac5.1673 .

Ridde V, Pérez D, Robert E. Using implementation science theories and frameworks in global health[J]. BMJ Glob Health. 2020;5(4):e002269. https://doi.org/10.1136/bmjgh-2019-002269 .

Damschroder LJ, Reardon CM, Widerquist MAO, et al. The updated Consolidated Framework for Implementation Research based on user feedback[J]. Implement Sci. 2022;17(1):75. https://doi.org/10.1186/s13012-022-01245-0 .

Birken SA, Powell BJ, Shea CM, et al. Criteria for selecting implementation science theories and frameworks: results from an international survey[J]. Implement Sci. 2017;12(1):124. https://doi.org/10.1186/s13012-017-0656-y .

Chan PS, Fang Y, Wong MC, et al. Using Consolidated Framework for Implementation Research to investigate facilitators and barriers of implementing alcohol screening and brief intervention among primary care health professionals: a systematic review[J]. Implement Sci. 2021;16(1):99. https://doi.org/10.1186/s13012-021-01170-8 .

Curran GM, Bauer M, Mittman B, et al. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact[J]. Med Care. 2012;50(3):217–26. https://doi.org/10.1097/MLR.0b013e3182408812 .

Joanna Briggs institute. Critical Appraisal Tools. https://jbi.global/critical-appraisal-tools . Accessed 17 May 2024.

Hong Q N, Pluye P, Fàbregues S, et al. Mixed Methods Appraisal Tool (MMAT) Version 2018 User Guide. Registration of Copyright (#1148552), Canadian Intellectual Property Office, Industry Canada. 2018. http://mixedmethodsappraisaltoolpublic.pbworks.com/w/file/fetch/127916259/MMAT_2018_criteria-manual_2018-08-01_ENG.pdf . Accessed 26 Apr 2024.

Duman-Lubberding S, van Uden-Kraan CF, Jansen F, et al. Durable usage of patient-reported outcome measures in clinical practice to monitor health-related quality of life in head and neck cancer patients[J]. Support Care Cancer. 2017;25(12):3775–83. https://doi.org/10.1007/s00520-017-3808-3 .

Article   PubMed   PubMed Central   CAS   Google Scholar  

Hartkopf AD, Graf J, Simoes E, et al. Electronic-Based Patient-Reported Outcomes: Willingness, Needs, and Barriers in Adjuvant and Metastatic Breast Cancer Patients[J]. JMIR Cancer. 2017;3(2):e11. https://doi.org/10.2196/cancer.6996 .

Iroz CB, Johnson JK, Ager MS, et al. Barriers and Facilitators to Implementing Patient-Reported Outcome Monitoring in Gastrointestinal Surgery[J]. J Surg Res. 2023;288:341–9. https://doi.org/10.1016/j.jss.2023.03.011 .

Jagsi R, Chiang AC, Medeiros BC, et al. Qualitative analysis of practicing oncologists’ attitudes and experiences regarding HIT-facilitated collection of patient-reported outcomes[J]. J Clin Oncol. 2012;30(34):290–7. https://doi.org/10.1200/JOP.2012.000823 .

Article   Google Scholar  

Kiderlen TR, Schnack A, de Wit M. Essential barriers and considerations for the implementation of electronic patient-reported outcome (ePRO) measures in oncological practice: contextualizing the results of a feasibility study with existing literature[J]. Journal of Public Health (Germany). 2023;31(12):2071–88. https://doi.org/10.1007/s10389-022-01767-3 .

Lai-Kwon J, Rutherford C, Jefford M, et al. Using Implementation Science Frameworks to Guide the Use of Electronic Patient-Reported Outcome Symptom Monitoring in Routine Cancer Care[J]. JCO Oncol Pract. 2024;20(3):335–49. https://doi.org/10.1200/op.23.00462 .

Lopez CJ, Jones JM, Campbell KL, et al. A pre-implementation examination of barriers and facilitators of an electronic prospective surveillance model for cancer rehabilitation: a qualitative study[J]. BMC Health Serv Res. 2024;24(1):17. https://doi.org/10.1186/s12913-023-10445-3 .

Minteer SA, Cheville A, Tesch N, et al. Implementing cancer symptom management interventions utilizing patient-reported outcomes: a pre-implementation evaluation of barriers and facilitators[J]. Support Care Cancer. 2023;31(12):697. https://doi.org/10.1007/s00520-023-08114-6 .

Naehrig DN, Dhillon HM, Asher R, et al. Patient-reported outcome measures and supportive care need assessment in patients attending an Australian comprehensive care centre: a multi-method study[J]. Support Care Cancer. 2021;29(9):5037–46. https://doi.org/10.1007/s00520-021-06028-9 .

Roberts NA, Alexander K, Wyld D, et al. What is needed by staff to implement PROMs into routine oncology care? A qualitative study with the multi-disciplinary team[J]. Eur J Cancer Care. 2019;28(6):13167. https://doi.org/10.1111/ecc.13167 .

Roberts NA, Janda M, Stover AM, et al. The utility of the implementation science framework “Integrated Promoting Action on Research Implementation in Health Services” (i-PARIHS) and the facilitator role for introducing patient-reported outcome measures (PROMs) in a medical oncology outpatient department[J]. Qual Life Res. 2021;30(11):3063–71. https://doi.org/10.1007/s11136-020-02669-1 .

Sandhu S, King Z, Wong M, et al. Implementation of Electronic Patient-Reported Outcomes in Routine Cancer Care at an Academic Center: Identifying Opportunities and Challenges[J]. Jco Oncology Practice. 2020;16(11):1255–63. https://doi.org/10.1200/op.20.00357 .

Schepers SA, Haverman L, Zadeh S, et al. Healthcare Professionals’ Preferences and Perceived Barriers for Routine Assessment of Patient-Reported Outcomes in Pediatric Oncology Practice: Moving Toward International Processes of Change[J]. Pediatr Blood Cancer. 2016;63(12):2181–8. https://doi.org/10.1002/pbc.26135 .

Schepers SA, Nicolaas SMS, Haverman L, et al. Real-world implementation of electronic patient-reported outcomes in outpatient pediatric cancer care[J]. Psychooncology. 2017;26(7):951–9. https://doi.org/10.1002/pon.4242 .

Shunmugasundaram C, Sundaresan P, White K, et al. Development and implementation barriers of a new patient-reported measure: The Radiation therapy-related Inconvenience Questionnaire (RIQ)[J]. J Med Imaging Radiat Oncol. 2023;67(7):777–88. https://doi.org/10.1111/1754-9485.13586 .

Skare TS, Midtbust MH, Lund JA, et al. Barriers and Facilitators When Implementing Electronic Patient-Reported Outcome Measures at a Municipal Cancer Care Unit A Qualitative Study[J]. Cancer Nurs. 2023;46(4):268–75. https://doi.org/10.1097/ncc.0000000000001120 .

Tsangaris E, Hyland C, Liang G, et al. Feasibility of implementing patient-reported outcome measures into routine breast cancer care delivery using a novel collection and reporting platform[J]. JAMIA Open. 2023;6(4):ooad108. https://doi.org/10.1093/jamiaopen/ooad108 .

Abejas A G, Trullas A S, Sobral M A, et al. Improving the Understanding and Managing of the Quality of Life of Patients With Lung Cancer With Electronic Patient-Reported Outcome Measures: Scoping Review[J]. Journal of Medical Internet Research. 2023;25. https://doi.org/10.2196/46259 .

Easpaig BNG, Tran Y, Bierbaum M, et al. What are the attitudes of health professionals regarding patient reported outcome measures (PROMs) in oncology practice? A mixed-method synthesis of the qualitative evidence[J]. BMC Health Serv Res. 2020;20(1):102. https://doi.org/10.1186/s12913-020-4939-7 .

Howell D, Molloy S, Wilkinson K, et al. Patient-reported outcomes in routine cancer clinical practice: a scoping review of use, impact on health outcomes, and implementation factors[J]. Ann Oncol. 2015;26(9):1846–58. https://doi.org/10.1093/annonc/mdv181 .

Nguyen H, Butow P, Dhillon H, et al. A review of the barriers to using Patient-Reported Outcomes (PROs) and Patient-Reported Outcome Measures (PROMs) in routine cancer care[J]. J Med Radiat Sci. 2021;68(2):186–95. https://doi.org/10.1002/jmrs.421 .

van Egdom LSE, Oemrawsingh A, Verweij LM, et al. Implementing Patient-Reported Outcome Measures in Clinical Breast Cancer Care: A Systematic Review[J]. Value in Health. 2019;22(10):1197–226. https://doi.org/10.1016/j.jval.2019.04.1927 .

Mazariego C, Jefford M, Chan RJ, et al. Priority recommendations for the implementation of patient-reported outcomes in clinical cancer care: a Delphi study[J]. J Cancer Surviv. 2022;16(1):33–43. https://doi.org/10.1007/s11764-021-01135-2 .

Means AR, Kemp CG, Gwayi-Chore MC, et al. Evaluating and optimizing the consolidated framework for implementation research (CFIR) for use in low- and middle-income countries: a systematic review[J]. Implement Sci. 2020;15(1):17. https://doi.org/10.1186/s13012-020-0977-0 .

Kirk MA, Kelley C, Yankey N, et al. A systematic review of the use of the Consolidated Framework for Implementation Research[J]. Implement Sci. 2016;11:72. https://doi.org/10.1186/s13012-016-0437-z .

Hartzler AL, Izard JP, Dalkin BL, et al. Design and feasibility of integrating personalized PRO dashboards into prostate cancer care[J]. J Am Med Inform Assoc. 2016;23(1):38–47. https://doi.org/10.1093/jamia/ocv101 .

Foster A, Croot L, Brazier J, et al. The facilitators and barriers to implementing patient reported outcome measures in organisations delivering health related services: a systematic review of reviews[J]. J Patient Rep Outcomes. 2018;2:46. https://doi.org/10.1186/s41687-018-0072-3 .

Laitio AM, Giunti G, Halonen R. Perceived Barriers and Facilitators in Using Patient-Reported Outcome Systems for Cancer Care: Systematic Mapping Study[J]. JMIR Cancer. 2023;9:e40875. https://doi.org/10.2196/40875 .

Flannery MA, Mohile S, Culakova E, et al. Completion of Patient-Reported Outcome Questionnaires Among Older Adults with Advanced Cancer[J]. J Pain Symptom Manage. 2022;63(2):301–10. https://doi.org/10.1016/j.jpainsymman.2021.07.032 .

Ball AE, Russell EM, Seymour DG, et al. Problems in using health survey questionnaires in older patients with physical disabilities. Can proxies be used to complete the SF-36?[J]. Gerontology. 2001;47(6):334–40. https://doi.org/10.1159/000052824 .

Chan EKH, Edwards TC, Haywood K, et al. Implementing patient-reported outcome measures in clinical practice: a companion guide to the ISOQOL user’s guide[J]. Qual Life Res. 2019;28(3):621–7. https://doi.org/10.1007/s11136-018-2048-4 .

Haun JN, Alman AC, Melillo C, et al. Using Electronic Data Collection Platforms to Assess Complementary and Integrative Health Patient-Reported Outcomes: Feasibility Project[J]. JMIR Med Inform. 2020;8(6):e15609. https://doi.org/10.2196/15609 .

Bamgboje-Ayodele A, Avery S, Pearson J, et al. Adapting an integrated care pathway for implementing electronic patient reported outcomes assessment in routine oncology care: Lessons learned from a case study[J]. J Eval Clin Pract. 2022;28(6):1072–83. https://doi.org/10.1111/jep.13688 .

Knowles SE, Ercia A, Caskey F, et al. Participatory co-design and normalisation process theory with staff and patients to implement digital ways of working into routine care: the example of electronic patient-reported outcomes in UK renal services[J]. BMC Health Serv Res. 2021;21(1):706. https://doi.org/10.1186/s12913-021-06702-y .

Download references

Acknowledgements

Not applicable.

This work is supported through Graduate Research and Innovation Fund of Chengdu Medical College (YCX2023-01–51), Sichuan Nursing Society (H22023) and Sichuan Medical Association (S23058).

Author information

Hao Zhang and Hua Wang contributed equally to this work and share first authorship.

Authors and Affiliations

Department of Head and Neck Radiation Oncology, Sichuan Cancer Hospital, Chengdu, China

Jianxia Lyu, Hua Wang, Xia Liu, Yunhua Jing & Li Yin

School of Nursing, Chengdu Medical College, Chengdu, China

Department of Nursing, First Affiliated Hospital of China Medical University, Shenyang, China

Aiping Wang

You can also search for this author in PubMed   Google Scholar

Contributions

LJX, ZH: proposed the research question and designed the theoretical construct of the systematic review; ZH, WH, LX, JYH: completed the systematic search of the literature, quality assessment, and data extraction; LJX, ZH, WH: wrote the manuscript; YL, WAP: critically revised the intellectual content of the manuscript. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Li Yin or Aiping Wang .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Lyu, J., Zhang, H., Wang, H. et al. Facilitators and barriers to implementing patient-reported outcomes in clinical oncology practice: a systematic review based on the consolidated framework for implementation research. Implement Sci Commun 5 , 120 (2024). https://doi.org/10.1186/s43058-024-00654-0

Download citation

Received : 17 June 2024

Accepted : 30 September 2024

Published : 29 October 2024

DOI : https://doi.org/10.1186/s43058-024-00654-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Patient-reported outcomes
• Implementation science
• Consolidated framework for implementation research
• Facilitators

Implementation Science Communications

ISSN: 2662-2211

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Farm-level economics of innovative tillage technologies: the case of no-till in the Altai Krai in Russian Siberia

Affiliations.

• 1 Institute of Agricultural and Nutritional Sciences, Martin-Luther-University-Halle-Wittenberg, Karl-Freiherr-von-Fritsch-Str. 4, 06120, Halle (Saale), Germany. [email protected].
• 2 Institute of Agricultural and Nutritional Sciences, Martin-Luther-University-Halle-Wittenberg, Karl-Freiherr-von-Fritsch-Str. 4, 06120, Halle (Saale), Germany.
• 3 Department of Mathematics and Information Technology, Altai State University of Barnaul, Lenina 61, 656049, Barnaul, Altai Krai, Russia.
• PMID: 28573561
• DOI: 10.1007/s11356-017-9268-y

In the agricultural Altai Krai in Russian Siberia, soil degradation problems are prevalent. Agronomists recommend "reduced tillage systems," especially no-till, as a sustainable way to cultivate land that is threatened by soil degradation. In the Altai Krai, less is known about the technologies in practice. In this paper, we provide information on plant cultivation technologies used in the Altai Krai and on selected factors preventing farm managers in this region from adopting no-till technology based on our own quantitative survey conducted across 107 farms in 2015 and 2016. The results of the quantitative survey show that farm managers have high uncertainty regarding the use of no-till technology including its economics. To close this gap, we provide systematic analysis of factors influencing the economy of the plant production systems by using a farm optimization model (linear programming) for a real farm, together with expert estimations. The farm-specific results of the optimization model show that under optimal management and climatic conditions, the expert Modern Canadian no-till technology outperforms the farm min-till technology, but this is not the case for suboptimal conditions with lower yields.

Keywords: Erosion; Linear programming; No-till; Profitability; Reduced soil cultivation technology; Russia.

• Agriculture / instrumentation
• Agriculture / methods*
• Farms / economics*
• Models, Economic

The Virgin Lands Campaign (1954–1963) Until the Breakdown of the Former Soviet Union (FSU): With Special Focus on Western Siberia

• First Online: 19 October 2019

Cite this chapter

• M. Frühauf 7 ,
• T. Meinel 8 &
• G. Schmidt 7  

Part of the book series: Innovations in Landscape Research ((ILR))

889 Accesses

10 Citations

1 Altmetric

The goal of the Virgin Lands Campaign under the leadership of Nikita Khrushchev was an increase in the agricultural production rate to alleviate food shortage in the Soviet Union after World War II. The campaign was targeted at converting the vast steppe, mainly in northern Kazakhstan and the Altai Region, into arable land for grain production. During the 10 years of the Virgin Lands Campaign, 420,000-km 2 steppe were transformed. The largest area was converted in the Kazakh part of the former Soviet Union. This was globally the largest ecosystem conversion of the temperate grassland in the twentieth century, and it had an important demographic dimension as well, which is reflected in the influx of population. The re-settlers found their home in new settlements, which were also the centres of newly established agricultural businesses. The campaign’s economic success turned out to be very depended on weather conditions—especially on precipitation. The annual crop yield was extremely variable, and the political goals of the campaign were unattainable. At the same time, increasing soil degradation affected the agricultural land use. After Khrushchev’s fall, the new party leaders enforced new land management methods, which addressed the ecological problems, and slowly, yield stability was achieved.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info
• Durable hardcover edition

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

The Impasse of Contemporary Agro-pastoralism in Central Tanzania: Environmental Pressures in the Face of Land Scarcity and Commercial Agricultural Investment

Elucidating Revival Measures to Extenuate Expanse of Fallow Lands and Climate Change: An Empirical Analysis of Jharkhand

Land Use Expansion in the Brazilian Cerrado

ASK (2003) Republika Kazahstan: 50-let nachala osvoenija celinnyh i zalezhnyh zemel’. Statistischeskij sbornik 1953–2003 (Republic of Kazakhstan: 50 Years since the Beginning of Virgin Lands Campaign. Statistical digest 1953–2003). Agency of Statistics of the Republic of Kazakhstan, Almaty (in Russian)

Google Scholar  

Bazhenova OI, Martanova GN (2003) The problem of Siberian steppe landscape transformation at climate and land-use change. In: Chibilyov AA Steppes of Northern Eurasia, pp 59–61

Beloglasov EV et al (1999) Slavim zemliu (We glorify the earth). Holding Blitz-Inform, Kiev (in Russian)

Brezhnev L (1979) Virgin Lands: two years in Kazakhstan (1954–55). Pergamon Press, Oxford

Bulygin IS (2000) Spisok naselennykh punktov Altaiskogo kraia: Spravochnik. Barnaul – 237s. Demographic transformation of municipal districts and settlements of Altai Krai in the Post-soviet period. In: Madry C (ed) The social transformation of the cities and regions in the post-communist countries. Bogucki Wydawnictwo Naukowe, Poznan, 2014, pp 23–33

Durgin FA Jr (1962) The Virgin Lands Program 1954–1960. Soviet Stud 13(3):255–280

Article   Google Scholar  

Eule W (1962) Das Problem der Neulandgewinnung in der Sowjetunion (The problem of reclaiming land in the Soviet Union). Dissertation. University of Bonn (in German)

Frühauf M, Meinel T (2007) Desertification in the agricultural used dry steppes in central Asia. In: Proceedings of the international conference soil and desertification—integrated research for the sustainable management of soils in Drylands, 5–6 May 2006, Hamburg, Germany, Organised by Desert*Net Germany and AK BoGeo ( http://www.desertnet.de/proceedings/start )

Frühauf M, Meinel T (2014) Ökosystemkonversion in den temperierten Grasländern Südwestsibiriens: Steuerfaktoren, Etappen und Geoökologische Konsequenzen (Ecosystem conversion in the temperate grasslands of southwestern Siberia: key factors, stages and geoecological consequences). Geoöko XXXV(1–2):5–38 (in German)

Georgiev AV (1955) God rabota po osvoeniiu tselinnykh I zalezhnykh zemel’ v Altaiskom Krae (A year of work on the development of virgin and fallow lands in the Altai Krai). Publisher, Place [in Russian]

GKS (1956–1990) National Economy of Russian Soviet Federative Socialist Republic. State Statistical Publishing House, Moscow

GKS SSSR (1991) Narodnoe Khoziaistvo SSSR v 1990. Finansi I Statistika, Moscow

Glauben T et al (2014) Eastern breadbasket obstructs its market and growth opportunities. IAMO Policy Brief 16, 1–4. Available at: www.iamo.de/dok/IAMOPolicyBrief16_en.pdf

Hahn R (1964) Klimatische und bodenkundliche Bedingungen der Neulanderschließung in Kasachstan (Climatic and geological conditions of land reclamation in Kazakhstan). Osteuropa 14:260–266

IAMO (2015) Pressemitteilung 07/2015. https://www.iamo.de/presse/pressemitteilungen/artikel/eine-analyse-von-langfristigen-landnutzungsaenderungen-in-nordkasachstan-zeigt-ein-geringes-potenzia/ . Accessed (in German)

Ioffe G, Nefedova T, Zaslavsky I (2004) From spatial continuity to fragmentation the case of Russian farming. Ann Assoc Am Geogr 94(4):913–943

Jackson WAD (1956) The Virgin and Idle Lands of Western Siberia and Northern Kazakhstan: a geographical appraisal. Geogr Rev 46(1):1–19

Jakutin NB (2005) Erschließungsetappen und aktuelle Probleme der Neulandregion. In: 50 Jahre Neulanderschließung – Vestnik der Altai-Universität Barnaul. Nr. 1 (13) 2004, pp 38–48 (in Russian)

Josephson P, Dronin N, Mnatsakanian R, Cherp A, Efremenko D, Larin V (2013) An environmental history of Russia. Cambridge University Press, Cambridge

Book   Google Scholar  

Kazancev VI (2004) K istorii rasseleniia krest’jan na territorii Altaiskogo kraia (On the history of peasant settlement on the territory of Altai Krai). Nauchnye chtenija pamjati Ju.S. Bulygina: Sbornik nauchynh statej/pod red. Ju.M. Goncharova. Barnaul – 124 s

Kazstat (2003) Respublika Kazakhstan: 50 let nachala Tselinnoi kompanii (1953–2002) [Republic of Kazakstan: 50 years of the Virgin Lands campaign (1953–2002)]. Agency of Kazakhstan on Statistics, Almaty

Kowalev RW, Trofimov SS (1968) Allgemeine Charakteristik der Bodendecke Westsibiriens. In: Agrochemische Charakteristik der Böden der UdSSr. Nauka, Moscow (in Russian)

Kostrowski P (1959) Pyl’nye buri v Altaiskikh stepiakh (Dust storms in the Altai steppes). In: Sel’skoe khosiaistvo Sibiri, Omsk 2, pp 35 et seqq (in Russian)

Kraemer R, Prishchepov AV, Müller D, Kuemmerle T, Radeloff VC, Dara A, Terekhov A, Frühauf M (2015) Long-term agricultural land-cover change and potential for cropland expansion in the former Virgin Lands area of Kazakhstan. Environ Res Lett 10(5):054012

Kurganova I, Gerenyu VL (2012) Russian carbon balance update after 20 years of farming system collapse—oral presentation paper by the IAMO FORUM “Land in Transition”. Halle, Germany, 20–22 June 2012, Available at: http://www.iamo.de/forum0/forum2012/#.Uv4oLPu2zpM

Lenk M (2005) 50 Jahre Neulandsteppe in Kasachstan: eine kritische Bilanz, Archiv für Naturschutz und Landschaftsforschung (50 years of virgin land steppe in Kazakhstan: a critical balance, archive for nature conservation and landscape research). Archiv für Naturschutz und Landschaftsforschung 44(1):37–62 (in German)

Lyuiri DI, Goryachkin SV, Karavaeva NA, Denisenko EA, Nefedova TG (2010) Dinamika sel’skokhoziaistvennykh zemel’ Rossi v XX veke I postagrogennoe vosstanovlenie rastitel’nosti i pochv (Dynamics of agricultural lands of Russia in the 20th century and postagrogenic restoration of vegetation and soils). GEOS, Moscow (in Russian)

McCauley M (1976) Khrushchev and the development of Soviet agriculture: The Virgin Land Programme 1953–1964. Holmes & Meier Publishers Inc, New York

Meinel T (2002) Die geoökologischen Folgewirkungen der Steppenumbrüche in den 50er Jahren in Westsibirien. Ein Beitrag für zukünftige Nutzungskonzepte unter besonderer Berücksichtigung der Winderosion (The geoecological consequences of the steppe ploughing in the 1950s in Western Siberia. A contribution for future usage concepts with special consideration of the wind erosion). Dissertation, Martin Luther University of Halle-Wittenberg (in German)

Milanova EV et al (1999) Land use/land cover change in Russia: mapping and GIS. Land Use Policy 16(3):153–159 (Elsevier Science)

Mitchell JG (2004) Neue Horizonte (New Horizons). Natl Geogr 5:132–161 (in German)

Nikonov AA, Schulze E (2004) Drei Jahrhunderte Agrarwissenschaft in Russland: Von 1700 bis zur Gegenwart (Drei Jahrhunderte Agrarwissenschaft in Russland: Von 1700 bis zur Gegenwart). In: Studies on the Agricultural and Food Sector in Central and Eastern Europe, IAMO 27 (in German)

N.N. (2005) Stepi Evrazii: Sokhranenie prirodnogo raznoobraziia i monitoring sostoianiia ekosistem (Steppe of Eurasia: Preservation of Biodiversity and Monitoring of the Eco-Systems). The Steppe Institute Press, Orenburg (in Russian)

N.N. (2011) Sel’skie naselennye punkty Altaiskogo kraia: statisticheskii sbornik (Rural settlements of the Altai Krai: a statistical anthology). Territorial’nyi organ Federal’noi sluzhby gosudarstvennoi statistiki, Barnaul (in Russian)

N.N. (2017) Access (10.11.2017): http://t3.gstatic.com/images?q=tbn:ANd9GcT83Th5mJweG2e3xSlb8ged18sClMobS34JZ0p6-brRqWAALRbLO13t

N.N. (2017) Access (10.11.2017): http://kazakhsteppe.com/uploads/Threads/zelina_small.jpg

Orlowski NV (1955) Osvoenie tselinnykh i zalezhnykh zemel’ v Altaiskom Krae (The development of virgin and fallow lands in the Altai Krai). Akademi Nauk, Moscow (in Russian)

Plit F, Plit J, Zakowski W (1995) Drylands development and combating desertification: bibliographic study of experiences in countries of the CIS. Food and Agriculture Organization of the United Nations, Rome

Prishchepov AV, Müller D, Dubinin M, Baumann M, Radeloff VC (2013) Determinants of agricultural land abandonment in post-Soviet European Russia. Land Use Policy 30:873–884

Prishchepov AV, Petrick M, Müller D, Schierhorn F, Kraemer R, Kurganova I, Kopsidis M (2015) Sechzig Jahre Neulandkampagne in Russland und Kasachstan: Eine Bewertung aus ökonomischer, ökologischer und politischer Sicht (Sixty years of virgin land campaign in Russia and Kazakhstan: An evaluation from an economic, ecological and political point of view). In: IAMO. Jahresbericht 2015, Halle (Saale): IAMO, pp 41–58 (in German)

Rostankowski P (1979) Agrarraum und Getreideanbau in der Sowjetunion 1948–1985 (Agrarian area and cereal cultivation in the Soviet Union 1948–1985). Duncker & Humblot, Berlin (in German)

Rusalimova O, Savenkov O, Smirnova N, Barshukov P (2006) Soil carbon losses from seasonally-frozen soils of agricultural ecosystems in West Siberis from the 20th century. In: Hatano R, Guggenberger G (eds) Symptom of environmental change in Siberian Permafrost Region. Hokkaido University Press, Sapporo

Schierhorn F, Müller D (2011) Russlands Beitrag zur Welternährung – Brachliegende Flächen und niedrige Erträge bieten ungenutztes Potential (Russia’s contribution to world food supply—fallow land and low yields offer untapped potential). Forschungsreport 2:18–20 (in German)

Schierhorn F, Müller D, Beringer T, Prishchepo AV, Kuemmerle T, Balmann A (2013) Post-Soviet cropland abandonment and carbon sequestration in European Russia, Ukraine and Belarus. Glob Biogeochem Cycles 27:1175–1185

Schierhorn F, Müller D, Prishchepov AV, Faramarzi M, Balmann A (2014) The potential of Russia to increase its wheat production through cropland expansion and intensification. Glob Food Secur 3:133–141

Shahgedanova M (ed) (2003) The physical geography of Northern Eurasia. Oxford Regional Environments. Oxford University Press, Oxford

Spaar D, Schuhmann P (eds) (2000) Natürliche Grundlagen der Pflanzenproduktion in den Ländern der Gemeinschaft Unabhängiger Staaten und des Baltikums (Natural foundations of plant production in the countries of the Commonwealth of Independent States and the Baltic). Agrimedia, Bergen/Dumme, pp 129–145 (in German)

Späth HJ (1980) Die agro-ökologische Trockengrenze (The agro-ecological dry border). Erdkunde 34:224–231 (in German)

Stadelbauer J (1987) Neuland und Getreideversorgung. Möglichkeiten und Grenzen agrarpolitisch motivierter Raumerschließung in den Steppen der Sowjetunion, der VR China und der Mongolischen VR (Virgin land and grain supply. Possibilities and limits of agricultural policy-driven spatial development in the steppes of the Soviet Union, the People’s Republic of China and the Mongolian People’s Republic). Freiburger Universitätsblätter 96:129–145

Wein N (1983) Agriculture in the pioneering regions of Siberia. Soviet Geogr 8:67–90

Wein N (1999) Sibirien (Siberia). Perthes Regionalprofile. Klett-Cotta, Gotha, Stuttgart (in German)

Zentralarchiv-Altai Krai (1954) Bericht zur bodenkundlichen Erhebung (1951) des staatlichen Vermessungsamtes des Altai-Krai. Barnaul (unpublished, in Russian)

Download references

Author information

Authors and affiliations.

Institute of Geosciences and Geography, Martin-Luther-Universität Halle-Wittenberg, Von-Seckendorff-Platz 4, 06120, Halle, Germany

M. Frühauf & G. Schmidt

AMAZONEN-WERKE H. Dreyer GmbH & Co. KG, Am Amazonenwerk 9-13, 49205, Hasbergen-Gaste, Germany

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to M. Frühauf .

Editor information

Editors and affiliations.

Institute of Geoscience and Geography, Martin-Luther-Universität Halle-Wittenberg, Halle (Saale), Germany

Manfred Frühauf

Institut für Bodenkunde, Leibniz Universität Hannover, Hannover, Germany

Georg Guggenberger

AMAZONEN-Werke H. Dreyer GmbH & Co. KG, Astana, Kazakhstan

Tobias Meinel

Agricultural, Environmental and Food Policy, Martin-Luther-Universität Halle-Wittenberg, Halle (Saale), Germany

Insa Theesfeld

Leibniz-Institut für Länderkunde (IfL), Universität Leipzig, Leipzig, Germany

Sebastian Lentz

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Frühauf, M., Meinel, T., Schmidt, G. (2020). The Virgin Lands Campaign (1954–1963) Until the Breakdown of the Former Soviet Union (FSU): With Special Focus on Western Siberia. In: Frühauf, M., Guggenberger, G., Meinel, T., Theesfeld, I., Lentz, S. (eds) KULUNDA: Climate Smart Agriculture. Innovations in Landscape Research. Springer, Cham. https://doi.org/10.1007/978-3-030-15927-6_8

Download citation

DOI : https://doi.org/10.1007/978-3-030-15927-6_8

Published : 19 October 2019

Publisher Name : Springer, Cham

Print ISBN : 978-3-030-15926-9

Online ISBN : 978-3-030-15927-6

eBook Packages : Earth and Environmental Science Earth and Environmental Science (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research