risk benefit assessment in research

Online ordering is currently unavailable due to technical issues. We apologise for any delays responding to customers while we resolve this. For further updates please visit our website: https://www.cambridge.org/news-and-insights/technical-incident

We use cookies to distinguish you from other users and to provide you with a better experience on our websites. Close this message to accept cookies or find out how to manage your cookie settings .

Login Alert

• > The Cambridge Handbook of Health Research Regulation
• > Risk-Benefit Analysis

Book contents

• The Cambridge Handbook of Health Research Regulation
• Copyright page
• Contributors
• Acknowledgements
• Introduction
• Part I Concepts, Tools, Processes
• Section IA Concepts
• Section IB Tools, Processes and Actors
• 11 Forms of Engagement
• 12 Participatory Governance in Health Research
• 13 Risk-Benefit Analysis
• 14 The Regulatory Role of Patents in Innovative Health Research and Its Translation from the Laboratory to the Clinic
• 15 Benefit Sharing
• 16 Taking Failure Seriously
• 17 Rules, Principles and the Added Value of Best Practice in Health Research Regulation
• 18 Research Ethics Review
• 19 Data Access Governance
• 20 Is the Red Queen Sitting on the Throne?
• 21 Regulatory Authorities and Decision-Making in Health Research
• 22 The Once and Future Role of Policy Advice for Health Regulation by Experts and Advisory Committees
• Part II Reimagining Health Research Regulation

13 - Risk-Benefit Analysis

from Section IB - Tools, Processes and Actors

Published online by Cambridge University Press:  09 June 2021

This chapter explores the concept of risk-benefit analysis in health research regulation, as well as ethical and practical questions raised by identifying, quantifying, and weighing risks and benefits. It argues that the pursuit of objectivity in risk-benefit analysis is ultimately futile, as the very concepts of risk and benefit depend on attitudes and preferences about which reasonable people disagree. Building on the work of previous authors, the discussion draws on contemporary examples to show how entities reviewing proposed research can improve the process of risk-benefit assessment by incorporating diverse perspectives into their decision-making and engaging in a systematic analytical approach.

13.1 Introduction

This chapter explores the concept of risk-benefit analysis in health research regulation, as well as ethical and practical questions raised by identifying, quantifying, and weighing risks and benefits. It argues that the pursuit of objectivity in risk-benefit analysis is ultimately futile, as the very concepts of risk and benefit depend on attitudes and preferences about which reasonable people disagree. Building on the work of previous authors, the discussion draws on contemporary examples to show how entities reviewing proposed research can improve the process of risk-benefit assessment by incorporating diverse perspectives into their decision-making and engaging in a systematic analytical approach.

13.2 Identifying Risks

The term ‘risk’ refers to the possibility of experiencing a harm. The concept incorporates two different dimensions: (1) the magnitude or severity of the potential harm; and (2) the likelihood that this harm will occur. The significance of a risk depends on the interaction of these two considerations. Thus, a low chance of a serious harm, such as death, would be considered significant, as would a high chance of a lesser harm, such as temporary pain.

In the context of research, the assessment of risk focuses on the additional risks participants will experience as a result of participating in a study, which will often be less than the total level of risks to which participants are exposed. For example, a study might involve the administration of various standard-of-care procedures, such as biopsies or CT scans. If the participants would have received these same procedures even if they were not participating in the study, the risks of those interventions would not be taken into account in the risk-benefit analysis. As a result, it is possible that a study comparing two interventions that are routinely used in clinical practice could be considered low risk, even if the interventions themselves are associated with a significant potential for harm. This is the case with a significant proportion of research conducted in ‘learning health systems’, which seek to integrate research into the delivery of healthcare. Because many of the research activities in such systems involve the evaluation of interventions patients would be undergoing anyway, the risks of the research are often minimal, even when the risks of the interventions themselves may be high. Footnote 1

The risks associated with health-related research are not limited to potential physical injuries. For example, in some studies, participants may be asked to engage in discussions of emotionally sensitive topics, such as a history of previous trauma. Such discussions entail a risk of psychological distress. In other studies, a primary risk is the potential for unauthorised disclosure of sensitive personal information, such as information about criminal activity, or stigmatised conditions such as HIV, or mental disorders. If such disclosures occur, participants could suffer adverse social, legal, or economic consequences.

Research-related risks can extend beyond the individuals participating in a study. For example, studies of novel interventions for preventing or treating infectious diseases could affect the likelihood that participants will transmit the disease to third parties. Footnote 2 Similarly, studies in which psychiatric patients are taken off their medications could increase the risk that participants will engage in violent behaviour. Footnote 3 Third-party risks are an inherent feature of research on genetic characteristics, given that information about individuals’ genomes necessarily has implications for their blood relatives. Footnote 4 Thus, if a genetic study results in the discovery that a participant is genetically predisposed to a serious disease, other persons who did not consent to participate in the study might be confronted with distressing, and potentially stigmatising, information that they never wanted to know.

In some cases, third-party risks extend beyond individuals to broader social groups. As the Council for International Organizations of Medical Sciences (CIOMS) has recognised, research on particular racial or ethnic groups ‘could indicate – rightly or wrongly – that a group has a higher than average prevalence of alcoholism, mental illness or sexually transmitted disease, or that it is particularly susceptible to certain genetic disorders’, Footnote 5 thereby exposing the group to potential stigma or discrimination. One example was a study in which researchers took blood samples from members of the Havasupai tribe in an effort to identify a genetic link to type 2 diabetes. After the study was completed, the researchers used the blood samples for a variety of unrelated studies without the tribe members’ informed consent, including research related to schizophrenia, inbreeding and migration patterns. Tribe members claimed that the schizophrenia and inbreeding studies were stigmatising, and that they never would have agreed to participate in the migration research because it conflicted with the tribe’s origin story, which maintained that the tribe had originated in the Grand Canyon. The researcher institution reached a settlement with the tribe that included monetary compensation and a formal apology. Footnote 6

Despite the prevalence of third-party risks in research, most ethics codes and regulations do not mention risks to anyone other than research participants. This omission is striking given that some of these same sources explicitly state that benefits to non-participants should be factored into the risk-benefit analysis. A notable exception is the EU Clinical Trials Regulation, which states that the anticipated benefits of the study must be justified by ‘the foreseeable risks and inconveniences’, Footnote 7 without specifying that those risks and inconveniences must be experienced by the participants themselves.

In addition to omitting any reference to third-party risks, the US Federal Regulations on Research With Human Participants state that entities reviewing proposed research ‘should not consider possible long-range effects of applying knowledge gained in the research (e.g. the possible effects of the research on public policy) as among those research risks that fall within the purview of its responsibility’. Footnote 8 This provision is intended ‘to prevent scientifically valuable research from being stifled because of how sensitive or controversial findings might be used at a social level’ . Footnote 9

13.3 Identifying Benefits

The primary potential benefit of research is the production of generalisable knowledge – i.e. knowledge that has relevance beyond the specific individuals participating in the study. For example, in a clinical trial of an investigational drug, data sufficient to establish the drug’s safety and efficacy would be a benefit of research. Data showing that an intervention is not safe or effective – or that it is inferior to the existing standard of care – would also count as a benefit of research, as such knowledge can protect future patients from potentially harmful and/or ineffective treatments they might otherwise undergo.

Whether a study has the potential to produce generalisable knowledge depends in part on how it is designed. The randomised controlled clinical trial (RCT) is often described as the ‘gold standard’ of research, as it includes methodological features designed to eliminate bias and control for potential confounding variables. Footnote 10 However, in some types of research, conducting an RCT may not be a realistic option. For example, if researchers want to understand the impact of different lifestyle factors on health, it might not be feasible to randomly assign participants to engage in different behaviours, particularly over a long period of time. Footnote 11 In addition, ethical considerations may sometimes preclude the use of RCTs. For example, researchers investigating the impact of smoking on health could not ethically conduct a study in which non-smokers are asked to take up smoking. Footnote 12 In these situations, alternative study designs may be used, such as cohort or case-control studies. These alternative designs can provide valuable scientific information, but the results may be prone to various biases, a factor that should be considered in assessing the potential benefits of the research. Footnote 13

A recent example of ethical challenges to RCTs arose during the Ebola outbreak of 2013–2016, when the international relief organisation Médicins Sans Frontières refused to participate in any RCTs of experimental Ebola treatments. The group argued that it would be unethical to withhold the experimental interventions from persons in a control group when ‘conventional care offers little benefit and mortality is extremely high’. Footnote 14 The difficulty with this argument was that, in the context of a rapidly evolving epidemic, the results of studies conducted without concurrent control groups would be difficult to interpret, meaning that an ineffective or even harmful intervention could erroneously be deemed effective. Some deviations from the ‘methodologically ideal approach’, such as the use of adaptive trial designs, could have been justified by the need ‘to accommodate the expectations of participants and to promote community trust’. Footnote 15 However, any alternative methodologies would need to offer a reasonable likelihood of producing scientifically valid information, or else it would not have been ethical to expose participants to any risk at all .

The potential benefit of scientific knowledge also depends on the size of a study, as studies with very small sample sizes may lack sufficient statistical power to produce reliable information. Some commentators maintain that underpowered studies lack any potential benefit, making them inherently unethical. Footnote 16 Others point out that small studies might be unavoidable in certain situations, such as research on rare diseases, and that their results can still be useful, particularly when they are aggregated using Bayesian techniques. Footnote 17

Often, choices about study design can require trade-offs between internal and external validity. While an RCT with tightly controlled inclusion and exclusion requirements is the most reliable way to establish whether an experimental intervention is causally linked to an observable result – thereby producing a high level of internal validity – if the study population does not reflect the diversity of patients in the real world, the results might have little relevance to clinical practice – thereby producing a low level of external validity. Footnote 18 In assessing the potential benefits of a study, decision-makers should take both of these considerations into account.

In addition to the potential benefit of generalisable knowledge, some research also offers potential benefits to the individuals participating in the study. Benefits to study participants can be divided into ‘direct’ and ‘indirect’ (or ‘collateral’) benefits. Footnote 19 Direct benefits refer to those that result directly from the interventions being studied, such as an improvement in symptoms that results from taking an investigational drug. In some studies, there is no realistic possibility that participants will directly benefit from the study interventions; this would be the case in a Phase I drug study involving healthy volunteers, where the purpose is simply to identify the highest dose humans can tolerate without serious side effects. Indirect benefits include those that result from ancillary features of the study, such as access to free health screenings, as well as the psychological benefits that some participants receive from engaging in altruistic activities. Study participants may also consider any payments or other remuneration they receive in exchange for their participation as a type of research-related benefit.

Most commentators take the position that only potential direct benefits to participants and potential contributions to generalisable knowledge should be factored into the risk-benefit analysis. The concern is that, otherwise, ‘simply increasing payment or adding more unrelated services could make the benefits outweigh even the riskiest research’. Footnote 20 Other commentators reject this position on the ground that it is not consistent with the ethical imperative to respect participants’ autonomy, and that it could preclude studies that would advance the interests of participants, investigators, and society. Footnote 21 The US Food and Drug Administration has stated that payments to participants should not be considered in the context of risk-benefit assessment, Footnote 22 but it has not taken a position on consideration of other indirect benefits, such as access to free health screenings .

13.4 Quantifying Risks and Benefits

Once the risks and benefits of a proposed study have been identified, the next step is to quantify them. Doing this is complicated by the fact that the significance of a particular risk or benefit is highly subjective. For example, a common risk in health-related research is the potential for unauthorised disclosure of participants’ medical records. This risk could be very troubling to individuals who place a high degree of value on personal privacy, but for persons who share intimate information freely, the risk of unauthorised disclosure might be a minor concern. In fact, in some studies, the same experience might be perceived by some participants as a harm and by others as a benefit. For example, in a study in which participants are asked to discuss prior traumatic experiences, some participants might experience psychological distress, while others might welcome the opportunity to process past experiences with a sympathetic listener. Footnote 23

In addition to differing attitudes about the potential outcomes of research, individuals differ in their perceptions about risk-taking itself. Many people are risk averse, meaning that they would prefer to forego a higher potential benefit if it enables them to reduce the potential for harm. Others are risk neutral, or even risk preferring. Similarly, individuals exhibit different levels of willingness to trade harmful outcomes for good ones. Footnote 24 For example, some people are willing to tolerate medical treatments with significant side effects, such as chemotherapy, because they place greater value on the potential therapeutic benefits. Others place greater weight on avoiding pain or discomfort and would be disinclined to accept high-risk interventions even when the potential benefits are substantial.

Another challenge in attempting to quantify risks and benefits is that the way that risks and benefits are perceived can be influenced by a variety of cognitive biases. For example, one study asked subjects to imagine that they had lung cancer and had to decide between surgery and radiation. One group was told that 68 per cent of surgical patients survived after one year, while a second group was told that 32 per cent of surgical patients died after one year. Even though the information being conveyed was identical, framing the information in terms of a risk of death increased the number of subjects who chose radiation from 18 per cent to 44 per cent. Footnote 25 Another common cognitive bias is the ‘availability heuristic’, which leads people to attach greater weight to information that is readily called to mind. Footnote 26 For example, if a well-known celebrity recently died after being implanted with a pacemaker, the risk of pacemaker-related deaths may be perceived as greater than it actually is.

Individuals’ perceptions of risks and benefits can also be influenced by their level of social trust, which has been defined as ‘the willingness to rely on those who have the responsibility for making decisions and taking actions related to the management of technology, the environment, medicine, or other realms of public health and safety’. Footnote 27 In particular, research suggests that, when individuals are considering the risks and benefits of new technologies, their level of social trust has ‘a positive influence on perceived benefits and a negative influence on perceived risks’. Footnote 28 This is not surprising: those who trust that decision-makers will act in their best interests are less likely to be fearful of changes, while those who lack such trust are more likely to be worried about the potential for harm (see Aitken and Cunningham-Burley, Chapter 11 , in this volume).

Compounding these subjective variables is the fact that risk-benefit analysis typically takes place against a backdrop of scientific uncertainty. This is true for all risk-benefit assessments, but it is especially pronounced in research, as the very reason research is conducted is to fill an evidentiary gap. While evaluators can sometimes rely on prior research, including animal studies, to identify the potential harms and benefits of proposed studies, most health-related research takes place in highly controlled environments, over short periods of time. As a result, prior research results are unlikely to provide much information about rare safety risks, long-term dangers or harms and benefits that are limited to discrete population subgroups .

13.5 Weighing Risks and Benefits

Those responsible for reviewing proposed research must ultimately weigh the risks and benefits to determine whether the relationship between them is acceptable. This process is complicated by the fact that risks and benefits often cannot be measured on a uniform scale. First, ‘risks and benefits for subjects may affect different domains of health status’, Footnote 29 as when a risk of physical injury is incurred in an effort to achieve a potential psychological benefit. Second, ‘risks and benefits may affect different people’; Footnote 30 risks are typically borne by the participants in the research, but most of the benefits will be experienced by patients in the future.

Several approaches have been suggested for systematising the process of risk-benefit analysis in research. The first, and most influential, approach is known as ‘component analysis’. This approach calls on decision-makers to independently assess the risks and potential benefits of each intervention or procedure to be used in a study, distinguishing those that have the potential to provide direct benefits to participants (‘therapeutic’) from those that are administered solely for the purpose of developing generalisable knowledge (‘non-therapeutic’). For therapeutic interventions, there must be genuine uncertainty regarding the relative therapeutic benefits of the intervention as compared to those of the standard of care for treating the participants’ condition or disorder (a standard known as ‘clinical equipoise’ Footnote 31 ). For non-therapeutic interventions, the risks must be minimised to the extent consistent with sound scientific design, and the remaining risks must be reasonable in relation to the knowledge that is expected to result. In addition, when a study involves a vulnerable population, such as children or adults who lack decision-making capacity, the risks posed by nontherapeutic procedures may not exceed a ‘minor increase above minimal risk’. Footnote 32

Component analysis has been influential, but it is not universally supported. Some critics maintain that the distinction between therapeutic and non-therapeutic procedures is inherently ambiguous, as ‘all interventions offer at least some very low chance of clinical benefit’. Footnote 33 Others argue that the approach’s reliance on clinical equipoise rests on the mistaken assumption that researchers have a duty to promote each participant’s medical best interests, which conflates the ethics of research with those of clinical care . Footnote 34

One alternative to component analysis is known as the ‘net risk test’, which is based on the principle that the fundamental ethical requirement of research is ‘to protect research participants from being exposed to excessive risks of harm for the benefit of others’. Footnote 35 The approach has four elements. First, for each procedure involved in a study, the risks to participants should be minimised and the potential clinical benefits to participants enhanced, to the extent doing so is consistent with the study’s scientific design. Second, instead of clinical equipoise, the approach requires that, ‘when compared to the available alternatives, a research procedure must not present an excessive increase in risk, or an excessive decrease in potential benefit, for the participant’. Footnote 36 Third, to the extent particular procedures involve greater risks than benefits, those net risks ‘must be justified by the expected knowledge gained from using that procedure in the study’. Footnote 37 Finally, the cumulative net risks of all of the procedures in a study must not be excessive. Footnote 38

Both component analysis and the net risk test can add structure to the process of risk-benefit analysis by focusing attention on the risks and potential benefits of each intervention in a study. The advantage of this approach is that it reduces the likelihood that potential direct benefits from one intervention will be used as a justification for exposing participants to risks from unrelated interventions that offer no direct benefits. However, neither approach eliminates the need for subjective determinations. Under component analysis, the principle of clinical equipoise offers a benchmark for judging the risks and potential benefits of therapeutic procedures, but for non-therapeutic procedures, the only guidance offered is that the risks must be ‘reasonable’ in relation to the knowledge expected to result. The net benefit test dispenses with clinical equipoise entirely, instead relying on a general principle of avoiding ‘excessive risk’. Whether a particular mix of risks and potential benefits is ‘reasonable’ or ‘excessive’ is ultimately left to the judgment of those charged with reviewing the study .

Most regulations and ethics codes provide little guidance on the process of weighing the risks and potential benefits of research. The primary exception is the CIOMS guidelines, which adopts what it describes as a ‘middle ground’ between component analysis and the net risk test. In most respects, the CIOMS approach reflects component analysis, including its reliance on clinical equipoise as a standard for evaluating interventions or procedures that have the potential to provide direct benefits to participants. However, the guidelines also call for a judgment that ‘the aggregate risks of all research interventions or procedures … must be considered appropriate in light of the potential individual benefits to participants and the scientific social value of the research’, Footnote 39 a requirement that mirrors the final step of the net risk test.

Neither component analysis nor the net risk test explicitly sets an upper limit on permissible risk, at least in studies involving competent adults. However, one of the developers of component analysis has stated that ‘the notion of excessive net risks, and the underlying ethical principle of non-exploitation, clearly impose a cap on the risks that individuals are allowed to assume for the benefit of others’. Footnote 40 The notion of an upper limit on risk also appears in several ethical guidelines. For example, the CIOMS guidelines state that ‘some risks cannot be justified, even when the research has great social and scientific value and adults who are capable of giving informed consent would give their voluntary, informed consent to participate in the study’. Footnote 41 Similarly, the European Commission has suggested that certain ‘threats to human dignity and shared values’ should never be traded against the potential scientific benefits of research, including ‘commonly shared values like privacy or free movement … certain perceptions of the integrity of a person (e.g. cloning, technological modifications) … [and] widely shared view[s] of our place in the world (e.g. inhumane treatment of animals or threat to biodiversity)’. Footnote 42

In light of the inherent ambiguities involved in weighing the risks and benefits of research, the results of risk-benefit assessments can be heavily influenced by the type of decision-making process used. The next section looks at these procedural issues more closely .

13.6 Procedural Issues in Risk-Benefit Analysis

In most health-related research, the process of risk-benefit assessment is undertaken by interdisciplinary bodies known as research ethics committees (RECs), research ethics boards (REBs), or institutional review boards (IRBs). These committees make judgments based on predictions about the preferences and attitudes of typical research participants, which do not necessarily reflect how the actual participants would react to particular risk-benefit trade-offs. Footnote 43 In addition, because few committees rely on formal methods of risk-benefit analysis, decisions are likely to be influenced by individual members’ personal attitudes and cognitive biases. Footnote 44 For this reason, it is not surprising that different committees’ assessments of the risks and potential benefits of identical situations exhibit widespread variation. Footnote 45

Some commentators have proposed techniques to promote greater consistency in risk-benefit assessments. For example, it has been suggested that committees issue written assessments that could be entered into searchable databases. Footnote 46 Others have called on committees to engage in a formal process of ‘evidence-based research ethics review’, in which judgments about risks and potential benefits would be informed by a systematic retrieval and critical appraisal of the best available evidence. Footnote 47

Outside of research ethics, a variety of techniques have been developed to systematise the process of risk-benefit analysis. For example, several quantitative approaches to risk-benefit assessment exist, such as the Quality-Adjusted Time Without Symptoms and Toxicity (Q-TWIST) test, which ‘compares therapies in terms of achieved survival and quality-of-life outcomes’, Footnote 48 or the ‘standard gamble’, which assigns utility values to health outcomes based on individuals’ stated choice between hypothetical health risks. Footnote 49 Committees reviewing proposed studies can draw on these quantitative analyses when relevant ones exist.

In some cases, formal consultation with the community from which participants will be drawn can be an important component of assessing risks and benefits. For example, in the study of Havasupai tribe members discussed above, prior consultation with the community could have alerted researchers to the fact that research on migration patterns was threatening to the tribe’s cultural beliefs. In cancer research, consultation with patient advocacy groups may help identify concerns about potential adverse effects that might not have been sufficiently considered by the researchers. Footnote 50 Further lessons might be learned from the the analysis by Chuong and O'Doherty, Chapter 12, this volume .

13.7 Conclusion

Risk-benefit analysis is a critical part of the process of evaluating the ethical acceptability of health-related research. The primary challenge in risk-benefit assessment arises from the fact that perceptions about risks and potential benefits are inherently subjective. Those charged with assessing the ethical acceptability of research should make efforts to incorporate as many different perspectives into the process as possible, to ensure that their decisions do not simply reflect their own idiosyncratic views .

1 J. Lantos et al., ‘ Considerations in the Evaluation and Determination of Minimal Risk in Pragmatic Clinical Trials ’, ( 2015 ) Clinical Trials , 12 ( 5 ), 485 – 493 .

2 N. Eyal et al., ‘ Risk to Study Nonparticipants: A Procedural Approach ’, ( 2018 ) Proceedings of the National Academy of Sciences , 115 ( 32 ), 8051 – 8053 .

3 G. DuVal , ‘ Ethics in Psychiatric Research: Study Design Issues ’, ( 2004 ) Canadian Journal of Psychiatry , 49 ( 1 ), 55 – 59 .

4 A. McGuire et al., ‘ Research Ethics and the Challenge of Whole-Genome Sequencing ’, ( 2008 ) Nature Reviews Genetics , 9 ( 2 ), 152 – 156 .

5 Council for International Organizations of Medical Sciences, ‘International Ethical Guidelines for Health-Related Research Involving Humans’, (CIOMS, 2016), p. 13.

6 M. Mello and L. Wolf , ‘ The Havasupai Indian Tribe Case: Lessons for Research Involving Stored Biologic Samples ’, ( 2010 ) New England Journal of Medicine , 363 ( 3 ), 204 – 207 .

7 Article 28 of the European Union Clinical Trials Regulation 536/2014, OJ 2014 No. L 158/1.

8 The Federal Policy for the Protection of Human Subjects (‘Common Rule’), 45 C.F.R. § 46.111(a)(2) (1991).

9 A. London et al., ‘ Beyond Access vs. Protection in Trials of Innovative Therapies ’, ( 2010 ) Science , 328 ( 5980 ), 829 – 830 , 830.

10 J. Grossman and F. Mackenzie , ‘ The Randomized Controlled Trial: Gold Standard, or Merely Standard? ’, ( 2005 ) Perspectives in Biology & Medicine , 48 ( 4 ), 516 – 534 .

11 J. Younge et al., ‘ Randomized Study Designs for Lifestyle Interventions: A Tutorial ’, ( 2015 ) International Journal of Epidemiology , 44 ( 6 ), 2006 – 2019 .

12 C. J. Mann , ‘ Observational Research Methods. Research Design II: Cohort, Cross Sectional, and Case-Control Studies ’, ( 2003 ) Emergency Medicine Journal , 20 ( 1 ), 54 – 60 .

13 D. Grimes and K. Schulz , ‘ Bias and Causal Associations in Observational Research ’, ( 2002 ) Lancet , 359 ( 9302 ), 248 – 252 .

14 C. Adebamowo et al., ‘ Randomised Controlled Trials for Ebola: Practical and Ethical Issues ’, ( 2014 ) Lancet , 384 ( 9952 ), 1423 – 1424 , 1423.

15 C. Coleman , ‘ Control Groups on Trial: The Ethics of Testing Experimental Ebola Treatments ’, ( 2016 ) Journal of Biosecurity, Biosafety and Biodefense Law , 7 ( 1 ), 3 – 24 , 8.

16 E. Emanuel et al., ‘ What Makes Clinical Research Ethical? ’, ( 2000 ) JAMA , 283 ( 20 ), 2701 – 2711 .

17 R. Lilford and A. Stevens , ‘ Underpowered Studies ’, ( 2002 ) British Journal of Surgery , 89 ( 2 ), 129 – 131 .

18 B. Freedman and S. Shapiro , ‘ Ethics and Statistics in Clinical Research: Towards a More Comprehensive Examination ’, ( 1994 ) Journal of Statistical Planning and Inference , 42 ( 1 ), 223 – 240 .

19 N. King , ‘ Defining and Describing Benefit Appropriately in Clinical Trials ’, ( 2000 ) Journal of Law, Medicine & Ethics , 28 ( 4 ), 332 – 343 .

20 Emanuel et al., ‘What Makes Clinical Research Ethical?’, 2705.

21 See, e.g. A. Wertheimer , ‘ Is Payment a Benefit? ’, ( 2013 ) Bioethics , 27 ( 2 ), 105 – 116 .

22 US Food and Drug Administration, ‘Payment and Reimbursement to Research Subjects’, (US Food and Drug Administration, 2018), www.fda.gov/regulatory-information/search-fda-guidance-documents/payment-and-reimbursement-research-subjects .

23 T. Opsal et al., ‘ “There Are No Known Benefits …” Considering the Risk/Benefit Ratio of Qualitative Research ’, ( 2016 ) Qualitative Health Research , 26 ( 8 ), 1137 – 1150 .

24 C. Troche et al., ‘ Evaluation of Therapeutic Strategies: A New Method for Balancing Risk and Benefit ’, ( 2000 ) Value in Health , 3 ( 1 ), 12 – 22 .

25 P. Slovic , ‘ Trust, Emotion, Sex, Politics, and Science: Surveying the Risk-Assessment Battlefield ’, ( 1999 ) Risk Analysis , 19 ( 4 ), 689 – 701 .

26 T. Pachur et al., ‘ How Do People Judge Risks: Availability Heuristic, Affect Heuristic, or Both? ’, ( 2012 ) Journal of Experimental Psychology: Applied , 18 ( 3 ), 314 – 330 .

27 M. Siegrist et al., ‘ Salient Value Similarity, Social Trust, and Risk/Benefit Perception ’, ( 2000 ) Risk Analysis , 20 ( 3 ), 353 – 362 , 354.

28 Footnote Ibid ., 358.

29 D. Martin et al., ‘ The Incommensurability of Research Risks and Benefits: Practical Help for Research Ethics Committees ’, ( 1995 ) IRB: Ethics & Human Research , 17 ( 2 ), 8 – 10 , 9.

30 Footnote Ibid ., 8.

31 B. Freedman , ‘ Equipoise and the Ethics of Clinical Research ’, ( 1987 ) New England Journal of Medicine , 317 ( 3 ), 141 – 145 .

32 C. Weijer , ‘ The Ethical Analysis of Risks and Potential Benefits in Human Subjects Research: History, Theory, and Implications for US Regulation ’ in National Bioethics Advisory Commission, Ethical and Policy Issues in Research Involving Human Participants. Volume II – Commissioned Papers and Staff Analysis ( Bethesda, MD : National Bioethics Advisory Commission ), pp. 1 – 29 , p. 24.

33 A. Rid and D. Wendler , ‘ Risk-Benefit Assessment in Medical Research – Critical Review and Open Questions ’, ( 2010 ) Law, Probability and Risk , 9 ( 3 –4), 151 – 177 , 157.

34 Footnote Ibid ., 158.

35 Footnote Ibid ., 164.

36 Footnote Ibid.

37 Footnote Ibid.

38 D. Wendler and F. Miller , ‘ Assessing Research Risks Systematically: The Net Risks Test ’, ( 2007 ) Journal of Medical Ethics , 33 ( 8 ), 481 – 486 .

39 Council for International Organizations of Medical Sciences, ‘International Ethical Guidelines’, xi, 9.

40 Wendler and Miller, ‘Assessing Research Risks Systematically’, 165.

41 Council for International Organizations of Medical Sciences, ‘International Ethical Guidelines’, 10.

42 European Commission Directorate-General for Research and Innovation, ‘Research and Innovation, Research, Risk-Benefit Analyses, and Ethical Issues’, (European Union, 2013).

43 M. Meyer , ‘ Regulating the Production of Knowledge: Research Risk-Benefit Analysis and the Heterogeneity Problem ’, ( 2013 ) Administrative Law Review , 65 ( 2 ), 241 – 242 .

44 C. Coleman , ‘ Rationalizing Risk Assessment in Human Subject Research ’, ( 2004 ) Arizona Law Review , 46 ( 1 ), 1 – 51 .

45 T. Caulfield , ‘ Variation in Ethics Review of Multi-Site Research Initiatives ’, ( 2011 ) Amsterdam Law Forum , 3 ( 1 ), 85 – 100 .

46 Coleman, ‘Rationalizing Risk Assessment’, 1176–1179.

47 E. Anderson and J. DuBois , ‘ Decision-Making with Imperfect Knowledge: A Framework for Evidence-Based Research Ethics ’, ( 2012 ) Journal of Law, Medicine and Ethics , 40 ( 4 ), 951 – 966 .

48 Troche et al., ‘Evaluation of Therapeutic Strategies’, 13.

49 S. van Osch and A. Stiggelbout , ‘ The Construction of Standard Gamble Utilities ’, ( 2008 ) Health Economics , 17 ( 1 ), 31 – 40 .

50 N. Dickert and J. Sugarman , ‘ Ethical Goals of Community Consultation in Research ’, ( 2005 ) American Journal of Public Health , 95 ( 7 ), 1123 – 1127 .

Save book to Kindle

To save this book to your Kindle, first ensure [email protected] is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle .

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service .

• Risk-Benefit Analysis
• By Carl H. Coleman
• Edited by Graeme Laurie , University of Edinburgh , Edward Dove , University of Edinburgh , Agomoni Ganguli-Mitra , University of Edinburgh , Catriona McMillan , University of Edinburgh , Emily Postan , University of Edinburgh , Nayha Sethi , University of Edinburgh , Annie Sorbie , University of Edinburgh
• Book: The Cambridge Handbook of Health Research Regulation
• Online publication: 09 June 2021
• Chapter DOI: https://doi.org/10.1017/9781108620024.017

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox .

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive .

Conducting Risk-Benefit Assessments and Determining Level of IRB Review

Regulatory background.

Investigators should understand the concept of minimizing risk when designing research and conduct a risk-benefit assessment to determine the level of IRB review of the research. In the protocol application the Investigator should:

• Assess potential risks and discomforts associated with each intervention or research procedure;
• Estimate the probability that a given harm may occur and its severity;
• Explain measures that will be taken to prevent and minimize potential risks and discomforts;
• Describe the benefits that may accrue directly to subjects; and
• Discuss and the potential societal benefits that may be expected from the research.

Risks to subjects who participate in research should be justified by the anticipated benefits to the subject or society. This requirement is found in all codes of research ethics, and is a central requirement in the Federal regulations ( 45 CFR 46.111  and  21 CFR 56.111 ). Two of the required criteria for granting IRB approval of the research are:

• Risks to subjects are  minimized  by using procedures which are consistent with sound research design and which do not unnecessarily expose subjects to risk, and whenever appropriate, by using procedures already being performed on the subjects for diagnostic or treatment purposes.
• Risks to subjects are  reasonable  in relation to anticipated benefits, if any, to subjects, and the importance of the knowledge that may reasonably be expected to result. In evaluating risks and benefits, the IRB Committee will consider  only those risks and benefits that may result from the research , as distinguished from risks and benefits of therapies subjects would receive even if not participating in the research.

Definitions

Benefit:  A helpful or good effect, something intended to help, promote or enhance well-being; an advantage.

Risk: The probability of harm or injury (physical, psychological, social, or economic) occurring as a result of participation in a research study. Both the probability and magnitude of possible harm may vary from minimal to significant.

Minimal Risk:  A risk is minimal when “the probability and magnitude of harm or discomfort anticipated in the proposed research are not greater in and of themselves than those ordinarily encountered in  daily life of the general population  or during the performance of  routine physical or psychological examinations or tests .” Examples of procedures that typically are considered no more than minimal risk include: collection of blood or saliva, moderate exercise, medical record chart reviews, quality of life questionnaires and focus groups. See Expedited review categories for a complete listing.

Minimal Risk for Research involving Prisoners:  The definition of minimal risk for research involving prisoners differs somewhat from that given for non-institutionalized adults. A risk is minimal when, "the probability and magnitude of physical or psychological harm that is normally encountered in the daily lives, or in the routine medical, dental or psychological examinations of  healthy persons ."

Privacy:  Privacy is about people and their sense of being in control of others access to them or to information about themselves.

Confidentiality:  Confidentiality is about how identifiable, private information that has been disclosed to others is used and stored. People share private information in the context of research with the expectation that it be kept confidential and will not be divulged except in ways that have been agreed upon.

Types of Risks to Research Subjects

Physical Harms:  Medical research often involves exposure to pain, discomfort, or injury from invasive medical procedures, or harm from possible side effects of drugs, devices or new procedures. All of these should be considered "risks" for purposes of IRB review.

• Some medical research is designed only to measure the effects of therapeutic or diagnostic procedures applied in the course of caring for an illness. Such research may not entail any significant risks beyond those presented by medically indicated interventions.
• Research designed to evaluate new drugs, devices or procedures typically present more than minimal risk and involve risks that are unforeseeable that could cause serious or disabling injuries.

Psychological Harms:  Participation in research may result in undesired changes in thought processes and emotion (e.g., episodes of depression, confusion, feelings of stress, guilt, and loss of self-esteem). Most psychological risks are minimal or transitory, but some research has the potential for causing serious psychological harm.

• Stress and feelings of guilt or embarrassment may arise from thinking or talking about one's own behavior or attitudes on sensitive topics such as drug use, sexual preferences, selfishness, and violence.
• Stress may be induced when the researchers manipulate the subjects' environment to observe their behaviors and reactions. The possibility of psychological harm is heightened when behavioral research involves an element of deception.

Social and Economic Harms:  Some losses of privacy and breaches of confidentiality may result in embarrassment within one's business or social group, loss of employment, or criminal prosecution.

• Areas of particular sensitivity involve information regarding alcohol or drug abuse, mental illness, illegal activities, and sexual behavior.
• Some social and behavioral research may yield information about individuals that could be considered stigmatizing to individual subjects or groups of subjects. (e.g., as actual or potential carriers of a gene; individuals prone to alcoholism). Confidentiality safeguards must be strong in these instances.
• Participation in research may result in additional actual costs to individuals. Any anticipated costs to research participants should be described to prospective subjects during the consent process.

Privacy Risks:  Loss of privacy in the research context usually involves either covert observation or participant observation of behavior that the subjects consider private. It can also involve access and use of private information about the subjects. The IRB must make two determinations:

• Is the loss of privacy involved acceptable in light of the subjects' reasonable expectations of privacy in the situation under study; and
• Is the research question of sufficient importance to justify the intrusion?

Breach of Confidentiality Risks:  Absolutely confidentiality cannot be guaranteed and is always a potential risk of participation in research. A breach of confidentiality is sometimes confused with loss of privacy, but it is a different risk. Loss of privacy concerns access to private information about a person or to a person's body or behavior without consent; confidentiality of data concerns safeguarding information that has been given voluntarily by one person to another. It is important to recognize that a breach of confidentiality may result in psychological harm to individuals (embarrassment, guilt, stress, etc.) or in social harm.

Conducting Risk-Benefit Assessments

Role of the Investigator:  When designing research studies, investigators are responsible for conducting an initial risk-benefit assessment using the steps outlined in the diagram below.

Role of the IRB:  The IRB ultimately is responsible for evaluating the potential risks and weighing the probability of the risk occurring and the magnitude of harm that may result. It must then judge whether the anticipated benefit, either of new knowledge or of improved health for the research subjects, justifies asking any person to undertake the risks. The IRB cannot approve research in which the risks are judged unreasonable in relation to the anticipated benefits. The IRB must:

• Identify the risks associated with the research, as distinguished from the risks of therapies the subjects would receive even if not participating in research;
• Determine that the risks will be minimized to the extent possible;
• Identify the probable benefits to be derived from the research;
• Determine that the risks are reasonable in relation to be benefits to subjects, if any, and the importance of the knowledge to be gained; and
• Assure that potential subjects will be provided with an accurate and fair description (during consent) of the risks or discomforts and the anticipated benefits.

Diagram 1: Steps for Conducting a Risk-Benefit Assessment

Ways to Minimize Risk

• Provide complete information in the protocol regarding the experimental design and the scientific rationale underlying the proposed research, including the results of previous animal and human studies.
• Assemble a research team with sufficient expertise and experience to conduct the research.
• Ensure that the projected sample size is sufficient to yield useful results.
• Collect data from conventional (standard) procedures to avoid unnecessary risk, particularly for invasive or risky procedures (e.g., spinal taps, cardiac catheterization).
• Incorporate adequate safeguards into the research design such as an appropriate data safety monitoring plan, the presence of trained personnel who can respond to emergencies.
• Store data in such a way that it is impossible to connect research data directly to the individuals from whom or about the data pertain; limit access to key codes and store separately from the data.
• Incorporate procedures to protect the confidentiality of the data (e.g., encryption, codes, and passwords) and follow UCLA IRB guidelines on  Data Security in Research .

Levels of IRB Review

Exempt research.

Although the category is called "exempt," this type of research does require IRB review and registration. The exempt registration process is much less rigorous than an expedited or full-committee review. To qualify, research must fall into 8 federally-defined exempt categories. These categories present the lowest amount of risk to potential subjects because, generally speaking, they involve either collection of anonymous or publicly-available data, or conduct of the least potentially-harmful research experiments. For additional information see OHRPP Exempt Guidance .

• Anonymous surveys or interviews
• Passive observation of public behavior without collection of identifiers
• Retrospective chart reviews with no recording of identifiers
• Analyses of discarded pathological specimens without identifiers

Expedited Research

To qualify for an expedited review, research must be no more than minimal risk and fall into nine (9) federally-defined expedited categories. These categories involve collection of samples and data in a manner that is not anonymous and that involves no more than minimal risk to subjects. For additional information see  OHRPP Expedited Guidance .

• Surveys and interviews with collection of identifiers
• Collection of biological specimens (e.g., hair, saliva) for research by noninvasive means
• Collection of blood samples from healthy volunteers
• Studies of existing pathological specimens with identifiers

Full Board Research

Proposed human subject research that does not fall into either the exempt or expedited review categories must be submitted for full committee review. This is the most rigorous level of review and, accordingly, is used for research projects that present greater than minimal risk to subjects. The majority of biomedical protocols submitted to the IRB require full Committee review. For additional information see  OHRPP Full Board Guidance .

• Clinical investigations of drugs and devices
• Studies involving invasive medical procedures or diagnostics
• Longitudinal interviews about illegal behavior or drug abuse
• Treatment interventions for suicidal ideation and behavior

Regulations and References

• DHHS 45 CFR 46.110
• DHHS 45 CFR 46.111(a)(1-2)
• FDA 21 CFR 56.110
• FDA 21 CFR 56.111(a)(1-2)
• OHRP IRB Guidebook, Chapter 3: Basic IRB Review, Section A, Risk/Benefit Analysis

Human Subjects Protection

You are here, protocol design - assessment of the risk-benefit relationship.

Investigators and IRB members have a responsibility to perform an assessment of the risks and potential benefits of a research protocol in accordance with the principle of beneficence, as defined in the Belmont Report, which states that risks must be minimized and the risks and benefits must be shown to be in a favorable ratio. As participants in research, subjects may be exposed to complex activities consisting of various procedures and interventions. These procedures and interventions may be administered to subjects for different reasons, either therapeutic or nontherapeutic. The first step in evaluating a protocol’s risk-benefit relationship is to classify the procedures and interventions, the research components, that present risks as either therapeutic or nontherapeutic. It is the intentof the intervention or procedure, therapeutic or nontherapeutic, that drives the moral analysis of these components. A therapeutic intervention or procedure is administered with the intent of providing direct benefit to the research subject. A nontherapeutic intervention or procedure is administered solely for scientific purposes. This distinction between research components prevents the justification of risky nontherapeutic procedures by the benefits that may flow from therapeutic procedures. Decisions regarding the appropriateness of interventions or procedures that are therapeutic are made exactly as they are in clinical practice (i.e., the associated risks are justified exclusively in terms of the degree of benefit that can be expected to accrue to the subject). Risks associated with nontherapeutic procedures or interventions must be justified by the importance of the generalizable knowledge that may be expected to result from the research study. Research risks are reasonable in relation to the anticipated benefits when the IRB determines that the moral standards for both therapeutic and nontherapeutic procedures are fulfilled. This ethical assessment of a protocol’s procedures and interventions is called a component analysis.

At first glance, a component​ analysis of a research protocol seems quite tedious and time-consuming. However, in practice it can typically be accomplished quite easily. The first step is to take a global look at the risks posed by the various therapeutic and nontherapeutic procedures and interventions in a protocol and determine if any exceed minimal risk. The definition of minimal risk and a description of how IRBs utilize the concept can be found at this  link . If no procedures or interventions exceed minimal risk, a component analysis is not required and the protocol is deemed of minimal risk. Investigators are simply responsible for ensuring that all risks are minimized before submitting the protocol for review by the IRB. The IRB will review minimal-risk protocols according to its policies. If a protocol contains therapeutic or nontherapeutic components that exceed minimal risk, then a component analysis is required. The specific assessment conducted on each greater-than-minimal-risk component is dependent on whether it is a therapeutic or nontherapeutic component. The following link  provides a social-behavioral and biomedical example of the distinction between therapeutic and nontherapeutic components of a protocol.

Assessing Nontherapeutic Procedures and Interventions

After distinguishing between therapeutic and nontherapeutic procedures and interventions and ensuring that all risks are minimized, all nontherapeutic procedures and interventions that present, or appear to present, more than minimal risk must undergo a component analysis to properly assess the risk-benefit relationship. A component analysis requires investigators to assess whether the risks expected to accrue to the subject from each nontherapeutic procedure or intervention are reasonable in relation to the anticipated benefits. The knowledge to be gained from that procedure (i.e., the scientific “value added” of this procedure) is the only likely benefit, although any indirect benefit that the subject derives directly and exclusively from the procedure may also be factored in. If the risks are found to be reasonable relative to these anticipated benefits, the nontherapeutic component is considered ethical. Healthy adult research subjects may assume significant risks, as long as these risks have been properly disclosed during the informed-consent process and are commensurate with the knowledge to be gained or the scientific value of the study and any other benefit derived from the research. The investigator’s risk-benefit assessment of the nontherapeutic procedure or intervention is then reviewed by the IRB. To assist in the deliberations, the IRB may draw upon the opinions of experts from relevant disciplines as well as representatives of the community. Once the IRB has concluded that the risks associated with specific nontherapeutic components of the protocol are justified by the potential value of the knowledge to be gained, the protocol can be approved. Because vulnerable subjects may have difficulty providing informed consent (understanding and protecting their own interests) or because their circumstances may subject them to intimidation and exploitation, the degree of risk associated with a nontherapeutic procedure or intervention to which a vulnerable subject may be exposed is limited to minimal or a minor increase over minimal (assuming other requirements for enrolling vulnerable subjects are met).  Module 7 contains additional information on vulnerable subjects.

Assessing Therapeutic Procedures and Interventions 

Therapeutic procedures and interventions are those administered with the intent of directly benefiting the subject. Once a specific procedure or intervention in a protocol has been categorized as therapeutic, its justification is determined in exactly the same manner as in clinical practice (i.e., the risks associated with specific procedures or interventions are justified exclusively in terms of the degree of benefit that is expected to accrue to the subject). It is a subject-specific risk-benefit judgment. The only ceiling for the probability and magnitude of risk from therapeutic or beneficial procedures is that they are not to exceed those of the benefits that can reasonably be expected to accrue to the subject. An additional requirement is that therapeutic procedures and interventions offered the subject while participating in the protocol must be at least as advantageous to the subject as any available alternative procedures and interventions (unless the subject has considered and refused to accept a superior alternative). Two alternative procedures or interventions that are considered to be equally advantageous to the subject are considered to be in a state of clinical equipoise. A brief discussion of clinical equipoise is available at this  link .

When evaluating a study with one or more therapeutic procedures or components, the IRB must take reasonable steps to be assured that the risks are reasonable relative to the benefits expected to accrue to the subject and that a state of clinical equipoise exists for each of the therapeutic procedures or components. This will involve a critical evaluation of the study’s justification and, in selected cases, a review of the medical literature or consultation with relevant experts. This assessment should take into account the efficacy of the treatment or procedure, side effects, ease of administration, and similar issues.

Note –  The framework or model for analyzing the risk-benefit relationship in research protocols was formulated by the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research during the late 1970s and culminated in the concept of component analysis. Some federal regulations were written before the National Commission had fully articulated the concept of component analysis (e.g., 45 CFR 46, Subparts B and C, which outline special protections for pregnant women, human fetuses, neonates and prisoners). The existence of regulations based on different models for analyzing the risk-benefit relationship has led to some confusion regarding the appropriate model investigators and IRB members should follow. The use of component analysis, as described above, will lead to ethical decisions.

• HRP Staff Directory
• Office Hours
• Quality Improvement Project vs. Research
• Self Exempt & UROP
• Case Reports
• Commercial IRB Reliance Agreements
• National Cancer Institute Central IRB (CIRB) Independent Review Process
• UCI as the Reviewing IRB
• Submitting the Application
• Lead Researcher Eligibility
• Training & Education
• Ethical Guidelines, Regulations and Statutes
• Other Institutional Requirements
• Department of Defense Research Requirements
• Levels of Review
• Artificial Intelligence and Human Subject Research
• Data Security
• Protected Health Information (HIPAA)
• European Union General Data Protection Regulation (EU GDPR)
• China’s Personal Information Protection Law
• Required Elements of Informed Consent
• Drafting the Informed Consent Form
• Consent and Non-English or Disabled Subjects
• Use Of Surrogate Consent In Research
• Vulnerable Populations
• Data and Safety Monitoring for Clinical Research
• Placebo-Controlled Studies
• Expanded Access to Unapproved Drugs or Biologics
• Right to Try: Unapproved Drugs or Biologics
• Use of Controlled Substances
• Expanded Access to Unapproved Medical Devices
• Humanitarian Use Devices
• Human Gene Transfer Research
• How To Register and Update Your Study
• Post-Review Responsibilities

Assessing Risks and Benefits

The IRB is responsible for evaluating the potential risks and weighing the probability of the risk occurring and the magnitude of harm that may result. It must then judge whether the anticipated benefit, either of new knowledge or of improved health for the research subjects, justifies inviting any person to undertake the risks.

Per DHHS and FDA regulations ( 45 CFR 46.111 and 21 CFR 56.111 ) two of the required criteria for granting IRB approval of research are:

• Risks to subjects are minimized by using procedures which are consistent with sound research design and which do not unnecessarily expose subjects to risk, and whenever appropriate, by using procedures already being performed on the subjects for diagnostic or treatment purposes.
• Risks to subjects are reasonable in relation to anticipated benefits, if any, to subjects, and the importance of the knowledge that may reasonably be expected to result. In evaluating risks and benefits, the IRB Committee will consider only those risks and benefits that may result from the research, as distinguished from risks and benefits of therapies subjects would receive even if not participating in the research.
• Benefit A valued or desired outcome; an advantage.
• Risk The probability of harm or injury (physical, psychological, social, or economic) occurring as a result of participation in a research study. Both the probability and magnitude of possible harm may vary from minimal to significant. Federal regulations define only "minimal risk."
• Minimal Risk A risk is minimal where the probability and magnitude of harm or discomfort anticipated in the proposed research are not greater, in and of themselves, than those ordinarily encountered in daily lives of the general population or during the performance of routine physical or psychological examinations or tests.
• Minimal Risk for Research involving Prisoners The definition of minimal risk for research involving prisoners differs somewhat from that given for non-institutionalized adults. Minimal risk is in this case is defined as, "the probability and magnitude of physical or psychological harm that is normally encountered in the daily lives, or in the routine medical, dental or psychological examinations of healthy persons."

There are two sources of confusion in the assessment of risks and benefits. One arises from the language employed in the discussion:

• "Risk" is a word expressing probabilities;
• "Benefits" is a word expressing a fact or state of affairs.

It is more accurate to speak as if both were in the realm of probability: i.e., risks and expected or anticipated benefits. Confusion also may arise because "risks" can refer to two quite different things:

• those chances that specific individuals are willing to undertake for some desired goal; or
• the conditions that make a situation harmful to a subject.

Researchers should provide detailed information in the IRB application about potential risks and benefits associated with the research, and provide information about the probability, magnitude and potential harms associated with each risk.

The IRB cannot approve research in which the risks are judged unreasonable in relation to the anticipated benefits. The IRB must:

• As applicable, evaluate the available clinical and nonclinical information on an investigational product to determine if the data is adequate to support the proposed clinical trial;
• Determine that the risks will be minimized to the extent possible [see below];
• Identify the probable benefits to be derived from the research;
• Determine that the risks are reasonable in relation to be benefits to subjects , if any, and the importance of the knowledge to be gained; and
• Assure that potential subjects will be provided with an accurate and fair description (during consent) of the risks or discomforts and the anticipated benefits.

The risks to which research subjects may be exposed have been classified as physical, psychological, social, and economic .

• Physical Harms Medical research often involves exposure to minor pain, discomfort, or injury from invasive medical procedures, or harm from possible side effects of drugs. All of these should be considered "risks" for purposes of IRB review. Some of the adverse effects that result from medical procedures or drugs can be permanent, but most are transient. Procedures commonly used in medical research usually result in no more than minor discomfort (e.g., temporary dizziness, the pain associated with venipuncture).Some medical research is designed only to measure more carefully the effects of therapeutic or diagnostic procedures applied in the course of caring for an illness. Such research may not entail any significant risks beyond those presented by medically indicated interventions. On the other hand, research designed to evaluate new drugs or procedures may present more than minimal risk, and, on occasion, can cause serious or disabling injuries.
• Psychological Harms Participation in research may result in undesired changes in thought processes and emotion (e.g., episodes of depression, confusion, or hallucination resulting from drugs, feelings of stress, guilt, and loss of self-esteem). These changes may be transitory, recurrent, or permanent. Most psychological risks are minimal or transitory, but some research has the potential for causing serious psychological harm.Stress and feelings of guilt or embarrassment may arise simply from thinking or talking about one's own behavior or attitudes on sensitive topics such as drug use, sexual preferences, selfishness, and violence. These feelings may be aroused when the subject is being interviewed or filling out a questionnaire. Stress may also be induced when the researchers manipulate the subjects' environment - as when "emergencies" or fake "assaults" are staged to observe how passersby respond. More frequently, however, is the possibility of psychological harm when behavioral research involves an element of deception.
• Is the invasion of privacy involved acceptable in light of the subjects' reasonable expectations of privacy in the situation under study;
• Is the research question of sufficient importance to justify the intrusion?
• The IRB must also consider whether the research design could be modified so that the study can be conducted without invading the privacy of the subjects.
• Note: Breach of confidentiality is sometimes confused with invasion of privacy, but it is really a different risk. Invasion of privacy concerns access to a person's body or behavior without consent; confidentiality of data concerns safeguarding information that has been given voluntarily by one person to another.
• Some research requires the use of a subject's hospital, school, or employment records. Access to such records for legitimate research purposes is generally acceptable, as long as the researcher protects the confidentiality of that information. However, it is important to recognize that a breach of confidentiality may result in psychological harm to individuals (in the form of embarrassment, guilt, stress, and so forth) or in social harm (see below).
• Social and Economic Harms Some invasions of privacy and breaches of confidentiality may result in embarrassment within one's business or social group, loss of employment, or criminal prosecution. Areas of particular sensitivity are information regarding alcohol or drug abuse, mental illness, illegal activities, and sexual behavior. Some social and behavioral research may yield information about individuals that could "label" or "stigmatize" the subjects. (e.g., as actual or potential delinquents or schizophrenics). Confidentiality safeguards must be strong in these instances. Participation in research may result in additional actual costs to individuals. Any anticipated costs to research participants should be described to prospective subjects during the consent process.
• Provide complete information in the protocol regarding the experimental design and the scientific rationale underlying the proposed research, including the results of previous animal and human studies.
• Assemble a research team with sufficient expertise and experience to conduct the research.
• Ensure that the projected sample size is sufficient to yield useful results.
• Collect data from standard-of-care procedures to avoid unnecessary risk, particularly for invasive or risky procedures (e.g., spinal taps, cardiac catheterization).
• Incorporate adequate safeguards into the research design such as an appropriate data safety monitoring plan, the presence of trained personnel who can respond to emergencies, and procedures to protect the confidentiality of the data (e.g., encryption, codes, and passwords).

• Open access
• Published: 20 April 2012

The risk-benefit task of research ethics committees: An evaluation of current approaches and the need to incorporate decision studies methods

• Rosemarie D L C Bernabe 1 ,
• Ghislaine J M W van Thiel 1 ,
• Jan A M Raaijmakers 2 &
• Johannes J M van Delden 1  

BMC Medical Ethics volume  13 , Article number:  6 ( 2012 ) Cite this article

15k Accesses

17 Citations

1 Altmetric

Metrics details

Research ethics committees (RECs) are tasked to assess the risks and the benefits of a trial. Currently, two procedure-level approaches are predominant, the Net Risk Test and the Component Analysis.

By looking at decision studies, we see that both procedure-level approaches conflate the various risk-benefit tasks, i.e., risk-benefit assessment, risk-benefit evaluation, risk treatment, and decision making. This conflation makes the RECs’ risk-benefit task confusing, if not impossible. We further realize that RECs are not meant to do all the risk-benefit tasks; instead, RECs are meant to evaluate risks and benefits, appraise risk treatment suggestions, and make the final decision.

As such, research ethics would benefit from looking beyond the procedure-level approaches and allowing disciplines like decision studies to be involved in the discourse on RECs’ risk-benefit task.

Peer Review reports

Research ethics committees (RECs) are tasked to do a risk-benefit assessment of proposed research with human subjects for at least two reasons: to verify the scientific/social validity of the research since an unscientific research is also an unethical research; and to ensure that the risks that the participants are exposed to are necessary, justified, and minimized [ 1 ].

Since 1979, specifically through the Belmont Report, the requirement for a “systematic, nonarbitrary analysis of risks and benefits” has been called for, though up to the present, commentaries about the lack of a generally acknowledged suitable risk-benefit assessment method continue [ 1 ]. The US National Bioethics Advisory Commission (US-NBAC), for example, stated the following in its 2001 report on Ethical and Policy issues in Research Involving Human Participants:

"An IRB’s 1

An institutional review board (IRB) is synonymous to an ethics committee. For consistency’s sake, we shall use REC throughout this paper.

assessment of risks and potential benefits is central to determining that a research study is ethically acceptable and would protect participants, which is not an easy task, because there are no clear criteria for IRBs to use in judging whether the risks of research are reasonable in relation to what might be gained by the research participant or society [ 2 ]."

The lack of a universally accepted risk-benefit assessment criteria does not mean that the research ethics literature says nothing about it. Within this same 2001 report, the US-NBAC recommended Weijer and Miller’s Component Analysis to RECs in evaluating clinical researches. As a reaction to Weijer and P. Miller, Wendler and F. Miller proposed the Net Risk Test. For convenience sake, we shall use the term “procedure-level approaches” [ 3 ] to refer to the models of Weijer et al. and Wendler et al.

In spite of their ideological differences, both procedure-level approaches are procedural in the sense that both approaches propose a step-by-step process in doing the risk-benefit assessment. In this paper, we shall not tackle their differences; rather, we are more interested in their similarities. We are of the position that both approaches fall short of providing an evaluation procedure that is systematic and nonarbitrary precisely because they conflate the various risk-benefit tasks, i.e., risk-benefit analysis, risk-benefit evaluation, risk treatment, and decision making [ 4 – 6 ]. As such, we recommend clarifying what these individual tasks refer to, and to whom these tasks must go. Lastly, we shall assert that RECs would benefit by looking into the current inputs of decision studies on the various risk-benefit tasks.

The procedure-level approaches

Charles Weijer and Paul Miller’s Component Analysis (Figure 1 ) requires research protocol procedures or “components” to be evaluated separately, since the probable benefits of one component must not be used to justify the risks that another component poses [ 2 ]. In this system, RECs would need to make a distinction between procedures in the protocol that are with and those that are without therapeutic warrant since therapeutic procedures would need to be analyzed differently compared to those that are non-therapeutic. It works on the assumption that a therapeutic warrant, that is, the reasonable belief that participants may directly benefit from a procedure, would justify more risks for the participants [ 7 ]. As such, therapeutic procedures ought to be evaluated based on the following conditions, in chronological order: that clinical equipoise exists, that is, that there is an “honest professional disagreement in the community of expert practitioners as to the preferred treatment” [ 8 ]; the “procedure is consistent with competent care; and risk is reasonable in relation to potential benefits to subjects” [ 7 ]. Non-therapeutic procedures, on the other hand, would need to be evaluated on the following conditions: the “risks are minimized and are consistent with sound scientific design; risks are reasonable in relation to knowledge to be gained; and if vulnerable population is involved, (there must be) no more than minor increase over minimal risk” [ 7 ]. Lastly, the REC would need to determine if both therapeutic and non-therapeutic procedures are acceptable [ 7 ]. If all components “pass”, then the “research risks are reasonable in relation to anticipated benefits” [ 7 ].

Component Analysis [ 7 , 9 ].

David Wendler and Franklin Miller, on the other hand, developed the Net-Risk Test (Figure 2 ) as a reaction to the Component Analysis. This system requires RECs to first “minimize the risks of all interventions included in the study” [ 10 ]. After which, the REC ought to review the remaining risks by first looking at each intervention in the study, and evaluating if the intervention “offers a potential for clinical benefit that compensates for its risks and burdens” [ 10 ]. If an intervention does offer a potential benefit that can compensate for the risks, then the intervention is acceptable; otherwise, the REC would need to determine whether the net risk is “sufficiently low and justified by the social value of the intervention” [ 10 ]. By net risk, they refer to the “risks of harm that are not, or not entirely, offset or outweighed by the potential clinical benefits for participants” [ 11 ]. If the net risks are sufficiently low and are justified by the social value of the intervention, then the intervention is acceptable; otherwise, it is not. Lastly, the REC would need to “calculate the cumulative net risks of all the interventions…and ensure that, taken together, the cumulative net risks are not excessive” [ 10 ].

The Net Risk Test [ 10 ].

Recently, Rid and Wendler elaborated the Net Risk Test through a seven-step framework (see Figure 3 ) that is meant to offer a chronological, “systematic and comprehensive guidance” for the risk-benefit evaluations of RECs [ 11 ]. As we could see from Figure 3 , most of the steps are the same as that of the previously explained Net Risk Test; the main addition of the framework is the first step, which is to ensure and enhance the study’s social value. In this first step, Rid and Wendler meant that RECs, at the start of their risk-benefit evaluation, ought to “ensure the study methods are sound”; “ensure that the study passes a minimum threshold of social value”; and “enhance the knowledge to be gained from the study” [ 11 ]. It is only after the social value of the study has been identified, evaluated, and enhanced could the RECs identify the individual interventions and then go through the other steps, i.e., the steps we have earlier discussed in the Net Risk Test.

Seven-step framework for risk-benefit evaluations in biomedical research [ 11 ].

The procedure-level approaches and the conflation of risk-benefit analysis, risk-benefit evaluation, risk treatment, and decision making

These procedure-level approaches may be credited for providing some form of a framework for the risk-benefit assessment tasks of RECs. They have also provided RECs with a framework that includes and puts into perspective certain ethical concepts that may or may not have been considered in REC evaluations, but are now procedurally necessary concepts. Weijer and Miller, for example, made it necessary for RECs to always consider therapeutic warrant, equipoise, and minimal risk when evaluating the risk-benefit balance of a study. Wendler and Miller on the other hand, provided RECs with the concept of net risk. In spite of these contributions, these approaches presuppose (maybe unwittingly) that risk-benefit analysis, risk-benefit evaluation, risk treatment, and decision making can all be conflated. This, in our view, is a major error that ought to be corrected since from this error flow other problems, problems that unavoidably make the procedures unsystematic and arbitrary. To substantiate our view, we first have to make a necessary detour by discussing the distinction between risk-benefit analysis, risk-benefit evaluation, risk treatment, and decision making [ 4 , 5 ]. After which, we shall show how the conflation is present in the procedure-level approaches and how such a conflation leads to difficult problems.

Distinction between risk-benefit analysis, risk-benefit evaluation, risk treatment, and decision making

Decisions on benefits and risks in fact involve four activities: risk-benefit analysis, risk-benefit evaluation, risk treatment, and decision making [ 4 – 6 ]. In the current debate, these terms are used as if they are interchangeable. Precisely because these four activities have four different demands, it must be made clear that the problem is not merely on terminological preference; that is, the problem cannot be solved by simply “agreeing” to use one term over another. In risk studies, the risk-benefit task concretely demands four separate activities [ 4 , 6 ]. Hence, these terms are not interchangeable, and their order must be chronological. The distinctions among these tasks and the necessity of their chronological ordering are as follows.

Risk-benefit analysis refers to the “systematic use of information to identify initiating events, causes, and consequences of these initiating events, and express risk (and benefit)” [ 4 ]. This, risk-benefit analysis refers to 1.) gathering of risk and benefit events, causes, and consequences; and 2.) presenting this wealth of information in a systematic and comprehensive way, in accordance with the purpose why such information is systematized in the first place. There are a number of risk analysis methods such as fault tree analysis, event tree analysis, Bayesian networks, Monte Carlo simulation, and others [ 4 ]. The multi criteria decision analysis (MCDA) method, mentioned by the EU Committee for Medicinal Products for Human Use (CHMP) in the Reflection Paper on Benefit Risk Assessment Methods in the Context of the Evaluation of Marketing Authorization Applications of Medicinal Products for Human Use [ 12 ] , proposes the use of a value tree in analyzing the risk-benefit balance of a drug, for example. Adjusted to drug trials, a drug trial risk-benefit analysis value tree could look like (Figure 4 ).

Risk-benefit analysis value tree .

In this value tree (Figure 4 ), we used King and Churchill’s typology of harms and benefits [ 1 ]. From each of the branches, the risk analyst would fill in information about a specific study. Of course, there could be more than one input under each category, depending on the nature of the drug trial being analyzed. Also, this value tree serves as an example; this is not the only way that benefits and risks may be analyzed within the context of drug trials. The best way to analyze risks and benefits within this context is something that ought to be further discussed and developed. Our aim is simply to show that a method such as a value tree is capable of encapsulating and framing the multidimensional nature of the causes and consequences of the benefits and risks of a study within one “tree.” This provides a functional risk-benefit picture from which the risks and the benefits may be evaluated, i.e., risk-benefit evaluation.

Risk-benefit evaluation refers to the “process of comparing risk (and benefit) against given risk (and benefit) criteria to determine the significance of the risk (and the benefit)” [ 4 ]. There are a number of methods to evaluate benefits and risks. Within the MCDA model for example, the “identification of the risk-benefit criteria; assessment of the performance of each option against the criteria; the assignment of weight to each criterion; and the calculation of the weighted scores at each level and the calculation of the overall weighted scores”[ 13 ] would constitute risk evaluation. The multriattribute utility theory (MAUT) is yet another example of an evaluation method. The MAUT is a theory that is basically “concerned with making tradoffs among different goals” [ 14 ]. This theory factors in human values, values defined as “the functions to use to assign utilities to outcomes” [ 14 ]. From the value tree “inputs,” the evaluator would then need to assign weights to each of these inputs. The purpose of plugging in weights is to establish the importance of each input, according to the evaluators. This is tantamount to establishing criteria, or identifying and making explicit the evaluators’ definition of acceptable risk. Next, the evaluators would need to plug in numerical values as the utility values of those that are being evaluated. These values would be multiplied to the weight. The latter values, when summed, would constitute the total utility value. To illustrate, if, for example, an REC wishes to make an evaluation of a psychotropic study drug and the standard drug, an REC may come up with MAUT chart like (Table 1 ).

Just like the value tree, our purpose is not to endorse only one way of doing the evaluation. Our purpose is merely to illustrate that such a decision study tool is capable of explicitly showing the following: a.) the inputs that the evaluators think must play a role in the evaluation; b.) the values of the evaluators, through the scores they have provided; c.) the importance they give to each of the factors/inputs through the weights that they have provided, d.) how the things compared (in this case, the study drug and the standard drug) fare given a, b and c ; and e.) a global perspective of what a, b, c, and d amount to, i.e., through the total utility value.

In the risk-benefit literature in research ethics, we find statements that such an algorithm is undesirable because it “yields one and only one verdict about the risk-benefit profile of each possible protocol” [ 11 ]. On this issue, CMHP’s Reflection is instructive. The scores in quantitative evaluations are valuable not because of some absolute value, but because these scores can

"…focus the discussion by highlighting the divergences between the assessors and stakeholders concerning choice for weights. The benefit of such analysis methods is that the degree and nature of these divergences can be assessed, even in advance of any compound’s review. The same method might be used with the weights (e.g., of different stakeholders) and make both the differences and the consequences of those differences more explicit. If the analyses agree, decision-makers can be more comfortable with a decision. If the analyses disagree, exact sources of the differences in view will be identified, and this will help focus the discussion on those topics [ 12 ]."

Thus, the scores are meant to allow the evaluators to know each others’ values, similarities, differences, and divergences. The divergences and differences could aid in focusing the REC discussion and figure out problem areas in a deliberate, transparent, coherent, and less intuitive manner [ 15 ].

Risk-benefit analysis and evaluation together constitute risk-benefit assessment [ 4 ] .

Once risks and benefits have been evaluated versus the evaluators’ given criteria, risk evaluation allows evaluators to decide “which risks need treatment and which do not” [ 6 ]. In decision studies, amplifying benefits and modifying risks are possible only after a global understanding of it through risk assessment has been achieved. Thus, after risk-benefit assessment comes risk treatment. By risk treatment, we refer to the “process of selection and implementation of measures to modify risk…measures may include avoiding, optimizing, transferring, or retaining risk” [ 4 ]. In terms of trials, risk treatment would refer to enhancing the trial’s social value, reducing the risks to the participants, and enhancing the participants’ benefits [ 11 ]. There may be concerns especially from REC members who have been used to minimizing risk immediately after its identification that this process necessitates them to suspend such move until risk evaluation is done, a procedure that may be counter-intuitive for some. However, the process of “immediately cutting the risks” also have passed through the process of evaluation, although intuitively and implicitly. An REC member who says that the risks of a certain procedure may be minimized or that the risks are unnecessary given the research question has already implicitly gone through a personal evaluation of what is and what is not necessary in such a clinical trial.

After investigating on the possibilities to modify risk and amplify the benefits, the decision makers would then have to finally decide whether the risks of the trial are justified given the benefits. By decision making , we refer to the final discussion of the REC on whether benefits truly outweigh risks, i.e., given all the information provided, are the risks of the trial ethically acceptable due to the merits of the probable benefits?

It is important to note that in the risk literature [ 4 , 13 ], the CHMP Reflection [ 12 ], and the CIOMS report [ 16 ], the risk-benefit tasks are assumed to be done interdependently and that the tasks are reflective of various values, interests, and ethical perspectives. At least for marketing authorization and marketed drug evaluation purposes, the sponsor and/or the investigator are assumed to be responsible for risk-benefit assessment and to a certain extent, the proposal of risk treatment measures. It makes a lot of sense that the sponsor ought to be responsible for risk analysis precisely because in this task, “experts on the systems and activities being studied are usually necessary to carry out the analysis” [ 4 ]. The regulatory authorities, on the other hand, are expected to provide guidelines for the risk-benefit analysis criteria. They also ought to provide their own version of risk-benefit evaluation to determine areas of divergences and differences, to extensively discuss risk treatment measures and options, and finally to deliberate and decide based on all these inputs.

Conflation of the various risk-benefit tasks by the procedure-level approaches

At the most superficial level, we notice that Wendler and Rid used the terms “risk-benefit assessment” and “risk-benefit evaluation” interchangeably to refer to the one and the same Net Risk Test [ 11 , 17 ]. Nevertheless, it could be argued that this is just a matter of misuse of terms, and that such does not substantially affect the approach that is proposed. Thus, we would need to look deeper into the Net Risk Test to justify our claim that it conflates the various risk-benefit tasks.

In the latest seven-step framework of the Net Risk Test, what ought to be a framework for risk-benefit evaluation of RECs ended up incorporating aspects of risk-benefit assessment, risk treatment, and decision making. The first step, that is, ensuring and enhancing the study’s social value, is risk treatment. The second step, that is, identifying the research interventions, is risk analysis. The third and fourth steps, which are the evaluation and reduction of risks to participants, and the evaluation and enhancing of potential benefits to participants, both fall into risk-benefit evaluation and risk treatment. It is worthwhile to note that in the Net Risk Test, the evaluation and the treatment of risks and benefits were not preceded by the identification of these risks and benefits; instead, prior to the third and fourth steps is the step to identify research interventions, a necessary but incomplete step in risk-benefit analysis. The fifth step, that is, the evaluation whether the interventions pose net risks, is risk-benefit evaluation. The sixth step, which is to evaluate whether the net risks are justified by the potential benefits of other interventions, is decision making. The last step, which is to evaluate whether the remaining net risks are justified by the study’s social value, is also decision making. Thus, the Net Risk Test in principle encompasses all the risk-benefit tasks without taking into account the distinctions, the chronological order among the various tasks, nor the division of labor in the various risk-benefit tasks.

The Component Analysis, just like the Net Risk Test, does the same conflation. In the process of distinguishing procedures into either therapeutic or non-therapeutic, the REC members would first need to identify the procedures to assess, i.e., risk analysis. The REC members would then need to evaluate therapeutic procedures differently compared to non-therapeutic procedures. Therapeutic procedures have to be evaluated on whether clinical equipoise exists, and whether the procedure is consistent with competent care. These two criteria may be considered as ethical principles that ought to be present in the deliberation towards decision making. Thus, these are decision making tasks. Next, the REC members would need to determine if the therapeutic procedure is reasonable in relation to the potential benefits to subjects. Since REC members need to answer questions of “reasonability,” this is a decision making task that presupposes risk-benefit evaluation. Non-therapeutic procedures, on the other hand, would necessitate the assessor to evaluate if risks are minimized and if risks are consistent with sound scientific design. This is risk treatment. Next, the assessor would need to verify if the risk of the non-therapeutic procedure is reasonable in relation to knowledge to be gained. Again, this is a decision making task that presupposes risk-benefit evaluation. In cases where vulnerable patients are involved, the REC members would need to verify if no more than minor increase over minimal risk is involved; this is a discussion that is likely to be present in the deliberation towards decision making, which also presupposes risk-benefit evaluation. Lastly, the assessor would need to make a decision if both therapeutic and non-therapeutic procedures pass. This is decision making. Hence, again, what we have is a system that touches on each of the risk-benefit tasks without making a distinction among the various tasks.

Since the risk-benefit tasks are conflated, the various tasks are necessarily simplified and confused. We have seen that the various risk-benefit tasks are resource intensive (since various experts must be involved), necessarily complex (since a drug trial is rarely simple), and time consuming. This is the reason why they are done separately. To conflate the various tasks into one system that ought to be accomplished within the few hours that the REC convenes is an impossibility. Precisely because of this conflation, plus the consideration that all the risk-benefit tasks ought to be done within the time restrictions of an REC, both procedure-level approaches cursorily and confusedly “accomplish” the various tasks. As such, we cannot expect the procedure-level approaches to have the same level of robustness, transparency, explicitness, and coherence as the various approaches of decision studies have. Neither of the procedure-level approaches could have the same robustness that the value-tree had, for example, in expressing and illustrating the relations between the nature, cause, consequences, as well as the uncertainties, of both risk and benefit components. Neither is also transparent, explicit, and rigorous enough to capture the acceptable risk definitions and the various weights and scores that are reflective of the various values and ethical dispositions that the MAUT method provided. The two procedure-level approaches simply do not require evaluators to be explicit in terms of their evaluative values. Though risk treatment is largely present in both procedure-level approaches, risk treatment, at least in the Net Risk Test, is sometimes confounded with risk evaluation. In the procedure-level approaches, RECs would also not have the benefit of systematically focusing the discussion on divergences and differences that a good risk evaluation makes possible. Lastly, because of the conflation and confusion of the various risk-benefit tasks, REC members are left to their own devices and intuition to decide on what is important to discuss and which is not, and eventually, to decide if the risks are justifiable relative to the benefits. Such a “procedure” could be categorized as a “taking into account and bearing in mind” process, a process that Dowie rightfully criticized as vague, general, and plainly intuitive [ 15 ].

Recommendations

We have seen that the methods from decision studies are more robust, transparent, and coherent than any of the procedure-level approaches. This is not surprising considering the fact that decision studies have been utilized in many various fields for quite some time now. The robustness of the decision studies methods stems from the clear distinction between risk-benefit analysis, risk-benefit evaluation, risk treatment, and decision making. In decision studies, each of the risk-benefit tasks is a system in itself that ought not to be conflated. In addition, in contrast to “taking into account and bearing in mind” processes, decision studies encourage the exposure of beliefs and values [ 15 ] precisely because it is from this explicitness that discussions can be defined and ordered. As such, we recommend the following:

RECs should make clear what their task is. RECs do not have the time and are not in the best position to do risk analyses. As such, risk analysis must be a task for the sponsor. As regards risk evaluation, RECs ought to provide their own risk-benefit evaluation to pair with the sponsor’s/investigator’s evaluation since this is the best way to systematically point out areas of divergence/convergence. These areas would aid in putting order in REC discussions. The evaluation of risk treatment suggestions and possibly coming up with a revised or different risk treatment appraisal ought to also form part of REC discussions. Lastly, it is obviously the REC’s task to make the final decision on whether the risks of the trial are justified given the benefits.

Precisely because such a clarification of tasks is so essential if the REC is to function efficiently, RECs must look into how decision studies may be incorporated in its risk-benefit tasks. This is something we will do in our next article. For now, it is imperative to lay the theoretical groundwork for the urgency of such incorporation.

The procedure-level approaches emphasize on the role of the various ethical concepts such as net risk, minimum risk, clinical equipoise, in the risk-benefit task of RECs. These are legitimate concerns; nevertheless, RECs must know when these concepts play a role in the various risk-benefit tasks. Minimal risk, for example, is a concept that ought to be present in risk treatment and/or deliberation towards final decision making.

Both the Net Risk Test and the Component Analysis conflate risk-benefit analysis, risk-benefit evaluation, risk treatment, and decision making. This makes the risk-benefit task of RECs confusing, if not impossible. It is necessary to make a distinction between these four different tasks if RECs are to be clear about what their task truly is. By looking at decision studies, we realize that RECs ought to evaluate risks and benefits, appraise risk treatment suggestions, and make the final decision. Further clarification and elaboration of these tasks would necessitate research ethicists to look beyond the procedure-level approaches. It further requires research ethicists to allow decision studies discourses into the current discussion on the risk-benefit tasks of RECs. Admittedly, this would take a lot of time and research effort. Nevertheless, the discussion on the REC’s risk-benefit task would be more fruitful and democratic if research ethics opens its doors to other disciplines that could truly help clarify risk-benefit task distinctions.

King NM, Churchill LR: Assessing and comparing potential benefits and risks of harm. The Oxford textbook of clinical research ethics. Edited by: Emanuel E, Grady C, Crouch RA, Lie RA, Miller FG, Wendler D. 2008, Oxford University Press, New York, 514-26.

Google Scholar  

US National Bioethics Advisory Commission: Ethical and Policy issues in Research Involving Human Participants. 2001

Westra AE, de Beufort ID: The merits of procedure-level risk-benefit assessment. 2011, Ethics and Human Research, IRB

Aven T: Risk analysis: asssessing uncertainties beyond expected values and probabilities. 2008, Wiley, Chichester

Book   Google Scholar  

Vose D: Risk analysis: a quantitative guide. 2008, John Wiley & Sons, Chichester, 3

European Network and Information Security Agency. Risk assessment. European Network and Information Security Agency. 2012, Available from: http://www.enisa.europa.eu/act/rm/cr/risk-management-inventory/rm-process/risk-assessment

Miller P, Weijer C: Evaluating benefits and harms in clinical research. Principles of Health Care Ethics, Second Edition. Edited by: Ashcroft RE, Dawson A, Draper H, McMillan JR. 2007, Wiley & Sons

Weijer C, Miller PB: When are research risks reasonable in relation to anticipated benefits?. Nat Med. 2004, 10 (6): 570-573. 10.1038/nm0604-570.

Article   Google Scholar  

Weijer C: When are research risks reasonable in relation to anticipated benefits?. J Law Med Ethics. 2000, 28: 344-361. 10.1111/j.1748-720X.2000.tb00686.x.

Wendler D, Miller FG: Assessing research risks systematically: the net risks test. J Med Ethics. 2007, 33 (8): 481-486. 10.1136/jme.2005.014043.

Rid A, Wendler D: A framework for risk-benefit evaluations in biomedical research. Kennedy Inst Ethics J. 2011 June, 21 (2): 141-179. 10.1353/ken.2011.0007.

European Medicines Agency -- Committee for Medicinal Products for Human Use. Reflection paper on benefit-risk assessment methods in the context of the evaluation of marketing authorization applications of medicinal products for human use. 2008

Mussen F, Salek S, Walker S: Benefit-risk appraisal of medicines. 2009, John Wiley & Sons, Chichester

Baron J: Thinking and deciding. 2008, Cambridge University Press, New York, 4

Dowie J: Decision analysis: the ethical approach to most health decision making. Principles of Health Care Ethics. Edited by: Ashcroft RE, Dawson A, Draper H, McMillan JR. 2007, John Wiley & Sons, 577-83.

Council for International Organizations of Medical Sciences’s Working Group IV. Benefit-risk balance for marketed drugs: evaluating safety signals. 1998

Rid A, Wendler D: Risk-benefit assessment in medical research– critical review and open questions. Law, Probability Risk. 2010, 9: 151-177. 10.1093/lpr/mgq006.

Pre-publication history

The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1472-6939/13/6/prepub

Download references

Acknowledgements

This study was performed in the context of the Escher project (T6-202), a project of the Dutch Top Institute Pharma, Leiden, The Netherlands.

Author information

Authors and affiliations.

Julius Center for Health Sciences and Primary Care, Utrecht University Medical Center, Heidelberglaan 100, Utrecht, 3584CX, The Netherlands

Rosemarie D L C Bernabe, Ghislaine J M W van Thiel & Johannes J M van Delden

GlaxoSmithKline, Huis ter Heideweg 62, Zeist, 3705, LZ, The Netherlands

Jan A M Raaijmakers

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Rosemarie D L C Bernabe .

Additional information

Competing interests.

RB’s PhD project is funded by the Dutch Top Institute Pharma. JR works for and holds stocks in GlaxoSmithKline. JvD and GvT have no competing interests to declare.

Authors’ contributions

All authors were involved in the design of the manuscript. RB did the research and wrote the draft and final manuscript; GvT commented on the drafts, wrote parts of the manuscript, and approved the final version; JR commented on the drafts and approved the final version of the manuscript; JvD commented on the drafts and approved the final version of the manuscript. All authors read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2, authors’ original file for figure 3, authors’ original file for figure 4, authors’ original file for figure 5, rights and permissions.

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article.

Bernabe, R.D.L.C., van Thiel, G.J.M.W., Raaijmakers, J.A.M. et al. The risk-benefit task of research ethics committees: An evaluation of current approaches and the need to incorporate decision studies methods. BMC Med Ethics 13 , 6 (2012). https://doi.org/10.1186/1472-6939-13-6

Download citation

Received : 05 April 2012

Accepted : 11 April 2012

Published : 20 April 2012

DOI : https://doi.org/10.1186/1472-6939-13-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Risk benefit assessment
• Ethics committee
• Decision theory
• Net risk test
• Component analysis

BMC Medical Ethics

ISSN: 1472-6939

• General enquiries: [email protected]

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Limits to research risks

Affiliation.

• 1 Department of Bioethics, Clinical Center, National Institutes of Health, Building 10, Bethesda, Maryland 208921-156, USA. [email protected]
• PMID: 19567696
• DOI: 10.1136/jme.2008.026062

Risk-benefit assessment is a routine requirement for research ethics committees that review and oversee biomedical research with human subjects. Nevertheless, it remains unclear how to weigh and balance risks to research participants against the social benefits that flow from generating biomedical knowledge. In this article, we address the question of whether there are any reasonable criteria for defining the limit of permissible risks to individuals who provide informed consent for research participation. We argue against any a priori limit to permissible research risks. However, attention to the uncertainty of potential social benefit that can be derived from any particular study warrants caution in exposing prospective research participants to a substantial likelihood of serious harm.

PubMed Disclaimer

Similar articles

• [Medical research-ethics applied to social sciences: relevance, limits, issues and necessary adjustments]. Desclaux A. Desclaux A. Bull Soc Pathol Exot. 2008 Apr;101(2):77-84. Bull Soc Pathol Exot. 2008. PMID: 18543697 Review. French.
• Are risks and benefits of oncological research protocols both incommensurable and incompensable? Musschenga AW, Van Luijn HE, Keus RB, Aaronson NK. Musschenga AW, et al. Account Res. 2007 Jul-Sep;14(3):179-96. doi: 10.1080/08989620701455217. Account Res. 2007. PMID: 17877107
• Assessing research risks systematically: the net risks test. Wendler D, Miller FG. Wendler D, et al. J Med Ethics. 2007 Aug;33(8):481-6. doi: 10.1136/jme.2005.014043. J Med Ethics. 2007. PMID: 17664310 Free PMC article.
• Randomized, controlled trials as minimal risk: an ethical analysis. Morris MC, Nelson RM. Morris MC, et al. Crit Care Med. 2007 Mar;35(3):940-4. doi: 10.1097/01.CCM.0000257333.95528.B8. Crit Care Med. 2007. PMID: 17255879
• Research participants telling the truth about their lives: the ethics of asking and not asking about abuse. Becker-Blease KA, Freyd JJ. Becker-Blease KA, et al. Am Psychol. 2006 Apr;61(3):218-26. doi: 10.1037/0003-066X.61.3.218. Am Psychol. 2006. PMID: 16594838 Review.
• A few remarks on limits of research risks and research payments. Różyńska J. Różyńska J. Med Health Care Philos. 2023 Mar;26(1):155-156. doi: 10.1007/s11019-022-10125-9. Epub 2022 Nov 9. Med Health Care Philos. 2023. PMID: 36350534 Free PMC article. No abstract available.
• Reevaluating the Ethical Issues in Porcine-to-Human Heart Xenotransplantation. Silverman H, Odonkor PN. Silverman H, et al. Hastings Cent Rep. 2022 Sep;52(5):32-42. doi: 10.1002/hast.1419. Hastings Cent Rep. 2022. PMID: 36226875 Free PMC article.
• Pandemic vaccine testing: Combining conventional and challenge studies. Gerhard T, Strom BL, Eyal N. Gerhard T, et al. Pharmacoepidemiol Drug Saf. 2022 Jun;31(6):710-715. doi: 10.1002/pds.5429. Epub 2022 Mar 31. Pharmacoepidemiol Drug Saf. 2022. PMID: 35297119 Free PMC article.
• Prioritizing second-generation SARS-CoV-2 vaccines through low-dosage challenge studies. Steuwer B, Jamrozik E, Eyal N. Steuwer B, et al. Int J Infect Dis. 2021 Apr;105:307-311. doi: 10.1016/j.ijid.2021.02.038. Epub 2021 Feb 13. Int J Infect Dis. 2021. PMID: 33592338 Free PMC article.
• Key criteria for the ethical acceptability of COVID-19 human challenge studies: Report of a WHO Working Group. Jamrozik E, Littler K, Bull S, Emerson C, Kang G, Kapulu M, Rey E, Saenz C, Shah S, Smith PG, Upshur R, Weijer C, Selgelid MJ; WHO Working Group for Guidance on Human Challenge Studies in COVID-19. Jamrozik E, et al. Vaccine. 2021 Jan 22;39(4):633-640. doi: 10.1016/j.vaccine.2020.10.075. Epub 2020 Oct 28. Vaccine. 2021. PMID: 33341309 Free PMC article.
• Search in MeSH

LinkOut - more resources

Full text sources.

• Ovid Technologies, Inc.
• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Risk-Benefit Assessment in Research Ethics

Introduction.

What is research ethics?

Research ethics deals with acceptable norms in the design and application of a research study. It governs the standards of conduct of investigators in scientific research in order to uphold important principles including autonomy, justice and beneficence (WHO).

What is Risk-Benefit Assessment?

Since the Belmont Report, most biomedical research guidelines require a sort of risk-benefits assessment in order to carry the research scientifically and ethically. The principles underlying this sort of assessment are beneficence and non-maleficence.

Justification for risk-benefit assessment are twofold “to verify the scientific/social validity of the research since an unscientific research is also an unethical research; and to ensure that the risks that the participants are exposed to are necessary, justified, and minimized.” (Bernabade et al 2012)

Therefore, participation in research trials is best when participants derive something health, or something other, valuable from it, or when it significantly improves generalizable knowledge without inducing unreasonable and unjust harm.

Assigned Reading

Rid, A., & Wendler, D. (2011). A framework for risk-benefit evaluations in biomedical research.  Kennedy Institute of Ethics journal ,  21 (2), 141–179. https://doi.org/10.1353/ken.2011.0007

Thesis : In their paper Rid and Wendler (2011) note the importance of risk-benefit assessment and highlight the paucity of comprehensive and concrete guidance to perform these assessment. This has resulted in rather unsystematic methods often largely based on intuitions and a disparity of assessment across several research studies. In order to address this gap the authors, designed the first step-by-step comprehensive guiding framework for risks benefits assessments in research ethics. Their framework is based on extant guidelines and regulations, other relevant literature and normative analysis.

Discussion Questions

The following questions were considered by seminar participants prior to the discussion:

• What were some of the strengths of the framework?
• What were some of the weaknesses or more vague elements of the framework?

Reflection Points

1. Evaluation of the requirements that studies meet a minimum threshold of “social value” – social value is a normatively laden concept (what ones person thinks as contributing to social value, another might think of as subtracting). It is also vague and may be too easy to apply (can hand waive that every study has social value of some sort).

2. “Enhancement” as an idea in their framework (requirement to “enhance” the social value the study and “enhance” potential benefits to participants). What is the normative justification for this? Why do researchers have an obligation to do this? Or IRB members? Also, consider potential unintended consequences of this – may negatively affect some of the science in the study, may impose costs on researchers or society, etc.

3. Evaluation of “clinical” benefit to participants. Is this biasing against non-clinical studies? Is this too narrow (benefits of psychological or other nature – such as fulfilled desire to be altruistic)? On the other hand, too broad or hand-waving (“benefit” from talking to researchers about feelings)? Violates equipoise/therapeutic misconception?

4. Steps 5, 6, 7/ the “weighing steps” – steps where the identified risks (and their likelihood), clinical benefits, net risks, and social value are weighed to establish whether the assessment is favorable toward allowing the study. How exactly is this weighing done? Ideas of “informed and impartial social arbitrator” and “informed clinician” are introduced to help, but these constructs are vague and can allow for introduction of bias.

5. Lack of context – avoids questions such as whether the study is therapeutic or non-therapeutic? Whether the study involves people who are dying, etc.6. May be too biased towards approval since it primes people to consider “social value” in first and last step–social value is easy to come by.

References and Additional Resources

Bernabe, R.D.L.C., van Thiel, G.J.M.W., Raaijmakers, J.A.M.  et al.  (2012). The risk-benefit task of research ethics committees: An evaluation of current approaches and the need to incorporate decision studies methods.  BMC Med Ethics   13 ,  6 https://doi.org/10.1186/1472-6939-13-6

Abdalla M.E. (2017) Ethical Issues Involved with the Analysis of Risks and Benefits. In: Silverman H. (eds) Research Ethics in the Arab Region. Research Ethics Forum, vol 5. Springer, Cham. https://doi.org/10.1007/978-3-319-65266-5_8

Rid, Annette & Wendler, David. (2010). Risk-benefit assessment in medical research–critical review and open questions. Law, Probability and Risk . 9 . 151-177. 10.1093/lpr/mgq006.

• Get IGI Global News

• All Products
• Book Chapters
• Journal Articles
• Video Lessons
• Teaching Cases
• Recommend to Librarian
• Recommend to Colleague
• Fair Use Policy

• Access on Platform

Export Reference

• Advances in Library and Information Science
• e-Book Collection
• Library and Information Science e-Book Collection
• Social Sciences and Humanities e-Book Collection
• e-Book Collection Select
• Education Knowledge Solutions e-Book Collection
• Social Sciences Knowledge Solutions e-Book Collection

Risk-Benefit Evaluation in Clinical Research Practice

Introduction.

Assuring a favorable risk-benefit balance is crucial to human subjects' protection in clinical research and thus complies to the principle of beneficence in the Belmont report, written and published by the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research in 1979. It is a critical requirement for all research regulations that guide each step in research. It is also the main criteria for determining the approvability of categories of participants and reviewing those categories, the degree of protection required for each category of participants, according to the type and frequency of monitoring study.

The objective of this chapter is to highlight the:

Concepts, principles, and types of risks and benefits;

Methodology of risk-benefit analysis;

Applications and impact of data science technology in clinical research practice;

Challenges to risk-benefit analysis and suggested resolutions.

Evaluation of the risk-benefit balance in research is a complex process, which is sometimes straightforward, but in other cases it is more challenging.

Balancing risks to patients against the total benefits to science or to future patients is one of the most difficult issues, especially in research involving vulnerable populations.

The aim of this chapter is to lead the readers through the decision-making process of risk-benefit evaluation of clinical research.

The scope: Throughout the three main sections of this paper, we discuss the main principles, approaches and challenges of risk and benefit assessment in clinical research.

Section I: Principles Of Risk-Benefit Assessment In Clinical Research

Risk-benefit assessment is an universal requirement for protecting human participants in clinical trials. The concept and perception of risks and benefits, their types, degree, and relative weight are complex attributes which can vary depending on the context in which the study is being conducted.

Defining Risk and Benefit

In research, risk is defined as: “a potential harm, discomfort, or inconvenience that a reasonable person in the subject's position would likely consider significant in deciding whether or not to participate in the study”. A comprehensive definition is given in the Belmont report (NCPHSBBR, 1979) as: “a possibility that harm may occur both in chance (probability) and severity (magnitude) of envision. The term 'benefit' refers to something of positive value related to health or welfare”. Accordingly, risk/benefit assessments are concerned with the probabilities and magnitudes of possible harm and benefits.

Key Terms in this Chapter

Precision Medicine : A strategy of medical treatment tailored for the medical necessities of an individual patient based on the collection of a large amount of data about that patient's (e.g., genomic study or other molecular or cellular analysis, personal, and disease data).

Omics : Is the part of biotechnology which analyzes the structure and functions of the whole makeup of a given biological function, at different levels, including the molecular gene level (“genomics”), the protein level (“proteomics”), and the metabolic level (“metabolomics”).

Equipoise : Means an equal view. It is used in research to describe an equal standing of the clinical investigator in therapeutic trials in relation to both the trial drug and an alternative treatment. This is important in order to avoid bias.

Patient-Centricity : This term defines a collaborative partnership between patients, caregivers, and pharmaceutical companies, focusing on patients' needs and preferences in the research enterprise. The term was at first created collaboratively by AstraZeneca, patients, and caregivers, and was demonstrated to be a valid approach by testing conducted on many patients' groups.

Ultraism : It is the sacrifice of self-interest for others benefit. In research, it is used when a participant is exposed to risk for the benefit to society.

Therapeutic Misconception : When the research participant in a clinical trial doesn't perceive the difference between the research intervention and the clinical care, which could impact the validity of his informed consent.

Deep Learning : Is the complex, unsupervised processing of unstructured data in order to create patterns used in decision making, patterns that are analogous to those of the human brain.

Complete Chapter List

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Publish with us
• About the journal
• Meet the editors
• Specialist reviews
• BMJ Journals

You are here

• Volume 3, Issue 1
• Assessing the methodological quality and risk of bias of systematic reviews: primer for authors of overviews of systematic reviews
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0002-7825-6765 Carole Lunny 1 , 2 ,
• Salmaan Kanji 3 , 4 ,
• Pierre Thabet 5 ,
• Anna-Bettina Haidich 6 ,
• Konstantinos I Bougioukas 6 and
• Dawid Pieper 7
• 1 Knowledge Translation Program, Li Ka Shing Knowledge Institute , St Michael's Hospital , Toronto , ON , Canada
• 2 Cochrane Hyptertension Review Group , Cochrane Canada , Vancouver , BC , Canada
• 3 Ottawa Hospital , Ottawa , ON , Canada
• 4 Ottawa Hospital Research Institute , Ottawa , ON , Canada
• 5 School of Pharmaceutical Sciences University of Ottawa , Hôpital Montfort , Ottawa , ON , Canada
• 6 Department of Hygiene, Social-Preventive Medicine and Medical Statistics , Aristotle University of Thessaloniki Faculty of Health Sciences , Thessaloniki , Central Macedonia , Greece
• 7 Faculty of Health Sciences Brandenburg , Brandenburg Medical School Theodor Fontane Ruppin Clinics , Neuruppin , Brandenburg , Germany
• Correspondence to Dr Carole Lunny, Knowledge Translation Program, Li Ka Shing Knowledge Institute, St Michael's Hospital, Toronto, Canada; carole.lunny{at}ubc.ca

https://doi.org/10.1136/bmjmed-2023-000604

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

• Epidemiology
• Research design
• Public health

Key messages

Systematic reviews underpin evidence based healthcare decision making, but flaws in their conduct may lead to biased estimates of intervention effects and hence invalid recommendations

Overviews of reviews (also known as umbrella reviews, meta-reviews, or reviews of reviews) evaluate biases at the systematic review level, among others, but proper use of tools for this purpose require training, time, and an appreciation of their strengths and limitations

AMSTAR-2 and ROBIS are the two most popular and rigorous critical appraisal tools used for appraising systematic reviews

The AMSTAR-2 16-item checklist focuses on methodological quality of systematic reviews of healthcare interventions, and incorporates aspects of review conduct, reporting comprehensiveness, and risk of bias as specific items

ROBIS is a domain based tool with 19 items focusing on risk of biases in a systematic review (eg, selective reporting of outcomes or analyses) of healthcare interventions and contains items related to risk of bias in results and conclusions, relevance, and an item about risk of interpretation bias or ’spin’

Carole Lunny and colleagues consider methods such as AMSTAR-2 and ROBIS tools to evaluate the methodological quality and risk of bias of systematic reviews of intervention effects that are included in overviews of reviews

Introduction

Overviews of reviews, which synthesise the findings of systematic reviews, 1 have significantly increased in publication over the past decade. 2 However, the terminology used to describe them is not agreed in consensus, with terms such as umbrella reviews, meta-reviews, and reviews of reviews being used interchangeably to mean overviews of reviews. Methods research has been ongoing since the 2010s to develop effective approaches for conducting overviews of reviews and addressing their unique characteristics. 3–7 Overview authors use various approaches to assess the methodological quality and risk of bias in their included systematic reviews, and they apply these assessments to inform the overviews' results and conclusions. However, proper use of tools for this purpose require training, time, and an appreciation of their strengths and limitations. This methods primer aims to address the inconsistency in assessing and reporting bias in systematic reviews of intervention effects included within overviews, and focuses on presenting the different validated tools, comparing them, and providing guidance on the interpretation and reporting of these assessments.

Assessing the methodological quality and risk of bias

Assessment tools.

In 2016, ROBIS was developed to assess risk of bias in systematic reviews, 11 ROBIS consists of three phases: assessment of relevance (optional), identification of bias concerns with the review process, and judgement of the overall risk of bias in the review. The tool focuses on four domains: study eligibility criteria, identification and selection of studies, data collection and study appraisal, and synthesis and findings. ROBIS helps reviewers identify potential biases in these domains by asking specific questions related to the review's methods and reporting. The tool underwent content validity and reliability testing to ensure its accuracy and consistency in assessing the risk of bias in systematic reviews.

In 2017, an update to AMSTAR, called AMSTAR-2, 10 aimed to assess methodological quality of systematic reviews, and involved inter-rater reliability and usability testing. AMSTAR-2 consists of 16 items that evaluate various aspects of the systematic review process, including the research question formulation, study selection and data extraction, assessment of risk of bias in individual studies, consideration of publication bias, and appropriate statistical analysis. This tool also assesses the overall methodological quality and risk of bias in the review, providing a comprehensive evaluation.

The decision about how to evaluate overall risk of bias for ROBIS is made at the assessors' discretion, as opposed to the AMSTAR-2 overall judgement, which is prescribed by AMSTAR-2 guidance. Examples of how to interpret methodological quality and risk of bias assessments, and how to make an overall judgement are found in box 1 .

Decision rules: how to decide that the results of a review are of high quality or at low risk of bias overall

Decision rules are a priori strategies used to specify rules to define explicitly how each item is rated, as well as how an overall judgement is made about a specific systematic review with the AMSTAR-2 and ROBIS tools. In the case of AMSTAR-2, the authors who are using the tool stipulate how to come to an overall high quality rating in the results of the review, but not how to rate each item. For example, item 15 of AMSTAR-2 asks assessors whether an adequate investigation of publication bias (small study bias) was conducted and whether its likely effect on the results was discussed. However, the AMSTAR-2 team did not specify what happens when 10 studies or fewer were included (ie, the analysis will be underpowered to detect publication bias), what methods to detect publication bias are recommended, and if publication bias is detected, how it should be discussed (ie, as a systematic review limitation).

The ROBIS tool equally does not specify what decision rules should be used for assessment of risk of bias, nor how to come to an overall judgement. For example, item 4.6 of ROBIS ("Were biases in primary studies minimal or addressed in the synthesis?") is similar to item 12 of AMSTAR-2 ("If meta-analysis was performed, did the review authors assess the potential impact of risk of bias in individual studies on the results of the meta-analysis?"). Of note, risk of bias should be assessed in any systematic review regardless of whether a meta-analysis was performed. A possible decision rule for answering these two questions when considering whether bias was adressed and considered in the results and their interpretation could be to respond "Yes" or "Probably/Partial Yes" if:

All studies received a low risk of bias rating; and

Studies were judged at high risk of bias and sensitivity analyses (grouping high v low risk studies in a meta-analysis) or adjustment approaches were used

For a "No" response:

Important biases were suspected to have been in the included studies that have been ignored by the review authors; or

Risk of bias was not assessed at all in the included studies; or

Bias was assessed but authors did not incorporate it into findings, discussion, and conclusions

Based on the above decision rules, how would the following statement be rated? "We planned on conducting sensitivity analysis on the studies based on their level of risk of bias. Most of the included studies had a similar risk of bias across all the domains except for industry sponsorship bias and incomplete data for total testosterone. Due to the inadequate number of studies, we were not able to conduct a sensitivity analysis on the included studies based on industry sponsorship."

For overall judgements, a decision rule could be that if one or more ROBIS domains are at high risk of bias, then the overall study is deemed at high risk of bias. For AMSTAR-2, the authors of the tool have stipulated that the review is considered of low or critical low quality when any of the subset of seven ‘critical’ items have one or more critical flaws. While the decisions about how to rate the items and make overall judgements can be debated, the grounds on which overview authors make these decisions should be noted explicitly in the manuscript or in an appendix, as then the assessment results will be transparent and reproducible.

Cautionary note: empirical evidence does not currently support the assignment of scores to items that are met in a risk of bias tool followed by the summation or averaging of these scores to produce a numerical measure of risk of bias. A thoughtful, nuanced, and customised overall judgement is required that considers all items with suspected bias on the basis of specific context.

The AMSTAR-2 and ROBIS tools were designed to assess systematic reviews with pairwise meta-analysis only. A more recent tool under development aims to assess the potential biases and limitations in network meta-analyses. 12 13 Guidance documents (eg, Cochrane 14 and JBI 15 ) recommend overview authors use ROBIS or AMSTAR-2 when comparing and critically appraising systematic reviews over other available tools. Figure 1 presents two example assessments conducted by our team, the ROBIS assessment of Normansell and colleagues 16 is presented at the domain level, and the AMSTAR-2 assessment of Puig and colleagues 17 is presented by item. Items are backed by quotes and rationales to support the answers chosen, for full transparency, and to help when comparing assessments between two independent assessors ( figure 2 ).

• Download figure
• Open in new tab
• Download powerpoint

Example assessments using ROBIS of Normansell 16 and AMSTAR-2 of Puig 17 . The ROBIS assessment is presented by domain and the AMSTAR-2 assessment by individual items. ROBIS's phase one, where the assessor considers the relevance of the systematic review questions to the overview's question, is not shown. The decision about how to evaluate overall risk of bias for ROBIS is made at the assessors' discretion, as opposed to the AMSTAR-2 overall judgement, which is prescribed by AMSTAR-2 guidance

PICO framework stands for patient or problem, intervention or exposure, comparison or control, and outcomes. DLQI=dermatology life quality index; DMARDs=disease modifying anti-rheumatic drugs; PASI=psoriasis area and severity index; RCT=randomised controlled trial

Comparison of AMSTAR-2 and ROBIS

Both the AMSTAR-2 and ROBIS tools provide structured guidelines for reviewers to evaluate and report on methodological strengths and weaknesses as well as potential biases in systematic reviews, contributing to the overall reliability and credibility of the evidence presented.Considerable overlap exists between the items of the two tools ( figure 1 ). In the documentation for each tool, AMSTAR-2 states that it was developed for systematic reviews of healthcare interventions whereas ROBIS states that it is aimed at reviews of healthcare interventions, diagnosis, prognosis, and biological cause. In practice, the ROBIS tool is generic and its signalling questions relate to interventions in the clinical or public health fields. Questions specific to systematic reviews of diagnosis, prognosis, and biological cause are not found in the tool. AMSTAR-2 was developed to assess methodological quality (which includes indicators of risk of bias) while ROBIS was developed primarily to assess risk of bias but also includes items that address methodological quality.

AMSTAR-2 focuses more on reporting comprehensiveness (eg, reporting of study designs for inclusion and reporting on excluded studies with justification) and methodological quality or transparency constructs (eg, pre-established protocol, sources of funding of primary studies, and reviewers' competing interests). Whereas ROBIS focuses on items related to identification of the different biases (eg, selective reporting of outcomes or analyses and publication bias). Bias occurs when factors systematically affect the results and conclusions of a review and cause them to be systematically different from the truth. 1 Systematic reviews affected by bias can be inaccurate; for example, finding false positive or false negative intervention effects by systematically over or under estimating the true effect in the target population. Methodological quality focuses on methodological features associated with internal validity. In theory, assessing risk of bias is the preferred approach because a review might have good methodological quality while still being at high risk of bias. For example, a systematic review might have been conducted according to stated guidance, but some relevant databases were not searched for evidence (database selection bias) leaving out crucial primary studies that may affect the results of the review.

In general, assessors found that AMSTAR-2 was more straightforward and user friendly than ROBIS. 18 19 The two tools had similar inter-rater reliability. 18 20 21 The range in time taken to use AMSTAR-2 was similar to ROBIS (14-60 v 16-60 min) across three comparison studies 18 20 21 ( table 1 ). ROBIS users required training and practice in using the tool 22 23 and it was often understood and applied differently. 20 AMSTAR-2 has been criticised for unclear guidance on some items, 24–26 which can lead to varying interpretations and applications. ROBIS is accompanied by voluminous guidance, which can be difficult to manage by the user. 21–23

• View inline

Comparison of the AMSTAR-2 and ROBIS tools

While AMSTAR-2 and ROBIS are both widely used tools for assessing systematic reviews, in some situations, one may be preferred over the other. AMSTAR-2 may be preferred when:

the primary focus is evaluating the methodological quality of a systematic review of interventions;

the aim is to broadly assess aspects of review conduct, reporting comprehensiveness, and risk of bias; or

a relatively quick and easy to use tool is sought, because AMSTAR-2 has fewer items compared with ROBIS.

ROBIS may be preferred when:

the aim is to identify concerns with the review conduct that may point to risk of biases in the results and conclusions, as well as assessing relevance and minimising interpretation bias or ‘spin’;

a more nuanced tool is sought, which may involve more thoughtful assessment and time, because ROBIS contains more items compared with AMSTAR-2;

the aim is to assess multiple types of systematic reviews to compare risk of bias across them (eg, when preparing a clinical practice guideline).

Reporting and interpretation

When reporting and interpreting the overview results, assessors should note some key considerations with AMSTAR-2 and ROBIS assessments. Authors should first report methodological quality or bias assessment results by item, domain, and overall judgement. In addition, assessment should be reported at the outcome level as opposed to the systematic review level. 18 Several responses to AMSTAR-2 item 13 (whether risk of bias was discussed or interpreted) are possible when multiple outcomes (eg, mortality and adverse events) are reported in one systematic review. Ideally, results of intervention overviews should be reported by qualifying the inherent methodological quality or risk of bias in the included systematic reviews as potential limitations.

Subgrouping systematic reviews by low and high risk of bias using ROBIS can be a great way to determine whether authors of reviews of interventions that have a high risk of bias over emphasised their findings and conclusions. Subgrouping also allows overview authors to exclude systematic reviews that are at a high risk of bias from the synthesis. However, using only one single criteria (ie, the systematic reviews at low risk of bias) for inclusion in analyses can result in unintended loss of information through exclusion of important systematic review data (eg, by excluding the systematic review with the greatest number of unique trials).

Conclusions

Overviews are used by guideline developers and policy makers to summarise large bodies of evidence in consideration of interventions of interest on a given topic. Using the appropriate tools to critically appraise included systematic reviews of intervention effects means that a complete assessment of methodological quality and all the potential biases are considered. Systematic reviews vary considerably by method, how data are synthesised, and how results and conclusions are reported, therefore. an assessment of potential biases is necessary to consider their reproducibility, trustworthiness, and usefulness for end users. At this time, the recommended tools to assess methodological quality and bias among systematic reviews included in overviews are AMSTAR-2 and ROBIS. Proper use of these tools for this purpose requires training, time, and methodological insight.

• Higgins JP ,
• Chandler J , et al
• Neelakant T ,
• Chen A , et al
• Hartling L ,
• Chisholm A ,
• Thomson D , et al
• Pollock A ,
• Campbell P , et al
• Buechter R ,
• Jerinic P , et al
• Begley CM , et al
• Thomson D ,
• Russell K ,
• Becker L , et al
• Brennan SE ,
• McDonald S , et al
• Whiting P ,
• Savović J , et al
• Reeves BC ,
• Wells G , et al
• Savović J ,
• Higgins JPT , et al
• Veroniki A-A ,
• Veroniki AA ,
• Hutton B , et al
• Pollock M ,
• Fernandes RM ,
• Becker LA , et al
• Aromataris E ,
• Fernandez RS ,
• Godfrey C , et al
• Normansell R ,
• Waterson S , et al
• Mollon P , et al
• Whitmarsh A ,
• Leach V , et al
• Eikermann M
• Duarte G , et al
• González-Lorenzo M , et al
• Cinquini M ,
• Gonzalez-Lorenzo M , et al
• Prengel P , et al
• Holmer HK ,
• Wegewitz U ,
• Weikert B ,
• Fishta A , et al

X @carole_lunny, @PierreThabet, @Bugiukas

Contributors CL conceived the idea of the study and drafted the manuscript; CL, PS, DW, KB, SK, and ABH edited the manuscript and read and approved the final manuscript.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests We have read and understood the BMJ policy on declaration of interests and declare the following interests: none.

Provenance and peer review Commissioned; externally peer reviewed.

Read the full text or download the PDF:

Official websites use .gov

A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS

A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

A Framework for Patient-Centered Health Risk Assessments

The Affordable Care Act (ACA), Section 4103 , requires that a health risk assessment be included in the annual wellness visit benefit authorized for Medicare beneficiaries under the Act. CDC has collaborated with the Centers for Medicare and Medicaid Services (CMS) to develop an evidence-informed framework document for this type of assessment, A Framework for Patient-Centered Health Risk Assessments: Providing Health Promotion and Disease Prevention Services to Medicare Beneficiaries [PDF, 3 MB] .

This framework includes sections on Use of HRAs and Follow-Up Interventions that evidence suggests can influence health behaviors; Defining the HRA Framework (i.e., HRA Plus process ) and rationale for its use; History of Health Risk Assessments, and a Suggested Set of HRA Questions.

Background on Developing this Document

In addition to reviewing the most recent literature, CDC convened internal and external workgroups to provide input into the development process for this framework.

A public request for information published in the Federal Register on November 16, 2010, was open for comment until January 3, 2011. All of those comments are available in this document, Final Comments on HRAs [PDF, 3.5MB] .

CDC convened a public forum in Atlanta at CDC headquarters on February 1–2, 2011. A compilation of the public forum proceedings provides additional input into the guidance.

Exit Notification / Disclaimer Policy

• The Centers for Disease Control and Prevention (CDC) cannot attest to the accuracy of a non-federal website.
• Linking to a non-federal website does not constitute an endorsement by CDC or any of its employees of the sponsors or the information and products presented on the website.
• You will be subject to the destination website's privacy policy when you follow the link.
• CDC is not responsible for Section 508 compliance (accessibility) on other federal or private website.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

FDA’s Benefit–Risk Framework for Human Drugs and Biologics: Role in Benefit–Risk Assessment and Analysis of Use for Drug Approvals

Leila lackey.

1 Center for Drug Evaluation and Research (CDER), Food and Drug Administration (FDA), Silver Spring, USA

2 10903 New Hampshire Ave, WO Bldg 51 Rm 1141, Silver Spring, MD 20993, USA

Graham Thompson

Sara eggers.

Author Contributions

Associated Data

Structured, descriptive approaches are utilized by drug regulatory agencies to support and communicate approval decisions about human drugs and biologics. The US Food and Drug Administration (FDA) uses the Benefit–Risk Framework (BRF), which has been integrated into its drug review process. This paper reviews how FDA review teams have used the BRF to communicate approval decisions.

This paper (1) uses content analysis to systematically review the decision factors communicated by FDA review teams in all BRFs associated with novel drugs approved by FDA in 2017–2018 and (2) presents a case study about how the BRF was used for three drugs approved for HIV-1 in 2018–2019.

The content analysis found most BRFs for novel drug approvals communicate what we call an “urgent” context and complicating decision factors around benefit and/or risk; the HIV-1 case study highlights the flexibility of the structured BRF tool.

Conclusions

FDA’s BRF provides a flexible mechanism for communicating important decision factors, allowing it to support the diversity of drug approval decisions made by FDA.

Introduction

Marketing authorization of human drugs and biologics—drugs for short—by the US Food and Drug Administration (FDA) involves first an assessment of whether the drug is effective, and then an assessment of whether the benefits of the drug outweigh its risks [ 1 ]. To support this assessment, interdisciplinary review teams assess large amounts of scientific and clinical data and use the assessment to come to a regulatory decision [ 2 ]. The body of evidence available at the time of a regulatory decision is inevitably imperfect, thus creating the need for scientific and regulatory judgment to determine whether the product’s benefits outweigh the risks despite this uncertainty, and whether additional measures are needed to address this uncertainty. The quantity and diversity of information considered, and the potential impact of the decision, makes drug approval a complex decision problem.

Structured approaches to regulatory decision-making can help with synthesizing the scientific and clinical trial evidence, evaluating the benefit–risk profile, and communicating the rationale for the decision [ 3 , 4 ]. FDA and other drug regulatory bodies generally utilize systematic, structured, and qualitative approaches [ 5 ]. While there are similarities, a common approach is not used by all regulators. FDA’s approach, development of which started in 2009, is described in greater detail below. Other examples include the European Medicines Agency (EMA), which follows the PrOACT-URL framework and includes an effects table and a structured, narrative benefit–risk assessment. Japan’s Pharmaceuticals and Medical Devices Agency uses “points to be considered by the review staff,” including a check sheet for the major review considerations that would lead to a favorable or unfavorable decision [ 6 ]. Guidance to industry from the International Council on Harmonization advises inclusion of a narrative assessment of the therapeutic context, benefits and risks of the proposed product, and an overall assessment [ 7 ]. Similar approaches have been developed by the pharmaceutical industry, including the Pharmaceutical Research and Manufacturers of America Benefit–Risk Assessment Team (PhRMA BRAT) [ 8 – 10 ]. Reports from the Innovative Medicines Initiative Pharmacoepidemiological Research on Outcomes of Therapeutics by a European Consortium (IMI-PROTECT) provide a thorough review of structured qualitative benefit–risk assessment frameworks [ 11 ]. All of these approaches focus on integration of the important components of the decision: the therapeutic or decision context, expectations for benefits and risks in the real world, and assessment of the balance of those benefits and risks given the context.

FDA’s approach to structured drug benefit–risk assessment is found in the Benefit–Risk framework (BRF) [ 12 ]. Developed jointly by the Center for Drug Evaluation and Research and the Center for Biologics Evaluation and Research, the BRF was phased into new drug review for the two centers between 2013 and 2017 [ 13 , 14 ]. While visually structured in a tabular format, the BRF is fundamentally a narrative discussion of the therapeutic context (the clinical characteristics of the condition or disease, the population affected, the currently available treatments, and the degree of unmet medical need), what is known about the benefits and risks of the drug under review, an integrated assessment of the benefit–risk balance, need for risk management, and the regulatory decision. A 2019 public meeting, organized by FDA in partnership with the Duke Margolis Center for Health Policy, highlighted the key considerations FDA review teams may take into account for each dimension of the decision problem ( Table 1 provides a simplification of the key considerations table from the discussion document for the 2019 meeting) [ 15 ]. The BRF includes questions to consider to facilitate completion of the BRF, which are intended to guide the review team through the decision-making process and to provide a mechanism for documenting uncertainties and identifying and resolving disagreements. The final result provides a structured mechanism to support a decision, and also serves as an efficient communication mechanism of FDA’s rationale for the final regulatory decision.

Sample of Key Considerations for FDA’s Premarket Benefit–Risk Assessment of Human Drugs Taken from the Discussion Document for the 2019 Public Meeting on the Topic [ 15 ].

This paper looks at how the BRF has been used by FDA review teams to communicate approval decisions. We first conducted a systematic content analysis [ 16 , 17 ] for all BRFs published for novel drug approvals in 2017 and 2018 to assess how often review teams communicated different aspects of their assessment. Second, we considered BRFs for three drugs approved for HIV-1 between 2018 and 2019 to highlight the flexibility of the BRF as a structured approach for communicating drug regulatory decision-making. While these three products have the same high-level therapeutic context, differences in the specific indication and expected benefits and risks were communicated by their review team through the respective BRFs.

Content Analysis: Novel Drug Approvals 2017–2018

Materials and methods.

We considered the 105 new molecular entities or original biologics (referred to here as novel drugs) approved by FDA between January 1, 2017 and December 31, 2018 [ 18 , 19 ]. We obtained all available BRFs ( n = 237; several products had multiple BRFs, for instance if multiple members of the review team completed BRFs as part of separate review documents) in review documents. Of the 105 novel drug approvals, 104 had at least one BRF; the drug with no BRFs, safinamide, was approved in 2017 on the second cycle and was dropped from the analysis. The unit of analysis was the drug; Table 2 provides the review division for all 104 products.

Clinical Review Divisions for the Novel Drug Approvals Included in the Content Analysis.

Content analysis is a systematic method for decomposing and classifying written documents to identify common themes [ 16 , 17 ]. Analysts carefully and systematically review each document and assign codes to portions of the text. The coded portions can then be qualitatively reviewed across documents to identify common themes. In this paper, the codes were also quantitatively assessed by determining the number of drugs that had a code assigned to one or more of its BRFs.

The set of codes is defined in a codebook, which in this case was based on the guiding prompts given to FDA review teams for completing the BRF and was similar to the key considerations in Table 1 . The codebook had only minor additions and modifications after coding began and the final definitions are provided in supplemental materials . Codes for the therapeutic context elements of the BRF (analysis of condition and current treatment options) were designed to be comprehensive and to indicate different degrees of condition severity and unmet medical need: for example, life-threatening or life-shortening , severe for a subset of patients , or mild and/or self-limiting for the large majority of patients . Codes pertaining to the benefit, risk, and integrated assessment elements of the BRF were designed to be used when the text of the BRF communicated that the factor was an actual or potential decision issue that was either resolved, or remained unresolved, at the time of approval. This allowed the content analysis to focus on BRFs that communicated situations where nuanced FDA judgement was required for one or more decision component. These differentiated from cases where the BRF communicated a straightforward assessment of the decision component. While few decisions are “close-calls,” many, especially for novel drug therapies, have at least one component in their development requiring careful consideration and judgement by FDA.

Coding was not limited to the corresponding rows of the table as review teams sometimes discussed concepts outside of a particular row. A few BRFs, particularly in 2017, used a different tabular format but were coded anyway as the text clearly aligned with the dimensions of the BRF. In particular, some BRF tables did not include an integrated assessment section. In these cases, we considered review text either immediately preceding or following the BRF, depending on the document, to comprise the integrated assessment.

The majority of drugs had more than one BRF because individual members of the review team authored separate BRFs for the same application and indication (32 drugs in 2017 and 23 in 2018). In a subset of these (2 each year), drugs were also reviewed for two separate indications, either in a single BRF (1 each year) or in two separate BRFs (1 each year). All BRFs in review documents were analyzed and in most cases the coded results were similar between BRFs. Of the 55 drugs with more than one BRF, only 10 had multiple codes related to the therapeutic context (5 each in the 2017 and 2018 cohorts). For example, one indication may have had some approved treatments available , but the second indication sought had no approved treatments available . Multiple codes could also occur if BRFs authored by different members of the review team communicated different therapeutic contexts.

All coding was completed in NVivo Pro (versions 10, 11, and 12 were used over the course of the project). To begin, we randomly selected four drugs from each year to be fully coded by each of the two coders. Discussion of the eight drugs served to align the two coders on the codebook and its application. Then, each of the 96 remaining drugs were assigned a primary coder who reviewed all BRFs for that drug and coded in NVivo according to the agreed-upon codebook. Final codes were determined by consensus among the two coders. Because our review was limited to the text of the BRF, we can make no judgement about the degree to which the text accurately reflects the internal deliberation process or the full set of decision factors the review team considered.

When coding was complete, we used NVivo’s cross-tabulation function to tally the number of times each code was used among BRFs for each drug. We utilized the cross-tabulation, together with review of coded excerpts, to identify the themes presented below. Results discussed below are not the full accounting of all coding. Selected quotes are used solely to illustrate how the code or theme was communicated by review teams in the BRF and no significance should be ascribed to the selection of any drug.

The majority of novel drug approvals in our analysis cohort, 76% in 2017 ( n = 45) and 86% in 2018 ( n = 59), had BRFs that described both a condition that is severe or life-threatening for all or a subset of patients (93% in 2017 and 97% in 2018) and to have an unmet medical need for all or a subset of patients (78% in 2017 and 86% in 2018) ( Fig. 1 ); we describe the overlap ( severe or life-threatening and unmet medical need ) as an “urgent context.” For example, a BRF for cerliponase alfa, approved in 2017 for ceroid lipofuscinosis type 2, describes the condition as: “a rare, devastating neurological degenerative disease of early childhood…death typically occurs between 10 and 16 years of age…there are no pharmacological treatments.” Unmet medical need was not limited to conditions where no treatments were available. For example, a BRF for burosumab, approved in 2018 for an inherited form of rickets, stated: “conventional therapy has major drawbacks…better therapies are urgently needed.”

Percent of novel drug approvals (2017–2018) with BRFs that communicated decision factors related to the therapeutic context (2017 n = 45; 2018 n = 59); urgent context is the combination of severe or life-threatening and unmet medical need for all or a subset of patients.

Issues, or potential issues, related to establishing the benefit of a novel drug were discussed in BRFs for 58% of approvals in 2017 and 68% of approvals in 2018 ( Fig. 2 ). As these are all approvals, these issues or potential issues were discussed by the review team but ultimately did not preclude approval. These issues included clinical relevance of the endpoint , clinical meaningfulness of the effect , real-world benefit , limitations or weaknesses in the trial , and whether the evidentiary standard has been met . A single BRF, or the collection of BRFs for a given drug, could have more than one issue coded. For example, a BRF for tagraxofusp, approved in 2018 for blastic plasmacytoid dendritic cell neoplasm, described the steps taken to determine the clinical relevance of the endpoint : “this novel endpoint was assessed by qualification in the patients in the initial cohorts of the study and affirmed by FDA prior to analysis of the pivotal cohort.” An example the clinical meaningfulness of the effect comes from a BRF for edaravone, approved in 2017 for amyotrophic lateral sclerosis (ALS): “prevention of even 1 unit of worsening in a single domain seems meaningful and desirable for individuals with ALS.” An example of the uncertainty about real-world benefit comes from a BRF for ivosidenib, approved in 2018 for myeloid leukemia: “follow-up is too short to determine whether there is a long-term benefit or substantial effect on survival from use of this differentiating agent.” As an example of limitations or weaknesses in the trial that did not preclude approval, a BRF for benralizumab, approved in 2017 for severe asthma with an eosinophilic phenotype, states: “a sufficiently powered study to demonstrate a treatment benefit would be impractical to conduct given the rarity of this severe asthma phenotype.” Finally, an example of discussion of whether the evidentiary standard has been met can be found in a BRF for fish oil triglycerides, approved in 2018 as a source of calories and fatty acids in pediatric patients with parenteral nutrition-associated cholestasis, where: “additional evidence was reviewed to support the proposed indication due to the above limitations. We determined [fish oil triglycerides]’s effect by comparing [fish oil triglyceride] study data to gender and age-standardized growth charts.”

Percent of novel drug approvals (2017–2018) with BRFs that communicated decision factors related to benefit (2017 n = 45; 2018 n = 59).

Serious safety concerns or limitations to assessing the safety profile were discussed in BRFs for 64% of approvals in 2017 and 56% of approvals in 2018 ( Fig. 3 ). Serious safety concerns included novel, unexpected, or otherwise significant risks, as well as risks that were described as being serious but well known —expected or routine for the drug class or indication. As these are all approvals, the review teams ultimately concluded that these risk-related issues did not preclude approval. An example of a serious safety concern comes from a BRF for brodalumab, approved in 2017 for psoriasis, which discussed: “a unique safety concern that emerged…[was] the observed incidence of completed suicide…this number of suicides in a psoriasis development program is unprecedented.” In contrast, an example of a serious but well-known safety risk comes from a BRF for avelumab, approved in 2017 for metastatic Merkel cell carcinoma, which determined: “avelumab demonstrated [adverse reactions] previously identified for this class of molecules…the incidences… were not excessive for this class of molecule.” An example of limitations to assessing the safety profile is found in a BRF for larotrectinib, approved in 2018 for cancer with specific genetic features: “due to the small number of pediatric and adult patients, the single arm design of clinical studies, and confounding factors…it is not possible to determine whether differences in the incidence of adverse reactions…are related to patient age or other factors.”

Percent of novel drug approvals (2017–2018) with BRFs that communicated decision factors related to risk (2017 n = 45; 2018 n = 59).

Overall, a little more than a third of all novel drug approvals (38% in 2017 and 37% in 2018) had BRFs that described an urgent therapeutic context, discussed one or more issue related to establishing benefit, and discussed a serious safety concern (or serious but well known ) or limitations to assessing the safety profile . These complicated tradeoffs frequently were discussed in the BRFs through extended discussions of how decision factors were weighed against one another (18% in 2017 and 19% in 2018). For example, a BRF for lofexidine, approved in 2018 for opioid withdrawal, discussed the benefit–risk tradeoffs for the drug and concluded it to be favorable for patients: “who are not at undue risk for clinically significant cardiovascular adverse effects,” and for patients: “at increased risk for the cardiovascular adverse effects of lofexidine provided that they can be adequately monitored and that their withdrawal could interfere with reaching their treatment goals or is otherwise clinically significant enough to justify the risk.” Occasionally, disagreements among the review team were noted (7% of 45 novel drugs approved in 2017 and 3% of 59 novel drugs approved in 2018). For example, a BRF for ocrelizumab, approved in 2017 for multiple sclerosis, states: “The review team is in partial agreement… that the applicant has provided substantial evidence of effectiveness for the use of [ocrelizumab] for the treatment of patients with primary progressive multiple sclerosis.” Not every approval identified or discussed issues related to benefit or risk in their BRFs; 33% of approvals in 2017 and 19% of approvals in 2018 had BRFs that did not communicate decision issues related to benefit or risk. For example, a BRF for netarsudil, approved in 2017 for glaucoma, noted that the product represents a “new class” of therapy; “was equivalent to a known effective product” as assessed using the “currently accepted standard;” and “no long term consequences of netarsudil administration have been identified.”

Case Study: Selected Drugs Approved for HIV-1 in 2018–2019

After completing the systematic content analysis, we conducted a case study to explore how the BRF has been used to capture nuanced considerations of benefit, risk, and clinical context. This case study examined BRFs for three drugs approved for human immunodeficiency virus 1 (HIV-1) infection between 2018 and 2019:

• Doravirine: novel drug approved in 2018 as part of a three-drug fixed-dose combination product (doravirine/lamivudine/tenofovir); taken orally once daily; indicated as a complete regimen for the treatment of HIV-1 infection in adult patients with no antiretroviral treatment history.
• Dolutegravir/lamivudine: fixed-dose combination product approved in 2019; combination of two previously approved drugs; taken orally once daily; indicated to treat HIV-1 infection in patients with no antiretroviral treatment history.
• Ibalizumab: novel monoclonal antibody approved in 2018; administered by intravenous injection every two weeks; indicated for the treatment of HIV-1 infection in heavily treatment-experienced adult patients with documented multi-drug resistant HIV (MDR HIV).

We drew from the publicly available BRFs to develop a table highlighting key benefit–risk considerations for each product. Table 3 summarizes the distinguishing considerations from each BRF using quotes from the published frameworks.

Selected Quotes from HIV-1 BRFs (2018–2019).

The content analysis of BRFs for novel drugs approvals between 2017 and 2018 shows how the BRF was used by FDA review teams to communicate the therapeutic context, benefit–risk assessment, and regulatory decision. The large percentage of drugs with BRFs communicating an urgent decision context is notable but perhaps not unexpected in the context of novel drugs and given recent, industry-wide trends to focus novel drug development on serious, rare diseases. Many times, review teams communicated an urgent context alongside discussion of potential concerns or issued related to establishing benefit and discussion of potentially serious safety concerns.

The subjectivity of the coding process in the content analysis paralleled the judgements inherent in drug regulatory decision-making. The greatest discussion between the two coders centered around how to code BRF communication of the therapeutic context; as coding requires some amount of judgement and inference based on sometimes limited text, discussion between the coders was necessary to resolve divergent codes. For FDA review teams, the therapeutic context defines the unmet medical need and, by extension, regulatory judgement about tolerance for uncertainty and acceptability of benefit–risk tradeoffs.

The case study of BRFs for drugs approved for HIV-1 highlights the flexibility of a structured, descriptive approach for benefit–risk assessment for drug regulatory decision-making. From the BRF for dolutegravir/lamivudine: “optimal management of HIV-1 is complex and must consider patients’ individual needs.” While each of these products was developed for treatment of HIV-1 infection, the benefit–risk assessment for each product had distinct considerations. Differences in the therapeutic context, unique therapeutic benefits of each drug, safety considerations, and limitations due to available trial data led to different benefit–risk profiles and different effects on the treatment armamentarium for HIV-1. Doravirine was approved as part of a three-drug regimen and provides an additional antiretroviral treatment option. Dolutegravir/lamivudine is the first two-drug complete regimen approved for HIV-1 and may provide a safety advantage for some patients by removing the concerns associated with a third drug. Ibalizumab had trial design issues, which introduced some uncertainty about the durability of benefit, but is intended for use in a high-risk subset of HIV-1 patients with few, if any, treatment options.

None of the reviewed BRFs in the content analysis or case study incorporated explicit statements of preferences or utilized benefit–risk modeling or analysis approaches. Indeed, it would be challenging to incorporate all of the complex decision factors communicated in each BRF into an integrated model. Nevertheless, there is potential for certain approaches to provide useful information for decision-making. Where additional benefit–risk analyses, sometimes including quantitative benefit–risk analysis approaches, may be useful by FDA to inform the decision, results can be communicated within the context of the BRF. Identification of the necessary situations, and the appropriate methodologies, remains an area of exploration for FDA and others [ 14 , 20 , 21 ].

Our content analysis and case study show the diversity of drug regulatory decisions and decision factors that can be communicated through a structured, descriptive benefit–risk assessment. FDA continues to refine its BRF approach and to more systematically integrate it into the drug review process [ 14 ]. The results presented here focus exclusively on how FDA’s benefit–risk assessment was communicated through completed BRFs. A potential future inquiry could be to compare FDA’s BRF to the benefit–risk approaches of EMA [ 22 ] or other organizations—both in terms of methodology and by comparing published benefit–risk assessments for products reviewed by both organizations. Various attempts have been made throughout the years to develop “unified” [ 23 ] benefit–risk assessment frameworks and models but so far only structured, descriptive approaches have been widely implemented [ 5 ].

This paper demonstrates the utility of having a succinct, structured, descriptive benefit–risk assessment tool to highlight and communicate important decision factors. In 2017 and 2018, BRFs for novel drug approvals by FDA tend to communicate serious conditions with unmet medical need, complex considerations for establishing benefit, and the potential for significant risks. A case study of selected 2018–2019 approvals for HIV-1 illustrates the flexibility of the BRF for communicating the nuances of the therapeutic context and drug regulatory decision. FDA is continuing to develop the qualitative framework and to explore decision analysis techniques that could support assessment of these complicated issues.

Supplementary Material

Online supplemental material, acknowledgements.

We thank the Division of Antiviral Products (Office of New Drugs, CDER, FDA) for helpful discussion to inform the case study. We also thank staff and leadership in the Office of New Drugs (CDER, FDA) for helpful feedback.

Conflict of interest

The authors are employed by the Center for Drug Evaluation and Research at the FDA. They have no other conflicts to disclose. This publication reflects the views of the authors and should not be construed to represent FDA’s views or policies.

Electronic supplementary material

The online version of this article ( https://doi.org/10.1007/s43441-020-00203-6 ) contains supplementary material , which is available to authorized users.

• Skip to main content
• Skip to FDA Search
• Skip to in this section menu
• Skip to footer links

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

U.S. Food and Drug Administration

•   Search
•   Menu
• For Consumers
• Consumer Updates

Antibacterial Soap? You Can Skip It, Use Plain Soap and Water

When you buy soaps and body washes, do you reach for products labeled “antibacterial” hoping they’ll keep your family safer? Do you think those products will lower your risk of getting sick, spreading germs or being infected?

According to the U.S. Food and Drug Administration (FDA), there isn’t enough science to show that over-the-counter (OTC) antibacterial soaps are better at preventing illness than washing with plain soap and water. To date, the benefits of using antibacterial hand soap haven’t been proven. In addition, the wide use of these products over a long time has raised the question of potential negative effects on your health.

After studying the issue, including reviewing available literature and hosting public meetings, in 2013 the FDA issued a proposed rule requiring safety and efficacy data from manufacturers, consumers, and others if they wanted to continue marketing antibacterial products containing those ingredients, but very little information has been provided. That’s why the FDA is issuing a final rule under which OTC consumer antiseptic wash products (including liquid, foam, gel hand soaps, bar soaps, and body washes) containing the majority of the antibacterial active ingredients—including triclosan and triclocarban—will no longer be able to be marketed.

Why? Because the manufacturers haven’t proven that those ingredients are safe for daily use over a long period of time. Also, manufacturers haven’t shown that these ingredients are any more effective than plain soap and water in preventing illnesses and the spread of certain infections. Some manufacturers have already started removing these ingredients from their products, ahead of the FDA’s final rule.

“Following simple handwashing practices is one of the most effective ways to prevent the spread of many types of infection and illness at home, at school and elsewhere,” says Theresa M. Michele, MD, of the FDA’s Division of Nonprescription Drug Products. “We can’t advise this enough. It’s simple, and it works.”

The FDA’s final rule covers only consumer antibacterial soaps and body washes that are used with water. It does not apply to hand sanitizers or hand wipes. It also does not apply to antibacterial soaps that are used in health care settings, such as hospitals and nursing homes.

What Makes Soap ‘Antibacterial’

Antibacterial soaps (sometimes called antimicrobial or antiseptic soaps) contain certain chemicals not found in plain soaps. Those ingredients are added to many consumer products with the intent of reducing or preventing bacterial infection.

Many liquid soaps labeled antibacterial contain triclosan, an ingredient of concern to many environmental, academic and regulatory groups. Animal studies have shown that triclosan alters the way some hormones work in the body and raises potential concerns for the effects of use in humans. We don’t yet know how triclosan affects humans and more research is needed.

“There’s no data demonstrating that these drugs provide additional protection from diseases and infections. Using these products might give people a false sense of security,” Michele says. “If you use these products because you think they protect you more than soap and water, that’s not correct. If you use them because of how they feel, there are many other products that have similar formulations but won’t expose your family to unnecessary chemicals. And some manufacturers have begun to revise these products to remove these ingredients.”

How do you tell if a product is antibacterial? For OTC drugs, antibacterial products generally have the word “antibacterial” on the label. Also, a Drug Facts label on a soap or body wash is a sign a product contains antibacterial ingredients.

Triclosan and Health Concerns

Triclosan can be found in many places today. It has been added to many consumer products—including clothing, kitchenware, furniture, and toys—to prevent bacterial contamination. Because of that, people’s long-term exposure to triclosan is higher than previously thought, raising concerns about the potential risks associated with the use of this ingredient over a lifetime.

In addition, laboratory studies have raised the possibility that triclosan contributes to making bacteria resistant to antibiotics. Some data shows this resistance may have a significant impact on the effectiveness of medical treatments, such as antibiotics.

The FDA and the Environmental Protection Agency (EPA) have been closely collaborating on scientific and regulatory issues related to triclosan. This joint effort will help to ensure government-wide consistency in the regulation of this chemical. The two agencies are reviewing the effects of triclosan from two different perspectives.

The EPA regulates the use of triclosan as a pesticide, and is in the process of updating its assessment of the effects of triclosan when it is used in pesticides. The FDA’s focus is on the effects of triclosan when it is used by consumers on a regular basis in hand soaps and body washes. By sharing information, the two agencies will be better able to measure the exposure and effects of triclosan and how these differing uses of triclosan may affect human health.

The EPA reevaluates each pesticide active ingredient every 15 years. The EPA’s Final Work Plan for the triclosan risk assessment can be found in docket EPA-HQ-OPP-2012-0811.

More on the FDA’s Rule

The FDA’s rule doesn’t yet apply to three chemicals (benzalkonium chloride, benzethonium chloride and chloroxylenol). Manufacturers are developing and planning to submit new safety and effectiveness data for these ingredients.

With the exception of those three ingredients that are still under study, all products that use the other 19 active ingredients will need to change their formulas or they will no longer be available to consumers. Manufacturers will have one year to comply with the rule.

This rule doesn’t apply to hand sanitizers. The FDA recently issued a final rule on OTC hand sanitizers and will continue to review the three active ingredients commonly used in hand sanitizers. To learn about the difference between consumer hand sanitizers and consumer antibacterial soaps, visit our consumer information page .

Consumers, Keep Washing with Plain Soap and Water

So what should consumers do? Wash your hands with plain soap and water. That’s still one of the most important steps you can take to avoid getting sick and to prevent spreading germs.

Consumer Updates Email

Subscribe to receive Consumer Updates email notifications.

The state of AI in 2023: Generative AI’s breakout year

You have reached a page with older survey data. please see our 2024 survey results here ..

The latest annual McKinsey Global Survey  on the current state of AI confirms the explosive growth of generative AI (gen AI) tools . Less than a year after many of these tools debuted, one-third of our survey respondents say their organizations are using gen AI regularly in at least one business function. Amid recent advances, AI has risen from a topic relegated to tech employees to a focus of company leaders: nearly one-quarter of surveyed C-suite executives say they are personally using gen AI tools for work, and more than one-quarter of respondents from companies using AI say gen AI is already on their boards’ agendas. What’s more, 40 percent of respondents say their organizations will increase their investment in AI overall because of advances in gen AI. The findings show that these are still early days for managing gen AI–related risks, with less than half of respondents saying their organizations are mitigating even the risk they consider most relevant: inaccuracy.

The organizations that have already embedded AI capabilities have been the first to explore gen AI’s potential, and those seeing the most value from more traditional AI capabilities—a group we call AI high performers—are already outpacing others in their adoption of gen AI tools. 1 We define AI high performers as organizations that, according to respondents, attribute at least 20 percent of their EBIT to AI adoption.

The expected business disruption from gen AI is significant, and respondents predict meaningful changes to their workforces. They anticipate workforce cuts in certain areas and large reskilling efforts to address shifting talent needs. Yet while the use of gen AI might spur the adoption of other AI tools, we see few meaningful increases in organizations’ adoption of these technologies. The percent of organizations adopting any AI tools has held steady since 2022, and adoption remains concentrated within a small number of business functions.

Table of Contents

• It’s early days still, but use of gen AI is already widespread
• Leading companies are already ahead with gen AI
• AI-related talent needs shift, and AI’s workforce effects are expected to be substantial
• With all eyes on gen AI, AI adoption and impact remain steady

About the research

1. it’s early days still, but use of gen ai is already widespread.

The findings from the survey—which was in the field in mid-April 2023—show that, despite gen AI’s nascent public availability, experimentation with the tools  is already relatively common, and respondents expect the new capabilities to transform their industries. Gen AI has captured interest across the business population: individuals across regions, industries, and seniority levels are using gen AI for work and outside of work. Seventy-nine percent of all respondents say they’ve had at least some exposure to gen AI, either for work or outside of work, and 22 percent say they are regularly using it in their own work. While reported use is quite similar across seniority levels, it is highest among respondents working in the technology sector and those in North America.

Organizations, too, are now commonly using gen AI. One-third of all respondents say their organizations are already regularly using generative AI in at least one function—meaning that 60 percent of organizations with reported AI adoption are using gen AI. What’s more, 40 percent of those reporting AI adoption at their organizations say their companies expect to invest more in AI overall thanks to generative AI, and 28 percent say generative AI use is already on their board’s agenda. The most commonly reported business functions using these newer tools are the same as those in which AI use is most common overall: marketing and sales, product and service development, and service operations, such as customer care and back-office support. This suggests that organizations are pursuing these new tools where the most value is. In our previous research , these three areas, along with software engineering, showed the potential to deliver about 75 percent of the total annual value from generative AI use cases.

In these early days, expectations for gen AI’s impact are high : three-quarters of all respondents expect gen AI to cause significant or disruptive change in the nature of their industry’s competition in the next three years. Survey respondents working in the technology and financial-services industries are the most likely to expect disruptive change from gen AI. Our previous research shows  that, while all industries are indeed likely to see some degree of disruption, the level of impact is likely to vary. 2 “ The economic potential of generative AI: The next productivity frontier ,” McKinsey, June 14, 2023. Industries relying most heavily on knowledge work are likely to see more disruption—and potentially reap more value. While our estimates suggest that tech companies, unsurprisingly, are poised to see the highest impact from gen AI—adding value equivalent to as much as 9 percent of global industry revenue—knowledge-based industries such as banking (up to 5 percent), pharmaceuticals and medical products (also up to 5 percent), and education (up to 4 percent) could experience significant effects as well. By contrast, manufacturing-based industries, such as aerospace, automotives, and advanced electronics, could experience less disruptive effects. This stands in contrast to the impact of previous technology waves that affected manufacturing the most and is due to gen AI’s strengths in language-based activities, as opposed to those requiring physical labor.

Responses show many organizations not yet addressing potential risks from gen AI

According to the survey, few companies seem fully prepared for the widespread use of gen AI—or the business risks these tools may bring. Just 21 percent of respondents reporting AI adoption say their organizations have established policies governing employees’ use of gen AI technologies in their work. And when we asked specifically about the risks of adopting gen AI, few respondents say their companies are mitigating the most commonly cited risk with gen AI: inaccuracy. Respondents cite inaccuracy more frequently than both cybersecurity and regulatory compliance, which were the most common risks from AI overall in previous surveys. Just 32 percent say they’re mitigating inaccuracy, a smaller percentage than the 38 percent who say they mitigate cybersecurity risks. Interestingly, this figure is significantly lower than the percentage of respondents who reported mitigating AI-related cybersecurity last year (51 percent). Overall, much as we’ve seen in previous years, most respondents say their organizations are not addressing AI-related risks.

2. Leading companies are already ahead with gen AI

The survey results show that AI high performers—that is, organizations where respondents say at least 20 percent of EBIT in 2022 was attributable to AI use—are going all in on artificial intelligence, both with gen AI and more traditional AI capabilities. These organizations that achieve significant value from AI are already using gen AI in more business functions than other organizations do, especially in product and service development and risk and supply chain management. When looking at all AI capabilities—including more traditional machine learning capabilities, robotic process automation, and chatbots—AI high performers also are much more likely than others to use AI in product and service development, for uses such as product-development-cycle optimization, adding new features to existing products, and creating new AI-based products. These organizations also are using AI more often than other organizations in risk modeling and for uses within HR such as performance management and organization design and workforce deployment optimization.

AI high performers are much more likely than others to use AI in product and service development.

Another difference from their peers: high performers’ gen AI efforts are less oriented toward cost reduction, which is a top priority at other organizations. Respondents from AI high performers are twice as likely as others to say their organizations’ top objective for gen AI is to create entirely new businesses or sources of revenue—and they’re most likely to cite the increase in the value of existing offerings through new AI-based features.

As we’ve seen in previous years , these high-performing organizations invest much more than others in AI: respondents from AI high performers are more than five times more likely than others to say they spend more than 20 percent of their digital budgets on AI. They also use AI capabilities more broadly throughout the organization. Respondents from high performers are much more likely than others to say that their organizations have adopted AI in four or more business functions and that they have embedded a higher number of AI capabilities. For example, respondents from high performers more often report embedding knowledge graphs in at least one product or business function process, in addition to gen AI and related natural-language capabilities.

While AI high performers are not immune to the challenges of capturing value from AI, the results suggest that the difficulties they face reflect their relative AI maturity, while others struggle with the more foundational, strategic elements of AI adoption. Respondents at AI high performers most often point to models and tools, such as monitoring model performance in production and retraining models as needed over time, as their top challenge. By comparison, other respondents cite strategy issues, such as setting a clearly defined AI vision that is linked with business value or finding sufficient resources.

The findings offer further evidence that even high performers haven’t mastered best practices regarding AI adoption, such as machine-learning-operations (MLOps) approaches, though they are much more likely than others to do so. For example, just 35 percent of respondents at AI high performers report that where possible, their organizations assemble existing components, rather than reinvent them, but that’s a much larger share than the 19 percent of respondents from other organizations who report that practice.

Many specialized MLOps technologies and practices  may be needed to adopt some of the more transformative uses cases that gen AI applications can deliver—and do so as safely as possible. Live-model operations is one such area, where monitoring systems and setting up instant alerts to enable rapid issue resolution can keep gen AI systems in check. High performers stand out in this respect but have room to grow: one-quarter of respondents from these organizations say their entire system is monitored and equipped with instant alerts, compared with just 12 percent of other respondents.

3. AI-related talent needs shift, and AI’s workforce effects are expected to be substantial

Our latest survey results show changes in the roles that organizations are filling to support their AI ambitions. In the past year, organizations using AI most often hired data engineers, machine learning engineers, and Al data scientists—all roles that respondents commonly reported hiring in the previous survey. But a much smaller share of respondents report hiring AI-related-software engineers—the most-hired role last year—than in the previous survey (28 percent in the latest survey, down from 39 percent). Roles in prompt engineering have recently emerged, as the need for that skill set rises alongside gen AI adoption, with 7 percent of respondents whose organizations have adopted AI reporting those hires in the past year.

The findings suggest that hiring for AI-related roles remains a challenge but has become somewhat easier over the past year, which could reflect the spate of layoffs at technology companies from late 2022 through the first half of 2023. Smaller shares of respondents than in the previous survey report difficulty hiring for roles such as AI data scientists, data engineers, and data-visualization specialists, though responses suggest that hiring machine learning engineers and AI product owners remains as much of a challenge as in the previous year.

Looking ahead to the next three years, respondents predict that the adoption of AI will reshape many roles in the workforce. Generally, they expect more employees to be reskilled than to be separated. Nearly four in ten respondents reporting AI adoption expect more than 20 percent of their companies’ workforces will be reskilled, whereas 8 percent of respondents say the size of their workforces will decrease by more than 20 percent.

Looking specifically at gen AI’s predicted impact, service operations is the only function in which most respondents expect to see a decrease in workforce size at their organizations. This finding generally aligns with what our recent research  suggests: while the emergence of gen AI increased our estimate of the percentage of worker activities that could be automated (60 to 70 percent, up from 50 percent), this doesn’t necessarily translate into the automation of an entire role.

AI high performers are expected to conduct much higher levels of reskilling than other companies are. Respondents at these organizations are over three times more likely than others to say their organizations will reskill more than 30 percent of their workforces over the next three years as a result of AI adoption.

4. With all eyes on gen AI, AI adoption and impact remain steady

While the use of gen AI tools is spreading rapidly, the survey data doesn’t show that these newer tools are propelling organizations’ overall AI adoption. The share of organizations that have adopted AI overall remains steady, at least for the moment, with 55 percent of respondents reporting that their organizations have adopted AI. Less than a third of respondents continue to say that their organizations have adopted AI in more than one business function, suggesting that AI use remains limited in scope. Product and service development and service operations continue to be the two business functions in which respondents most often report AI adoption, as was true in the previous four surveys. And overall, just 23 percent of respondents say at least 5 percent of their organizations’ EBIT last year was attributable to their use of AI—essentially flat with the previous survey—suggesting there is much more room to capture value.

Organizations continue to see returns in the business areas in which they are using AI, and they plan to increase investment in the years ahead. We see a majority of respondents reporting AI-related revenue increases within each business function using AI. And looking ahead, more than two-thirds expect their organizations to increase their AI investment over the next three years.

The online survey was in the field April 11 to 21, 2023, and garnered responses from 1,684 participants representing the full range of regions, industries, company sizes, functional specialties, and tenures. Of those respondents, 913 said their organizations had adopted AI in at least one function and were asked questions about their organizations’ AI use. To adjust for differences in response rates, the data are weighted by the contribution of each respondent’s nation to global GDP.

The survey content and analysis were developed by Michael Chui , a partner at the McKinsey Global Institute and a partner in McKinsey’s Bay Area office, where Lareina Yee is a senior partner; Bryce Hall , an associate partner in the Washington, DC, office; and senior partners Alex Singla and Alexander Sukharevsky , global leaders of QuantumBlack, AI by McKinsey, based in the Chicago and London offices, respectively.

They wish to thank Shivani Gupta, Abhisek Jena, Begum Ortaoglu, Barr Seitz, and Li Zhang for their contributions to this work.

This article was edited by Heather Hanselman, an editor in the Atlanta office.

Explore a career with us

Related articles.

The economic potential of generative AI: The next productivity frontier

What is generative AI?

Exploring opportunities in the generative AI value chain