is quantitative research biased

Research Bias 101: What You Need To Know

By: Derek Jansen (MBA) | Expert Reviewed By: Dr Eunice Rautenbach | September 2022

If you’re new to academic research, research bias (also sometimes called researcher bias) is one of the many things you need to understand to avoid compromising your study. If you’re not careful, research bias can ruin the credibility of your study. 

In this post, we’ll unpack the thorny topic of research bias. We’ll explain what it is , look at some common types of research bias and share some tips to help you minimise the potential sources of bias in your research.

Overview: Research Bias 101

• What is research bias (or researcher bias)?
• Bias #1 – Selection bias
• Bias #2 – Analysis bias
• Bias #3 – Procedural (admin) bias

So, what is research bias?

Well, simply put, research bias is when the researcher – that’s you – intentionally or unintentionally skews the process of a systematic inquiry , which then of course skews the outcomes of the study . In other words, research bias is what happens when you affect the results of your research by influencing how you arrive at them.

For example, if you planned to research the effects of remote working arrangements across all levels of an organisation, but your sample consisted mostly of management-level respondents , you’d run into a form of research bias. In this case, excluding input from lower-level staff (in other words, not getting input from all levels of staff) means that the results of the study would be ‘biased’ in favour of a certain perspective – that of management.

Of course, if your research aims and research questions were only interested in the perspectives of managers, this sampling approach wouldn’t be a problem – but that’s not the case here, as there’s a misalignment between the research aims and the sample .

Now, it’s important to remember that research bias isn’t always deliberate or intended. Quite often, it’s just the result of a poorly designed study, or practical challenges in terms of getting a well-rounded, suitable sample. While perfect objectivity is the ideal, some level of bias is generally unavoidable when you’re undertaking a study. That said, as a savvy researcher, it’s your job to reduce potential sources of research bias as much as possible.

To minimize potential bias, you first need to know what to look for . So, next up, we’ll unpack three common types of research bias we see at Grad Coach when reviewing students’ projects . These include selection bias , analysis bias , and procedural bias . Keep in mind that there are many different forms of bias that can creep into your research, so don’t take this as a comprehensive list – it’s just a useful starting point.

Bias #1 – Selection Bias

First up, we have selection bias . The example we looked at earlier (about only surveying management as opposed to all levels of employees) is a prime example of this type of research bias. In other words, selection bias occurs when your study’s design automatically excludes a relevant group from the research process and, therefore, negatively impacts the quality of the results.

With selection bias, the results of your study will be biased towards the group that it includes or favours, meaning that you’re likely to arrive at prejudiced results . For example, research into government policies that only includes participants who voted for a specific party is going to produce skewed results, as the views of those who voted for other parties will be excluded.

Selection bias commonly occurs in quantitative research , as the sampling strategy adopted can have a major impact on the statistical results . That said, selection bias does of course also come up in qualitative research as there’s still plenty room for skewed samples. So, it’s important to pay close attention to the makeup of your sample and make sure that you adopt a sampling strategy that aligns with your research aims. Of course, you’ll seldom achieve a perfect sample, and that okay. But, you need to be aware of how your sample may be skewed and factor this into your thinking when you analyse the resultant data.

Need a helping hand?

Bias #2 – Analysis Bias

Next up, we have analysis bias . Analysis bias occurs when the analysis itself emphasises or discounts certain data points , so as to favour a particular result (often the researcher’s own expected result or hypothesis). In other words, analysis bias happens when you prioritise the presentation of data that supports a certain idea or hypothesis , rather than presenting all the data indiscriminately .

For example, if your study was looking into consumer perceptions of a specific product, you might present more analysis of data that reflects positive sentiment toward the product, and give less real estate to the analysis that reflects negative sentiment. In other words, you’d cherry-pick the data that suits your desired outcomes and as a result, you’d create a bias in terms of the information conveyed by the study.

Although this kind of bias is common in quantitative research, it can just as easily occur in qualitative studies, given the amount of interpretive power the researcher has. This may not be intentional or even noticed by the researcher, given the inherent subjectivity in qualitative research. As humans, we naturally search for and interpret information in a way that confirms or supports our prior beliefs or values (in psychology, this is called “confirmation bias”). So, don’t make the mistake of thinking that analysis bias is always intentional and you don’t need to worry about it because you’re an honest researcher – it can creep up on anyone .

To reduce the risk of analysis bias, a good starting point is to determine your data analysis strategy in as much detail as possible, before you collect your data . In other words, decide, in advance, how you’ll prepare the data, which analysis method you’ll use, and be aware of how different analysis methods can favour different types of data. Also, take the time to reflect on your own pre-conceived notions and expectations regarding the analysis outcomes (in other words, what do you expect to find in the data), so that you’re fully aware of the potential influence you may have on the analysis – and therefore, hopefully, can minimize it.

Bias #3 – Procedural Bias

Last but definitely not least, we have procedural bias , which is also sometimes referred to as administration bias . Procedural bias is easy to overlook, so it’s important to understand what it is and how to avoid it. This type of bias occurs when the administration of the study, especially the data collection aspect, has an impact on either who responds or how they respond.

A practical example of procedural bias would be when participants in a study are required to provide information under some form of constraint. For example, participants might be given insufficient time to complete a survey, resulting in incomplete or hastily-filled out forms that don’t necessarily reflect how they really feel. This can happen really easily, if, for example, you innocently ask your participants to fill out a survey during their lunch break.

Another form of procedural bias can happen when you improperly incentivise participation in a study. For example, offering a reward for completing a survey or interview might incline participants to provide false or inaccurate information just to get through the process as fast as possible and collect their reward. It could also potentially attract a particular type of respondent (a freebie seeker), resulting in a skewed sample that doesn’t really reflect your demographic of interest.

The format of your data collection method can also potentially contribute to procedural bias. If, for example, you decide to host your survey or interviews online, this could unintentionally exclude people who are not particularly tech-savvy, don’t have a suitable device or just don’t have a reliable internet connection. On the flip side, some people might find in-person interviews a bit intimidating (compared to online ones, at least), or they might find the physical environment in which they’re interviewed to be uncomfortable or awkward (maybe the boss is peering into the meeting room, for example). Either way, these factors all result in less useful data.

Although procedural bias is more common in qualitative research, it can come up in any form of fieldwork where you’re actively collecting data from study participants. So, it’s important to consider how your data is being collected and how this might impact respondents. Simply put, you need to take the respondent’s viewpoint and think about the challenges they might face, no matter how small or trivial these might seem. So, it’s always a good idea to have an informal discussion with a handful of potential respondents before you start collecting data and ask for their input regarding your proposed plan upfront.

Let’s Recap

Ok, so let’s do a quick recap. Research bias refers to any instance where the researcher, or the research design , negatively influences the quality of a study’s results, whether intentionally or not.

The three common types of research bias we looked at are:

• Selection bias – where a skewed sample leads to skewed results
• Analysis bias – where the analysis method and/or approach leads to biased results – and,
• Procedural bias – where the administration of the study, especially the data collection aspect, has an impact on who responds and how they respond.

As I mentioned, there are many other forms of research bias, but we can only cover a handful here. So, be sure to familiarise yourself with as many potential sources of bias as possible to minimise the risk of research bias in your study.

Psst... there’s more!

This post was based on one of our popular Research Bootcamps . If you're working on a research project, you'll definitely want to check this out ...

This is really educational and I really like the simplicity of the language in here, but i would like to know if there is also some guidance in regard to the problem statement and what it constitutes.

Do you have a blog or video that differentiates research assumptions, research propositions and research hypothesis?

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

Understanding the different types of bias in research (2024 guide)

Last updated

6 October 2023

Reviewed by

Miroslav Damyanov

Short on time? Get an AI generated summary of this article instead

Research bias is an invisible force that overly highlights or dismisses the chosen study topic’s traits. When left unchecked, it can significantly impact the validity and reliability of your research.

In a perfect world, every research project would be free of any trace of bias—but for this to happen, you need to be aware of the most common types of research bias that plague studies.

Read this guide to learn more about the most common types of bias in research and what you can do to design and improve your studies to create high-quality research results.

• What is research bias?

Research bias is the tendency for qualitative and quantitative research studies to contain prejudice or preference for or against a particular group of people, culture, object, idea, belief, or circumstance.

Bias is rarely based on observed facts. In most cases, it results from societal stereotypes, systemic discrimination, or learned prejudice.

Every human develops their own set of biases throughout their lifetime as they interact with their environment. Often, people are unaware of their own biases until they are challenged—and this is why it’s easy for unintentional bias to seep into research projects .

Left unchecked, bias ruins the validity of research . So, to get the most accurate results, researchers need to know about the most common types of research bias and understand how their study design can address and avoid these outcomes.

• The two primary types of bias

Historically, there are two primary types of bias in research:

Conscious bias

Conscious bias is the practice of intentionally voicing and sharing a negative opinion about a particular group of people, beliefs, or concepts.

Characterized by negative emotions and opinions of the target group, conscious bias is often defined as intentional discrimination.

In most cases, this type of bias is not involved in research projects, as they are unjust, unfair, and unscientific.

Unconscious bias

An unconscious bias is a negative response to a particular group of people, beliefs, or concepts that is not identified or intentionally acted upon by the bias holder.

Because of this, unconscious bias is incredibly dangerous. These warped beliefs shape and impact how someone conducts themselves and their research. The trouble is that they can’t identify the moral and ethical issues with their behavior.

• Examples of commonly occurring research bias

Humans use countless biases daily to quickly process information and make sense of the world. But, to create accurate research studies and get the best results, you must remove these biases from your study design.

Here are some of the most common types of research biases you should look out for when planning your next study:

Information bias

During any study, tampering with data collection is widely agreed to be bad science. But what if your study design includes information biases you are unaware of?

Also known as measurement bias, information bias occurs when one or more of the key study variables are not correctly measured, recorded, or interpreted. As a result, the study’s perceived outcome may be inaccurate due to data misclassification, omission, or obfuscation (obscuring). 

Observer bias

Observer bias occurs when researchers don’t have a clear understanding of their own personal assumptions and expectations. During observational studies, it’s possible for a researcher’s personal biases to impact how they interpret the data. This can dramatically affect the study’s outcome.

The study should be double-blind to combat this type of bias. This is where the participants don’t know which group they are in, and the observers don’t know which group they are observing.

Regression to the mean (RTM)

Bias can also impact research statistics.

Regression of the mean (RTM) refers to a statistical bias that if a first clinical reading is extreme in value (i.e., it’s very high or very low compared to the average), the second reading will provide a more statistically normal result.

Here’s an example: you might be nervous when a doctor takes your blood pressure in the doctor’s surgery. The first result might be quite high. This is a phenomenon known as “white coat syndrome.” When your blood pressure is retaken to double-check the value, it is more likely to be closer to typical values.

So, which value is more accurate, and which should you record as the truth?

The answer depends on the specific design of your study. However, using control groups is usually recommended for studies with a high risk of RTM.

Performance bias

A performance bias can develop if participants understand the study’s nature or desired outcomes. This can harm the study’s accuracy, as participants may adjust their behavior outside of their normal to improve their performance. This results in inaccurate data and study results.

This is a common bias type in medical and health studies, particularly those studying the differences between two lifestyle choices.

To reduce performance bias, researchers should strive to keep members of the control and study groups unaware of the other group’s activities. This method is known as “blinding.”

Recall bias

How good is your memory? Chances are, it’s not as good as you think—and the older the memory, the more inaccurate and biased it will become.

A recall bias commonly occurs in self-reporting studies requiring participants to remember past information. While people can remember big-picture events (like the day they got married or landed their first job), routine occurrences like what they do after work every Tuesday are harder to recall.

To offset this type of bias, design a study that engages with participants on both short- and long-term periods to help keep the content more top of mind.

Researcher bias

Researcher bias (also known as interviewer bias) occurs due to the researcher’s personal beliefs or tendencies that influence the study’s results or outcomes.

These types of biases can be intentional or unintentional, and most are driven by personal feelings, historical stereotypes, and assumptions about the study’s outcome before it has even begun.

Question order bias

Survey design and question order is a huge area of contention for researchers. These elements are essential for quality study design and can prevent or invite answer bias.

When designing a research study that collects data via survey questions , the order of the questions presented can impact how the participants answer each subsequent question. Leading questions (questions that guide participants toward a particular answer) are perfect examples of this. When included early in the survey, they can sway a participant’s opinions and answers as they complete the questionnaire .

This is known as systematic distortion, meaning each question answered after the guiding questions is impacted or distorted by the wording of the questions before.

Demand characteristics

Body language and social cues play a significant role in human communication—and this also rings true for the validity of research projects . 

A demand characteristic bias can occur due to a verbal or non-verbal cue that encourages research participants to behave in a particular way.

Imagine a researcher is studying a group of new grad business students about their experience applying to new jobs one, three, and six months after graduation. They scowl every time a participant mentions they don’t use a cover letter. This reaction may encourage participants to change their answers, harming the study’s outcome and resulting in less accurate results.

Courtesy bias

Courtesy bias arises from not wanting to share negative or constructive feedback or answers—a common human tendency.

You’ve probably been in this situation before. Think of a time when you had a negative opinion or perspective on a topic, but you felt the need to soften or reduce the harshness of your feedback to prevent someone’s feelings from being hurt.

This type of bias also occurs in research. Without a comfortable and non-judgmental environment that encourages honest responses, courtesy bias can result in inaccurate data intake.

Studies based on small group interviews, focus groups , or any in-person surveys are particularly vulnerable to this type of bias because people are less likely to share negative opinions in front of others or to someone’s face.

Extreme responding

Extreme responding refers to the tendency for people to respond on one side of the scale or the other, even if these extreme answers don’t reflect their true opinion. 

This is a common bias in surveys, particularly online surveys asking about a person’s experience or personal opinions (think questionnaires that ask you to decide if you strongly disagree, disagree, agree, or strongly agree with a statement).

When this occurs, the data will be skewed. It will be overly positive or negative—not accurate. This is a problem because the data can impact future decisions or study outcomes.

Writing different styles of questions and asking for follow-up interviews with a small group of participants are a few options for reducing the impact of this type of bias.

Social desirability bias

Everyone wants to be liked and respected. As a result, societal bias can impact survey answers.

It’s common for people to answer questions in a way that they believe will earn them favor, respect, or agreement with researchers. This is a common bias type for studies on taboo or sensitive topics like alcohol consumption or physical activity levels, where participants feel vulnerable or judged when sharing their honest answers.

Finding ways to comfort participants with ensured anonymity and safe and respectful research practices are ways you can offset the impact of social desirability bias.

Selection bias

For the most accurate results, researchers need to understand their chosen population before accepting participants. Failure to do this results in selection bias, which is caused by an inaccurate or misrepresented selection of participants that don’t truly reflect the chosen population.

Self-selection bias

To collect data, researchers in many studies require participants to volunteer their time and experiences. This results in a study design that is automatically biased toward people who are more likely to get involved.

People who are more likely to voluntarily participate in a study are not reflective of the common experience of a broad, diverse population. Because of this, any information collected from this type of study will contain a self-selection bias .

To avoid this type of bias, researchers can use random assignment (using control versus treatment groups to divide the study participants after they volunteer).

Sampling or ascertainment bias

When choosing participants for a study, take care to select people who are representative of the overall population being researched. Failure to do this will result in sampling bias.

For example, if researchers aim to learn more about how university stress impacts sleep quality but only choose engineering students as participants, the study won’t reflect the wider population they want to learn more about.

To avoid sampling bias, researchers must first have a strong understanding of their chosen study population. Then, they should take steps to ensure that any person within that population has an equal chance of being selected for the study.

Attrition bias

People tend to be hard on themselves, so an attrition bias toward the impact of failure versus success can seep into research.

Many people find it easier to list things they struggle with rather than things they think they are good at. This also occurs in research, as people are more likely to value the impact of a negative experience (or failure) than that of a positive, successful outcome.

Survivorship bias

In medical clinical trials and studies, a survivorship bias may develop if only the results and data from participants who survived the trial are studied. Survivorship bias also includes participants who were unable to complete the entire trial, not just those who passed away during the duration of the study.

In long-term studies that evaluate new medications or therapies for high-mortality diseases like aggressive cancers, choosing to only consider the success rate, side effects, or experiences of those who completed the study eliminates a large portion of important information. This disregarded information may have offered insights into the quality, efficacy, and safety of the treatment being tested.

Nonresponse bias

A nonresponse bias occurs when a portion of chosen participants decide not to complete or participate in the study. This is a common issue in survey-based research (especially online surveys).

In survey-based research, the issue of response versus nonresponse rates can impact the quality of the information collected. Every nonresponse is a missed opportunity to get a better understanding of the chosen population, whether participants choose not to reply based on subject apathy, shame, guilt, or a lack of skills or resources.

To combat this bias, improve response rates using multiple different survey styles. These might include in-person interviews, mailed paper surveys, and virtual options. However, note that these efforts will never completely remove nonresponse bias from your study.

Cognitive bias

Cognitive biases result from repeated errors in thinking or memory caused by misinterpreting information, oversimplifying a situation, or making inaccurate mental shortcuts. They can be tricky to identify and account for, as everyone lives with invisible cognitive biases that govern how they understand and interact with their surrounding environment.

Anchoring bias

When given a list of information, humans have a tendency to overemphasize (or anchor onto) the first thing mentioned.

For example, if you ask people to remember a grocery list of items that starts with apples, bananas, yogurt, and bread, people are most likely to remember apples over any of the other ingredients. This is because apples were mentioned first, despite not being any more important than the other items listed.

This habit conflates the importance and significance of this one piece of information, which can impact how you respond to or feel about the other equally important concepts being mentioned.

Halo effect

The halo effect explains the tendency for people to form opinions or assumptions about other people based on one specific characteristic. Most commonly seen in studies about physical appearance and attractiveness, the halo effect can cause either a positive or negative response depending on how the defined trait is perceived.

Framing effect

Framing effect bias refers to how you perceive information based on how it’s presented to you. 

To demonstrate this, decide which of the following desserts sounds more delicious.

“Made with 95% natural ingredients!”

“Contains only 5% non-natural ingredients!”

Both of these claims say the same thing, but most people have a framing effect bias toward the first claim as it’s positive and more impactful.

This type of bias can significantly impact how people perceive or react to data and information.

The misinformation effect

The misinformation effect refers to the brain’s tendency to alter or misremember past experiences when it has since been fed inaccurate information. This type of bias can significantly impact how a person feels about, remembers, or trusts the authority of their previous experiences.

Confirmation bias

Confirmation bias occurs when someone unconsciously prefers or favors information that confirms or validates their beliefs and ideas.

In some cases, confirmation bias is so strong that people find themselves disregarding information that counters their worldview, resulting in poorer research accuracy and quality.

We all like being proven right (even if we are testing a research hypothesis ), so this is a commonly occurring cognitive bias that needs to be addressed during any scientific study.

Availability heuristic

All humans contextualize and understand the world around them based on their past experiences and memories. Because of this, people tend to have an availability bias toward explanations they have heard before. 

People are more likely to assume or gravitate toward reasoning and ideas that align with past experience. This is known as the availability heuristic . Information and connections that are more available or accessible in your memory might seem more likely than other alternatives. This can impact the validity of research efforts.

• How to avoid bias in your research

Research is a compelling, complex, and essential part of human growth and learning, but collecting the most accurate data possible also poses plenty of challenges.

Should you be using a customer insights hub?

Do you want to discover previous research faster?

Do you share your research findings with others?

Do you analyze research data?

Start for free today, add your research, and get to key insights faster

Editor’s picks

Last updated: 18 April 2023

Last updated: 27 February 2023

Last updated: 6 February 2023

Last updated: 6 October 2023

Last updated: 5 February 2023

Last updated: 16 April 2023

Last updated: 9 March 2023

Last updated: 12 December 2023

Last updated: 11 March 2024

Last updated: 4 July 2024

Last updated: 6 March 2024

Last updated: 5 March 2024

Last updated: 13 May 2024

Latest articles

Related topics, .css-je19u9{-webkit-align-items:flex-end;-webkit-box-align:flex-end;-ms-flex-align:flex-end;align-items:flex-end;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-flex-direction:row;-ms-flex-direction:row;flex-direction:row;-webkit-box-flex-wrap:wrap;-webkit-flex-wrap:wrap;-ms-flex-wrap:wrap;flex-wrap:wrap;-webkit-box-pack:center;-ms-flex-pack:center;-webkit-justify-content:center;justify-content:center;row-gap:0;text-align:center;max-width:671px;}@media (max-width: 1079px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}}@media (max-width: 799px){.css-je19u9{max-width:400px;}.css-je19u9>span{white-space:pre;}} decide what to .css-1kiodld{max-height:56px;display:-webkit-box;display:-webkit-flex;display:-ms-flexbox;display:flex;-webkit-align-items:center;-webkit-box-align:center;-ms-flex-align:center;align-items:center;}@media (max-width: 1079px){.css-1kiodld{display:none;}} build next, decide what to build next.

Users report unexpectedly high data usage, especially during streaming sessions.

Users find it hard to navigate from the home page to relevant playlists in the app.

It would be great to have a sleep timer feature, especially for bedtime listening.

I need better filters to find the songs or artists I’m looking for.

Log in or sign up

Get started for free

• Research Bias: Definition, Types + Examples

Sometimes, in the cause of carrying out a systematic investigation, the researcher may influence the process intentionally or unknowingly. When this happens, it is termed as research bias, and like every other type of bias , it can alter your findings. 

Research bias is one of the dominant reasons for the poor validity of research outcomes. There are no hard and fast rules when it comes to research bias and this simply means that it can happen at any time; if you do not pay adequate attention. 

The spontaneity of research bias means you must take care to understand what it is, be able to identify its feature, and ultimately avoid or reduce its occurrence to the barest minimum. In this article, we will show you how to handle bias in research and how to create unbiased research surveys with Formplus. 

What is Research Bias? 

Research bias happens when the researcher skews the entire process towards a specific research outcome by introducing a systematic error into the sample data. In other words, it is a process where the researcher influences the systematic investigation to arrive at certain outcomes. 

When any form of bias is introduced in research, it takes the investigation off-course and deviates it from its true outcomes. Research bias can also happen when the personal choices and preferences of the researcher have undue influence on the study. 

For instance, let’s say a religious conservative researcher is conducting a study on the effects of alcohol. If the researcher’s conservative beliefs prompt him or her to create a biased survey or have sampling bias , then this is a case of research bias. 

Types of Research Bias 

• Design Bias

Design bias has to do with the structure and methods of your research. It happens when the research design, survey questions, and research method is largely influenced by the preferences of the researcher rather than what works best for the research context. 

In many instances, poor research design or a pack of synergy between the different contributing variables in your systematic investigation can infuse bias into your research process. Research bias also happens when the personal experiences of the researcher influence the choice of the research question and methodology. 

Example of Design Bias  

A researcher who is involved in the manufacturing process of a new drug may design a survey with questions that only emphasize the strengths and value of the drug in question. 

• Selection or Participant Bias

Selection bias happens when the research criteria and study inclusion method automatically exclude some part of your population from the research process. When you choose research participants that exhibit similar characteristics, you’re more likely to arrive at study outcomes that are uni-dimensional. 

Selection bias manifests itself in different ways in the context of research. Inclusion bias is particularly popular in quantitative research and it happens when you select participants to represent your research population while ignoring groups that have alternative experiences. 

Examples of Selection Bias  

• Administering your survey online; thereby limiting it to internet savvy individuals and excluding members of your population without internet access. 
• Collecting data about parenting from a mother’s group. The findings in this type of research will be biased towards mothers while excluding the experiences of the fathers. 
• Publication Bias

Peer-reviewed journals and other published academic papers, in many cases, have some degree of bias. This bias is often imposed on them by the publication criteria for research papers in a particular field. Researchers work their papers to meet these criteria and may ignore information or methods that are not in line with them. 

For example, research papers in quantitative research are more likely to be published if they contain statistical information. On the other hand, Non-publication in qualitative studies is more likely to occur because of a lack of depth when describing study methodologies and findings are not presented. 

• Analysis Bias

This is a type of research bias that creeps in during data processing. Many times, when sorting and analyzing data, the researcher may focus on data samples that confirm his or her thoughts, expectations, or personal experiences; that is, data that favors the research hypothesis. 

This means that the researcher, albeit deliberately or unintentionally, ignores data samples that are inconsistent and suggest research outcomes that differ from the hypothesis. Analysis bias can be far-reaching because it alters the research outcomes significantly and provides a false presentation of what is obtainable in the research environment. 

Example of Analysis Bias  

While researching cannabis, a researcher pays attention to data samples that reinforce the negative effects of cannabis while ignoring data that suggests positives.

• Data Collection Bias

Data collection bias is also known as measurement bias and it happens when the researcher’s personal preferences or beliefs affect how data samples are gathered in the systematic investigation. Data collection bias happens in both q ualitative and quantitative research methods. 

In quantitative research, data collection methods can occur when you use a data-gathering tool or method that is not suitable for your research population. For example, asking individuals who do not have access to the internet, to complete a survey via email or your website. 

In qualitative research, data collection bias happens when you ask bad survey questions during a semi-structured or unstructured interview . Bad survey questions are questions that nudge the interviewee towards implied assumptions. Leading and loaded questions are common examples of bad survey questions. 

• Procedural Bias

Procedural is a type of research bias that happens when the participants in a study are not given enough time to complete surveys. The result is that respondents end up providing half-thoughts and incomplete information that does not provide a true representation of their thoughts. 

There are different ways to subject respondents to procedural respondents. For instance, asking respondents to complete a survey quickly to access an incentive, may force them to fill in false information to simply get things over with. 

Example of Procedural Bias

• Asking employees to complete an employee feedback survey during break time. This timeframe puts respondents under undue pressure and can affect the validity of their responses.  

Bias in Quantitative Research

In quantitative research, the researcher often tries to deny the existence of any bias, by eliminating any type of bias in the systematic investigation. Sampling bias is one of the most types of quantitative research biases and it is concerned with the samples you omit and/or include in your study. 

Types of Quantitative Research Bias

Design bias occurs in quantitative research when the research methods or processes alter the outcomes or findings of a systematic investigation. It can occur when the experiment is being conducted or during the analysis of the data to arrive at a valid conclusion. 

Many times, design biases result from the failure of the researchers to take into account the likely impact of the bias in the research they conduct. This makes the researcher ignore the needs of the research context and instead, prioritize his or her preferences. 

• Sampling Bias

Sampling bias in quantitative research occurs when some members of the research population are systematically excluded from the data sample during research. It also means that some groups in the research population are more likely to be selected in a sample than the others. 

Sampling bias in quantitative research mainly occurs in systematic and random sampling. For example, a study about breast cancer that has just male participants can be said to have sampling bias since it excludes the female group in the research population. 

Bias in Qualitative Research

In qualitative research, the researcher accepts and acknowledges the bias without trying to deny its existence. This makes it easier for the researcher to clearly define the inherent biases and outline its possible implications while trying to minimize its effects. 

Qualitative research defines bias in terms of how valid and reliable the research results are. Bias in qualitative research distorts the research findings and also provides skewed data that defeats the validity and reliability of the systematic investigation. 

Types of Bias in Qualitative Research

• Bias from Moderator

The interviewer or moderator in qualitative data collection can impose several biases on the process. The moderator can introduce bias in the research based on his or her disposition, expression, tone, appearance, idiolect, or relation with the research participants. 

• Biased Questions

The framing and presentation of the questions during the research process can also lead to bias. Biased questions like leading questions , double- barrelled questions, negative questions, and loaded questions , can influence the way respondents provide answers and the authenticity of the responses they present. 

The researcher must identify and eliminate biased questions in qualitative research or rephrase them if they cannot be taken out altogether. Remember that questions form the main basis through which information is collected in research and so, biased questions can lead to invalid research findings. 

• Biased Reporting

Biased reporting is yet another challenge in qualitative research. It happens when the research results are altered due to personal beliefs, customs, attitudes, culture, and errors among many other factors. It also means that the researcher must have analyzed the research data based on his/her beliefs rather than the views perceived by the respondents. 

Bias in Psychology

Cognitive biases can affect research and outcomes in psychology. For example, during a stop-and-search exercise, law enforcement agents may profile certain appearances and physical dispositions as law-abiding. Due to this cognitive bias, individuals who do not exhibit these outlined behaviors can be wrongly profiled as criminals. 

Another example of cognitive bias in psychology can be observed in the classroom. During a class assessment, an invigilator who is looking for physical signs of malpractice might mistakenly classify other behaviors as evidence of malpractice; even though this may not be the case. 

Bias in Market Research

There are 5 common biases in market research – social desirability bias, habituation bias, sponsor bias, confirmation bias, and cultural bias. Let’s find out more about them.

• Social desirability bias happens when respondents fill in incorrect information in market research surveys because they want to be accepted or liked. It happens when respondents are seeking social approval and so, fail to communicate how they truly feel about the statement or question being considered. 

A good example will be market research to find out preferred sexual enhancement methods for adults. Some persons may not want to admit that they use sexual enhancement drugs to avoid criticism or disapproval.

• Habituation bias happens when respondents give similar answers to questions that are structured in the same way. Lack of variety in survey questions can make respondents lose interest, become non-responsive, and simply regurgitate answers.  

For example, multiple-choice questions with the same set of answer options can cause habituation bias in your survey. What you get is that respondents just choose answer options without reflecting on how well their choices represent their thoughts, feelings, and ideas. 

• Sponsor bias takes place when respondents have an idea of the brand or organization that is conducting the research. In this case, their perceptions, opinions, experiences, and feelings about the sponsor may influence how they answer the questions about that particular brand. 

For example, let’s say Formplus is carrying out a study to find out what the market’s preferred form builder is. Respondents may mention the sponsor for the survey (Formplus) as their preferred form builder out of obligation; especially when the survey has some incentives.

• Confirmation bias happens when the overall research process is aimed at confirming the researcher’s perception or hypothesis about the research subjects. In other words, the research process is merely a formality to reinforce the researcher’s existing beliefs. 

Electoral polls often fall into the confirmation bias trap. For example, civil society organizations that are in support of one candidate can create a survey that paints the opposing candidate in a bad light to reinforce beliefs about their preferred candidate. 

• Cultural bias arises from the assumptions we have about other cultures based on the values and standards we have for our own culture . For example, when asked to complete a survey about our culture, we may tilt towards positive answers. In the same vein, we are more likely to provide negative responses in a survey for a culture we do not like. 

How to Identify Bias in a Research

• Pay attention to research design and methods. 
• Observe the data collection process. Does it lean overwhelmingly towards a particular group in the survey population? 
• Look out for bad survey questions like loaded questions and negative questions. 
• Observe the data sample you have to confirm if it is a fair representation of your research population.

How to Avoid Research Bias 

• Gather data from multiple sources: Be sure to collect data samples from the different groups in your research population. 
• Verify your data: Before going ahead with the data analysis, try to check in with other data sources, and confirm if you are on the right track. 
• If possible, ask research participants to help you review your findings: Ask the people who provided the data whether your interpretations seem to be representative of their beliefs. 
• Check for alternative explanations: Try to identify and account for alternative reasons why you may have collected data samples the way you did. 
• Ask other members of your team to review your results: Ask others to review your conclusions. This will help you see things that you missed or identify gaps in your argument that need to be addressed.

How to Create Unbiased Research Surveys with Formplus 

Formplus has different features that would help you create unbiased research surveys. Follow these easy steps to start creating your Formplus research survey today: 

• Go to your Formplus dashboard and click on the “create new form” button. You can access the Formplus dashboard by signing into your Formplus account here. 

• After you click on the “create new form” button, you’d be taken to the form builder. This is where you can add different fields into your form and edit them accordingly. Formplus has over 30 form fields that you can simply drag and drop into your survey including rating fields and scales. 

• After adding form fields and editing them, save your form to access the builder’s customization features. You can tweak the appearance of your form here by changing the form theme and adding preferred background images to it. 

• Copy the form link and share it with respondents. 

Conclusion 

The first step to dealing with research bias is having a clear idea of what it is and also, being able to identify it in any form. In this article, we’ve shared important information about research bias that would help you identify it easily and work on minimizing its effects to the barest minimum. 

Formplus has many features and options that can help you deal with research bias as you create forms and questionnaires for quantitative and qualitative data collection. To take advantage of these, you can sign up for a Formplus account here. 

Connect to Formplus, Get Started Now - It's Free!

• examples of research bias
• types of research bias
• what is research bias
• busayo.longe

You may also like:

Selection Bias in Research: Types, Examples & Impact

In this article, we’ll discuss the effects of selection bias, how it works, its common effects and the best ways to minimize it.

Quota Sampling: Definition, Types, Pros, Cons & Examples

In this article, we’ll explore the concept of quota sampling, its types, and some real-life examples of it can be applied in rsearch

How to do a Meta Analysis: Methodology, Pros & Cons

In this article, we’ll go through the concept of meta-analysis, what it can be used for, and how you can use it to improve how you...

Systematic Errors in Research: Definition, Examples

In this article, we are going to explore the types of systematic error, the causes of this error, how to identify, and how to avoid it.

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 11 December 2020

Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences

• Alec P. Christie   ORCID: orcid.org/0000-0002-8465-8410 1 ,
• David Abecasis   ORCID: orcid.org/0000-0002-9802-8153 2 ,
• Mehdi Adjeroud 3 ,
• Juan C. Alonso   ORCID: orcid.org/0000-0003-0450-7434 4 ,
• Tatsuya Amano   ORCID: orcid.org/0000-0001-6576-3410 5 ,
• Alvaro Anton   ORCID: orcid.org/0000-0003-4108-6122 6 ,
• Barry P. Baldigo   ORCID: orcid.org/0000-0002-9862-9119 7 ,
• Rafael Barrientos   ORCID: orcid.org/0000-0002-1677-3214 8 ,
• Jake E. Bicknell   ORCID: orcid.org/0000-0001-6831-627X 9 ,
• Deborah A. Buhl 10 ,
• Just Cebrian   ORCID: orcid.org/0000-0002-9916-8430 11 ,
• Ricardo S. Ceia   ORCID: orcid.org/0000-0001-7078-0178 12 , 13 ,
• Luciana Cibils-Martina   ORCID: orcid.org/0000-0002-2101-4095 14 , 15 ,
• Sarah Clarke 16 ,
• Joachim Claudet   ORCID: orcid.org/0000-0001-6295-1061 17 ,
• Michael D. Craig 18 , 19 ,
• Dominique Davoult 20 ,
• Annelies De Backer   ORCID: orcid.org/0000-0001-9129-9009 21 ,
• Mary K. Donovan   ORCID: orcid.org/0000-0001-6855-0197 22 , 23 ,
• Tyler D. Eddy 24 , 25 , 26 ,
• Filipe M. França   ORCID: orcid.org/0000-0003-3827-1917 27 ,
• Jonathan P. A. Gardner   ORCID: orcid.org/0000-0002-6943-2413 26 ,
• Bradley P. Harris 28 ,
• Ari Huusko 29 ,
• Ian L. Jones 30 ,
• Brendan P. Kelaher 31 ,
• Janne S. Kotiaho   ORCID: orcid.org/0000-0002-4732-784X 32 , 33 ,
• Adrià López-Baucells   ORCID: orcid.org/0000-0001-8446-0108 34 , 35 , 36 ,
• Heather L. Major   ORCID: orcid.org/0000-0002-7265-1289 37 ,
• Aki Mäki-Petäys 38 , 39 ,
• Beatriz Martín 40 , 41 ,
• Carlos A. Martín 8 ,
• Philip A. Martin 1 , 42 ,
• Daniel Mateos-Molina   ORCID: orcid.org/0000-0002-9383-0593 43 ,
• Robert A. McConnaughey   ORCID: orcid.org/0000-0002-8537-3695 44 ,
• Michele Meroni 45 ,
• Christoph F. J. Meyer   ORCID: orcid.org/0000-0001-9958-8913 34 , 35 , 46 ,
• Kade Mills 47 ,
• Monica Montefalcone 48 ,
• Norbertas Noreika   ORCID: orcid.org/0000-0002-3853-7677 49 , 50 ,
• Carlos Palacín 4 ,
• Anjali Pande 26 , 51 , 52 ,
• C. Roland Pitcher   ORCID: orcid.org/0000-0003-2075-4347 53 ,
• Carlos Ponce 54 ,
• Matt Rinella 55 ,
• Ricardo Rocha   ORCID: orcid.org/0000-0003-2757-7347 34 , 35 , 56 ,
• María C. Ruiz-Delgado 57 ,
• Juan J. Schmitter-Soto   ORCID: orcid.org/0000-0003-4736-8382 58 ,
• Jill A. Shaffer   ORCID: orcid.org/0000-0003-3172-0708 10 ,
• Shailesh Sharma   ORCID: orcid.org/0000-0002-7918-4070 59 ,
• Anna A. Sher   ORCID: orcid.org/0000-0002-6433-9746 60 ,
• Doriane Stagnol 20 ,
• Thomas R. Stanley 61 ,
• Kevin D. E. Stokesbury 62 ,
• Aurora Torres 63 , 64 ,
• Oliver Tully 16 ,
• Teppo Vehanen   ORCID: orcid.org/0000-0003-3441-6787 65 ,
• Corinne Watts 66 ,
• Qingyuan Zhao 67 &
• William J. Sutherland 1 , 42  

Nature Communications volume  11 , Article number:  6377 ( 2020 ) Cite this article

15k Accesses

56 Citations

69 Altmetric

Metrics details

• Environmental impact
• Scientific community
• Social sciences

Building trust in science and evidence-based decision-making depends heavily on the credibility of studies and their findings. Researchers employ many different study designs that vary in their risk of bias to evaluate the true effect of interventions or impacts. Here, we empirically quantify, on a large scale, the prevalence of different study designs and the magnitude of bias in their estimates. Randomised designs and controlled observational designs with pre-intervention sampling were used by just 23% of intervention studies in biodiversity conservation, and 36% of intervention studies in social science. We demonstrate, through pairwise within-study comparisons across 49 environmental datasets, that these types of designs usually give less biased estimates than simpler observational designs. We propose a model-based approach to combine study estimates that may suffer from different levels of study design bias, discuss the implications for evidence synthesis, and how to facilitate the use of more credible study designs.

Similar content being viewed by others

Citizen science in environmental and ecological sciences

Improving quantitative synthesis to achieve generality in ecology

Empirical evidence of widespread exaggeration bias and selective reporting in ecology

Introduction.

The ability of science to reliably guide evidence-based decision-making hinges on the accuracy and credibility of studies and their results 1 , 2 . Well-designed, randomised experiments are widely accepted to yield more credible results than non-randomised, ‘observational studies’ that attempt to approximate and mimic randomised experiments 3 . Randomisation is a key element of study design that is widely used across many disciplines because of its ability to remove confounding biases (through random assignment of the treatment or impact of interest 4 , 5 ). However, ethical, logistical, and economic constraints often prevent the implementation of randomised experiments, whereas non-randomised observational studies have become popular as they take advantage of historical data for new research questions, larger sample sizes, less costly implementation, and more relevant and representative study systems or populations 6 , 7 , 8 , 9 . Observational studies nevertheless face the challenge of accounting for confounding biases without randomisation, which has led to innovations in study design.

We define ‘study design’ as an organised way of collecting data. Importantly, we distinguish between data collection and statistical analysis (as opposed to other authors 10 ) because of the belief that bias introduced by a flawed design is often much more important than bias introduced by statistical analyses. This was emphasised by Light, Singer & Willet 11 (p. 5): “You can’t fix by analysis what you bungled by design…”; and Rubin 3 : “Design trumps analysis.” Nevertheless, the importance of study design has often been overlooked in debates over the inability of researchers to reproduce the original results of published studies (so-called ‘reproducibility crises’ 12 , 13 ) in favour of other issues (e.g., p-hacking 14 and Hypothesizing After Results are Known or ‘HARKing’ 15 ).

To demonstrate the importance of study designs, we can use the following decomposition of estimation error equation 16 :

This demonstrates that even if we improve the quality of modelling and analysis (to reduce modelling bias through a better bias-variance trade-off 17 ) or increase sample size (to reduce statistical noise), we cannot remove the intrinsic bias introduced by the choice of study design (design bias) unless we collect the data in a different way. The importance of study design in determining the levels of bias in study results therefore cannot be overstated.

For the purposes of this study we consider six commonly used study designs; differences and connections can be visualised in Fig.  1 . There are three major components that allow us to define these designs: randomisation, sampling before and after the impact of interest occurs, and the use of a control group.

A hypothetical study set-up is shown where the abundance of birds in three impact and control replicates (e.g., fields represented by blocks in a row) are monitored before and after an impact (e.g., ploughing) that occurs in year zero. Different colours represent each study design and illustrate how replicates are sampled. Approaches for calculating an estimate of the true effect of the impact for each design are also shown, along with synonyms from different disciplines.

Of the non-randomised observational designs, the Before-After Control-Impact (BACI) design uses a control group and samples before and after the impact occurs (i.e., in the ‘before-period’ and the ‘after-period’). Its rationale is to explicitly account for pre-existing differences between the impact group (exposed to the impact) and control group in the before-period, which might otherwise bias the estimate of the impact’s true effect 6 , 18 , 19 .

The BACI design improves upon several other commonly used observational study designs, of which there are two uncontrolled designs: After, and Before-After (BA). An After design monitors an impact group in the after-period, while a BA design compares the state of the impact group between the before- and after-periods. Both designs can be expected to yield poor estimates of the impact’s true effect (large design bias; Equation (1)) because changes in the response variable could have occurred without the impact (e.g., due to natural seasonal changes; Fig.  1 ).

The other observational design is Control-Impact (CI), which compares the impact group and control group in the after-period (Fig.  1 ). This design may suffer from design bias introduced by pre-existing differences between the impact group and control group in the before-period; bias that the BACI design was developed to account for 20 , 21 . These differences have many possible sources, including experimenter bias, logistical and environmental constraints, and various confounding factors (variables that change the propensity of receiving the impact), but can be adjusted for through certain data pre-processing techniques such as matching and stratification 22 .

Among the randomised designs, the most commonly used are counterparts to the observational CI and BACI designs: Randomised Control-Impact (R-CI) and Randomised Before-After Control-Impact (R-BACI) designs. The R-CI design, often termed ‘Randomised Controlled Trials’ (RCTs) in medicine and hailed as the ‘gold standard’ 23 , 24 , removes any pre-impact differences in a stochastic sense, resulting in zero design bias (Equation ( 1 )). Similarly, the R-BACI design should also have zero design bias, and the impact group measurements in the before-period could be used to improve the efficiency of the statistical estimator. No randomised equivalents exist of After or BA designs as they are uncontrolled.

It is important to briefly note that there is debate over two major statistical methods that can be used to analyse data collected using BACI and R-BACI designs, and which is superior at reducing modelling bias 25 (Equation (1)). These statistical methods are: (i) Differences in Differences (DiD) estimator; and (ii) covariance adjustment using the before-period response, which is an extension of Analysis of Covariance (ANCOVA) for generalised linear models — herein termed ‘covariance adjustment’ (Fig.  1 ). These estimators rely on different assumptions to obtain unbiased estimates of the impact’s true effect. The DiD estimator assumes that the control group response accurately represents the impact group response had it not been exposed to the impact (‘parallel trends’ 18 , 26 ) whereas covariance adjustment assumes there are no unmeasured confounders and linear model assumptions hold 6 , 27 .

From both theory and Equation (1), with similar sample sizes, randomised designs (R-BACI and R-CI) are expected to be less biased than controlled, observational designs with sampling in the before-period (BACI), which in turn should be superior to observational designs without sampling in the before-period (CI) or without a control group (BA and After designs 7 , 28 ). Between randomised designs, we might expect that an R-BACI design performs better than a R-CI design because utilising extra data before the impact may improve the efficiency of the statistical estimator by explicitly characterising pre-existing differences between the impact group and control group.

Given the likely differences in bias associated with different study designs, concerns have been raised over the use of poorly designed studies in several scientific disciplines 7 , 29 , 30 , 31 , 32 , 33 , 34 , 35 . Some disciplines, such as the social and medical sciences, commonly undertake direct comparisons of results obtained by randomised and non-randomised designs within a single study 36 , 37 , 38 or between multiple studies (between-study comparisons 39 , 40 , 41 ) to specifically understand the influence of study designs on research findings. However, within-study comparisons are limited in their scope (e.g., a single study 42 , 43 ) and between-study comparisons can be confounded by variability in context or study populations 44 . Overall, we lack quantitative estimates of the prevalence of different study designs and the levels of bias associated with their results.

In this work, we aim to first quantify the prevalence of different study designs in the social and environmental sciences. To fill this knowledge gap, we take advantage of summaries for several thousand biodiversity conservation intervention studies in the Conservation Evidence database 45 ( www.conservationevidence.com ) and social intervention studies in systematic reviews by the Campbell Collaboration ( www.campbellcollaboration.org ). We then quantify the levels of bias in estimates obtained by different study designs (R-BACI, R-CI, BACI, BA, and CI) by applying a hierarchical model to approximately 1000 within-study comparisons across 49 raw environmental datasets from a range of fields. We show that R-BACI, R-CI and BACI designs are poorly represented in studies testing biodiversity conservation and social interventions, and that these types of designs tend to give less biased estimates than simpler observational designs. We propose a model-based approach to combine study estimates that may suffer from different levels of study design bias, discuss the implications for evidence synthesis, and how to facilitate the use of more credible study designs.

Prevalence of study designs

We found that the biodiversity-conservation (conservation evidence) and social-science (Campbell collaboration) literature had similarly high proportions of intervention studies that used CI designs and After designs, but low proportions that used R-BACI, BACI, or BA designs (Fig.  2 ). There were slightly higher proportions of R-CI designs used by intervention studies in social-science systematic reviews than in the biodiversity-conservation literature (Fig.  2 ). The R-BACI, R-CI, and BACI designs made up 23% of intervention studies for biodiversity conservation, and 36% of intervention studies for social science.

Intervention studies from the biodiversity-conservation literature were screened from the Conservation Evidence database ( n =4260 studies) and studies from the social-science literature were screened from 32 Campbell Collaboration systematic reviews ( n =1009 studies – note studies excluded by these reviews based on their study design were still counted). Percentages for the social-science literature were calculated for each systematic review (blue data points) and then averaged across all 32 systematic reviews (blue bars and black vertical lines represent mean and 95% Confidence Intervals, respectively). Percentages for the biodiversity-conservation literature are absolute values (shown as green bars) calculated from the entire Conservation Evidence database (after excluding any reviews). Source data are provided as a Source Data file. BA before-after, CI control-impact, BACI before-after-control-impact, R-BACI randomised BACI, R-CI randomised CI.

Influence of different study designs on study results

In non-randomised datasets, we found that estimates of BACI (with covariance adjustment) and CI designs were very similar, while the point estimates for most other designs often differed substantially in their magnitude and sign. We found similar results in randomised datasets for R-BACI (with covariance adjustment) and R-CI designs. For ~30% of responses, in both non-randomised and randomised datasets, study design estimates differed in their statistical significance (i.e., p < 0.05 versus p  > =0.05), except for estimates of (R-)BACI (with covariance adjustment) and (R-)CI designs (Table  1 ; Fig.  3 ). It was rare for the 95% confidence intervals of different designs’ estimates to not overlap – except when comparing estimates of BA designs to (R-)BACI (with covariance adjustment) and (R-)CI designs (Table  1 ). It was even rarer for estimates of different designs to have significantly different signs (i.e., one estimate with entirely negative confidence intervals versus one with entirely positive confidence intervals; Table  1 , Fig.  3 ). Overall, point estimates often differed greatly in their magnitude and, to a lesser extent, in their sign between study designs, but did not differ as greatly when accounting for the uncertainty around point estimates – except in terms of their statistical significance.

t-statistics were obtained from two-sided t-tests of estimates obtained by each design for different responses in each dataset using Generalised Linear Models (see Methods). For randomised datasets, BACI and CI axis labels refer to R-BACI and R-CI designs (denoted by ‘R-’). DiD Difference in Differences; CA covariance adjustment. Lines at t-statistic values of 1.96 denote boundaries between cells and colours of points indicate differences in direction and statistical significance ( p  < 0.05; grey = same sign and significance, orange = same sign but difference in significance, red = different sign and significance). Numbers refer to the number of responses in each cell. Source data are provided as a Source Data file. BA Before-After, CI Control-Impact, BACI Before-After-Control-Impact.

Levels of bias in estimates of different study designs

We modelled study design bias using a random effect across datasets in a hierarchical Bayesian model; σ is the standard deviation of the bias term, and assuming bias is randomly distributed across datasets and is on average zero, larger values of σ will indicate a greater magnitude of bias (see Methods). We found that, for randomised datasets, estimates of both R-BACI (using covariance adjustment; CA) and R-CI designs were affected by negligible amounts of bias (very small values of σ; Table  2 ). When the R-BACI design used the DiD estimator, it suffered from slightly more bias (slightly larger values of σ), whereas the BA design had very high bias when applied to randomised datasets (very large values of σ; Table  2 ). There was a highly positive correlation between the estimates of R-BACI (using covariance adjustment) and R-CI designs (Ω[R-BACI CA, R-CI] was close to 1; Table  2 ). Estimates of R-BACI using the DiD estimator were also positively correlated with estimates of R-BACI using covariance adjustment and R-CI designs (moderate positive mean values of Ω[R-BACI CA, R-BACI DiD] and Ω[R-BACI DiD, R-CI]; Table  2 ).

For non-randomised datasets, controlled designs (BACI and CI) were substantially less biased (far smaller values of σ) than the uncontrolled BA design (Table  2 ). A BACI design using the DiD estimator was slightly less biased than the BACI design using covariance adjustment, which was, in turn, slightly less biased than the CI design (Table  2 ).

Standard errors estimated by the hierarchical Bayesian model were reasonably accurate for the randomised datasets (see λ in Methods and Table  2 ), whereas there was some underestimation of standard errors and lack-of-fit for non-randomised datasets.

Our approach provides a principled way to quantify the levels of bias associated with different study designs. We found that randomised study designs (R-BACI and R-CI) and observational BACI designs are poorly represented in the environmental and social sciences; collectively, descriptive case studies (the After design), the uncontrolled, observational BA design, and the controlled, observational CI design made up a substantially greater proportion of intervention studies (Fig.  2 ). And yet R-BACI, R-CI and BACI designs were found to be quantifiably less biased than other observational designs.

As expected the R-CI and R-BACI designs (using a covariance adjustment estimator) performed well; the R-BACI design using a DiD estimator performed slightly less well, probably because the differencing of pre-impact data by this estimator may introduce additional statistical noise compared to covariance adjustment, which controls for these data using a lagged regression variable. Of the observational designs, the BA design performed very poorly (both when analysing randomised and non-randomised data) as expected, being uncontrolled and therefore prone to severe design bias 7 , 28 . The CI design also tended to be more biased than the BACI design (using a DiD estimator) due to pre-existing differences between the impact and control groups. For BACI designs, we recommend that the underlying assumptions of DiD and CA estimators are carefully considered before choosing to apply them to data collected for a specific research question 6 , 27 . Their levels of bias were negligibly different and their known bracketing relationship suggests they will typically give estimates with the same sign, although their tendency to over- or underestimate the true effect will depend on how well the underlying assumptions of each are met (most notably, parallel trends for DiD and no unmeasured confounders for CA; see Introduction) 6 , 27 . Overall, these findings demonstrate the power of large within-study comparisons to directly quantify differences in the levels of bias associated with different designs.

We must acknowledge that the assumptions of our hierarchical model (that the bias for each design (j) is on average zero and normally distributed) cannot be verified without gold standard randomised experiments and that, for observational designs, the model was overdispersed (potentially due to underestimation of statistical error by GLM(M)s or positively correlated design biases). The exact values of our hierarchical model should therefore be treated with appropriate caution, and future research is needed to refine and improve our approach to quantify these biases more precisely. Responses within datasets may also not be independent as multiple species could interact; therefore, the estimates analysed by our hierarchical model are statistically dependent on each other, and although we tried to account for this using a correlation matrix (see Methods, Eq. ( 3 )), this is a limitation of our model. We must also recognise that we collated datasets using non-systematic searches 46 , 47 and therefore our analysis potentially exaggerates the intrinsic biases of observational designs (i.e., our data may disproportionately reflect situations where the BACI design was chosen to account for confounding factors). We nevertheless show that researchers were wise to use the BACI design because it was less biased than CI and BA designs across a wide range of datasets from various environmental systems and locations. Without undertaking costly and time-consuming pre-impact sampling and pilot studies, researchers are also unlikely to know the levels of bias that could affect their results. Finally, we did not consider sample size, but it is likely that researchers might use larger sample sizes for CI and BA designs than BACI designs. This is, however, unlikely to affect our main conclusions because larger sample sizes could increase type I errors (false positive rate) by yielding more precise, but biased estimates of the true effect 28 .

Our analyses provide several empirically supported recommendations for researchers designing future studies to assess an impact of interest. First, using a controlled and/or randomised design (if possible) was shown to strongly reduce the level of bias in study estimates. Second, when observational designs must be used (as randomisation is not feasible or too costly), we urge researchers to choose the BACI design over other observational designs—and when that is not possible, to choose the CI design over the uncontrolled BA design. We acknowledge that limited resources, short funding timescales, and ethical or logistical constraints 48 may force researchers to use the CI design (if randomisation and pre-impact sampling are impossible) or the BA design (if appropriate controls cannot be found 28 ). To facilitate the usage of less biased designs, longer-term investments in research effort and funding are required 43 . Far greater emphasis on study designs in statistical education 49 and better training and collaboration between researchers, practitioners and methodologists, is needed to improve the design of future studies; for example, potentially improving the CI design by pairing or matching the impact group and control group 22 , or improving the BA design using regression discontinuity methods 48 , 50 . Where the choice of study design is limited, researchers must transparently communicate the limitations and uncertainty associated with their results.

Our findings also have wider implications for evidence synthesis, specifically the exclusion of certain observational study designs from syntheses (the ‘rubbish in, rubbish out’ concept 51 , 52 ). We believe that observational designs should be included in systematic reviews and meta-analyses, but that careful adjustments are needed to account for their potential biases. Exclusion of observational studies often results from subjective, checklist-based ‘Risk of Bias’ or quality assessments of studies (e.g., AMSTRAD 2 53 , ROBINS-I 54 , or GRADE 55 ) that are not data-driven and often neglect to identify the actual direction, or quantify the magnitude, of possible bias introduced by observational studies when rating the quality of a review’s recommendations. We also found that there was a small proportion of studies that used randomised designs (R-CI or R-BACI) or observational BACI designs (Fig.  2 ), suggesting that systematic reviews and meta-analyses risk excluding a substantial proportion of the literature and limiting the scope of their recommendations if such exclusion criteria are used 32 , 56 , 57 . This problem is compounded by the fact that, at least in conservation science, studies using randomised or BACI designs are strongly concentrated in Europe, Australasia, and North America 31 . Systematic reviews that rely on these few types of study designs are therefore likely to fail to provide decision makers outside of these regions with locally relevant recommendations that they prefer 58 . The Covid-19 pandemic has highlighted the difficulties in making locally relevant evidence-based decisions using studies conducted in different countries with different demographics and cultures, and on patients of different ages, ethnicities, genetics, and underlying health issues 59 . This problem is also acute for decision-makers working on biodiversity conservation in the tropical regions, where the need for conservation is arguably the greatest (i.e., where most of Earth’s biodiversity exists 60 ) but they either have to rely on very few well-designed studies that are not locally relevant (i.e., have low generalisability), or more studies that are locally relevant but less well-designed 31 , 32 . Either option could lead decision-makers to take ineffective or inefficient decisions. In the long-term, improving the quality and coverage of scientific evidence and evidence syntheses across the world will help solve these issues, but shorter-term solutions to synthesising patchy evidence bases are required.

Our work furthers sorely needed research on how to combine evidence from studies that vary greatly in their design. Our approach is an alternative to conventional meta-analyses which tend to only weight studies by their sample size or the inverse of their variance 61 ; when studies vary greatly in their study design, simply weighting by inverse variance or sample size is unlikely to account for different levels of bias introduced by different study designs (see Equation (1)). For example, a BA study could receive a larger weight if it had lower variance than a BACI study, despite our results suggesting a BA study usually suffers from greater design bias. Our model provides a principled way to weight studies by both their variance and the likely amount of bias introduced by their study design; it is therefore a form of ‘bias-adjusted meta-analysis’ 62 , 63 , 64 , 65 , 66 . However, instead of relying on elicitation of subjective expert opinions on the bias of each study, we provide a data-driven, empirical quantification of study biases – an important step that was called for to improve such meta-analytic approaches 65 , 66 .

Future research is needed to refine our methodology, but our empirically grounded form of bias-adjusted meta-analysis could be implemented as follows: 1.) collate studies for the same true effect, their effect size estimates, standard errors, and the type of study design; 2.) enter these data into our hierarchical model, where effect size estimates share the same intercept (the true causal effect), a random effect term due to design bias (whose variance is estimated by the method we used), and a random effect term for statistical noise (whose variance is estimated by the reported standard error of studies); 3.) fit this model and estimate the shared intercept/true effect. Heuristically, this can be thought of as weighting studies by both their design bias and their sampling variance and could be implemented on a dynamic meta-analysis platform (such as metadataset.com 67 ). This approach has substantial potential to develop evidence synthesis in fields (such as biodiversity conservation 31 , 32 ) with patchy evidence bases, where reliably synthesising findings from studies that vary greatly in their design is a fundamental and unavoidable challenge.

Our study has highlighted an often overlooked aspect of debates over scientific reproducibility: that the credibility of studies is fundamentally determined by study design. Testing the effectiveness of conservation and social interventions is undoubtedly of great importance given the current challenges facing biodiversity and society in general and the serious need for more evidence-based decision-making 1 , 68 . And yet our findings suggest that quantifiably less biased study designs are poorly represented in the environmental and social sciences. Greater methodological training of researchers and funding for intervention studies, as well as stronger collaborations between methodologists and practitioners is needed to facilitate the use of less biased study designs. Better communication and reporting of the uncertainty associated with different study designs is also needed, as well as more meta-research (the study of research itself) to improve standards of study design 69 . Our hierarchical model provides a principled way to combine studies using a variety of study designs that vary greatly in their risk of bias, enabling us to make more efficient use of patchy evidence bases. Ultimately, we hope that researchers and practitioners testing interventions will think carefully about the types of study designs they use, and we encourage the evidence synthesis community to embrace alternative methods for combining evidence from heterogeneous sets of studies to improve our ability to inform evidence-based decision-making in all disciplines.

Quantifying the use of different designs

We compared the use of different study designs in the literature that quantitatively tested interventions between the fields of biodiversity conservation (4,260 studies collated by Conservation Evidence 45 ) and social science (1,009 studies found by 32 systematic reviews produced by the Campbell Collaboration: www.campbellcollaboration.org ).

Conservation Evidence is a database of intervention studies, each of which has quantitatively tested a conservation intervention (e.g., sowing strips of wildflower seeds on farmland to benefit birds), that is continuously being updated through comprehensive, manual searches of conservation journals for a wide range of fields in biodiversity conservation (e.g., amphibian, bird, peatland, and farmland conservation 45 ). To obtain the proportion of studies that used each design from Conservation Evidence, we simply extracted the type of study design from each study in the database in 2019 – the study design was determined using a standardised set of criteria; reviews were not included (Table  3 ). We checked if the designs reported in the database accurately reflected the designs in the original publication and found that for a random subset of 356 studies, 95.1% were accurately described.

Each systematic review produced by the Campbell Collaboration collates and analyses studies that test a specific social intervention; we collated systematic reviews that tested a variety of social interventions across several fields in the social sciences, including education, crime and justice, international development and social welfare (Supplementary Data  1 ). We retrieved systematic reviews produced by the Campbell Collaboration by searching their website ( www.campbellcollaboration.org ) for reviews published between 2013‒2019 (as of 8th September 2019) — we limited the date range as we could not go through every review. As we were interested in the use of study designs in the wider social-science literature, we only considered reviews (32 in total) that contained sufficient information on the number of included and excluded studies that used different study designs. Studies may be excluded from systematic reviews for several reasons, such as their relevance to the scope of the review (e.g., testing a relevant intervention) and their study design. We only considered studies if the sole reason for their exclusion from the systematic review was their study design – i.e., reviews clearly reported that the study was excluded because it used a particular study design, and not because of any other reason, such as its relevance to the review’s research questions. We calculated the proportion of studies that used each design in each systematic review (using the same criteria as for the biodiversity-conservation literature – see Table  3 ) and then averaged these proportions across all systematic reviews.

Within-study comparisons of different study designs

We wanted to make direct within-study comparisons between the estimates obtained by different study designs (e.g., see 38 , 70 , 71 for single within-study comparisons) for many different studies. If a dataset contains data collected using a BACI design, subsets of these data can be used to mimic the use of other study designs (a BA design using only data for the impact group, and a CI design using only data collected after the impact occurred). Similarly, if data were collected using a R-BACI design, subsets of these data can be used to mimic the use of a BA design and a R-CI design. Collecting BACI and R-BACI datasets would therefore allow us to make direct within-study comparisons of the estimates obtained by these designs.

We collated BACI and R-BACI datasets by searching the Web of Science Core Collection 72 which included the following citation indexes: Science Citation Index Expanded (SCI-EXPANDED) 1900-present; Social Sciences Citation Index (SSCI) 1900-present Arts & Humanities Citation Index (A&HCI) 1975-present; Conference Proceedings Citation Index - Science (CPCI-S) 1990-present; Conference Proceedings Citation Index - Social Science & Humanities (CPCI-SSH) 1990-present; Book Citation Index - Science (BKCI-S) 2008-present; Book Citation Index - Social Sciences & Humanities (BKCI-SSH) 2008-present; Emerging Sources Citation Index (ESCI) 2015-present; Current Chemical Reactions (CCR-EXPANDED) 1985-present (Includes Institut National de la Propriete Industrielle structure data back to 1840); Index Chemicus (IC) 1993-present. The following search terms were used: [‘BACI’] OR [‘Before-After Control-Impact’] and the search was conducted on the 18th December 2017. Our search returned 674 results, which we then refined by selecting only ‘Article’ as the document type and using only the following Web of Science Categories: ‘Ecology’, ‘Marine Freshwater Biology’, ‘Biodiversity Conservation’, ‘Fisheries’, ‘Oceanography’, ‘Forestry’, ‘Zoology’, Ornithology’, ‘Biology’, ‘Plant Sciences’, ‘Entomology’, ‘Remote Sensing’, ‘Toxicology’ and ‘Soil Science’. This left 579 results, which we then restricted to articles published since 2002 (15 years prior to search) to give us a realistic opportunity to obtain the raw datasets, thus reducing this number to 542. We were able to access the abstracts of 521 studies and excluded any that did not test the effect of an environmental intervention or threat using an R-BACI or BACI design with response measures related to the abundance (e.g., density, counts, biomass, cover), reproduction (reproductive success) or size (body length, body mass) of animals or plants. Many studies did not test a relevant metric (e.g., they measured species richness), did not use a BACI or R-BACI design, or did not test the effect of an intervention or threat — this left 96 studies for which we contacted all corresponding authors to ask for the raw dataset. We were able to fully access 54 raw datasets, but upon closer inspection we found that three of these datasets either: did not use a BACI design; did not use the metrics we specified; or did not provide sufficient data for our analyses. This left 51 datasets in total that we used in our preliminary analyses (Supplementary Data  2 ).

All the datasets were originally collected to evaluate the effect of an environmental intervention or impact. Most of them contained multiple response variables (e.g., different measures for different species, such as abundance or density for species A, B, and C). Within a dataset, we use the term “response” to refer to the estimation of the true effect of an impact on one response variable. There were 1,968 responses in total across 51 datasets. We then excluded 932 responses (resulting in the exclusion of one dataset) where one or more of the four time-period and treatment subsets (Before Control, Before Impact, After Control, and After Impact data) consisted of entirely zero measurements, or two or more of these subsets had more than 90% zero measurements. We also excluded one further dataset as it was the only one to not contain repeated measurements at sites in both the before- and after-periods. This was necessary to generate reliable standard errors when modelling these data. We modelled the remaining 1,036 responses from across 49 datasets (Supplementary Table  1 ).

We applied each study design to the appropriate components of each dataset using Generalised Linear Models (GLMs 73 , 74 ) because of their generality and ability to implement the statistical estimators of many different study designs. The model structure of GLMs was adjusted for each response in each dataset based on the study design specified, response measure and dataset structure (Supplementary Table  2 ). We quantified the effect of the time period for the BA design (After vs Before the impact) and the effect of the treatment type for the CI and R-CI designs (Impact vs Control) on the response variable (Supplementary Table  2 ). For BACI and R-BACI designs, we implemented two statistical estimators: 1.) a DiD estimator that estimated the true effect using an interaction term between time and treatment type; and 2.) a covariance adjustment estimator that estimated the true effect using a term for the treatment type with a lagged variable (Supplementary Table  2 ).

As there were large numbers of responses, we used general a priori rules to specify models for each response; this may have led to some model misspecification, but was unlikely to have substantially affected our pairwise comparison of estimates obtained by different designs. The error family of each GLM was specified based on the nature of the measure used and preliminary data exploration: count measures (e.g., abundance) = poisson; density measures (e.g., biomass or abundance per unit area) = quasipoisson, as data for these measures tended to be overdispersed; percentage measures (e.g., percentage cover) = quasibinomial; and size measures (e.g., body length) = gaussian.

We treated each year or season in which data were collected as independent observations because the implementation of a seasonal term in models is likely to vary on a case-by-case basis; this will depend on the research questions posed by each study and was not feasible for us to consider given the large number of responses we were modelling. The log link function was used for all models to generate a standardised log response ratio as an estimate of the true effect for each response; a fixed effect coefficient (a variable named treatment status; Supplementary Table  2 ) was used to estimate the log response ratio 61 . If the response had at least ten ‘sites’ (independent sampling units) and two measurements per site on average, we used the random effects of subsample (replicates within a site) nested within site to capture the dependence within a site and subsample (i.e., a Generalised Linear Mixed Model or GLMM 73 , 74 was implemented instead of a GLM); otherwise we fitted a GLM with only the fixed effects (Supplementary Table  2 ).

We fitted all models using R version 3.5.1 75 , and packages lme4 76 and MASS 77 . Code to replicate all analyses is available (see Data and Code Availability). We compared the estimates obtained using each study design (both in terms of point estimates and estimates with associated standard error) by their magnitude and sign.

A model-based quantification of the bias in study design estimates

We used a hierarchical Bayesian model motivated by the decomposition in Equation (1) to quantify the bias in different study design estimates. This model takes the estimated effects of impacts and their standard errors as inputs. Let \(\hat \beta _{ij}\) be the true effect estimator in study \(i\) using design \(j\) and \(\hat \sigma _{ij}\) be its estimated standard error from the corresponding GLM or GLMM. Our hierarchical model assumes:

where β i is the true effect for response \(i\) , \(\gamma _{ij}\) is the bias of design j in response \(i\) , and \(\varepsilon _{ij}\) is the sampling noise of the statistical estimator. Although \(\gamma _{ij}\) technically incorporates both the design bias and any misspecification (modelling) bias due to using GLMs or GLMMs (Equation (1)), we expect the modelling bias to be much smaller than the design bias 3 , 11 . We assume the statistical errors \(\varepsilon _i\) within a response are related to the estimated standard errors through the following joint distribution:

where \({\Omega}\) is the correlation matrix for the different estimators in the same response and λ is a scaling factor to account for possible over/under-estimation of the standard errors.

This model effectively quantifies the bias of design \(j\) using the value of \(\sigma _j\) (larger values = more bias) by accounting for within-response correlations using the correlation matrix \({\Omega}\) and for possible under-estimation of the standard error using \(\lambda\) . We ensured that the prior distributions we used had very large variances so they would have a very small effect on the posterior distribution — accordingly we placed the following disperse priors on the variance parameters:

We fitted the hierarchical Bayesian model in R version 3.5.1 using the Bayesian inference package rstan 78 .

Data availability

All data analysed in the current study are available from Zenodo, https://doi.org/10.5281/zenodo.3560856 .  Source data are provided with this paper.

Code availability

All code used in the current study is available from Zenodo, https://doi.org/10.5281/zenodo.3560856 .

Donnelly, C. A. et al. Four principles to make evidence synthesis more useful for policy. Nature 558 , 361–364 (2018).

Article   ADS   CAS   PubMed   Google Scholar  

McKinnon, M. C., Cheng, S. H., Garside, R., Masuda, Y. J. & Miller, D. C. Sustainability: map the evidence. Nature 528 , 185–187 (2015).

Rubin, D. B. For objective causal inference, design trumps analysis. Ann. Appl. Stat. 2 , 808–840 (2008).

Article   MathSciNet   MATH   Google Scholar  

Peirce, C. S. & Jastrow, J. On small differences in sensation. Mem. Natl Acad. Sci. 3 , 73–83 (1884).

Fisher, R. A. Statistical methods for research workers . (Oliver and Boyd, 1925).

Angrist, J. D. & Pischke, J.-S. Mostly harmless econometrics: an empiricist’s companion . (Princeton University Press, 2008).

de Palma, A. et al . Challenges with inferring how land-use affects terrestrial biodiversity: study design, time, space and synthesis. in Next Generation Biomonitoring: Part 1 163–199 (Elsevier Ltd., 2018).

Sagarin, R. & Pauchard, A. Observational approaches in ecology open new ground in a changing world. Front. Ecol. Environ. 8 , 379–386 (2010).

Article   Google Scholar  

Shadish, W. R., Cook, T. D. & Campbell, D. T. Experimental and quasi-experimental designs for generalized causal inference . (Houghton Mifflin, 2002).

Rosenbaum, P. R. Design of observational studies . vol. 10 (Springer, 2010).

Light, R. J., Singer, J. D. & Willett, J. B. By design: Planning research on higher education. By design: Planning research on higher education . (Harvard University Press, 1990).

Ioannidis, J. P. A. Why most published research findings are false. PLOS Med. 2 , e124 (2005).

Article   PubMed   PubMed Central   Google Scholar  

Open Science Collaboration. Estimating the reproducibility of psychological science. Science 349 , aac4716 (2015).

Article   CAS   Google Scholar  

John, L. K., Loewenstein, G. & Prelec, D. Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23 , 524–532 (2012).

Article   PubMed   Google Scholar  

Kerr, N. L. HARKing: hypothesizing after the results are known. Personal. Soc. Psychol. Rev. 2 , 196–217 (1998).

Zhao, Q., Keele, L. J. & Small, D. S. Comment: will competition-winning methods for causal inference also succeed in practice? Stat. Sci. 34 , 72–76 (2019).

Article   MATH   Google Scholar  

Friedman, J., Hastie, T. & Tibshirani, R. The Elements of Statistical Learning . vol. 1 (Springer series in statistics, 2001).

Underwood, A. J. Beyond BACI: experimental designs for detecting human environmental impacts on temporal variations in natural populations. Mar. Freshw. Res. 42 , 569–587 (1991).

Stewart-Oaten, A. & Bence, J. R. Temporal and spatial variation in environmental impact assessment. Ecol. Monogr. 71 , 305–339 (2001).

Eddy, T. D., Pande, A. & Gardner, J. P. A. Massive differential site-specific and species-specific responses of temperate reef fishes to marine reserve protection. Glob. Ecol. Conserv. 1 , 13–26 (2014).

Sher, A. A. et al. Native species recovery after reduction of an invasive tree by biological control with and without active removal. Ecol. Eng. 111 , 167–175 (2018).

Imbens, G. W. & Rubin, D. B. Causal Inference in Statistics, Social, and Biomedical Sciences . (Cambridge University Press, 2015).

Greenhalgh, T. How to read a paper: the basics of Evidence Based Medicine . (John Wiley & Sons, Ltd, 2019).

Salmond, S. S. Randomized Controlled Trials: Methodological Concepts and Critique. Orthopaedic Nursing 27 , (2008).

Geijzendorffer, I. R. et al. How can global conventions for biodiversity and ecosystem services guide local conservation actions? Curr. Opin. Environ. Sustainability 29 , 145–150 (2017).

Dimick, J. B. & Ryan, A. M. Methods for evaluating changes in health care policy. JAMA 312 , 2401 (2014).

Article   CAS   PubMed   Google Scholar  

Ding, P. & Li, F. A bracketing relationship between difference-in-differences and lagged-dependent-variable adjustment. Political Anal. 27 , 605–615 (2019).

Christie, A. P. et al. Simple study designs in ecology produce inaccurate estimates of biodiversity responses. J. Appl. Ecol. 56 , 2742–2754 (2019).

Watson, M. et al. An analysis of the quality of experimental design and reliability of results in tribology research. Wear 426–427 , 1712–1718 (2019).

Kilkenny, C. et al. Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS ONE 4 , e7824 (2009).

Christie, A. P. et al. The challenge of biased evidence in conservation. Conserv, Biol . 13577, https://doi.org/10.1111/cobi.13577 (2020).

Christie, A. P. et al. Poor availability of context-specific evidence hampers decision-making in conservation. Biol. Conserv. 248 , 108666 (2020).

Moscoe, E., Bor, J. & Bärnighausen, T. Regression discontinuity designs are underutilized in medicine, epidemiology, and public health: a review of current and best practice. J. Clin. Epidemiol. 68 , 132–143 (2015).

Goldenhar, L. M. & Schulte, P. A. Intervention research in occupational health and safety. J. Occup. Med. 36 , 763–778 (1994).

CAS   PubMed   Google Scholar  

Junker, J. et al. A severe lack of evidence limits effective conservation of the World’s primates. BioScience https://doi.org/10.1093/biosci/biaa082 (2020).

Altindag, O., Joyce, T. J. & Reeder, J. A. Can Nonexperimental Methods Provide Unbiased Estimates of a Breastfeeding Intervention? A Within-Study Comparison of Peer Counseling in Oregon. Evaluation Rev. 43 , 152–188 (2019).

Chaplin, D. D. et al. The Internal And External Validity Of The Regression Discontinuity Design: A Meta-Analysis Of 15 Within-Study Comparisons. J. Policy Anal. Manag. 37 , 403–429 (2018).

Cook, T. D., Shadish, W. R. & Wong, V. C. Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. J. Policy Anal. Manag. 27 , 724–750 (2008).

Ioannidis, J. P. A. et al. Comparison of evidence of treatment effects in randomized and nonrandomized studies. J. Am. Med. Assoc. 286 , 821–830 (2001).

dos Santos Ribas, L. G., Pressey, R. L., Loyola, R. & Bini, L. M. A global comparative analysis of impact evaluation methods in estimating the effectiveness of protected areas. Biol. Conserv. 246 , 108595 (2020).

Benson, K. & Hartz, A. J. A Comparison of Observational Studies and Randomized, Controlled Trials. N. Engl. J. Med. 342 , 1878–1886 (2000).

Smokorowski, K. E. et al. Cautions on using the Before-After-Control-Impact design in environmental effects monitoring programs. Facets 2 , 212–232 (2017).

França, F. et al. Do space-for-time assessments underestimate the impacts of logging on tropical biodiversity? An Amazonian case study using dung beetles. J. Appl. Ecol. 53 , 1098–1105 (2016).

Duvendack, M., Hombrados, J. G., Palmer-Jones, R. & Waddington, H. Assessing ‘what works’ in international development: meta-analysis for sophisticated dummies. J. Dev. Effectiveness 4 , 456–471 (2012).

Sutherland, W. J. et al. Building a tool to overcome barriers in research-implementation spaces: The Conservation Evidence database. Biol. Conserv. 238 , 108199 (2019).

Gusenbauer, M. & Haddaway, N. R. Which academic search systems are suitable for systematic reviews or meta-analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res. Synth. Methods 11 , 181–217 (2020).

Konno, K. & Pullin, A. S. Assessing the risk of bias in choice of search sources for environmental meta‐analyses. Res. Synth. Methods 11 , 698–713 (2020).

PubMed   Google Scholar  

Butsic, V., Lewis, D. J., Radeloff, V. C., Baumann, M. & Kuemmerle, T. Quasi-experimental methods enable stronger inferences from observational data in ecology. Basic Appl. Ecol. 19 , 1–10 (2017).

Brownstein, N. C., Louis, T. A., O’Hagan, A. & Pendergast, J. The role of expert judgment in statistical inference and evidence-based decision-making. Am. Statistician 73 , 56–68 (2019).

Article   MathSciNet   Google Scholar  

Hahn, J., Todd, P. & Klaauw, W. Identification and estimation of treatment effects with a regression-discontinuity design. Econometrica 69 , 201–209 (2001).

Slavin, R. E. Best evidence synthesis: an intelligent alternative to meta-analysis. J. Clin. Epidemiol. 48 , 9–18 (1995).

Slavin, R. E. Best-evidence synthesis: an alternative to meta-analytic and traditional reviews. Educ. Researcher 15 , 5–11 (1986).

Shea, B. J. et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ (Online) 358 , 1–8 (2017).

Google Scholar  

Sterne, J. A. C. et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355 , i4919 (2016).

Guyatt, G. et al. GRADE guidelines: 11. Making an overall rating of confidence in effect estimates for a single outcome and for all outcomes. J. Clin. Epidemiol. 66 , 151–157 (2013).

Davies, G. M. & Gray, A. Don’t let spurious accusations of pseudoreplication limit our ability to learn from natural experiments (and other messy kinds of ecological monitoring). Ecol. Evolution 5 , 5295–5304 (2015).

Lortie, C. J., Stewart, G., Rothstein, H. & Lau, J. How to critically read ecological meta-analyses. Res. Synth. Methods 6 , 124–133 (2015).

Gutzat, F. & Dormann, C. F. Exploration of concerns about the evidence-based guideline approach in conservation management: hints from medical practice. Environ. Manag. 66 , 435–449 (2020).

Greenhalgh, T. Will COVID-19 be evidence-based medicine’s nemesis? PLOS Med. 17 , e1003266 (2020).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Barlow, J. et al. The future of hyperdiverse tropical ecosystems. Nature 559 , 517–526 (2018).

Gurevitch, J. & Hedges, L. V. Statistical issues in ecological meta‐analyses. Ecology 80 , 1142–1149 (1999).

Stone, J. C., Glass, K., Munn, Z., Tugwell, P. & Doi, S. A. R. Comparison of bias adjustment methods in meta-analysis suggests that quality effects modeling may have less limitations than other approaches. J. Clin. Epidemiol. 117 , 36–45 (2020).

Rhodes, K. M. et al. Adjusting trial results for biases in meta-analysis: combining data-based evidence on bias with detailed trial assessment. J. R. Stat. Soc.: Ser. A (Stat. Soc.) 183 , 193–209 (2020).

Article   MathSciNet   CAS   Google Scholar  

Efthimiou, O. et al. Combining randomized and non-randomized evidence in network meta-analysis. Stat. Med. 36 , 1210–1226 (2017).

Article   MathSciNet   PubMed   Google Scholar  

Welton, N. J., Ades, A. E., Carlin, J. B., Altman, D. G. & Sterne, J. A. C. Models for potentially biased evidence in meta-analysis using empirically based priors. J. R. Stat. Soc. Ser. A (Stat. Soc.) 172 , 119–136 (2009).

Turner, R. M., Spiegelhalter, D. J., Smith, G. C. S. & Thompson, S. G. Bias modelling in evidence synthesis. J. R. Stat. Soc.: Ser. A (Stat. Soc.) 172 , 21–47 (2009).

Shackelford, G. E. et al. Dynamic meta-analysis: a method of using global evidence for local decision making. bioRxiv 2020.05.18.078840, https://doi.org/10.1101/2020.05.18.078840 (2020).

Sutherland, W. J., Pullin, A. S., Dolman, P. M. & Knight, T. M. The need for evidence-based conservation. Trends Ecol. evolution 19 , 305–308 (2004).

Ioannidis, J. P. A. Meta-research: Why research on research matters. PLOS Biol. 16 , e2005468 (2018).

Article   PubMed   PubMed Central   CAS   Google Scholar  

LaLonde, R. J. Evaluating the econometric evaluations of training programs with experimental data. Am. Econ. Rev. 76 , 604–620 (1986).

Long, Q., Little, R. J. & Lin, X. Causal inference in hybrid intervention trials involving treatment choice. J. Am. Stat. Assoc. 103 , 474–484 (2008).

Article   MathSciNet   CAS   MATH   Google Scholar  

Thomson Reuters. ISI Web of Knowledge. http://www.isiwebofknowledge.com (2019).

Stroup, W. W. Generalized linear mixed models: modern concepts, methods and applications . (CRC press, 2012).

Bolker, B. M. et al. Generalized linear mixed models: a practical guide for ecology and evolution. Trends Ecol. Evolution 24 , 127–135 (2009).

R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing (2019).

Bates, D., Mächler, M., Bolker, B. & Walker, S. Fitting linear mixed-effects models using lme4. J. Stat. Softw. 67 , 1–48 (2015).

Venables, W. N. & Ripley, B. D. Modern Applied Statistics with S . (Springer, 2002).

Stan Development Team. RStan: the R interface to Stan. R package version 2.19.3 (2020).

Download references

Acknowledgements

We are grateful to the following people and organisations for contributing datasets to this analysis: P. Edwards, G.R. Hodgson, H. Welsh, J.V. Vieira, authors of van Deurs et al. 2012, T. M. Grome, M. Kaspersen, H. Jensen, C. Stenberg, T. K. Sørensen, J. Støttrup, T. Warnar, H. Mosegaard, Axel Schwerk, Alberto Velando, Dolores River Restoration Partnership, J.S. Pinilla, A. Page, M. Dasey, D. Maguire, J. Barlow, J. Louzada, Jari Florestal, R.T. Buxton, C.R. Schacter, J. Seoane, M.G. Conners, K. Nickel, G. Marakovich, A. Wright, G. Soprone, CSIRO, A. Elosegi, L. García-Arberas, J. Díez, A. Rallo, Parks and Wildlife Finland, Parc Marin de la Côte Bleue. Author funding sources: T.A. was supported by the Grantham Foundation for the Protection of the Environment, Kenneth Miller Trust and Australian Research Council Future Fellowship (FT180100354); W.J.S. and P.A.M. were supported by Arcadia, MAVA, and The David and Claudia Harding Foundation; A.P.C. was supported by the Natural Environment Research Council via Cambridge Earth System Science NERC DTP (NE/L002507/1); D.A. was funded by Portugal national funds through the FCT – Foundation for Science and Technology, under the Transitional Standard – DL57 / 2016 and through the strategic project UIDB/04326/2020; M.A. acknowledges Koniambo Nickel SAS, and particularly Gregory Marakovich and Andy Wright; J.C.A. was funded through by Dirección General de Investigación Científica, projects PB97-1252, BOS2002-01543, CGL2005-04893/BOS, CGL2008-02567 and Comunidad de Madrid, as well as by contract HENARSA-CSIC 2003469-CSIC19637; A.A. was funded by Spanish Government: MEC (CGL2007-65176); B.P.B. was funded through the U.S. Geological Survey and the New York City Department of Environmental Protection; R.B. was funded by Comunidad de Madrid (2018-T1/AMB-10374); J.A.S. and D.A.B. were funded through the U.S. Geological Survey and NextEra Energy; R.S.C. was funded by the Portuguese Foundation for Science and Technology (FCT) grant SFRH/BD/78813/2011 and strategic project UID/MAR/04292/2013; A.D.B. was funded through the Belgian offshore wind monitoring program (WINMON-BE), financed by the Belgian offshore wind energy sector via RBINS—OD Nature; M.K.D. was funded by the Harold L. Castle Foundation; P.M.E. was funded by the Clackamas County Water Environment Services River Health Stewardship Program and the Portland State University Student Watershed Research Project; T.D.E., J.P.A.G. and A.P. were supported by funding from the New Zealand Department of Conservation (Te Papa Atawhai) and from the Centre for Marine Environmental & Economic Research, Victoria University of Wellington, New Zealand; F.M.F. was funded by CNPq-CAPES grants (PELD site 23 403811/2012-0, PELD-RAS 441659/2016-0, BEX5528/13-5 and 383744/2015-6) and BNP Paribas Foundation (Climate & Biodiversity Initiative, BIOCLIMATE project); B.P.H. was funded by NOAA-NMFS sea scallop research set-aside program awards NA16FM1031, NA06FM1001, NA16FM2416, and NA04NMF4720332; A.L.B. was funded by the Portuguese Foundation for Science and Technology (FCT) grant FCT PD/BD/52597/2014, Bat Conservation International student research fellowship and CNPq grant 160049/2013-0; L.C.M. acknowledges Secretaría de Ciencia y Técnica (UNRC); R.A.M. acknowledges Alaska Fisheries Science Center, NOAA Fisheries, and U.S. Department of Commerce for salary support; C.F.J.M. was funded by the Portuguese Foundation for Science and Technology (FCT) grant SFRH/BD/80488/2011; R.R. was funded by the Portuguese Foundation for Science and Technology (FCT) grant PTDC/BIA-BIC/111184/2009, by Madeira’s Regional Agency for the Development of Research, Technology and Innovation (ARDITI) grant M1420-09-5369-FSE-000002 and by a Bat Conservation International student research fellowship; J.C. and S.S. were funded by the Alabama Department of Conservation and Natural Resources; A.T. was funded by the Spanish Ministry of Education with a Formacion de Profesorado Universitario (FPU) grant AP2008-00577 and Dirección General de Investigación Científica, project CGL2008-02567; C.W. was funded by Strategic Science Investment Funding of the Ministry of Business, Innovation and Employment, New Zealand; J.S.K. acknowledges Boreal Peatland LIFE (LIFE08 NAT/FIN/000596), Parks and Wildlife Finland and Kone Foundation; J.J.S.S. was funded by the Mexican National Council on Science and Technology (CONACYT 242558); N.N. was funded by The Carl Tryggers Foundation; I.L.J. was funded by a Discovery Grant from the Natural Sciences and Engineering Research Council of Canada; D.D. and D.S. were funded by the French National Research Agency via the “Investment for the Future” program IDEALG (ANR-10-BTBR-04) and by the ALGMARBIO project; R.C.P. was funded by CSIRO and whose research was also supported by funds from the Great Barrier Reef Marine Park Authority, the Fisheries Research and Development Corporation, the Australian Fisheries Management Authority, and Queensland Department of Primary Industries (QDPI). Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government. The scientific results and conclusions, as well as any views or opinions expressed herein, are those of the author(s) and do not necessarily reflect those of NOAA or the Department of Commerce.

Author information

Authors and affiliations.

Conservation Science Group, Department of Zoology, University of Cambridge, The David Attenborough Building, Downing Street, Cambridge, CB3 3QZ, UK

Alec P. Christie, Philip A. Martin & William J. Sutherland

Centre of Marine Sciences (CCMar), Universidade do Algarve, Campus de Gambelas, 8005-139, Faro, Portugal

David Abecasis

Institut de Recherche pour le Développement (IRD), UMR 9220 ENTROPIE & Laboratoire d’Excellence CORAIL, Université de Perpignan Via Domitia, 52 avenue Paul Alduy, 66860, Perpignan, France

Mehdi Adjeroud

Museo Nacional de Ciencias Naturales, CSIC, Madrid, Spain

Juan C. Alonso & Carlos Palacín

School of Biological Sciences, University of Queensland, Brisbane, 4072, QLD, Australia

Tatsuya Amano

Education Faculty of Bilbao, University of the Basque Country (UPV/EHU). Sarriena z/g E-48940 Leioa, Basque Country, Spain

Alvaro Anton

U.S. Geological Survey, New York Water Science Center, 425 Jordan Rd., Troy, NY, 12180, USA

Barry P. Baldigo

Universidad Complutense de Madrid, Departamento de Biodiversidad, Ecología y Evolución, Facultad de Ciencias Biológicas, c/ José Antonio Novais, 12, E-28040, Madrid, Spain

Rafael Barrientos & Carlos A. Martín

Durrell Institute of Conservation and Ecology (DICE), School of Anthropology and Conservation, University of Kent, Canterbury, CT2 7NR, UK

Jake E. Bicknell

U.S. Geological Survey, Northern Prairie Wildlife Research Center, Jamestown, ND, 58401, USA

Deborah A. Buhl & Jill A. Shaffer

Northern Gulf Institute, Mississippi State University, 1021 Balch Blvd, John C. Stennis Space Center, Mississippi, 39529, USA

Just Cebrian

MARE – Marine and Environmental Sciences Centre, Dept. Life Sciences, University of Coimbra, Coimbra, Portugal

Ricardo S. Ceia

CFE – Centre for Functional Ecology, Dept. Life Sciences, University of Coimbra, Coimbra, Portugal

Departamento de Ciencias Naturales, Universidad Nacional de Río Cuarto (UNRC), Córdoba, Argentina

Luciana Cibils-Martina

CONICET, Buenos Aires, Argentina

Marine Institute, Rinville, Oranmore, Galway, Ireland

Sarah Clarke & Oliver Tully

National Center for Scientific Research, PSL Université Paris, CRIOBE, USR 3278 CNRS-EPHE-UPVD, Maison des Océans, 195 rue Saint-Jacques, 75005, Paris, France

Joachim Claudet

School of Biological Sciences, University of Western Australia, Nedlands, WA, 6009, Australia

Michael D. Craig

School of Environmental and Conservation Sciences, Murdoch University, Murdoch, WA, 6150, Australia

Sorbonne Université, CNRS, UMR 7144, Station Biologique, F.29680, Roscoff, France

Dominique Davoult & Doriane Stagnol

Flanders Research Institute for Agriculture, Fisheries and Food (ILVO), Ankerstraat 1, 8400, Ostend, Belgium

Annelies De Backer

Marine Science Institute, University of California Santa Barbara, Santa Barbara, CA, 93106, USA

Mary K. Donovan

Hawaii Institute of Marine Biology, University of Hawaii at Manoa, Honolulu, HI, 96822, USA

Baruch Institute for Marine & Coastal Sciences, University of South Carolina, Columbia, SC, USA

Tyler D. Eddy

Centre for Fisheries Ecosystems Research, Fisheries & Marine Institute, Memorial University of Newfoundland, St. John’s, Canada

School of Biological Sciences, Victoria University of Wellington, P O Box 600, Wellington, 6140, New Zealand

Tyler D. Eddy, Jonathan P. A. Gardner & Anjali Pande

Lancaster Environment Centre, Lancaster University, LA1 4YQ, Lancaster, UK

Filipe M. França

Fisheries, Aquatic Science and Technology Laboratory, Alaska Pacific University, 4101 University Dr., Anchorage, AK, 99508, USA

Bradley P. Harris

Natural Resources Institute Finland, Manamansalontie 90, 88300, Paltamo, Finland

Department of Biology, Memorial University, St. John’s, NL, A1B 2R3, Canada

Ian L. Jones

National Marine Science Centre and Marine Ecology Research Centre, Southern Cross University, 2 Bay Drive, Coffs Harbour, 2450, Australia

Brendan P. Kelaher

Department of Biological and Environmental Science, University of Jyväskylä, Jyväskylä, Finland

Janne S. Kotiaho

School of Resource Wisdom, University of Jyväskylä, Jyväskylä, Finland

Centre for Ecology, Evolution and Environmental Changes – cE3c, Faculty of Sciences, University of Lisbon, 1749-016, Lisbon, Portugal

Adrià López-Baucells, Christoph F. J. Meyer & Ricardo Rocha

Biological Dynamics of Forest Fragments Project, National Institute for Amazonian Research and Smithsonian Tropical Research Institute, 69011-970, Manaus, Brazil

Granollers Museum of Natural History, Granollers, Spain

Adrià López-Baucells

Department of Biological Sciences, University of New Brunswick, PO Box 5050, Saint John, NB, E2L 4L5, Canada

Heather L. Major

Voimalohi Oy, Voimatie 23, Voimatie, 91100, Ii, Finland

Aki Mäki-Petäys

Natural Resources Institute Finland, Paavo Havaksen tie 3, 90014 University of Oulu, Oulu, Finland

Fundación Migres CIMA Ctra, Cádiz, Spain

Beatriz Martín

Intergovernmental Oceanographic Commission of UNESCO, Marine Policy and Regional Coordination Section Paris 07, Paris, France

BioRISC, St. Catharine’s College, Cambridge, CB2 1RL, UK

Philip A. Martin & William J. Sutherland

Departamento de Ecología e Hidrología, Universidad de Murcia, Campus de Espinardo, 30100, Murcia, Spain

Daniel Mateos-Molina

RACE Division, Alaska Fisheries Science Center, National Marine Fisheries Service, NOAA, 7600 Sand Point Way NE, Seattle, WA, 98115, USA

Robert A. McConnaughey

European Commission, Joint Research Centre (JRC), Ispra, VA, Italy

Michele Meroni

School of Science, Engineering and Environment, University of Salford, Salford, M5 4WT, UK

Christoph F. J. Meyer

Victorian National Park Association, Carlton, VIC, Australia

Department of Earth, Environment and Life Sciences (DiSTAV), University of Genoa, Corso Europa 26, 16132, Genoa, Italy

Monica Montefalcone

Department of Ecology, Swedish University of Agricultural Sciences, Uppsala, Sweden

Norbertas Noreika

Chair of Plant Health, Institute of Agricultural and Environmental Sciences, Estonian University of Life Sciences, Tartu, Estonia

Biosecurity New Zealand – Tiakitanga Pūtaiao Aotearoa, Ministry for Primary Industries – Manatū Ahu Matua, 66 Ward St, PO Box 40742, Wallaceville, New Zealand

Anjali Pande

National Institute of Water & Atmospheric Research Ltd (NIWA), 301 Evans Bay Parade, Greta Point Wellington, New Zealand

CSIRO Oceans & Atmosphere, Queensland Biosciences Precinct, 306 Carmody Road, ST. LUCIA QLD, 4067, Australia

C. Roland Pitcher

Museo Nacional de Ciencias Naturales, CSIC, José Gutiérrez Abascal 2, E-28006, Madrid, Spain

Carlos Ponce

Fort Keogh Livestock and Range Research Laboratory, 243 Fort Keogh Rd, Miles City, Montana, 59301, USA

Matt Rinella

CIBIO-InBIO, Research Centre in Biodiversity and Genetic Resources, University of Porto, Vairão, Portugal

Ricardo Rocha

Departamento de Sistemas Físicos, Químicos y Naturales, Universidad Pablo de Olavide, ES-41013, Sevilla, Spain

María C. Ruiz-Delgado

El Colegio de la Frontera Sur, A.P. 424, 77000, Chetumal, QR, Mexico

Juan J. Schmitter-Soto

Division of Fish and Wildlife, New York State Department of Environmental Conservation, 625 Broadway, Albany, NY, 12233-4756, USA

Shailesh Sharma

University of Denver Department of Biological Sciences, Denver, CO, USA

Anna A. Sher

U.S. Geological Survey, Fort Collins Science Center, Fort Collins, CO, 80526, USA

Thomas R. Stanley

School for Marine Science and Technology, University of Massachusetts Dartmouth, New Bedford, MA, USA

Kevin D. E. Stokesbury

Georges Lemaître Earth and Climate Research Centre, Earth and Life Institute, Université Catholique de Louvain, 1348, Louvain-la-Neuve, Belgium

Aurora Torres

Center for Systems Integration and Sustainability, Department of Fisheries and Wildlife, 13 Michigan State University, East Lansing, MI, 48823, USA

Natural Resources Institute Finland, Latokartanonkaari 9, 00790, Helsinki, Finland

Teppo Vehanen

Manaaki Whenua – Landcare Research, Private Bag 3127, Hamilton, 3216, New Zealand

Corinne Watts

Statistical Laboratory, Department of Pure Mathematics and Mathematical Statistics, University of Cambridge, Wilberforce Road, Cambridge, CB3 0WB, UK

Qingyuan Zhao

You can also search for this author in PubMed   Google Scholar

Contributions

A.P.C., T.A., P.A.M., Q.Z., and W.J.S. designed the research; A.P.C. wrote the paper; D.A., M.A., J.C.A., A.A., B.P.B, R.B., J.B., D.A.B., J.C., R.S.C., L.C.M., S.C., J.C., M.D.C, D.D., A.D.B., M.K.D., T.D.E., P.M.E., F.M.F., J.P.A.G., B.P.H., A.H., I.L.J., B.P.K., J.S.K., A.L.B., H.L.M., A.M., B.M., C.A.M., D.M., R.A.M, M.M., C.F.J.M.,K.M., M.M., N.N., C.P., A.P., C.R.P., C.P., M.R., R.R., M.C.R., J.J.S.S., J.A.S., S.S., A.A.S., D.S., K.D.E.S., T.R.S., A.T., O.T., T.V., C.W. contributed datasets for analyses. All authors reviewed, edited, and approved the manuscript.

Corresponding author

Correspondence to Alec P. Christie .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Peer review information Nature Communications thanks Casper Albers, Samuel Scheiner, and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer reviewer reports are available.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information, peer review file, description of additional supplementary information, supplementary data 1, supplementary data 2, source data, source data, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Christie, A.P., Abecasis, D., Adjeroud, M. et al. Quantifying and addressing the prevalence and bias of study designs in the environmental and social sciences. Nat Commun 11 , 6377 (2020). https://doi.org/10.1038/s41467-020-20142-y

Download citation

Received : 29 January 2020

Accepted : 13 November 2020

Published : 11 December 2020

DOI : https://doi.org/10.1038/s41467-020-20142-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: Anthropocene newsletter — what matters in anthropocene research, free to your inbox weekly.

The Ultimate Guide to Qualitative Research - Part 1: The Basics

• Introduction and overview
• What is qualitative research?
• What is qualitative data?
• Examples of qualitative data
• Qualitative vs. quantitative research
• Mixed methods
• Qualitative research preparation
• Theoretical perspective
• Theoretical framework
• Literature reviews
• Research question
• Conceptual framework
• Conceptual vs. theoretical framework
• Data collection
• Qualitative research methods
• Focus groups
• Observational research
• Case studies
• Ethnographical research
• Ethical considerations
• Confidentiality and privacy

What is research bias?

Understanding unconscious bias, how to avoid bias in research, bias and subjectivity in research.

• Power dynamics
• Reflexivity

Bias in research

In a purely objective world, research bias would not exist because knowledge would be a fixed and unmovable resource; either one knows about a particular concept or phenomenon, or they don't. However, qualitative research and the social sciences both acknowledge that subjectivity and bias exist in every aspect of the social world, which naturally includes the research process too. This bias is manifest in the many different ways that knowledge is understood, constructed, and negotiated, both in and out of research.

Understanding research bias has profound implications for data collection methods and data analysis , requiring researchers to take particular care of how to account for the insights generated from their data .

Research bias, often unavoidable, is a systematic error that can creep into any stage of the research process , skewing our understanding and interpretation of findings. From data collection to analysis, interpretation , and even publication , bias can distort the truth we seek to capture and communicate in our research.

It’s also important to distinguish between bias and subjectivity, especially when engaging in qualitative research . Most qualitative methodologies are based on epistemological and ontological assumptions that there is no such thing as a fixed or objective world that exists “out there” that can be empirically measured and understood through research. Rather, many qualitative researchers embrace the socially constructed nature of our reality and thus recognize that all data is produced within a particular context by participants with their own perspectives and interpretations. Moreover, the researcher’s own subjective experiences inevitably shape how they make sense of the data. These subjectivities are considered to be strengths, not limitations, of qualitative research approaches, because they open new avenues for knowledge generation. This is also why reflexivity is so important in qualitative research. When we refer to bias in this guide, on the other hand, we are referring to systematic errors that can negatively affect the research process but that can be mitigated through researchers’ careful efforts.

To fully grasp what research bias is, it's essential to understand the dual nature of bias. Bias is not inherently evil. It's simply a tendency, inclination, or prejudice for or against something. In our daily lives, we're subject to countless biases, many of which are unconscious. They help us navigate our world, make quick decisions, and understand complex situations. But when conducting research, these same biases can cause significant issues.

Research bias can affect the validity and credibility of research findings, leading to erroneous conclusions. It can emerge from the researcher's subconscious preferences or the methodological design of the study itself. For instance, if a researcher unconsciously favors a particular outcome of the study, this preference could affect how they interpret the results, leading to a type of bias known as confirmation bias.

Research bias can also arise due to the characteristics of study participants. If the researcher selectively recruits participants who are more likely to produce desired outcomes, this can result in selection bias.

Another form of bias can stem from data collection methods . If a survey question is phrased in a way that encourages a particular response, this can introduce response bias. Moreover, inappropriate survey questions can have a detrimental effect on future research if such studies are seen by the general population as biased toward particular outcomes depending on the preferences of the researcher.

Bias can also occur during data analysis . In qualitative research for instance, the researcher's preconceived notions and expectations can influence how they interpret and code qualitative data, a type of bias known as interpretation bias. It's also important to note that quantitative research is not free of bias either, as sampling bias and measurement bias can threaten the validity of any research findings.

Given these examples, it's clear that research bias is a complex issue that can take many forms and emerge at any stage in the research process. This section will delve deeper into specific types of research bias, provide examples, discuss why it's an issue, and provide strategies for identifying and mitigating bias in research.

What is an example of bias in research?

Bias can appear in numerous ways. One example is confirmation bias, where the researcher has a preconceived explanation for what is going on in their data, and any disconfirming evidence is (unconsciously) ignored. For instance, a researcher conducting a study on daily exercise habits might be inclined to conclude that meditation practices lead to greater engagement in exercise because that researcher has personally experienced these benefits. However, conducting rigorous research entails assessing all the data systematically and verifying one’s conclusions by checking for both supporting and refuting evidence.

What is a common bias in research?

Confirmation bias is one of the most common forms of bias in research. It happens when researchers unconsciously focus on data that supports their ideas while ignoring or undervaluing data that contradicts their ideas. This bias can lead researchers to mistakenly confirm their theories, despite having insufficient or conflicting evidence.

What are the different types of bias?

There are several types of research bias, each presenting unique challenges. Some common types include:

Confirmation bias: As already mentioned, this happens when a researcher focuses on evidence supporting their theory while overlooking contradictory evidence.

Selection bias: This occurs when the researcher's method of choosing participants skews the sample in a particular direction.

Response bias: This happens when participants in a study respond inaccurately or falsely, often due to misleading or poorly worded questions.

Observer bias (or researcher bias): This occurs when the researcher unintentionally influences the results because of their expectations or preferences.

Publication bias: This type of bias arises when studies with positive results are more likely to get published, while studies with negative or null results are often ignored.

Analysis bias: This type of bias occurs when the data is manipulated or analyzed in a way that leads to a particular result, whether intentionally or unintentionally.

What is an example of researcher bias?

Researcher bias, also known as observer bias, can occur when a researcher's expectations or personal beliefs influence the results of a study. For instance, if a researcher believes that a particular therapy is effective, they might unconsciously interpret ambiguous results in a way that supports the efficacy of the therapy, even if the evidence is not strong enough.

Even quantitative research methodologies are not immune from bias from researchers. Market research surveys or clinical trial research, for example, may encounter bias when the researcher chooses a particular population or methodology to achieve a specific research outcome. Questions in customer feedback surveys whose data is employed in quantitative analysis can be structured in such a way as to bias survey respondents toward certain desired answers.

Turn your data into findings with ATLAS.ti

Key insights are at your fingertips with our powerful interface. See how with a free trial.

Identifying and avoiding bias in research

As we will remind you throughout this chapter, bias is not a phenomenon that can be removed altogether, nor should we think of it as something that should be eliminated. In a subjective world involving humans as researchers and research participants , bias is unavoidable and almost necessary for understanding social behavior. The section on reflexivity later in this guide will highlight how different perspectives among researchers and human subjects are addressed in qualitative research. That said, bias in excess can place the credibility of a study's findings into serious question. Scholars who read your research need to know what new knowledge you are generating, how it was generated, and why the knowledge you present should be considered persuasive. With that in mind, let's look at how bias can be identified and, where it interferes with research, minimized.

How do you identify bias in research?

Identifying bias involves a critical examination of your entire research study involving the formulation of the research question and hypothesis , the selection of study participants, the methods for data collection, and the analysis and interpretation of data. Researchers need to assess whether each stage has been influenced by bias that may have skewed the results. Tools such as bias checklists or guidelines, peer review , and reflexivity (reflecting on one's own biases) can be instrumental in identifying bias.

How do you identify research bias?

Identifying research bias often involves careful scrutiny of the research methodology and the researcher's interpretations. Was the sample of participants relevant to the research question ? Were the interview or survey questions leading? Were there any conflicts of interest that could have influenced the results? It also requires an understanding of the different types of bias and how they might manifest in a research context. Does the bias occur in the data collection process or when the researcher is analyzing data?

Research transparency requires a careful accounting of how the study was designed, conducted, and analyzed. In qualitative research involving human subjects, the researcher is responsible for documenting the characteristics of the research population and research context. With respect to research methods, the procedures and instruments used to collect and analyze data are described in as much detail as possible.

While describing study methodologies and research participants in painstaking detail may sound cumbersome, a clear and detailed description of the research design is necessary for good research. Without this level of detail, it is difficult for your research audience to identify whether bias exists, where bias occurs, and to what extent it may threaten the credibility of your findings.

How to recognize bias in a study?

Recognizing bias in a study requires a critical approach. The researcher should question every step of the research process: Was the sample of participants selected with care? Did the data collection methods encourage open and sincere responses? Did personal beliefs or expectations influence the interpretation of the results? External peer reviews can also be helpful in recognizing bias, as others might spot potential issues that the original researcher missed.

The subsequent sections of this chapter will delve into the impacts of research bias and strategies to avoid it. Through these discussions, researchers will be better equipped to handle bias in their work and contribute to building more credible knowledge.

Unconscious biases, also known as implicit biases, are attitudes or stereotypes that influence our understanding, actions, and decisions in an unconscious manner. These biases can inadvertently infiltrate the research process, skewing the results and conclusions. This section aims to delve deeper into understanding unconscious bias, its impact on research, and strategies to mitigate it.

What is unconscious bias?

Unconscious bias refers to prejudices or social stereotypes about certain groups that individuals form outside their conscious awareness. Everyone holds unconscious beliefs about various social and identity groups, and these biases stem from a tendency to organize social worlds into categories.

How does unconscious bias infiltrate research?

Unconscious bias can infiltrate research in several ways. It can affect how researchers formulate their research questions or hypotheses , how they interact with participants, their data collection methods, and how they interpret their data . For instance, a researcher might unknowingly favor participants who share similar characteristics with them, which could lead to biased results.

Implications of unconscious bias

The implications of unconscious research bias are far-reaching. It can compromise the validity of research findings , influence the choice of research topics, and affect peer review processes . Unconscious bias can also lead to a lack of diversity in research, which can severely limit the value and impact of the findings.

Strategies to mitigate unconscious research bias

While it's challenging to completely eliminate unconscious bias, several strategies can help mitigate its impact. These include being aware of potential unconscious biases, practicing reflexivity , seeking diverse perspectives for your study, and engaging in regular bias-checking activities, such as bias training and peer debriefing .

By understanding and acknowledging unconscious bias, researchers can take steps to limit its impact on their work, leading to more robust findings.

Why is researcher bias an issue?

Research bias is a pervasive issue that researchers must diligently consider and address. It can significantly impact the credibility of findings. Here, we break down the ramifications of bias into two key areas.

How bias affects validity

Research validity refers to the accuracy of the study findings, or the coherence between the researcher’s findings and the participants’ actual experiences. When bias sneaks into a study, it can distort findings and move them further away from the realities that were shared by the research participants . For example, if a researcher's personal beliefs influence their interpretation of data , the resulting conclusions may not reflect what the data show or what participants experienced.

The transferability problem

Transferability is the extent to which your study's findings can be applied beyond the specific context or sample studied. Applying knowledge from one context to a different context is how we can progress and make informed decisions. In quantitative research , the generalizability of a study is a key component that shapes the potential impact of the findings. In qualitative research , all data and knowledge that is produced is understood to be embedded within a particular context, so the notion of generalizability takes on a slightly different meaning. Rather than assuming that the study participants are statistically representative of the entire population, qualitative researchers can reflect on which aspects of their research context bear the most weight on their findings and how these findings may be transferable to other contexts that share key similarities.

How does bias affect research?

Research bias, if not identified and mitigated, can significantly impact research outcomes. The ripple effects of research bias extend beyond individual studies, impacting the body of knowledge in a field and influencing policy and practice. Here, we delve into three specific ways bias can affect research.

Distortion of research results

Bias can lead to a distortion of your study's findings. For instance, confirmation bias can cause a researcher to focus on data that supports their interpretation while disregarding data that contradicts it. This can skew the results and create a misleading picture of the phenomenon under study.

Undermining scientific progress

When research is influenced by bias, it not only misrepresents participants’ realities but can also impede scientific progress. Biased studies can lead researchers down the wrong path, resulting in wasted resources and efforts. Moreover, it could contribute to a body of literature that is skewed or inaccurate, misleading future research and theories.

Influencing policy and practice based on flawed findings

Research often informs policy and practice. If the research is biased, it can lead to the creation of policies or practices that are ineffective or even harmful. For example, a study with selection bias might conclude that a certain intervention is effective, leading to its broad implementation. However, suppose the transferability of the study's findings was not carefully considered. In that case, it may be risky to assume that the intervention will work as well in different populations, which could lead to ineffective or inequitable outcomes.

While it's almost impossible to eliminate bias in research entirely, it's crucial to mitigate its impact as much as possible. By employing thoughtful strategies at every stage of research, we can strive towards rigor and transparency , enhancing the quality of our findings. This section will delve into specific strategies for avoiding bias.

How do you know if your research is biased?

Determining whether your research is biased involves a careful review of your research design, data collection , analysis , and interpretation . It might require you to reflect critically on your own biases and expectations and how these might have influenced your research. External peer reviews can also be instrumental in spotting potential bias.

Strategies to mitigate bias

Minimizing bias involves careful planning and execution at all stages of a research study. These strategies could include formulating clear, unbiased research questions , ensuring that your sample meaningfully represents the research problem you are studying, crafting unbiased data collection instruments, and employing systematic data analysis techniques. Transparency and reflexivity throughout the process can also help minimize bias.

Mitigating bias in data collection

To mitigate bias in data collection, ensure your questions are clear, neutral, and not leading. Triangulation, or using multiple methods or data sources, can also help to reduce bias and increase the credibility of your findings.

Mitigating bias in data analysis

During data analysis , maintaining a high level of rigor is crucial. This might involve using systematic coding schemes in qualitative research or appropriate statistical tests in quantitative research . Regularly questioning your interpretations and considering alternative explanations can help reduce bias. Peer debriefing , where you discuss your analysis and interpretations with colleagues, can also be a valuable strategy.

By using these strategies, researchers can significantly reduce the impact of bias on their research, enhancing the quality and credibility of their findings and contributing to a more robust and meaningful body of knowledge.

Impact of cultural bias in research

Cultural bias is the tendency to interpret and judge phenomena by standards inherent to one's own culture. Given the increasingly multicultural and global nature of research, understanding and addressing cultural bias is paramount. This section will explore the concept of cultural bias, its impacts on research, and strategies to mitigate it.

What is cultural bias in research?

Cultural bias refers to the potential for a researcher's cultural background, experiences, and values to influence the research process and findings. This can occur consciously or unconsciously and can lead to misinterpretation of data, unfair representation of cultures, and biased conclusions.

How does cultural bias infiltrate research?

Cultural bias can infiltrate research at various stages. It can affect the framing of research questions , the design of the study, the methods of data collection , and the interpretation of results . For instance, a researcher might unintentionally design a study that does not consider the cultural context of the participants, leading to a biased understanding of the phenomenon being studied.

Implications of cultural bias

The implications of cultural bias are profound. Cultural bias can skew your findings, limit the transferability of results, and contribute to cultural misunderstandings and stereotypes. This can ultimately lead to inaccurate or ethnocentric conclusions, further perpetuating cultural bias and inequities.

As a result, many social science fields like sociology and anthropology have been critiqued for cultural biases in research. Some of the earliest research inquiries in anthropology, for example, have had the potential to reduce entire cultures to simplistic stereotypes when compared to mainstream norms. A contemporary researcher respecting ethical and cultural boundaries, on the other hand, should seek to properly place their understanding of social and cultural practices in sufficient context without inappropriately characterizing them.

Strategies to mitigate cultural bias

Mitigating cultural bias requires a concerted effort throughout the research study. These efforts could include educating oneself about other cultures, being aware of one's own cultural biases, incorporating culturally diverse perspectives into the research process, and being sensitive and respectful of cultural differences. It might also involve including team members with diverse cultural backgrounds or seeking external cultural consultants to challenge assumptions and provide alternative perspectives.

By acknowledging and addressing cultural bias, researchers can contribute to more culturally competent, equitable, and valid research. This not only enriches the scientific body of knowledge but also promotes cultural understanding and respect.

Ready to jumpstart your research with ATLAS.ti?

Conceptualize your research project with our intuitive data analysis interface. Download a free trial today.

Keep in mind that bias is a force to be mitigated, not a phenomenon that can be eliminated altogether, and the subjectivities of each person are what make our world so complex and interesting. As things are continuously changing and adapting, research knowledge is also continuously being updated as we further develop our understanding of the world around us.

Ready to analyze your data with ATLAS.ti?

See how our intuitive software can draw key insights from your data with a free trial today.

• Search Menu
• Sign in through your institution
• Supplements
• Cohort Profiles
• Education Corner
• Author Guidelines
• Submission Site
• Open Access
• About the International Journal of Epidemiology
• About the International Epidemiological Association
• Editorial Team
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Contact the IEA
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

Introduction, when is bias analysis practical and productive, how does one select the biases that ought to be addressed, how does one select a method to model biases, how does one assign values to the parameters of a bias model, how does one present and interpret a bias analysis, conclusions, acknowledgments.

• < Previous

Good practices for quantitative bias analysis

• Article contents
• Figures & tables
• Supplementary Data

Timothy L Lash, Matthew P Fox, Richard F MacLehose, George Maldonado, Lawrence C McCandless, Sander Greenland, Good practices for quantitative bias analysis, International Journal of Epidemiology , Volume 43, Issue 6, December 2014, Pages 1969–1985, https://doi.org/10.1093/ije/dyu149

• Permissions Icon Permissions

Quantitative bias analysis serves several objectives in epidemiological research. First, it provides a quantitative estimate of the direction, magnitude and uncertainty arising from systematic errors. Second, the acts of identifying sources of systematic error, writing down models to quantify them, assigning values to the bias parameters and interpreting the results combat the human tendency towards overconfidence in research results, syntheses and critiques and the inferences that rest upon them. Finally, by suggesting aspects that dominate uncertainty in a particular research result or topic area, bias analysis can guide efficient allocation of sparse research resources.

The fundamental methods of bias analyses have been known for decades, and there have been calls for more widespread use for nearly as long. There was a time when some believed that bias analyses were rarely undertaken because the methods were not widely known and because automated computing tools were not readily available to implement the methods. These shortcomings have been largely resolved. We must, therefore, contemplate other barriers to implementation. One possibility is that practitioners avoid the analyses because they lack confidence in the practice of bias analysis.

The purpose of this paper is therefore to describe what we view as good practices for applying quantitative bias analysis to epidemiological data, directed towards those familiar with the methods. We focus on answering questions often posed to those of us who advocate incorporation of bias analysis methods into teaching and research. These include the following. When is bias analysis practical and productive? How does one select the biases that ought to be addressed? How does one select a method to model biases? How does one assign values to the parameters of a bias model? How does one present and interpret a bias analysis?.

We hope that our guide to good practices for conducting and presenting bias analyses will encourage more widespread use of bias analysis to estimate the potential magnitude and direction of biases, as well as the uncertainty in estimates potentially influenced by the biases.

Quantitative bias analysis provides an estimate of uncertainty arising from systematic errors, combats overconfidence in research results and guides future research.

Methods of bias analysis have been well known for decades and endorsed for widespread use, yet bias analysis is rarely implemented.

One possible barrier to implementation is lack of guidance focused more on practice and less on bias models or methods. The purpose of this paper is to provide this missing guidance, and thereby to encourage more widespread use of bias analysis.

Quantitative bias analysis models nonrandom errors that may distort results of epidemiological research. The primary objective of bias analysis is to estimate the potential magnitude and direction of biases, and to quantify the uncertainty about these biases. Models to quantify the direction and magnitude of biases have been known for decades. 1–10 There have been hundreds of articles on adjustment methods for measured bias sources such as confounders, measurement error (including misclassification) and missing data, resulting in several textbooks dealing with these topics. 11–15 Most textbook methods assume that data are available to allow an analyst to estimate parameters used in an adjustment method, for example by imputation of the missing correct values. 11 , 13 , 16

Only a small proportion of the literature deals with cases in which available data are inadequate to support these methods, although these cases are probably more often encountered in practice. This problem has led to development of methods for sensitivity analysis 17–23 and extensions for simulation of bias effects under scenarios deemed plausible based on background information. 15 , 23–32 There has, however, been only limited guidance on when particular bias analysis methods are helpful and on what constitutes good practices in conducting such analyses. 15 , 23 , 28 , 31 This lack of guidance may partly explain the relative dearth of applications in published research.

There are many parallels between good practices for epidemiological research and good practices for applying bias analysis to epidemiological data. 15 For example, good research practices and good bias analysis practices both include: (i) development of a protocol to guide the work; (ii) documentation of revisions to the protocol that are made once the work is under way, along with reasons for (Color online) these revisions; (iii) detailed description of the data used; (iv) a complete description of all analytical methods used and their results, along with reasons for emphasizing particular results for presentation; and (v) discussion of underlying assumptions and limitations of the methods used. Good practices in presentation provide (i)–(v) along with (vi), description of possible explanations for the results. If inferences beyond the study are attempted, they should be prudent, circumspect and integrated with prior knowledge on the topic at hand; inferences based on single studies can be especially misleading given that most inferences require careful synthesis of diverse and extensive literature. 33–35

Even if everyone agreed on certain principles, however, both good research practices and good bias analysis practices would require a presumption that researchers, analysts, authors and reviewers have made in good faith an effort to follow these principles. This presumption can never be guaranteed, but can be bolstered by transparent declaration of competing interests, by symmetrical consideration of bias sources and by other evidence of attempts at neutrality. 36

The purpose of this paper is not to review the methods of bias analysis or ethical research practices, however, but rather to describe what we view as good practices for applying quantitative bias analysis to epidemiological data. Thus we will presume that the data to which these methods will be applied have been gathered and analysed according to good research practices and ethical research conduct. Our focus will instead be on answering questions often posed to those of us who advocate incorporation of bias analysis methods into teaching and research. These questions include the following.

Box 1 summarizes our recommendations in reply to these questions. We do not intend to provide absolute or complete rules of conduct or a definitive checklist to evaluate the quality of a bias analysis. Instead, we provide some initial guidelines for answering the above questions, with the goals of easing the task for those interested in applying bias analysis and encouraging others to view bias analysis as a viable and desirable tool in their own work. Another benefit would be for these guidelines to improve the quality of bias analyses. In turn, we hope that these guidelines will themselves be improved by feedback from readers and users. Eventually, such feedback along with wider experience may lead to more detailed and extensive collaborative guidelines, perhaps along the lines of CONSORT, STROBE and other community efforts to improve research conduct and reporting.

Advisable when a report of an association that is not dramatic goes beyond description and alternative explanations for results, and attempts to draw inferences about causality.

Essential when a report makes action or policy recommendations, or has been developed specifically as a synthesis for decision making, and the decisions are sensitive to biases.

Begin with a review of selection and retention of study subjects, data collection methods and opportunities for confounding, selection bias and measurement error.

Create a graphical display of presumed causal relations among variables and their measurements, and present these DAGs to display the underlying assumptions.

Complete simplified bias calculations to prioritize biases likely to have the greatest influence.

Biases that could credibly explain a finding merit more attention than biases that could not.

A realistic model of bias sources is likely to be complex. Balance realistic modelling and practicality, just as with conventional epidemiological data analysis.

Transparency and credibility are essential. Increasing complexity can reduce transparency and hence the credibility of an analysis.Publish code to improve transparency, aid future implementations and identify algorithmic errors.

Assign credible values and distributions to bias parametersthat reflect available data, including internal sub-studies, external validation data and expert judgment.

Choose a range of plausible values to yield a grid of adjustments that can be examined for consistency and to understand the dependence of results on values or their combinations.

Assign distributions to bias parameters, rather than sets of values, for probabilistic bias analysis and Bayesian analysis to describe the central tendency and spread of adjustments.

Include a sensitivity analysis of the bias analysis to evaluate the dependence of the results on underlying assumptions.

Begin with a clear statement of objectives, which should relate directly to some aspect of the conventional methods description.

Link objectives to a bias model, which relates measured variables to the bias analysis result through the bias parameters. Give values or distributions assigned to these bias parameters, the basis for the assignments and reasons for rejecting alternatives.

Provide an example of the calculations completed using the bias model.

Use diagnostics to assure bias modelling results correspond to the intended model.

Use tables and figures to depict the complete set of bias analysis results.

Interpret bias analysis results beginning with a restatement of the underlying assumptions, and concluding with a description of any resulting change in the study inferences.

Bias analysis covers a broad range of methods, from simple sensitivity analyses to in-depth probabilistic analyses requiring considerable labour. Choosing a method depends on judging the point at which the likely benefit of doing further analyses no longer justifies the labour. This question is complicated by the fact that we rarely have more than vague ideas of the cost or benefit of further analyses. The benefit in particular may be largely unknown until we do the analysis. Fortunately, our analysis decisions are subject to challenge and revision as long as the data remain available.

Later sections will outline what we think is needed for a ‘good’ bias analysis, which will provide some sense of cost. As for benefits, a good bias analysis provides an effect estimate that accounts for plausible sources of bias, aiding in scientific inference. Further, it can provide a sense of the uncertainty warranted given the assumptions incorporated into the bias analysis. As with any methodology, however, bias analysis is not foolproof: poor choice of models or parameter values could harm inferences. To aid decisions on whether and how much bias analysis is needed, we have created a rough classification scheme of situations, ranging from those in which bias analysis seems unnecessary to those in which it appears essential.

Cases in which bias analysis is not essential

Bias analysis is not essential when a research report strictly limits itself to description of its motivation, conduct and data, and stops short of discussing causality or other inferences beyond the observations. Although such purely descriptive reports are unusual and are even discouraged by many, they have been recommended as preferable to opposite extremes in which single studies attempt to argue for or against causality without regard to studies of the same topic or other relevant research. 35

Bias analysis may be helpful, but not necessary, when a report stops short of drawing inferences about causality or other targets beyond the observations, and instead offers alternative explanations for observations. This sort of report is among the most cautious seen in the literature, focusing on data limitations and needs for further research but refraining from substantive conclusions.

Bias analysis may be unnecessary when ordinary statistical analyses, encompassing only random error, show the study is incapable of discriminating among the alternative hypotheses under consideration within the broader topic community. Common examples include studies where the precision of the effect estimate is so poor that the confidence interval includes all associations that are taken seriously by the topic community as possible effect sizes. This situation commonly arises when modest point estimates (e.g. relative risks around 2) are reported, the null value is included in the conventional frequentist confidence interval and no one seriously argues that the effect, if any, could be large (e.g. relative risks above 5). Attempts to argue for or against causality in such cases would be ill-advised even if bias were absent, and discussion may be adequately restrained by considering both limits of the interval estimates with equal weight. 37 In the above situations, however, bias analysis becomes necessary if a reader attempts to draw substantive conclusions beyond those of the original study report, such as in public health policy, legal and regulatory settings.

Bias analysis may also be unnecessary when the observed associations are dramatic, consistent across studies and coherent to the point that bias claims appear unreasonable or motivated by obfuscation goals. Classic examples include associations between smoking and lung cancer, occupational exposure to vinyl chloride and angiosarcoma, estrogen replacement therapy and endometrial cancer, and outbreaks from infectious or toxic sources. In these situations, bias analysis may still be helpful to improve accuracy of uncertainty assessment. It may also be helpful for policy makers seeking to incorporate the size of an effect estimate and its total uncertainty into hazard prioritization and regulation. Finally, bias analysis may be useful in this setting to demonstrate the unreasonableness of denialist claims, as did Cornfield et al. 3 in response to claims that the smoking-lung cancer association could be attributed to a genetic factor affecting both tendency to smoke and cancer risk. As a historical note, this paper is often cited as the first sensitivity analysis, although Berkson 1 is an earlier example of quantitative bias analysis.

Cases in which bias analysis is advisable

Bias analysis is advisable when a report of an association that is not dramatic goes beyond description and possible alternative explanations for results, and attempts to draw inferences about causality or other targets beyond the immediate observations. In these cases, the inferences drawn from conventional statistics may not hold up under the scrutiny afforded by bias analysis, especially when conventional statistical analyses make it appear that the study is capable of discriminating among importantly different alternatives or there is any attempt to interpret the study as if it does so. In public health policy, legal and regulatory settings involving hazards, this situation frequently arises when the lower relative-risk confidence limit is above 1 or the upper limit is below 2.

When conventional statistics appear decisive to some in the topic community, discussion needs to be adequately restrained by considering the potential impact of bias. Simple bias-sensitivity analyses will often suffice to adequately demonstrate robustness or sensitivity of inferences to specific biases. The aforementioned Cornfield et al . 3 paper is an example that addressed an extreme and unreasonable bias explanation (complete genetic confounding) for an extreme and consistent association (which was being promoted as calling for policy change). The analysis by Cornfield et al . demonstrated the extremity of the bias explanation relative to what was known at the time (and has since been borne out by genetic and twin studies). It is uncertain whether they would have gone through this exercise had not a highly influential scientist raised this challenge, but the paper established the notion that one could not explain away the association between smoking and lung cancer as confounding alone without invoking associations and effects at least as large as the one in question.

Cases in which bias analysis is arguably essential

Bias analysis becomes essential when a report makes action or policy recommendations, or has been developed specifically as a research synthesis for decision making, and the decisions (as opposed to the statistical estimates) are sensitive to biases considered reasonable by the topic community. As with Cornfield et al ., 3 simple bias-sensitivity analyses might suffice to demonstrate robustness or sensitivity of inferences. Nonetheless, multiple-bias analysis might be necessary in direct policy or decision settings, and that in turn usually requires probabilistic inputs to deal with the large number of bias parameters.

As an example, by the early 2000s, over a dozen studies exhibited relatively consistent but weak (relative-risk estimates dispersed around 1.7) associations of elevated residential electromagnetic fields (EMFs) and childhood leukaemia. Conventional meta-analyses gave relative-risk interval estimates in the range of 1.3 to 2.3 ( P  = 0.0001). 38 , 39 Consequently, there were calls by some groups for costly remediation (e.g. relocation of power lines). Probabilistic bias analysis found that more credible interval estimates could easily include the null (no effect), 28 as well as very large effects that were inconsistent with surveillance data. 40 Thus, bias analysis showed that the evidence provided by the conventional meta-analysis should be downweighted when considering remediation. In settings where immediate policy action is not needed, bias analysis results can provide a rationale for continued collection of better evidence and can even provide a guide for further research. 41

In summary, simple bias analyses seldom strain resources and so are often worthwhile. They are, however, not necessary until research reports contemplate alternative hypotheses and draw inferences. At this point, and certainly once policy decisions are contemplated, bias quantification by simple bias modelling becomes essential and more complex modelling may also be needed.

When a bias analysis is advisable, the next order of business is to decide which sources of bias to examine. Most bias analyses will have to consider the possibility that results are affected by uncontrolled confounding, selection bias and measurement error (including misclassification) because most epidemiological studies are susceptible to these biases. Which biases to account for with quantitative analysis will depend on the goals of the analysis (e.g. full quantification of study error vs bounding the impact of a single source of bias) and which biases, if any, were ruled out by study features (e.g. a study with mortality as outcome may have no concern about outcome misclassification).

After defining a clear causal question, the analyst should describe the bias sources. This description begins with a detailed review of selection and retention of study subjects in comparison with the source population they are meant to represent, data collection methods, and opportunities for confounding, selection bias and measurement error. Although these descriptions provide a sound foundation, they may miss certain types of biases such as bias from conditioning on colliders. 42 Directed acyclic graphs (DAGs) 42–46 can be useful for identifying potential bias sources, hence we recommend, as a preliminary step to guide the analysis, creating a graphical display of presumed causal relations among analysis variables and their measurements. Further, we recommend presenting these DAGs along with the analysis to help display the assumptions underlying the methods used.

In terms of effort, biases that could credibly explain a finding may merit more attention than biases that could not. For example, in a null study of vaccination and autism risk, an analysis that examined misclassification would be critical if the inference is one of no association. Nondifferential misclassification is typically expected to produce bias toward the null, but small departures from nondifferentiality may lead to bias away from the null, 47 and some forms of differential misclassification may lead to bias toward the null. 48 In contrast, in a non-null study of the association between neighbourhood quality and physical function, correlation of errors between measures of neighbourhood quality and measures of physical function may be most important to evaluate before inferring that poor neighbourhood quality causes poor physical function. 49

Finally, there will often be restrictions on what can be done given available software. Missing-data and Bayesian software can sometimes effectively be used 30 , 50 and procedures for Excel, SAS and Stata have been published. 15 , 27 , 29

Potential sources of bias to be considered

Uncontrolled confounding arises from failure to adjust for important confounders that account, in part, for lack of exchangeability between groups. Failure to adjust properly is due to either failure to measure these confounders, or inappropriate use of statistical adjustment, or variable-selection procedures such as stepwise regression. Incomplete adjustment may also arise from use of inaccurately measured or categorized confounders or from misspecification of the functional form of the relationship between the confounder and the outcome variable (e.g. smoker/non-smoker vs full smoking history). Many bias analysis methods assume no effect-measure modification by the unmeasured confounder, although methods to account for effect-measure modification are available. 15 , 28

A mirror problem to uncontrolled confounding is overadjustment bias, which arises from adjustment for inappropriate variables (such as intermediates and other variables affected by exposure). Failure to adjust for well-measured confounders and overadjustment bias can be remedied given the original data by adding and deleting adjustment variables as appropriate, but these data are usually unavailable for subsequent meta-analyses or risk analyses.

Selection bias arises from biased subject sampling, losses to follow-up, subject nonresponse, subject selection after susceptibles have left the pool of subjects and other mechanisms. Selection bias is often a major concern in selection of controls in case-control studies, but can also arise in case-control and cohort studies when loss to follow-up is related to both the exposure and the outcome, when follow-up begins after the onset of exposure 51 , 52 or when there is matching on inappropriate variables (overmatching). Information on the relation of both exposure and outcome to selection is rarely available. Validation studies of selection proportions are difficult to conduct, because the subjects under consideration are not, and may never have been, in the study. Even when such validation studies are done, as when data from a study can be compared with population registries, the results may not easily translate to the source population for the study subjects. Nonetheless, available information can be used to bound the magnitude of bias due to nonrandom subject selection.

Mismeasurement of variables can be expected in almost all studies. Exposure mismeasurement is common in most nonexperimental designs because of the nature of data collection. Self-reports, medical records, laboratory tests etc. can all result in measurement errors. Approximately nondifferential mismeasurement of exposures and covariates with respect to the study outcome may be plausible when measurements are collected before the outcome occurs. Differential mismeasurement may arise, however, when exposure and covariate measurements are influenced by or share influences with the outcome. Classic examples arise in studies that interview subjects about exposure history after the study outcome, as knowledge of the outcome can influence recall of exposures (recall bias). Contrary to common lore, the net bias that results need not be away from the null. 48 When independent of other errors, nondifferential confounder mismeasurement usually leads to bias in the direction of the original confounding. 7 , 53 Regardless of nondifferentiality, mismeasurement of a covariate that is a strong confounder can lead to substantial bias.

Measurement errors in one variable may also be correlated with measurement errors in other variables. 54 , 55 Such correlated or dependent errors should be expected whenever measurements are obtained or constructed using the same instrument or data. For example, a survey of self-perceived physical function and neighbourhood quality may yield an association between them, even if none exists, because respondents who overstate or understate the true quality of their neighbourhood may do the same with regard to the quality of their own physical function. 49 Errors in occupational exposure histories constructed from the same job-exposure matrix will be correlated since they will incorporate the same errors in the job histories. Similarly, errors in nutrient intakes calculated from the same food-nutrient table will be dependent since they will incorporate the same errors in the diet histories. Less extreme but nonetheless important error dependence can arise among questionnaire responses, especially within related items (e.g. long-term recall of life events or habits). Even when errors are nondifferential, the presence of dependent error between the exposure and the outcome variable can create bias away from the null, and may even create the appearance of a strong association when there is no association at all. 54

Which sources of bias to model

Once the sources of bias have been identified, one must prioritize which biases to include in the analysis. We recommend prioritizing biases likely to have the greatest influence on study results. Judging this often requires relatively quick, simplified bias calculations (described in the next section) based on review of the subject literature and expert subject knowledge. Each of the sources of bias described above may be evaluated tentatively using simple bias analyses. Such an approach will often require a fair amount of labour, but is essential to informing the main part of the bias analysis and any conclusions that follow from it.

As an example, if little or no association has been observed, priority might be given to analysing single biases or combinations of biases that are likely to be toward the null (e.g. independent nondifferential misclassification) and thus might explain the observation. In this regard, signed DAGs 56 , 57 can sometimes indicate the direction of bias and thus help to identify explanatory biases. A danger, however, is that by selecting biases to analyse based on expected direction, one will analyse a biased set of biases and thus reach biased conclusions. We thus advise that any bias that may be of substantively important magnitude be included in the final analyses, without regard to its likely direction.

Investigators may think that a source of bias is present, but that the magnitude of the bias is unimportant relative to the other errors present. For example, if the literature indicates that the association between an uncontrolled confounder and the exposure or outcome is small (e.g. as with socioeconomic status and childhood leukaemia), then the amount of uncontrolled bias from this confounder is also likely to be small. 3 , 58 A number of authors give bounds on the magnitude of bias due to uncontrolled confounding based on bounds for the component associations, 17 , 59–61 which allow the analyst to judge whether that bias is important in their application.

Soliciting expert opinion about possible bias sources can be a useful complement to, but no substitute for, the process described above in conjunction with a full literature review. Experts in the field may be aware of sources of bias that are not commonly mentioned in the literature. It is unlikely, however, that one will be able to obtain a random sample of expert opinions, a concern of special importance in controversial topic areas where experts may disagree vehemently.

Balancing computational intensity and sophistication

Quantitative bias analysis encompasses an array of methods ranging from the relatively simple to the very complex ( Table 1 ). Bias analysts consider such factors as computational intensity and the sophistication needed to implement the method when selecting from among the options. All methods require specifying a bias model and its parameters, but the method’s computational intensity is dictated, in part, by how the bias parameters are specified.

Summary of quantitative bias analysis techniques

Reprinted from Lash, Fox and Fink (2009). 15

In simple bias-sensitivity analysis, the user treats the bias parameters as fixed quantities which are then sometimes varied systematically together (multidimensional bias analysis 15 , 23 ). For example, to study bias due to confounding by an unmeasured covariate, the analyst may examine many combinations of the confounder distribution and its relations to exposure and to the outcome. Similarly, to study bias from exposure misclassification, the analyst might explore different pairs of sensitivity and specificity. 15 , 23 These analyses can be computationally straightforward and require no detailed specification of a distribution for the bias parameters. Once the bias model and its initial values have been coded in a spreadsheet, for example, it is usually a small matter to change the values assigned to the bias parameters to generate a multidimensional analysis. However, such analyses do not explicitly incorporate uncertainty about the bias parameters in interval estimates or tests of the target parameter. Whereas an analyst may wish to begin with simple and multidimensional methods, we recommend formal sensitivity analysis in cases where plausible changes in values of bias parameters result in drastic changes in the bias-adjusted estimate, as often occurs in exposure-misclassification problems 6 , 62 or when more complete depictions of uncertainty are indicated. 27

One way to incorporate this uncertainty into statistical results is to use probabilistic bias analysis (PBA). PBA is a generalization of simple bias analysis in which the bias parameters are assigned a joint probability distribution. This distribution is known as a bias-parameter distribution or, in Bayesian terms, a joint prior distribution, and is supposed to represent the analyst’s uncertainty regarding the true value of a bias parameter.

PBA can be implemented in various ways. 15 , 23 , 24 The simplest approach (sometimes called Monte-Carlo sensitivity analysis, or MCSA) is to repeatedly sample bias parameters from their joint distribution and to use the sampled values in the same basic sensitivity formulas as used in simple fixed-value analysis. Unlike simple bias analysis, however, summaries of the adjusted estimates (e.g. histograms) from PBA reflect the uncertainty about the target parameter due to uncertainty about the bias parameters, provided the latter uncertainty is properly captured by the distribution used.

When multiple sources of bias are of concern, effect estimates can be adjusted for each source simultaneously using multiple bias modelling (MBM). In these situations there are usually far too many bias parameters to carry out simple fixed-value bias analysis, and PBA becomes essential. 28 This type of analysis is more realistic since it can incorporate all biases that are of serious concern, but there is little distributed software to do it. It is possible to combine single-bias algorithms to create a multiple bias adjustment, but care is needed in doing so. In particular, the order of adjustment is important. Adjustments need to be modelled in the reverse order from that in which the biases actually occur, 15 , 23 which depends on the study design. Often confounding occurs first in the source population, selection bias second as the researcher selects subjects, and measurement error last as exposure, covariates and outcomes are measured. These biases should be analysed in the reverse order. Exceptions to this order are also common. For example, when a study sample is drawn from a database, and the inclusion criteria are based on measurements in the database, then selection-bias adjustment should precede adjustment for measurement error. If subsequent measurements are made on patients (including interviews), then adjustment for errors in those measurements should precede selection-bias adjustment. It is essential that analysts report and explain the order in which the adjustments were made to allow evaluation by interested parties.

In typical probabilistic multiple-bias analyses, each bias source receives its own distribution. This modelling implicitly assumes that the distributions, and hence the biases, are independent (e.g. that selection probabilities tell us nothing about misclassification probabilities, and vice versa), which may not always be accurate. Dependencies can, however, be introduced directly as prior correlations 63 or indirectly by using hierarchical models. 28 , 32 A more statistically refined approach is to use the bias-parameter distributions as prior distributions in Bayesian posterior computations. 30 , 50 , 64–67 Fully Bayesian bias analysis can be difficult to implement, however, requiring special software packages and special checks for convergence of the fitting algorithm, which may fail more easily than in conventional analyses. Fortunately, MCSA appears to provide a good approximation to a partial-Bayesian analysis in which only the bias parameters are given prior distributions, provided that these distributions do not include values that are in outright conflict with the data being analysed. 23 , 28 , 68 , 69 In particular, if the bias parameters are completely unidentified from the data (there is no data information about them) and the priors used apply only to these parameters, the resulting MCSA procedure can be viewed as a method of generating samples from the posterior distribution. 30 , 70

Although needless complexity should be avoided, there are areas in which too much simplification should also be avoided. When examining misclassification parameters, it is unreasonable to assume that the parameters are independent of one another. For example, when examining exposure misclassification in a case-control study, we should ordinarily expect the sensitivity among the cases and the sensitivity among the controls to be similar, at least if the same instrument is used to collect exposure information in each group. That is, a higher sensitivity in one group will usually imply a higher sensitivity in the other group and this is modelled by specifying a high positive correlation in the joint prior distribution. The same is true for specificity. In fact, under nondifferentiality these correlations will be 1 (although perfect correlation does not by itself imply nondifferentiality). Failure to include correlations among related bias parameters can result in the sampling of unlikely parameter combinations, which in turn could result in adjusted-estimate distributions that are more misleading than the original unadjusted confidence interval for the target parameter.

Balancing realistic modelling against simplicity and ease of presentation

Any realistic model of bias sources is likely to be complex. As with conventional epidemiological data analysis, tradeoffs must be made between realistic modelling and practicality. Simplifying assumptions are always required, and it is important that these assumptions are made explicit. For example, if the decision is made to omit some biases (perhaps because they are not viewed as being overly influential), the omissions and their rationales should be reported.

We encourage researchers using complex models to also examine simpler approximations to these models as a way to both check coding and gain intuition about the more complex model. For instance, multiple bias models can provide realistic estimates of total study error but may obscure the impacts of distinct bias sources. We thus advise researchers implementing a multiple-bias model to examine each source of bias individually, which helps identify adjustments with the greatest impact on results. One can also compare estimates obtained from probabilistic analysis with the estimate obtained when the bias parameters are fixed at the modes, medians or means of their prior distributions. In the event that the results of the simpler and more complex analyses do not align, the author should provide an explanation as to why.

Implications regarding transparency and credibility

Transparency and credibility are integral to any quantitative bias analysis. Unfortunately, increasing model complexity can lead to less transparency and hence, reduce the credibility of an analysis. Researchers should take several steps to increase the transparency of the methods they use. As with all analyses, researchers should avoid using models that they do not fully understand. Giving a full explanation of why the model specification produced the given results can increase transparency. We also encourage authors to make the data and code from their bias analyses publicly available. With the advent of electronic appendices in most major journals, providing bias analysis code as web appendices poses little problem. Published code will aid future researchers who need to implement bias analyses. Further, quantitative bias modelling is complex and public dissemination of code can help to identify and correct algorithmic errors.

Using available resources versus writing a new model

Numerous resources are available to help researchers implement quantitative bias analysis. Many sources we cite contain detailed examples that illustrate the analyses. Several have provided code so that future researchers could implement their analyses as well. 15 , 27 , 68 , 71 When possible, we encourage authors to adopt code that has been previously developed, because it should help to identify and reduce coding errors. Existing resources may be difficult to adapt to new situations, however, particularly for multiple bias models. In that case, researchers have to write their own programs.

After choosing a bias model that is specified by a collection of bias parameters, the next step is to assign values or distributions to the bias parameters. Here one must wrestle with which value assignments are reasonable, based on the subject matter literature and on experience, and what other considerations should be made when assigning values to the bias parameters. Sometimes only summary data from a publication are available, whereas the original authors would have access to record level data.

Sources of information about bias parameters

Internal validation data.

Credible values and distributions assigned to bias parameters should reflect relevant available data. Some studies obtain bias parameter information from an internal second-phase or validation sub-study, in which additional data are collected that allow adjustment for bias or estimation of values or distributions to assign to bias parameters (e.g. measurements of confounders that are not recorded in the full data set, such as full smoking histories, or laboratory measurements that are collected from a subsample to validate self-reported exposure status). 14 , 15 Internal validation may be the best source of data on the bias parameters in the study in which it was conducted, which implies that a substantial proportion of study resources should be expended on validation sub-studies, even if it requires a reduction in total sample size. The results of such studies often do more to improve the yield from the research than expending these resources on a larger sample size or longer follow-up.

Many statistical methods are available for joint analysis of primary data with internal validation data, including missing-data and measurement-error correction methods. 13 , 16 , 72 Nonetheless, these methods assume that the internal validation data are themselves free of bias. This assumption is often unreasonable, and if violated will result in bias in the bias-adjusted estimate. For example, to adjust for bias due to non-response, after initial requests we could ask all original invitees (including non-responders) to answer a brief supplementary questionnaire. Data provided by those initial non-responders who responded to this call-back survey might provide individual-level information about basic confounders like age and sex, and perhaps exposure and disease status, to identify the determinants of non-response. We should expect however that many initial non-responders will also not respond to this survey, and those that do are unlikely to be a random sample of all initial non-responders. Similarly, internal measurement-validation studies are themselves prone to selection bias when they place an additional burden on study participants, such as filling out further questionnaires, keeping diaries or supplying biological specimens. Those who agree to this additional burden are likely to differ from those who refuse, and these differences may relate to the size of the measurement errors characterized by the validation sub-study. The validation data they supply and adjustments based on them may therefore also be subject to unknown degrees of bias. Consequently, although a validation sub-study can supply valuable information, that information may have to be analysed with allowance for sources of bias in the sub-study.

External validation data

External validation data and external adjustment describe the scenario where we obtain bias parameter information from data outside of the study. 15 , 23 Data from external validation studies can supplement internal validation data (which are often sparse) and are often the only direct source of information about bias parameters. Examples include individual-level data from a second study population, or parameter estimates obtained from a systematic review or meta-analysis. For example, to adjust for bias from an unmeasured confounder, we could conduct a review of the literature to identify published estimates of the distribution of the confounder in the population and the associations between the confounder and the exposure and outcome variables.

As described above, internal and external validation data are themselves subject to systematic as well as random errors, and thus provide imperfect estimates of bias parameters. Nonetheless, such data can help set the range and distribution of values to assign those parameters. Uncertainty about the resulting bias parameter estimates can be incorporated into bias adjustments via sensitivity analyses, as described below.

Input from experts

Validation data are often unavailable, forcing reliance on expert opinion and educated guesses to specify the bias parameters and their distributions. Formulating knowledge or beliefs about unknown parameters into a joint probability distribution is called elicitation of the prior distribution. 73 One formal approach is to ask each expert for an interval within which they think the parameter falls and the odds or percentage they would bet on the parameter falling in this interval. From this interval one may specify a member of a convenient parametric family, such as a lognormal or normal distribution. For example, suppose an expert would give a certain odds or probability that a false-positive probability p (p = 1−specificity) falls between 0.05 and 0.20. If we modelled this expert’s bet as arising from a distribution for logit(p) that was symmetrical (thus having a mean equal to its median), the expert’s implied prior median for p would be expit[(logit(0.20) + logit(0.05))/2] = 0.10. Further modelling the expert’s uncertainty as roughly normal on the logit scale, we would deduce that the standard deviation of this normal distribution is (logit(0.20)−logit(0.05))/(2*1.96) = 0.40.

There is little evidence about which methods of constructing priors are more accurate; research on the quality of reasoning under uncertainty in general suggests that direct expert elicitations are unlikely to provide reliably accurate estimates of values or distributions for assignment to bias parameters. 74 , 75 Of great concern is that expert opinions are highly susceptible to bias. Experts are often influenced by their selective knowledge, reading and interpretation of the literature, as well as personal preferences (‘wish bias’). They can also be overconfident and understate the uncertainty about bias that would be warranted by available evidence, 76 which in turn results in overconfidence about the size of effect under study. 15 , 23 Furthermore, experts may seriously misjudge the quality of the literature and the extent to which bias accounts for previous findings. Such misjudgments may be aggravated by expert overconfidence or poor judgment about the reliability or quality of articles (e.g. over-rating their own studies or those that agree with their views, and under-rating those that conflict with what they expect). As a result, we recommend that analysts inspect the literature directly rather than rely solely on expert opinions. In doing so the analyst should bear in mind that, like reviews, judgment may also be distorted by publication bias and by lack of information on study problems in published reports.

Assigning values and distributions to bias parameters

A parsimonious strategy that does not require specifying the bias parameter values or distributions is to use target-adjustment sensitivity analysis. 26 In this approach, one back-calculates from conventional results to find combinations of bias-parameter values that would qualitatively change or explain the conventional statistics (e.g. that would shift an estimated effect measure to the null value or to a doubling of risk). Target-adjustment sensitivity analysis can be easier to implement and understand than bias modelling with best estimates assigned as values for the bias parameters, for it demands only qualitative assumptions about the bias parameters.

Nonetheless, there are several objections to target adjustment. Most obviously, it only examines how the difference between the conventional estimate and the targeted value might be entirely an artefact of bias 26 and thus is of little use if the goal is to estimate plausible ranges for the effect measure. Target-adjustment sensitivity analysis is also difficult to evaluate when there are multiple biases, for then many plausible as well as implausible patterns of bias could explain the difference between estimate and target. Finally, target adjustment incurs a risk of contaminating subsequent analyses, since once one knows what values would change a conventional estimate to a targeted value, that knowledge can bias one’s view of the plausibility (and hence probability) of such parameter combinations. Thus, target adjustment may be useful only when one bias source is to be evaluated and the only question is whether plausible values for the bias parameters might explain the difference between the study’s result and a targeted effect size of particular interest.

Instead of focusing on a value of the target parameter, one may assign one or more values to the bias parameters based on estimates drawn from external validation studies, internal validation studies or the investigator’s experience working in the topic area. This process may be called fixed bias-parameter analysis (FBA). It is crucial to explain the basis for the selected values. Investigators often choose a range of plausible values. The extreme limits of plausibility may also be selected to avoid understating the uncertainty. When the bias model involves more than one bias parameter, this method ultimately yields a grid of adjustments corresponding to combinations of values assigned to the different parameters of the bias model. The resulting adjusted estimates can be examined for consistency and to understand the dependence of results on different values, or combinations of values, assigned to the bias parameters.

Instead of focusing on fixed sets of values, probabilistic bias analysis (PBA) assigns distributions to the bias parameters. The location and spread of the distributions may be determined by the same information used to assign sets of values for simple and multidimensional bias analysis. For example, suppose we wish to restrict the sensitivity of exposure classification to fall between a and c , and b is considered a most likely value (mode). Among other possibilities, one could then assign: (i) a uniform distribution ranging between a and c ; (ii) a triangular distribution with minimum a, maximum c and mode b ; (iii) a trapezoidal distribution with minimum a, maximum c and lower and upper modes equidistant from b ; (iv) a distribution that is normal on the logit scale, translated to fall between a and c , with mode b ; or (v) a beta distribution, again translated to fall between a and c , with mode b .

A simplicity advantage of the uniform and triangular distributions is that they are determined completely by the specified range a to c and most likely value b . The uniform distribution is exceptionally unrealistic, however, because it has a sudden drop in probability at its boundaries and makes no distinction within those boundaries; for example, if a  = 0.6, b = 0.9, it states that 0.599 is impossible yet 0.601 is as probable as any other possibility including 0.7 and 0.8. Among more mild criticisms of the triangular and trapezoid distributions is that they are not smooth (although they entail no sudden change in probability), whereas logit-normal and beta-distributions may become bimodal with low-precision parameter settings. Thus, to help avoid implausible distributional features, we recommend that distributions be graphed before use. Nonetheless, form (shape) can be particularly difficult to judge visually and intuitively; for example, normal, logistic, and t-distributions are all unimodal symmetrical and are not strikingly different in appearance, yet switching from a normal to a logistic distribution triples the prior probability that the true parameter is over 3 standard deviations from the mean.

An objection to all range-restricted distributions is that we may have no basis for being completely certain a parameter is within the boundaries a , b unless those are purely logical limits (e.g. 1 is the upper limit for a sensitivity and specificity). This problem can be addressed by extending the range between a and b (e.g. to the logical limits of 0 and 1 for sensitivity and specificity). However, this extension can create another problem: when the data convey some information about parameters in a bias model, some values for those parameters inside the chosen range may conflict with the original data, as manifested by impossible adjusted data such as negative adjusted cell counts. This can easily occur, for example, when adjusting for misclassification using sensitivity and specificity, and the minimum allowed value a is too low or the maximum value b is too high, creating an incompatibility between the observed data and the proposed values of sensitivities and specificities. 15 , 23 , 27 It is important to describe these bias-parameter values and see why they produce impossible data. It is also important that the estimates from such values are not used in subsequent inferential statistics, especially when aggregating estimates into simulation summaries (as in MCSA). If only a small proportion of values result in impossible adjusted results, there may be little harm from simply discarding these values and using summaries based on the remaining bias-parameter values, a strategy that truncates the bias-parameter distribution away from values that produce impossible adjustments. 23 , 27 , 28 , 69

One may avoid impossible adjustments by using the priors in proper Bayesian procedures, or by using a bias model whose parameters are independent of the data. 30 , 70 Nonetheless, encountering impossible adjusted data is often of substantive importance, as it may represent a fundamental disconnect between the priors and the data or data model, and may signal poor prior information, poor data modelling or unrecognized data problems.

Sensitivity analysis of the bias analysis

The values assigned to the location and spread of a given bias-parameter distribution can greatly influence the results of a bias analysis. We thus recommend that a sensitivity analysis of the bias analysis, at least to location and spread, should be included where space permits, for example as supplementary appendices. Increasing the spread of a prior distribution (e.g. the prior variance) will usually increase the spread of the bias-adjusted effect measures, and it can be crucial to assess this increase.

Other potentially important sources of sensitivity in prior distributions, and hence uncertainty about final results, include form (e.g. trapezoidal or beta), and dependencies (e.g . correlations) among parameters. Few attempts have been made to compare bias analysis results when different distribution types are assigned to the bias parameters of a bias model, holding the location and spread (e.g. mean and variance) constant. The one published example we are aware of found little difference from use of different distributions with the same location and spread, 15 but more study is needed of sensitivity of bias analysis to distributional forms.

Prior dependencies among bias parameters can be of special concern because there is rarely any validation data to support choices, and yet typical statistical default values (such as zero correlation between case and control misclassification probabilities) may be contextually nonsensical, as discussed above. 23 , 27 Nonetheless, it may be possible to re-parameterize the bias model so that its parameters are approximately independent; 30 , 77 comparisons between results from an independent-parameter model and the original model can reveal sensitivity of results to parameterization.

Diagnostics

An important element of bias analysis, and especially probabilistic bias analysis, is model diagnostics. If the analyst assigns distributions to the parameters of a bias model, then it is good practice to generate histograms of the values selected from the distributions and used in the analysis, and to plot these histograms against the probability densities of the distributions to assure that the sampling corresponds well enough to the expectation given the density. Among other problems, in some situations (e.g . when combinations of sensitivity and specificity lead to negative adjusted cell counts as described earlier) the histogram of values used in MCSA may not correspond to the assigned probability density. Graphical diagnostics are also essential in a full Bayesian bias analysis because of the risk of poor convergence of the fitting algorithm.

Presentation of probabilistic bias analysis results may focus on the median, 2.5th percentile and 97.5th percentile of the modelling results, but the analyst should examine the entire histogram of adjusted estimates for implausible results and unexpected shapes. If results from some modelling iterations were discarded, for example due to negative cell frequencies in contingency tables, then the frequency of discarded iterations should be presented. If discarded results influenced the selection of values or distributions assigned to the bias parameters, then this influence should be described. Complete diagnostic results may be too detailed for presentation in a publication, but the description of the methods should explain which diagnostics were undertaken and that the model and computing code were found to perform adequately.

Presenting bias analysis methods

Bias analysis methods are unfamiliar to many readers of epidemiological research, so presentations of these methods should be as complete and detailed as reasonably possible. A good presentation of a bias analysis should begin with a clear statement of its objectives, which should relate directly to some aspect of the conventional methods description. That is, the conventional methods section should foreshadow the bias analysis methods. The stated objective should then link to a bias model, such as an equation that links measured variables to the bias analysis result through the non-identifiable bias parameters. The presentation should then give values or distributions assigned to these bias parameters, explain the basis for the assignments in terms of plausibility with respect to background literature and give reasons for rejecting other reasonable alternatives that were explicitly considered. A good methods presentation should also provide an example of the calculations completed using the bias model. For multiple bias analysis, this presentation should be repeated for each bias and the order of analysis should be described and explained.

To illustrate these recommendations, consider a bias analysis to address misclassification of a binary exposure by varying assumed sensitivity and specificity. It should state that the objective of the bias analysis is to evaluate the influence of exposure misclassification. The bias model equations link the measured cell frequencies to the adjusted cell frequencies as a function of the sensitivities and specificities. Values assigned to these parameters might come from internal or external validation data, or probability distributions may be assigned using the methods described above. A 2 × 2 table of the measured frequencies should be linked to a 2 × 2 table of the adjusted frequencies with the bias model equation, where the terms of the model are replaced by the measured frequencies, adjusted frequencies and single values drawn from the assigned distributions (e.g. their ranges and modes). This presentation of the methods allows the reader to trace from the objective, to the bias model, to the information and judgments used to assign values to the bias model and finally to the output that provides one possible answer to the objective, conditional on the bias model and assigned values. The example calculation, although perhaps extraneous, ties all of the elements together for the reader.

Presenting bias analysis results

Presentation of bias analysis results might be as simple as presenting the adjusted estimate when only a single simple bias analysis has been completed. It is only when multiple values or multiple combinations of values are assigned to bias parameters, or when distributions are assigned to the bias parameters, that presentation of the results of a bias analysis become more difficult. In general, one should present results of all bias analyses that were completed, not just those with interesting results. The main problem then becomes presentation of complete results in a manner that respects the word and space limitations enforced by most journals. Online supplements provide one common alternative to assure completeness.

Using tables allows the reader to evaluate different bias scenarios created by different assignment values or different combinations of assigned values, which is especially important for presenting the results of multidimensional bias analyses. The disadvantage of using tables is that data reduction is often necessary to deal with complexity, and tables provide no summary of the final uncertainty that arises from uncertainty about the bias parameters. For example, in a multiple-bias analysis there may be several equally important sources of bias. If so, the results need to be presented using high-dimensional tables that are unwieldy, difficult to interpret and which may needlessly highlight implausible parameter combinations. Further, it is cumbersome to incorporate uncertainty from random error into such tables in enough detail so that someone can repeat the bias analysis under different assumptions.

When table complexity overwhelms comprehension, figures usually provide a workable alternative. Three-dimensional column charts with multiple bars along each axis can present the results of even complex multidimensional bias analyses (see Flanders and Khoury, 17 for example). For PBA, one can use tornado diagrams to compare multiple 95% interval estimates that are computed by incorporating uncertainty from each different bias source individually or in subsets (see Stonebraker et al ., 78 for example). Histograms that depict the frequency of adjusted estimates from the iterations of a probabilistic bias analysis can be used to compare bias analysis results with the conventional results, the results of various bias models with one another, and the progression of results across the sequence of models applied in multiple bias analysis. 15 , 23

Space and word count limitations may preclude presentation of all important results in tables or figures. In this case, bias analysis results can be presented in text: as single values yielded by the model and a single set of values assigned to the bias parameters (simple bias analysis); as a range of values yielded by the model and multiple values or multiple combinations of values assigned to the bias parameters (multidimensional bias analysis); or as median and simulation intervals (2.5th and 97.5th percentiles of the adjusted estimates) and medians yielded by a probabilistic bias analysis. Good practice will usually require a more complete presentation of results online or, less preferably, as a posting on the author’s own internet site. In no case should concerns about space limitations or word limits deter the most suitable bias analysis from being undertaken.

Interpreting bias analysis results

One of the advantages of bias analysis is to counteract the human tendency towards overconfidence in research results and inferences based on them. 76 It would be counterproductive, therefore, if the interpretation of a bias analysis exaggerated that overconfidence rather than diminished it, or if it substituted overconfidence in the bias analysis for overconfidence in the conventional analysis. We encourage interpretations of the bias analysis results to begin with a clear restatement of the assumptions underlying it, including the choice of biases to examine, the choice of bias models used in the analysis and the choice of values or distributions assigned to the non-identifiable parameters of the bias model. This restatement should then be summarized in any further presentation or interpretation of the bias analysis results with a preface such as: ‘given the methods used and the assumptions of the analysis’.

The focus of the bias analysis interpretation should then turn to a description of any change in the inferences that might result from the bias analysis. One might write, for example, that the bias analysis suggests that confounding by an unmeasured variable might, or might not, plausibly account for the association observed in the conventional result, conditional on the accuracy of the bias model. Similar statements could be made regarding selection bias, measurement errors or combinations of biases. Recommendations for interpreting simple, multidimensional or and probabilistic bias analyses have been made elsewhere. 15 We recommend against interpreting bias analysis results as proving or otherwise definitively answering whether a bias might, or might not, account for a given conventional result, because of the dependence on the accuracy of the underlying and non-verifiable assumptions.

By identifying the largest sources of uncertainty, sensitivity analyses of the bias analysis results, or the bias analysis results themselves, offer an opportunity for discussion of productive avenues for research improvement, such as where more accurate measurements, validation studies or more confounder measurements are needed. We recognize that general calls for further research are of little utility, but these specific avenues for further research are a direct product of bias analysis, so somewhat different from general statements.

Quantitative bias analysis serves several important goals in epidemiological research. First, it provides a quantitative estimate of the direction, magnitude and uncertainty arising from systematic errors. 15 , 23 , 26 , 28 , 79 Second, the very acts of identifying sources of systematic error, writing down models to quantify them, assigning values to the bias parameters and interpreting the results, combat the human tendency towards overconfidence in research results and the inferences that rest upon them. 76 , 80 , 81 Finally, in an era of diminishing research funds, efficient allocation of sparse resources is becoming even more important. By suggesting aspects that dominate uncertainty in a particular research result or topic area, quantitative bias analysis can provide a guide for productive avenues of additional research; 41 and, as happened with smoking and lung cancer 3 and exogenous estrogens and endometrial cancer, 82 quantitative bias analysis may reinforce more basic results by showing that a particular bias is probably not as important as some claim.

We advocate transparency in description of the methods by which biases were identified for analysis, models were developed and values were assigned to the model parameters. We also encourage bias analysts to make their data and computer code available for use by others, so that the results can be challenged by modifications to the model or by different choices for the values assigned to the model parameters. When data cannot be made freely available, bias analysts at a minimum should offer to incorporate credible modelling modifications and changes to the values assigned to the model parameters, when these are suggested by other stakeholders, and to report completely the results of these revised analyses. Bias models cannot be verified as empirically correct, nor are values assigned to the model parameters identifiable. It is, therefore, crucial that credible alternatives be given thorough examination.

Bias analysis is not a panacea. It cannot resolve fundamental problems with poor epidemiological research design or reporting, although it can account for uncertainties arising from design limitations. If there is investigator bias that introduces fraud into the data collection or analysis, 36 or incompletely represents the data collection and analysis process, 83 then no analysis can be expected to correct the resulting bias. Because the bias analyses we have discussed are designed for unselected analyses of individual studies, they cannot resolve inferential errors arising from selective reporting of research results, whether this is due to selective reporting of ‘significant’ associations or suppression of undesired associations. 84–86 Methods of publication-bias analysis 84–86 and forensic statistics 87–89 can help to investigate these problems.

We hope to increase the proportion of health research that includes a quantitative estimate of the influence of systematic errors on research results. This quantification has been long advocated. 18 , 90 There was a time when some of us believed that such quantification was rarely undertaken because the methods were not widely known and because automated computing tools were not readily available to implement the methods. These shortcomings have been largely resolved. 15 , 25 , 27 , 29 , 71 We must, therefore, contemplate other barriers to implementation.

One sociological barrier is the lack of demand for quantitative bias analysis by reviewers and editors of peer-reviewed journals. 15 So long as the authors confine themselves to description of their study and resulting data, along with thorough discussion of possible explanations for the results and contrasts to other studies, this may not be a problem. But many reports still attempt to extend their own results into general inferences about causes, effects and their policy implications, often overweighting their results relative to other relevant research. In such cases, reviewers and editors are often too willing to excuse study imperfections, if they are confessed in the discussion section, 26 providing little motivation for researchers to use quantitative bias analysis. With rare exceptions, 91 such analyses will only expand the uncertainty interval and call into question the validity of the inferences; that is, after all, a major point of quantitative bias analysis. Researchers have little motivation, aside from scientific integrity, to call their own inferences into question in this way, so the demand must come from the gatekeepers to publication. We hope that our guide to good practices for conducting and presenting bias analyses will make it easier for editors and reviewers to request quantitative bias analysis in lieu of narrative description of study imperfections when investigators insist on drawing broad conclusions about general relations and policy implications.

The authors thank Charles Poole, Paul Gustafson and the reviewers for their valuable comments and suggestions about earlier drafts of the manuscript. Any errors that remain are the sole responsibility of the authors.

Conflict of interest: None declared.

Berkson J . Limitations of the application of fourfold table analysis to hospital data . Biomet Bull 1946 ; 2 : 47 – 53 .

Google Scholar

Bross I . Misclassification in 2x2 tables . Biometrics 1954 ; 10 : 478 – 89 .

Cornfield J Haenszel W Hammond EC Lilienfeld AM Shimkin MB Wydner EL . Smoking and lung cancer: recent evidence and a discussion of some questions . J Natl Cancer Inst 1959 ; 22 : 173 – 203 .

Bross ID . Spurious effects from an extraneous variable . J Chron Dis 1966 ; 19 : 637 – 47 .

Barron B . The effects of misclassification on the estimation of relative risks . Biometrics 1977 ; 33 : 414 – 18 .

Copeland KT Checkoway H McMichael AJ Holbrook RH . Bias due to misclassification in the estimation of relative risk . Am J Epidemiol 1977 ; 105 : 488 – 95 .

Greenland S . The effect of misclassification in the presence of covariates . Am J Epidemiol 1980 ; 112 : 564 – 69 .

Kupper LL Karon JM Kleinbaum DG Morgenstern H Lewis DK . Matching in epidemiological studies: validity and efficiency considerations . Biometrics 1981 ; 37 : 271 – 91 .

Greenland S Kleinbaum DG . Correcting for misclassification in two-way tables and matched-pair studies . Int J Epidemiol 1983 ; 12 : 93 – 97 .

Gail MH Wacholder S Lubin JH . Indirect corrections for confounding under multiplicative and additive risk models . Am J Ind Med 1988 ; 13 : 119 – 30 .

Little RJA Rubin DB . Statistical Analysis with Missing Data . New York : Wiley , 2002 .

Google Preview

Gustafson P . Measurement Error and Misclassification in Statistics and Epidemiology: Impacts and Bayesian Adjustments . Boca Raton, FL : Chapman and Hall/CRC , 2003 .

Carroll RJ Ruppert D Stefanski LA Crainiceanu C . Measurement Error in Nonlinear Models . Boca Raton, FL : Chapman and Hall , 2006 .

Rothman KJ Greenland S Lash TL . Modern Epidemiology . 3rd edn. Philadelphia, PA : Lippincott Williams and Wilkins , 2008 .

Lash TL Fox MP Fink AK . Applying Quantitative Bias Analysis to Epidemiological Data . New York : Springer , 2009 .

Cole SR Chu H Greenland S . Multiple-imputation for measurement-error correction . Int J Epidemiol 2006 ; 35 : 1074 – 81 .

Flanders WD Khoury MJ . Indirect assessment of confounding: graphic description and limits on effect of adjusting for covariates . Epidemiology 1990 ; 1 : 239 – 46 .

Greenland S . Basic methods for sensitivity analysis of biases . Int J Epidemiol 1996 ; 25 : 1107 – 16 .

Scharfstein D Rotnitzky A Robins JM . Adjusting for nonignorable drop-out using semiparametric nonresponse models . J Am Stat Assoc 1999 ; 94 : 1096 – 120 .

Robins JM Rotnitzky A Scharfstein D . Sensitivity analysis for selection bias and unmeasured confounding in missing data and causal inference models . In: Halloran E Berry D (eds). Statistical Models in Epidemiology, the Environment, and Clinical Trials . New York : Springer , 1999 .

Rosenbaum PR . Observational Studies . 2nd edn. New York : Springer , 2002 .

Checkoway H Pearce N Kriebel D . Research Methods in Occupational Epidemiology . 2nd edn. New York : Oxford University Press , 2004 .

Greenland S Lash TL . Bias analysis . In: Rothman KJ Greenland S Lash TL (eds). Modern Epidemiology . 3rd edn. Philadelphia, PA : Lippincott Williams and Wilkins ; 2008 .

Eddy DM Hasselblad V Schachter R . Meta-analysis by the Confidence Profile Method . New York : Academic Press , 1992 .

Lash TL Fink AK . Semi-automated sensitivity analysis to assess systematic errors in observational data . Epidemiology 2003 ; 14 : 451 – 58 .

Phillips CV LaPole LM . Quantifying errors without random sampling . BMC Med Res Methodol 2003 ; 3 : 9 .

Fox MP Lash TL Greenland S . A method to automate probabilistic sensitivity analyses of misclassified binary variables . Int J Epidemiol 2005 ; 34 : 1370 – 76 .

Greenland S . Multiple-bias modeling for analysis of observational data . J R Stat Soc Ser A 2005 ; 168 : 267 – 308 .

Orsini N Bellocco R Bottai M Wolk A Greenland S . A tool for deterministic and probabilistic sensitivity analysis of epidemiological studies . Stata J 2008 ; 8 : 29 – 48 .

Greenland S . Bayesian perspectives for epidemiological research: III. Bias analysis via missing-data methods . Int J Epidemiol 2009 ; 38 : 1662 – 73 .

Greenland S . Sensitivty analysis and bias analysis . In: Ahrens W Pigeot I (eds). Handbook of Epidemiology . 2nd edn. New York : Springer , 2013 .

Greenland S . The impact of prior distributions for uncontrolled confounding and response bias: a case study of the relation of wire codes and magnetic fields to childhood leukemia . J Am Stat Assoc 2003 ; 98 : 47 – 54 .

Lanes SF Poole C . ‘Truth in packaging?’ The unwrapping of epidemiological research . J Occup Med 1984 ; 26 : 571 – 74 .

Rothman KJ Poole C . Science and policy making . Am J Public Health 1985 ; 75 : 340 – 41 .

Greenland S Gago-Dominguez M Castelao JE . The value of risk-factor (‘black-box’) epidemiology . Epidemiology 2004 ; 15 : 529 – 35 .

Greenland S . Transparency and disclosure, neutrality and balance: shared values or just shared words? J Epidemiol Community Health 2012 ; 66 : 967 – 70 .

Poole C . Low P-values or narrow confidence intervals: which are more durable? Epidemiology 2001 ; 12 : 291 – 94 .

Ahlbom A Day N Feychting M et al.  . A pooled analysis of magnetic fields and childhood leukaemia . Br J Cancer 2000 ; 83 : 692 – 98 .

Greenland S Sheppard AR Kaune WT Poole C Kelsh MA . A pooled analysis of magnetic fields, wire codes, and childhood leukemia. Childhood Leukemia-EMF Study Group . Epidemiology 2000 ; 11 : 624 – 34 .

Greenland S . Bayesian perspectives for epidemiological research: I. Foundations and basic methods . Int J Epidemiol 2006 ; 35 : 765 – 75 .

Lash TL Ahern TP . Bias analysis to guide new data collection . Int J Biostat 2012 ; 8 : 4679 .

Greenland S Pearl J Robins JM . Causal diagrams for epidemiological research . Epidemiology 1999 ; 10 : 37 – 48 .

Cole SR Hernan MA . Fallibility in estimating direct effects . Int J Epidemiol 2002 ; 31 : 163 – 65 .

Hernan MA Hernandez-Diaz S Robins JM . A structural approach to selection bias . Epidemiology 2004 ; 15 : 615 – 25 .

Glymour MM Greenland S . Causal diagrams . In: Rothman KJ Greenland S Lash TL (eds). Modern Epidemiology . 3rd edn. Philadelphia, PA : Lippincott Williams and Wilkins , 2008 .

Hernan MA Cole SR . Invited Commentary: Causal diagrams and measurement bias . Am J Epidemiol 2009 ; 170 : 959 – 62 ; discussion 63-64 .

Jurek AM Greenland S Maldonado G . How far from non-differential does exposure or disease misclassification have to be to bias measures of association away from the null? Int J Epidemiol 2008 ; 37 : 382 – 85 .

Drews CD Greeland S . The impact of differential recall on the results of case-control studies . Int J Epidemiol 1990 ; 19 : 1107 – 12 .

Lash TL Fink AK . Re: ‘Neighborhood environment and loss of physical function in older adults: evidence from the Alameda County Study’ . Am J Epidemiol 2003 ; 157 : 472 – 73 .

Chu H Wang Z Cole SR Greenland S . Sensitivity analysis of misclassification: a graphical and a Bayesian approach . Ann Epidemiol 2006 ; 16 : 834 – 41 .

Hernan MA . The hazards of hazard ratios . Epidemiology 2010 ; 21 : 13 – 35 .

Flanders WD Klein M . Properties of 2 counterfactual effect definitions of a point exposure . Epidemiology 2007 ; 18 : 453 – 60 .

Ogburn EL VanderWeele TJ . On the nondifferential misclassification of a binary confounder . Epidemiology 2012 ; 23 : 433 – 39 .

Kristensen P . Bias from nondifferential but dependent misclassification of exposure and outcome . Epidemiology 1992 ; 3 : 210 – 15 .

Chavance M Dellatolas G . Bias from dependent misclassification . Epidemiology 1993 ; 4 : 180 – 82 .

VanderWeele TJ Robins JM . Signed directed acyclic graphs for causal inference . J R Stat Soc Ser B 2010 ; 72 : 111 - 27 .

VanderWeele TJ Hernan MA . Results on differential and dependent measurement error of the exposure and the outcome using signed directed acyclic graphs . Am J Epidemiol 2012 ; 175 : 1303 – 10 .

Bross ID . Pertinency of an extraneous variable . J Chron Dis 1967 ; 20 : 487 – 95 .

Yanagawa T . Case-control studies: Assessing the effect of a confouding factor . Biometrika 1984 ; 71 : 191 – 94 .

Arah OA Chiba Y Greenland S . Bias formulas for external adjustment and sensitivity analysis of unmeasured confounders . Ann Epidemiol 2008 ; 18 : 637 – 46 .

Vanderweele TJ Arah OA . Bias formulas for sensitivity analysis of unmeasured confounding for general outcomes, treatments, and confounders . Epidemiology 2011 ; 22 : 42 – 52 .

Gustafson P Le ND Saskin R . Case-control analysis with partial knowledge of exposure misclassification probabilities . Biometrics 2001 ; 57 : 598 – 609 .

Greenland S . Generalized conjugate priors for Bayesian analysis of risk and survival regressions . Biometrics 2003 ; 59 : 92 – 99 .

Gustafson P . On model expansion, model contraction, identifiability and prior information: Two illustrative scenarios involving mismeasured variables . Stat Sci 2005 ; 20 : 111 – 29 .

Gustafson P . On model expansion, model contraction, identifiability and prior information: Two illustrative scenarios involving mismeasured variables - Rejoinder . Stat Sci 2005 ; 20 : 137 – 40 .

Turner RM Spiegelhalter DJ Smith GC Thompson SG . Bias modelling in evidence synthesis . J R Stat Soc Ser A 2009 ; 172 : 21 – 47 .

Gustafson P McCandless LC . Probabilistic approaches to better quantifying the results of epidemiological studies . Int J Environ Res Public Health 2010 ; 7 : 1520 – 39 .

Steenland K Greenland S . Monte Carlo sensitivity analysis and Bayesian analysis of smoking as an unmeasured confounder in a study of silica and lung cancer . Am J Epidemiol 2004 ; 160 : 384 – 92 .

MacLehose RF Gustafson P . Is probabilistic bias analysis approximately Bayesian? Epidemiology 2012 ; 23 : 151 – 58 .

Greenland S . Relaxation penalties and priors for plausible modeling of nonidentified bias sources . Stat Sci 2009 ; 24 : 195 – 210 .

MacLehose RF Olshan AF Herring AH et al.  . Bayesian methods for correcting misclassification: an example from birth defects epidemiology . Epidemiology 2009 ; 20 : 27 – 35 .

Robins JM Rotnitzky A Zhao LP . Estimation of regression coefficients when some regressors are not always observed . J Am Stat Assoc 1994 ; 89 : 846 – 66 .

Garthwaite PH Kadane JB O’Hagan A . Statistical methods for eliciting probability distributions . J Am Stat Assoc 2005 ; 100 : 680 – 700 .

Kahneman D Slovic P Tversky A . Judgment Under Uncertainty: Heuristics and Biases . New York : Cambridge University Press , 1982 .

Gilovich T Griffin D Kahneman D . Heuristics and Biases: The Psychology of Intuitive Judgment . New York : Cambridge University Press , 2002 .

Lash TL . Heuristic thinking and inference from observational epidemiology . Epidemiology 2007 ; 18 : 67 – 72 .

Greenland S . Sensitivity analysis, Monte Carlo risk analysis, and Bayesian uncertainty assessment . Risk Analysis 2001 ; 21 : 579 – 83 .

Stonebraker JS Farrugia A Gathmann B Party ERW Orange JS . Modeling primary immunodeficiency disease epidemiology and its treatment to estimate latent therapeutic demand for immunoglobulin . J Clin Iimmunol 2014 ; 34 : 233 – 44 .

Maldonado G . Adjusting a relative-risk estimate for study imperfections . J Epidemiol Community Health 2008 ; 62 : 655 – 63 .

Maclure M Schneeweiss S . Causation of bias: the episcope . Epidemiology 2001 ; 12 : 114 – 22 .

Savitz DA . Interpreting Epidemiological Evidence Strategies for Study Design and Analysis . Oxford, UK : Oxford University Press , 2003 .

Greenland S Neutra R . An analysis of detection bias and proposed corrections in the study of estrogens and endometrial cancer . J Chron Dis 1981 ; 34 : 433 – 3-8 .

Phillips CV . Publication bias in situ . BMC Med Res Methodol 2004 ; 4 : 20 .

Henmi M Copas JB Eguchi S . Confidence intervals and P-values for meta-analysis with publication bias . Biometrics 2007 ; 63 : 475 – 82 .

Copas J Dwan K Kirkham J Williamson P . A model-based correction for outcome reporting bias in meta-analysis . Biostatistics 2014 ; 15 : 370 – 83 .

Greenland S O’Rourke K . Meta-analysis . In: Rothman KJ Greenland S Lash TL (eds). Modern Epidemiology . 3rd edn. Philadelphia, PA : Lippincott Williams and Wilkins ; 2008 .

Al-Marzouki S Evans S Marshall T Roberts I . Are these data real? Statistical methods for the detection of data fabrication in clinical trials . BMJ 2005 ; 331 : 267 – 70 .

Baggerly KA Coombes KR . Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology . Ann Appl Stat 2009 ; 3 : 1309 – 34 .

Pogue JM Devereaux PJ Thorlund K Yusuf S . Central statistical monitoring: detecting fraud in clinical trials . Clin Trials 2013 ; 10 : 225 – 35 .

Blair A Steenland K Shy C O’Berg M Halperin W Thomas T . Control of smoking in occupational epidemiological studies: methods and needs . Am J Ind Med 1988 ; 13 : 3 – 4 .

Gustafson P Greenland S . Curious phenomena in Bayesian adjustment for exposure misclassification . Stat Med 2006 ; 25 : 87 – 103 .

• misclassification
• sensitivity analysis
• interpretation of findings
• measurement error
• transparency

Email alerts

Citing articles via, looking for your next opportunity.

• About International Journal of Epidemiology
• Recommend to your Library

Affiliations

• Online ISSN 1464-3685
• Copyright © 2024 International Epidemiological Association
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Write for Us
• BMJ Journals

You are here

• Volume 17, Issue 4
• Bias in research
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• Joanna Smith 1 ,
• Helen Noble 2
• 1 School of Human and Health Sciences, University of Huddersfield , Huddersfield , UK
• 2 School of Nursing and Midwifery, Queens's University Belfast , Belfast , UK
• Correspondence to : Dr Joanna Smith , School of Human and Health Sciences, University of Huddersfield, Huddersfield HD1 3DH, UK; j.e.smith{at}hud.ac.uk

https://doi.org/10.1136/eb-2014-101946

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

The aim of this article is to outline types of ‘bias’ across research designs, and consider strategies to minimise bias. Evidence-based nursing, defined as the “process by which evidence, nursing theory, and clinical expertise are critically evaluated and considered, in conjunction with patient involvement, to provide the delivery of optimum nursing care,” 1 is central to the continued development of the nursing professional. Implementing evidence into practice requires nurses to critically evaluate research, in particular assessing the rigour in which methods were undertaken and factors that may have biased findings.

What is bias in relation to research and why is understanding bias important?

Although different study designs have specific methodological challenges and constraints, bias can occur at each stage of the research process ( table 1 ). In quantitative research, the validity and reliability are assessed using statistical tests that estimate the size of error in samples and calculating the significance of findings (typically p values or CIs). The tests and measures used to establish the validity and reliability of quantitative research cannot be applied to qualitative research. However, in the broadest context, these terms are applicable, with validity referring to the integrity and application of the methods and the precision in which the findings accurately reflect the data, and reliability referring to the consistency within the analytical processes. 4

• View inline

Types of research bias

How is bias minimised when undertaken research?

Bias exists in all study designs, and although researchers should attempt to minimise bias, outlining potential sources of bias enables greater critical evaluation of the research findings and conclusions. Researchers bring to each study their experiences, ideas, prejudices and personal philosophies, which if accounted for in advance of the study, enhance the transparency of possible research bias. Clearly articulating the rationale for and choosing an appropriate research design to meet the study aims can reduce common pitfalls in relation to bias. Ethics committees have an important role in considering whether the research design and methodological approaches are biased, and suitable to address the problem being explored. Feedback from peers, funding bodies and ethics committees is an essential part of designing research studies, and often provides valuable practical guidance in developing robust research.

In quantitative studies, selection bias is often reduced by the random selection of participants, and in the case of clinical trials randomisation of participants into comparison groups. However, not accounting for participants who withdraw from the study or are lost to follow-up can result in sample bias or change the characteristics of participants in comparison groups. 7 In qualitative research, purposeful sampling has advantages when compared with convenience sampling in that bias is reduced because the sample is constantly refined to meet the study aims. Premature closure of the selection of participants before analysis is complete can threaten the validity of a qualitative study. This can be overcome by continuing to recruit new participants into the study during data analysis until no new information emerges, known as data saturation. 8

In quantitative studies having a well-designed research protocol explicitly outlining data collection and analysis can assist in reducing bias. Feasibility studies are often undertaken to refine protocols and procedures. Bias can be reduced by maximising follow-up and where appropriate in randomised control trials analysis should be based on the intention-to-treat principle, a strategy that assesses clinical effectiveness because not everyone complies with treatment and the treatment people receive may be changed according to how they respond. Qualitative research has been criticised for lacking transparency in relation to the analytical processes employed. 4 Qualitative researchers must demonstrate rigour, associated with openness, relevance to practice and congruence of the methodological approach. Although other researchers may interpret the data differently, appreciating and understanding how the themes were developed is an essential part of demonstrating the robustness of the findings. Reducing bias can include respondent validation, constant comparisons across participant accounts, representing deviant cases and outliers, prolonged involvement or persistent observation of participants, independent analysis of the data by other researchers and triangulation. 4

In summary, minimising bias is a key consideration when designing and undertaking research. Researchers have an ethical duty to outline the limitations of studies and account for potential sources of bias. This will enable health professionals and policymakers to evaluate and scrutinise study findings, and consider these when applying findings to practice or policy.

• Wakefield AJ ,
• Anthony A ,
• ↵ The Lancet . Retraction—ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children . Lancet 2010 ; 375 : 445 . OpenUrl CrossRef PubMed Web of Science
• Easterbrook PJ ,
• Berlin JA ,
• Gopalan R ,
• Petticrew M ,
• Thomson H ,
• Francis J ,
• Johnston M ,
• Robertson C ,

Competing interests None.

Read the full text or download the PDF:

• Skip to main content
• Skip to primary sidebar
• Skip to footer
• QuestionPro

• Solutions Industries Gaming Automotive Sports and events Education Government Travel & Hospitality Financial Services Healthcare Cannabis Technology Use Case NPS+ Communities Audience Contactless surveys Mobile LivePolls Member Experience GDPR Positive People Science 360 Feedback Surveys
• Resources Blog eBooks Survey Templates Case Studies Training Help center

Home Market Research Research Tools and Apps

Research bias: What it is, Types & Examples

The researcher sometimes unintentionally or actively affects the process while executing a systematic inquiry. It is known as research bias, and it can affect your results just like any other sort of bias.

When it comes to studying bias, there are no hard and fast guidelines, which simply means that it can occur at any time. Experimental mistakes and a lack of concern for all relevant factors can lead to research bias.

One of the most common causes of study results with low credibility is study bias. Because of its informal nature, you must be cautious when characterizing bias in research. To reduce or prevent its occurrence, you need to be able to recognize its characteristics. 

This article will cover what it is, its type, and how to avoid it.

Content Index

What is research bias?

How does research bias affect the research process, types of research bias with examples, how questionpro helps in reducing bias in a research process.

Research bias is a technique in which the researchers conducting the experiment modify the findings to present a specific consequence. It is often known as experimenter bias.

Bias is a characteristic of the research technique that makes it rely on experience and judgment rather than data analysis. The most important thing to know about bias is that it is unavoidable in many fields. Understanding research bias and reducing the effects of biased views is an essential part of any research planning process.

For example, it is much easier to become attracted to a certain point of view when using social research subjects, compromising fairness.

Research bias can majorly affect the research process, weakening its integrity and leading to misleading or erroneous results. Here are some examples of how this bias might affect the research process:

Distorted research design

When bias is present, study results can be skewed or wrong. It can make the study less trustworthy and valid. If bias affects how a study is set up, how data is collected, or how it is analyzed, it can cause systematic mistakes that move the results away from the true or unbiased values.

Invalid conclusions

It can make it hard to believe that the findings of a study are correct. Biased research can lead to unjustified or wrong claims because the results may not reflect reality or give a complete picture of the research question.

Misleading interpretations

Bias can lead to inaccurate interpretations of research findings. It can alter the overall comprehension of the research issue. Researchers may be tempted to interpret the findings in a way that confirms their previous assumptions or expectations, ignoring alternate explanations or contradictory evidence.

Ethical concerns

This bias poses ethical considerations. It can have negative effects on individuals, groups, or society as a whole. Biased research can misinform decision-making processes, leading to ineffective interventions, policies, or therapies.

Damaged credibility

Research bias undermines scientific credibility. Biased research can damage public trust in science. It may reduce reliance on scientific evidence for decision-making.

Bias can be seen in practically every aspect of quantitative research and qualitative research , and it can come from both the survey developer and the participants. The sorts of biases that come directly from the survey maker are the easiest to deal with out of all the types of bias in research. Let’s look at some of the most typical research biases.

Design bias

Design bias happens when a researcher fails to capture biased views in most experiments. It has something to do with the organization and its research methods. The researcher must demonstrate that they realize this and have tried to mitigate its influence.

Another design bias develops after the research is completed and the results are analyzed. It occurs when the researchers’ original concerns are not reflected in the exposure, which is all too often these days.

For example, a researcher working on a survey containing questions concerning health benefits may overlook the researcher’s awareness of the sample group’s limitations. It’s possible that the group tested was all male or all over a particular age.

Selection bias or sampling bias

Selection bias occurs when volunteers are chosen to represent your research population, but those with different experiences are ignored. 

In research, selection bias manifests itself in a variety of ways. When the sampling method puts preference into the research, this is known as sampling bias . Selection bias is also referred to as sampling bias.

For example, research on a disease that depended heavily on white male volunteers cannot be generalized to the full community, including women and people of other races or communities.

Procedural bias

Procedural bias is a sort of research bias that occurs when survey respondents are given insufficient time to complete surveys. As a result, participants are forced to submit half-thoughts with misinformation, which does not accurately reflect their thinking.

Another sort of study bias is using individuals who are forced to participate, as they are more likely to complete the survey fast, leaving them with enough time to accomplish other things.

For Example, If you ask your employees to survey their break, they may be pressured, which may compromise the validity of their results.

Publication or reporting bias

A sort of bias that influences research is publication bias. It is also known as reporting bias. It refers to a condition in which favorable outcomes are more likely to be reported than negative or empty ones. Analysis bias can also make it easier for reporting bias to happen.

The publication standards for research articles in a specific area frequently reflect this bias on them. Researchers sometimes choose not to disclose their outcomes if they believe the data do not reflect their theory.

As an example, there was seven research on the antidepressant drug Reboxetine. Among them, only one got published, and the others were unpublished.

Measurement of data collecting bias

A defect in the data collection process and measuring technique causes measurement bias. Data collecting bias is also known as measurement bias. It occurs in both qualitative and quantitative research methodologies. 

Data collection methods might occur in quantitative research when you use an approach that is not appropriate for your research population. Instrument bias is one of the most common forms of measurement bias in quantitative investigations. A defective scale would generate instrument bias and invalidate the experimental process in a quantitative experiment.

For example, you may ask those who do not have internet access to survey by email or on your website.

Data collection bias occurs in qualitative research when inappropriate survey questions are asked during an unstructured interview. Bad survey questions are those that lead the interviewee to make presumptions. Subjects are frequently hesitant to provide socially incorrect responses for fear of criticism.

For example, a topic can avoid coming across as homophobic or racist in an interview.

Some more types of bias in research include the ones listed here. Researchers must understand these biases and reduce them through rigorous study design, transparent reporting, and critical evidence review: 

• Confirmation bias: Researchers often search for, evaluate, and prioritize material that supports their existing hypotheses or expectations, ignoring contradictory data. This can lead to a skewed perception of results and perhaps biased conclusions.
• Cultural bias: Cultural bias arises when cultural norms, attitudes, or preconceptions influence the research process and the interpretation of results.
• Funding bias: Funding bias takes place when powerful motives support research. It can bias research design, data collecting, analysis, and interpretation toward the funding source.
• Observer bias: Observer bias arises when the researcher or observer affects participants’ replies or behavior. Collecting data might be biased by accidental clues, expectations, or subjective interpretations.

LEARN ABOUT: Theoretical Research

QuestionPro offers several features and functionalities that can contribute to reducing bias in the research process. Here’s how QuestionPro can help:

Randomization

QuestionPro allows researchers to randomize the order of survey questions or response alternatives. Randomization helps to remove order effects and limit bias from the order in which participants encounter the items.

Branching and skip logic

Branching and skip logic capabilities in QuestionPro allow researchers to design customized survey pathways based on participants’ responses. It enables tailored questioning, ensuring that only pertinent questions are asked of participants. Bias generated by such inquiries is reduced by avoiding irrelevant or needless questions.

Diverse question types

QuestionPro supports a wide range of questions kinds, including multiple-choice, Likert scale, matrix, and open-ended questions. Researchers can choose the most relevant question kinds to get unbiased data while avoiding leading or suggestive questions that may affect participants’ responses.

Anonymous responses

QuestionPro enables researchers to collect anonymous responses, protecting the confidentiality of participants. It can encourage participants to provide more unbiased and equitable feedback, especially when dealing with sensitive or contentious issues.

Data analysis and reporting

QuestionPro has powerful data analysis and reporting options, such as charts, graphs, and statistical analysis tools. These properties allow researchers to examine and interpret obtained data objectively, decreasing the role of bias in interpreting results.

Collaboration and peer review

QuestionPro supports peer review and researcher collaboration. It helps uncover and overcome biases in research planning, questionnaire formulation, and data analysis by involving several researchers and soliciting external opinions.

You must comprehend biases in research and how to deal with them. Knowing the different sorts of biases in research allows you to readily identify them. It is also necessary to have a clear idea to recognize it in any form.

QuestionPro provides many research tools and settings that can assist you in dealing with research bias. Try QuestionPro today to undertake your original bias-free quantitative or qualitative research.

LEARN MORE         FREE TRIAL

Frequently Asking Questions

Research bias affects the validity and dependability of your research’s findings, resulting in inaccurate interpretations of the data and incorrect conclusions.

Bias should be avoided in research to ensure that findings are accurate, valid, and objective.

 To avoid research bias, researchers should take proactive steps throughout the research process, such as developing a clear research question and objectives, designing a rigorous study, following standardized protocols, and so on.

MORE LIKE THIS

A Case for Empowerment and Being Bold — Tuesday CX Thoughts

Jul 30, 2024

Typeform vs. Google Forms: Which one is best for my needs?

Microsoft Forms vs SurveyMonkey: Complete Analysis

Jul 29, 2024

Explore The Photo Elicitation Power in Qualitative Research

Jul 25, 2024

Other categories

• Academic Research
• Artificial Intelligence
• Assessments
• Brand Awareness
• Case Studies
• Communities
• Consumer Insights
• Customer effort score
• Customer Engagement
• Customer Experience
• Customer Loyalty
• Customer Research
• Customer Satisfaction
• Employee Benefits
• Employee Engagement
• Employee Retention
• Friday Five
• General Data Protection Regulation
• Insights Hub
• Life@QuestionPro
• Market Research
• Mobile diaries
• Mobile Surveys
• New Features
• Online Communities
• Question Types
• Questionnaire
• QuestionPro Products
• Release Notes
• Research Tools and Apps
• Revenue at Risk
• Survey Templates
• Training Tips
• Tuesday CX Thoughts (TCXT)
• Uncategorized
• What’s Coming Up
• Workforce Intelligence

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• What Is Quantitative Research? | Definition, Uses & Methods

What Is Quantitative Research? | Definition, Uses & Methods

Published on June 12, 2020 by Pritha Bhandari . Revised on June 22, 2023.

Quantitative research is the process of collecting and analyzing numerical data. It can be used to find patterns and averages, make predictions, test causal relationships, and generalize results to wider populations.

Quantitative research is the opposite of qualitative research , which involves collecting and analyzing non-numerical data (e.g., text, video, or audio).

Quantitative research is widely used in the natural and social sciences: biology, chemistry, psychology, economics, sociology, marketing, etc.

• What is the demographic makeup of Singapore in 2020?
• How has the average temperature changed globally over the last century?
• Does environmental pollution affect the prevalence of honey bees?
• Does working from home increase productivity for people with long commutes?

Table of contents

Quantitative research methods, quantitative data analysis, advantages of quantitative research, disadvantages of quantitative research, other interesting articles, frequently asked questions about quantitative research.

You can use quantitative research methods for descriptive, correlational or experimental research.

• In descriptive research , you simply seek an overall summary of your study variables.
• In correlational research , you investigate relationships between your study variables.
• In experimental research , you systematically examine whether there is a cause-and-effect relationship between variables.

Correlational and experimental research can both be used to formally test hypotheses , or predictions, using statistics. The results may be generalized to broader populations based on the sampling method used.

To collect quantitative data, you will often need to use operational definitions that translate abstract concepts (e.g., mood) into observable and quantifiable measures (e.g., self-ratings of feelings and energy levels).

Note that quantitative research is at risk for certain research biases , including information bias , omitted variable bias , sampling bias , or selection bias . Be sure that you’re aware of potential biases as you collect and analyze your data to prevent them from impacting your work too much.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Once data is collected, you may need to process it before it can be analyzed. For example, survey and test data may need to be transformed from words to numbers. Then, you can use statistical analysis to answer your research questions .

Descriptive statistics will give you a summary of your data and include measures of averages and variability. You can also use graphs, scatter plots and frequency tables to visualize your data and check for any trends or outliers.

Using inferential statistics , you can make predictions or generalizations based on your data. You can test your hypothesis or use your sample data to estimate the population parameter .

First, you use descriptive statistics to get a summary of the data. You find the mean (average) and the mode (most frequent rating) of procrastination of the two groups, and plot the data to see if there are any outliers.

You can also assess the reliability and validity of your data collection methods to indicate how consistently and accurately your methods actually measured what you wanted them to.

Quantitative research is often used to standardize data collection and generalize findings . Strengths of this approach include:

• Replication

Repeating the study is possible because of standardized data collection protocols and tangible definitions of abstract concepts.

• Direct comparisons of results

The study can be reproduced in other cultural settings, times or with different groups of participants. Results can be compared statistically.

• Large samples

Data from large samples can be processed and analyzed using reliable and consistent procedures through quantitative data analysis.

• Hypothesis testing

Using formalized and established hypothesis testing procedures means that you have to carefully consider and report your research variables, predictions, data collection and testing methods before coming to a conclusion.

Despite the benefits of quantitative research, it is sometimes inadequate in explaining complex research topics. Its limitations include:

• Superficiality

Using precise and restrictive operational definitions may inadequately represent complex concepts. For example, the concept of mood may be represented with just a number in quantitative research, but explained with elaboration in qualitative research.

• Narrow focus

Predetermined variables and measurement procedures can mean that you ignore other relevant observations.

• Structural bias

Despite standardized procedures, structural biases can still affect quantitative research. Missing data , imprecise measurements or inappropriate sampling methods are biases that can lead to the wrong conclusions.

• Lack of context

Quantitative research often uses unnatural settings like laboratories or fails to consider historical and cultural contexts that may affect data collection and results.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Chi square goodness of fit test
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Inclusion and exclusion criteria

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

In mixed methods research , you use both qualitative and quantitative data collection and analysis methods to answer your research question .

Data collection is the systematic process by which observations or measurements are gathered in research. It is used in many different contexts by academics, governments, businesses, and other organizations.

Operationalization means turning abstract conceptual ideas into measurable observations.

For example, the concept of social anxiety isn’t directly observable, but it can be operationally defined in terms of self-rating scores, behavioral avoidance of crowded places, or physical anxiety symptoms in social situations.

Before collecting data , it’s important to consider how you will operationalize the variables that you want to measure.

Reliability and validity are both about how well a method measures something:

• Reliability refers to the  consistency of a measure (whether the results can be reproduced under the same conditions).
• Validity   refers to the  accuracy of a measure (whether the results really do represent what they are supposed to measure).

If you are doing experimental research, you also have to consider the internal and external validity of your experiment.

Hypothesis testing is a formal procedure for investigating our ideas about the world using statistics. It is used by scientists to test specific predictions, called hypotheses , by calculating how likely it is that a pattern or relationship between variables could have arisen by chance.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bhandari, P. (2023, June 22). What Is Quantitative Research? | Definition, Uses & Methods. Scribbr. Retrieved July 30, 2024, from https://www.scribbr.com/methodology/quantitative-research/

Is this article helpful?

Pritha Bhandari

Other students also liked, descriptive statistics | definitions, types, examples, inferential statistics | an easy introduction & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Opinion Stage » survey » Types of Bias

Types of Bias in Research: How to Identify & Prevent Them in Your Surveys

Bias exists in all forms of research and every discipline. Even the most seasoned researchers acknowledge the fact that the different types of bias in research can exist at any phase of the study – from survey design and data collection to analysis.

Why is bias a problem in research? More importantly, how can we eliminate bias to produce high-quality results?

Here, we will delve into the different types of bias in research that we should strive to eliminate, as well as how we can avoid them.

What Is Bias in Research?

Bias in research pertains to unfair and prejudiced practices that influence the results of the study. From sampling bias to asking leading questions, unfair practices can seep into different phases of research. Thus, it’s important for researchers to be well aware of its many forms in order to prevent or eliminate them from the study.

Whether or not researchers do it intentionally, bias can negatively affect the outcomes of the study. It makes the results irrelevant and insignificant. Eliminating bias in research is quite a taxing endeavor, but it’s all worth it. You ought to know how to identify the different kinds of bias researchers and respondents can introduce into the study. That’s the first step to ensuring the quality of your results.

In eliminating or, at least, minimizing bias, you can produce research with reliable and valid results that can benefit your business, community, or society in general. Publishing false claims can do more harm than good to the people and organizations that rely on these studies.

Unfortunately, many turn a blind eye to research bias. Sometimes, it could be a lack of resources or time that drives researchers to ignore these unfair practices.

Opinion Stage is an interactive content creation platform that can help you minimize bias in research. As it comes with an intuitive  survey maker  with pre-made yet highly-customizable  survey templates , you can write unbiased questions that will produce valid and reliable results. The survey maker even delivers comprehensive reports on the results and performance of the survey, allowing you to improve the quality of your research.

Write unbiased survey questions

Types of Bias in Qualitative Research

Human error causes bias in research. Some people may do it intentionally. However, most researchers unknowingly add all kinds of biases into their studies at various phases of the study.

Bias in research, whether quantitative or qualitative, may come in different types.

Due to the nature of qualitative data, bias is more likely to occur in this form of research. Not only that, it can exist in all parts of the study. However, qualitative research has more room for creativity and flexibility. Thus, it can produce more insights that one cannot generate from quantitative research.

In qualitative research, bias can either be caused by respondents or researchers.

Respondent Bias

Respondents can add bias to your research by answering questions untruthfully. If they choose answers that are more socially acceptable instead of ones that reflect what they truly think or feel, they unknowingly create bias. Sometimes, respondents also introduce bias when they know the researchers or sponsors of the study. For example, respondents may simply agree to everything that’s recommended to them.

Researcher Bias

When researchers conduct their study in a manner that influences the outcomes, they commit one of the two main forms of bias in qualitative studies—researcher bias.

Much like respondents, researchers can commit different types of bias in research. They may introduce bias when they ask questions that influence the respondents’ answers or when they interpret data to match their hypothesis.

Avoiding bias in qualitative studies is challenging. At best, you can eliminate such occurrences in the study to protect the quality of the data you gather and the integrity of the research itself.

Identifying Respondent Bias

Survey participants are often unaware of it, but they tend to introduce bias into research by answering questions untruthfully. Response bias occurs when they feel pressured to answer questions in a more socially acceptable way. Or, they might feel compelled to provide responses that enable researchers to achieve their desired outcomes. Non-response bias is also common, which occurs when response rates are low.

Let’s take a closer look at the different forms of respondent bias.

Friendliness Bias

Also dubbed acquiescence bias, friendliness bias occurs when respondents simply tend to agree with the ideas that they are presented within the survey. They may even tend to give more positive ratings or feedback. Sometimes, this happens when participants perceive the brand or researchers as professionals or authoritative figures. However, it may still happen even when respondents don’t necessarily have an inherent affinity towards the brand surveying them. Their acquiescent personalities may enable them to introduce such biases to the research.

There are also instances where friendliness bias occurs because of the length of the survey itself. When it drags on, respondents may feel disengaged and tired that they will start to agree with everything that’s presented to them.

Social Desirability Bias

Another form of bias that’s created when respondents answer questions in a particular manner is called social desirability bias. This occurs when people give responses that they think are more socially acceptable. Let’s say you’re asking controversial or sensitive questions. Respondents may provide inaccurate answers just to put themselves in the best possible light.

You can prevent this from happening by choosing your words carefully. Make respondents feel that there is no right answer and that any answer is acceptable. You may also want to consider phrasing the questions indirectly, such as asking them what a third party might do in certain scenarios. In doing so, respondents can project their own perspectives and answer questions more accurately.

Sponsor Bias

Sponsor bias usually occurs when respondents who are familiar with the researchers provide answers which they think they will want to hear. Knowing a brand’s mission or core principles, for example, may influence how they respond to the survey questions.

Maintaining a neutral stance in your questions will prevent you from influencing the respondents’ answers. Furthermore, you should refrain from providing details about the sponsors, including their logo or the goal of the research.

Habituation Bias

A survey must always be brief and engaging. If not, respondents may find it tedious and boring. Some might drop out, while others might continue answering questions without fully paying attention.

Halfway through the questionnaire, they might start providing similar responses to questions that are phrased in the same manner just so they can get it over with. This is one form of bias in qualitative studies. It’s called habituation bias, which is mostly how our brain responds to conserve more energy.

Language is key to preventing this type of bias. In designing the survey, it’s best to keep the questions conversational and engaging. At the same time, researchers should change the wording of the questions.

Opinion Stage offers a  Survey Maker  tool in their online platform which provides visually appealing pre-made templates that respondents want to participate in. With higher response rates, you can expect to get better and more accurate results from respondents.

Identifying Researcher Bias

Researchers tend to introduce bias when they want their study to produce certain outcomes, particularly ones that meet their hypothesis. If researchers manipulate questions to prompt the desired responses, they are only compromising the quality of their data. As a result, they will end up wasting precious resources on insights that won’t move the needle.

Here are some of the most common forms of researcher bias.

Leading Questions

Using biased language in  survey questions  can affect the answers of the respondents. This is evident in leading questions, where respondents are often influenced to answer in a particular manner. Much like other forms of bias, these questions should be avoided at all costs as they produce inaccurate results that can hurt the quality of your research.

Examples of leading questions:

• Was our excellent customer support team helpful?
• Do you have any problems with customer service?

Question-Order Bias

Aside from the language used in a survey, the order of questions, as well as their level of specificity, can affect respondents’ answers. If one question influences a respondent’s answer to succeeding questions, it constitutes research bias. This, in particular, is called question-order bias.

Let’s say you want respondents to rate how satisfied they are with your brand in general and then with a specific product. If you ask them to report on their brand experience first, you will see a slight correlation between the questions and their responses. Reversing the order, however, may lead to different outcomes, where happy consumers of a particular product rate higher satisfaction levels with the brand. By starting with the specific question, you gain biased results on the more general query.

Question-order bias is often inevitable. However, you can strive to minimize this type of bias by designing your surveys carefully and meticulously. You may also want to ask general questions before delving into more specific ones.

Asking unaided questions before delving into aided questions can help minimize bias. Moreover, you may want to start with positive questions before going into negative ones.

The Opinion Stage intuitive survey maker utilizes branch logic to deliver the right order of questions based on respondents’ answers. Using this highly-customizable intuitive solution, you can improve response rates, gather accurate data, and produce actionable insights.

Eliminate researcher bias from your survey

Confirmation Bias

One of the most pervasive types of bias in qualitative research happens when researchers establish a particular hypothesis and shape their entire methodology to confirm the premise. This is called confirmation bias.

When researchers determine the value of responses based on their capacity to support their hypothesis, confirmation bias occurs.

Cultural Bias

Researchers who judge people based on their own culture’s values and standards introduce cultural bias into their study. To eliminate this form of bias, researchers must strive to understand the influences and motivations of a particular group of respondents in terms of their own culture.

Eliminating this type of bias might not be completely possible. However, by being conscious of your own cultural assumptions, you can significantly minimize such instances in your research.

Types of Bias in Quantitative Research

As with qualitative studies, bias in quantitative research can affect the validity of the results. Researchers must be very careful of the methods they use in this type of research to prove the accuracy and integrity of their study.

Let’s find out what types of biases can occur in quantitative research.

Historical Bias

Long-term experiments and studies are susceptible to historical bias because, along the way, respondents may experience different events that influence their thoughts and attitudes. In turn, it may skew the results of your experiment.

If, for instance, you conducted an experiment around the time of an earthquake, respondents who saw its effects first-hand may have different beliefs and attitudes. To prevent history bias, you should establish experimental and control groups that have experienced the same events. You may select respondents from the same communities or organizations.

Maturation Bias

Aside from events and experiences, time can change the attitudes, feelings, and thoughts of respondents. If you conduct a long-term study, their maturity might skew the outcome of your research. You can, however, prevent this from happening by selecting participants from the same age group. This way, they will grow at the same pace as everyone else throughout the duration of your study.

Measurement Bias

In quantitative research, another type of bias you might encounter is measurement bias. It refers to a systematic error that happens during the data collection phase of research. It occurs when you measure outcomes poorly. When you overstate or understate the value of measurement, favoring a certain result, you create this type of bias.

Overcoming Different Types of Bias in Research

Quantitative and qualitative studies utilize different methodologies. Similarly, both approaches require different methods and processes when it comes to avoiding bias.

Qualitative Research

In terms of qualitative studies, researchers can avoid bias by being aware of its many forms. Knowing what to avoid is an excellent first step toward accurate and valid research. Given that language plays a crucial role in qualitative studies, you must also be very careful in designing survey questions. Neutral language must be favored over biased and loaded words. Not to mention, you must make your surveys quick, simple, conversational, and engaging.

Quantitative Research

Unlike qualitative studies, researchers can eliminate bias in quantitative studies. You can utilize different statistical tests such as z-test and t-test to determine the authenticity and integrity of your results. You may want to choose your respondents wisely. Randomization, for example, can help eliminate bias. However, if you’re doing long-term research, you may want to pick respondents from the same age group or the same community.

Of course, you must, at all times, refrain from using methods that steer results to confirm your hypothesis.

The Importance of Avoiding Research Bias

Biased research has very little to no value. Such studies produce distorted impressions and present false conclusions. Often, the findings of biased research can impede important decision-making processes and may cause harm to the people or businesses that rely on them.

Acknowledging one’s susceptibility to bias and understanding its many forms will help you design a balanced survey. As long as you know what types of bias in research to look for and how to eliminate them, you can produce valuable results.

The Opinion Stage intuitive  survey maker  has a series of templates that help you make engaging and unbiased questions that produce accurate and actionable results.

Popular Resources

You can easily do it yourself, no need for a developer

Qualitative vs Quantitative Research Methods & Data Analysis

Saul Mcleod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul Mcleod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Learn about our Editorial Process

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

On This Page:

What is the difference between quantitative and qualitative?

The main difference between quantitative and qualitative research is the type of data they collect and analyze.

Quantitative research collects numerical data and analyzes it using statistical methods. The aim is to produce objective, empirical data that can be measured and expressed in numerical terms. Quantitative research is often used to test hypotheses, identify patterns, and make predictions.

Qualitative research , on the other hand, collects non-numerical data such as words, images, and sounds. The focus is on exploring subjective experiences, opinions, and attitudes, often through observation and interviews.

Qualitative research aims to produce rich and detailed descriptions of the phenomenon being studied, and to uncover new insights and meanings.

Quantitative data is information about quantities, and therefore numbers, and qualitative data is descriptive, and regards phenomenon which can be observed but not measured, such as language.

What Is Qualitative Research?

Qualitative research is the process of collecting, analyzing, and interpreting non-numerical data, such as language. Qualitative research can be used to understand how an individual subjectively perceives and gives meaning to their social reality.

Qualitative data is non-numerical data, such as text, video, photographs, or audio recordings. This type of data can be collected using diary accounts or in-depth interviews and analyzed using grounded theory or thematic analysis.

Qualitative research is multimethod in focus, involving an interpretive, naturalistic approach to its subject matter. This means that qualitative researchers study things in their natural settings, attempting to make sense of, or interpret, phenomena in terms of the meanings people bring to them. Denzin and Lincoln (1994, p. 2)

Interest in qualitative data came about as the result of the dissatisfaction of some psychologists (e.g., Carl Rogers) with the scientific study of psychologists such as behaviorists (e.g., Skinner ).

Since psychologists study people, the traditional approach to science is not seen as an appropriate way of carrying out research since it fails to capture the totality of human experience and the essence of being human.  Exploring participants’ experiences is known as a phenomenological approach (re: Humanism ).

Qualitative research is primarily concerned with meaning, subjectivity, and lived experience. The goal is to understand the quality and texture of people’s experiences, how they make sense of them, and the implications for their lives.

Qualitative research aims to understand the social reality of individuals, groups, and cultures as nearly as possible as participants feel or live it. Thus, people and groups are studied in their natural setting.

Some examples of qualitative research questions are provided, such as what an experience feels like, how people talk about something, how they make sense of an experience, and how events unfold for people.

Research following a qualitative approach is exploratory and seeks to explain ‘how’ and ‘why’ a particular phenomenon, or behavior, operates as it does in a particular context. It can be used to generate hypotheses and theories from the data.

Qualitative Methods

There are different types of qualitative research methods, including diary accounts, in-depth interviews , documents, focus groups , case study research , and ethnography.

The results of qualitative methods provide a deep understanding of how people perceive their social realities and in consequence, how they act within the social world.

The researcher has several methods for collecting empirical materials, ranging from the interview to direct observation, to the analysis of artifacts, documents, and cultural records, to the use of visual materials or personal experience. Denzin and Lincoln (1994, p. 14)

Here are some examples of qualitative data:

Interview transcripts : Verbatim records of what participants said during an interview or focus group. They allow researchers to identify common themes and patterns, and draw conclusions based on the data. Interview transcripts can also be useful in providing direct quotes and examples to support research findings.

Observations : The researcher typically takes detailed notes on what they observe, including any contextual information, nonverbal cues, or other relevant details. The resulting observational data can be analyzed to gain insights into social phenomena, such as human behavior, social interactions, and cultural practices.

Unstructured interviews : generate qualitative data through the use of open questions.  This allows the respondent to talk in some depth, choosing their own words.  This helps the researcher develop a real sense of a person’s understanding of a situation.

Diaries or journals : Written accounts of personal experiences or reflections.

Notice that qualitative data could be much more than just words or text. Photographs, videos, sound recordings, and so on, can be considered qualitative data. Visual data can be used to understand behaviors, environments, and social interactions.

Qualitative Data Analysis

Qualitative research is endlessly creative and interpretive. The researcher does not just leave the field with mountains of empirical data and then easily write up his or her findings.

Qualitative interpretations are constructed, and various techniques can be used to make sense of the data, such as content analysis, grounded theory (Glaser & Strauss, 1967), thematic analysis (Braun & Clarke, 2006), or discourse analysis .

For example, thematic analysis is a qualitative approach that involves identifying implicit or explicit ideas within the data. Themes will often emerge once the data has been coded .

Key Features

• Events can be understood adequately only if they are seen in context. Therefore, a qualitative researcher immerses her/himself in the field, in natural surroundings. The contexts of inquiry are not contrived; they are natural. Nothing is predefined or taken for granted.
• Qualitative researchers want those who are studied to speak for themselves, to provide their perspectives in words and other actions. Therefore, qualitative research is an interactive process in which the persons studied teach the researcher about their lives.
• The qualitative researcher is an integral part of the data; without the active participation of the researcher, no data exists.
• The study’s design evolves during the research and can be adjusted or changed as it progresses. For the qualitative researcher, there is no single reality. It is subjective and exists only in reference to the observer.
• The theory is data-driven and emerges as part of the research process, evolving from the data as they are collected.

Limitations of Qualitative Research

• Because of the time and costs involved, qualitative designs do not generally draw samples from large-scale data sets.
• The problem of adequate validity or reliability is a major criticism. Because of the subjective nature of qualitative data and its origin in single contexts, it is difficult to apply conventional standards of reliability and validity. For example, because of the central role played by the researcher in the generation of data, it is not possible to replicate qualitative studies.
• Also, contexts, situations, events, conditions, and interactions cannot be replicated to any extent, nor can generalizations be made to a wider context than the one studied with confidence.
• The time required for data collection, analysis, and interpretation is lengthy. Analysis of qualitative data is difficult, and expert knowledge of an area is necessary to interpret qualitative data. Great care must be taken when doing so, for example, looking for mental illness symptoms.

Advantages of Qualitative Research

• Because of close researcher involvement, the researcher gains an insider’s view of the field. This allows the researcher to find issues that are often missed (such as subtleties and complexities) by the scientific, more positivistic inquiries.
• Qualitative descriptions can be important in suggesting possible relationships, causes, effects, and dynamic processes.
• Qualitative analysis allows for ambiguities/contradictions in the data, which reflect social reality (Denscombe, 2010).
• Qualitative research uses a descriptive, narrative style; this research might be of particular benefit to the practitioner as she or he could turn to qualitative reports to examine forms of knowledge that might otherwise be unavailable, thereby gaining new insight.

What Is Quantitative Research?

Quantitative research involves the process of objectively collecting and analyzing numerical data to describe, predict, or control variables of interest.

The goals of quantitative research are to test causal relationships between variables , make predictions, and generalize results to wider populations.

Quantitative researchers aim to establish general laws of behavior and phenomenon across different settings/contexts. Research is used to test a theory and ultimately support or reject it.

Quantitative Methods

Experiments typically yield quantitative data, as they are concerned with measuring things.  However, other research methods, such as controlled observations and questionnaires , can produce both quantitative information.

For example, a rating scale or closed questions on a questionnaire would generate quantitative data as these produce either numerical data or data that can be put into categories (e.g., “yes,” “no” answers).

Experimental methods limit how research participants react to and express appropriate social behavior.

Findings are, therefore, likely to be context-bound and simply a reflection of the assumptions that the researcher brings to the investigation.

There are numerous examples of quantitative data in psychological research, including mental health. Here are a few examples:

Another example is the Experience in Close Relationships Scale (ECR), a self-report questionnaire widely used to assess adult attachment styles .

The ECR provides quantitative data that can be used to assess attachment styles and predict relationship outcomes.

Neuroimaging data : Neuroimaging techniques, such as MRI and fMRI, provide quantitative data on brain structure and function.

This data can be analyzed to identify brain regions involved in specific mental processes or disorders.

For example, the Beck Depression Inventory (BDI) is a clinician-administered questionnaire widely used to assess the severity of depressive symptoms in individuals.

The BDI consists of 21 questions, each scored on a scale of 0 to 3, with higher scores indicating more severe depressive symptoms. 

Quantitative Data Analysis

Statistics help us turn quantitative data into useful information to help with decision-making. We can use statistics to summarize our data, describing patterns, relationships, and connections. Statistics can be descriptive or inferential.

Descriptive statistics help us to summarize our data. In contrast, inferential statistics are used to identify statistically significant differences between groups of data (such as intervention and control groups in a randomized control study).

• Quantitative researchers try to control extraneous variables by conducting their studies in the lab.
• The research aims for objectivity (i.e., without bias) and is separated from the data.
• The design of the study is determined before it begins.
• For the quantitative researcher, the reality is objective, exists separately from the researcher, and can be seen by anyone.
• Research is used to test a theory and ultimately support or reject it.

Limitations of Quantitative Research

• Context: Quantitative experiments do not take place in natural settings. In addition, they do not allow participants to explain their choices or the meaning of the questions they may have for those participants (Carr, 1994).
• Researcher expertise: Poor knowledge of the application of statistical analysis may negatively affect analysis and subsequent interpretation (Black, 1999).
• Variability of data quantity: Large sample sizes are needed for more accurate analysis. Small-scale quantitative studies may be less reliable because of the low quantity of data (Denscombe, 2010). This also affects the ability to generalize study findings to wider populations.
• Confirmation bias: The researcher might miss observing phenomena because of focus on theory or hypothesis testing rather than on the theory of hypothesis generation.

Advantages of Quantitative Research

• Scientific objectivity: Quantitative data can be interpreted with statistical analysis, and since statistics are based on the principles of mathematics, the quantitative approach is viewed as scientifically objective and rational (Carr, 1994; Denscombe, 2010).
• Useful for testing and validating already constructed theories.
• Rapid analysis: Sophisticated software removes much of the need for prolonged data analysis, especially with large volumes of data involved (Antonius, 2003).
• Replication: Quantitative data is based on measured values and can be checked by others because numerical data is less open to ambiguities of interpretation.
• Hypotheses can also be tested because of statistical analysis (Antonius, 2003).

Antonius, R. (2003). Interpreting quantitative data with SPSS . Sage.

Black, T. R. (1999). Doing quantitative research in the social sciences: An integrated approach to research design, measurement and statistics . Sage.

Braun, V. & Clarke, V. (2006). Using thematic analysis in psychology . Qualitative Research in Psychology , 3, 77–101.

Carr, L. T. (1994). The strengths and weaknesses of quantitative and qualitative research : what method for nursing? Journal of advanced nursing, 20(4) , 716-721.

Denscombe, M. (2010). The Good Research Guide: for small-scale social research. McGraw Hill.

Denzin, N., & Lincoln. Y. (1994). Handbook of Qualitative Research. Thousand Oaks, CA, US: Sage Publications Inc.

Glaser, B. G., Strauss, A. L., & Strutzel, E. (1968). The discovery of grounded theory; strategies for qualitative research. Nursing research, 17(4) , 364.

Minichiello, V. (1990). In-Depth Interviewing: Researching People. Longman Cheshire.

Punch, K. (1998). Introduction to Social Research: Quantitative and Qualitative Approaches. London: Sage

Further Information

• Mixed methods research
• Designing qualitative research
• Methods of data collection and analysis
• Introduction to quantitative and qualitative research
• Checklists for improving rigour in qualitative research: a case of the tail wagging the dog?
• Qualitative research in health care: Analysing qualitative data
• Qualitative data analysis: the framework approach
• Using the framework method for the analysis of
• Qualitative data in multi-disciplinary health research
• Content Analysis
• Grounded Theory
• Thematic Analysis

Popular searches

• How to Get Participants For Your Study
• How to Do Segmentation?
• Conjoint Preference Share Simulator
• MaxDiff Analysis
• Likert Scales
• Reliability & Validity

Request consultation

Do you need support in running a pricing or product study? We can help you with agile consumer research and conjoint analysis.

Looking for an online survey platform?

Conjointly offers a great survey tool with multiple question types, randomisation blocks, and multilingual support. The Basic tier is always free.

Catherine Chipeta

Monthly newsletter.

Get the latest updates about market research, automated tools, product testing and pricing techniques.

Bias in market research is inevitable, but you can minimise its effects on your study. Learn the common types of survey bias and how to avoid them.

Bias is an unavoidable occurrence in market research ( Suchman 1962 ) but researchers can limit its effects on survey results by understanding what causes it and taking proactive measures. By implementing effective survey design and ensuring questions are both well written and well-formatted , researchers can be more confident that respondents’ answers are more accurate and autonomous ( Summers and Hammonds 1969 ).

In this article, we explore some of the common types of bias in survey research, and how to minimise their impact on survey results.

Acquiescence bias or ‘Yes’ bias

Acquiescence bias AKA friendliness bias occurs when a respondent feels inclined to answer a question in a positive or agreeable way. This can happen when the respondent selects options which seem “right”.

For example, in a customer satisfaction survey , the respondent may select “Very satisfied” because it is the most positive option and pleasing to the researcher. Acquiescence bias can also occur if respondents are fatigued and begin to answer questions with minimal thought. It is more prevalent in Asian cultures than in Western countries and varies by other characteristics of respondent groups ( Johnson et al. 2005 ).

To minimise acquiescence bias, the researcher should review and adjust any questions which might elicit a favourable answer including binary response formats such as “Yes/No”, “True/False”, and “Agree/Disagree”. The dual negative-positive scale helps avoid this bias, making results more comparable across countries and subgroups.

Social desirability bias

Social desirability bias is caused by respondents who choose answers based on what they think is socially acceptable ( Lindsay and Roxas 2011 ). This results in answers which either depict a higher number of “desirable” responses or a lower number of “undesirable” responses.

Questions about topics such as health, income, politics, and religion tend to be affected by social bias. For example, respondents might answer the question “How often do you drink alcohol?” with a lower frequency than is actually true. To reduce this form of bias, researchers should anonymise respondents and assure confidentiality, whilst using neutral and non suggestive question wording.

Habituation bias

Respondents can be affected by habituation bias when questions are repetitive or phrased similarly. This lowers assertiveness, causing respondents to answer questions based on similar questions they have previously answered.

For example, a survey that uses the question “On a scale of 1-5, how likely would you buy this product”, is likely to suffer from habituation bias. To prevent this, researchers must differentiate question wording, and use an engaging tone to keep respondents alert .

Confirmation bias

Confirmation bias is characterised as the human tendency to seek out information that supports their pre-existing beliefs or opinions and ignore information that does not. It affects researchers during the analysis process as they may use respondent data to validate their original hypothesis but ignore any data that contradicts it.

To reduce the effects of confirmation bias, researchers must remain open-minded and consider all data when evaluating existing hypotheses whilst acknowledging it could be disproven during analysis.

Extreme response bias

Extreme response bias occurs when respondents answer a question in the extreme way, regardless of if it reflects their actual views. It is usually the result of another underlying form of bias.

For example, if a respondent is affected by acquiescence bias and selects the most “positive” answers during a customer satisfaction survey, this is also an instance of extreme response bias. Habituation bias from habit can also cause extreme response bias as respondents may choose the lowest or highest option as a habit or due to fatigue.

Researchers should ensure there is question variation, unsuggestive phrasing, and respondent anonymity to help reduce extreme response bias.

Non-response bias

Non-response bias aka participation bias occurs when potential respondents do not participate in or complete a survey ( Shultz and Luloff 2009 ). This can happen for a number of reasons such as respondent fatigue, privacy concerns, complex survey design, poor question wording, or if a survey is irrelevant to the respondent.

For example, if a survey asks too many open response questions, respondents may get overwhelmed and opt not to complete it. To avoid non-response bias, researchers should keep surveys as simple as possible, with clear wording and instruction and seek respondents who are relevant to the survey topic.

Need high quality responses?

We can source the right participants to minimise potential bias in your research.

Read these articles next:

How to market a new product.

Here, you'll find three distinct methods and tips for marketing a new product that will help you adequately prepare for product launch and fully realise the potential of your hard work.

4 Ways Behavioural Segmentation Aids Marketing

Behavioural segmentation helps you understand how consumers make purchasing decisions. Learn how it benefits your product development and marketing efforts.

Do Russians really trust Putin? And can you trust WCIOM?

We conducted our own online poll amongst 372 panellists to validate WCIOM’s claims about the levels of support for Russian politicians.

Which one are you?

I am new to conjointly, i am already using conjointly, cookie consent.

Conjointly uses essential cookies to make our site work. We also use additional cookies in order to understand the usage of the site, gather audience analytics, and for remarketing purposes.

For more information on Conjointly's use of cookies, please read our Cookie Policy .

Do you want to be updated on new features from Conjointly?

We send occasional emails to keep our users informed about new developments on Conjointly , such as new types of analysis and features.

Subscribe to updates from Conjointly

You can always unsubscribe later. Your email will not be shared with other companies.

Qualitative vs. Quantitative Data: 7 Key Differences

Qualitative data is information you can describe with words rather than numbers. 

Quantitative data is information represented in a measurable way using numbers. 

One type of data isn’t better than the other. 

To conduct thorough research, you need both. But knowing the difference between them is important if you want to harness the full power of both qualitative and quantitative data. 

In this post, we’ll explore seven key differences between these two types of data. 

#1. The Type of Data

The single biggest difference between quantitative and qualitative data is that one deals with numbers, and the other deals with concepts and ideas. 

The words “qualitative” and “quantitative” are really similar, which can make it hard to keep track of which one is which. I like to think of them this way: 

• Quantitative = quantity = numbers-related data
• Qualitative = quality = descriptive data

Qualitative data—the descriptive one—usually involves written or spoken words, images, or even objects. It’s collected in all sorts of ways: video recordings, interviews, open-ended survey responses, and field notes, for example. 

I like how researcher James W. Crick defines qualitative research in a 2021 issue of the Journal of Strategic Marketing : “Qualitative research is designed to generate in-depth and subjective findings to build theory.”

In other words, qualitative research helps you learn more about a topic—usually from a primary, or firsthand, source—so you can form ideas about what it means. This type of data is often rich in detail, and its interpretation can vary depending on who’s analyzing it. 

Here’s what I mean: if you ask five different people to observe how 60 kittens behave when presented with a hamster wheel, you’ll get five different versions of the same event. 

Quantitative data, on the other hand, is all about numbers and statistics. There’s no wiggle room when it comes to interpretation. In our kitten scenario, quantitative data might show us that of the 60 kittens presented with a hamster wheel, 40 pawed at it, 5 jumped inside and started spinning, and 15 ignored it completely.

There’s no ifs, ands, or buts about the numbers. They just are. 

#2. When to Use Each Type of Data

You should use both quantitative and quantitative data to make decisions for your business. 

Quantitative data helps you get to the what . Qualitative data unearths the why .

Quantitative data collects surface information, like numbers. Qualitative data dives deep beneath these same numbers and fleshes out the nuances there. 

Research projects can often benefit from both types of data, which is why you’ll see the term “mixed-method” research in peer-reviewed journals. The term “mixed-method” refers to using both quantitative and qualitative methods in a study. 

So, maybe you’re diving into original research. Or maybe you’re looking at other peoples’ studies to make an important business decision. In either case, you can use both quantitative and qualitative data to guide you.

Imagine you want to start a company that makes hamster wheels for cats. You run that kitten experiment, only to learn that most kittens aren’t all that interested in the hamster wheel. That’s what your quantitative data seems to say. Of the 60 kittens who participated in the study, only 5 hopped into the wheel. 

But 40 of the kittens pawed at the wheel. According to your quantitative data, these 40 kittens touched the wheel but did not get inside. 

This is where your qualitative data comes into play. Why did these 40 kittens touch the wheel but stop exploring it? You turn to the researchers’ observations. Since there were five different researchers, you have five sets of detailed notes to study. 

From these observations, you learn that many of the kittens seemed frightened when the wheel moved after they pawed it. They grew suspicious of the structure, meowing and circling it, agitated.

One researcher noted that the kittens seemed desperate to enjoy the wheel, but they didn’t seem to feel it was safe. 

So your idea isn’t a flop, exactly. 

It just needs tweaking. 

According to your quantitative data, 75% of the kittens studied either touched or actively participated in the hamster wheel. Your qualitative data suggests more kittens would have jumped into the wheel if it hadn’t moved so easily when they pawed at it. 

You decide to make your kitten wheel sturdier and try the whole test again with a new set of kittens. Hopefully, this time a higher percentage of your feline participants will hop in and enjoy the fun. 

This is a very simplistic and fictional example of how a mixed-method approach can help you make important choices for your business. 

#3. Data You Have Access To

When you can swing it, you should look at both qualitative and quantitative data before you make any big decisions. 

But this is where we come to another big difference between quantitative vs. qualitative data: it’s a lot easier to source qualitative data than quantitative data. 

Why? Because it’s easy to run a survey, host a focus group, or conduct a round of interviews. All you have to do is hop on SurveyMonkey or Zoom and you’re on your way to gathering original qualitative data. 

And yes, you can get some quantitative data here. If you run a survey and 45 customers respond, you can collect demographic data and yes/no answers for that pool of 45 respondents.

But this is a relatively small sample size. (More on why this matters in a moment.) 

To tell you anything meaningful, quantitative data must achieve statistical significance. 

If it’s been a while since your college statistics class, here’s a refresh: statistical significance is a measuring stick. It tells you whether the results you get are due to a specific cause or if they can be attributed to random chance. 

To achieve statistical significance in a study, you have to be really careful to set the study up the right way and with a meaningful sample size.

This doesn’t mean it’s impossible to get quantitative data. But unless you have someone on your team who knows all about null hypotheses and p-values and statistical analysis, you might need to outsource quantitative research. 

Plenty of businesses do this, but it’s pricey. 

When you’re just starting out or you’re strapped for cash, qualitative data can get you valuable information—quickly and without gouging your wallet. 

#4. Big vs. Small Sample Size

Another reason qualitative data is more accessible? It requires a smaller sample size to achieve meaningful results. 

Even one person’s perspective brings value to a research project—ever heard of a case study?

The sweet spot depends on the purpose of the study, but for qualitative market research, somewhere between 10-40 respondents is a good number. 

Any more than that and you risk reaching saturation. That’s when you keep getting results that echo each other and add nothing new to the research.

Quantitative data needs enough respondents to reach statistical significance without veering into saturation territory. 

The ideal sample size number is usually higher than it is for qualitative data. But as with qualitative data, there’s no single, magic number. It all depends on statistical values like confidence level, population size, and margin of error.

Because it often requires a larger sample size, quantitative research can be more difficult for the average person to do on their own. 

#5. Methods of Analysis

Running a study is just the first part of conducting qualitative and quantitative research. 

After you’ve collected data, you have to study it. Find themes, patterns, consistencies, inconsistencies. Interpret and organize the numbers or survey responses or interview recordings. Tidy it all up into something you can draw conclusions from and apply to various situations. 

This is called data analysis, and it’s done in completely different ways for qualitative vs. quantitative data. 

For qualitative data, analysis includes: 

• Data prep: Make all your qualitative data easy to access and read. This could mean organizing survey results by date, or transcribing interviews, or putting photographs into a slideshow format. 
• Coding: No, not that kind. Think color coding, like you did for your notes in school. Assign colors or codes to specific attributes that make sense for your study—green for positive emotions, for instance, and red for angry emotions. Then code each of your responses. 
• Thematic analysis: Organize your codes into themes and sub-themes, looking for the meaning—and relationships—within each one. 
• Content analysis: Quantify the number of times certain words or concepts appear in your data. If this sounds suspiciously like quantitative research to you, it is. Sort of. It’s looking at qualitative data with a quantitative eye to identify any recurring themes or patterns. 
• Narrative analysis: Look for similar stories and experiences and group them together. Study them and draw inferences from what they say.
• Interpret and document: As you organize and analyze your qualitative data, decide what the findings mean for you and your project.

You can often do qualitative data analysis manually or with tools like NVivo and ATLAS.ti. These tools help you organize, code, and analyze your subjective qualitative data. 

Quantitative data analysis is a lot less subjective. Here’s how it generally goes: 

• Data cleaning: Remove all inconsistencies and inaccuracies from your data. Check for duplicates, incorrect formatting (mistakenly writing a 1.00 value as 10.1, for example), and incomplete numbers. 
• Summarize data with descriptive statistics: Use mean, median, mode, range, and standard deviation to summarize your data. 
• Interpret the data with inferential statistics: This is where it gets more complicated. Instead of simply summarizing stats, you’ll now use complicated mathematical and statistical formulas and tests—t-tests, chi-square tests, analysis of variance (ANOVA), and correlation, for starters—to assign meaning to your data. 

Researchers generally use sophisticated data analysis tools like RapidMiner and Tableau to help them do this work. 

#6. Flexibility 

Quantitative research tends to be less flexible than qualitative research. It relies on structured data collection methods, which researchers must set up well before the study begins.

This rigid structure is part of what makes quantitative data so reliable. But the downside here is that once you start the study, it’s hard to change anything without negatively affecting the results. If something unexpected comes up—or if new questions arise—researchers can’t easily change the scope of the study. 

Qualitative research is a lot more flexible. This is why qualitative data can go deeper than quantitative data. If you’re interviewing someone and an interesting, unexpected topic comes up, you can immediately explore it.

Other qualitative research methods offer flexibility, too. Most big survey software brands allow you to build flexible surveys using branching and skip logic. These features let you customize which questions respondents see based on the answers they give.  

This flexibility is unheard of in quantitative research. But even though it’s as flexible as an Olympic gymnast, qualitative data can be less reliable—and harder to validate. 

#7. Reliability and Validity

Quantitative data is more reliable than qualitative data. Numbers can’t be massaged to fit a certain bias. If you replicate the study—in other words, run the exact same quantitative study two or more times—you should get nearly identical results each time. The same goes if another set of researchers runs the same study using the same methods.

This is what gives quantitative data that reliability factor. 

There are a few key benefits here. First, reliable data means you can confidently make generalizations that apply to a larger population. It also means the data is valid and accurately measures whatever it is you’re trying to measure. 

And finally, reliable data is trustworthy. Big industries like healthcare, marketing, and education frequently use quantitative data to make life-or-death decisions. The more reliable and trustworthy the data, the more confident these decision-makers can be when it’s time to make critical choices. 

Unlike quantitative data, qualitative data isn’t overtly reliable. It’s not easy to replicate. If you send out the same qualitative survey on two separate occasions, you’ll get a new mix of responses. Your interpretations of the data might look different, too. 

There’s still incredible value in qualitative data, of course—and there are ways to make sure the data is valid. These include: 

• Member checking: Circling back with survey, interview, or focus group respondents to make sure you accurately summarized and interpreted their feedback. 
• Triangulation: Using multiple data sources, methods, or researchers to cross-check and corroborate findings.
• Peer debriefing: Showing the data to peers—other researchers—so they can review the research process and its findings and provide feedback on both. 

Whether you’re dealing with qualitative or quantitative data, transparency, accuracy, and validity are crucial. Focus on sourcing (or conducting) quantitative research that’s easy to replicate and qualitative research that’s been peer-reviewed.

With rock-solid data like this, you can make critical business decisions with confidence.

Make your website better. Instantly.

Keep reading about user experience.

Qualitative data is information you can describe with words rather than numbers.  Quantitative data is information represented in a measurable way using numbers.  One type…

6 Real Ways AI Has Improved the User Experience

It seems like every other company is bragging about their AI-enhanced user experiences. Consumers and the UX professionals responsible for designing great user experiences are…

12 Key UX Metrics: What They Mean + How To Calculate Each

UX metrics help identify where users struggle when using an app or website and where they are successful. The data collected helps designers, developers, and…

5 Key Principles Of Good Website Usability

Ease of use is a common expectation for a site to be considered well designed. Over the past few years, we have been used to…

20 Ways to Speed Up Your Website and Improve Conversion in 2024

Think that speeding up your website isn’t important? Big mistake. A one-second delay in page load time yields: Your site taking a few extra seconds to…

How to Do Usability Testing Right

User experience is one of the most important aspects of having a successful website, app, piece of software, or any other product that you’ve built. …

Website Navigation: Tips, Examples and Best Practices

Your website’s navigation structure has a huge impact on conversions, sales, and bounce rates. If visitors can’t figure out where to find what they want,…

How to Create a Heatmap Online for Free in Less Than 15 Minutes

A heatmap is an extremely valuable tool for anyone with a website. Heatmaps are a visual representation of crucial website data. With just a simple…

Website Analysis: Our 4-Step Process Plus the Tools and Examples You Need to Get Started

The problem with a lot of the content that covers website analysis is that the term “website analysis” can refer to a lot of different things—and…

How to Improve My Website: Grow Engagement and Conversions by Fixing 3 Common Website Problems

Here, we show you how to use Google Analytics together with Crazy Egg’s heatmap reports to easily identify and fix 3 common website problems.

Comprehensive Guide to Website Usability Testing (With Tools and Software to Help)

We share the 3-step process for the website usability testing we recommend to our customers, plus the tools to pull actionable insights out of the process.

Hotjar Alternatives: 21 Tools for Optimizing Your Website and Conversion Rate

Hotjar is a great tool for website optimization, but some marketers may need something a little different. If that’s you, here are 21 Hotjar alternatives.

How to Interpret and Use Clickmaps to Improve Your Website’s UX

To use the info in a clickmap to improve your website experience and effectiveness, you have to know how to interpret what you see.

User Experience Design: 6 Simple Steps for Developing Your UX Design Process

User Experience Design or UX design is the process for improving the satisfaction of your website visitors by making your site more usable, accessible, and…

The Best Way to Increase User Engagement on Your Website

When setting out to create a successful website, you should always be mindful of user engagement. But what does that mean, exactly? When visitors land…

Over 300,000 websites use Crazy Egg to improve what's working, fix what isn't and test new ideas.

Last Updated on January 15, 2019

• Open access
• Published: 29 July 2024

Factors associated with health CEO turnover - a scoping review

• Nebu Varughese Mathew 1 ,
• Chaojie Liu 1 &
• Hanan Khalil 1  

BMC Health Services Research volume  24 , Article number:  861 ( 2024 ) Cite this article

66 Accesses

Metrics details

Chief Executive Officer’s (CEO) are integral leaders of health care organisation. Over the last two decades in United States (US) hospitals, it has been noted that CEO turnover rates are increasing, and it was reported that the CEO turnover rates have augmented from 14% in 2008 to 18% in 2017 in the private sector. In Australia, it was discovered that during two years, 41 executives had 18 distinct positions. It has been highlighted that the increasing CEO turnover is a major issue for Australian and international health care organisations. Some of the negative consequences of CEO turnover include organisational instability, high financial costs, loss of human capital and adverse effects on staff morale and patient care.

Our scoping review aimed to map and summarise the evidence associated with CEO turnovers in both health and non-health setting, and answer the following questions: 1. What are the reasons for CEO departure?, 2. What are the strategies to minimise CEO turnover?

A protocol explaining the objectives, inclusion criteria and methods for this scoping review were specified in advance and published. This scoping review included 17 studies (13 health and 4 non-health setting) published over a 31-year period that investigated and described the increasing CEO turnover rates. All 17 studies identified causes of CEO turnover along with certain studies identifying facilitators of CEO retention. We classified CEO’s departure reasons into three major themes: organizational, performance, and personal. Organisational factors include CEO departures due to issues within the organisation, performance factors include issues with CEO’s work and personal factors captures personal reasons for CEO’s leaving their job.

CEOs are under immense pressure to deliver good results and drive growth while satisfying the interests of internal and external stakeholders. There are various reasons for CEO’s departure however the most common factor identified is organisational.

Contribution of paper statement

What is already known

Causes and consequences of CEO turnover in both health and non-health settings.

What this paper adds

Three main factors associated with CEO turnover such as organisational, performance and personal factors.

Peer Review reports

A Chief Executive Officer (CEO) is a top rank executive in an organisation who manages daily operations, makes corporate decisions and is the public face of the organisation [ 1 ]. With changing economies, CEOs are finding their job to be increasingly difficult as they must manage tight budgets, model a positive work culture, manage daily operations and follow the Board’s initiatives and strategic plans. The CEO role demands significant time, effort and critical thinking which makes the job one of the busiest roles within health care [ 2 ]. CEOs deal with demanding scenarios like long, difficult work hours, separation from loved ones, severe mental health issues, and battling false online narratives and campaigns by social media [ 3 ]. It was reported that almost a third (29%) of all the CEOs were forced out from their roles due to poor performance, a scandal or a strategic disagreement. Further, 7% of executives were formally dismissed and the remaining 66% quit with no reasons highlighted as such [ 4 , 5 , 6 , 7 ].

It has been highlighted that the increasing CEO turnover is a matter of concern for Australian and international health care organisations [ 8 ]. Deleterious consequences of CEO turnover include organisational unsteadiness, budgetary issues, loss of human resources and adverse effects on staff wellbeing and patient care [ 5 ]. Two thirds of health CEOs claimed that their decision to leave the organisation was solely their own [ 9 ]. Majority of the CEOs perceive their departure as detrimental towards their organisation i.e., community relations, medical staff relations, hospital culture and employee morale get negatively impacted. Interestingly, it was found that CEO’s turnover has a major knock-on effect on their executives. For example, the American College of Healthcare Executives (ACHE) report found that within one year of the CEO’s departure, the chief medical officer and the chief operating officer also changed, 77% and 52% of the time [ 10 ].

Over the years, organisations have realised the importance of succession planning as it enables an organization to sustain itself in their leader’s absence [ 1 ]. Organizations need to take seriously how they develop leaders because a strong leadership team can make a big difference on how health can be advanced for everybody [ 11 ]. In the health setting, it was noted that the average tenure of a CEO was about five years, and just over half (51%) had previously been a CEO at another facility [ 9 ]. In Australia, long-term employment contracts are considered for non-health settings to ensure that CEOs have sufficient time and outlook for decision making. Information on contract length among Australian CEOs is hard to gather, because not all organizations uncover the contract in their yearly report [ 12 ]. Among those Top 100 organizations that do uncover contract length stated that the contracts ranged between 3-and 5-years. Similarly, in the health setting, most CEO contract terms in USA are for three to four years and include provisions for renewal [ 13 ].

Two recent studies conducted in non-health settings have reported various reasons behind the turnover such as death, illness, dismissed for job performance, terminated for behavioural or policy-related problems, voluntary retirement, to pursue a new venture and departure following a merger or acquisition [ 13 , 14 ]. However, in the health setting CEOs face different issues and challenges [ 5 ] hence may have various reasons to leave an organisation. The factors influencing CEO turnover in the healthcare sector differ significantly from those in non-health setting due to the industry’s unique organizational structures and compensation models. Healthcare organizations, whether for-profit or not-for-profit, operate within a complex regulatory environment and face distinct financial pressures related to reimbursement and healthcare policies [ 15 ]. Moreover, compensation structures for healthcare CEOs often include performance metrics related to quality of care and patient outcomes, which may not be as prominent in other industries [ 16 ]. These differences in setting underscore the importance of considering industry-specific factors when interpreting results on CEO turnover in healthcare, as they reflect the intricacies of managing healthcare delivery amidst regulatory, financial, and mission-driven demands. To date, no scoping reviews were published to identify the evidence associated with CEOs turnover. Detailing this evidence will clarify the gaps in this area and enable further research therefore, this scoping review aims to examine the evidence associated with CEO turnovers in both the health and non-health setting. Barriers and facilitators for CEOs retention were also identified. These finding will immensely help the health care industry in retaining their CEOs leading to lesser disruption in health services resulting in quality patient care.

The scoping review methodology followed the guidance published by Peters et al. 2020 [ 17 ].

Inclusion criteria

Types of participants.

Studies with CEO turnover in health and non-health settings are included. For the purpose of this study, CEOs were defined as the highest ranked officer within a registered corporate entity, such as a company or non-profit organisation. They usually report to a board of directors [ 18 ].

The factors or causes of CEO departure, barriers and facilitators contributing to CEO retention.

The context of this review included both the health and non-health setting for comparative purposes. Examples of these included hospitals and corporate organisations.

Types of studies

Health care and non-health CEO turnover rates nationally and internationally were searched. Both quantitative and qualitative study designs were included in this scoping review. We also included systematic reviews to explore turnover and retention rates of CEOs by pearling of the reference list. Organisational reports through manual search, non-peer review literature was also included in the review. A protocol detailing the methodology for the current review was registered in Open science framework; Registration DOI- https://doi.org/10.17605/OSF.IO/8UXKH .

Search strategy

In consultation with a librarian, a three-step search strategy was utilised in this review. An initial limited search of Proquest (ABI/Inform), Medline (Ovid), CINAHL (Ebsco), Cochrane Library was undertaken which was followed by examining the text words contained in the title, abstract and of the index terms used to explain the article. Several studies were identified and deemed to be relevant to the review. A second search was conducted by identifying the keywords and index terms retrieved from our initial search was undertaken across all included databases. The below given databases were searched on 04 July 2021 using the support of a librarian: Proquest (select suitable relevant databases – e.g. ABI/Inform), Medline (Ovid), Pubmed, CINAHL (Ebsco), Cochrane Library, Informit databases, Scopus and Google Scholar. The above data bases were searched as they were anticipated to capture the relevant studies. In addition to the above, databases such as Web of Science, Business Source Ultimate and Emerald Insight were searched to identify the studies in non-health settings. All the above databases were explored during the final search. The search strategy of all the databases followed the same strategy mentioned in Appendix I . The reference lists of all identified reports and articles were searched for additional studies. We included literature in English in our review because of limited resources. Studies published from 1990 onward until 04 July 2021, were included as the CEO turnover issue has been identified since the changes in governmental funding mechanism for medical care [ 19 ]. Funding system has evolved over the past three decades, which has imposed a significant impact on the way that hospitals and other health organisations are managed [ 20 ]. The world economy and business models have experienced substantial transformation between 1990 and 2021, marked by the rise of globalization, the proliferation of digital technologies, and the emergence of new economic powers. These changes have reshaped industries, altered consumer behaviour, and necessitated adaptation to new market dynamics [ 20 ]. The following keywords were used to identify health studies: CEO, Chief executive officer, Vice-president, Top executive, Turnover, Replacement, Retention, Churn, Succession, Cause, Factors, Succession policy/policies, Health care, Hospital/hospitals. Appendix 1 details the search strategy used for health studies. A similar search was used for non-health studies by adding key terms such as firms, company, business, industry, organisation, enterprise and corporation.

Exclusion criteria

Studies not identifying causes of CEO turnover were excluded. Publications in a language other than English were also excluded due to limited resources.

Data extraction

To address the review question, pertinent data were extracted from the included studies. To minimise the risk of bias and promote transparency, we registered and published the protocol before starting the review. Moreover, extraction was conducted by two authors to minimise the risk of errors and bias. When disagreement arose during data extraction conducted between authors, it was typically resolved through discussion, consultation with another senior researcher, or referral to established criteria, ensuring transparency and accuracy in the extraction process. The data extracted included the following: CEO turnover rates, causes, barriers, facilitators, authors, date of publication, country where study was originally conducted, aims/purpose, study population, methodology/methods, context, details, and key findings and outcome measures that relate to the scoping review question. The extracted data were represented in a logical format and three step coding process was followed to extract high level themes from the literature where the data was categorised into personal, organisational and performance related factors behind a CEO’s departure to enable a detailed map of the evidence [ 21 ].

The database searches yielded a total of 138 citations (122 non-health and 16 health studies) after duplicates were removed and an additional two citations were found via hand searching (health studies). The titles and abstracts for these 140 citations were screened, with 86 papers excluded. The remaining 54 citations were considered for further detailed assessment of the full paper, and 37 were excluded due to having irrelevant aims (12 studies), study population (i.e., not relating to CEOs) (10 studies), not identifying cause behind CEO turnover (8 studies), and describing only facilitators of CEO turnover (7 studies). The search yielded a total of 17 citations for inclusion in this review i.e., 13 health and 4 non-health studies were included. A protocol detailing the methodology for the current review was followed. A flowchart showing the number of citations at each stage is detailed in Fig.  1 : PRISMA flowchart of study selection and inclusion process.

Prisma flowchart of study selection and inclusion process [ 22 ]

Date & country where study was origihnally published

Table  1 illustrates studies that were published between 1990 and 2021. All the studies described in the review were undertaken in developed countries. Of the studies, 94% were conducted in the USA, and 6% in Australia. There were no studies found in non-Western countries.

Aim/Purpose

Most of the health studies (69%) included in this review aimed to investigate causes for the increasing turnover of CEOs with two studies reporting on turnover in rural hospital in USA [ 25 , 26 ]. Only one study was identified from Australia that reported on health care executive turnover including CEOs. Two studies measured the organisational impacts of CEO departure [ 5 ]. Interestingly, a study explored the career trajectories of 4 CEOs i.e., what they are doing after leaving their CEO roles.

Study population

One of the studies which was conducted in Australia focused on health care executive’s turnover in general. The health care executives are CEO, Chief Nursing officers (CNO), chief finance officer (CFO) etc. [ 5 ]. Other included studies focused specifically on CEO population. The population size for the included studies ranged from 4 to 2711 participants [ 26 , 29 ].

Study types

The types of studies ranged from Quantitative (23.53%), Qualitative research (11.76%) and report reviews (64.71%). Four of the health studies have used quantitative research to answer their research question [ 5 , 9 , 25 , 27 ]. Two studies used a qualitative approach via conducting interviews [ 29 , 30 ] to explore CEO’s perspective on roles, responsibilities and challenges of being a CEO therefore identified certain factors behind their turnover. The remaining studies reviewed the available data on CEO turnover from a hospital or health care college/organizations. On the other hand, one of the non-health studies have also used a qualitative approach to investigate the turnover rates [ 30 ]. The remaining three non-health studies reviewed the available data across various corporate organisations.

CEO turnover rates

In Australia, 16 executives left seven positions in 24 months [ 5 ] whereas the USA is experiencing similar issues and turnover rates is expediting with average CEO tenure of 3 years. Non-health industry reports similar trajectory of CEO tenure of 4 years [ 14 , 30 , 31 ].

Health studies have identified certain factors behind turnover which ranged from lack of career advancement opportunities [ 4 , 5 ], lack of remuneration [ 9 ], pressure from board [ 13 ], lack of engagement of teams within the organisation [ 14 ], job being demanding and stressful [ 4 ], poor financial performance [ 14 ], low occupancy in hospital [ 23 ], lack of leadership support programs [ 28 ], inadequate salary and desire for better positions [ 27 ].

Non-health studies highlighted some causes for their turnover such as poor financial performance [ 14 ], pressure from broad [ 30 ] and high job demands [ 32 ].

Figure  2 details the classification of each of the studies. The studies ranged from having only one to all three of the components being organizational, performance and personal factors. The organisational component is aimed at focusing on organizational related causes due to which CEO leaves such as under-resourced hospitals, policy issues, merger and acquisitions. Performance components are aimed at the departure causes that are related to CEO’s unsatisfactory performance such as being terminated by the Board. Personal factors focus on CEO’s personal reasons behind resigning from their job such as family commitments, desiring a career change etc.

Classification of included health and non-health studies based on causes

Sixteen studies have reported that CEOs depart due to organisational factors. Some of the reason’s identified behind their departure were poor financial performance, under-resourced services, low bed occupancy etc. [ 9 , 14 , 23 ]. CEOs have significant pressure in relation to finance and resource management, particularly if a hospital is under-resourced and has low bed occupancy rate which places more pressure on CEOs [ 8 ].

The second most common reason identified by the review behind their departure is due to their own personal reasons such as desiring for a career change, retirement, illness etc. [ 4 , 14 , 23 , 24 ]. It is important to note that CEOs may desire for a career change or be willing to step down from their role for family commitments due to an increase in work demands [ 27 , 29 ].

Six of the studies shows that CEOs contracts get terminated by the board due to their poor performance in not being able to meet the target goals [ 14 , 25 , 26 , 27 , 31 ]. Some other reasons include lack of leadership, conflict with board members or medical staff, not aligning the goals with organisational vision, ethical misconduct etc. [ 14 , 26 ].

Facilitators

Health studies have identified that certain strategies may help minimise the turnover such as supporting CEOs by exposing them to leadership development programs to enhance team engagement [ 4 , 11 ]. Exposure to leadership programs will help them to reflect upon their own leadership style and learn new skills and knowledge in terms of managing their organisation/team. Moreover, CEOs get some time to detach themselves from the busy environment to meet with other CEOs/leaders from other organisation and share experiences [ 4 ]. The review suggests that developing good relationships with the board and its members may help in retaining a CEO as he/she may feel well supported [ 25 , 26 , 27 , 28 , 29 ]. Boards plays a vital role in supporting their CEOs as they are expected to intervene at the government level, where CEOs may not have the power or authority to do this [ 27 ]. It is also imperative for the CEO to have a good relationship with board chair so that the CEO can discuss issues concerning their organisation and get support as they need [ 28 , 29 ]. Other factors to minimise turnover were variety in the job, good pay, good fringe benefits and the opportunity for professional development. CEOs also appreciate variety in their role where they get the opportunity to act in another CEO related role. They also feel valued if their efforts or achievements have been recognised and acknowledged by receiving incentives or fringe benefits [ 27 ].

The current scoping review consists of 13 health and 4 non-health related studies published over 31-year period that investigated and described the increasing CEO turnover rates. This review identified multiple factors associated with CEOs turnover in both health and non-health settings which is reported by the organisation. These included organisational, performance and personal.

A range of outcomes were measured in this review; they were the rate of turnover, causes behind turnover and consequences of turnover. Only one study identified the career trajectories of Ex-CEO’s. Regardless of health or non-health background, most of the studies have suggested the need for having a succession plan in place within their organisation [ 4 , 5 , 9 , 23 , 24 , 25 ]. Researchers reports that an organisation performance suffers less from a CEO departure if they give other executive directors an opportunity to act in the CEO’s role when they are on leave or seconded to manage another organisation. Therefore, sudden CEO’s departure will be less impactful for the organisation as one of their own senior leaders will backfill the position until the recruitment process finishes, and a new CEO takes over [ 4 ].

The recruitment process in hospitals for CEOs varies, but many institutions are actively seeking capable and skilled leaders who possess a blend of healthcare expertise, strategic vision, and managerial acumen. However, challenges such as competitive market demands, complex regulatory environments, and the evolving landscape of healthcare delivery can impact the attractiveness of these positions to top-tier candidates [ 15 , 24 ]. In Australia, long term contracts are considered for CEO’s in non-health settings however the downside to these contracts is a large termination pay-out which could be salary for the entire remaining term of the contract. Nonetheless, a much bigger cost which is mostly overlooked is the implication for all other layers of management within the organisation [ 12 ]. While long term contracts relieve the issue of CEO’s short-term decision making, it limits the prospects of senior executives aspiring for a CEO role thus, they end up leaving the organisation [ 12 ]. This review has identified similar trajectory of CEO turnover in both health and non-health settings which is due to similar reasons such as unsatisfactory CEO performance and personal factors such as death and illness. It has been noted that CEO’s left mainly due to performance and personal factors within non-health setting when compared to organisational factors in the health setting. Apart from that, both settings have highlighted the importance of the board in CEO retention. Other factors such as investing in leadership programs and providing incentives to the CEO are some of the measures that may help in CEO retention. Moreover, having some realistic and clear key performance indicators (KPIs) will be a facilitator for CEO retention [ 14 ]. One of the reasons why CEO’s may fail to perform well in their role is that the board may not have set clear KPIs for them to achieve. Therefore, if not addressed, this may result in poor performance [ 31 ]. Research shows that boards of directors play a significant role in CEO retention by implementing effective governance practices and strategic oversight, thereby influencing CEO turnover rates [ 35 ]. By adopting measures such as succession planning, performance evaluations, and incentivizing long-term organizational goals, boards can mitigate CEO turnover and enhance stability within the executive leadership [ 36 ]. Hospital CEO’s come from a wide variety of backgrounds and college education. Generally, Licensing and certifications are not required however having clinical knowledge and commitment to the profession can be helpful in advancing toward becoming a hospital CEO [ 33 , 34 ]. One of the retired hospital CEO stated that starting his career as a clinician has immensely helped in gaining the respect that he developed while working alongside other medical professionals which was critical to his success as CEO [ 33 ].

Further research is required to identify CEOs turnover and which interventions are likely to increase their retentions. Developing a retention model for this group of employees has the potential to enhance both recruitment and is likely to stabilise the turnover rates for this population.

Limitations of this review

The results described here are difficult to compare as both the health and non-health settings and structure of organisations are quite different. Additional limitation to the review was the language restriction as we only included studies published in the English Language as well as the how old some of the included studies were, spanning over a 31-year period. This review does not include an assessment of the quality of evidence, in keeping with a scoping review.

This review has identified multiple causes for the CEOs departure which has been classified under the following components as: Organisational factors, performance factors and personal factors. More studies are needed to explore the reasons behind CEO’s departure so that retention strategies can be put in place to support CEOs at workplace. To minimize CEO turnover in hospitals, strategies includes fostering a supportive organizational culture that values transparency, employee development, and work-life balance, while also implementing robust succession planning programs to ensure smooth transitions and continuity in leadership.

Data availability

The authors confirm that the data supporting the findings of this study are available within the article [and/or] its supplementary materials.

Wilson CN, Stranahan H. Organizational characteristics associated with hospital CEO turnover. J Healthc Manag. 2000;45(6):395–404.

CAS   PubMed   Google Scholar  

Zimring C, Augenbroe GL, Malone EB, Sadler BL. Implementing healthcare excellence: the vital role of the CEO in evidence-based design. HERD. 2008 Spring;1(3):7–21.

Jha MK. An Integrated Framework of Leadership for Healthcare Organizations to Navigate through COVID-19 Crisis. Asia Pac J Health Manage. 2021;16(3):16–20.

Google Scholar  

American College of health care Executive.Hospital CEO Turnover Rate. https://www.ache.org/learning-center/research/about-ceos/hospital-ceo-turnover/hospital-ceo-turnover-rate-2017;Ache.org . 2020.

Duffield C, Roche M, Blay N, Thoms D, Stasa H. The consequences of executive turnover. J Res Nurs. 2011;16(6):503–14.

Article   Google Scholar  

Ricketts LCEO, Turnover Hits A, Record High. 2020; Available at: < https://www.fnlondon.com/articles/ceo-churn-hits-a-high-as-pandemic-spooked-investors-seek-level-heads-20200728 .

Stafford P. CEO turnover soars during downturn, survey shows. Smart Co. 2009.

Corey M. High Turnover of Hospital CEOs: A New Risk Factor for Health Systems and Bond Investors. Phillipsdipisa.com. 2013. https://phillipsdipisa.com/wp%20content/uploads/2013/07/Corey_CEOTurnover_Final_PDAwebsite1.pdf .

Khaliq AA, Walston SL, Thompson DM. Is chief executive officer turnover good for the hospital? Health Care Manag (Frederick). 2007;26(4):341–6.

Article   PubMed   Google Scholar  

Mallory H. Hospital CEO turnover rate dipped in 2019 for the first time in five years. Healthc Finance News. https://www.healthcarefinancenews.com/news/hospital-ceo-turnover . 2020, September 16.

Hearld LR, Opoku-Agyeman W, Kim DH, Landry AYCEO, Turnover Among. U.S. Acute Care hospitals, 2006–2015. J Healthc Manag, 64(1), 28–42. https://doi.org/10.1097/jhm-d-18-00019 .

Chaudhri V. CEO Contracts and Succession [Internet]., Melbourne. ISS PROXY AUSTRALIA; 2005 p. 5–12. https://www.pc.gov.au/inquiries/completed/executive-remuneration/submissions/sub071-attachment3.pdf .

5 Basic Features of Today’s Hospital Executive Contracts. Becker’s Hospital Review [Internet]. 2012 [cited 27 November 2021];. https://www.beckershospitalreview.com/compensation-issues/5-basic-features-of-todays-hospital-executive-contracts.html .

Gentry R, Harrison J, Quigley T, Boivie S. A database of CEO turnover and dismissal in S&P 1500 firms, 2000–2018. Strateg Manag J. 2021;42(5):968–91.

Rosko MD. The relationships among hospital strategic orientation, leadership, and performance in a changing environment: a study of hospitals in different organizational contexts. Health Care Manage Rev. 2014;39(3):252–66.

Pfarrer MD, Pollock TG, Rindova VP. A tale of two assets: the effects of firm reputation and celebrity on earnings surprises and investors’ reactions. Acad Manag J. 2010;53(5):1131–52.

Peters MDJ, Marnie C, Tricco AC, Pollock D, Munn Z, Alexander L, et al. Updated methodological guidance for the conduct of scoping reviews. JBI Evid Synth. 2020;18(10):2119–26.

Peterdy K. CEO - Understanding the Roles and Responsibilities of a CEO [Internet]. Corporate Finance Institute. 2018. https://corporatefinanceinstitute.com/resources/careers/jobs/what-is-a-ceo-chief-executive-officer/ .

Weil P, Timmerberg J, Hospital. CEO turnover: modelling more or less durable places to work. Health Serv Manage Res. 1990;3(3):208–17.

Article   CAS   PubMed   Google Scholar  

Baxter PE, Hewko SJ, Pfaff KA, Cleghorn L, Cunningham BJ, Elston D, et al. Leaders’ experiences and perceptions implementing activity-based funding and pay-for-performance hospital funding models: a systematic review. Health Policy. 2015;119(8):1096–110.

Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol [Internet]. 2006;3(2):77–101. https://doi.org/10.1191/1478088706qp063oa .

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, Moher D, Peters MD, Horsley T, Weeks L, Hempel S, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73. https://doi.org/10.7326/M18-0850 .

Hospital CEO. turnover rises, retirement up (USA). Leadersh Health Serv (Bradf Engl) [Internet]. 2010;23(3). https://doi.org/10.1108/lhs.2010.21123cab.003 .

Commins J. Hospital CEO turnover hits record high. Health Governance Report. 2014;24(5):3. https://www-proquest-com.ez.library.latrobe.edu.au/trade-journals/hospital-ceo-turnover-hits-record-high/docview/1534076864/se-2?accountid=12001 .

Leibert M, Leaming LE. Critical access hospital chief executive officer turnover: implications and challenges for governing boards. Health Care Manag (Frederick). 2010;29(1):22–8.

Hart LG, Robertson DG, Lishner DM, Rosenblatt RA. CEO turnover in rural northwest hospitals. Hosp Health Serv Adm. 1993 Autumn;38(3):353–74.

Weil PA, Kimball PA. A model of voluntary turnover among hospital CEOs. Hosp Health Serv Adm. 1995 Autumn;40(3):362–85.

Dwore RB, Murray BP. A comparison of Utah hospital CEO turnover between 1973–1987 and 1988–1992. Health Care Manage Rev. 1996 Spring;21(2):62–73.

Hudson T. Ex-hospital CEOs. Where are they now? Hosp Health Netw. 1995;69(24):30. 36–8.

Coyne KP, Coyne EJ Sr. Surviving your new CEO. Harv Bus Rev. 2007;85(5):62–9.

PubMed   Google Scholar  

Balsam S, Miharjo S. The effect of equity compensation on voluntary executive turnover. J Account Econ. 2007;43(1):95–119.

Kaplan SN, Minton BA. How has CEO turnover changed? How has CEO turnover changed? Int Rev Fin. 2012;12(1):57–87.

Becoming a hospital CEO —. an overview with real-life examples [Internet]. Dhge.org. 2021 [cited 27 November 2021]. https://dhge.org/about-us/blog/how-to-become-ceo-of-a-hospital .

How to Become a CEO Your Path to Chief Executive Officer. - The Quantic Blog [Internet]. Quantic School of Business and Technology. 2020 [cited 27 November 2021]. https://blog.quantic.edu/2020/08/28/how-to-become-a-ceo-your-path-to-chief-executive-officer/ .

Brown WA, Caylor ML. Corporate governance and firm performance. International Corporate Governance. Emerald Group Publishing Limited; 2016. pp. 107–26.

Chen G, Hambrick DC, Pollock TG. CEO career variety: effects on firm-level strategic and social novelty. Acad Manag J. 2015;58(6):1916–38.

Download references

Acknowledgements

The authors wish to express their gratitude to all who assisted with this study.

Not applicable.

Author information

Authors and affiliations.

La Trobe University, Melbourne, VIC, 3086, Australia

Nebu Varughese Mathew, Chaojie Liu & Hanan Khalil

You can also search for this author in PubMed   Google Scholar

Contributions

We confirm that the manuscript has been read and approved by all named authors and that there are no other persons who satisfied the criteria for authorship but are not listed. We further confirm that the order of authors listed in the manuscript has been approved by all of us.We understand that the Corresponding Author is the sole contact for the Editorial process. He is responsible for communicating with the other authors about progress, submissions of revisions and final approval of proofs. We confirm that we have provided a current, correct email address which is accessible by the Corresponding Author.NM, CL, HKNebu Varughese Mathew (corresponding author)La Trobe University, Melbourne VIC 3086, Australia E: [email protected] LiuLa Trobe University, Melbourne VIC 3086, AustraliaE: [email protected] KhalilLa Trobe University, Melbourne VIC 3086, AustraliaE: [email protected].

Corresponding author

Correspondence to Nebu Varughese Mathew .

Ethics declarations

Ethics approval and consent to participate.

We further confirm that any aspect of the work covered in this manuscript that has involved human has been conducted with the ethical approval of all relevant bodies.

Consent for publication

Not Applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mathew, N.V., Liu, C. & Khalil, H. Factors associated with health CEO turnover - a scoping review. BMC Health Serv Res 24 , 861 (2024). https://doi.org/10.1186/s12913-024-11246-y

Download citation

Received : 08 May 2023

Accepted : 25 June 2024

Published : 29 July 2024

DOI : https://doi.org/10.1186/s12913-024-11246-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Chief executive officer
• Top executive
• Turnover rates
• Replacement

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

• Open access
• Published: 30 July 2024

siqRNA-seq is a spike-in-independent technique for quantitative mapping of mRNA landscape

• Zhenzhen Wang 1   na1 ,
• Kehan Tao 1   na1 ,
• Jiaojiao Ji 1 ,
• Changbin Sun 1   na1 &

BMC Genomics volume  25 , Article number:  743 ( 2024 ) Cite this article

Metrics details

RNA sequencing (RNA-seq) is widely used for gene expression profiling and quantification. Quantitative RNA sequencing usually requires cell counting and spike-in, which is not always applicable to many samples. Here, we present a novel quantitative RNA sequencing method independent of spike-ins or cell counting, named siqRNA-seq, which can be used to quantitatively profile gene expression by utilizing gDNA as an internal control. Single-stranded library preparation used in siqRNA-seq profiles gDNA and cDNA with equal efficiency.

To quantify mRNA expression levels, siqRNA-seq constructs libraries for total nucleic acid to establish a model for expression quantification. Compared to Relative Quantification RNA-seq, siqRNA-seq is technically reliable and reproducible for expression profiling but also can sequence reads from gDNA which can be used as an internal reference for accurate expression quantification. Applying siqRNA-seq to investigate the effects of actinomycin D on gene expression in HEK293T cells, we show the advantages of siqRNA-seq in accurately identifying differentially expressed genes between samples with distinct global mRNA levels. Furthermore, we analyzed factors influencing the downward trend of gene expression regulated by ActD using siqRNA-seq and found that mRNA with m 6 A modification exhibited a faster decay rate compared to mRNA without m 6 A modification. Additionally, applying this technique to the quantitative analysis of seven tumor cell lines revealed a high degree of diversity in total mRNA expression among tumor cell lines.

Conclusions

Collectively, siqRNA-seq is a spike-in independent quantitative RNA sequencing method, which creatively uses gDNA as an internal reference to absolutely quantify gene expression. We consider that siqRNA-seq provides a convenient and versatile method to quantitatively profile the mRNA landscape in various samples.

Peer Review reports

RNA sequencing (RNA-seq) and almost 100 derivatives have revolutionized our understanding of gene expression in various aspects of biology [ 1 ], such as common markers for multiple cancers [ 2 ]. Studies of single-cell omics techniques and recent spatial transcriptomics are increasingly being explored to reshape the current cell-type classification system and preserve spatial information [ 3 , 4 , 5 ]. Nonetheless standard bulk RNA-seq remains a convenient and routine research tool for studies on gene expression, especially for a large number of samples to be sequenced [ 1 ].

For RNA-seq analysis, gene expression is quantified to identify differentially expressed genes (DEGs), to infer regulatory networks, and/or to reveal cellular states and function [ 6 ]. Generally, sequencing reads are mapped to the reference genome after quality control and assigned to each feature to quantify gene expression [ 7 ]. To account for differences in read depth, genes are typically normalized to RPKM (Reads Per Kilobase Million) or FPKM (Fragments Per Kilobase Million) for comparative studies. These normalization methods are largely based on assumptions that most genes are at the same expression level and that the total mRNA levels do not show much difference across samples [ 8 ]. However, when groups with high heterogeneity of gene expression levels are sequenced for comparison, these assumptions fail to hold and sequencing depth-based normalization methods may result in erroneous conclusions [ 9 ], such as cells with high levels of c-Myc showing two to three times more total RNA than their low-Myc counterparts [ 10 , 11 ]. To address such issues, one approach is the use of spike-in control RNAs that are added during library construction [ 12 , 13 ]. Spike-in controls are a useful resource for evaluating the sensitivity and accuracy of RNA-seq experiments for transcriptome discovery and quantification [ 9 ]. Although spike-in controls are commercially available, such as a common set of external RNA controls that has been developed by the External RNA Controls Consortium (ERCC), they are not yet widely adopted [ 1 ].

Here, we developed a spike-in independent quantitative RNA sequencing (siqRNA-seq) method, which uses genomic DNA (gDNA) as an internal reference to normalize mRNA expression levels. By comparison, we showed that the relative gene expression pattern profiled by siqRNA-seq is similar to that profiled by Relative Quantification RNA-seq. We further demonstrated that siqRNA-seq enables us to assess the copy number of mRNA per cell/genome without cell counting, RNA quantification, or spike-ins. Finally, we exemplify the application of siqRNA-seq for differential gene expression analysis on samples with distinct global mRNA levels. Together, siqRNA-seq provides a convenient and versatile method to quantitatively profile the mRNA landscape.

Design of siqRNA-seq

To develop a spike-in-independent method for gene expression quantification, we designed siqRNA-seq that utilized genomic DNA as an internal reference for normalization (Fig.  1 A). As we know, in general, each cell during the G1 phase has a complete set of gDNA, so we considered that the gDNA can be taken as a stable internal reference for normalizing mRNA copy number to “per cell”. For non-interphase cells or other special samples, the copy number of gDNA in each cell is not constant; thus, the “per genome” normalization method is suitable. With that goal, in siqRNA-seq, total nucleic acids were extracted from samples to construct two libraries in parallel, an mRNA library and an mRNA&gDNA library for each sample, as shown in Fig.  1 A. The only difference between the mRNA library and the mRNA&gDNA library preparation was that DNase I digestion was performed after nucleic acid extraction to remove gDNA in mRNA library (Fig.  1 A). For both libraries, mRNA was reverse-transcribed with oligo(dT) primers to synthesize complementary DNA (cDNA). To reduce bias and simplify the pipeline of library preparation, reverse transcription products were fragmented by sonication and denatured by heat, following library preparation using a highly efficient single-strand DNA (ssDNA) ligation technique, Adaptase™ from xGen™ ssDNA&Low-Input DNA Library Prep Kit (Integrated DNA Technologies). Denaturation makes cDNA-mRNA hybrid and gDNA to ssDNA. Adaptase is a commercial enzyme mixture with very high ligation efficiency and low bias for single-stranded DNA (ssDNA) library construction with strand-specific information, which we have used to develop ssDRIP-seq and ULI-ssDRIP-seq for R-loop profiling [ 14 , 15 ] and DEtail-seq for DNA break detection [ 16 ]. Compared to Relative Quantification RNA-seq, the mRNA library in siqRNA-seq was directly prepared from single-stranded cDNA, therefore we named this method ssRNA-seq (single-strand DNA ligation-basedR NA sequencing) hereafter.

Design of siqRNA-seq. A Flowchart of siqRNA-seq. Total nucleic acids were extracted and two types of libraries, mRNA library (ssRNA-seq) and mRNA&gDNA library, were constructed in parallel and sequenced for each sample in siqRNA-seq. B Principle of siqRNA-seq for RCPG calculation in mRNA&gDNA library. Depth of gDNA can be assessed by intergenic read depth. RCPG is equal to four times the ratio of mRNA read depth to gDNA depth. *: gDNA of diploid with two strands. RCPG: mRNA count per genome

As shown above, we can obtain two data sets from siqRNA-seq after sequencing, ssRNA-seq for mRNA profiling and the mRNA&gDNA library with reads from gDNA and mRNA. The next step is to integrate the two data sets to quantify gene expression with reads from gDNA as an internal reference for normalization (Fig.  1 B). In the mRNA&gDNA library, reads that are mapped to intergenic regions can be used to calculate the sequencing depth of gDNA, while reads that are assigned to exons include mRNA reads and gDNA reads. Consequently, the mRNA count per diploid genome (RCPG) for each gene can be obtained from the ratio of mRNA read depth to gDNA read depth multiplied by four (gDNA of diploid with two strands). If cells are at interphase with constant diploid genomic DNA, we can infer that the mRNA count per cell (RCPC) is equal to RCPG (Fig.  1 B). Considering that it is difficult to distinguish cDNA reads on low-expression genes from the gDNA background, we use ssRNA-seq data to calculate FPKM (Fragments Per Kilobase Million) for all genes and combine FPKM and RCPG values of highly expressed genes to establish a model for normalization (Fig.  1 B). After normalization, FPKM values of ssRNA-seq data are transferred to RCPG values for all expressed genes. Together, siqRNA-seq is a ssDNA ligation-based method that constructs libraries for cDNA and gDNA in parallel, which can be applied to quantify gene expression by using gDNA as an internal reference.

Validation of siqRNA-seq for expression profiling

To investigate the reliability and reproducibility of siqRNA-seq for gene expression profiling, we performed siqRNA-seq on the cell lines HEK293T, IOSE-80, and HCT116 (Fig.  2 A). The results of correlation analysis for ssRNA-seq showed that the expression profiles of replicates were highly correlated with each other (Pearson correlation coefficients were 0.991, 0.985, and 0.987 for HEK293T, IOSE-80, and HCT116, respectively), suggesting that whole nucleic acids can supply high-quality materials for siqRNA-seq library preparation (Additional file 1: Fig.S1A). By comparison, high correlations were also showed between ssRNA-seq and public RNA-seq data (Additional file 1: Fig.S1B), suggesting that ssRNA-seq is reliable and reproducible for gene expression profiling.

Validation of siqRNA-seq for quantitative expression profiling. A IGV showing snapshots of siqRNA-seq signals (RPCG) in the human genome. Both RNA and gDNA signals can be sequenced in the mRNA&gDNA libraries. B Scatter plots showing the correlation between mRNA&gDNA library repeats. C Scatter plots showing the correlation of ssRNA-seq with mRNA&gDNA libraries

Next, we analyzed the ability of the mRNA&gDNA library to perform gene expression profiling. Although there was a high gDNA background, approximately 11.84% of the total mapped reads could be assigned to exons in mRNA&gDNA libraries, while only approximately 6.74% reads could be assigned to exons in gDNA libraries (Not all shown) (Additional file 2: Table S1), suggesting high efficiency for RNA profiling in the mRNA&gDNA library with gDNA. Similar to the ssRNA-seq results, RNA profiles from the mRNA&gDNA libraries remained well-correlated with each replicate (Fig.  2 B), and correlated well with the ssRNA-seq data (Fig.  2 C). These results demonstrated that our mRNA&gDNA library not only can be used as a reliable and reproducible method for gene expression profiling but also supply information on reads from gDNA for expression quantification.

Taken together, our data validated that both the mRNA&gDNA library and ssRNA-seq in siqRNA-seq are technically reliable and reproducible, which are the basis for accurate qualification of gene expression.

Pipeline and model of siqRNA-seq for gene expression quantification

Normally, gDNA in a diploid cell includes two copies for each site in double-stranded DNA formation. However, repeated sequences [ 17 ] and regions such as blacklists of ChIP-seq data in the genome may produce erroneous signals after alignment [ 18 ]. Therefore, we designed a pipeline to extract reliable regions to accurately assess the sequencing depth of gDNA (Fig.  3 A). First, the genome was divided into consecutive windows with a 10 kb bin size and windows overlapping with any gene and downstream of the gene were removed. Then, counts of reads from the gDNA library on each window were calculated. After sorting by sequencing depth, windows only with sequencing depths higher or lower than 10% of the median were selected to exclude windows with extreme depth. Considering that intergenic transcripts may lead to overestimation of gDNA depth, windows that could be mapped by reads from ssRNA-seq were finally removed. The remaining windows, named intergenic regions (IRs), were used to assess gDNA depth in the mRNA&gDNA library. Applying the pipeline for HEK293T cells, 125.78 Mb IRs were screened to calculate gDNA depth in the mRNA&gDNA library (Additional file 1: Fig.S2A).

Pipeline and model of siqRNA-seq for gene expression quantification. A Schematic diagram showing the pipeline for intergenic regions (IRs) screening in the gDNA library. There IRs are used for assessment of gDNA depth in siqRNA-seq quantitative analysis. B Schematic diagram showing the model of siqRNA-seq for gene expression quantification. In the mRNA&gDNA library, a set of genes was selected for constructing a linear model using RCPG values from the mRNA&gDNA library and FPKM values from ssRNA-seq. Then, the established linear model was applied to transform the FPKM values of all genes in the ssRNA-seq to RCPG. C Bar plot showing the number of mRNA molecules per cell in HEK293T and IOSE-80 cells based on siqRNA-seq quantification. D The quantitative results of siqRNA-seq were verified by RT-qPCR in HEK293T cells. n.s.: not significant

Next, the model for siqRNA-seq was established to quantify gene expression as shown in Fig.  3 B. First, the mapped reads of the mRNA&gDNA library were assigned to IRs, and genesto calculate their depth. Then, the RCPG of each gene can be calculated according to the formula shown in Fig.  3 B (see Methods ). The diploid genome is dsDNA with two copies for each region, while cDNA reverse-transcribed from mRNA is ssDNA. Thus, one mRNA read is equal to four gDNA reads sequenced in the mRNA&gDNA library. Due to the gDNA background, lowly expressed genes might not be accurately calculated by data from the mRNA&gDNA library. Then, we built a linear model to calibrate the RCPG values of all expressed genes through the integration of the mRNA&gDNA library and the ssRNA-seq (Not all shown) (Fig.  3 B, Additional file 1: Figs.S2C and S4B). To establish the linear model with RANSAC [ 19 ] for siqRNA-seq, we selected a set of genes according to criteria described in Methods to obtain values of RCPG from the mRNA&gDNA library and expression levels from the ssRNA-seq for each sample (Not all shown) (Additional file 1: Figs.S2C, S4B and Additional file 2: Table S2). Eventually, the model can be used to calibrate the RCPG of each expressed gene using expression data from ssRNA-seq (Fig.  3 B).

To validate the reliability of our model, we applied siqRNA-seq in the HEK293T and IOSE-80 cell lines to estimate the number of mRNA molecules per genome. In these cells, we detected approximately 118,15 and 153,469 mRNA molecules (Fig.  3 C). In addition, we performed qPCR of seven genes with moderate gene expression and three intergenic regions as references to validate our sequencing data (see Methods ). The results showed that the expression of these genes quantified by siqRNA-seq was consistent with the signals from qPCR (Fig.  3 D), further demonstrating the reliability of our siqRNA-seq for gene expression quantification. Together, we established a reliable analysis pipeline and model in siqRNA-seq to quantify gene expression.

Example of siqRNA-seq for differential gene expression analysis

RNA-seq remains the primary tool for differential expression gene (DEG) analysis to study the change in expression of genes or transcripts under different conditions [ 1 ]. To exemplify the merits of siqRNA-seq for quantitative mapping of the mRNA landscape and DEG analysis, we performed siqRNA-seq on HEK293T cells treated with actinomycin D (ActD). ActD, which has been widely used to study mRNA stability, is a transcription inhibitor that does not significantly affect DNA replication or protein synthesis at low concentrations [ 20 , 21 ]. Compared to the untreated control, the abundance of mRNA quantified by siqRNA-seq was greatly reduced in treated cells as expected (Fig.  4 A, Additional file 1: Fig.S4A, and Additional file 2: Table S4). The results of DEG analysis using the siqRNA-seq method showed that almost all DEGs were downregulated (4,746 genes) except one gene that showed upregulation in our data (Fig.  4 B). In contrast, if we use the traditional FPKM normalization method for ssRNA-seq data to quantify gene expression, the fold change trend of all genes in the traditional normalization method was higher than that in the siqRNA-seq method (Fig.  4 C and Additional file 1: Fig.S4C). In summary, approximately 50% of genes showed an upregulation trend (Fig.  4 C), and 2,119 genes were identified as upregulated genes by the FPKM normalization method (Fig.  4 D). In addition, some downregulated genes of siqRNA-seq were assigned to upregulated genes in the FPKM normalization method, such as FUCA2, ARF5, GGCT, CCDC124, RPS20, CSDE1, MDH1, FHL1, and GRN. In line with siqRNA-seq, RT-qPCR confirmed that these genes were downregulated in ActD-treated cells (Additional file 1: Fig.S4D). These results together suggested that siqRNA-seq shows advantages in accurately identifying differentially expressed genes between samples with distinct global mRNA levels using gDNA as an internal reference (Additional file 2: Table S5).

Analysis of factors influencing the downward trend of ActD-regulated gene expression by siqRNA-seq. A Bar plot showing absolute quantification of mRNA molecules per cell for ActD-treated and untreated HEK293T cells by siqRNA-seq. B Volcano plot showing the results of the DEGs identified by siqRNA-seq for HEK293T cells treated with ActD compared to untreated control. Genes with fold change greater than 2 and FDR less than 0.01 were assigned as DEGs. C The Sankey diagram showing the trend of genes with different fold changes(log 2 ) of Relative Quantification RNA-seq and siqRNA-seq for HEK293T cells treated with ActD. In Relative Quantification RNA-seq, nearly 50% of genes showed an upregulation trend, while siqRNA-seq showed almost no upregulation. D Volcano plot showing the results of DEGs identified by Relative Quantification RNA-seq analysis for HEK293T cells treated with ActD compared to untreated control. Genes with a fold change greater than 2 and FDR less than 0.01 were assigned as DEGs

ActD intercalates into DNA to form a very stable complex with DNA to prevent the unwinding of the DNA double-helix; thus, transcription could be globally inhibited [ 21 ]. However, the degradation rates may be distinct among these expressed genes, as our data shown (Fig.  4 B and D). Next, we investigated some potential factors that may impact the degradation of genes responding to ActD. First, we performed a correlation analysis between the fold change (FC) of gene expression after ActD treatment and the gene expression levels in the control. Our results showed that the Pearson correlation coefficient was approximately 0.400 between them (Fig.  4 E), suggesting that the gene degradation rate is positively correlated with the expression level. We also observed that the FC of genes downregulated by ActD was slightly positively correlated with gene length (Pearson r  = 0.065) (Fig.  4 F). Additionally, we investigated the influence of N6-methyladenosine (m 6 A) modification on changes in expression induced by ActD. m 6 A, a chemical derivative of adenosine in RNA, has been increasingly reported as an important RNA modification involved in RNA metabolism, such as altering pre-mRNA processing, promoting mRNA nuclear export, changing mRNA stability, and increasing translation efficiency [ 22 , 23 ]. It has been reported that some genes, such as YTHDF1, YTHDF2, and YTHDF3, trigger the rapid decay of m 6 A-modified transcripts [ 24 , 25 ]. Comparing m 6 A-modified mRNA with mRNA without m 6 A modification, we observed a larger change in m 6 A modified mRNA than mRNA without m 6 A modification, indicating that m 6 A mediates mRNA decay as previously reported (Fig.  4 G) [ 24 , 25 ].

High diversity of total mRNA expression in tumor cell lines

In recent years, due to the continuously rising incidence of cancer, it has become a significant global public health issue. Consequently, research on tumor cells remains a hotspot [ 26 , 27 , 28 ]. We applied siqRNA-seq to tumor cell lines HCT116, A498, DU145, NCI-H226, SK-OV-3, MDA-MB-468, and SW620 to estimate the number of mRNA molecules per genome. In these cells, we detected approximately 64,410, 76,500, 142,008, 55,742, 56,865, 72,380, and 47,147 mRNA molecules (Fig.  5 A), indicating a high degree of diversity in total mRNA expression among tumor cell lines. Through analysis of the mRNA quantification data from these tumor cell lines, we found that the total mRNA content in DU145 tumor cell line was significantly higher than in other tumor cell lines. Therefore, we further investigated the underlying reasons for this phenomenon and discovered that it was due to the overall higher levels of gene expression rather than a few specific genes (Fig.  5 B). Together, with the assistance of siqRNA-seq technology, it was revealed that there is a rich diversity in total RNA expression among tumor cells. Regarding the diversity of total RNA expression in tumor cells, we speculate that it may be attributed to changes in genomic polyploidy [ 29 , 30 , 31 ].

High diversity of total mRNA expression in tumor lines. A The bar chart showing the quantitative results of total mRNA content in seven tumor cell lines obtained through siqRNA-seq. B The bar charts showing the density distribution of fold changes in differential expression compared to tumor HCT116 across these tumor cell lines

In this study, we present a novel method, siqRNA-seq, to profile and quantify global gene expression (Figs. 1 A and 3 A). Without spike-ins, siqRNA-seq integrates reads from gDNA and mRNA to construct a model for quantification of mRNA copy number per genome or per cell (Figs. 1 B and 3 B). Thus, siqRNA-seq allows the accurate identification of differentially expressed genes between samples with distinct mRNA abundances to study RNA dynamics under different conditions (Additional file 1: Fig.S4A).

Although RNA-seq has been applied in various fields for decades [ 32 ], DEG analysis remains at the essential stage for gene expression studies [ 1 ]. Current normalization methods for DEG analysis generally assume that cells have similar RNA abundance, however, this can result in erroneous conclusions for cells globally expressing different RNA levels, such as increased mRNA abundance correlating with high metabolic activity in active cell cycling cells [ 33 , 34 , 35 ]. To quantify the abundance of gene expression, several methods have been reported, such as counting cell numbers to measure different RNA abundances based on the Quant-iT assay [ 33 ], counting cell numbers combined with adding a known quantity of spike-in RNAs [ 36 ], or constructing a linear correlation between sequencing read count and RNA abundance [ 37 ]. However, these methods use absorbance or fluorescence measurements to estimate RNA content per cell with risks of large technical error. The process of cell counting might delay the time point and change the gene expression pattern. Although spike-in RNAs, such as commercially available ERCC, can be used to measure sensitivity, accuracy, and biases in RNA-seq experiments as well as to derive standard curves for quantifying the abundance of transcripts, they are still highly expensive [ 9 ]. Besides, RNase is ubiquitous. RNA standards must be carefully processed to prevent them degradation. In contrast, siqRNA-seq, which uses reads from gDNA as the internal reference to quantify gene expression levels, enables us to assess the copy number of mRNA per cell or per genome, with the advantages of no requirement of cell counting and spike-in controls. In addition, the ratio of mRNA and DNA in whole nucleic acids is stable during dilution to a needed concentration for library construction. For materials, such as multinucleated skeletal muscle or polyploid plants, it may be interesting to investigate their expression profiles per genome. Therefore, we consider siqRNA-seq to be an extended RNA-seq method to quantify expression in cells with multiple nuclei.

As with all high-throughput sequencing methods, siqRNA-seq has limitations. In siqRNA-seq, RNAs were reverse-transcribed with oligo(dT) to synthesize cDNA. The efficiency of reverse transcription greatly impacts the accuracy of quantification. However, the efficiency can be improved by high-efficient reverse transcriptase in optimized reaction buffer. In addition, the efficiency is equal for samples in the same experiment without influencing the expression comparison. Another limitation of our siqRNA-seq is that we only tested the whole nucleic acid extraction method on mammalian cells. For samples, such as microorganisms or plant tissues with cell walls, we may need to optimize the extraction method for different materials. Additionally, cells in the G1 phase of meiosis are required to normalize mRNA copy number to "per cell" by siqRNA-seq, it might be not easy for many samples to isolate cells in the G1 phase. Thus, the “per genome” method may be more practical to normalize data for comparative analysis of gene expression among conditions.

In summary, siqRNA-seq is a spike-in independent quantitative RNA sequencing method, which creatively uses gDNA as an internal reference to quantify gene expression. siqRNA-seq enables us to assess the copy number of mRNA per genome or cell with no requirement of cell counting and spike-ins. We consider that siqRNA-seq can be supplied as a complementary tool to profile and quantify gene expression in many fields.

Cell preparation

Human embryonic kidney cells (HEK293T), human normal ovarian cells (IOSE-80), human colon cancer cells (HCT116), Human renal cell carcinoma cells (A498), human prostate cancer cells (DU145), human squamous lung cancer cells (NCl-H226), human ovarian cancer cells (SK-OV-3), human breast cancer cells (MDA-MB-468), and human colorectal adenocarcinoma cells (SW-620) were purchased from Xiaofan Technology, Guangzhou. They were cultured simultaneously in a medium containing 10% fetal bovine serum (Gibco) and 1% penicillin–streptomycin (Gibco) at 37 °C with 5% CO 2 . When cells reached approximately 90% confluence, they were collected for nucleic acid extraction.

For actinomycin D treatment, cells were cultured to 80% convergence and changed to fresh medium with actinomycin D at a final concentration of 5 μg/ml for 12 h. Then, the medium was removed, and the cells were carefully washed once with PBS. To collect cells, cultures were digested with 0.25% Trypsin–EDTA (Gibco) at 37 °C for 5 min. Untreated HEK293T cells were cultured simultaneously as controls.

Nucleic acid extraction from cells

Total nucleic acids were extracted from the collected cells by SDS and proteinase K methods. Briefly, approximately 1 × 10 6 cells were resuspended in 4 ml TE buffer with 0.5% SDS and 0.1 mg/ml proteinase K and incubated at 37 °C in a shaker at 200 rpm for 4 h. Then, 1/4 volume 5 M potassium acetate was added, mixed well, and placed on ice for 15 min. Finally, the total nucleic acids were purified by the phenol–chloroform extraction method and precipitated with 1 volume of isopropanol. Purified total nucleic acids can be stored at -80 °C until later use.

Genomic DNA (gDNA) library construction

During siqRNA-seq quantification, the gDNA depth of the mRNA&gDNA libraries was assessed by intergenic regions (IRs) that were picked out from gDNA libraries. To construct the gDNA library, the extracted total nucleic acid was dissolved in enzyme-free water and digested with RNase A at 37 °C for 40 min to remove RNA. Then, the gDNA was fragmented by sonication for library preparation by the Accel-NGS 1S Plus DNA Library Kit (Swift Accel-NGS) according to the manual. We compared the IRs of three cell lines (HEK293T, IOSE-80, and HCT116) and found that their IRs are almost identical. Therefore, this study utilized the IRs identified in HEK293T cells for quantitative analysis of siqRNA-seq (Additional file 1: Fig.S2B).

siqRNA-seq library preparation

siqRNA-seq includes two libraries for each sample, the mRNA&gDNA library and ssRNA-seq library. Different from the mRNA&gDNA library, total nucleic acids need to be digested with DNase I at 37 °C for 40 min to remove DNA for the ssRNA-seq library. To reduce potential DNA contamination due to incomplete DNase I digestion as possible, we have used the same amount of the total nucleic acid in the digestion reaction and monitored by Qubit™ 1X dsDNA HS Assay Kit. Then, reverse transcription was performed with oligo(dT) to synthesize the first strand of complementary DNA (cDNA) for both libraries. After extraction and purification, approximately one-fifth of reversed products without sonication were left for qPCR and others were ultrasonically fragmented to 300 bp using ME220 (Covaris, 70 W, 20% Duty factor, 1,000 cycles per burst, 130 s, at 4 °C) for library preparation using the Accel-NGS 1S Plus DNA Library Kit according to the manual, which can use ssDNA as substrate for library preparation.

Routine transcriptome analysis

The libraries were sequenced on NovaSeq 6000 platform. The data were checked by Fastp (version 0.23.1) software [ 38 ] and mapped to the GRCh38 genome via HISAT2 (version 2.2.1) software [ 39 ]. Using two normalization methods to output BigWig files, one is RPCG normalization: BamCoverage from Deeptools (version 3.5.1) [ 40 ] is used to convert Bam format to BigWig format and normalize it to 1 × sequencing depth (RPGC). Another type of normalization is based on IRs: the multiBigwigSummary BED-file (version 3.5.1) for IRs was used to calculate the average score of the BigWig files and the following formula was used to calculate the scale factor. Corrected BigWig files were generated using the scale Factor parameter in BamCoverage.

SRs avg is the average of the average scores for all regions. IRs are the intergenic regions. IRs avg represents the average depth of IRs .

The gene expression profiles were quantified to obtain the fragments per kilobase of the exon model per million mapped fragments (FPKM). Read counts were quantified with FeatureCounts software [ 41 ]. PCA in R was performed using the prcomp function. The correlation of the samples was determined by Pearson's method, using the cor function. The differential expression analysis was performed using edgeR (version 3.26) [ 42 ]. Cut-off values |log2(FC)|> 1 and Padj < 0.05 were used for differential gene expression analysis. The Python package pyecharts was used to draw Sankey diagrams.

To determine the minimal number of reads required for mRNA and gDNA libraries to confidently normalize FPKM values in ssRNA-seq to RCPG values, we randomly subsampled raw fastq files with different numbers of reads and compared their correlations to the original data (Additional file 1: Fig.S5). According to the results, we found that at least 10 million reads are needed for mRNA & gDNA libraries to establish a reliable normalization model.

Quantitative analysis of siqRNA-seq

The siqRNA-seq quantification process was initiated with the step of splitting the gDNA library into 10 kb windows using Bedtools make windows (version 2.30.0) [ 43 ]. Then, windows with sequencing depths 10% above or below the median were sorted out after statistical calculation by multiBamSummary of Deeptools (version 3.5.1) [ 40 ]. In particular, the windows overlapping with any gene or its 10 kb flanking region were discarded, and those overlapping with regions that could be mapped by the ssRNA-seq reads were also removed. The remaining windows were named intergenic regions (IRs). Mapped reads of the mRNA&gDNA library for the genomic region, IRs, and genes were counted to gain the depth of each feature. The RCPG was calculated as follows:

In the formula, “ i ” represents gene i, “ Gi ” represents the depth of gene i, and IRs avg represents the average depth of IRs . The diploid genome is double-stranded DNA, which means that there will be two copies of each region. cDNA reversely transcribed from mRNA has single-stranded DNA. Thus, one mRNA read is equal to four gDNA sequencing reads in the mRNA&gDNA library as shown in the formula.

The gDNA reads in the mRNA&gDNA library may result in some low-expressed genes not being accurately calculated from the data in the mRNA&gDNA library. Therefore, correction of the expressed gene RCPG in ssRNA-seq was performed by constructing a linear model. To construct a linear model, we selected a set of genes that met the following criteria.

Considering that some genes have multiple copies or pseudogenes in the genome. Genes with outlier depth in the gDNA library were removed.

Lengths of genes less than 1,000 bp were removed to reduce bias in genes with as short a length as possible.

Genes with RCPG values obtained from mRNA&gDNA library should be between 5 and 100 to reduce interference from possible outliers. The number of genes with RCPG values between 5 and 100 is consistent with the expected moderate expression levels of genes that exclude quantitative biases caused by under- or overexpression of genes (Additional file 2: Table S6).

Overall, we used RANSAC [ 19 ] to screen eligible genes in mRNA&gDNA libraries and construct a linear model (Additional file 1: Fig.  2 C). Then the FPKM of all expressed genes in ssRNA-seq was calculated. and corrected by this linear model. The final result was the RCPG expressed by each gene.

Quantitative verification by RT-qPCR

To test the accuracy of our siqRNA-seq quantification, RT-qPCR was used for validation. According to the results of the siqRNA-seq sequencing, seven genes with moderate gene expression and three intergenic regions were selected. Primers were designed for genes and intergenic regions for RT-qPCR validation. First, a standard curve was established for the ten pairs of primers (Additional file 1: Fig.S3). Since the gene primers are located in the exon, gDNA can be used as the amplification template. Using gDNA as a template, five gradients (20 ng/μl, 10 ng/μl, 5 ng/μl, 2.5 ng/μl, 1.25 ng/μl) were diluted by 2 × dilution. qPCR was performed with designed primers, and each primer was repeated 3 times. According to the output Cq value of each dilution gradient, the standard curve and the standard curve equation of each primer were calculated. Our experimental samples were then diluted within the dilution gradient range (1.25—20 ng/μl), and RT-qPCR was performed on the samples using gene and intergenic region primers. One Cq value was output for each gene and intergenic region primer, and the Cq values of the two primers were substituted into the equation for their respective primers. The amount of gDNA per diploid genome is four, so the ratio of four times the gene primer values to intergenic regional primer values is RCPG. RT-qPCR quantification values and siqRNA-seq quantification values differed within 10%. Primers for HEK293T cell genes and intergenic regions were designed according to the design principle of fluorescence primers (Additional file 2: Table S7).

Verification of gene differential expression analysis

The transcriptome of HEK293T cells treated with actinomycin D for 12 h was analyzed by Relative Quantification RNA-seq and siqRNA-seq. We found that many genes were upregulated in Relative Quantification RNA-seq, while the expression of these genes in siqRNA-seq was downregulated. Examples include FUCA2, ARF5, GGCT, CCDC124, RPS20, CSDE1, MDH1, FHL1, and GRN. Primers were designed for these nine genes and a housekeeping gene, GAPDH, and RT-qPCR confirmed that these genes were downregulated in ActD-treated cells. The results show that siqRNA-seq shows advantages in accurately identifying differentially expressed genes between samples with different global mRNA levels. Primer design was performed on selected genes according to fluorescent primer design principles (Additional file 2: Table S8).

Availability of data and materials

All data for this study are publicly available and can be accessed via a link. ( https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE223145 ).

Abbreviations

Spike-in-independent quantitative RNA sequencing

Differentially expressed genes

Genomic DNA

Complementary DNA

Single-strand DNA

Single-strand DNA ligation-based RNA sequencing

Intergenic regions

The mRNA count per diploid genome

The mRNA count per cell

Fragments Per Kilobase Million

Actinomycin D

The fold change

N6-methyladenosine

Phosphate buffered saline

Ethylene Diamine Tetraacetie Acid

Dulbecco's Modified Eagle Medium

Quantitative reverse transcription PCR

Sodium dodecyl sulfate

Principal component analysis

Reads Per Genomic Content, defined as (total number of mapped reads * fragment length) / effective genome size

Stark R, Grzelak M, Hadfield J. RNA sequencing: the teenage years. Nat Rev Genet. 2019;20:631–56

Article   CAS   PubMed   Google Scholar  

Kaczkowski B, Tanaka Y, Kawaji H, Sandelin A, Andersson R, Itoh M, et al. Transcriptome Analysis of Recurrently Deregulated Genes across Multiple Cancers Identifies New Pan-Cancer Biomarkers. Cancer Res. 2016;76:216–26.

Satija R, Farrell JA, Gennert D, Schier AF, Regev A. Spatial reconstruction of single-cell gene expression data. Nat Biotechnol. 2015;33:495–502.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Wen L, Li G, Huang T, Geng W, Pei H, Yang J, et al. Single-cell technologies: From research to application. Innovation (Camb). 2022;3:100342.

CAS   PubMed   Google Scholar  

Zhang Y, Lin X, Yao Z, Sun D, Lin X, Wang X, et al. onvolution algorithms for inference of the cell-type composition of the spatial transcriptome. Comput Struct Biotechnol J. 2023;21:176–84.

Janjic A, Wange LE, Bagnoli JW, Geuder J, Nguyen P, Richter D, et al. Prime-seq, efficient and powerful bulk RNA sequencing. Genome Biol. 2022;23:88.

Conesa A, Madrigal P, Tarazona S, Gomez-Cabrero D, Cervera A, McPherson A, et al. A survey of best practices for RNA-seq data analysis. Genome Biol. 2016;17:13.

Article   PubMed   PubMed Central   Google Scholar  

Zyprych-Walczak J, Szabelska A, Handschuh L, Górczak K, Klamecka K, Figlerowicz M, et al. The Impact of Normalization Methods on RNA-Seq Data Analysis. Biomed Res Int. 2015;2015: 621690.

Jiang L, Schlesinger F, Davis CA, Zhang Y, Li R, Salit M, et al. Synthetic spike-in standards for RNA-seq experiments. Genome Res. 2011;21:1543–51.

Lin CY, Lovén J, Rahl PB, Paranal RM, Burge CB, Bradner JE, et al. Transcriptional amplification in tumor cells with elevated c-Myc. Cell. 2012;151:56–67.

Nie Z, Hu G, Wei G, Cui K, Yamane A, Resch W, et al. c-Myc Is a Universal Amplifier of Expressed Genes in Lymphocytes and Embryonic Stem Cells. Cell. 2012;151:68–79.

Lovén J, Orlando DA, Sigova AA, Lin CY, Rahl PB, Burge CB, et al. Revisiting Global Gene Expression Analysis. Cell. 2012;151:476–82.

Risso D, Ngai J, Speed TP, Dudoit S. Normalization of RNA-seq data using factor analysis of control genes or samples. Nat Biotechnol. 2014;32:896–902.

Xu W, Li K, Li Q, Li S, Zhou J, Sun Q. Quantitative, Convenient, and Efficient Genome-Wide R-Loop Profiling by ssDRIP-Seq in Multiple Organisms. Methods Mol Biol. 2022;2528:445–64.

Xu W, Liu X, Li J, Sun C, Chen L, Zhou J, et al. ULI-ssDRIP-seq revealed R-loop dynamics during vertebrate early embryogenesis. Cell Insight. 2024;3:100179.

Xu W, Liu C, Zhang Z, Sun C, Li Q, Li K, et al. DEtail-Seq is an Ultra-efficient and Convenient Method for Meiotic DNA Break Profiling in Multiple Organisms. SCLS. 2023. https://doi.org/10.1007/s11427-022-2277-y .

Article   Google Scholar  

Alexander RP, Fang G, Rozowsky J, Snyder M, Gerstein MB. Annotating non-coding regions of the genome. Nat Rev Genet. 2010;11:559–71.

Amemiya HM, Kundaje A, Boyle AP. The ENCODE Blacklist: Identification of Problematic Regions of the Genome. Sci Rep. 2019;9:9354.

Martínez-Otzeta JM, Rodríguez-Moreno I, Mendialdua I, Sierra B. RANSAC for Robotic Applications: A Survey. Sensors (Basel). 2022;23:327.

Article   PubMed   Google Scholar  

Khanna KK, Jackson SP. DNA double-strand breaks: signaling, repair and the cancer connection. Nat Genet. 2001;27:247–54.

Ratnadiwakara M, Änkö M-L. mRNA Stability Assay Using transcription inhibition by Actinomycin D in Mouse Pluripotent Stem Cells. Bio Protoc. 2018;8:e3072.

He PC, He C. m6 A RNA methylation: from mechanisms to therapeutic potential. EMBO J. 2021;40:e105977.

Lee Y, Choe J, Park OH, Kim YK. Molecular Mechanisms Driving mRNA Degradation by m6A Modification. Trends Genet. 2020;36:177–88.

Du H, Zhao Y, He J, Zhang Y, Xi H, Liu M, et al. YTHDF2 destabilizes m6A-containing RNA through direct recruitment of the CCR4–NOT deadenylase complex. Nat Commun. 2016;7:12626.

Shi H, Wang X, Lu Z, Zhao BS, Ma H, Hsu PJ, et al. YTHDF3 facilitates translation and decay of N6-methyladenosine-modified RNA. Cell Res. 2017;27:315–28.

Bray F, Ferlay J, Soerjomataram I, Siegel RL, Torre LA, Jemal A. Global cancer statistics 2018: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA A Cancer J Clinicians. 2018;68:394–424.

McGuire S. World Cancer Report. Geneva, Switzerland: World Health Organization, International Agency for Research on Cancer, WHO Press, 2015. Adv Nutr. 2014;2016(7):418–9.

Google Scholar  

Shen B, Wang Z, Li Z, Song H, Ding X. Circular RNAs: an emerging landscape in tumor metastasis. Am J Cancer Res. 2019;9:630–43.

CAS   PubMed   PubMed Central   Google Scholar  

Yao Y, Dai W. Genomic Instability and Cancer. J Carcinog Mutagen. 2014;5:1000165.

Lau TY, Poon RYC. Whole-Genome Duplication and Genome Instability in Cancer Cells: Double the Trouble. IJMS. 2023;24:3733.

Bielski CM, Zehir A, Penson AV, Donoghue MTA, Chatila W, Armenia J, et al. Genome doubling shapes the evolution and prognosis of advanced cancers. Nat Genet. 2018;50:1189–95.

Emrich SJ, Barbazuk WB, Li L, Schnable PS. Gene discovery and annotation using LCM-454 transcriptome sequencing. Genome Res. 2007;17:69–73.

Monaco G, Lee B, Xu W, Mustafah S, Hwang YY, Carré C, et al. RNA-Seq Signatures Normalized by mRNA Abundance Allow Absolute Deconvolution of Human Immune Cell Types. Cell Rep. 2019;26:1627-1640.e7.

Wagner GP, Kin K, Lynch VJ. Measurement of mRNA abundance using RNA-seq data: RPKM measure is inconsistent among samples. Theory Biosci. 2012;131:281–5.

Yu R, Vorontsov E, Sihlbom C, Nielsen J. Quantifying absolute gene expression profiles reveals distinct regulation of central carbon metabolism genes in yeast. Elife. 2021;10:e65722.

Schertzer MD, Murvin MM, Calabrese JM. Using RNA Sequencing and Spike-in RNAs to Measure Intracellular Abundance of lncRNAs and mRNAs. Bio Protoc. 2020;10:e3772.

Hu JF, Yim D, Ma D, Huber SM, Davis N, Bacusmo JM, et al. Quantitative mapping of the cellular small RNA landscape with AQRNA-seq. Nat Biotechnol. 2021;39:978–88.

Chen S, Zhou Y, Chen Y, Gu J. fastp: an ultra-fast all-in-one FASTQ preprocessor. Bioinformatics. 2018;34:i884–90.

Kim D, Paggi JM, Park C, Bennett C, Salzberg SL. Graph-based genome alignment and genotyping with HISAT2 and HISAT-genotype. Nat Biotechnol. 2019;37:907–15.

Ramírez F, Dündar F, Diehl S, Grüning BA, Manke T. deepTools: a flexible platform for exploring deep-sequencing data. Nucleic Acids Res. 2014;42 Web Server issue:W187–191.

Liao Y, Smyth GK, Shi W. featureCounts: an efficient general purpose program for assigning sequence reads to genomic features. Bioinformatics. 2014;30:923–30.

Robinson MD, McCarthy DJ, Smyth GK. edgeR : a Bioconductor package for differential expression analysis of digital gene expression data. Bioinformatics. 2010;26:139–40.

Quinlan AR, Hall IM. BEDTools: a flexible suite of utilities for comparing genomic features. Bioinformatics. 2010;26:841–2.

Download references

Acknowledgements

We greatly appreciate all the Wei Xu Lab for useful discussions. We would like to thank Jinjin Li and Jianli Yan for laboratory support.

This study was supported by National Key R&D Program of China (Grant No. 2021YFF1000600), the Youth Innovation Program of Chinese Academy of Agricultural Sciences (No. Y2022QC33), National Natural Science Foundation of China (Grant Nos. 32071437 and 32100423), and China Postdoctoral Science Foundation (Grant No. 2022M713420).

Author information

Zhenzhen Wang, Kehan Tao and Changbin Sun contributed equally to this work.

Authors and Affiliations

Shenzhen Branch, Guangdong Laboratory of Lingnan Modern Agriculture, Key Laboratory of Livestock and Poultry Multi-omics of MARA, Agricultural Genomics Institute at Shenzhen, Chinese Academy of Agricultural Sciences, Shenzhen, 518120, China

Zhenzhen Wang, Kehan Tao, Jiaojiao Ji, Changbin Sun & Wei Xu

You can also search for this author in PubMed   Google Scholar

Contributions

WX and CS conceived, designed, and supervised the experiments. ZW, CS, and KT contributed equally to this work. ZW developed the new technique with the help of CS. Z W and JJ assisted KT in completing the data analysis. WX, ZW, and CS wrote the manuscript. All authors read and approved the final manuscript.

Corresponding authors

Correspondence to Changbin Sun or Wei Xu .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

12864_2024_10650_moesm1_esm.pptx.

Additional file 1: Fig. S1. Quality control of siqRNA-seq. A Scatter plots showing the correlation between repeats of ssRNA-seq. B Scatter plots showing the correlation of ssRNA-seq and data from public databases. Fig. S2. siqRNA-seq for gene expression quantification. A Pie chart showing the size of abnormal sequencing depth regions, ssRNA-seq signal regions, and IRs in the genome. B Bar graph showing the region of IRs of the three cell lines HEK293T, IOSE-80, and HCT116. C Scatter plots showing the construction of linear models in the quantitative process. Fig. S3. RT-qPCR standard curve and standard curve equation. Validate the standard curve and standard curve equation for siqRNA-seq quantification. Fig. S4. siqRNA-seq for differential gene expression analysis. A IGV snapshots showing siqRNA-seq signals (IRs) in the human genome. Gene expression was greatly reduced in ActD drug-treated HEK293T cells compared with untreated controls. B Scatter plots showing the construction of linear models for gene expression quantification of untreated HEK293T cells and cells treated with ActD for 12 h. C Scatter plot showing the trend of fold change for all genes analyzed by siqRNA-seq compared to Relative Quantification RNA-seq for HEK293T cells treated with ActD for 12 h. D Bar plot showing RT-qPCR validation of FUCA2, ARF5, GGCT, CCDC124, RPS20, CSDE1, MDH1, FHL1, and GRN genes downregulated in HEK293T cells with ActD treatment. **: p - Value < 0.01; ***: p - Value < 0.001. Figure S5. Subsampling analysis showing the minimal number of reads required for siqRNA-seq. A Correlation analysis for data between subsampling mRNA & gDNA libraries and the raw mRNA & gDNA library. B Correlation analysis for data between subsampling ssRNA-seq and the raw ssRNA-seq. C Correlation analysis for data between subsampling mRNA & gDNA libraries and the raw ssRNA-seq.

Additional file 2. 

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Wang, Z., Tao, K., Ji, J. et al. siqRNA-seq is a spike-in-independent technique for quantitative mapping of mRNA landscape. BMC Genomics 25 , 743 (2024). https://doi.org/10.1186/s12864-024-10650-2

Download citation

Received : 04 April 2024

Accepted : 22 July 2024

Published : 30 July 2024

DOI : https://doi.org/10.1186/s12864-024-10650-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Transcriptome
• Gene expression
• Quantification
• RNA sequencing
• Next-generation sequencing

BMC Genomics

ISSN: 1471-2164

• Submission enquiries: [email protected]
• General enquiries: [email protected]

How Africa’s ‘ticket’ to prosperity fueled a debt bomb

Protesters took to the streets of Accra, Ghana, in October 2023, accusing the government of economic mismanagement. REUTERS/Francis Kokoroko

Credit ratings were meant to help sub-Saharan countries tap global investors to fund much-needed development. But low scores, heavy borrowing and bad luck have left many struggling with crushing bond debt.

• Medium Text

‘NO CLEAR, OUTRIGHT BIAS’

I never doubted that there were going to be defaults.

David Beers, S&P’s former global head of sovereign ratings

‘A TICKET TO THE BENEFITS’

ALLURING TO INVESTORS

‘DANTE’S INFERNO’

If Africa wants to really become the continent it’s projected to be, you can’t avoid the capital markets. That’s the way of the world.

Daniel Cash, associate professor of law at Aston University in Birmingham, England

Sign up here.

Reporting by Libby George, Tom Bergin and Tom Wilson in London and Lawrence Delevingne in Boston. Additional reporting by Karin Strohecker in London, Maxwell Akalaare Adombila in Accra, MacDonald Dzirutwe in Lagos, Duncan Miriri in Nairobi, Bate Felix in Dakar, Chris Mfula in Lusaka, Nyasha Nyaungwa in Windhoek, and Loucoumane Coulibaly in Abidjan. Photo editing by Simon Newman. Graphics by Prinz Magtulis and Sam Hart. Design by Jillian Kumagai. Edited by Tom Lasseter and John Blanton.

Our Standards: The Thomson Reuters Trust Principles. , opens new tab

Thomson Reuters

Libby George is a London-based journalist on the Reuters emerging markets team. She was part of a team named as Pulitzer finalists in 2023, and who won the Selden Ring Award for International Investigative Reporting, for a series of stories revealing abuses by Nigeria’s military. After launching her career as a political journalist in Washington, D.C., she joined Reuters in 2015 covering oil, and from 2019-2023, she was senior correspondent and acting bureau chief based in Lagos, Nigeria.

Tom Bergin is an investigative journalist specializing in regulatory abuses, fraud, tax avoidance and illicit money flows. His work has won awards including The Gerald Loeb Prize for Distinguished Business and Financial Journalism and the Orwell Prize for Journalism. He is the author of two books: ‘Spills & Spin: The Inside Story of BP’ and ‘Free Lunch Thinking: 8 Economic Myths and Why Politicians Fall for Them’. Tips via email

Tom Wilson covers investigative stories across the finance and business worlds. He previously led Reuters' coverage of cryptocurrencies, and was named the news agency's Reporter of the Year for his series on the Binance crypto exchange. In a decade at Reuters, he was previously posted to Tokyo, where he uncovered abuses in Japan's immigration system and reported on the tobacco giant Philip Morris. His work has won a Gerald Loeb Award, and prizes from Society for Advancement of Business Editing and Writing and the Overseas Press Club.

Do you have a news tip?

How to contact Reuters securely

Investigations

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Protecting against researcher bias in secondary data analysis: challenges and potential solutions

Jessie r. baldwin.

1 Department of Clinical, Educational and Health Psychology, Division of Psychology and Language Sciences, University College London, London, WC1H 0AP UK

2 Social, Genetic and Developmental Psychiatry Centre, Institute of Psychiatry, Psychology and Neuroscience, King’s College London, London, UK

Jean-Baptiste Pingault

Tabea schoeler, hannah m. sallis.

3 MRC Integrative Epidemiology Unit at the University of Bristol, Bristol Medical School, University of Bristol, Bristol, UK

4 School of Psychological Science, University of Bristol, Bristol, UK

5 Centre for Academic Mental Health, Population Health Sciences, University of Bristol, Bristol, UK

Marcus R. Munafò

6 NIHR Biomedical Research Centre, University Hospitals Bristol NHS Foundation Trust and University of Bristol, Bristol, UK

Analysis of secondary data sources (such as cohort studies, survey data, and administrative records) has the potential to provide answers to science and society’s most pressing questions. However, researcher biases can lead to questionable research practices in secondary data analysis, which can distort the evidence base. While pre-registration can help to protect against researcher biases, it presents challenges for secondary data analysis. In this article, we describe these challenges and propose novel solutions and alternative approaches. Proposed solutions include approaches to (1) address bias linked to prior knowledge of the data, (2) enable pre-registration of non-hypothesis-driven research, (3) help ensure that pre-registered analyses will be appropriate for the data, and (4) address difficulties arising from reduced analytic flexibility in pre-registration. For each solution, we provide guidance on implementation for researchers and data guardians. The adoption of these practices can help to protect against researcher bias in secondary data analysis, to improve the robustness of research based on existing data.

Introduction

Secondary data analysis has the potential to provide answers to science and society’s most pressing questions. An abundance of secondary data exists—cohort studies, surveys, administrative data (e.g., health records, crime records, census data), financial data, and environmental data—that can be analysed by researchers in academia, industry, third-sector organisations, and the government. However, secondary data analysis is vulnerable to questionable research practices (QRPs) which can distort the evidence base. These QRPs include p-hacking (i.e., exploiting analytic flexibility to obtain statistically significant results), selective reporting of statistically significant, novel, or “clean” results, and hypothesising after the results are known (HARK-ing [i.e., presenting unexpected results as if they were predicted]; [ 1 ]. Indeed, findings obtained from secondary data analysis are not always replicable [ 2 , 3 ], reproducible [ 4 ], or robust to analytic choices [ 5 , 6 ]. Preventing QRPs in research based on secondary data is therefore critical for scientific and societal progress.

A primary cause of QRPs is common cognitive biases that affect the analysis, reporting, and interpretation of data [ 7 – 10 ]. For example, apophenia (the tendency to see patterns in random data) and confirmation bias (the tendency to focus on evidence that is consistent with one’s beliefs) can lead to particular analytical choices and selective reporting of “publishable” results [ 11 – 13 ]. In addition, hindsight bias (the tendency to view past events as predictable) can lead to HARK-ing, so that observed results appear more compelling.

The scope for these biases to distort research outputs from secondary data analysis is perhaps particularly acute, for two reasons. First, researchers now have increasing access to high-dimensional datasets that offer a multitude of ways to analyse the same data [ 6 ]. Such analytic flexibility can lead to different conclusions depending on the analytical choices made [ 5 , 14 , 15 ]. Second, current incentive structures in science reward researchers for publishing statistically significant, novel, and/or surprising findings [ 16 ]. This combination of opportunity and incentive may lead researchers—consciously or unconsciously—to run multiple analyses and only report the most “publishable” findings.

One way to help protect against the effects of researcher bias is to pre-register research plans [ 17 , 18 ]. This can be achieved by pre-specifying the rationale, hypotheses, methods, and analysis plans, and submitting these to either a third-party registry (e.g., the Open Science Framework [OSF]; https://osf.io/ ), or a journal in the form of a Registered Report [ 19 ]. Because research plans and hypotheses are specified before the results are known, pre-registration reduces the potential for cognitive biases to lead to p-hacking, selective reporting, and HARK-ing [ 20 ]. While pre-registration is not necessarily a panacea for preventing QRPs (Table ​ (Table1), 1 ), meta-scientific evidence has found that pre-registered studies and Registered Reports are more likely to report null results [ 21 – 23 ], smaller effect sizes [ 24 ], and be replicated [ 25 ]. Pre-registration is increasingly being adopted in epidemiological research [ 26 , 27 ], and is even required for access to data from certain cohorts (e.g., the Twins Early Development Study [ 28 ]). However, pre-registration (and other open science practices; Table ​ Table2) 2 ) can pose particular challenges to researchers conducting secondary data analysis [ 29 ], motivating the need for alternative approaches and solutions. Here we describe such challenges, before proposing potential solutions to protect against researcher bias in secondary data analysis (summarised in Fig.  1 ).

Limitations in the use of pre-registration to address QRPs

Challenges and potential solutions regarding sharing pre-existing data

Challenges in pre-registering secondary data analysis and potential solutions (according to researcher motivations). Note : In the “Potential solution” column, blue boxes indicate solutions that are researcher-led; green boxes indicate solutions that should be facilitated by data guardians

Challenges of pre-registration for secondary data analysis

Prior knowledge of the data.

Researchers conducting secondary data analysis commonly analyse data from the same dataset multiple times throughout their careers. However, prior knowledge of the data increases risk of bias, as prior expectations about findings could motivate researchers to pursue certain analyses or questions. In the worst-case scenario, a researcher might perform multiple preliminary analyses, and only pursue those which lead to notable results (perhaps posting a pre-registration for these analyses, even though it is effectively post hoc). However, even if the researcher has not conducted specific analyses previously, they may be biased (either consciously or subconsciously) to pursue certain analyses after testing related questions with the same variables, or even by reading past studies on the dataset. As such, pre-registration cannot fully protect against researcher bias when researchers have previously accessed the data.

Research may not be hypothesis-driven

Pre-registration and Registered Reports are tailored towards hypothesis-driven, confirmatory research. For example, the OSF pre-registration template requires researchers to state “specific, concise, and testable hypotheses”, while Registered Reports do not permit purely exploratory research [ 30 ], although a new Exploratory Reports format now exists [ 31 ]. However, much research involving secondary data is not focused on hypothesis testing, but is exploratory, descriptive, or focused on estimation—in other words, examining the magnitude and robustness of an association as precisely as possible, rather than simply testing a point null. Furthermore, without a strong theoretical background, hypotheses will be arbitrary and could lead to unhelpful inferences [ 32 , 33 ], and so should be avoided in novel areas of research.

Pre-registered analyses are not appropriate for the data

With pre-registration, there is always a risk that the data will violate the assumptions of the pre-registered analyses [ 17 ]. For example, a researcher might pre-register a parametric test, only for the data to be non-normally distributed. However, in secondary data analysis, the extent to which the data shape the appropriate analysis can be considerable. First, longitudinal cohort studies are often subject to missing data and attrition. Approaches to deal with missing data (e.g., listwise deletion; multiple imputation) depend on the characteristics of missing data (e.g., the extent and patterns of missingness [ 34 ]), and so pre-specifying approaches to dealing with missingness may be difficult, or extremely complex. Second, certain analytical decisions depend on the nature of the observed data (e.g., the choice of covariates to include in a multiple regression might depend on the collinearity between the measures, or the degree of missingness of different measures that capture the same construct). Third, much secondary data (e.g., electronic health records and other administrative data) were never collected for research purposes, so can present several challenges that are impossible to predict in advance [ 35 ]. These issues can limit a researcher’s ability to pre-register a precise analytic plan prior to accessing secondary data.

Lack of flexibility in data analysis

Concerns have been raised that pre-registration limits flexibility in data analysis, including justifiable exploration [ 36 – 38 ]. For example, by requiring researchers to commit to a pre-registered analysis plan, pre-registration could prevent researchers from exploring novel questions (with a hypothesis-free approach), conducting follow-up analyses to investigate notable findings [ 39 ], or employing newly published methods with advantages over those pre-registered. While this concern is also likely to apply to primary data analysis, it is particularly relevant to certain fields involving secondary data analysis, such as genetic epidemiology, where new methods are rapidly being developed [ 40 ], and follow-up analyses are often required (e.g., in a genome-wide association study to further investigate the role of a genetic variant associated with a phenotype). However, this concern is perhaps over-stated – pre-registration does not preclude unplanned analyses; it simply makes it more transparent that these analyses are post hoc. Nevertheless, another understandable concern is that reduced analytic flexibility could lead to difficulties in publishing papers and accruing citations. For example, pre-registered studies are more likely to report null results [ 22 , 23 ], likely due to reduced analytic flexibility and selective reporting. While this is a positive outcome for research integrity, null results are less likely to be published [ 13 , 41 , 42 ] and cited [ 11 ], which could disadvantage researchers’ careers.

In this section, we describe potential solutions to address the challenges involved in pre-registering secondary data analysis, including approaches to (1) address bias linked to prior knowledge of the data, (2) enable pre-registration of non-hypothesis-driven research, (3) ensure that pre-planned analyses will be appropriate for the data, and (4) address potential difficulties arising from reduced analytic flexibility.

Challenge: Prior knowledge of the data

Declare prior access to data.

To increase transparency about potential biases arising from knowledge of the data, researchers could routinely report all prior data access in a pre-registration [ 29 ]. This would ideally include evidence from an independent gatekeeper (e.g., a data guardian of the study) stating whether data and relevant variables were accessed by each co-author. To facilitate this process, data guardians could set up a central “electronic checkout” system that records which researchers have accessed data, what data were accessed, and when [ 43 ]. The researcher or data guardian could then provide links to the checkout histories for all co-authors in the pre-registration, to verify their prior data access. If it is not feasible to provide such objective evidence, authors could self-certify their prior access to the dataset and where possible, relevant variables—preferably listing any publications and in-preparation studies based on the dataset [ 29 ]. Of course, self-certification relies on trust that researchers will accurately report prior data access, which could be challenging if the study involves a large number of authors, or authors who have been involved on many studies on the dataset. However, it is likely to be the most feasible option at present as many datasets do not have available electronic records of data access. For further guidance on self-certifying prior data access when pre-registering secondary data analysis studies on a third-party registry (e.g., the OSF), we recommend referring to the template by Van den Akker, Weston [ 29 ].

The extent to which prior access to data renders pre-registration invalid is debatable. On the one hand, even if data have been accessed previously, pre-registration is likely to reduce QRPs by encouraging researchers to commit to a pre-specified analytic strategy. On the other hand, pre-registration does not fully protect against researcher bias where data have already been accessed, and can lend added credibility to study claims, which may be unfounded. Reporting prior data access in a pre-registration is therefore important to make these potential biases transparent, so that readers and reviewers can judge the credibility of the findings accordingly. However, for a more rigorous solution which protects against researcher bias in the context of prior data access, researchers should consider adopting a multiverse approach.

Conduct a multiverse analysis

A multiverse analysis involves identifying all potential analytic choices that could justifiably be made to address a given research question (e.g., different ways to code a variable, combinations of covariates, and types of analytic model), implementing them all, and reporting the results [ 44 ]. Notably, this method differs from the traditional approach in which findings from only one analytic method are reported. It is conceptually similar to a sensitivity analysis, but it is far more comprehensive, as often hundreds or thousands of analytic choices are reported, rather than a handful. By showing the results from all defensible analytic approaches, multiverse analysis reduces scope for selective reporting and provides insight into the robustness of findings against analytical choices (for example, if there is a clear convergence of estimates, irrespective of most analytical choices). For causal questions in observational research, Directed Acyclic Graphs (DAGs) could be used to inform selection of covariates in multiverse approaches [ 45 ] (i.e., to ensure that confounders, rather than mediators or colliders, are controlled for).

Specification curve analysis [ 46 ] is a form of multiverse analysis that has been applied to examine the robustness of epidemiological findings to analytic choices [ 6 , 47 ]. Specification curve analysis involves three steps: (1) identifying all analytic choices – termed “specifications”, (2) displaying the results graphically with magnitude of effect size plotted against analytic choice, and (3) conducting joint inference across all results. When applied to the association between digital technology use and adolescent well-being [ 6 ], specification curve analysis showed that the (small, negative) association diminished after accounting for adequate control variables and recall bias – demonstrating the sensitivity of results to analytic choices.

Despite the benefits of the multiverse approach in addressing analytic flexibility, it is not without limitations. First, because each analytic choice is treated as equally valid, including less justifiable models could bias the results away from the truth. Second, the choice of specifications can be biased by prior knowledge (e.g., a researcher may choose to omit a covariate to obtain a particular result). Third, multiverse analysis may not entirely prevent selective reporting (e.g., if the full range of results are not reported), although pre-registering multiverse approaches (and specifying analytic choices) could mitigate this. Last, and perhaps most importantly, multiverse analysis is technically challenging (e.g., when there are hundreds or thousands of analytic choices) and can be impractical for complex analyses, very large datasets, or when computational resources are limited. However, this burden can be somewhat reduced by tutorials and packages which are being developed to standardise the procedure and reduce computational time [see 48 , 49 ].

Challenge: Research may not be hypothesis-driven

Pre-register research questions and conditions for interpreting findings.

Observational research arguably does not need to have a hypothesis to benefit from pre-registration. For studies that are descriptive or focused on estimation, we recommend pre-registering research questions, analysis plans, and criteria for interpretation. Analytic flexibility will be limited by pre-registering specific research questions and detailed analysis plans, while post hoc interpretation will be limited by pre-specifying criteria for interpretation [ 50 ]. The potential for HARK-ing will also be minimised because readers can compare the published study to the original pre-registration, where a-priori hypotheses were not specified.

Detailed guidance on how to pre-register research questions and analysis plans for secondary data is provided in Van den Akker’s [ 29 ] tutorial. To pre-specify conditions for interpretation, it is important to anticipate – as much as possible – all potential findings, and state how each would be interpreted. For example, suppose that a researcher aims to test a causal relationship between X and Y using a multivariate regression model with longitudinal data. Assuming that all potential confounders have been fully measured and controlled for (albeit a strong assumption) and statistical power is high, three broad sets of results and interpretations could be pre-specified. First, an association between X and Y that is similar in magnitude to the unadjusted association would be consistent with a causal relationship. Second, an association between X and Y that is attenuated after controlling for confounders would suggest that the relationship is partly causal and partly confounded. Third, a minimal, non-statistically significant adjusted association would suggest a lack of evidence for a causal effect of X on Y. Depending on the context of the study, criteria could also be provided on the threshold (or range of thresholds) at which the effect size would justify different interpretations [ 51 ], be considered practically meaningful, or the smallest effect size of interest for equivalence tests [ 52 ]. While researcher biases might still affect the pre-registered criteria for interpreting findings (e.g., toward over-interpreting a small effect size as meaningful), this bias will at least be transparent in the pre-registration.

Use a holdout sample to delineate exploratory and confirmatory research

Where researchers wish to integrate exploratory research into a pre-registered, confirmatory study, a holdout sample approach can be used [ 18 ]. Creating a holdout sample refers to the process of randomly splitting the dataset into two parts, often referred to as ‘training’ and ‘holdout’ datasets. To delineate exploratory and confirmatory research, researchers can first conduct exploratory data analysis on the training dataset (which should comprise a moderate fraction of the data, e.g., 35% [ 53 ]. Based on the results of the discovery process, researchers can pre-register hypotheses and analysis plans to formally test on the holdout dataset. This process has parallels with cross-validation in machine learning, in which the dataset is split and the model is developed on the training dataset, before being tested on the test dataset. The approach enables a flexible discovery process, before formally testing discoveries in a non-biased way.

When considering whether to use the holdout sample approach, three points should be noted. First, because the training dataset is not reusable, there will be a reduced sample size and loss of power relative to analysing the whole dataset. As such, the holdout sample approach will only be appropriate when the original dataset is large enough to provide sufficient power in the holdout dataset. Second, when the training dataset is used for exploration, subsequent confirmatory analyses on the holdout dataset may be overfitted (due to both datasets being drawn from the same sample), so replication in independent samples is recommended. Third, the holdout dataset should be created by an independent data manager or guardian, to ensure that the researcher does not have knowledge of the full dataset. However, it is straightforward to randomly split a dataset into a holdout and training sample and we provide example R code at: https://github.com/jr-baldwin/Researcher_Bias_Methods/blob/main/Holdout_script.md .

Challenge: Pre-registered analyses are not appropriate for the data

Use blinding to test proposed analyses.

One method to help ensure that pre-registered analyses will be appropriate for the data is to trial the analyses on a blinded dataset [ 54 ], before pre-registering. Data blinding involves obscuring the data values or labels prior to data analysis, so that the proposed analyses can be trialled on the data without observing the actual findings. Various types of blinding strategies exist [ 54 ], but one method that is appropriate for epidemiological data is “data scrambling” [ 55 ]. This involves randomly shuffling the data points so that any associations between variables are obscured, whilst the variable distributions (and amounts of missing data) remain the same. We provide a tutorial for how to implement this in R (see https://github.com/jr-baldwin/Researcher_Bias_Methods/blob/main/Data_scrambling_tutorial.md ). Ideally the data scrambling would be done by a data guardian who is independent of the research, to ensure that the main researcher does not access the data prior to pre-registering the analyses. Once the researcher is confident with the analyses, the study can be pre-registered, and the analyses conducted on the unscrambled dataset.

Blinded analysis offers several advantages for ensuring that pre-registered analyses are appropriate, with some limitations. First, blinded analysis allows researchers to directly check the distribution of variables and amounts of missingness, without having to make assumptions about the data that may not be met, or spend time planning contingencies for every possible scenario. Second, blinded analysis prevents researchers from gaining insight into the potential findings prior to pre-registration, because associations between variables are masked. However, because of this, blinded analysis does not enable researchers to check for collinearity, predictors of missing data, or other covariances that may be necessary for model specification. As such, blinded analysis will be most appropriate for researchers who wish to check the data distribution and amounts of missingness before pre-registering.

Trial analyses on a dataset excluding the outcome

Another method to help ensure that pre-registered analyses will be appropriate for the data is to trial analyses on a dataset excluding outcome data. For example, data managers could provide researchers with part of the dataset containing the exposure variable(s) plus any covariates and/or auxiliary variables. The researcher can then trial and refine the analyses ahead of pre-registering, without gaining insight into the main findings (which require the outcome data). This approach is used to mitigate bias in propensity score matching studies [ 26 , 56 ], as researchers use data on the exposure and covariates to create matched groups, prior to accessing any outcome data. Once the exposed and non-exposed groups have been matched effectively, researchers pre-register the protocol ahead of viewing the outcome data. Notably though, this approach could help researchers to identify and address other analytical challenges involving secondary data. For example, it could be used to check multivariable distributional characteristics, test for collinearity between multiple predictor variables, or identify predictors of missing data for multiple imputation.

This approach offers certain benefits for researchers keen to ensure that pre-registered analyses are appropriate for the observed data, with some limitations. Regarding benefits, researchers will be able to examine associations between variables (excluding the outcome), unlike the data scrambling approach described above. This would be helpful for checking certain assumptions (e.g., collinearity or characteristics of missing data such as whether it is missing at random). In addition, the approach is easy to implement, as the dataset can be initially created without the outcome variable, which can then be added after pre-registration, minimising burden on data guardians. Regarding limitations, it is possible that accessing variables in advance could provide some insight into the findings. For example, if a covariate is known to be highly correlated with the outcome, testing the association between the covariate and the exposure could give some indication of the relationship between the exposure and the outcome. To make this potential bias transparent, researchers should report the variables that they already accessed in the pre-registration. Another limitation is that researchers will not be able to identify analytical issues relating to the outcome data in advance of pre-registration. Therefore, this approach will be most appropriate where researchers wish to check various characteristics of the exposure variable(s) and covariates, rather than the outcome. However, a “mixed” approach could be applied in which outcome data is provided in scrambled format, to enable researchers to also assess distributional characteristics of the outcome. This would substantially reduce the number of potential challenges to be considered in pre-registered analytical pipelines.

Pre-register a decision tree

If it is not possible to access any of the data prior to pre-registering (e.g., to enable analyses to be trialled on a dataset that is blinded or missing outcome data), researchers could pre-register a decision tree. This defines the sequence of analyses and rules based on characteristics of the observed data [ 17 ]. For example, the decision tree could specify testing a normality assumption, and based on the results, whether to use a parametric or non-parametric test. Ideally, the decision tree should provide a contingency plan for each of the planned analyses, if assumptions are not fulfilled. Of course, it can be challenging and time consuming to anticipate every potential issue with the data and plan contingencies. However, investing time into pre-specifying a decision tree (or a set of contingency plans) could save time should issues arise during data analysis, and can reduce the likelihood of deviating from the pre-registration.

Challenge: Lack of flexibility in data analysis

Transparently report unplanned analyses.

Unplanned analyses (such as applying new methods or conducting follow-up tests to investigate an interesting or unexpected finding) are a natural and often important part of the scientific process. Despite common misconceptions, pre-registration does not permit such unplanned analyses from being included, as long as they are transparently reported as post-hoc. If there are methodological deviations, we recommend that researchers should (1) clearly state the reasons for using the new method, and (2) if possible, report results from both methods, to ideally show that the change in methods was not due to the results [ 57 ]. This information can either be provided in the manuscript or in an update to the original pre-registration (e.g., on the third-party registry such as the OSF), which can be useful when journal word limits are tight. Similarly, if researchers wish to include additional follow-up analyses to investigate an interesting or unexpected finding, this should be reported but labelled as “exploratory” or “post-hoc” in the manuscript.

Ensure a paper’s value does not depend on statistically significant results

Researchers may be concerned that reduced analytic flexibility from pre-registration could increase the likelihood of reporting null results [ 22 , 23 ], which are harder to publish [ 13 , 42 ]. To address this, we recommend taking steps to ensure that the value and success of a study does not depend on a significant p-value. First, methodologically strong research (e.g., with high statistical power, valid and reliable measures, robustness checks, and replication samples) will advance the field, whatever the findings. Second, methods can be applied to allow for the interpretation of statistically non-significant findings (e.g., Bayesian methods [ 58 ] or equivalence tests, which determine whether an observed effect is surprisingly small [ 52 , 59 , 60 ]. This means that the results will be informative whatever they show, in contrast to approaches relying solely on null hypothesis significance testing, where statistically non-significant findings cannot be interpreted as meaningful. Third, researchers can submit the proposed study as a Registered Report, where it will be evaluated before the results are available. This is arguably the strongest way to protect against publication bias, as in-principle study acceptance is granted without any knowledge of the results. In addition, Registered Reports can improve the methodology, as suggestions from expert reviewers can be incorporated into the pre-registered protocol.

Under a system that rewards novel and statistically significant findings, it is easy for subconscious human biases to lead to QRPs. However, researchers, along with data guardians, journals, funders, and institutions, have a responsibility to ensure that findings are reproducible and robust. While pre-registration can help to limit analytic flexibility and selective reporting, it involves several challenges for epidemiologists conducting secondary data analysis. The approaches described here aim to address these challenges (Fig.  1 ), to either improve the efficacy of pre-registration or provide an alternative approach to address analytic flexibility (e.g., a multiverse analysis). The responsibility in adopting these approaches should not only fall on researchers’ shoulders; data guardians also have an important role to play in recording and reporting access to data, providing blinded datasets and hold-out samples, and encouraging researchers to pre-register and adopt these solutions as part of their data request. Furthermore, wider stakeholders could incentivise these practices; for example, journals could provide a designated space for researchers to report deviations from the pre-registration, and funders could provide grants to establish best practice at the cohort level (e.g., data checkout systems, blinded datasets). Ease of adoption is key to ensure wide uptake, and we therefore encourage efforts to evaluate, simplify and improve these practices. Steps that could be taken to evaluate these practices are presented in Box 1.

More broadly, it is important to emphasise that researcher biases do not operate in isolation, but rather in the context of wider publication bias and a “publish or perish” culture. These incentive structures not only promote QRPs [ 61 ], but also discourage researchers from pre-registering and adopting other time-consuming reproducible methods. Therefore, in addition to targeting bias at the individual researcher level, wider initiatives from journals, funders, and institutions are required to address these institutional biases [ 7 ]. Systemic changes that reward rigorous and reproducible research will help researchers to provide unbiased answers to science and society’s most important questions.

Box 1. Evaluation of approaches

To evaluate, simplify and improve approaches to protect against researcher bias in secondary data analysis, the following steps could be taken.

Co-creation workshops to refine approaches

To obtain feedback on the approaches (including on any practical concerns or feasibility issues) co-creation workshops could be held with researchers, data managers, and wider stakeholders (e.g., journals, funders, and institutions).

Empirical research to evaluate efficacy of approaches

To evaluate the effectiveness of the approaches in preventing researcher bias and/or improving pre-registration, empirical research is needed. For example, to test the extent to which the multiverse analysis can reduce selective reporting, comparisons could be made between effect sizes from multiverse analyses versus effect sizes from meta-analyses (of non-pre-registered studies) addressing the same research question. If smaller effect sizes were found in multiverse analyses, it would suggest that the multiverse approach can reduce selective reporting. In addition, to test whether providing a blinded dataset or dataset missing outcome variables could help researchers develop an appropriate analytical protocol, researchers could be randomly assigned to receive such a dataset (or no dataset), prior to pre-registration. If researchers who received such a dataset had fewer eventual deviations from the pre-registered protocol (in the final study), it would suggest that this approach can help ensure that proposed analyses are appropriate for the data.

Pilot implementation of the measures

To assess the practical feasibility of the approaches, data managers could pilot measures for users of the dataset (e.g., required pre-registration for access to data, provision of datasets that are blinded or missing outcome variables). Feedback could then be collected from researchers and data managers via about the experience and ease of use.

Acknowledgements

The authors are grateful to Professor George Davey for his helpful comments on this article.

Author contributions

JRB and MRM developed the idea for the article. The first draft of the manuscript was written by JRB, with support from MRM and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

J.R.B is funded by a Wellcome Trust Sir Henry Wellcome fellowship (grant 215917/Z/19/Z). J.B.P is a supported by the Medical Research Foundation 2018 Emerging Leaders 1 st Prize in Adolescent Mental Health (MRF-160–0002-ELP-PINGA). M.R.M and H.M.S work in a unit that receives funding from the University of Bristol and the UK Medical Research Council (MC_UU_00011/5, MC_UU_00011/7), and M.R.M is also supported by the National Institute for Health Research (NIHR) Biomedical Research Centre at the University Hospitals Bristol National Health Service Foundation Trust and the University of Bristol.

Declarations

Author declares that they have no conflict of interest.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Use of bedinvetmab (librela ® ) for canine osteoarthritis in france, germany, italy, spain, and the uk: quantitative analysis of veterinarian satisfaction and real-world treatment patterns, simple summary, 1. introduction, 2. materials and methods, 2.1. survey respondents and patient record selection, 2.1.1. targeted profile for study participants, 2.1.2. timing of bedinvetmab prescription, 2.1.3. patient record selection, 2.2. patient record forms, 2.2.1. data collection, security, confidentiality, and quality assurance, 2.2.2. data processing and quality assurance, 2.3. data analysis, 3.1. 1932 prfs were collected from 375 veterinarians in the five countries, 3.2. over half of patients received other therapies before initiation of bedinvetmab, 3.2.1. nsaids were the most frequently prescribed medication prior to initiation of bedinvetmab, 3.2.2. non-nsaid therapies were also prescribed before initiation of bedinvetmab (particularly physical therapy), 3.3. bedinvetmab was initiated as first-line therapy or following lack of response to previous therapy, 3.3.1. bedinvetmab was used across age, weight, and oa pain severity levels, 3.3.2. most patients were compliant with their treatment regimen, 3.4. overall veterinarian-reported satisfaction with bedinvetmab (on a ten-point scale) appeared high in this study and increased at later doses, 3.5. most patients did not require other therapies following initiation of bedinvetmab; a minority added other therapies, or continued previous therapies, 4. discussion, 5. conclusions, supplementary materials, author contributions, institutional review board statement, informed consent statement, data availability statement, acknowledgments, conflicts of interest.

• American College of Veterinary Surgeons. Osteoarthritis in Dogs. 2020. Available online: https://www.acvs.org/small-animal/osteoarthritis-in-dogs (accessed on 23 June 2023).
• Centers for Disease Control and Prevention. Osteoarthritis (OA). 2020. Available online: https://archive.cdc.gov/www_cdc_gov/arthritis/types/osteoarthritis.htm (accessed on 23 June 2023).
• Belshaw, Z.; Yeates, J. Assessment of quality of life and chronic pain in dogs. Vet. J. 2018 , 239 , 59–64. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Wright, A.; Amodie, D.; Cernicchiaro, N.; Lascelles, B.; Pavlock, A. PVM1 Diagnosis and treatment rates of osteoarthritis in dogs using a health risk assessment (HRA) or health questionaire for osteoarthritis in general veterinary practice. Value Health 2019 , 22 , S387. [ Google Scholar ]
• Lavan, R. Development and validation of a survey for quality of life assessment by owners of healthy dogs. Vet. J. 2013 , 197 , 578–582. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Wright, A.; Amodie, D.M.; Cernicchiaro, N.; Lascelles, B.D.X.; Pavlock, A.M.; Roberts, C.; Bartram, D.J. Identification of canine osteoarthritis using an owner-reported questionnaire and treatment monitoring using functional mobility tests. J. Small Anim. Pract. 2022 , 63 , 609–618. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Pettitt, R.A.; German, A.J. Investigation and management of canine osteoarthritis. In Pract. 2015 , 37 , 1–8. [ Google Scholar ] [ CrossRef ]
• Pye, C.; Bruniges, N.; Peffers, M.; Comerford, E. Advances in the pharmaceutical treatment options for canine osteoarthritis. J. Small Anim. Pract. 2022 , 63 , 721–738. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Monteiro, B.; Lascelles, B.; Murrell, J.; Robertson, S.; Steagall, P.; Wright, B. 2022 WSAVA guidelines for the recognition, assessment and treatment of pain. J. Small Anim. Pract. 2023 , 64 , 177–254. [ Google Scholar ] [ CrossRef ]
• Osteoarthritis Research Society International. Definition of Osteoarthritis. 2015. Available online: https://oarsi.org/research/standardization-osteoarthritis-definitions (accessed on 24 March 2022).
• Johnston, S.A.; Budsberg, S.C. Nonsteroidal anti-inflammatory drugs and corticosteroids for the management of canine osteoarthritis. Vet. Clin. N. Am. Small Anim. Pract. 1997 , 27 , 841–862. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Holtsinger, R.; Parker, R.; Beale, B.; Friedman, R. The therapeutic efficacy of carprofen (Rimadyl-V™) in 209 clinical cases of canine degenerative joint disease. Vet. Comp. Orthop. Traumatol. 1992 , 5 , 140–144. [ Google Scholar ]
• Vasseur, P.B.; Johnson, A.; Budsberg, S.; Lincoln, J.; Toombs, J.; Whitehair, J.; Lentz, E. Randomized, controlled trial of the efficacy of carprofen, a nonsteroidal anti-inflammatory drug, in the treatment of osteoarthritis in dogs. J. Am. Vet. Med. Assoc. 1995 , 206 , 807–811. [ Google Scholar ] [ CrossRef ]
• Budsberg, S.C.; Johnston, S.A.; Schwarz, P.; DeCamp, C.E.; Claxton, R. Efficacy of etodolac for the treatment of osteoarthritis of the hip joints in dogs. J. Am. Vet. Med. Assoc. 1999 , 214 , 206–210. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Lascelles, B.D.; Gaynor, J.S.; Smith, E.S.; Roe, S.C.; Marcellin-Little, D.J.; Davidson, G.; Boland, E.; Carr, J. Amantadine in a multimodal analgesic regimen for alleviation of refractory osteoarthritis pain in dogs. J. Vet. Intern. Med. 2008 , 22 , 53–59. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Belshaw, Z.; Asher, L.; Dean, R.S. The attitudes of owners and veterinary professionals in the United Kingdom to the risk of adverse events associated with using non-steroidal anti-inflammatory drugs (NSAIDs) to treat dogs with osteoarthritis. Prev. Vet. Med. 2016 , 131 , 121–126. [ Google Scholar ] [ CrossRef ]
• Bell, A.; Helm, J.; Reid, J. Veterinarians’ attitudes to chronic pain in dogs. Vet. Rec. 2014 , 175 , 428. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Autefage, A.; Gosselin, J. Efficacy and safety of the long-term oral administration of carprofen in the treatment of osteoarthritis in dogs. Rev. Méd. Vét. 2007 , 1 , 119–127. [ Google Scholar ]
• Innes, J.F.; Clayton, J.; Lascelles, B.D.X. Review of the safety and efficacy of long-term NSAID use in the treatment of canine osteoarthritis. Vet. Rec. 2010 , 166 , 226–230. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Mansa, S.; Palmér, E.; Grøndahl, C.; Lønaas, L.; Nyman, G. Long-term treatment with carprofen of 805 dogs with osteoarthritis. Vet. Rec. 2007 , 160 , 427–430. [ Google Scholar ] [ CrossRef ]
• Enomoto, M.; Mantyh, P.W.; Murrell, J.; Innes, J.F.; Lascelles, B.D.X. Anti-nerve growth factor monoclonal antibodies for the control of pain in dogs and cats. Vet. Rec. 2019 , 184 , 23. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Ricciotti, E.; FitzGerald, G.A. Prostaglandins and inflammation. Arterioscler Thromb. Vasc. Biol. 2011 , 31 , 986–1000. [ Google Scholar ] [ CrossRef ]
• Isola, M.; Ferrari, V.; Miolo, A.; Stabile, F.; Bernardini, D.; Carnier, P.; Busetto, R. Nerve growth factor concentrations in the synovial fluid from healthy dogs and dogs with secondary osteoarthritis. Vet. Comp. Orthop. Traumatol. 2011 , 24 , 279–284. [ Google Scholar ] [ CrossRef ]
• European Medicines Agency. Librela. Summary of Product Characteristics. 2020. Available online: https://www.ema.europa.eu/en/documents/product-information/librela-epar-product-information_en.pdf (accessed on 21 April 2022).
• Veterinary Medicines Directorate. Product Details—Librela 5 mg Solution for Injection for Dogs. 2020. Available online: https://www.vmd.defra.gov.uk/productinformationdatabase/product/A011012 (accessed on 22 June 2023).
• Corral, M.J.; Moyaert, H.; Fernandes, T.; Escalada, M.; Tena, J.K.S.; Walters, R.R.; Stegemann, M.R. A prospective, randomized, blinded, placebo-controlled multisite clinical study of bedinvetmab, a canine monoclonal antibody targeting nerve growth factor, in dogs with osteoarthritis. Vet. Anaesth. Analg. 2021 , 48 , 943–955. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Keizer, R.J.; Huitema, A.D.; Schellens, J.H.; Beijnen, J.H. Clinical pharmacokinetics of therapeutic monoclonal antibodies. Clin. Pharmacokinet. 2010 , 49 , 493–507. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Tabrizi, M.; Bornstein, G.G.; Suria, H. Biodistribution mechanisms of therapeutic monoclonal antibodies in health and disease. AAPS J. 2010 , 12 , 33–43. [ Google Scholar ] [ CrossRef ]
• FMR Global Health. 2022. Available online: https://fmrglobalhealth.com/ (accessed on 27 February 2022).
• Gruen, M.E.; Lascelles, B.D.X.; Colleran, E.; Gottlieb, A.; Johnson, J.; Lotsikas, P.; Marcellin-Little, D.; Wright, B. 2022 AAHA Pain Management Guidelines for Dogs and Cats. J. Am. Anim. Hosp. Assoc. 2022 , 58 , 55–76. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Salichs, M.; Badiella, L.; Sarasola, P.; Homedes, J. Efficacy and safety of enflicoxib for treatment of canine osteoarthritis: A 6-week randomised, controlled, blind, multicentre clinical trial. Vet. Rec. 2022 , 191 , e949. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Belshaw, Z.; Dean, R.; Asher, L. “You can be blind because of loving them so much”: The impact on owners in the United Kingdom of living with a dog with osteoarthritis. BMC Vet. Res. 2020 , 16 , 190. [ Google Scholar ] [ CrossRef ]
• Booth, S.; Meller, S.; Packer, R.M.; Farquhar, R.; Maddison, J.E.; Volk, H.A. Owner compliance in canine epilepsy. Vet. Rec. 2021 , 188 , e16. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Gildea, E.; North, C.; Walker, K.; Adriaens, F.; Lascelles, B.D.X. Use of Bedinvetmab (Librela ® ) for Canine Osteoarthritis in France, Germany, Italy, Spain, and the UK: Quantitative Analysis of Veterinarian Satisfaction and Real-World Treatment Patterns. Animals 2024 , 14 , 2231. https://doi.org/10.3390/ani14152231

Gildea E, North C, Walker K, Adriaens F, Lascelles BDX. Use of Bedinvetmab (Librela ® ) for Canine Osteoarthritis in France, Germany, Italy, Spain, and the UK: Quantitative Analysis of Veterinarian Satisfaction and Real-World Treatment Patterns. Animals . 2024; 14(15):2231. https://doi.org/10.3390/ani14152231

Gildea, Edwina, Cyndy North, Kate Walker, Francis Adriaens, and Benedict Duncan X. Lascelles. 2024. "Use of Bedinvetmab (Librela ® ) for Canine Osteoarthritis in France, Germany, Italy, Spain, and the UK: Quantitative Analysis of Veterinarian Satisfaction and Real-World Treatment Patterns" Animals 14, no. 15: 2231. https://doi.org/10.3390/ani14152231

Article Metrics

Article access statistics, supplementary material.

ZIP-Document (ZIP, 255 KiB)

Further Information

Mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals