no significant relationship meaning in research

What is a Statistically Significant Relationship Between Two Variables?

How do you decide if, indeed, there is a statistically significant relationship between two variables in your study? What does the p-value output in statistical software analysis mean? This article explains the concept and provides examples with computations and video tutorial.

What does a researcher mean if he says there is a statistically significant relationship between two variables in his study? What makes the relationship statistically significant?

These questions imply that a test for correlation between two variables was made in that particular study. The specific statistical test could either be the parametric Pearson Product-Moment Correlation or the non-parametric Spearman’s Rho test.

Table of Contents

Statistical software applications to test a statistically significant relationship.

Once the statistical software has finished processing the data, You will get a range of correlation coefficient values along with their corresponding p-values denoted by the letter p and a decimal number for a one-tailed and two-tailed test . The p-value is the one that matters when trying to judge whether there is a statistically significant relationship between two variables.

Confusing Definition of p-value

Many of my students in the statistics course I teach are confused about the meaning of p-value. I understand this dilemma because the references I see online do not explain in easily understandable language the meaning of the p-value.

For example, Investopedia, the top resource that returns out of 83 million-plus searches on the  meaning of p-value,  is challenging to understand for a beginning researcher, an undergraduate, or even a graduate student for that matter. If you click the link I provided on the meaning of p-value, you will understand what I mean.

Hence, I explain the meaning of p-value in the next section in the best way I can in the simplest manner possible. I provide a detailed explanation of what it means in the next section.

The Meaning of p-value

What does the p-value mean? This value never exceeds 1. Why?

Therefore, this means that as a researcher, you should be clear about what you want to test in the first place.

For example, your null hypothesis that will lend itself to statistical analysis should be written like this:

H 0 : There is no relationship between the long quiz score and the number of hours devoted by students in studying their lessons.

That p-value means a 100% probability (read simply as 100 percent sure) that the long quiz score and the number of hours devoted by the students in studying their lessons are correlated. The greater the number of hours devoted by students in learning their courses, the higher their long quiz scores. As simple as that.

Conversely, if the p-value is 0 , this means there is no correlation at all. It means that whether the students study or not, their long quiz scores are not affected at all.

Why in Reality the p-value of 1 is Not Possible

Now, this means that the p-value should not be 1 or numbers greater than that. If you get a p-value of more than 1 in your computation, that’s nonsense. Your p-value, I repeat once again, should range between 1 and 0.

Deciding Whether the Relationship is Significant

Suppose the probability in the example given above is p = 0.05. Is it good enough to say that there is a statistically significant relationship between long quiz scores and the number of hours spent by students studying their lessons?

Comparing the computed p-value with the pre-chosen probabilities of 5% and 1% will help you decide whether the relationship between the two variables is significant or not. If, say, the p-values you obtained in your computation are 0.5, 0.4, or 0.06, you should accept the null hypothesis. That is if you set alpha at 0.05 (α = 0.05). If the value you got is below 0.05 or p < 0.05, then you should accept your alternative hypothesis.

In the above example, the alternative hypothesis that should be accepted when the p-value is less than 0.05 will be:

Computation of Correlation in SPSS

If you want to learn about how to use SPSS in computing correlations, the eight-minute tutorial by Dr. Bogdan Kostic below on the correlation between Intelligence Quotient (IQ) and Grade Point Average (GPA) will guide you. He demonstrates in detail how the data are encoded in SPSS, how the labels are written, and the process of statistical test selection. In this instance, the correlation between IQ and GPA using Pearson’s product-moment correlation with the accompanying computer output. This video will further strengthen your knowledge on how to determine if there is a significant relationship between two groups of variables.

More examples and demonstrations on how to find out if there is a statistically significant relationship between variables are given in the two articles below. These articles provide example computer outputs and how these are interpreted.

More Easy-to-Follow Tips

For very easy-to-follow tips on how to select the appropriate statistical tests for your study, see my eBook on statistics at the middle of the page in Simplyeducate.me ‘s eBook store.

Guilford, J. P., 1956. Fundamental statistics in psychology and education. New York: McGraw-Hill. p. 145.

© 2014 May 29 P. A. Regoniel Updated 13 November 2020

Related Posts

Information system and decision making: 3 essential components for effective management, statistical sampling: how to determine sample size, regression analysis: 5 steps and 4 applications, about the author, patrick regoniel.

Dr. Regoniel, a hobbyist writer, served as consultant to various environmental research and development projects covering issues and concerns on climate change, coral reef resources and management, economic valuation of environmental and natural resources, mining, and waste management and pollution. He has extensive experience on applied statistics, systems modelling and analysis, an avid practitioner of LaTeX, and a multidisciplinary web developer. He leverages pioneering AI-powered content creation tools to produce unique and comprehensive articles in this website.

Thanks for this article,It’s really helpfull.I have a question regarding my project.It is “Relationship between ocupational stress and Marital Adjustment among working women in defence”. Here,Is it necessary to do t-test as I have already calculated correlation and the sample size is 30? Please help me out from this.

The article is really very useful to me as it helps me analyze the meaning of significant relationship between two variables. However, I have yet to go over the article, read it several times to finally come with a better grasp of the idea behind relationship. Thanks anyway.

SimplyEducate.Me Privacy Policy

User Preferences

Content preview.

Arcu felis bibendum ut tristique et egestas quis:

• Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris
• Duis aute irure dolor in reprehenderit in voluptate
• Excepteur sint occaecat cupidatat non proident

Keyboard Shortcuts

5.2 - correlation & significance.

This lesson expands on the statistical methods for examining the relationship between two different measurement variables.  Remember that overall statistical methods are one of two types: descriptive methods  (that describe attributes of a data set) and inferential methods (that try to draw conclusions about a population based on sample data).

Correlation

Many relationships between two measurement variables tend to fall close to a straight line . In other words, the two variables exhibit a linear relationship . The graphs in Figure 5.2 and Figure 5.3 show approximately linear relationships between the two variables.

It is also helpful to have a single number that will measure the strength of the linear relationship between the two variables. This number is the correlation . The correlation is a single number that indicates how close the values fall to a straight line.  In other words, the correlation quantifies both the strength and direction of the linear relationship between the two measurement variables. Table 5.1 shows the correlations for data used in Example 5.1  to  Example 5.3 . (Note: you would use software to calculate a correlation.)

Watch the movie below to get a feel for how the correlation relates to the strength of the linear association in a scatterplot.

Features of correlation

Below are some features about the correlation .

• The correlation of a sample is represented by the letter r .
• The range of possible values for a correlation is between -1 to +1.
• A positive correlation indicates a positive linear association like the one in example 5.8. The strength of the positive linear association increases as the correlation becomes closer to +1.
• A negative correlation indicates a negative linear association. The strength of the negative linear association increases as the correlation becomes closer to -1.
• A correlation of either +1 or -1 indicates a perfect linear relationship. This is hard to find with real data.
• there is no linear relationship between the two variables, and/or
• the best straight line through the data is horizontal.
• The correlation is independent of the original units of the two variables. This is because the correlation depends only on the relationship between the standard scores of each variable.
• The correlation is calculated using every observation in the data set.
• The correlation is a descriptive result.

As you compare the scatterplots of the data from the three examples with their actual correlations, you should notice that findings are consistent for each example.

• In Example 5.1 , the scatterplot shows a positive association between weight and height.  However, there is still quite a bit of scatter around the pattern. Consequently, a correlation of .541 is reasonable. It is common for a correlation to decrease as sample size increases.
• In Example 5.2 , the scatterplot shows a negative association between monthly rent and distance from campus. Since the data points are very close to a straight line it is not surprising the correlation is -.903.
• In Example 5.3 , the scatterplot does not show any strong association between exercise hours/week and study hours/week. This lack of association is supported by a correlation of .109.

Statistical Significance Section  

A statistically significant relationship is one that is large enough to be unlikely to have occurred in the sample if there's no relationship in the population. The issue of whether a result is unlikely to happen by chance is an important one in establishing cause-and-effect relationships from experimental data.  If an experiment is well planned, randomization makes the various treatment groups similar to each other at the beginning of the experiment except for the luck of the draw that determines who gets into which group.  Then, if subjects are treated the same during the experiment (e.g. via double blinding), there can be two possible explanations for differences seen: 1) the treatment(s) had an effect or 2) differences are due to the luck of the draw.  Thus, showing that random chance is a poor explanation for a relationship seen in the sample provides important evidence that the treatment had an effect.

The issue of statistical significance is also applied to observational studies - but in that case, there are many possible explanations for seeing an observed relationship, so a finding of significance cannot help in establishing a cause-and-effect relationship.  For example, an explanatory variable may be associated with the response because:

• Changes in the explanatory variable cause changes in the response;
• Changes in the response variable cause changes in the explanatory variable;
• Changes in the explanatory variable contribute, along with other variables, to changes in the response;
• A confounding variable or a common cause affects both the explanatory and response variables;
• Both variables have changed together over time or space; or
• The association may be the result of coincidence (the only issue on this list that is addressed by statistical significance).

Remember the key lesson:  correlation demonstrates association - but the association is not the same as causation, even with a finding of significance.  

What is The Null Hypothesis & When Do You Reject The Null Hypothesis

Julia Simkus

Editor at Simply Psychology

BA (Hons) Psychology, Princeton University

Julia Simkus is a graduate of Princeton University with a Bachelor of Arts in Psychology. She is currently studying for a Master's Degree in Counseling for Mental Health and Wellness in September 2023. Julia's research has been published in peer reviewed journals.

Learn about our Editorial Process

Saul McLeod, PhD

Editor-in-Chief for Simply Psychology

BSc (Hons) Psychology, MRes, PhD, University of Manchester

Saul McLeod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.

Olivia Guy-Evans, MSc

Associate Editor for Simply Psychology

BSc (Hons) Psychology, MSc Psychology of Education

Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.

On This Page:

A null hypothesis is a statistical concept suggesting no significant difference or relationship between measured variables. It’s the default assumption unless empirical evidence proves otherwise.

The null hypothesis states no relationship exists between the two variables being studied (i.e., one variable does not affect the other).

The null hypothesis is the statement that a researcher or an investigator wants to disprove.

Testing the null hypothesis can tell you whether your results are due to the effects of manipulating ​ the dependent variable or due to random chance. 

How to Write a Null Hypothesis

Null hypotheses (H0) start as research questions that the investigator rephrases as statements indicating no effect or relationship between the independent and dependent variables.

It is a default position that your research aims to challenge or confirm.

For example, if studying the impact of exercise on weight loss, your null hypothesis might be:

There is no significant difference in weight loss between individuals who exercise daily and those who do not.

Examples of Null Hypotheses

When Do We Reject The Null Hypothesis? 

We reject the null hypothesis when the data provide strong enough evidence to conclude that it is likely incorrect. This often occurs when the p-value (probability of observing the data given the null hypothesis is true) is below a predetermined significance level.

If the collected data does not meet the expectation of the null hypothesis, a researcher can conclude that the data lacks sufficient evidence to back up the null hypothesis, and thus the null hypothesis is rejected. 

Rejecting the null hypothesis means that a relationship does exist between a set of variables and the effect is statistically significant ( p > 0.05).

If the data collected from the random sample is not statistically significance , then the null hypothesis will be accepted, and the researchers can conclude that there is no relationship between the variables. 

You need to perform a statistical test on your data in order to evaluate how consistent it is with the null hypothesis. A p-value is one statistical measurement used to validate a hypothesis against observed data.

Calculating the p-value is a critical part of null-hypothesis significance testing because it quantifies how strongly the sample data contradicts the null hypothesis.

The level of statistical significance is often expressed as a  p  -value between 0 and 1. The smaller the p-value, the stronger the evidence that you should reject the null hypothesis.

Usually, a researcher uses a confidence level of 95% or 99% (p-value of 0.05 or 0.01) as general guidelines to decide if you should reject or keep the null.

When your p-value is less than or equal to your significance level, you reject the null hypothesis.

In other words, smaller p-values are taken as stronger evidence against the null hypothesis. Conversely, when the p-value is greater than your significance level, you fail to reject the null hypothesis.

In this case, the sample data provides insufficient data to conclude that the effect exists in the population.

Because you can never know with complete certainty whether there is an effect in the population, your inferences about a population will sometimes be incorrect.

When you incorrectly reject the null hypothesis, it’s called a type I error. When you incorrectly fail to reject it, it’s called a type II error.

Why Do We Never Accept The Null Hypothesis?

The reason we do not say “accept the null” is because we are always assuming the null hypothesis is true and then conducting a study to see if there is evidence against it. And, even if we don’t find evidence against it, a null hypothesis is not accepted.

A lack of evidence only means that you haven’t proven that something exists. It does not prove that something doesn’t exist. 

It is risky to conclude that the null hypothesis is true merely because we did not find evidence to reject it. It is always possible that researchers elsewhere have disproved the null hypothesis, so we cannot accept it as true, but instead, we state that we failed to reject the null. 

One can either reject the null hypothesis, or fail to reject it, but can never accept it.

Why Do We Use The Null Hypothesis?

We can never prove with 100% certainty that a hypothesis is true; We can only collect evidence that supports a theory. However, testing a hypothesis can set the stage for rejecting or accepting this hypothesis within a certain confidence level.

The null hypothesis is useful because it can tell us whether the results of our study are due to random chance or the manipulation of a variable (with a certain level of confidence).

A null hypothesis is rejected if the measured data is significantly unlikely to have occurred and a null hypothesis is accepted if the observed outcome is consistent with the position held by the null hypothesis.

Rejecting the null hypothesis sets the stage for further experimentation to see if a relationship between two variables exists. 

Hypothesis testing is a critical part of the scientific method as it helps decide whether the results of a research study support a particular theory about a given population. Hypothesis testing is a systematic way of backing up researchers’ predictions with statistical analysis.

It helps provide sufficient statistical evidence that either favors or rejects a certain hypothesis about the population parameter. 

Purpose of a Null Hypothesis 

• The primary purpose of the null hypothesis is to disprove an assumption. 
• Whether rejected or accepted, the null hypothesis can help further progress a theory in many scientific cases.
• A null hypothesis can be used to ascertain how consistent the outcomes of multiple studies are.

Do you always need both a Null Hypothesis and an Alternative Hypothesis?

The null (H0) and alternative (Ha or H1) hypotheses are two competing claims that describe the effect of the independent variable on the dependent variable. They are mutually exclusive, which means that only one of the two hypotheses can be true. 

While the null hypothesis states that there is no effect in the population, an alternative hypothesis states that there is statistical significance between two variables. 

The goal of hypothesis testing is to make inferences about a population based on a sample. In order to undertake hypothesis testing, you must express your research hypothesis as a null and alternative hypothesis. Both hypotheses are required to cover every possible outcome of the study. 

What is the difference between a null hypothesis and an alternative hypothesis?

The alternative hypothesis is the complement to the null hypothesis. The null hypothesis states that there is no effect or no relationship between variables, while the alternative hypothesis claims that there is an effect or relationship in the population.

It is the claim that you expect or hope will be true. The null hypothesis and the alternative hypothesis are always mutually exclusive, meaning that only one can be true at a time.

What are some problems with the null hypothesis?

One major problem with the null hypothesis is that researchers typically will assume that accepting the null is a failure of the experiment. However, accepting or rejecting any hypothesis is a positive result. Even if the null is not refuted, the researchers will still learn something new.

Why can a null hypothesis not be accepted?

We can either reject or fail to reject a null hypothesis, but never accept it. If your test fails to detect an effect, this is not proof that the effect doesn’t exist. It just means that your sample did not have enough evidence to conclude that it exists.

We can’t accept a null hypothesis because a lack of evidence does not prove something that does not exist. Instead, we fail to reject it.

Failing to reject the null indicates that the sample did not provide sufficient enough evidence to conclude that an effect exists.

If the p-value is greater than the significance level, then you fail to reject the null hypothesis.

Is a null hypothesis directional or non-directional?

A hypothesis test can either contain an alternative directional hypothesis or a non-directional alternative hypothesis. A directional hypothesis is one that contains the less than (“<“) or greater than (“>”) sign.

A nondirectional hypothesis contains the not equal sign (“≠”).  However, a null hypothesis is neither directional nor non-directional.

A null hypothesis is a prediction that there will be no change, relationship, or difference between two variables.

The directional hypothesis or nondirectional hypothesis would then be considered alternative hypotheses to the null hypothesis.

Gill, J. (1999). The insignificance of null hypothesis significance testing.  Political research quarterly ,  52 (3), 647-674.

Krueger, J. (2001). Null hypothesis significance testing: On the survival of a flawed method.  American Psychologist ,  56 (1), 16.

Masson, M. E. (2011). A tutorial on a practical Bayesian alternative to null-hypothesis significance testing.  Behavior research methods ,  43 , 679-690.

Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy.  Psychological methods ,  5 (2), 241.

Rozeboom, W. W. (1960). The fallacy of the null-hypothesis significance test.  Psychological bulletin ,  57 (5), 416.

• Skip to secondary menu
• Skip to main content
• Skip to primary sidebar

Statistics By Jim

Making statistics intuitive

Null Hypothesis: Definition, Rejecting & Examples

By Jim Frost 6 Comments

What is a Null Hypothesis?

The null hypothesis in statistics states that there is no difference between groups or no relationship between variables. It is one of two mutually exclusive hypotheses about a population in a hypothesis test.

• Null Hypothesis H 0 : No effect exists in the population.
• Alternative Hypothesis H A : The effect exists in the population.

In every study or experiment, researchers assess an effect or relationship. This effect can be the effectiveness of a new drug, building material, or other intervention that has benefits. There is a benefit or connection that the researchers hope to identify. Unfortunately, no effect may exist. In statistics, we call this lack of an effect the null hypothesis. Researchers assume that this notion of no effect is correct until they have enough evidence to suggest otherwise, similar to how a trial presumes innocence.

In this context, the analysts don’t necessarily believe the null hypothesis is correct. In fact, they typically want to reject it because that leads to more exciting finds about an effect or relationship. The new vaccine works!

You can think of it as the default theory that requires sufficiently strong evidence to reject. Like a prosecutor, researchers must collect sufficient evidence to overturn the presumption of no effect. Investigators must work hard to set up a study and a data collection system to obtain evidence that can reject the null hypothesis.

Related post : What is an Effect in Statistics?

Null Hypothesis Examples

Null hypotheses start as research questions that the investigator rephrases as a statement indicating there is no effect or relationship.

After reading these examples, you might think they’re a bit boring and pointless. However, the key is to remember that the null hypothesis defines the condition that the researchers need to discredit before suggesting an effect exists.

Let’s see how you reject the null hypothesis and get to those more exciting findings!

When to Reject the Null Hypothesis

So, you want to reject the null hypothesis, but how and when can you do that? To start, you’ll need to perform a statistical test on your data. The following is an overview of performing a study that uses a hypothesis test.

The first step is to devise a research question and the appropriate null hypothesis. After that, the investigators need to formulate an experimental design and data collection procedures that will allow them to gather data that can answer the research question. Then they collect the data. For more information about designing a scientific study that uses statistics, read my post 5 Steps for Conducting Studies with Statistics .

After data collection is complete, statistics and hypothesis testing enter the picture. Hypothesis testing takes your sample data and evaluates how consistent they are with the null hypothesis. The p-value is a crucial part of the statistical results because it quantifies how strongly the sample data contradict the null hypothesis.

When the sample data provide sufficient evidence, you can reject the null hypothesis. In a hypothesis test, this process involves comparing the p-value to your significance level .

Rejecting the Null Hypothesis

Reject the null hypothesis when the p-value is less than or equal to your significance level. Your sample data favor the alternative hypothesis, which suggests that the effect exists in the population. For a mnemonic device, remember—when the p-value is low, the null must go!

When you can reject the null hypothesis, your results are statistically significant. Learn more about Statistical Significance: Definition & Meaning .

Failing to Reject the Null Hypothesis

Conversely, when the p-value is greater than your significance level, you fail to reject the null hypothesis. The sample data provides insufficient data to conclude that the effect exists in the population. When the p-value is high, the null must fly!

Note that failing to reject the null is not the same as proving it. For more information about the difference, read my post about Failing to Reject the Null .

That’s a very general look at the process. But I hope you can see how the path to more exciting findings depends on being able to rule out the less exciting null hypothesis that states there’s nothing to see here!

Let’s move on to learning how to write the null hypothesis for different types of effects, relationships, and tests.

Related posts : How Hypothesis Tests Work and Interpreting P-values

How to Write a Null Hypothesis

The null hypothesis varies by the type of statistic and hypothesis test. Remember that inferential statistics use samples to draw conclusions about populations. Consequently, when you write a null hypothesis, it must make a claim about the relevant population parameter . Further, that claim usually indicates that the effect does not exist in the population. Below are typical examples of writing a null hypothesis for various parameters and hypothesis tests.

Related posts : Descriptive vs. Inferential Statistics and Populations, Parameters, and Samples in Inferential Statistics

Group Means

T-tests and ANOVA assess the differences between group means. For these tests, the null hypothesis states that there is no difference between group means in the population. In other words, the experimental conditions that define the groups do not affect the mean outcome. Mu (µ) is the population parameter for the mean, and you’ll need to include it in the statement for this type of study.

For example, an experiment compares the mean bone density changes for a new osteoporosis medication. The control group does not receive the medicine, while the treatment group does. The null states that the mean bone density changes for the control and treatment groups are equal.

• Null Hypothesis H 0 : Group means are equal in the population: µ 1 = µ 2 , or µ 1 – µ 2 = 0
• Alternative Hypothesis H A : Group means are not equal in the population: µ 1 ≠ µ 2 , or µ 1 – µ 2 ≠ 0.

Group Proportions

Proportions tests assess the differences between group proportions. For these tests, the null hypothesis states that there is no difference between group proportions. Again, the experimental conditions did not affect the proportion of events in the groups. P is the population proportion parameter that you’ll need to include.

For example, a vaccine experiment compares the infection rate in the treatment group to the control group. The treatment group receives the vaccine, while the control group does not. The null states that the infection rates for the control and treatment groups are equal.

• Null Hypothesis H 0 : Group proportions are equal in the population: p 1 = p 2 .
• Alternative Hypothesis H A : Group proportions are not equal in the population: p 1 ≠ p 2 .

Correlation and Regression Coefficients

Some studies assess the relationship between two continuous variables rather than differences between groups.

In these studies, analysts often use either correlation or regression analysis . For these tests, the null states that there is no relationship between the variables. Specifically, it says that the correlation or regression coefficient is zero. As one variable increases, there is no tendency for the other variable to increase or decrease. Rho (ρ) is the population correlation parameter and beta (β) is the regression coefficient parameter.

For example, a study assesses the relationship between screen time and test performance. The null states that there is no correlation between this pair of variables. As screen time increases, test performance does not tend to increase or decrease.

• Null Hypothesis H 0 : The correlation in the population is zero: ρ = 0.
• Alternative Hypothesis H A : The correlation in the population is not zero: ρ ≠ 0.

For all these cases, the analysts define the hypotheses before the study. After collecting the data, they perform a hypothesis test to determine whether they can reject the null hypothesis.

The preceding examples are all for two-tailed hypothesis tests. To learn about one-tailed tests and how to write a null hypothesis for them, read my post One-Tailed vs. Two-Tailed Tests .

Related post : Understanding Correlation

Neyman, J; Pearson, E. S. (January 1, 1933).  On the Problem of the most Efficient Tests of Statistical Hypotheses .  Philosophical Transactions of the Royal Society A .  231  (694–706): 289–337.

Share this:

Reader Interactions

January 11, 2024 at 2:57 pm

Thanks for the reply.

January 10, 2024 at 1:23 pm

Hi Jim, In your comment you state that equivalence test null and alternate hypotheses are reversed. For hypothesis tests of data fits to a probability distribution, the null hypothesis is that the probability distribution fits the data. Is this correct?

January 10, 2024 at 2:15 pm

Those two separate things, equivalence testing and normality tests. But, yes, you’re correct for both.

Hypotheses are switched for equivalence testing. You need to “work” (i.e., collect a large sample of good quality data) to be able to reject the null that the groups are different to be able to conclude they’re the same.

With typical hypothesis tests, if you have low quality data and a low sample size, you’ll fail to reject the null that they’re the same, concluding they’re equivalent. But that’s more a statement about the low quality and small sample size than anything to do with the groups being equal.

So, equivalence testing make you work to obtain a finding that the groups are the same (at least within some amount you define as a trivial difference).

For normality testing, and other distribution tests, the null states that the data follow the distribution (normal or whatever). If you reject the null, you have sufficient evidence to conclude that your sample data don’t follow the probability distribution. That’s a rare case where you hope to fail to reject the null. And it suffers from the problem I describe above where you might fail to reject the null simply because you have a small sample size. In that case, you’d conclude the data follow the probability distribution but it’s more that you don’t have enough data for the test to register the deviation. In this scenario, if you had a larger sample size, you’d reject the null and conclude it doesn’t follow that distribution.

I don’t know of any equivalence testing type approach for distribution fit tests where you’d need to work to show the data follow a distribution, although I haven’t looked for one either!

February 20, 2022 at 9:26 pm

Is a null hypothesis regularly (always) stated in the negative? “there is no” or “does not”

February 23, 2022 at 9:21 pm

Typically, the null hypothesis includes an equal sign. The null hypothesis states that the population parameter equals a particular value. That value is usually one that represents no effect. In the case of a one-sided hypothesis test, the null still contains an equal sign but it’s “greater than or equal to” or “less than or equal to.” If you wanted to translate the null hypothesis from its native mathematical expression, you could use the expression “there is no effect.” But the mathematical form more specifically states what it’s testing.

It’s the alternative hypothesis that typically contains does not equal.

There are some exceptions. For example, in an equivalence test where the researchers want to show that two things are equal, the null hypothesis states that they’re not equal.

In short, the null hypothesis states the condition that the researchers hope to reject. They need to work hard to set up an experiment and data collection that’ll gather enough evidence to be able to reject the null condition.

February 15, 2022 at 9:32 am

Dear sir I always read your notes on Research methods.. Kindly tell is there any available Book on all these..wonderfull Urgent

Comments and Questions Cancel reply

Stack Exchange Network

Stack Exchange network consists of 183 Q&A communities including Stack Overflow , the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Interpreting non-significant regression coefficients

Out of seven, six of the independent variables (predictors) are not significant ( $p>0.05$ ), but their correlation values are small to moderate. Moreover, the $p$ -value of the regression itself is significant ( $p<0.005$ ; Table 2).

I understand in a partial-least squares analysis or SEM, the weights (standardized coefficients in Table 1) are considered rather than the correlation coefficient $r$ (Table 4). I am trying to achieve the same using multiple regression analysis.

Should I ignore the var that are non-significant in the coefficient table?

Do we account for significance or non-signficance from the corresponding 1-tailed sig in Table 4 (correlations) for each variable or should we consider the 2-tailed sig in Table 1 (coefficients)?

I am planning to investigate how each variable in a framework are related to each other (directly and indirectly) using multiple regression. Kindly advise.

• multiple-regression
• regression-coefficients
• exploratory-data-analysis

• 1 $\begingroup$ I understand that the path coefficients are the beta weights (standardized coefficients) in the coefficient table, between the independent var and the dependent var. Do we actually make use of the values of Pearson Correlation in a Path Analysis run through Multiple Regression? $\endgroup$ –  Vyas Commented Sep 26, 2018 at 3:14
• $\begingroup$ Please include the output you are referring to. $\endgroup$ –  Frans Rodenburg Commented Sep 26, 2018 at 11:30

I'll try to address your questions in order, but I don't think this is the right approach, so you may also skip to the third quote.

Should I ignore the variables that are non-significant in the coefficient table?

Non-significant results are also results and you should definitely include them in the results. However, you should not focus too much on what the implications of their estimated coefficients might be. Namely, their large standard errors (or similarly: high $p$ -values) suggest that you might as well have observed an effect this large if the true effect were zero.

Table 1 shows the estimated coefficients of your explanatory variables. While bearing in mind that no causal relationship has been demonstrated, you can interpret significance here as: Does a unit change in this explanatory variable correspond to a significant change in the response variable?

Table 4 appears to show the correlations of your fixed effects. Whether or not your explanatory variables strongly correlate to each other says more about whether you might have potential problems with estimating this model, rather then their effect on the outcome. Significance here could mean that you have collinearity issues, but there are better ways to diagnose collinearity .

Now as to why I don't think you can best answer your research objective with multiple regression:

I am planning to investigate how each variable in a framework are related to each other

Unless you have a variable that can clearly be considered the outcome of the others, and you have some idea of which interactions to test for, I don't think multiple regression is the way to go here. Using multiple regression, you would have to regress all variables on all other variables and interpret a multitude of output tables. You are almost guaranteed to find spurious correlations and I doubt any $p$ -values would be significant after correcting for multiple testing .

If you really want to use multiple regression, I suggest you forget about significance and instead construct a set of confidence intervals using the reported standard errors in table 1. You should clearly state that the goal is exploration and then you can propose which variables might correlate with which. A future study could then try to confirm/refute these findings.

Instead, you might be interested in graphical models :

In brief, you can find the partial correlations between variables by standardizing the precision matrix (the inverse of the covariance matrix). $^1$ Using a form of regularization (e.g. LASSO ), you can shrink the smallest partial correlations to zero, such that variables with zero partial correlation can be considered conditionally independent. The remaining non-zero partial correlations can then form an undirected graph, which gives you a single, intuitive representation of which variables 'interact' with one another.

$^1$ This also has the interpretation of regressing all variables on each other, but with a single resulting network to interpret the results with.

I don't know of any SPSS implementations, but you can download $\textsf{R}$ for free and use the glasso package (or try rags2ridges for a ridge regularization approach).

• Friedman, Jerome and Hastie, Trevor and Tibshirani, Robert (2008): "Sparse inverse covariance estimation with the graphical lasso"
• $\begingroup$ Thank you for the elaboration. Given my limited knowledge to conduct anlayses beyond Multiple Regression, I believe it may be a wiser approach to explore the relationships among the Independent Var. By ignoring p-value > 0.05, do you imply to focus only on the beta standardized coefficient from Table 1? How do we construct a set of confidence intervals using the reported standard errors in table 1? Please provide an example. (2) How complicated is it to run a PLS and analyze the output in SPSS which I have access to? Can we use more than one dependent var in PLS? Thanks again $\endgroup$ –  Vyas Commented Sep 27, 2018 at 2:18
• 1 $\begingroup$ @Vyas You're welcome! I don't use SPSS, but you could calculate $t$-based confidence intervals as $\hat{\beta} \pm t_{1-\frac{\alpha}{2}, \text{df}} \cdot \text{SE}$. If you have enough degrees of freedom, this is almost the same as a normal-based confidence interval: $\text{Estimate} \pm 1.96 \cdot \text{standard error}$. $\endgroup$ –  Frans Rodenburg Commented Sep 27, 2018 at 2:31
• $\begingroup$ Also note that if you have new questions, you should post them separately, the comments are only for clarification. $\endgroup$ –  Frans Rodenburg Commented Sep 27, 2018 at 2:32
• $\begingroup$ Hi Frans, are you referring to the confidence intervals generated (coefficients first table as shown above) relating to each variable? Looking at each confidence interval value, except for BX_Adv_HQ variable, I note that the confidence intervals include zero, meaning that we can confidently state that there is no relation between X and Y. Is this how I can interpret the multiple regression results using confidence interval? Thank you $\endgroup$ –  Vyas Commented Sep 27, 2018 at 13:16
• $\begingroup$ Almost. You can state that there is no significant relationship between variables which CI includes the null-hypothesis ($\text{H}_0:\, \beta_j = 0$). However, you cannot confidently conclude that there is no relationship at all, because that isn't what the confidence interval shows. In fact, the true $\beta$ could be any value on (or even outside) the confidence interval. Have a look here about inference about the null-hypothesis: stats.stackexchange.com/q/85903/176202 . If you want to demonstrate that a variable has no effect, you would need a test of equivalence. $\endgroup$ –  Frans Rodenburg Commented Sep 28, 2018 at 1:14

Your Answer

Sign up or log in, post as a guest.

Required, but never shown

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy .

Not the answer you're looking for? Browse other questions tagged regression multiple-regression spss regression-coefficients exploratory-data-analysis or ask your own question .

• Featured on Meta
• Bringing clarity to status tag usage on meta sites
• We've made changes to our Terms of Service & Privacy Policy - July 2024
• Announcing a change to the data-dump process

Hot Network Questions

• Dropper post won't actuate. Is this fixable?
• How to get the value of a specified index number from the sorting of a column and fill it with null if missing?
• Automotive Controller LDO Failures
• The minimal Anti-Sudoku
• Why didn't Walter White choose to work at Gray Matter instead of becoming a drug lord in Breaking Bad?
• What are these on my cabbages?
• Does full erase create all 0s or all 1s on the CD-RW?
• What mode of transport is ideal for the cold post-apocalypse?
• Why is Excel not counting time with COUNTIF?
• Is there a plurality of persons in the being of king Artaxerxes in his use of the pronoun "us" in Ezra 4:18?
• "Undefined" when attempting analytical expression for a RegionIntersection and its Area in V14.0
• Package 'gettext' has no installation candidate, but package exists
• how to include brackets in pointform latex
• The meaning of "κ" in coordination compound
• "Not many can say that is there."
• Did Newton predict the deflection of light by gravity?
• Shift right by half a trit
• What are those bars in subway train or bus called?
• As a resident of a Schengen country, do I need to list every visit to other Schengen countries in my travel history in visa applications?
• Molecule that is placed on the equal sign
• Unstable output C++: running the same thing twice gives different output
• Could a 3D sphere of fifths reveal more insights than the 2D circle of fifths?
• Functional derivative under a path integral sign
• Using "where" to modify people

View the latest institution tables

View the latest country/territory tables

5 tips for dealing with non-significant results

Credit: Image Source/Getty Images

It might look like failure, but don’t let go just yet.

16 September 2019

Image Source/Getty Images

When researchers fail to find a statistically significant result, it’s often treated as exactly that – a failure. Non-significant results are difficult to publish in scientific journals and, as a result, researchers often choose not to submit them for publication.

This means that the evidence published in scientific journals is biased towards studies that find effects.

A study published in Science by a team from Stanford University who investigated 221 survey-based experiments funded by the National Science Foundation found that nearly two-thirds of the social science experiments that produced null results were filed away, never to be published.

By comparison, 96% of the studies with statistically strong results were written up .

“These biases imperil the robustness of scientific evidence,” says David Mehler, a psychologist at the University of Münster in Germany . “But they also harm early career researchers in particular who depend on building up a track record.”

Mehler is the co-author of a recent article published in the Journal of European Psychology Students about appreciating the significance of non-significant findings.

So, what can researchers do to avoid unpublishable results?

#1: Perform an equivalence test

The problem with a non-significant result is that it’s ambiguous, explains Daniël Lakens , a psychologist at Eindhoven University of Technology in the Netherlands .

It could mean that the null hypothesis is true – there really is no effect. But it could also indicate that the data are inconclusive either way.

Lakens says performing an ‘equivalence test’ can help you distinguish between these two possibilities. It can’t tell you that there is no effect, but it can tell you that an effect – if it exists – is likely to be of negligible practical or theoretical significance.

Bayesian statistics offer an alternative way of performing this test, and in Lakens’ experience, “either is better than current practice”.

#2 Collaborate to collect more data

Equivalence tests and Bayesian analyses can be helpful, but if you don’t have enough data, their results are likely to be inconclusive.

“The root problem remains that researchers want to conduct confirmatory hypothesis tests for effects that their studies are mostly underpowered to detect,” says Mehler.

This, he adds, is a particular problem for students and early career researchers, whose limited resources often constrain them to small sample sizes.

One solution is to collaborate with other researchers to collect more data. In psychology, the StudySwap website is one way for researchers to team up and combine resources.

#3 Use directional tests to increase statistical power

If resources are scarce, it’s important to use them as efficiently as possible. Lakens suggests a number of ways in which researchers can tweak their research design to increase statistical power – the likelihood of finding an effect if it really does exist.

In some circumstances, he says, researchers should consider ‘directional’ or ‘one-sided’ tests.

For example, if your hypothesis clearly states that patients receiving a new drug should have better outcomes than those receiving a placebo, it makes sense to test that prediction rather than looking for a difference between the groups in either direction.

“It’s basically free statistical power just for making a prediction,” says Lakens.

#4 Perform sequential analyses to improve data collection efficiency

Efficiency can also be increased by conducting sequential analyses, whereby data collection is terminated if there is already enough evidence to support the hypothesis, or it’s clear that further data will not lead to it being supported.

This approach is often taken in clinical trials where it might be unethical to test patients beyond the point that the efficacy of the treatment can already be determined.

A common concern is that performing multiple analyses increases the probability of finding an effect that doesn’t exist. However, this can be addressed by adjusting the threshold for statistical significance, Lakens explains.

#5 Submit a Registered Report

Whichever approach is taken, it’s important to describe the sampling and analyses clearly to permit a fair evaluation by peer reviewers and readers, says Mehler.

Ideally, studies should be preregistered. This allows authors to demonstrate that the tests were determined before rather than after the results were known. In fact, Mehler argues, the best way to ensure that results are published is to submit a Registered Report.

In this format, studies are evaluated and provisionally accepted based on the methods and analysis plan. The paper is then guaranteed to be published if the researchers follow this preregistered plan – whatever the results.

In a recent investigation , Mehler and his colleague, Chris Allen from Cardiff University in the UK , found that Registered Reports led to a much increased rate of null results: 61% compared with 5 to 20% for traditional papers.

First analysis of ‘pre-registered’ studies shows sharp rise in null findings

This simple tool shows you how to choose your mentors

Q&A Niamh Brennan: 100 rules for publishing in top journals

Sign up to the Nature Index newsletter

Get regular news, analysis and data insights from the editorial team delivered to your inbox.

• Sign up to receive the Nature Index newsletter. I agree my information will be processed in accordance with the Nature and Springer Nature Limited Privacy Policy .

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Section 4.1: Examining Relationships

Learning Objectives

At the end of this section you should be able to answer the following questions:

• What is a statistical relationship?
• What is the difference between a positive and negative relationship?
• What does a Pearson correlation coefficient indicate?

When we discuss relationships or associations between variables, the terms “relationship” and “association” mean that two variables change, or vary, together. However, just because two variables change together does not mean that this change is statistically significant. Thus, there are two types of variation or relationships – significant and nonsignificant.

As a researcher, looking into the relationships between psychological constructs can tell me a great deal about how certain mental health concerns and behaviours effects individuals with respect to behavioural and emotional levels.  For example, does a person with greater levels of social connection and support have greater feelings of well-being? Do people with greater levels of mindfulness have lower levels of perceived stress? These are questions that can be answered using correlational analyses.

So what is a statistical relationship? A statistical relationship is the association between two variables that is statistically significant.  This significance is based on the level of a probability test, which is a p-value in the case of Pearson correlation coefficients. If one variable increases or decreases, an associated variable will also show an increase or decrease, and it is statistically significant if this variability can be attributed to more than chance. An example is calorie intake and weight, as more calories are consumed, weight will likely increase. This type of example shows a positive relationship between the variables which means that as one variable increases the other also increases (e.g., as height increases, weight usually increases). However, relationships can be either positive or negative. A negative relationship is present when one variable increases, the other decreases (e.g., as stress levels increase, health will likely decrease).

A Pearson correlation coefficient (represented as an r value statistically) is a very useful tool in psychological research. However, there are many other types of correlation coefficients, such as the Spearman rank-order correlation coefficient, which is a nonparametric measure of association between two variables.

A Pearson correlation draws on a “line of best fit” that will be imposed through the two variables in the data to establish the relationship between two variables. Using the linear model, the Pearson’s correlation coefficient (which is represented by an r) , represents the strength of the association. This means that the distance from the data points show the line of best fit and how strongly the two variables are related. Mathematically, the Pearson correlation is calculated from the central tendency statistic of the mean and the standard deviation for each of the variables.  Have a look at the below illustration by clicking on the link labelled “Chapter Four – Line of Best Fit,”  which displays a graph with the Line of Best Fit for the two variables mental distress and physical illness.  The variables are in fact correlated with a significant Pearson correlation coefficient ( r = .472, p < .000).

PowerPoint: Line of Best Fit

Take a look at the following PowerPoint slides:

• Chapter Four – Line of Best Fit

The Pearson correlation coefficient can range from +1 to -1, with positive values indicating that as one value increases (e.g., as height increases, weight increases) the other also increases, or negative values which show that as one value increases, the other decreases (e.g., as stress increases health decreases). The stronger the association between the two variables, the closer to 1 the value will be.

Statistics for Research Students Copyright © 2022 by University of Southern Queensland is licensed under a Creative Commons Attribution 4.0 International License , except where otherwise noted.

Share This Book

My results were not significant… now what?

So, you have collected your data and conducted your statistical analysis, but all of those pesky p -values were above .05. You didn’t get significant results. Now you may be asking yourself, “What do I do now?” “What went wrong?” “How do I fix my study?”

One of the most common concerns that I see from students is about what to do when they fail to find significant results. They might be disappointed. They might be worried about how they are going to explain their results. They might panic and start furiously looking for ways to “fix” their study. Whatever your level of concern may be, here are a few things to keep in mind…

First, just know that this situation is not uncommon. Research studies at all levels fail to find statistical significance all the time. So if this happens to you, know that you are not alone.

Next, this does NOT necessarily mean that your study failed or that you need to do something to “fix” your results. Rest assured, your dissertation committee will not (or at least SHOULD not) refuse to pass you for having non-significant results. They will not dangle your degree over your head until you give them a p -value less than .05. Further, blindly running additional analyses until something turns out significant (also known as “fishing for significance”) is generally frowned upon.

Finally, and perhaps most importantly, failing to find significance is not necessarily a bad thing. Findings that are different from what you expected can make for an interesting and thoughtful discussion chapter. Specifically, your discussion chapter should be an avenue for raising new questions that future researchers can explore.

Need help with your research?

Schedule a time to speak with an expert using the calendar below.

Your discussion can include potential reasons why your results defied expectations. Maybe there are characteristics of your population that caused your results to turn out differently than expected. Or perhaps there were outside factors (i.e., confounds) that you did not control that could explain your findings. You will also want to discuss the implications of your non-significant findings to your area of research. Talk about how your findings contrast with existing theories and previous research and emphasize that more research may be needed to reconcile these differences. Lastly, you can make specific suggestions for things that future researchers can do differently to help shed more light on the topic. You might suggest that future researchers should study a different population or look at a different set of variables. If you conducted a correlational study, you might suggest ideas for experimental studies. You also can provide some ideas for qualitative studies that might reconcile the discrepant findings, especially if previous researchers have mostly done quantitative studies.

The bottom line is: do not panic. This happens all the time and moving forward is often easier than you might think.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.336(7634); 2008 Jan 5

Listen to the data when results are not significant

Unexpected non-significant results from randomised trials can be difficult to accept. Catherine Hewitt, Natasha Mitchell, and David Torgerson find that some authors continue to support interventions despite evidence that they might be harmful

When randomised controlled trials show a difference that is not statistically significant there is a risk of interpretive bias. 1 Interpretive bias occurs when authors and readers overemphasise or underemphasise results. For example, authors may claim that the non-significant result is due to lack of power rather than lack of effect, using terms such as borderline significance 2 or stating that no firm conclusions can be drawn because of the modest sample size. 3 In contrast, if the study shows a non-significant effect that opposes the study hypothesis, it may be downplayed by emphasising the results are not statistically significant. We investigated the problem of interpretive bias in a sample of recently published trials with findings that did not support the study hypothesis.

Why interpretive bias occurs

A non-significant difference between two groups in a randomised controlled trial may have several explanations. The observed difference may be real and the study is underpowered or the observed difference may occur simply by chance. Bias can also produce a non-significant difference, but we will not include this in our discussion below.

Trialists are rarely neutral about their research. If they are testing a novel intervention they usually suspect that it is effective otherwise they could not convince themselves, their peers, or research funders that it is worth evaluating. This lack of equipoise, however, can affect the way they interpret negative results. They have often invested a large amount of intellectual capital in developing the treatment under evaluation. Naturally, therefore, it is difficult to accept that it may be ineffective.

A trial with statistically significant negative results should, generally, overwhelm any preconceptions and prejudices of the trialists. However, negative results that are not statistically significant are more likely to be affected by preconceived notions of effectiveness, resulting in interpretive bias. This interpretive bias may lead authors to continue to recommend interventions that should be withdrawn.

Extent of problem

To assess the effect of interpretive bias in trials we hand searched studies published in the BMJ during 2002 to 2006 for trials that showed a non-significant difference in the opposite direction of that hypothesised. Two researchers (CEH, NM) identified the papers with a P value above 0.05 and below 0.3 on the primary outcome, which they agreed between them. Our choice of the limits for the P value was arbitrary and was driven by our decision to identify trials where there was an unexpected difference that could potentially be important and not be statistically significant because of lack of statistical power (type II error).

The decision to use a P value of 0.05 or a 95% confidence interval to determine statistical significance is arbitrary but widely accepted. 4 Ideally, we should judge the findings of a study not only on its statistical significance but in terms of its relative harms and benefits. Statistical significance is important, however, to guide us in the interpretation of a study’s results.

We found 17 papers where there was a difference between the two groups and this difference had a P value of between 0.05 and 0.30. Of these 17 trials, seven (table 1 ​ 1) ) showed differences in the opposite direction to that specified by the hypothesis.

 Trials with negative non-significant results published in BMJ, 2002-6

*67% and 51% confidence intervals were calculated from the data presented in the articles not from the original raw data.

We calculated three confidence intervals for each identified trial: 95%, 67%, and 51%. We chose 67% as this is half of 95% (that is, the z value for the 67% confidence interval is about half the z value for the 95% interval) and 51% because this range shows where, more often than not, the true treatment estimate will lie. Obviously, each value within the confidence interval is not equally plausible. Values that are close to the point estimate are more likely to correspond to the true value than estimates towards the extreme of the confidence interval.

We used the information in the box in each paper entitled “What this study adds” to determine whether the authors recommended the intervention. We then assessed the data in the paper and used the three confidence intervals to make our recommendation. The authors seem to recommend that the intervention should or could be used in four studies (table 2 ​ 2). ). We disagreed with this conclusion for three of these studies and were unsure for the other one, as discussed below.

 Interpretation of trials with non-significant negative results published in BMJ

Sex education programme for 13-15 year olds

Twenty five schools in Scotland were randomised to receive either normal sex education or an enhanced package. 5 The trial was powered to show a 33% reduction in termination rates and had over 99% follow-up after 4.5 years. The intervention schools had an increase of 15.7 terminations per 1000 compared with the control schools (P=0.26). Although the 95% confidence intervals did not exclude an 11% decrease in terminations, they included a 42% increase in terminations. The 67% confidence intervals did not pass through zero, thus on balance the intervention was more likely to be associated with an increase in terminations than a decrease. The cost of the intervention was up to 45 times greater than usual sex education.

To support use of the intervention the authors refer to an earlier report that “pupils and teachers preferred the SHARE programme . . . It also increased pupils’ knowledge of sexual health . . . and had a small but beneficial effect on beliefs about alternatives to sexual intercourse and intentions to resist unwanted sexual activities and to discuss condoms with partners.” Although the authors admit that the programme “was not more effective than conventional provision,” they do not discuss the possibility that the increase in termination rates might be real and that the programme should be withdrawn until further research supported its implementation. Indeed, the Scottish Executive supports its use in Scottish schools.

Providing free child safety equipment to prevent injuries

A total of 3428 families were randomised to provide 80% power to show a 10% reduction in medically attended injuries. 9 Free safety equipment was offered to families living in deprived areas along with advice from health visitors. Data on injuries attended in primary care were available for >80% of participants and secondary care >92%. There was an increased risk of having medically attended injuries in the intervention group (P=0.08). The 67% confidence intervals suggested that on balance the most likely value for the true effect is to increase the risk of injuries. The intervention is associated with increased cost and increased risk.

Despite this, the authors seem to use proxy measures of outcome as justification for the intervention: “Our findings in relation to safety practices and degrees of satisfaction are encouraging for safety equipment schemes such as those organised by SureStart.” The authors also note that it was unlikely that intervention would not reduce injury rates because “several observational studies have shown a lower risk of injury among people with a range of safety practices.” Observational studies are potentially biased, which is one of the main reasons we do randomised trials. It is, therefore, surprising to seek reassurance from non-randomised data when a randomised trial shows the “wrong” result. The authors suggest that bias could have been introduced because of differential raised parental awareness, although they acknowledge that the intervention could have increased injury through the process of risk compensation.

Oral misoprostol for induction of labour

In this trial, 741 pregnant women with an indication for prostaglandin induction of labour were randomised to oral misoprostol or vaginal dinoprostone gel. 10 The trial was powered to show a 30% difference in vaginal birth after 24 hours. Follow-up rates were 100% in both groups, allocated treatment adherence was greater than 99%. 46% of women in the oral misoprostol group did not achieve a vaginal birth within 24 hours compared with 41% of the vaginal dinoprostone group. The 95% confidence intervals suggested, at best, the intervention could be associated with 0.95 relative risk improvement.

The authors stated that there was no difference between the two treatments but women preferred oral treatment. However, the 67% confidence interval was significant, suggesting that oral treatment increased the risk of delayed vaginal birth. We could not make a definite recommendation because the risk of caesarean section was reduced for the intervention group (0.82, P=0.13), and the 67% confidence interval (0.73 to 0.91) on this outcome favours the intervention.

Lidocaine spray to reduce pain during vaginal delivery

This trial randomised 185 women to receive a topically applied anaesthetic spray or placebo. 11 The primary outcome was pain during delivery. Follow-up was 100% at delivery. The pain on delivery was increased by 4.8 points in the intervention group, although the 95% confidence intervals suggested that it could reduce pain by 1.7 points or increase it by 11.2 points. The 67% interval suggested that the true difference was an increase in pain. An adjusted analysis suggested a bigger difference in pain scores. Therefore, this intervention should not be used.

Acting on evidence

Randomised trials are usually considered the best method of establishing effectiveness. All of the trials we identified were well designed and powered to test a two tailed hypothesis, which by implication accepts that the intervention could cause benefit or harm. The results on proxy measures or from observational studies cannot justify ignoring the main results of the trial.

The use of measures of uncertainty, such as confidence intervals, inform the need for further research not necessarily policy decisions. The Scottish Executive implemented the sex education programme described above on the basis of proxy markers of effect. The main follow-up has been completed. The decision should now be made on a combination of effectiveness and costs. We know the point estimate favours the control group, and we know that on balance when we examine both the 67% and 51% confidence intervals that the likely true estimate of effect is an increase in terminations , and finally we know the costs also favour the control group. The logical interpretation, therefore, of this evidence is to withdraw this programme until further research shows another sex education programme is effective at reducing unwanted pregnancies. A similar argument applies to the accident prevention programme.

Journal editors, readers, and authors need to listen to the data presented in the paper. Sometimes the data speak clearly. Often, however, the data speak more softly and we must be more careful in our interpretation. Journal editors and peer reviewers have an important role in making sure that authors do not make recommendations that are not supported by the data presented.

Summary points

• Bias can occur when interpreting randomised controlled trials that produce unexpected results that are not statistically significant
• Some authors seem to support interventions despite evidence that they might be ineffective
• Authors should be careful when they interpret non-significant negative results

We thank Nick Freemantle, Cathie Sudlow, and BMJ editors for helpful comments.

Contributors and sources: DJT has published widely on the design of randomised controlled trials in health care and education. This article arose from discussions around interpretation bias within a single study. DJT suggested the idea of the study. CEH and NM identified the studies and extracted the data. All authors interpreted the data. All authors drafted and revised the manuscript critically. All authors give final approval of the version to be published. DJT is guarantor.

Competing interests None declared.

Provenance and peer review: Not commissioned; externally peer reviewed.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Pearson Correlation Coefficient (r) | Guide & Examples

Pearson Correlation Coefficient (r) | Guide & Examples

Published on May 13, 2022 by Shaun Turney . Revised on February 10, 2024.

The Pearson correlation coefficient ( r ) is the most common way of measuring a linear correlation. It is a number between –1 and 1 that measures the strength and direction of the relationship between two variables.

Table of contents

What is the pearson correlation coefficient, visualizing the pearson correlation coefficient, when to use the pearson correlation coefficient, calculating the pearson correlation coefficient, testing for the significance of the pearson correlation coefficient, reporting the pearson correlation coefficient, other interesting articles, frequently asked questions about the pearson correlation coefficient.

The Pearson correlation coefficient ( r ) is the most widely used correlation coefficient and is known by many names:

• Pearson’s r
• Bivariate correlation
• Pearson product-moment correlation coefficient (PPMCC)
• The correlation coefficient

The Pearson correlation coefficient is a descriptive statistic , meaning that it summarizes the characteristics of a dataset. Specifically, it describes the strength and direction of the linear relationship between two quantitative variables.

Although interpretations of the relationship strength (also known as effect size ) vary between disciplines, the table below gives general rules of thumb:

The Pearson correlation coefficient is also an inferential statistic , meaning that it can be used to test statistical hypotheses . Specifically, we can test whether there is a significant relationship between two variables.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Another way to think of the Pearson correlation coefficient ( r ) is as a measure of how close the observations are to a line of best fit .

The Pearson correlation coefficient also tells you whether the slope of the line of best fit is negative or positive. When the slope is negative, r is negative. When the slope is positive, r is positive.

When r is 1 or –1, all the points fall exactly on the line of best fit:

When r is greater than .5 or less than –.5, the points are close to the line of best fit:

When r is between 0 and .3 or between 0 and –.3, the points are far from the line of best fit:

When r is 0, a line of best fit is not helpful in describing the relationship between the variables:

The Pearson correlation coefficient ( r ) is one of several correlation coefficients that you need to choose between when you want to measure a correlation. The Pearson correlation coefficient is a good choice when all of the following are true:

• Both variables are quantitative : You will need to use a different method if either of the variables is qualitative .
• The variables are normally distributed : You can create a histogram of each variable to verify whether the distributions are approximately normal. It’s not a problem if the variables are a little non-normal.
• The data have no outliers : Outliers are observations that don’t follow the same patterns as the rest of the data. A scatterplot is one way to check for outliers—look for points that are far away from the others.
• The relationship is linear: “Linear” means that the relationship between the two variables can be described reasonably well by a straight line. You can use a scatterplot to check whether the relationship between two variables is linear.

Pearson vs. Spearman’s rank correlation coefficients

Spearman’s rank correlation coefficient is another widely used correlation coefficient. It’s a better choice than the Pearson correlation coefficient when one or more of the following is true:

• The variables are ordinal .
• The variables aren’t normally distributed .
• The data includes outliers.
• The relationship between the variables is non-linear and monotonic.

Below is a formula for calculating the Pearson correlation coefficient ( r ):

The formula is easy to use when you follow the step-by-step guide below. You can also use software such as R or Excel to calculate the Pearson correlation coefficient for you.

Step 1: Calculate the sums of x and y

Start by renaming the variables to “ x ” and “ y .” It doesn’t matter which variable is called x and which is called y —the formula will give the same answer either way.

Next, add up the values of x and y . (In the formula, this step is indicated by the Σ symbol, which means “take the sum of”.)

Σ x = 3.63 + 3.02 + 3.82 + 3.42 + 3.59 + 2.87 + 3.03 + 3.46 + 3.36 + 3.30

Σ y = 53.1 + 49.7 + 48.4 + 54.2 + 54.9 + 43.7 + 47.2 + 45.2 + 54.4 + 50.4

Step 2: Calculate x 2 and y 2 and their sums

Create two new columns that contain the squares of x and y . Take the sums of the new columns.

Σ x 2  = 13.18 + 9.12 + 14.59 + 11.70 + 12.89 +  8.24 +  9.18 + 11.97 + 11.29 + 10.89

Σ x 2  = 113.05

Σ y 2  = 2 819.6 + 2 470.1 + 2 342.6 + 2 937.6 + 3 014.0 + 1 909.7 + 2 227.8 + 2 043.0 + 2 959.4 + 2 540.2

Step 3: Calculate the cross product and its sum

In a final column, multiply together x and y (this is called the cross product). Take the sum of the new column.

Σ xy = 192.8 + 150.1 + 184.9 + 185.4 + 197.1 + 125.4 + 143.0 + 156.4 + 182.8 + 166.3

Step 4: Calculate r

Use the formula and the numbers you calculated in the previous steps to find r .

Prevent plagiarism. Run a free check.

The Pearson correlation coefficient can also be used to test whether the relationship between two variables is significant .

The Pearson correlation of the sample is r . It is an estimate of rho ( ρ ), the Pearson correlation of the population . Knowing r and n (the sample size), we can infer whether ρ is significantly different from 0.

• Null hypothesis ( H 0 ): ρ = 0
• Alternative hypothesis ( H a ): ρ ≠ 0

To test the hypotheses , you can either use software like R or Stata or you can follow the three steps below.

Step 1: Calculate the t value

Calculate the t value (a test statistic ) using this formula:

Step 2: Find the critical value of t

You can find the critical value of t ( t* ) in a t table. To use the table, you need to know three things:

• The degrees of freedom ( df ): For Pearson correlation tests, the formula is df = n – 2.
• Significance level (α): By convention, the significance level is usually .05.
• One-tailed or two-tailed: Most often, two-tailed is an appropriate choice for correlations.

Step 3: Compare the t value to the critical value

Determine if the absolute t value is greater than the critical value of t . “Absolute” means that if the t value is negative you should ignore the minus sign.

Step 4: Decide whether to reject the null hypothesis

• If the t value is greater than the critical value, then the relationship is statistically significant ( p <  α ). The data allows you to reject the null hypothesis and provides support for the alternative hypothesis.
• If the t value is less than the critical value, then the relationship is not statistically significant ( p >  α ). The data doesn’t allow you to reject the null hypothesis and doesn’t provide support for the alternative hypothesis.

If you decide to include a Pearson correlation ( r ) in your paper or thesis, you should report it in your results section . You can follow these rules if you want to report statistics in APA Style :

• You don’t need to provide a reference or formula since the Pearson correlation coefficient is a commonly used statistic.
• You should italicize r when reporting its value.
• You shouldn’t include a leading zero (a zero before the decimal point) since the Pearson correlation coefficient can’t be greater than one or less than negative one.
• You should provide two significant digits after the decimal point.

When Pearson’s correlation coefficient is used as an inferential statistic (to test whether the relationship is significant), r is reported alongside its degrees of freedom and p value. The degrees of freedom are reported in parentheses beside r .

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Chi square test of independence
• Statistical power
• Descriptive statistics
• Degrees of freedom
• Null hypothesis

Methodology

• Double-blind study
• Case-control study
• Research ethics
• Data collection
• Hypothesis testing
• Structured interviews

Research bias

• Hawthorne effect
• Unconscious bias
• Recall bias
• Halo effect
• Self-serving bias
• Information bias

You should use the Pearson correlation coefficient when (1) the relationship is linear and (2) both variables are quantitative and (3) normally distributed and (4) have no outliers.

You can use the cor() function to calculate the Pearson correlation coefficient in R. To test the significance of the correlation, you can use the cor.test() function.

You can use the PEARSON() function to calculate the Pearson correlation coefficient in Excel. If your variables are in columns A and B, then click any blank cell and type “PEARSON(A:A,B:B)”.

There is no function to directly test the significance of the correlation.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Turney, S. (2024, February 10). Pearson Correlation Coefficient (r) | Guide & Examples. Scribbr. Retrieved August 12, 2024, from https://www.scribbr.com/statistics/pearson-correlation-coefficient/

Is this article helpful?

Shaun Turney

Other students also liked, simple linear regression | an easy introduction & examples, coefficient of determination (r²) | calculation & interpretation, hypothesis testing | a step-by-step guide with easy examples, what is your plagiarism score.

Shanker Blog

In research, what does a "significant effect" mean.

If you follow education research – or quantitative work in any field – you’ll often hear the term “significant effect." For example, you will frequently read research papers saying that a given intervention, such as charter school attendance or participation in a tutoring program, had “significant effects," positive or negative, on achievement outcomes.

This term by itself is usually sufficient to get people who support the policy in question extremely excited, and to compel them to announce boldly that their policy “works." They’re often overinterpreting the results, but there’s a good reason for this. The problem is that “significant effect” is a statistical term, and it doesn’t always mean what it appears to mean. As most people understand the words, “significant effects” are often neither significant nor necessarily effects.

Let’s very quickly clear this up, one word at a time, working backwards.

In education research, the term “effect” usually refers to an estimate from a model, such as a regression. For example, I might want to see how education influences income, but, in order to isolate this relationship, I need to control for other factors that also affect income, such as industry and experience. Put more simply, I want to look at the average relationship between education and income among people who have the same level of experience, work in the same industry and share other characteristics that shape income. That quantified relationship – usually controlling for a host of different variables - is often called an "effect."

But we can’t randomly assign education to people the way we would a pharmaceutical drug. And there are dozens of interrelated variables that might affect income, many of which, such as ability or effort, can’t even be measured directly.

In good models using large, detailed datasets with a thorough set of control variables, a statistically significant “effect” might serve as pretty good tentative evidence that there is a causal relationship between two variables – e.g., that having more education leads to higher earnings, at least to some degree, all else being equal. Sometimes, it’s even possible for social scientists to randomly assign “treatment” (e.g., merit pay programs), or exploit this when it happens (e.g., charter school lotteries). One can be relatively confident that the results from studies using random assignment, assuming they're well-executed, are not only causal per se , but also less likely to reflect bias from unmeasured influences. Even in these cases, however, there are usually validity-related questions left open, such as whether a program’s effect in one context/location will be the same elsewhere.

So, in general, when you hear about “effects," especially those estimated without the benefit of random assignment, it's best to think of them as relationships or associations that are often (but not nearly always) causal to some extent, though the estimate of that association’s size varies in its precision, and the degree to which it reflects the influence of unmeasured factors.

Then there’s the term “significant." “Significant” is of course a truncated form of “statistically significant." Statistical significance means we can be confident that a given relationship is not zero . That is, the relationship or difference is probably not just random “noise." A significant effect can be either positive (we can be confident it’s greater than zero) or negative (we can be confident it’s less than zero). In other words, it is “significant” insofar as it’s not nothing. The better way to think about it is “discernible." There’s something there.

In our education/income example, a “significant positive effect” of education on income means that one can be confident that, on average, more educated people earn more than people with less education, even when we control for experience, industry and, presumably, a bunch of other variables that might be associated with income.

(Side note: One can also test for statistical significance of simpler relationships that are not properly called "effects," such as whether there is a difference between test scores in one year compared with a prior year.)

Most importantly, as I mentioned in a previous post , an “effect” that is statistically significant is not necessarily educationally meaningful . Remember – significant means that the relationship is not zero, but that doesn’t mean it’s big or even moderate. Quite often, “significant” effects are so small as to be rather meaningless, especially when using big datasets. You need to check the size of the "effect," the proper interpretation of which depends on the outcome used, the type and duration of “treatment” in question and other factors.

For example, today's NAEP results indicated a "significant increase" in fourth and eighth grade math and eighth grade reading, but in all three cases, the increase was as modest as it gets - just one scale score point, roughly a month of "learning." Certainly, this change warrants attention, but it may not square with most people's definition of "significant" (and it may also reflect differences in the students taking the test ).

So, summing up, a when you hear that something has a “statistically significant effect” on something else, remember that it’s not necessarily significant or an effect in the common use of those words. It’s best to think of them as “statistically discernible relationships." They can be big or small, they’re not necessarily causal, and they can vary widely in terms of precision.

- Matt Di Carlo

Great post. I think another thing to consider, unfortunately, is that interpretation of effects often requires some background knowledge of the phenomena being studied. Even other scientists, who can look for effect size, and know that when you have census data, everything is "significant," can still misinterpret things like NAEP scores, where one needs to have an understanding of the history of the test, the content, and the testing situation. But definitely, I am on board with urging caution and humility in interpreting effects.

Thanks for explaining this, I can use this information in conversation and debates. I even have to keep reminding myself not to fall for the rhetoric of people using their ideology to mislead the public with authoritarian, "scientific facts".

And - the size of an effect shouldn't be compared to doing nothing in a schools context as an acton is invariably taken instead of some other possible approach. Educational researchers ought to care about an effect that is greater than the stuff a teacher/school normally does. It's unusual for a set of teachers to do something new and it not to have an effect of some kind. The important question is; is this approach better than the other possible approaches we know about for improving boys reading, for example. Comparison with best known strategy is what good medical research undertakes. There really isn't any point in demonstrating that a particular strategy improves boys reading unless the strategy is better than the best one presently known.

A look at what makes the effect size meaningful (as opposed to significant) one may check here: http://www2.ed.gov/rschstat/eval/choice/implementation/achievementanaly… and here: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.126.8383&rep=r…

And even when some studies do everything right, and do find significant effects, whether that makes a big difference in reality is another question altogether. Check this one. http://ies.ed.gov/ncee/pubs/20114001/pdf/20114002.pdf

What's your view on Professor Hattie's table 'effect sizes'? Clearly he has gone further than claiming interventions are more then merely 'significant effects' and has tried to quantify these effects. I can accept that there are limits to the type of meta-analyses he conducts, but does that mean we can't trust his research at all? How else should we make decisions on what's effective and what's not?

• What it means when “no significant differences were found”

It means nothing. Really.

[Note: Perhaps this is part two of “ zero, naught, and nothing ” because it is related to the concepts introduced in that section.]

In research, participants are divided up into two (or more) groups, and theoretically, at least, they are randomly assigned to those groups. One is a control group and the other is an intervention or experimental group. Some type of intervention is applied (administration of a type of therapy or drug under investigation, for example) to the experimental group, and the outcome measures of the two groups are compared to see if the difference is significant . Those two words are critical, because they are the key to understanding this type of research.

How do we know if there is a significant difference?

Let’s start with an actual example from the academic literature. Some well-known, researchers for whom I have the highest respect (e.g., Mitchell Krucoff, Frank Oz, Harold Koenig, and others) wanted to study prayer to see if they could scientifically demonstrate an effect of prayer on heart patients who underwent surgery (i.e., cardiac catheterization). In all fairness, these researchers were pioneers in their efforts to scientifically validate the spiritual dimension (which I will be blogging about soon). Their design was fairly simple—divide cardiac patients in two groups, and enroll prayer groups to pray for the experimental group. However, maintaining anonymity of the patients was a factor, so the researchers negotiated that the prayed-for patients would be identified only by their first name and last initial; “John Smith” would be “John S.” No picture or other identifying information would be transmitted to the three groups around the country who were praying daily for the experimental-group cardiac patients.

No significant differences were found between the patients who were prayed for and the control group. The researchers concluded that “prayer was not effective” in cardiac patients. This research was published in one of the most prestigious medical journals, The Lancet . I read the article the day after it was published, and immediately called one of my colleagues. We decided to write a letter to the editors to point out that the conclusion was not possible; the letter was published.

In other words, what was done here was the equivalent of testing batteries by putting them in a flashlight and turning on the switch. If the flashlight lights up, then we can  conclude that the batteries are good.  This is the equivalent of finding a statistically significant difference (the bulb was off, then it turned on).

But what if the flashlight did not light up? In that case, we could not conclude that the batteries are not good. Why not? We do not know if the flashlight works; it could be that there is not a good contact with the batteries, or the bulb is burned out, or the switch does not work. It is the equivalent of failing to find a significant difference in research; just as we cannot conclude that the batteries are not good, we cannot conclude that there is no difference between the two groups. Our only conclusion is: all possibilities remain.

[This part gets academic-boring-technical, so if you want to cut to the chase scroll on down to the brackets below.]

In research, how do we know if there’s a statistically significant difference? That’s where the use of statistical methodology comes in to play. Scientists use statistical tests to examine the differences between the two groups to see how likely it is that those results could have been derived by chance. For example, if we flip a coin 20 times and get 12 heads, how likely is that? If two people each flip a coin 20 times and one person (let’s call her “control”) records 13 heads and another person (let’s call him “experimental”) gets 10, are those results what we could normally expect, given the odds of getting these results by chance?

It’s not as simple as it looks. We can’t just say the two groups need to have a difference of, say, 10% in order to have a significant difference. This is because the results could be grouped together. For example, if we are comparing a control group that receives a placebo, and the experimental group receives a new high-tech drug to increase the rate of hair growth, then we can measure the hair growth in each group and compare them. So if the average growth of hair in the control group is 10 cm and the average growth in the intervention group is 12 cm we don’t know if the difference is meaningful or due to factors of chance. One factor that comes into play is how “spread out” the rates of growth are between members of the two groups. We might have a significant overlap between actual growth rates of members in the two groups.

An average of 10cm vs. an average of 12cm might look almost the same, however, if the two groups are spread out widely. And having 4 people in each group is different from having 1000 people in each one.

The “spread-outness” of the data has a name: we call it variance or standard deviation (standard deviation is the square root of the variance, so these are related closely enough statistically that we can use them interchangeably). Statistical formulae that are used to determine whether the difference between two groups is statistically different must account for the number of people in each group, the variance, and the actual difference between the averages for the groups.

But we have to back up just a second and talk about “hypothesis testing.” The scientist has a null hypothesis and an alternate hypothesis. The scientist begins with the null, which in this case is “There is no difference between the control group and the experimental group.” The alternate hypothesis, then, would be “there is a significant difference between the two groups” (and therefore the intervention has an effect). If the researcher finds a statistically significant difference between the two groups, he or she rejects the null and accepts the alternate hypothesis. But if the researcher fails to find a difference between the two groups, then the only conclusion that can be made is that “all possibilities remain.”

[OK. End of academic speak. What it really means when “No significant differences were found.”]

The trick is what happens when the differences between the groups aren’t large enough to find a statistically significant difference. Perhaps the two groups overlap too much, or there just aren’t enough people in the two groups to establish a significant difference; when the researcher fails to find a significant difference, only one conclusion is possible: “all possibilities remain.” In other words, failure to find a significant difference means that nothing was found. So it means nothing. Really.

(c) 2012 Charles L. McLafferty, Jr.

Leave a Reply Cancel reply

You must be logged in to post a comment.

• Search for:

Recent Posts

• New York Times: “Study Links Autism with Antidepressant Use During Pregnancy”–Misleading or just plain wrong?
• Nature versus nurture: Questioning a cornerstone of psychology
• Is the Saffir-Simpson Hurricane Scale Outdated (When are Statistics too Simple)?
• How can we study what we can’t measure?

Recent Comments

• November 2012
• September 2012
• January 2012
• September 2011
• January 2011
• December 2010
• November 2010
• October 2010
• assumptions
• bell-shaped curve
• Car Fax report
• certification
• credit history
• dimensionality
• experimental method
• financial reporting
• grade inflation
• Great Delusion
• human dimensions
• hypothesis testing
• mixed methods research
• nature vs nurture
• noëtic dimension
• normalization of scores
• qualitative research
• quantitative research
• research methods
• standard deviation
• statistical significance
• Uncategorized
• Entries feed
• Comments feed
• WordPress.org

Conor Murphy

STOR-i PhD Student

The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant

This short paper caught my eye recently when scouring the internet for something interesting to (attempt to) explain clearly in my first blog post. When I initially read the title, I was a bit shocked as thoughts rushed through my mind such as “All of those modules where I learned about p-values and statistical significance never mentioned this fairly crucial fact!”. After a few breaths, I began to read it and of course, realised the paper is not discounting this widely-used method of determining the validity of a variable in a model but it is simply making known a common error often made when using this method for comparisons.

Introduction

This common statistical error comes about when comparisons are summarised by declarations of statistical significance and results are sharply distinguished between “significant” and “not significant”. The reason this is important is that changes in statistical significance are not themselves statistically significant. The significance level of a quantity can be changed largely by a small ( non-significant ) change in some statistical quantity such as a mean or regression coefficient.

Quick Example

As a simple example, say, we have run two independent studies in different areas to determine the number of days/nights people had spent inside in the last month when compared to the same month in 2019, i.e looking at the effect of lockdown/Covid-19 on the amount of days/nights a person spends inside. Now, say, we obtained effect estimates of 27 in study 1 and 12 in study 2 with respective standard errors of 12.5 and 12 . The first study would be statistically significant while the second would not. A naive conclusion to make here but one that might be tempting is to declare that there is a large difference between the two studies. Unfortunately, this difference is certainly not statistically significant with an estimated difference of 15 and standard error of \sqrt{12.5^2 + 12^2} = 17.3 .

In the paper, they also explain how it can be problematic to compare estimates with different levels of information. Say, there was another independent study conducted with a far larger sample size and the effect estimate obtained was 2.7 with a standard error of 1.2 . This study could now obtain the same significance level as in study 1 but the difference between the two is significant! If we focussed just on significance, we might say study 1 and 3 replicate each other but looking at the effect estimated, clearly this is not true.

This is dangerous as “significance” often aids decision making and conclusions could be made based on the first study while disregarding the second, when actually the two don’t differ significantly from one another. As the paper explains, one way of interpreting this lack of statistical significance is that further information might change the conclusion/decision.

In essence, the paper is urging one to err on the side of caution when interpreting significance. Essentially, comparing statistical significance levels is not a good idea and one should look at the significance of the difference and not the difference in significance.

I hope you found this post interesting. If you’d like to read the full paper, see the link below and feel free to leave a comment (even if just to say you never want to hear the word “significance” again!).

The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant – Andrew GELMAN and Hal STERN

• Next Efficient Experimental Design
• Previous Hello world!

You may also like...

• The Monty Hall Problem

• Data Farming

• Dependence in Extremes

Recent Posts

• Efficient Experimental Design

• Bipolar Disorder
• Therapy Center
• When To See a Therapist
• Types of Therapy
• Best Online Therapy
• Best Couples Therapy
• Managing Stress
• Sleep and Dreaming
• Understanding Emotions
• Self-Improvement
• Healthy Relationships
• Student Resources
• Personality Types
• Sweepstakes
• Guided Meditations
• Verywell Mind Insights
• 2024 Verywell Mind 25
• Mental Health in the Classroom
• Editorial Process
• Meet Our Review Board
• Crisis Support

Do We Have the Same Relationship Non-Negotiables? An Investigation

What my mom considers pickiness, I consider non-negotiable

Justin Pumfrey/ Getty Images

Disrespect Is a Big No No

• What's a Relationship Without Trust?

Have the Same (Or Similar) Goals

Spirituality, religion, politics, and personal values, identifying your non-negotiables.

• Communicating and Honoring Non-negotiables

My mother says I've become incredibly picky about the partners I've been choosing. After much therapy and self-reflection, I made a list of red flags that, if ever presented, would immediately end the relationship—no ifs, ands, or buts. What my mother considers pickiness, I consider non-negotiable. 

These red flags are not open to discussion or reconsideration—aka they’re non-negotiable. Being viewed as picky is the least of my concerns; I want a healthy relationship, and studies show self-awareness plays a significant part. But, identifying non-negotiable is only half the battle. Communicating and reinforcing them seals the deal, and advocating for yourself is not as easy as it looks.

Which is why I tagged some experts and got their advice on how to identify and communicate your non-negotiables and because I like to do my due diligence, I also asked a few folks about their relationship non-negotiables. Because we all don't have the same boundaries...right? I mean, what's intolerable across the board, and what's a preference? The answers to these questions (and more) are just a few paragraphs away.

Respect is *always* a must. According to a 2002 study, mutual respect is not just necessary; it's crucial for problem-solving and relationship resilience. Caylia Wallace , 28, says mutual respect is her non-negotiable because of previous experiences where her partners disrespected her and created “volatile situations.”

A disrespectful partner is no joke. Who wants to feel small, belittled, invisible, and unheard by the person you love (and who's  supposed to love you)? I pray I never find a love like that.

What's a Relationship Without Trust?

“Without trust, you have nothing.” It may be overstated, but this quote still rings true. It's why trust is lifestyle content creator Melody Njoku's , 27, non-negotiable. For without it, there's no connection or intimacy, she says.

While Njoku holds trust as one of her core relationship values, the same can't be said for everyone else. Trust may be the foundation of a relationship, but nearly a third of adults have trust issues.

Look—I get it. We've all been burned before in dating, and it's hard not to hold onto past fears in new relationships. But I'm with Njoku on this one. Trust issues are a death sentence to any relationship—they brew resentments, doubts, and suspiciousness.

We all have different goals—some of us dream of buying homes (yes, even in this economy!). Others want to jet-set around the globe and discover life's treasures. Whatever your goals are, it needs to align with your significant other.

Sure, you can compromise. But you can't go half on a baby or bargain on a marriage license. Some things don't have a middle ground—and that's OK. Don't force your partner to choose between their goals and the relationship because resentment, tension, and arguments can brew.

A person's value system is another factor that plays a huge role in relationships. Most people date others with similar beliefs , whether that's political, religious, or spiritual.

For digital creator Danteé Ramos , 30, her partners must “believe in something” for there to be a connection or relationship. And she's not the only one—research shows that people form relationships based on political homogeneity. The same goes for religion, with one study reporting that romantic relationships with religious homogeneity were more likely to last longer than their interfaith counterparts.

You might not have these negotiable, but you do have some. And you don't have to think too hard to discover them, says Charese L. Josie , a licensed clinical social worker.

People already know what their non-negotiables are, but they rely more on the hope that they can get over that and make a relationship work.

Early on, we might ignore warning signs because we believe it's too early to judge . (Hint: it's really not, tbh). But here's the thing: those warning signs aren't going away; they're just turning into relationship issues and conflict, Josie says.

Be serious about setting strict boundaries and not letting things fly. Trust your gut, but also have outlined and written non-negotiables.

A great starting point to developing your own negotiables is by “understanding your principles, morals, wants, and needs,” says relationship and sex therapist Nikquan Lewis , MS, LMFT. Josie and Lewis recommend asking yourself the following questions:

• What do I know is a violation of who I am?
• What causes me anxiety?
• What is my belief system on children, pets, careers, and lifestyles? 
• What makes me feel emotionally safe? Physically safe?

Communicating and Honoring Non-negotiables 

Now that you've identified your non-negotiables, enforce them. What's the first towards doing that? Talking with your partner. Will these be awkward and uncomfortable conversations? Potentially. But, remember these conversations are for your  benefit.

If you can, try to hold these conversations in the early stages of dating, Lewis says. “When you communicate where you stand and where your values and principles lie—plus ask what theirs are—you can identify if there's alignment,” she explains. “Alignment is key in healthy relationships.” Getting to know someone and establishing deal breakers early helps prevent potentially toxic and painful relationships. Dating is already hard—let's not make it any harder.

Enforcing boundaries isn't a one-way street. Your partner also has deal breakers that you must respect.

“When you don't respect someone's non-negotiables, you are setting yourself up to be in a relationship where your partner's needs won't be met, and that's a problem,” Lewis says.

Relationships are give and take. When there's too much sacrifice on either end, a non-negotiable—a boundary—gets crossed. That's why it is important to identify your non-negotiables, communicate them early on, and ask your partner to do the same.

And when you deliver said boundaries, “be mindful” of your tone, approach, and delivery , Lewis says. You want to create a safe space for you and your partner to verbalize your needs. And nothing's unsafer than an angry tone or hostile body language .

Amazing—now we understand what non-negotiables are, why they're essential, and how to enforce them. The only advice left to share is to reflect periodically on your non-negotiables.

You're not the same person two years or five years ago—hell, even ten years ago. Anytime something significant happens, our value changes. Just look at your 20s and 30s—relationships, layoffs, deaths, weddings, and childbirth. All of these life events impact our outlook and influence what we find important.

It's a great thing to grow, evolve, and learn! Understanding how life shifts our worldview and perspective is even better. A happy and healthy relationship is a self-reflection moment away. 

Tajmirriyahi, M., & Ickes, W. (2022). Evidence that increasing self-concept clarity tends to reduce the role of emotional contagion in predicting one’s emotional intelligence regarding a romantic partner .  Personality and Individual Differences ,  185 , 111259. doi:10.1016/j.paid.2021.111259

Frei, J. R., & Shaver, P. R. (2002). Respect in close relationships: Prototype definition, self-report assessment, and initial correlates.  Personal Relationships ,  9 (2), 121–139. doi:10.1111/1475-6811.00008

Jackson C. 30% of adults say most people can be trusted . Ipsos. Published March, 24, 2022.

Huber, G. A., & Malhotra, N. (2017). Political homophily in social relationships: Evidence from online dating behavior .  The Journal of Politics ,  79 (1), 269–283. doi:10.1086/687533

Cassepp-Borges, V. (2021). Should i stay or should i go? Relationship satisfaction, love, love styles and religion compatibility predicting the fate of relationships .  Sexuality & Culture ,  25 (3), 871–883. doi:10.1007/s12119-020-09798-2

• Relationships

Why Do You Love Me? The Quest for Certainty in Relationships

Clarifying goals can help calm reassurance-seeking in romantic relationships..

Updated August 1, 2024 | Reviewed by Margaret Foley

• Why Relationships Matter
• Take our Relationship Satisfaction Test
• Find a therapist to strengthen relationships
• Fear of abandonment can fuel reassurance-seeking in romantic relationships.
• It's possible to learn strategies to stop the cycle of doubt and find peace in a relationship.
• Creating a Wise Mind Dating Plan can lead to healthier dating decisions.

Do you demand constant reassurance from your romantic partner? Do you feel uncertain about your partner's feelings or if this relationship is right for you? This craving for reassurance isn't just about needing to be sure; it goes much deeper.

For people with Relationship OCD , BPD, or betrayal trauma , reassurance-seeking is a quest for certainty and attachment .

Relationship betrayal trauma stems from past betrayals, abuse, or neglect. This creates trust issues and sensitivity to potential betrayal, leading to reassurance-seeking behaviors to ensure a partner's fidelity and commitment.

Relationship OCD involves intrusive doubts about your relationship or partner. Obsessive thinking generates anxiety , and compulsive reassurance-seeking provides short-term relief.

Borderline personality disorder involves intense fears of abandonment and difficulties with emotional regulation . These symptoms can complicate intimate relationships, causing a craving for reassurance about the stability of relationships.

Reassurance-seeking behaviors temporarily calm anxiety but create a cycle of dependence. Asking questions like "Do you love me?" offers momentary relief but reinforces the behavior and actually increases the need for reassurance. When anxiety returns, it's often more intense, leading to a vicious cycle of doubt and reassurance-seeking.

Reassurance-seeking strains relationships. Constantly asking for validation is a trap for both partners: One needs reassurance to calm anxiety, while the other feels overwhelmed and pressured to provide it. This cycle creates emotional disconnection and frustration, worsening the relationship—which then makes the need for reassurance more intense.

Our True Desire: Attachment and Connection

Reassurance-seeking is driven by a need for certainty and connection. It's a symptom of our craving for secure attachment. When we ask for reassurance, we're longing for emotional security and closeness.

The Quest for Certainty in an Uncertain World

As humans, we desire certainty in an uncertain world. This is especially true in relationships. We want proof that our partner loves us, that our relationship will last, or that we will never be hurt. But relationships, like much of life, are inherently uncertain. This relentless quest for certainty can lead to an endless cycle of doubt and reassurance-seeking, temporarily relieving the anxiety but never providing the security we crave.

There are two different types of reassurance-seeking behavior:

• Seeking validation from your partner about their love and commitment. Hunting for evidence of infidelity or commitment. Testing your partner's love. Overanalyzing your partner's words and actions, second-guessing their meaning.
• Seeking reassurance from others about your partner's suitability or the value of the relationship. Comparing your relationship or partner to other people's situations or your previous partners. Reliance on external validation undermines confidence in your judgment and weakens your capacity to stay in intimate partner relationships.

Discernment vs. Obsession

It can be challenging to distinguish between healthy discernment and obsessive thoughts. Discernment involves evaluating your relationship based on your values and needs. In contrast, obsession involves intrusive, repetitive thoughts that lead to anxiety and compulsive behaviors .

Creating a Wise Mind Dating Plan

A Wise Mind Dating Plan clarifies your relationship goals , including the type of relationship you want and the qualities you seek in a partner. Refer to it when doubts arise about your relationship. In Chapter 16 of my book, DBT for Life: Skills to Transform the Way You Live , I outline steps to create your Wise Mind Dating Plan. Here's a brief overview:

• Clarify your relationship goals : What do you want out of a relationship? What are your long-term goals? Clarifying what you want can help you stay grounded when doubts arise.
• Identify qualities you must have in a partner : These are non-negotiables, such as similar values, good hygiene, or a sense of humor . Knowing your needs helps you recognize if your partner aligns with your core values.
• Recognize your preferences : These are qualities you prefer but can be flexible with, such as specific hobbies or interests.
• Establish your deal-breakers : These are things you won't accept in a partner, such as dishonesty, disrespect, or incompatible life goals.
• Understand the things you are willing to put up with : These are the minor irritations that come with living with another person and a recognition of the other person's limitations. You don't like these behaviors, but you can live with them.

By having this plan, you create a concrete reference point to ground yourself when obsessive concerns flare up. Your Wise Mind Dating plan helps you differentiate between valid concerns and anxiety-driven doubts.

DBT Skills to the Rescue

• Check the Facts : Challenge irrational thoughts by identifying and disputing them with evidence, such as countering "If my partner doesn't reassure me, they don't love me" with factual evidence.
• Self-Compassion and Validation : Being kind to yourself is crucial. When you notice self-critical thoughts, try to replace them with self-compassionate ones. Remember, it's OK to feel anxious sometimes, and treating yourself with the kindness you would offer a friend is essential.
• Emotion Regulation : Understanding and managing your emotions can reduce the intensity of ROCD symptoms. Techniques like identifying and labeling emotions and using the Opposite Action skill can be very effective.
• Interpersonal Effectiveness : These skills help you communicate more effectively and build healthier relationships. Techniques such as assertiveness training and setting boundaries can empower you to manage reassurance-seeking behaviors.

Encouragement and Hope

If you're struggling with reassurance-seeking in an otherwise secure relationship, please know you are not alone. There is hope, and DBT skills can help you manage your symptoms and improve your relationships. Remember to be kind to yourself, practice self-compassion, and seek professional help if needed. You're stronger than you think, and you can overcome these anxieties with time and effort.

To find a therapist, visit the Psychology Today Therapy Directory .

Diana Partington, LPC, author of DBT for Life: Skills to Transform the Way You Live, is dedicated to making DBT skills fun and easy to learn.

• Find a Therapist
• Find a Treatment Center
• Find a Psychiatrist
• Find a Support Group
• Find Online Therapy
• United States
• Brooklyn, NY
• Chicago, IL
• Houston, TX
• Los Angeles, CA
• New York, NY
• Portland, OR
• San Diego, CA
• San Francisco, CA
• Seattle, WA
• Washington, DC
• Asperger's
• Bipolar Disorder
• Chronic Pain
• Eating Disorders
• Passive Aggression
• Personality
• Goal Setting
• Positive Psychology
• Stopping Smoking
• Low Sexual Desire
• Child Development
• Self Tests NEW
• Therapy Center
• Diagnosis Dictionary
• Types of Therapy

Sticking up for yourself is no easy task. But there are concrete skills you can use to hone your assertiveness and advocate for yourself.

• Emotional Intelligence
• Gaslighting
• Affective Forecasting
• Neuroscience