natural experiment and quasi experiment

• Technical advance
• Open access
• Published: 11 February 2021

Conceptualising natural and quasi experiments in public health

• Frank de Vocht   ORCID: orcid.org/0000-0003-3631-627X 1 , 2 , 3 ,
• Srinivasa Vittal Katikireddi 4 ,
• Cheryl McQuire 1 , 2 ,
• Kate Tilling 1 , 5 ,
• Matthew Hickman 1 &
• Peter Craig 4  

BMC Medical Research Methodology volume  21 , Article number:  32 ( 2021 ) Cite this article

21k Accesses

63 Citations

27 Altmetric

Metrics details

Natural or quasi experiments are appealing for public health research because they enable the evaluation of events or interventions that are difficult or impossible to manipulate experimentally, such as many policy and health system reforms. However, there remains ambiguity in the literature about their definition and how they differ from randomized controlled experiments and from other observational designs. We conceptualise natural experiments in the context of public health evaluations and align the study design to the Target Trial Framework.

A literature search was conducted, and key methodological papers were used to develop this work. Peer-reviewed papers were supplemented by grey literature.

Natural experiment studies (NES) combine features of experiments and non-experiments. They differ from planned experiments, such as randomized controlled trials, in that exposure allocation is not controlled by researchers. They differ from other observational designs in that they evaluate the impact of events or process that leads to differences in exposure. As a result they are, in theory, less susceptible to bias than other observational study designs. Importantly, causal inference relies heavily on the assumption that exposure allocation can be considered ‘as-if randomized’. The target trial framework provides a systematic basis for evaluating this assumption and the other design elements that underpin the causal claims that can be made from NES.

Conclusions

NES should be considered a type of study design rather than a set of tools for analyses of non-randomized interventions. Alignment of NES to the Target Trial framework will clarify the strength of evidence underpinning claims about the effectiveness of public health interventions.

Peer Review reports

When designing a study to estimate the causal effect of an intervention, the experiment (particularly the randomised controlled trial (RCT) is generally considered to be the least susceptible to bias. A defining feature of the experiment is that the researcher controls the assignment of the treatment or exposure. If properly conducted, random assignment balances unmeasured confounders in expectation between the intervention and control groups . In many evaluations of public health interventions, however, it is not possible to conduct randomised experiments. Instead, standard observational epidemiological study designs have traditionally been used. These are known to be susceptible to unmeasured confounding.

Natural experimental studies (NES) have become popular as an alternative evaluation design in public health research, as they have distinct benefits over traditional designs [ 1 ]. In NES, although the allocation and dosage of treatment or exposure are not under the control of the researcher, they are expected to be unrelated to other factors that cause the outcome of interest [ 2 , 3 , 4 , 5 ]. Such studies can provide strong causal information in complex real-world situations, and can generate effect sizes close to the causal estimates from RCTs [ 6 , 7 , 8 ]. The term natural experiment study is sometimes used synonymously with quasi-experiment; a much broader term that can also refer to researcher-led but non-randomised experiments. In this paper we argue for a clearer conceptualisation of natural experiment studies in public health research, and present a framework to improve their design and reporting and facilitate assessment of causal claims.

Natural and quasi-experiments have a long history of use for evaluations of public health interventions. One of the earliest and best-known examples is the case of ‘Dr John Snow and the Broad Street pump’ [ 9 ]. In this study, cholera deaths were significantly lower among residents served by the Lambeth water company, which had moved its intake pipe to an upstream location of the Thames following an earlier outbreak, compared to those served by the Southwark and Vauxhall water company, who did not move their intake pipe. Since houses in the study area were serviced by either company in an essentially random manner, this natural experiment provided strong evidence that cholera was transmitted through water [ 10 ].

Natural and quasi experiments

Natural and quasi experiments are appealing because they enable the evaluation of changes to a system that are difficult or impossible to manipulate experimentally. These include, for example, large events, pandemics and policy changes [ 7 , 11 ]. They also allow for retrospective evaluation when the opportunity for a trial has passed [ 12 ]. They offer benefits over standard observational studies because they exploit variation in exposure that arises from an exogenous ( i.e. not caused by other factors in the analytic model [ 1 ]) event or intervention. This aligns them to the ‘ do -operator’ in the work of Pearl [ 13 ]. Quasi experiments (QES) and NES thus combine features of experiments (exogenous exposure) and non-experiments (observations without a researcher-controlled intervention). As a result, they are generally less susceptible to confounding than many other observational study designs [ 14 ]. However, a common critique of QES and NES is that because the processes producing variation in exposure are outside the control of the research team, there is uncertainty as to whether confounding has been sufficiently minimized or avoided [ 7 ]. For example, a QES of the impact of a voluntary change by a fast food chain to label its menus with information on calories on subsequent purchasing of calories [ 15 ]. Unmeasured differences in the populations that visit that particular chain compared to other fast-food choices could lead to residual confounding.

A distinction is sometimes made between QES and NES. The term ‘natural experiment’ has traditionally referred to the occurrence of an event with a natural cause; a ‘force of nature‘(Fig.  1 a) [ 1 ]. These make for some of the most compelling studies of causation from non-randomised experiments. For example, the Canterbury earthquakes in 2010–2011 have been used to study the causal impact of such disasters because about half of an established birth cohort lived in the affected area with the remainder of the cohort living elsewhere [ 16 ]. More recently, the use of the term ‘natural’ has been understood more broadly as an event which did not involve the deliberate manipulation of exposure for research purposes (for example a policy change), even if human agency was involved [ 17 ]. Compared to natural experiments in QES the research team may be able to influence exposure allocation, even if the event or exposure itself is not under their full control; for example in a phased roll out of a policy [ 18 ]. A well-known example of a natural experiment is the “Dutch Hunger Winter” summarised by Lumey et al. [ 19 ]. During this period in the Second World War the German authorities blocked all food supplies to the occupied West of the Netherlands, which resulted in widespread starvation. Food supplies were restored immediately after the country was liberated, so the exposure was sharply defined by time as well as place. Because there was sufficient food in the occupied and liberated areas of the Netherlands before and after the Hunger Winter, exposure to famine occurred based on an individual’s time and place (of birth) only. Similar examples of such ‘political’ natural experiment studies are the study of the impact of China’s Great Famine [ 20 ] and the ‘special period’ in Cuba’s history following the collapse of the Soviet Union and the imposition of a US blockade [ 21 ]. NES that describe the evaluation of an event which did not involve the deliberate manipulation of an exposure but involved human agency, such as the impact of a new policy, are the mainstay of ‘natural experimental research’ in public health, and the term NES has become increasingly popular to indicate any quasi-experimental design (although it has not completely replaced it).

Different conceptualisations of natural and quasi experiments within wider evaluation frameworks

Dunning takes the distinction of a NES further. He defines a NES as a QES where knowledge about the exposure allocation process provides a strong argument that allocation, although not deliberately manipulated by the researcher, is essentially random. This concept is referred to as ‘as-if randomization’ (Fig. 1 b) [ 4 , 8 , 10 ]. Under this definition, NES differ from QES in which the allocation of exposure, whether partly controlled by the researcher or not, does not clearly resemble a random process.

A third distinction between QES and NES has been made that argues that NES describe the study of unplanned events whereas QES describe evaluations of events that are planned (but not controlled by the researcher), such as policies or programmes specifically aimed at influencing an outcome (Fig. 1 c) [ 17 ]. In practice however, the distinction between these can be ambiguous.

When the assignment of exposure is not controlled by the researcher, with rare exceptions (for example lottery-system [ 22 ] or military draft [ 23 ] allocations), it is typically very difficult to prove that true (as-if) randomization occurred. Because of the ambiguity of ‘as-if randomization’ and the fact that the tools to assess this are the same as those used for assessment of internal validity in any observational study [ 12 ], the UK Medical Research Council (MRC) guidance advocates a broader conceptualisation of a NES. Under the MRC guidance, a NES is defined as any study that investigates an event that is not under the control of the research team, and which divides a population into exposed and unexposed groups, or into groups with different levels of exposure (Fig. 1 d).

Here, while acknowledging the remaining ambiguity regarding the precise definition of a NES, in consideration of the definitions above [ 24 ], we argue that:

what distinguishes NES from RCTs is that allocation is not controlled by the researchers and;

what distinguishes NES from other observational designs is that they specifically evaluate the impact of a clearly defined event or process which result in differences in exposure between groups.

A detailed assessment of the allocation mechanism (which determines exposure status) is essential. If we can demonstrate that the allocation process approximates a randomization process, any causal claims from NES will be substantially strengthened. The plausibility of the ‘as-if random’ assumption strongly depends on detailed knowledge of why and how individuals or groups of individuals were assigned to conditions and how the assignment process was implemented [ 10 ]. This plausibility can be assessed quantitatively for observed factors using standard tools for assessment of internal validity of a study [ 12 ], and should ideally be supplemented by a qualitative description of the assignment process. Common with contemporary public health practice, we will use the term ‘natural experiment study’, or NES to refer to both NES and QES, from hereon.

Medline, Embase and Google Scholar were searched using search terms including quasi-experiment, natural experiment, policy evaluation and public health evaluation and key methodological papers were used to develop this work. Peer-reviewed papers were supplemented by grey literature.

Part 1. Conceptualisations of natural experiments

An analytic approach.

Some conceptualisations of NES place their emphasis on the analytic tools that are used to evaluate natural experiments [ 25 , 26 ]. In this conceptualisation NES are understood as being defined by the way in which they are analysed, rather than by their design. An array of different statistical methods is available to analyse natural experiments, including regression adjustments, propensity scores, difference-in-differences, interrupted time series, regression discontinuity, synthetic controls, and instrumental variables. Overviews including strengths and limitations of the different methods are provided in [ 12 , 27 ]. However, an important drawback of this conceptualisation is that it suggests that there is a distinct set of methods for the analysis of NES.

A study design

The popularity of NES has resulted in some conceptual stretching, where the label is applied to a research design that only implausibly meets the definitional features of a NES [ 10 ]. For example, observational studies exploring variation in exposures (rather than the study of an event or change in exposure) have sometimes also been badged as NES. A more stringent classification of NES as a type of study design, rather than a collection of analytic tools, is important because it prevents attempts to incorrectly cover observational studies with a ‘glow of experimental legitimacy’ [ 10 ]. If the design rather than the statistical methodology defines a NES, this allows an open-ended array of statistical tools. These tools are not necessarily constrained by those mentioned above, but could also, for example, include new methods such as synthetic controls that can be utilised to analyse the natural experiments. The choice of appropriate evaluation method should be based on what is most suitable for each particular study, and then depends on the knowledge about the event, the availability of data, and design elements such as its allocation process.

Dunning argues that it is the overall research design, rather than just the statistical methods, that compels conviction when making causal claims. He proposes an evaluation framework for NES along the three dimensions of (1) the plausibility of as-if randomization of treatment, (2) the credibility of causal and statistical models, and (3) the substantive relevance of the treatment. Here, the first dimension is considered key for distinguishing NES from other QES [ 4 ]. NES can be divided into those where a plausible case for ‘as-if random’ assignment can be made (which he defines as NES), and those where confounding from observed factors is directly adjusted for through statistical means. The validity of the latter (which Dunning defines as ‘other quasi experiments’, and we define as ‘weaker NES’) relies on the assumption that unmeasured confounding is absent [ 8 ], and is considered less credible in theory for making causal claims [ 4 ]. In this framework, the ‘as-if-randomised’ NES can be viewed as offering stronger causal evidence than other quasi-experiments. In principle, they offer an opportunity for direct estimates of effects (akin to RCTs) where control for confounding factors would not necessarily be required [ 4 ], rather than relying on adjustment to derive conditional effect estimates [ 10 ]. Of course, the latter may well reach valid and compelling conclusions as well, but causal claims suffer to a higher degree from the familiar threats of bias and unmeasured confounding.

Part 2. A target trial framework for natural experiment studies

In this section, we provide recommendations for evaluation of the ‘as if random’ assumption and provide a unifying Target Trial Framework for NES, which brings together key sets of criteria that can be used to appraise the strength of causal claims from NES and assist with study design and reporting.

In public health, there is considerable overlap between analytic and design-based uses of the term NES. Nevertheless, we argue that if we consider NES a type of study design, causal inference can be strengthened by clear appraisal of the likelihood of ‘as-if’ random allocation of exposure. This should be demonstrated by both empirical evidence and by knowledge and reasoning about the causal question and substantive domain under question [ 8 , 10 ]. Because the concept of ‘as-if’ randomization is difficult, if not impossible to prove, it should be thought of along a ‘continuum of plausibility’ [ 10 ]. Specifically, for claims of ‘as-if’ randomization to be plausible, it must be demonstrated that the variables that determine treatment assignment are exogenous. This means that they are: i) strongly correlated with treatment status but are not caused by the outcome of interest (i.e. no reverse causality) and ii) independent of any other (measured or unmeasured) causes of the outcome of interest [ 8 ].

Given this additional layer of justification, especially with respect to the qualitative knowledge of the assignment process and domain knowledge from practitioners more broadly, we argue where feasible for the involvement of practitioners. This could, for example, be formalized through co-production in which members of the public and policy makers are involved in the development of the evaluation. If we appraise NES as a type of study design, which distinguish themselves from other designs because i) there is a particular change in exposure that is evaluated and ii) causal claims are supported by an argument of the plausibility of as-if randomization, then we guard against conflating NES with other observational designs [ 10 , 28 ].

There is a range of ways of dealing with the problems of selection on measured and unmeasured confounders in NES [ 8 , 10 ] which can be understood in terms of a ‘target trial’ we are trying to emulate, had randomization been possible [ 29 ]. The protocol of a target trial describes seven components common to RCTs (‘eligibility criteria’, ‘treatment strategies’, ‘assignment procedures’, ‘follow-up period’, ‘outcome’, ‘causal contrasts of interest’, and the ‘analysis plan’), and provides a systematic way of improving, reporting and appraising NES relative to a ‘gold standard’ (but often not feasible in practice) trial. In the design phase of a NES deviations from the target trial in each domain can be used to evaluate where improvements and where concessions will have to be made. This same approach can be used to appraise existing NES. The target trial framework also provides a structured way for reporting NES, which will facilitate evaluation of the strength of NES, improve consistency and completeness of reporting, and benefit evidence syntheses.

In Table  1 , we bring together elements of the Target Trial framework and conceptualisations of NES to derive a framework to describe the Target Trial for NES [ 12 ]. By encouraging researchers to address the questions in Table 1 , the framework provides a structured approach to the design, reporting and evaluation of NES across the seven target trial domains. Table 1 also provides recommendations to improve the strength of causal claims from NES, focussing primarily on sensitivity analyses to improve internal validity.

An illustrative example of a well-developed NES based on the criteria outlined in Table 1 is by Reeves et al. [ 39 ]. The NES evaluates the impact of the introduction of a National Minimum Wage on mental health. The study compared a clearly defined intervention group of recipients of a wage increase up to 110% of pre-intervention wage with clearly defined control groups of (1) people ineligible to the intervention because their wage at baseline was just above (100–110%) minimum wage and (2) people who were eligible, but whose companies did not comply and did not increase minimum wage. This study also included several sensitivity tests to strengthen causal arguments. We have aligned this study to the Target Trial framework in Additional file  1 .

The Target Trial Approach for NES (outlined in Table 1 ) provides a straightforward approach to improve, report, and appraise existing NES and to assist in the design of future studies. It focusses on structural design elements and goes beyond the use of quantitative tools alone to assess internal validity [ 12 ]. This work complements the ROBINS-I tool for assessing risk of bias in non-randomised studies of interventions, which similarly adopted the Target Trial framework [ 40 ]. Our approach focusses on the internal validity of a NES, with issues of construct and external validity being outside of the scope of this work (guidelines for these are provided in for example [ 41 ]). It should be acknowledged that less methodologically robust studies can still reach valid and compelling conclusions, even without resembling the notional target trial. However, we believe that drawing on the target trial framework helps highlight occasions when causal inference can be made more confidently.

And finally, the framework does explicitly exclude observational studies that aim to investigate the effects of changes in behaviour without an externally forced driver to do so. For example, although a cohort study can be the basis for the evaluation of a NES in principle, effects of the change of diet of some participants (compared to those who did not change their diet) is not an external cause (i.e. exogenous) and does not fall within the definition of an experiment [ 11 ]. However, such studies are likely to be more convincing than those which do not study within-person changes and we note that the statistical methods used may be similar to NES.

Despite their advantages, NES remain based on observational data and thus biases in assignment of the intervention can never be completely excluded (although for plausibly ‘as if randomised’ natural experiments these should be minimal). It is therefore important that a robust assessment of different potential sources of bias is reported. It has additionally been argued that sensitivity analyses are required to assess whether a pattern of small biases could explain away any ostensible effect of the intervention, because confidence intervals and statistical tests do not do this [ 14 ]. Recommendations that would improve the confidence with which we can make causal claims from NES, derived from work by Rosenbaum [ 14 ], have been outlined in Table 1 . Although sensitivity analyses can place plausible limits on the size of the effects of hidden biases, because such analyses are susceptible to assumptions about the maximum size of omitted biases, they cannot completely rule out residual bias [ 34 ]. Of importance for the strength of causal claims therefore, is the triangulation of NES with other evaluations using different data or study designs susceptible to different sources of bias [ 5 , 42 ].

None of the recommendations outlined in Table 1 will by themselves eliminate bias in a NES, but neither is it required to implement all of them to be able to make a causal claim with some confidence. Instead, a continuum of confidence in the causal claims based on the study design and the data is a more appropriate and practical approach [ 43 ]. Each sensitivity analysis aims to minimise ambiguity of a particular potential bias or biases, and as such a combination of selected sensitivity analyses can strengthen causal claims [ 14 ]. We would generally, but not strictly, consider a well conducted RCT as the design where we are most confident about such claims, followed by natural experiments, and then other observational studies; this would be an extension of the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) framework [ 44 ]. GRADE provides a system for rating the quality (or certainty) of a body of evidence and grading the strength of recommendations for use in systematic reviews, health technology assessments (HTAs), and clinical practice guidelines. It typically only distinguishes between trials and observational studies when making these judgments (note however, that recent guidance does not make this explicit distinction when using ROBINS-I [ 45 ]). Given the increased contribution of NES in public health, especially those based on routine data [ 37 ], the specific inclusion of NES in this system might improve the rating of the evidence from these study designs.

Our recommendations are of particular importance for ensuring rigour in the context of (public) health research where natural experiments have become increasingly popular for a variety of reasons, including the availability of large routinely collected datasets [ 37 ]. Such datasets invite the discovery of natural experiments, even where the data may not be particularly applicable to this design, but also these enable many of the sensitivity analyses to be conducted from within the same dataset or through linkage to other routine datasets.

Finally, alignment to the Target Trial Framework also links natural experiment studies directly to other measures of trial validity, including pre-registration, reporting checklists, and evaluation through risk-of-bias-tools [ 40 ]. This aligns with previous recommendations to use established reporting guidelines such as STROBE, TREND [ 12 ], and TIDieR-PHP [ 46 ] for the reporting of natural experiment studies. These reporting guidelines could be customized to specific research areas (for example, as developed for a systematic review of quasi-experimental studies of prenatal alcohol use and birthweight and neurodevelopment [ 47 ]).

We provide a conceptualisation of natural experiment studies as they apply to public health. We argue for the appreciation of natural experiments as a type of study design rather than a set of tools for the analyses of non-randomised interventions. Although there will always remain some ambiguity about the strength of causal claims, there are clear benefits to harnessing NES rather than relying purely on observational studies. This includes the fact that NES can be based on routinely available data and that timely evidence of real-world relevance can be generated. The inclusion of a discussion of the plausibility of as-if randomization of exposure allocation will provide further confidence in the strength of causal claims.

Aligning NES to the Target Trial framework will guard against conceptual stretching of these evaluations and ensure that the causal claims about whether public health interventions ‘work’ are based on evidence that is considered ‘good enough’ to inform public health action within a ‘practice-based evidence’ framework. This framework describes how evaluations can help reducing critical uncertainties and adjust the compass bearing of existing policy (in contrast to the ‘evidence-based practice’ framework in which RCTs are used to generate ‘definitive’ evidence for particular interventions) [ 48 ].

Availability of data and materials

Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.

Abbreviations

Randomised Controlled Trial

Natural Experiment

Stable Unit Treatment Value Assumption

Intention-To-Treat

Shadish WR, Cook TD, Campbell DT. Experimental and Quasi-Experimental Designs. 2nd ed. Wadsworth, Cengage Learning: Belmont; 2002.

Google Scholar  

King G, Keohane RO, Verba S. The importance of research Design in Political Science. Am Polit Sci Rev. 1995;89:475–81.

Article   Google Scholar  

Meyer BD. Natural and quasi-experiments in economics. J Bus Econ Stat. 1995;13:151–61.

Dunning T. Natural experiments in the social sciences. A design-based approach. 6th edition. Cambridge: Cambridge University Press; 2012.

Book   Google Scholar  

Craig P, Cooper C, Gunnell D, Haw S, Lawson K, Macintyre S, et al. Using natural experiments to evaluate population health interventions: new medical research council guidance. J Epidemiol Community Health. 2012;66:1182–6.

Cook TD, Shadish WR, Wong VC. Three conditions under which experiments and observational studies produce comparable causal estimates: new findings from within-study comparisons. J Policy Anal Manag. 2008;27:724–50.

Bärnighausen T, Røttingen JA, Rockers P, Shemilt I, Tugwell P. Quasi-experimental study designs series—paper 1: introduction: two historical lineages. J Clin Epidemiol. 2017;89:4–11.

Waddington H, Aloe AM, Becker BJ, Djimeu EW, Hombrados JG, Tugwell P, et al. Quasi-experimental study designs series—paper 6: risk of bias assessment. J Clin Epidemiol. 2017;89:43–52.

Saeed S, Moodie EEM, Strumpf EC, Klein MB. Evaluating the impact of health policies: using a difference-in-differences approach. Int J Public Health. 2019;64:637–42.

Dunning T. Improving causal inference: strengths and limitations of natural experiments. Polit Res Q. 2008;61:282–93.

Bärnighausen T, Tugwell P, Røttingen JA, Shemilt I, Rockers P, Geldsetzer P, et al. Quasi-experimental study designs series—paper 4: uses and value. J Clin Epidemiol. 2017;89:21–9.

Craig P, Katikireddi SV, Leyland A, Popham F. Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Annu Rev Public Health. 2017;38:39–56.

Pearl J, Mackenzie D. The book of why: the new science of cause and effect. London: Allen Lane; 2018.

Rosenbaum PR. How to see more in observational studies: some new quasi-experimental devices. Annu Rev Stat Its Appl. 2015;2:21–48.

Petimar J, Ramirez M, Rifas-Shiman SL, Linakis S, Mullen J, Roberto CA, et al. Evaluation of the impact of calorie labeling on McDonald’s restaurant menus: a natural experiment. Int J Behav Nutr Phys Act. 2019;16. Article no: 99.

Fergusson DM, Horwood LJ, Boden JM, Mulder RT. Impact of a major disaster on the mental health of a well-studied cohort. JAMA Psychiatry. 2014;71:1025–31.

Remler DK, Van Ryzin GG. Natural and quasi experiments. In: Research methods in practice: strategies for description and causation. 2nd ed. Thousand Oaks: SAGE Publication Inc.; 2014. p. 467–500.

Cook PA, Hargreaves SC, Burns EJ, De Vocht F, Parrott S, Coffey M, et al. Communities in charge of alcohol (CICA): a protocol for a stepped-wedge randomised control trial of an alcohol health champions programme. BMC Public Health. 2018;18. Article no: 522.

Lumey LH, Stein AD, Kahn HS, Van der Pal-de Bruin KM, Blauw GJ, Zybert PA, et al. Cohort profile: the Dutch hunger winter families study. Int J Epidemiol. 2007;36:1196–204.

Article   CAS   Google Scholar  

Meng X, Qian N. The Long Term Consequences of Famine on Survivors: Evidence from a Unique Natural Experiment using China’s Great Famine. Natl Bur Econ Res Work Pap Ser. 2011;NBER Worki.

Franco M, Bilal U, Orduñez P, Benet M, Morejón A, Caballero B, et al. Population-wide weight loss and regain in relation to diabetes burden and cardiovascular mortality in Cuba 1980-2010: repeated cross sectional surveys and ecological comparison of secular trends. BMJ. 2013;346:f1515.

Angrist J, Bettinger E, Bloom E, King E, Kremer M. Vouchers for private schooling in Colombia: evidence from a randomized natural experiment. Am Econ Rev. 2002;92:1535–58.

Angrist JD. Lifetime earnings and the Vietnam era draft lottery: evidence from social security administrative records. Am Econ Rev. 1990;80:313–36.

Dawson A, Sim J. The nature and ethics of natural experiments. J Med Ethics. 2015;41:848–53.

Bärnighausen T, Oldenburg C, Tugwell P, Bommer C, Ebert C, Barreto M, et al. Quasi-experimental study designs series—paper 7: assessing the assumptions. J Clin Epidemiol. 2017;89:53-66.

Tugwell P, Knottnerus JA, McGowan J, Tricco A. Big-5 Quasi-Experimental designs. J Clin Epidemiol. 2017;89:1–3.

Reeves BC, Wells GA, Waddington H. Quasi-experimental study designs series—paper 5: a checklist for classifying studies evaluating the effects on health interventions—a taxonomy without labels. J Clin Epidemiol. 2017;89:30–42.

Rubin DB. For objective causal inference, design trumps analysis. Ann Appl Stat. 2008;2:808–40.

Hernán MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol. 2016;183:758–64.

Benjamin-Chung J, Arnold BF, Berger D, Luby SP, Miguel E, Colford JM, et al. Spillover effects in epidemiology: parameters, study designs and methodological considerations. Int J Epidemiol. 2018;47:332–47.

Munafò MR, Tilling K, Taylor AE, Evans DM, Smith GD. Collider scope: when selection bias can substantially influence observed associations. Int J Epidemiol. 2018;47:226–35.

Schwartz S, Gatto NM, Campbell UB. Extending the sufficient component cause model to describe the stable unit treatment value assumption (SUTVA). Epidemiol Perspect Innov. 2012;9:3.

Cawley J, Thow AM, Wen K, Frisvold D. The economics of taxes on sugar-sweetened beverages: a review of the effects on prices, sales, cross-border shopping, and consumption. Annu Rev Nutr. 2019;39:317–38.

Reichardt CS. Nonequivalent Group Designs. In: Quasi-Experimentation. A Guide to Design and Analysis. 1st edition. New York: The Guildford Press; 2019. p. 112–162.

Denzin N. Sociological methods: a sourcebook. 5th ed. New York: Routledges; 2006.

Matthay EC, Hagan E, Gottlieb LM, Tan ML, Vlahov D, Adler NE, et al. Alternative causal inference methods in population health research: evaluating tradeoffs and triangulating evidence. SSM - Popul Heal. 2020;10:10052.

Leatherdale ST. Natural experiment methodology for research: a review of how different methods can support real-world research. Int J Soc Res Methodol. 2019;22:19–35.

Reichardt CS. Quasi-experimentation. A guide to design and analysis. 1st ed. New York: The Guildford Press; 2019.

Reeves A, McKee M, Mackenbach J, Whitehead M, Stuckler D. Introduction of a National Minimum Wage Reduced Depressive Symptoms in Low-Wage Workers: A Quasi-Natural Experiment in the UK. Heal Econ (United Kingdom). 2017;26:639–55.

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

Shadish WR, Cook TD, Campbell DT. Generalized Causal Inference: A Grounded Theory. In: Experimental and Quasi-Experimental Designs for Generalized Causal Inference. 2nd ed. Belmont: Wadsworth, Cengage Learning; 2002. p. 341–73.

Lawlor DA, Tilling K, Smith GD. Triangulation in aetiological epidemiology. Int J Epidemiol. 2016;45:1866–86.

Hernán MA. The C-word: scientific euphemisms do not improve causal inference from observational data. Am J Public Health. 2018;108:616–9.

Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction - GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64:383–94.

Schünemann HJ, Cuello C, Akl EA, Mustafa RA, Meerpohl JJ, Thayer K, et al. GRADE guidelines: 18. How ROBINS-I and other tools to assess risk of bias in nonrandomized studies should be used to rate the certainty of a body of evidence. J Clin Epidemiol. 2019;111:105–14.

Campbell M, Katikireddi SV, Hoffmann T, Armstrong R, Waters E, Craig P. TIDieR-PHP: a reporting guideline for population health and policy interventions. BMJ. 2018;361:k1079.

Mamluk L, Jones T, Ijaz S, Edwards HB, Savović J, Leach V, et al. Evidence of detrimental effects of prenatal alcohol exposure on offspring birthweight and neurodevelopment from a systematic review of quasi-experimental studies. Int J Epidemiol. 2021;49(6):1972-95.

Ogilvie D, Adams J, Bauman A, Gregg EW, Panter J, Siegel KR, et al. Using natural experimental studies to guide public health action: turning the evidence-based medicine paradigm on its head. J Epidemiol Community Health. 2019;74:203–8.

Download references

Acknowledgements

This study is funded by the National Institute for Health Research (NIHR) School for Public Health Research (Grant Reference Number PD-SPH-2015). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The funder had no input in the writing of the manuscript or decision to submit for publication. The NIHR School for Public Health Research is a partnership between the Universities of Sheffield; Bristol; Cambridge; Imperial; and University College London; The London School for Hygiene and Tropical Medicine (LSHTM); LiLaC – a collaboration between the Universities of Liverpool and Lancaster; and Fuse - The Centre for Translational Research in Public Health a collaboration between Newcastle, Durham, Northumbria, Sunderland and Teesside Universities. FdV is partly funded by National Institute for Health Research Applied Research Collaboration West (NIHR ARC West) at University Hospitals Bristol NHS Foundation Trust. SVK and PC acknowledge funding from the Medical Research Council (MC_UU_12017/13) and Scottish Government Chief Scientist Office (SPHSU13). SVK acknowledges funding from a NRS Senior Clinical Fellowship (SCAF/15/02). KT works in the MRC Integrative Epidemiology Unit, which is supported by the Medical Research Council (MRC) and the University of Bristol [MC_UU_00011/3].

Author information

Authors and affiliations.

Population Health Sciences, Bristol Medical School, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, UK

Frank de Vocht, Cheryl McQuire, Kate Tilling & Matthew Hickman

NIHR School for Public Health Research, Newcastle, UK

Frank de Vocht & Cheryl McQuire

NIHR Applied Research Collaboration West, Bristol, UK

Frank de Vocht

MRC/CSO Social and Public Health Sciences Unit, University of Glasgow, Bristol, UK

Srinivasa Vittal Katikireddi & Peter Craig

MRC IEU, University of Bristol, Bristol, UK

Kate Tilling

You can also search for this author in PubMed   Google Scholar

Contributions

FdV conceived of the study. FdV, SVK,CMQ,KT,MH, PC interpretated the evidence and theory. FdV wrote the first version of the manuscript. SVK,CMQ,KT,MH, PC provided substantive revisions to subsequent versions. All authors have read and approved the manuscript. FdV, SVK,CMQ,KT,MH, PC agreed to be personally accountable for their own contributions and will ensure that questions related to the accuracy or integrity of any part of the work, even ones in which the author was not personally involved, are appropriately investigated, resolved, and the resolution documented in the literature.

Corresponding author

Correspondence to Frank de Vocht .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Online Supplementary Material. Table 1 . the Target Trial for Natural Experiments and Reeves et al. [ 28 ]. Alignment of Reeves et al. (Introduction of a National Minimum Wage Reduced Depressive Symptoms in Low-Wage Workers: A Quasi-Natural Experiment in the UK. Heal Econ. 2017;26:639–55) to the Target Trial framework.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

de Vocht, F., Katikireddi, S.V., McQuire, C. et al. Conceptualising natural and quasi experiments in public health. BMC Med Res Methodol 21 , 32 (2021). https://doi.org/10.1186/s12874-021-01224-x

Download citation

Received : 14 July 2020

Accepted : 28 January 2021

Published : 11 February 2021

DOI : https://doi.org/10.1186/s12874-021-01224-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Public health
• Public health policy
• Natural experiments
• Quasi experiments
• Evaluations

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Quasi-Experimental Design | Definition, Types & Examples

Quasi-Experimental Design | Definition, Types & Examples

Published on July 31, 2020 by Lauren Thomas . Revised on January 22, 2024.

Like a true experiment , a quasi-experimental design aims to establish a cause-and-effect relationship between an independent and dependent variable .

However, unlike a true experiment, a quasi-experiment does not rely on random assignment . Instead, subjects are assigned to groups based on non-random criteria.

Quasi-experimental design is a useful tool in situations where true experiments cannot be used for ethical or practical reasons.

Table of contents

Differences between quasi-experiments and true experiments, types of quasi-experimental designs, when to use quasi-experimental design, advantages and disadvantages, other interesting articles, frequently asked questions about quasi-experimental designs.

There are several common differences between true and quasi-experimental designs.

Example of a true experiment vs a quasi-experiment

However, for ethical reasons, the directors of the mental health clinic may not give you permission to randomly assign their patients to treatments. In this case, you cannot run a true experiment.

Instead, you can use a quasi-experimental design.

You can use these pre-existing groups to study the symptom progression of the patients treated with the new therapy versus those receiving the standard course of treatment.

Prevent plagiarism. Run a free check.

Many types of quasi-experimental designs exist. Here we explain three of the most common types: nonequivalent groups design, regression discontinuity, and natural experiments.

Nonequivalent groups design

In nonequivalent group design, the researcher chooses existing groups that appear similar, but where only one of the groups experiences the treatment.

In a true experiment with random assignment , the control and treatment groups are considered equivalent in every way other than the treatment. But in a quasi-experiment where the groups are not random, they may differ in other ways—they are nonequivalent groups .

When using this kind of design, researchers try to account for any confounding variables by controlling for them in their analysis or by choosing groups that are as similar as possible.

This is the most common type of quasi-experimental design.

Regression discontinuity

Many potential treatments that researchers wish to study are designed around an essentially arbitrary cutoff, where those above the threshold receive the treatment and those below it do not.

Near this threshold, the differences between the two groups are often so minimal as to be nearly nonexistent. Therefore, researchers can use individuals just below the threshold as a control group and those just above as a treatment group.

However, since the exact cutoff score is arbitrary, the students near the threshold—those who just barely pass the exam and those who fail by a very small margin—tend to be very similar, with the small differences in their scores mostly due to random chance. You can therefore conclude that any outcome differences must come from the school they attended.

Natural experiments

In both laboratory and field experiments, researchers normally control which group the subjects are assigned to. In a natural experiment, an external event or situation (“nature”) results in the random or random-like assignment of subjects to the treatment group.

Even though some use random assignments, natural experiments are not considered to be true experiments because they are observational in nature.

Although the researchers have no control over the independent variable , they can exploit this event after the fact to study the effect of the treatment.

However, as they could not afford to cover everyone who they deemed eligible for the program, they instead allocated spots in the program based on a random lottery.

Although true experiments have higher internal validity , you might choose to use a quasi-experimental design for ethical or practical reasons.

Sometimes it would be unethical to provide or withhold a treatment on a random basis, so a true experiment is not feasible. In this case, a quasi-experiment can allow you to study the same causal relationship without the ethical issues.

The Oregon Health Study is a good example. It would be unethical to randomly provide some people with health insurance but purposely prevent others from receiving it solely for the purposes of research.

However, since the Oregon government faced financial constraints and decided to provide health insurance via lottery, studying this event after the fact is a much more ethical approach to studying the same problem.

True experimental design may be infeasible to implement or simply too expensive, particularly for researchers without access to large funding streams.

At other times, too much work is involved in recruiting and properly designing an experimental intervention for an adequate number of subjects to justify a true experiment.

In either case, quasi-experimental designs allow you to study the question by taking advantage of data that has previously been paid for or collected by others (often the government).

Quasi-experimental designs have various pros and cons compared to other types of studies.

• Higher external validity than most true experiments, because they often involve real-world interventions instead of artificial laboratory settings.
• Higher internal validity than other non-experimental types of research, because they allow you to better control for confounding variables than other types of studies do.
• Lower internal validity than true experiments—without randomization, it can be difficult to verify that all confounding variables have been accounted for.
• The use of retrospective data that has already been collected for other purposes can be inaccurate, incomplete or difficult to access.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Degrees of freedom
• Null hypothesis
• Discourse analysis
• Control groups
• Mixed methods research
• Non-probability sampling
• Quantitative research
• Ecological validity

Research bias

• Rosenthal effect
• Implicit bias
• Cognitive bias
• Selection bias
• Negativity bias
• Status quo bias

A quasi-experiment is a type of research design that attempts to establish a cause-and-effect relationship. The main difference with a true experiment is that the groups are not randomly assigned.

In experimental research, random assignment is a way of placing participants from your sample into different groups using randomization. With this method, every member of the sample has a known or equal chance of being placed in a control group or an experimental group.

Quasi-experimental design is most useful in situations where it would be unethical or impractical to run a true experiment .

Quasi-experiments have lower internal validity than true experiments, but they often have higher external validity  as they can use real-world interventions instead of artificial laboratory settings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Thomas, L. (2024, January 22). Quasi-Experimental Design | Definition, Types & Examples. Scribbr. Retrieved August 21, 2024, from https://www.scribbr.com/methodology/quasi-experimental-design/

Is this article helpful?

Lauren Thomas

Other students also liked, guide to experimental design | overview, steps, & examples, random assignment in experiments | introduction & examples, control variables | what are they & why do they matter, get unlimited documents corrected.

✔ Free APA citation check included ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

• Search Menu

Sign in through your institution

• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Numismatics
• Classical Literature
• Classical Reception
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Epigraphy
• Greek and Roman Law
• Greek and Roman Papyrology
• Greek and Roman Archaeology
• Late Antiquity
• Religion in the Ancient World
• Social History
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Acquisition
• Language Evolution
• Language Reference
• Language Variation
• Language Families
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Modernism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Religion
• Music and Media
• Music and Culture
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Science
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Politics
• Law and Society
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Legal System - Costs and Funding
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Restitution
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Clinical Neuroscience
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Ethics
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Security
• Computer Games
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Strategy
• Business Ethics
• Business History
• Business and Government
• Business and Technology
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Social Issues in Business and Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic Systems
• Economic History
• Economic Methodology
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Management of Land and Natural Resources (Social Science)
• Natural Disasters (Environment)
• Pollution and Threats to the Environment (Social Science)
• Social Impact of Environmental Issues (Social Science)
• Sustainability
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• Ethnic Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Political Behaviour
• Political Economy
• Political Institutions
• Political Theory
• Politics and Law
• Politics of Development
• Public Administration
• Public Policy
• Qualitative Political Methodology
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Disability Studies
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

5.9 Natural and quasi-experiments

• Published: November 2021
• Cite Icon Cite
• Permissions Icon Permissions

Natural experiments, defined as events or processes outwith the control of a researcher, which divide a population into exposed and unexposed groups, have long been used in epidemiology and public health. Evaluation methods that rely on observational rather than experimental data are classified as second best in conventional hierarchies of evidence. Natural experimental approaches have attracted renewed interest from public health researchers and decision-makers because they widen the range of interventions that can usefully be evaluated beyond those that are politically, ethically, or practically amenable to testing in randomized controlled trials. This chapter argues for seeing trials and natural experiments as part of a common toolkit for producers and users of evidence about the effectiveness of policies and programmes. It describes the most commonly used natural experimental approaches to evaluating population health interventions, and provides examples of their use in a wide range of countries and policy settings.

Personal account

• Sign in with email/username & password
• Get email alerts
• Save searches
• Purchase content
• Activate your purchase/trial code
• Add your ORCID iD

Institutional access

Sign in with a library card.

• Sign in with username/password
• Recommend to your librarian
• Institutional account management
• Get help with access

Access to content on Oxford Academic is often provided through institutional subscriptions and purchases. If you are a member of an institution with an active account, you may be able to access content in one of the following ways:

IP based access

Typically, access is provided across an institutional network to a range of IP addresses. This authentication occurs automatically, and it is not possible to sign out of an IP authenticated account.

Choose this option to get remote access when outside your institution. Shibboleth/Open Athens technology is used to provide single sign-on between your institution’s website and Oxford Academic.

• Click Sign in through your institution.
• Select your institution from the list provided, which will take you to your institution's website to sign in.
• When on the institution site, please use the credentials provided by your institution. Do not use an Oxford Academic personal account.
• Following successful sign in, you will be returned to Oxford Academic.

If your institution is not listed or you cannot sign in to your institution’s website, please contact your librarian or administrator.

Enter your library card number to sign in. If you cannot sign in, please contact your librarian.

Society Members

Society member access to a journal is achieved in one of the following ways:

Sign in through society site

Many societies offer single sign-on between the society website and Oxford Academic. If you see ‘Sign in through society site’ in the sign in pane within a journal:

• Click Sign in through society site.
• When on the society site, please use the credentials provided by that society. Do not use an Oxford Academic personal account.

If you do not have a society account or have forgotten your username or password, please contact your society.

Sign in using a personal account

Some societies use Oxford Academic personal accounts to provide access to their members. See below.

A personal account can be used to get email alerts, save searches, purchase content, and activate subscriptions.

Some societies use Oxford Academic personal accounts to provide access to their members.

Viewing your signed in accounts

Click the account icon in the top right to:

• View your signed in personal account and access account management features.
• View the institutional accounts that are providing access.

Signed in but can't access content

Oxford Academic is home to a wide variety of products. The institutional subscription may not cover the content that you are trying to access. If you believe you should have access to that content, please contact your librarian.

For librarians and administrators, your personal account also provides access to institutional account management. Here you will find options to view and activate subscriptions, manage institutional settings and access options, access usage statistics, and more.

Our books are available by subscription or purchase to libraries and institutions.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Rights and permissions
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Europe PMC Author Manuscripts

Natural Experiments: An Overview of Methods, Approaches, and Contributions to Public Health Intervention Research

Associated data.

Population health interventions are essential to reduce health inequalities and tackle other public health priorities, but they are not always amenable to experimental manipulation. Natural experiment (NE) approaches are attracting growing interest as a way of providing evidence in such circumstances. One key challenge in evaluating NEs is selective exposure to the intervention. Studies should be based on a clear theoretical understanding of the processes that determine exposure. Even if the observed effects are large and rapidly follow implementation, confidence in attributing these effects to the intervention can be improved by carefully considering alternative explanations. Causal inference can be strengthened by including additional design features alongside the principal method of effect estimation. NE studies often rely on existing (including routinely collected) data. Investment in such data sources and the infrastructure for linking exposure and outcome data is essential if the potential for such studies to inform decision making is to be realized.

Introduction

Natural experiments (NEs) have a long history in public health research, stretching back to John Snow’s classic study of London’s cholera epidemics in the mid-nineteenth century. Since the 1950s, when the first clinical trials were conducted, investigators have emphasized randomized controlled trials (RCTs) as the preferred way to evaluate health interventions. Recently, NEs and other alternatives to RCTs have attracted interest because they are seen as the key to evaluating large-scale population health interventions that are not amenable to experimental manipulation but are essential to reducing health inequalities and tackling emerging health problems such as the obesity epidemic ( 15 , 27 , 40 , 68 , 76 ).

We follow the UK Medical Research Council guidance in defining NEs broadly to include any event not under the control of a researcher that divides a population into exposed and unexposed groups ( 16 ). NE studies use this naturally occurring variation in exposure to identify the impact of the event on some outcome of interest. Our focus here is on public health and other policy interventions that seek to improve population health or which may have important health impacts as a by-product of other policy goals. One key evaluation challenge is selective exposure to the intervention, leading exposed individuals or groups to differ from unexposed individuals or groups in characteristics associated with better or worse outcomes. Understanding and modeling the process(es) determining exposure to the intervention are therefore central to the design and conduct of NE studies.

Some authors define NEs more narrowly to include only those in which the process that determines exposure (often referred to as the assignment or data-generating process) is random or as-if random ( 22 , pp. 15 – 16 ). Truly random assignment, although not unknown ( 14 ), is extremely rare in policy and practice settings. As-if randomness lacks a precise definition, and the methods proposed to identify as-if random processes (such as a good understanding of the assignment process and checks on the balance of covariates between exposed and unexposed groups) are those used to assess threats to validity in any study that attempts to make causal inferences from observational data. In the absence of a clear dividing line, we prefer to adopt a more inclusive definition and to assess the plausibility of causal inference on a case-by-case basis.

In the next section, we set out a general framework for making causal inferences in experimental and observational studies. The following section discusses the main approaches used NE studies to estimate the impact of public health interventions and to address threats to the validity of causal inferences. We conclude with brief proposals for improving the future use of NEs.

Causal Inference in Trials and Observational Studies

The potential outcomes model provides a useful framework for clarifying similarities and differences between true experiments on the one hand and observational studies (including NEs) on the other hand ( 51 ). Potential outcomes refer to the outcomes that would occur if a person (or some other unit) were exposed simultaneously to an intervention and a control condition. As only one of those outcomes can be observed, causal effects must be inferred from a comparison of average outcomes among units assigned to an intervention or to a control group. If assignment is random, the groups are said to be exchangeable and the intervention’s average causal effect can be estimated from the difference in the average outcomes for the two groups. In a well-conducted RCT, randomization ensures exchangeability. In an observational study, knowledge of the assignment mechanism can be used to make the groups conditionally exchangeable, for example, by controlling for variables that influence both assignment and outcomes to the extent that these variables are known and accurately measured ( 34 ).

As well as showing why a control group is needed, this framework indicates why an understanding of the assignment process is so important to the design of an NE study. The methods discussed in the next section can be seen as different ways of achieving conditional exchangeability. The framework also usefully highlights the need to be clear about the kind of causal effect being estimated and, in particular, whether it applies to the whole population (such as an increase in alcohol excise duty) or a particular subset (such as a change in the minimum legal age for purchasing alcohol). A comparison of outcomes between groups assigned to the intervention or control condition provides an estimate of the effect of assignment, known as the intention-to-treat (ITT) effect, rather than the effect of the intervention itself. The two are necessarily the same only if there is perfect compliance. Some methods, such as fuzzy regression discontinuity (RD) and instrumental variables (IVs), estimate a different effect, the complier average causal effect (CACE), which is the effect of the intervention on those who comply with their allocation into the control or intervention group ( 11 ). Under certain assumptions, the CACE is equivalent to the ITT effect divided by the proportion of compliers. Which effect is relevant will depend on the substantive questions the study is asking. If the effect of interest is the ITT effect, as in a pragmatic effectiveness trial or a policy evaluation in which decision makers wish to know about the effect across the whole population, methods that estimate a more restricted effect may be less useful ( 20 ).

A related issue concerns extrapolation—using results derived from one population to draw conclusions about another. In a trial, all units have a known probability of being assigned to the intervention or control group. In an observational study, where exchangeability may be achieved by a method such as matching or by conditioning on covariates, intervention groups may be created whose members in practice have no chance of receiving the treatment ( 55 ). The meaning of treatment effects estimated in this way is unclear. Extrapolation may also be a problem for some NE studies, such as those using RD designs, which estimate treatment effects at a particular value of a variable used to determine assignment. Effects of this kind, known as local average treatment effects, may be relevant to the substantive concerns of the study, but researchers should bear in mind how widely results can be extrapolated, given the nature of the effects being estimated.

Table 1 summarizes similarities and contrasts between RCTs, NEs, and nonexperimental observational studies.

Abbreviations: NE, natural experiment; RCT, randomized controlled trial; RD, regression discontinuity.

Methods for Evaluating Natural Experiments

Key considerations when choosing an NE evaluation method are the source of variation in exposure and the size and nature of the expected effects. The source of variation in exposure may be quite simple, such as an implementation date, or quite subtle, such as a score on an eligibility test. Interventions that are introduced abruptly, that affect large populations, and that are implemented where it is difficult for individuals to manipulate their treatment status are more straightforward to evaluate. Likewise, effects that are large and follow rapidly after implementation are more readily detectable than more subtle or delayed effects. One example of the former is a study that assessed the impact of a complete ban in 1995 on the import of pesticides commonly used in suicide in Sri Lanka ( 32 ). Suicide rates had risen rapidly since the mid-1970s, then leveled off following a partial ban on pesticide imports in the early 1980s. After the complete ban, rates of suicide by self-poisoning fell by 50%. The decrease was specific to Sri Lanka, was barely offset by an increase in suicide by other methods, and could not be explained by changes in death recording or by wider socioeconomic or political trends.

Although NE studies are not restricted to interventions with rapid, large effects, more complicated research designs may be needed where effects are smaller or more gradual. Table 2 summarizes approaches to evaluating NEs. It includes both well-established and widely used methods such as difference-in-differences (DiD) and interrupted time series (ITS), as well as more novel approaches such as synthetic controls. Below, we describe these methods in turn, drawing attention to their strengths and limitations and providing examples of their use.

Abbreviations: LMIC, low- and middle-income countries; NE, natural experiment.

Regression Adjustment

Standard multivariable models, which control for observed differences between intervention and control groups, can be used to evaluate NEs when no important differences in unmeasured characteristics between intervention and control groups are expected (see Model 1 in Appendix 1 ). Goodman et al. used data from the UK Millennium Cohort Study to evaluate the impact of a school-based cycle training scheme on children’s cycling behavior ( 29 ). The timing of survey fieldwork meant that some interviews took place before and others after the children received training. Poisson models were used to estimate the effect of training on cycling behaviors, with adjustment for a wide range of potential confounders. Previous evaluations that compared children from participating and nonparticipating schools found substantial effects on cycling behavior. In contrast, this study found no difference, suggesting that the earlier findings reflected the selective provision of training. The key strength of the study by Goodman et al. is the way the timing of data gathering in relation to exposure created well-balanced intervention and control groups. Without this overlap between data gathering and exposure to the intervention, there was a significant risk that unobserved differences between the groups would bias the estimates, despite adjusting for a wide range of observed confounders.

Propensity Score–Based Methods

In a well-conducted RCT, random allocation ensures that intervention and control arms are balanced in terms of both measured and unmeasured covariates. In the absence of random allocation, the propensity score attempts to recreate the allocation mechanism, defined as the conditional probability of an individual being in the intervention group, given a number of covariates ( 65 ).

The propensity score is typically estimated using logistic regression, based on a large number of covariates, although alternative estimation methods are available. There are four principal ways to use the propensity score to obtain an estimated treatment effect: matching, stratification, inverse probability weighting, and covariate adjustment ( 7 ). Each method will adjust for differences in characteristics of the intervention and control groups and, in so doing, minimize the effects of confounding. The propensity score, however, is constrained by the covariates available and the extent to which they can collectively mimic the allocation to intervention and control groups.

Understanding the mechanism underlying allocation to intervention and control groups is key when deriving the propensity score. Sure Start Local Programmes (SSLPs), area-based interventions designed to improve the health and well-being of young children in England, were an example where, on an ITT basis, exposure to the intervention was determined by area of residence and would apply to everyone living in the area regardless of individual characteristics. Melhuish et al. ( 54 ) therefore constructed a propensity score at the area level, based on 85 variables, to account for differences between areas with and without SSLPs. Analysis was undertaken on individuals clustered within areas, stratified by the propensity of an area to receive the SSLP. The most deprived areas were excluded from the analysis because there were insufficient comparison areas.

Advantages of using the propensity score over simple regression adjustment include the complexity of the propensity score that can be created (through, for example, including higher-order terms and interactions), the ease of checking the adequacy of the propensity score as opposed to checking the adequacy of a regression model, and the ability to examine the extent to which intervention and control groups overlap in key covariates ( 7 , 18 ), and thereby avoid extrapolation. Although in statistical terms the use of propensity scores may produce results that differ little from those obtained through traditional regression adjustment ( 70 ), they encourage clearer thinking about study design and particularly the assignment mechanism ( 66 ). When membership of the treatment and control groups varies over time, inverse probability weighting can be used to account for time-varying confounding ( 34 ), as in the study by Pega et al. ( 59 ) of the cumulative impact of tax credits on self-rated health.

Difference-in-Differences

In its simplest form, the DiD approach compares change in an outcome among people who are newly exposed to an intervention with change among those who remain unexposed. Although these differences could be calculated from a 2 × 2 table of outcomes for each group at each time point, the effect is more usefully estimated from a regression with terms for group, period, and group-by-period interaction. The coefficient of the interaction term is the DiD estimator (Model 2 in Appendix 1 ).

DiD’s strength is that it controls for unobserved as well as observed differences in the fixed (i.e., time-invariant) characteristics of the groups and is therefore less prone to omitted variable bias caused by unmeasured confounders or measurement error. The method relies on the assumption that, in the absence of the intervention, preimplementation trends would continue. This common trends assumption may be violated by differential changes in the composition of the intervention or control groups or by other events (such as the introduction of another intervention) that affect one group but not the other. With data for multiple preimplementation time points, the common trends assumption can be investigated directly, and it can be relaxed by extending the model to include terms for group-specific trends. With more groups and time points, the risk that other factors may influence outcomes increases, but additional terms can be included to take account of time-varying characteristics of the groups.

De Angelo & Hansen ( 19 ) used a DiD approach to estimate the effectiveness of traffic policing in reducing road traffic injuries and fatalities by taking advantage of a NE provided by the state of Oregon’s failure to agree on a budget in 2003, which led to the layoff of more than one-third of Oregon’s traffic police force. A comparison of injury and fatality rates in Oregon with rates in two neighboring states before and after the layoff indicated that, after allowing for other factors associated with road traffic incidents, such as the weather and the number of young drivers, less policing led to a 12–14% increase in fatalities. Whereas De Angelo & Hansen’s study focused on an intervention in a single area, Nandi and colleagues ( 58 ) applied DiD methods to estimate the impact of paid maternity leave across a sample of 20 low- and middle-income countries.

DiD methods are not limited to area-based interventions. Dusheiko et al. ( 23 ) used the withdrawal of a financial incentive scheme for family doctors in the English National Health Service to identify whether it led to treatment rationing. Recent developments, such as the use of propensity scores, rather than traditional covariate adjustment, to account for group-specific time-varying characteristics, add additional complexity, but combining DiD with other approaches in this way may further strengthen causal inference.

Interrupted Time Series

Alongside DiD, ITS methods are among the most widely applied approaches to evaluating NEs. An ITS consists of a sequence of count or continuous data at evenly spaced intervals over time, with one or more well-defined change points that correspond to the introduction of an intervention ( 69 ). There are many approaches to analyzing time series data ( 44 ). A straightforward approach is to use a segmented regression model, which provides an estimate of changes in the level and trend of the outcome associated with the intervention, controlling for preintervention level and trend ( 43 , 75 ). Such models can be estimated by fitting a linear regression model, including a continuous variable for time since the start of the observation period, a dummy variable for time period (i.e., before/after intervention), and a continuous variable for time postintervention (Model 3 in Appendix 1 ). The coefficients of these variables measure the preintervention trend, the change in the level of the outcome immediately postintervention, and the change in the trend postintervention. Additional variables can be added to identify the effects of interventions introduced at other time points or to control for changes in level or trend of the outcome due to other factors. Lags in the effect of the intervention can be accounted for by omitting outcome values that occur during the lag period or by modeling the lag period as a separate segment ( 75 ). Successive observations in a time series are often related to one another, a problem known as serial autocorrelation. Unless autocorrelation is addressed, the standard errors will be underestimated, but models that allow for autocorrelation can be fitted using standard statistical packages.

By accounting for preintervention trends, well-conducted ITS studies permit stronger causal inference than do cross-sectional or simple prepost designs, but they may be subject to confounding by cointerventions or changes in population composition. Controlled ITS designs, which compare trends in exposed and unexposed groups or in outcomes that are not expected to change as a result of the intervention, can be used to strengthen causal inference still further; in addition, standardization can be used to control for changes in population composition. A common shortcoming in ITS analyses is a lack of statistical power ( 61 ). Researchers have published a range of recommendations for the number of data points required, but statistical power also depends on the expected effect size and the degree of autocorrelation. Studies with few data points will be underpowered unless the effect size is large. Zhang et al. ( 79 ) and Mcleod & Vingilis ( 53 ) provide methods for calculating statistical power for ITS studies.

Robinson et al. ( 64 ) applied controlled ITS methods to commercially available alcohol sales data to estimate the impact of a ban on the offer of multipurchase discounts by retailers in Scotland. Because alcohol sales vary seasonally, the researchers fitted models that took account of seasonal autocorrelation, as well as trends in sales in England and Wales where the legislation did not apply. After adjusting for sales in England and Wales, the study found a 2% decrease in overall sales, compared with a previous study’s finding of no impact using DiD methods applied to self-reported alcohol purchase data.

Synthetic Controls

The difficulty of finding control areas that closely match the background trends and characteristics of the intervention area is a significant challenge in many NE studies. One solution is to use a synthetic combination of areas rather than the areas themselves as controls. Methods for deriving synthetic controls and using them to estimate the impact of state-, region-, or national-level policies were developed by political scientists ( 1 – 4 ) and are now being applied to many health and social policies ( 8 , 9 , 17 , 30 , 45 , 62 , 67 ).

A synthetic control is a weighted average of control areas that provides the best visual and statistical match to the intervention area on the preintervention values of the outcome variable and of predictors of the outcome. Although the weights are based on observed characteristics, matching on the outcome in the preintervention period minimizes differences in unobserved fixed and time-varying characteristics. The difference between the postintervention trend in the intervention and synthetic control provides the effect estimate. Software to implement the method is available in a number of statistical packages ( 2 ).

Abadie et al. ( 1 ) used synthetic controls to evaluate a tobacco control program introduced in California in 1988, which increased tobacco taxes and earmarked the revenues for other tobacco control measures. The comparator was derived from a donor pool of other US states, excluding any states that had implemented extensive tobacco control interventions. A weighted combination of five states, based on pre-1988 trends in cigarette consumption and potential confounders, formed the synthetic control. Comparison of the postintervention trends in the real and synthetic California suggested a marked reduction in tobacco consumption as a result of the program.

The synthetic control method can be seen as an extension of the DiD method, with a number of advantages. In particular, it relaxes the requirement for a geographical control that satisfies the parallel trends assumption and relies less on subjective choices of control areas. A practical limitation, albeit one that prevents extrapolation, is that if the intervention area is an outlier, for example if California’s smoking rate in 1988 was higher than those of all other US states, then no combination of areas in the donor pool can provide an adequate match. Another limitation is that conventional methods of statistical inference cannot be applied, although Abadie et al. ( 1 ) suggest an alternative that compares the estimated effect for the intervention area with the distribution of placebo effects derived by comparing each area in the donor pool with its own synthetic control.

Instrumental Variables

IV methods address selective exposure to an intervention by replacing a confounded direct measure of exposure with an unconfounded proxy measure, akin to treatment assignment in an RCT ( 33 ). To work in this way, an IV must be associated with exposure to the intervention, must have no association with any other factors associated with exposure, and must be associated with outcomes only through its association with exposure to the intervention ( Figure 1 ).

( a ) The variable Z is associated with outcome Y only through its association with exposure X , so it can be considered a valid instrument of X . ( b ) Z is not a valid instrument owing to a lack of any association with outcome Y . ( c ) Z is not a valid instrument owing to its association with confounder C . ( d ) Z is not a valid instrument owing to its direct association with Y .

IVs that satisfy the three conditions offer a potentially valuable solution to the problem of unobserved as well as observed confounders. Estimating an intervention’s effect using IVs can be viewed as a two-stage process (Models 5.1 and 5.2 in Appendix 1 ). In the first stage, a prediction of treatment assignment is obtained from a regression of the treatment variable on the instruments. Fitted values from this model replace the treatment variable in the outcome regression ( 41 ).

IVs are widely used in econometric program evaluation and have attracted much recent interest in epidemiology, particularly in the context of Mendelian randomization studies, which use genetic variants as instruments for environmental exposures ( 25 , 36 , 48 ). IV methods have not yet been widely used to evaluate public health interventions because it can be difficult to find suitable instruments and to demonstrate convincingly, using theory or data, that they meet the second and third conditions above ( 35 , 71 ). A recent example is the study by Ichida et al. ( 39 ) of the effect of community centers on improving social participation among older people in Japan, using distance to the nearest center as an instrument for intervention receipt. Another study, by Yen et al. ( 78 ), considers the effect of food stamps on food insecurity, using a range of instruments, including aspects of program administration that might encourage or discourage participation in the food stamp program. Given the potential value of IVs, as one of a limited range of approaches for mitigating the problems associated with unobserved confounders, and their widespread use in related fields, they should be kept in mind should opportunities arise ( 35 ).

Regression Discontinuity

Age, income, and other continuous variables are often used to determine entitlement to social programs, such as means-tested welfare benefits. The RD design uses such assignment rules to estimate program impacts. RD is based on the insight that units with values of the assignment variable just above or below the cutoff for entitlement will be similar in other respects, especially if there is random error in the assignment variable ( 11 ). This similarity allows the effect of the program to be estimated from a regression of the outcome on the assignment variable (often referred to as the running or forcing variable) and a dummy variable denoting exposure (treatment), with the coefficient of the dummy identifying the treatment effect (Model 4 in Appendix 1 ). Additional terms are usually included in the model to allow slopes to vary above and below the cutoff, allow for nonlinearities in the relationship between the assignment and outcome variables, and deal with residual confounding.

Visual checks play an important role in RD studies. Plots of treatment probability ( Figure 2 ) and outcomes against the assignment variable can be used to identify discontinuities that indicate a treatment effect, and a histogram of the assignment variable can be plotted to identify bunching around the cutoff that would indicate manipulation of treatment assignment. Scatterplots of covariates against assignment can be used to check for continuity at the cutoff that would indicate whether units above and below the cutoff are indeed similar ( 57 ).

( a ) A fuzzy regression discontinuity: probability of treatment changes gradually at values of the assignment variable close to the cutoff. ( b ) A sharp regression discontinuity: probability of treatment changes from 0 to 1 at the cutoff. Source: Reproduced from Moscoe (2015) ( 57 ) with permission from Elsevier.

The RD estimator need not be interpreted only as the effect of a unit’s exposure to the program (treatment) right at the cutoff value ( 47 ), but the assumption that units above and below the cutoff are similar except in their exposure to the program becomes less tenable as distance from the cutoff increases. Usual practice is to fit local linear regressions for observations within a narrow band on either side of the cutoff. Restricting the analysis in this way also means that nonlinearities in the relationship between the forcing and outcome variables are less important. One drawback is that smaller numbers of observations will yield less precise estimates, so the choice involves a trade-off between bias and precision.

The above approach works when the probability of a treatment jumps from 0 to 1 at the cutoff, which is known as sharp RD ( Figure 2b ). If exposure is influenced by factors other than the value of the forcing variable, for example because administrators can exercise discretion over whom to include in the program or because individuals can, to some extent, manipulate their own assignment, the probability of treatment may take intermediate values close to the cutoff ( Figure 1b ) and a modified approach known as fuzzy RD should be applied. This process uses the same two-stage approach to estimation as does an IV analysis (Models 5.1 and 5.2 in Appendix 1 ).

One example of a sharp RD design is Ludwig & Miller’s ( 52 ) analysis of the US Head Start program. Help with applications for Head Start funding was targeted to counties with poverty rates of 59% or greater. This targeting led to a lasting imbalance in the receipt of Head Start funds among counties with poverty rates above and below the cutoff. Ludwig & Miller used local linear regressions of mortality on poverty rates for counties with poverty rates between 49% and 69%; the impact of the Head Start funding was defined as the difference between the estimated mortality rates at the upper and lower limits of this range. They found substantial reductions in mortality from causes amenable to Head Start but not from other causes of death or in children whose ages meant they were unlikely to benefit from the program.

Andalon ( 6 ) used a fuzzy RD design to investigate the impact of a conditional cash transfer program on obesity and overweight. Mexico’s Opportunidades program provided substantial cash subsidies to households in rural communities that scored below a poverty threshold, which were conditional on school and health clinic attendance. There were a range of other ad hoc adjustments to eligibility criteria, creating a fuzzy rather than a sharp discontinuity in participation at the poverty cutoff. Andalon used two-stage least squares regression to estimate the effect of eligibility (based on the poverty score) on program participation and the effect of predicted participation on obesity and overweight. The author found no effect for men but a substantial reduction in obesity among women. Further testing indicated no bunching of poverty scores around the cutoff and no significant discontinuity at the cutoff in a range of covariates. Inclusion of the covariates in the outcome regressions had little effect on the estimates, further supporting the assumption of local randomization.

RD methods are widely regarded as the closest approximation of an observational study to an RCT ( 5 ), but their real value derives from their wide applicability to the evaluation of social programs for which eligibility is determined by a score on some form of continuous scale and also from their reliance on relatively weak, directly testable assumptions. One key shortcoming is that restricting the bandwidth to reduce bias results in a loss of precision ( 46 , 73 ), and estimates that may hold over only a small segment of the whole population exposed to the intervention. This restriction to a subset of the population may not matter if the intervention is expected to affect outcomes locally, as in the case of a minimum legal drinking age or if the substantive focus of the study is on the effect of a small change in the assignment rule. It is more serious when the outcome of interest is the effect on the whole population.

Strengthening Inference in Natural Experimental Studies

Causal inference can be strengthened in NE studies by the inclusion of additional design features alongside the principal method of effect estimation. Studies should be based on a clear theoretical understanding of how the intervention achieves its effects and the processes that determine exposure. Even if the observed effects are large and rapidly follow implementation, confidence in attributing them to the intervention can be markedly improved by a detailed consideration of alternative explanations.

Qualitative research can strengthen the design of RCTs of complex public health interventions ( 10 , 56 ), and this argument applies equally to NEs ( 38 ). Qualitative research undertaken in preparation for, or alongside, NE studies can help to identify which outcomes might change as a consequence of the intervention and which are priorities for decision makers ( 42 ). It can also improve understanding of the processes that determine exposure, factors associated with intervention delivery and compliance, mechanisms by which outcomes are realized, and the strengths and limitations of routinely collected measures of exposures and outcomes ( 13 ). Qualitative studies conducted alongside the quantitative evaluation of Scotland’s smoke-free legislation have been used to assess compliance with the intervention ( 24 ) and to identify a range of secondary outcomes such as changes in smoking behavior within the home ( 60 ). Qualitative methods for identifying the effects of interventions have also been proposed, but further studies are needed to establish their validity and usefulness ( 63 , 72 , 77 ).

Quantitative methods for strengthening inference include the use of multiple estimation methods within studies, replication studies, and falsification tests. Tests specific to particular methods, such as visual checks for discontinuities in RD and ITS studies, can also be used. Good-quality NE studies typically use a range of approaches. Comparing results obtained using different methods can be used to assess the dependence of findings on particular assumptions ( 50 ). Such comparisons are particularly useful in early applications of novel methods whose strengths and weaknesses are not fully understood ( 49 ).

Falsification or placebo tests assess the plausibility of causal attribution by checking for the specificity of effects. One such approach is to use nonequivalent dependent variables to measure changes in outcomes that are not expected to respond to the intervention. They serve as indicators of residual confounding or the effects of other interventions introduced alongside the study intervention. A related approach is to use false implementation dates and to compare changes associated with those dates with effects estimated for the real implementation date. A similar test used in synthetic control studies involves generating placebo effects by replacing the intervention area with each of the areas in the donor pool in turn and then comparing the estimated intervention effect with the distribution of placebo effects ( 1 , 2 ).

Most NE studies are conducted retrospectively, using data collected before the study is planned. Ideally, an analysis protocol, setting out hypotheses and methods, should be developed before any data analysis is conducted ( 21 ). Even when such protocols are published, they do not provide a perfect safeguard against selective reporting of positive findings. Replication studies, which by definition retest a previously published hypothesis, are a valuable additional safeguard against retrospectively fitting hypotheses to known features of the data. Reporting of NE studies of all kinds may also be improved by following established reporting guidelines such as STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) ( 74 ) or TREND (Transparent Reporting of Evaluations with Nonrandomized Designs) ( 28 ).

Conclusions

NE approaches to evaluation have become topical because they address researchers’ and policy makers’ interests in understanding the impact of large-scale population health interventions that, for practical, ethical, or political reasons, cannot be manipulated experimentally. We have suggested a pragmatic approach to NEs. NEs are not the answer to every evaluation question, and it is not always possible to conduct a good NE study whenever an RCT would be impractical. Choices among evaluation approaches are best made according to specific features of the intervention in question, such as the allocation process, the size of the population exposed, the availability of suitable comparators, and the nature of the expected impacts, rather than on the basis of general rules about which methods are strongest, regardless of circumstances. Availability of data also constrains the choice of methods. Where data allow, combining methods and comparing results are good ways to avoid overdependence on particular assumptions. Having a clear theory of change based on a sound qualitative understanding of the causal mechanisms at work is just as important as sophisticated analytical methods.

Many of the examples discussed above use routinely collected data on outcomes such as mortality, road traffic accidents, and hospital admissions and data on exposures such as poverty rates, alcohol sales, and tobacco consumption. Continued investment in such data sources, and in population health surveys, is essential if the potential for NEs to contribute to the evidence base for policy making is to be realized. Recent investments in infrastructure to link data across policy sectors for research purposes are a welcome move that should increase opportunities to evaluate NEs ( 12 , 26 , 37 ). Funding calls for population health research proposals should take a similarly even-handed approach to specifying which approaches would be acceptable and should emphasize the importance of developing a clear theory of change, carefully testing assumptions, and comparing estimates from alternative methods.

Supplementary Material

Acknowledgments.

The authors receive core funding from the UK Medical Research Council (funding codes: MC_UU_12017/13, MC_UU_12017/15) and the Scottish Government Chief Scientist Office (funding codes: SPHSU13 and SPHSU15). In addition, S.V.K. is funded by an NHS Research Scotland Senior Clinical Fellowship (SCAF/15/02). The funders had no role in the preparation or submission of the manuscript, and the views expressed are those of the authors alone.

Disclosure Statement

The authors are not aware of any affiliations, memberships, funding, or financial holdings that might be perceived as affecting the objectivity of this review.

Literature Cited

• A-Z Publications

Annual Review of Public Health

Volume 38, 2017, review article, open access, natural experiments: an overview of methods, approaches, and contributions to public health intervention research.

• Peter Craig 1 , Srinivasa Vittal Katikireddi 1 , Alastair Leyland 1 , and Frank Popham 1
• View Affiliations Hide Affiliations Affiliations: MRC/CSO Social and Public Health Sciences Unit, University of Glasgow, Glasgow G2 3QB, United Kingdom; email: [email protected] , [email protected] , [email protected] , [email protected]
• Vol. 38:39-56 (Volume publication date March 2017) https://doi.org/10.1146/annurev-publhealth-031816-044327
• First published as a Review in Advance on January 11, 2017
• Copyright © 2017 Annual Reviews. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 (CC-BY-SA) International License, which permits unrestricted use, distribution, and reproduction in any medium and any derivative work is made available under the same, similar, or a compatible license. See credit lines of images or other third-party material in this article for license information.

Population health interventions are essential to reduce health inequalities and tackle other public health priorities, but they are not always amenable to experimental manipulation. Natural experiment (NE) approaches are attracting growing interest as a way of providing evidence in such circumstances. One key challenge in evaluating NEs is selective exposure to the intervention. Studies should be based on a clear theoretical understanding of the processes that determine exposure. Even if the observed effects are large and rapidly follow implementation, confidence in attributing these effects to the intervention can be improved by carefully considering alternative explanations. Causal inference can be strengthened by including additional design features alongside the principal method of effect estimation. NE studies often rely on existing (including routinely collected) data. Investment in such data sources and the infrastructure for linking exposure and outcome data is essential if the potential for such studies to inform decision making is to be realized.

Article metrics loading...

Full text loading...

Literature Cited

• Abadie A , Diamond A , Hainmueller J . 1.  2010 . Synthetic control methods for comparative case studies: estimating the effect of California's Tobacco Control Program. J. Am. Stat. Assoc. 105 : 493– 505 [Google Scholar]
• Abadie A , Diamond A , Hainmueller J . 2.  2011 . Synth: an R package for synthetic control methods in comparative case studies. J. Stat. Softw. 42 : 1– 17 [Google Scholar]
• Abadie A , Diamond A , Hainmueller J . 3.  2015 . Comparative politics and the synthetic control method. Am. J. Polit. Sci. 50 : 495– 510 [Google Scholar]
• Abadie A , Gardeazabal J . 4.  2003 . The economic costs of conflict: a case study of the Basque Country. Am. Econ. Rev. 93 : 113– 32 [Google Scholar]
• 5.  Acad. Med. Sci. 2007 . Identifying the Environmental Causes of Disease: How Should We Decide What to Believe and When to Take Action. London: Acad. Med. Sci. [Google Scholar]
• Andalon M . 6.  2011 . Impact of Oportunidades in overweight and obesity in Mexico. Health Econ 20 : Suppl. 1 1– 18 [Google Scholar]
• Austin PC . 7.  2011 . An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivar. Behav. Res. 46 : 399– 424 [Google Scholar]
• Basu S , Rehkopf DH , Siddiqi A , Glymour MM , Kawachi I . 8.  2016 . Health behaviors, mental health, and health care utilization among single mothers after welfare reforms in the 1990s. Am. J. Epidemiol. 83 : 531– 38 [Google Scholar]
• Bauhoff S . 9.  2014 . The effect of school district nutrition policies on dietary intake and overweight: a synthetic control approach. Econ. Hum. Biol. 12 : 45– 55 [Google Scholar]
• Bonell C , Fletcher A , Morton M , Lorenc T , Moore L . 10.  2012 . Realist randomised controlled trials: a new approach to evaluating complex public health interventions. Soc. Sci. Med. 75 : 2299– 306 [Google Scholar]
• Bor J , Moscoe E , Mutevedzi P , Newell M-L , Barnighausen T . 11.  2014 . Regression discontinuity designs in epidemiology: causal inference without randomized trials. Epidemiology 25 : 729– 37 [Google Scholar]
• Boyd J , Ferrante AM , O'Keefe C , Bass AJ , Randall AM . 12.  et al. 2012 . Data linkage infrastructure for cross-jurisdictional health-related research in Australia. BMC Health Serv. Res. 2 : 480 [Google Scholar]
• Brown J , Neary J , Katikireddi SV , Thomson H , McQuaid RW . 13.  et al. 2015 . Protocol for a mixed-methods longitudinal study to identify factors influencing return to work in the over 50s participating in the UK Work Programme: Supporting Older People into Employment (SOPIE). BMJ Open 5 : e010525 [Google Scholar]
• Chattopadhyay R , Duflo E . 14.  2004 . Women as policy makers: evidence from a randomised policy experiment in India. Econometrica 72 : 1409– 43 [Google Scholar]
• 15.  Comm. Soc. Determinants Health. 2008 . Closing the Gap in a Generation: Health Equity Through Action on the Social Determinants of Health. Final Report of the Commission on Social Determinants of Health. Geneva: World Health Organ. [Google Scholar]
• Craig P , Cooper C , Gunnell D , Macintyre S , Petticrew M . 16.  et al. 2012 . Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. J. Epidemiol. Community Health 66 : 1182– 86 [Google Scholar]
• Crifasi CK , Meyers JS , Vernick JS , Webster DW . 17.  2015 . Effects of changes in permit-to-purchase handgun laws in Connecticut and Missouri on suicide rates. Prev. Med. 79 : 43– 49 [Google Scholar]
• D'Agostino RB . 18.  1998 . Tutorial in biostatistics. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat. Med. 17 : 2265– 81 [Google Scholar]
• De Angelo G , Hansen B . 19.  2014 . Life and death in the fast lane: police enforcement and traffic fatalities. Am. Econ. J. Econ. Policy 6 : 231– 57 [Google Scholar]
• Deaton A . 20.  2010 . Instruments, randomisation and learning about development. J. Econ. Lit. 48 : 424– 55 [Google Scholar]
• Dundas R , Ouédraogo S , Bond L , Briggs AH , Chalmers J . 21.  et al. 2014 . Evaluation of health in pregnancy grants in Scotland: a protocol for a natural experiment. BMJ Open 4 : e006547 [Google Scholar]
• Dunning T . 22.  2012 . Natural Experiments in the Social Sciences: A Design-Based Approach Cambridge, UK: Cambridge Univ. Press [Google Scholar]
• Dusheiko M , Gravelle H , Jacobs R , Smith P . 23.  2006 . The effect of financial incentives on gatekeeping doctors: evidence from a natural experiment. J. Health Econ. 25 : 449– 78 [Google Scholar]
• Eadie D , Heim D , MacAskill S , Ross A , Hastings G , Davies J . 24.  2008 . A qualitative analysis of compliance with smoke-free legislation in community bars in Scotland: implications for public health. Addiction 103 : 1019– 26 [Google Scholar]
• Fall T , Hägg S , Mägi R , Ploner A , Fischer K . 25.  et al. 2013 . The role of adiposity in cardiometabolic traits: a Mendelian randomization analysis. PLOS Med 10 : e1001474 [Google Scholar]
• 26.  Farr Inst. Health Inf. Res. 2016 . Environmental and public health research. Farr Inst. Health Inf. Res. Dundee, UK: http://www.farrinstitute.org/research-education/research/environmental-and-public-health [Google Scholar]
• 27.  Foresight 2007 . Tackling Obesities: Future Choices. Challenges for Research and Research Management. London: Gov. Off. Sci. [Google Scholar]
• Fuller T , Peters J , Pearson M , Anderson R . 28.  2014 . Impact of the transparent reporting of evaluations with nonrandomized designs reporting guideline: ten years on. Am. J. Public Health 104 : e110– 17 [Google Scholar]
• Goodman A , van Sluijs EMF , Ogilvie D . 29.  2016 . Impact of offering cycle training in schools upon cycling behaviour: a natural experimental study. Int. J. Behav. Nutr. Phys. Act. 13 : 34 [Google Scholar]
• Green CP , Heywood JS , Navarro M . 30.  2014 . Did liberalising bar hours decrease traffic accidents?. J. Health Econ. 35 : 189– 98 [Google Scholar]
• Grundy C , Steinbach R , Edwards P , Green J , Armstrong B . 31.  et al. 2009 . Effect of 20 mph traffic speed zones on road injuries in London, 1986–2006: controlled interrupted time series analysis. BMJ 339 : b4469 [Google Scholar]
• Gunnell D , Fernando R , Hewagama M , Priyangika W , Konradsen F , Eddleston M . 32.  2007 . The impact of pesticide regulations on suicide in Sri Lanka. Int. J. Epidemiol. 36 : 1235– 42 [Google Scholar]
• Heckman JJ . 33.  1995 . Randomization as an instrumental variable. Rev. Econ. Stat. 78 : 336– 41 [Google Scholar]
• Hernán M , Robins J . 34.  2017 . Causal Inference Boca Raton, FL: Chapman Hall/CRC In press [Google Scholar]
• Hernán M , Robins JM . 35.  2006 . Instruments for causal inference. An epidemiologist's dream. Epidemiology 17 : 360– 72 [Google Scholar]
• Holmes MV , Dale CE , Zuccolo L , Silverwood RJ , Guo Y . 36.  et al. 2014 . Association between alcohol and cardiovascular disease: Mendelian randomisation analysis based on individual participant data. BMJ 349 g4164 [Google Scholar]
• 37.  House Commons Sci. Technol. Comm 2016 . The Big Data Dilemma. Fourth Report of Session 2015–16. HC 468 London: Station. Off. Ltd. [Google Scholar]
• Humphreys DK , Panter J , Sahlqvist S , Goodman A , Ogilvie D . 38.  2016 . Changing the environment to improve population health: a framework for considering exposure in natural experimental studies. J. Epidemiol. Community Health. doi: 10.1136/jech-2015-206381 [Google Scholar]
• Ichida Y , Hirai H , Kondo K , Kawachi I , Takeda T , Endo H . 39.  2013 . Does social participation improve self-rated health in the older population? A quasi-experimental intervention study. Soc. Sci. Med. 94 : 83– 90 [Google Scholar]
• 40.  IOM (Inst. Med.) 2010 . Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making Washington, DC: Natl. Acad. Press [Google Scholar]
• Jones A , Rice N . 41.  2009 . Econometric evaluation of health policies HEDG Work. Pap. 09/09 Univ. York [Google Scholar]
• Katikireddi SV , Bond L , Hilton S . 42.  2014 . Changing policy framing as a deliberate strategy for public health advocacy: a qualitative policy case study of minimum unit pricing of alcohol. Milbank Q 92 : 250– 83 [Google Scholar]
• Katikireddi SV , Der G , Roberts C , Haw S . 43.  2016 . Has childhood smoking reduced following smoke-free public places legislation? A segmented regression analysis of cross-sectional UK school-based surveys. Nicotine Tob. Res. 18 : 1670– 74 [Google Scholar]
• Kontopantelis E , Doran T , Springate DA , Buchan I , Reeves D . 44.  2015 . Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis. BMJ 350 : h2750 [Google Scholar]
• Kreif N , Grieve R , Hangartner D , Nikolova S , Turner AJ , Sutton M . 45.  2015 . Examination of the synthetic control method for evaluating health policies with multiple treated units. Health Econ doi: 10.1002/hec.3258 [Google Scholar]
• Labrecque JA , Kaufman JS . 46.  2016 . Can a quasi-experimental design be a better idea than an experimental one?. Epidemiology 27 : 500– 2 [Google Scholar]
• Lee DS , Lemieux T . 47.  2010 . Regression discontinuity designs in economics. J. Econ. Lit. 48 : 281– 355 [Google Scholar]
• Lewis SJ , Araya R , Davey Smith G , Freathy R , Gunnell D . 48.  et al. 2011 . Smoking is associated with, but does not cause, depressed mood in pregnancy—a Mendelian randomization study. PLOS ONE 6 : e21689 [Google Scholar]
• Linden A , Adams JL . 49.  2011 . Applying a propensity score-based weighting model to interrupted time series data: improving causal inference in programme evaluation. J. Eval. Clin. Pract. 17 : 1231– 38 [Google Scholar]
• Linden A , Adams JL . 50.  2012 . Combining the regression discontinuity design and propensity score-based weighting to improve causal inference in program evaluation. J. Eval. Clin. Pract. 18 : 317– 25 [Google Scholar]
• Little RJ , Rubin DB . 51.  2000 . Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches. Annu. Rev. Public Health 21 : 121– 45 [Google Scholar]
• Ludwig J , Miller D . 52.  2007 . Does Head Start improve children's life chances? Evidence from an RD design. Q. J. Econ. 122 : 159– 208 [Google Scholar]
• Mcleod AI , Vingilis ER . 53.  2008 . Power computations in time series analyses for traffic safety interventions. Accid. Anal. Prev. 40 : 1244– 48 [Google Scholar]
• Melhuish E , Belsky J , Leyland AH , Barnes J . 54.  Natl. Eval. Sure Start Res. Team. 2008 . Effects of fully-established Sure Start Local Programmes on 3-year-old children and their families living in England: a quasi-experimental observational study. Lancet 372 : 1641– 47 [Google Scholar]
• Messer LC , Oakes JM , Mason S . 55.  2010 . Effects of socioeconomic and racial residential segregation on preterm birth: a cautionary tale of structural confounding. Am. J. Epidemiol. 171 : 664– 73 [Google Scholar]
• Moore G , Audrey S , Barker M , Bond L , Bonell C . 56.  et al. 2015 . MRC process evaluation of complex intervention. Medical Research Council guidance. BMJ 350 : h1258 [Google Scholar]
• Moscoe E , Bor J , Barnighausen T . 57.  2015 . Regression discontinuity designs are under-used in medicine, epidemiology and public health: a review of current and best practice. J. Clin. Epidemiol. 68 : 132– 43 [Google Scholar]
• Nandi A , Hajizadeh M , Harper S , Koski A , Strumpf EC , Heymann J . 58.  2016 . Increased duration of paid maternity leave lowers infant mortality in low and middle-income countries: a quasi-experimental study. PLOS Med. 13 : e1001985 [Google Scholar]
• Pega F , Blakely T , Glymour MM , Carter KN , Kawachi I . 59.  2016 . Using marginal structural modelling to estimate the cumulative impact of an unconditional tax credit on self-rated health. Am. J. Epidemiol. 183 : 315– 24 [Google Scholar]
• Phillips R , Amos A , Ritchie D , Cunningham-Burley S , Martin C . 60.  2007 . Smoking in the home after the smoke-free legislation in Scotland: qualitative study. BMJ 335 : 553 [Google Scholar]
• Ramsay CR , Matowe L , Grilli R , Grimshaw JM , Thomas RE . 61.  2005 . Interrupted time series designs in health technology assessment: lessons from two systematic reviews of behaviour change strategies. Int. J. Technol. Assess. Health Care 19 : 613– 23 [Google Scholar]
• Restrepo BJ , Rieger M . 62.  2016 . Denmark's policy on artificial trans fat and cardiovascular disease. Am. J. Prev. Med. 50 : 69– 76 [Google Scholar]
• Rihoux B , Ragin C . 63.  2009 . Configurational Comparative Methods: Qualitative Comparative Analysis (QCA) and Related Techniques London: Sage [Google Scholar]
• Robinson M , Geue C , Lewsey J , Mackay D , McCartney G . 64.  et al. 2014 . Evaluating the impact of the Alcohol Act on off-trade alcohol sales: a natural experiment in Scotland. Addiction 109 : 2035– 43 [Google Scholar]
• Rosenbaum PR , Rubin DB . 65.  1983 . The central role of the propensity score in observational studies for causal effects. Biometrika 70 : 41– 55 [Google Scholar]
• Rubin DB . 66.  2008 . For objective causal inference, design trumps analysis. Ann. Appl. Stat. 2 : 808– 40 [Google Scholar]
• Ryan AM , Krinsky S , Kontopantelis E , Doran T . 67.  2016 . Long-term evidence for the effect of pay-for-performance in primary care on mortality in the UK: a population study. Lancet 388 : 268– 74 [Google Scholar]
• Sanson-Fisher RW , D'Este CS , Carey ML , Noble N , Paul CL . 68.  2014 . Evaluation of systems-oriented public health interventions: alternative research designs. Annu. Rev. Public Health 35 : 9– 27 [Google Scholar]
• Shadish WR , Cook TD , Campbell DT . 69.  2002 . Experimental and Quasi-Experimental Designs for Generalized Causal Inference New York: Houghton Mifflin [Google Scholar]
• Shah BR , Laupacis A , Hux JE , Austin PC . 70.  2005 . Propensity score methods gave similar results to traditional regression modeling in observational studies: a systematic review. J. Clin. Epidemiol. 58 : 550– 59 [Google Scholar]
• Swanson SA , Hernán MA . 71.  2013 . Commentary: how to report instrumental variable analyses (suggestions welcome). Epidemiology 24 : 370– 74 [Google Scholar]
• Thomas J , O'Mara-Eves A , Brunton G . 72.  2014 . Using qualitative comparative analysis (QCA) in systematic reviews of complex interventions: a worked example. Syst. Rev. 3 : 67 [Google Scholar]
• van Leeuwen N , Lingsma HF , de Craen T , Nieboer D , Mooijaart S . 73.  et al. 2016 . Regression discontinuity design: simulation and application in two cardiovascular trials with continuous outcomes. Epidemiology 27 : 503– 11 [Google Scholar]
• von Elm E , Altman DG , Egger M , Pocock SJ , Gøtzsche PC . 74.  et al. 2008 . Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J. Clin. Epidemiol. 61 : 344– 49 [Google Scholar]
• Wagner AK , Soumerai SB , Zhang F , Ross-Degnan D . 75.  2002 . Segmented regression analysis of interrupted time series studies in medication use research. J. Clin. Pharm. Ther. 27 : 299– 309 [Google Scholar]
• Wanless D . 76.  2004 . Securing Good Health for the Whole Population London: HM Treas. [Google Scholar]
• Warren J , Wistow J , Bambra C . 77.  2013 . Applying qualitative comparative analysis (QCA) to evaluate a public health policy initiative in the North East of England. Policy Soc 32 : 289– 301 [Google Scholar]
• Yen ST , Andrews M , Chen Z , Eastwood DB . 78.  2008 . Food stamp program participation and food insecurity: an instrumental variables approach. Am. J. Agric. Econ. 90 : 117– 32 [Google Scholar]

Data & Media loading...

• Article Type: Review Article

Most Read This Month

Most cited most cited rss feed, review of community-based research: assessing partnership approaches to improve public health, nature and health, measuring social class in us public health research: concepts, methodologies, and guidelines, the epidemiology of depression across cultures, acute respiratory effects of particulate air pollution, racism and health: evidence and needed research, the social determinants of health: coming of age, the role of behavioral science theory in development and implementation of public health interventions, mediation analysis: a practitioner's guide, the prescription opioid and heroin crisis: a public health approach to an epidemic of addiction.

• History & Society
• Science & Tech
• Biographies
• Animals & Nature
• Geography & Travel
• Arts & Culture
• Games & Quizzes
• On This Day
• One Good Fact
• New Articles
• Lifestyles & Social Issues
• Philosophy & Religion
• Politics, Law & Government
• World History
• Health & Medicine
• Browse Biographies
• Birds, Reptiles & Other Vertebrates
• Bugs, Mollusks & Other Invertebrates
• Environment
• Fossils & Geologic Time
• Entertainment & Pop Culture
• Sports & Recreation
• Visual Arts
• Demystified
• Image Galleries
• Infographics
• Top Questions
• Britannica Kids
• Saving Earth
• Space Next 50
• Student Center
• Introduction

Comparison with controlled study design

Natural experiments as quasi experiments, instrumental variables.

• Where was science invented?
• When did science begin?
• Is Internet technology "making us stupid"?
• What is the impact of artificial intelligence (AI) technology on society?

natural experiment

Our editors will review what you’ve submitted and determine whether to revise the article.

• Table Of Contents

natural experiment , observational study in which an event or a situation that allows for the random or seemingly random assignment of study subjects to different groups is exploited to answer a particular question. Natural experiments are often used to study situations in which controlled experimentation is not possible, such as when an exposure of interest cannot be practically or ethically assigned to research subjects. Situations that may create appropriate circumstances for a natural experiment include policy changes, weather events, and natural disasters. Natural experiments are used most commonly in the fields of epidemiology , political science , psychology , and social science .

Key features of experimental study design include manipulation and control. Manipulation, in this context , means that the experimenter can control which research subjects receive which exposures. For instance, subjects randomized to the treatment arm of an experiment typically receive treatment with the drug or therapy that is the focus of the experiment, while those in the control group receive no treatment or a different treatment. Control is most readily accomplished through random assignment, which means that the procedures by which participants are assigned to a treatment and control condition ensure that each has equal probability of assignment to either group. Random assignment ensures that individual characteristics or experiences that might confound the treatment results are, on average, evenly distributed between the two groups. In this way, at least one variable can be manipulated, and units are randomly assigned to the different levels or categories of the manipulated variables.

In epidemiology, the gold standard in research design generally is considered to be the randomized control trial (RCT). RCTs, however, can answer only certain types of epidemiologic questions, and they are not useful in the investigation of questions for which random assignment is either impracticable or unethical. The bulk of epidemiologic research relies on observational data, which raises issues in drawing causal inferences from the results. A core assumption for drawing causal inference is that the average outcome of the group exposed to one treatment regimen represents the average outcome the other group would have had if they had been exposed to the same treatment regimen. If treatment is not randomly assigned, as in the case of observational studies, the assumption that the two groups are exchangeable (on both known and unknown confounders) cannot be assumed to be true.

As an example, suppose that an investigator is interested in the effect of poor housing on health. Because it is neither practical nor ethical to randomize people to variable housing conditions, this subject is difficult to study using an experimental approach. However, if a housing policy change, such as a lottery for subsidized mortgages, was enacted that enabled some people to move to more desirable housing while leaving other similar people in their previous substandard housing, it might be possible to use that policy change to study the effect of housing change on health outcomes. In another example, a well-known natural experiment in Helena , Montana, smoking was banned from all public places for a six-month period. Investigators later reported a 60-percent drop in heart attacks for the study area during the time the ban was in effect.

Because natural experiments do not randomize participants into exposure groups, the assumptions and analytical techniques customarily applied to experimental designs are not valid for them. Rather, natural experiments are quasi experiments and must be thought about and analyzed as such. The lack of random assignment means multiple threats to causal inference , including attrition , history, testing, regression , instrumentation, and maturation, may influence observed study outcomes. For this reason, natural experiments will never unequivocally determine causation in a given situation. Nevertheless, they are a useful method for researchers, and if used with care they can provide additional data that may help with a research question and that may not be obtainable in any other way.

The major limitation in inferring causation from natural experiments is the presence of unmeasured confounding. One class of methods designed to control confounding and measurement error is based on instrumental variables (IV). While useful in a variety of applications, the validity and interpretation of IV estimates depend on strong assumptions, the plausibility of which must be considered with regard to the causal relation in question.

In particular, IV analyses depend on the assumption that subjects were effectively randomized, even if the randomization was accidental (in the case of an administrative policy change or exposure to a natural disaster) and adherence to random assignment was low. IV methods can be used to control for confounding in observational studies, to control for confounding due to noncompliance, and to correct for misclassification.

IV analysis, however, can produce serious biases in effect estimates. It can also be difficult to identify the particular subpopulation to which the causal effect IV estimate applies. Moreover, IV analysis can add considerable imprecision to causal effect estimates. Small sample size poses an additional challenge in applying IV methods.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• My Bibliography
• Collections
• Citation manager

Save citation to file

Email citation, add to collections.

• Create a new collection
• Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

• Search in PubMed
• Search in NLM Catalog
• Add to Search

Conceptualising natural and quasi experiments in public health

Affiliations.

• 1 Population Health Sciences, Bristol Medical School, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, UK. [email protected].
• 2 NIHR School for Public Health Research, Newcastle, UK. [email protected].
• 3 NIHR Applied Research Collaboration West, Bristol, UK. [email protected].
• 4 MRC/CSO Social and Public Health Sciences Unit, University of Glasgow, Bristol, UK.
• 5 Population Health Sciences, Bristol Medical School, University of Bristol, Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, UK.
• 6 NIHR School for Public Health Research, Newcastle, UK.
• 7 MRC IEU, University of Bristol, Bristol, UK.
• PMID: 33573595
• PMCID: PMC7879679
• DOI: 10.1186/s12874-021-01224-x

Background: Natural or quasi experiments are appealing for public health research because they enable the evaluation of events or interventions that are difficult or impossible to manipulate experimentally, such as many policy and health system reforms. However, there remains ambiguity in the literature about their definition and how they differ from randomized controlled experiments and from other observational designs. We conceptualise natural experiments in the context of public health evaluations and align the study design to the Target Trial Framework.

Methods: A literature search was conducted, and key methodological papers were used to develop this work. Peer-reviewed papers were supplemented by grey literature.

Results: Natural experiment studies (NES) combine features of experiments and non-experiments. They differ from planned experiments, such as randomized controlled trials, in that exposure allocation is not controlled by researchers. They differ from other observational designs in that they evaluate the impact of events or process that leads to differences in exposure. As a result they are, in theory, less susceptible to bias than other observational study designs. Importantly, causal inference relies heavily on the assumption that exposure allocation can be considered 'as-if randomized'. The target trial framework provides a systematic basis for evaluating this assumption and the other design elements that underpin the causal claims that can be made from NES.

Conclusions: NES should be considered a type of study design rather than a set of tools for analyses of non-randomized interventions. Alignment of NES to the Target Trial framework will clarify the strength of evidence underpinning claims about the effectiveness of public health interventions.

Keywords: Evaluations; Natural experiments; Public health; Public health policy; Quasi experiments.

PubMed Disclaimer

Conflict of interest statement

The authors declare that they have no competing interests.

Different conceptualisations of natural and…

Different conceptualisations of natural and quasi experiments within wider evaluation frameworks

Similar articles

• Overview of the epidemiology methods and applications: strengths and limitations of observational study designs. Colditz GA. Colditz GA. Crit Rev Food Sci Nutr. 2010;50 Suppl 1(s1):10-2. doi: 10.1080/10408398.2010.526838. Crit Rev Food Sci Nutr. 2010. PMID: 21132580 Free PMC article.
• The future of Cochrane Neonatal. Soll RF, Ovelman C, McGuire W. Soll RF, et al. Early Hum Dev. 2020 Nov;150:105191. doi: 10.1016/j.earlhumdev.2020.105191. Epub 2020 Sep 12. Early Hum Dev. 2020. PMID: 33036834
• Using natural experiments to improve public health evidence: a review of context and utility for obesity prevention. Crane M, Bohn-Goldbaum E, Grunseit A, Bauman A. Crane M, et al. Health Res Policy Syst. 2020 May 18;18(1):48. doi: 10.1186/s12961-020-00564-2. Health Res Policy Syst. 2020. PMID: 32423438 Free PMC article.
• Natural experiments for the evaluation of place-based public health interventions: a methodology scoping review. Albers PN, Rinaldi C, Brown H, Mason KE, d'Apice K, McGill E, McQuire C, Craig P, Laverty AA, Beeson M, Campbell M, Egan M, Gibson M, Fuller M, Dillon A, Taylor-Robinson D, Jago R, Tilling K, Barr B, Sniehotta FF, Hickman M, Millett CJ, de Vocht F. Albers PN, et al. Front Public Health. 2023 Jun 22;11:1192055. doi: 10.3389/fpubh.2023.1192055. eCollection 2023. Front Public Health. 2023. PMID: 37427271 Free PMC article. Review.
• A framework for conducting economic evaluations alongside natural experiments. Deidda M, Geue C, Kreif N, Dundas R, McIntosh E. Deidda M, et al. Soc Sci Med. 2019 Jan;220:353-361. doi: 10.1016/j.socscimed.2018.11.032. Epub 2018 Nov 27. Soc Sci Med. 2019. PMID: 30513485 Free PMC article. Review.
• A mentally healthy framework to guide employers and policy makers. Deady M, Sanatkar S, Tan L, Glozier N, Gayed A, Petrie K, Dalgaard VL, Stratton E, LaMontagne AD, Harvey SB. Deady M, et al. Front Public Health. 2024 Jul 22;12:1430540. doi: 10.3389/fpubh.2024.1430540. eCollection 2024. Front Public Health. 2024. PMID: 39109149 Free PMC article. Review.
• The association between physical availability of cannabis retail outlets and frequent cannabis use and related health harms: a systematic review. Cantor N, Silverman M, Gaudreault A, Hutton B, Brown C, Elton-Marshall T, Imtiaz S, Sikora L, Tanuseputro P, Myran DT. Cantor N, et al. Lancet Reg Health Am. 2024 Mar 7;32:100708. doi: 10.1016/j.lana.2024.100708. eCollection 2024 Apr. Lancet Reg Health Am. 2024. PMID: 38486811 Free PMC article. Review.
• Estimating the effectiveness of an enhanced 'Improving Access to Psychological Therapies' (IAPT) service addressing the wider determinants of mental health: a real-world evaluation. Porter A, Franklin M, De Vocht F, d'Apice K, Curtin E, Albers P, Kidger J. Porter A, et al. BMJ Open. 2024 Jan 30;14(1):e077220. doi: 10.1136/bmjopen-2023-077220. BMJ Open. 2024. PMID: 38296286 Free PMC article.
• The impact of the newly developed school-based 'Digital Health Contact'-Evaluating a health and wellbeing screening tool for adolescents in England. Porter A, d'Apice K, Albers P, Woodrow N, Fairbrother H, Breheny K, Mills C, Tebbett S, De Vocht F. Porter A, et al. PLoS One. 2024 Jan 12;19(1):e0297016. doi: 10.1371/journal.pone.0297016. eCollection 2024. PLoS One. 2024. PMID: 38215072 Free PMC article.
• The longitudinal effects of the built environment on transportation and recreational walking, and differences by age and sex: systematic review protocol. Bandara TN, Higgs C, Zapata-Diomedi B, Gunn L, Turrell G, De Livera A. Bandara TN, et al. Arch Public Health. 2023 Oct 17;81(1):184. doi: 10.1186/s13690-023-01194-0. Arch Public Health. 2023. PMID: 37848953 Free PMC article.
• Shadish WR, Cook TD, Campbell DT. Experimental and Quasi-Experimental Designs. 2. Wadsworth, Cengage Learning: Belmont; 2002.
• King G, Keohane RO, Verba S. The importance of research Design in Political Science. Am Polit Sci Rev. 1995;89:475–481. doi: 10.2307/2082445. - DOI
• Meyer BD. Natural and quasi-experiments in economics. J Bus Econ Stat. 1995;13:151–161.
• Dunning T. Natural experiments in the social sciences. A design-based approach. 6th edition. Cambridge: Cambridge University Press; 2012.
• Craig P, Cooper C, Gunnell D, Haw S, Lawson K, Macintyre S, et al. Using natural experiments to evaluate population health interventions: new medical research council guidance. J Epidemiol Community Health. 2012;66:1182–1186. doi: 10.1136/jech-2011-200375. - DOI - PMC - PubMed

Publication types

• Search in MeSH

Related information

• Cited in Books

Grants and funding

• MC_UU_00022/2/MRC_/Medical Research Council/United Kingdom
• SCAF/15/02/CSO_/Chief Scientist Office/United Kingdom
• SPHR-PROG-PCA-WP3/School for Public Health Research
• MC_UU_12017/13/MRC_/Medical Research Council/United Kingdom
• MC_UU_00011/3/MRC_/Medical Research Council/United Kingdom
• SPHSU13/CSO_/Chief Scientist Office/United Kingdom
• SPHSU17/CSO_/Chief Scientist Office/United Kingdom

LinkOut - more resources

Full text sources.

• BioMed Central
• Europe PubMed Central
• PubMed Central

Other Literature Sources

• scite Smart Citations

Research Materials

• NCI CPTC Antibody Characterization Program

• Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

6. Natural Experiments and Quasi-Experiments

Library resources.

Natural Experiments

You have just learned that it is hard to infer causality from an observational study. With observational studies, it is usually much harder to control for confounding variables. However, researchers have come up with ways to control for confounding variables even when treatment assignment is not under the control of the researcher. One example of this is a research design called the  natural experiment.

We have a natural experiment when  variation in the independent variable is randomly assigned, but not by the researcher.  One example of a natural experiment was conducted in Brazil: researchers wanted to know if voters punish politicians when they discover that politicians are involved in corruption.

The researchers noticed that the federal government of Brazil conducted audits to find out whether or not the mayors of local towns are involved in corruption. The federal government did not have the capacity to conduct such audits in every single town, so they had to choose a group of towns to be audited.

To avoid suspicion that the audits were used to persecute opponents,  the government decided that the group of towns to be audited would be randomly selected.  This gave researchers an opportunity to test the following question: what happens to the reelection chances of mayors who get audited, when compared to mayors who do not get audited?

They found that, when compared to mayors who do not get audited, honest mayors who were audited and came clean were more likely to be reelected. By contrast, corrupt mayors who were audited and got caught were less likely to be reelected than mayors who were not audited. We know that these outcomes were caused by the audits because audits were randomly assigned.  Because of random assignment, the researchers were able to rule out confounding variables: towns that were audited are similar, on average, to towns that were not audited.

When a researcher notices that variation in an independent variable is random, and uses this random assignment to draw causal inference, we see an example of a natural experiment. The researchers exploited the fact that the Brazilian government randomly assigned variation in which towns get audited--the independent variable of interest.

Quasi-Experiments

A quasi-experiment is an experiment that looks like a true, randomized experiment, but lacks random assignment. One example of a quasi-experiment is when treatment is not randomly assigned, but we have reasons to believe that the treatment group is similar to the control group.

For example, think about a high school that wants to measure the impact of the Covid pandemic on student performance. At this school, the class of 2019 suffered no impact from the pandemic, the class of 2020 suffered some impact, and the class of 2021 suffered a lot of impact.  So long as we believe that students in the classes of 19, 20 and 21 are similar to each other , a comparison between these three classes should reveal something about the impact of the pandemic on student performance.

This research design  has its flaws  because assignment to treatment is not random. For example, the senior year of the class of 2021 had the added stress of a hectic election campaign and its violent aftermath. These additional stressors are potential confounds in this research design. But because we cannot randomly assign the impact of the pandemic on students' lives, a research design that compares across graduating classes may be as good as it gets.

• Ferraz, Claudio, and Frederico Finan. 2008. “Exposing Corrupt Politicians: The Effects of Brazil’s Publicly Released Audits on Electoral Outcomes.”  The Quarterly Journal of Economics  123 (2): 703–45.  https://www.jstor.org/stable/25098913 Links to an external site.
• Leatherdale, Scott T. 2019. “Natural Experiment Methodology for Research: A Review of How Different Methods Can Support Real-World Research.”  International Journal of Social Research Methodology  22 (1): 19–35.  https://doi.org/10.1080/13645579.2018.1488449 Links to an external site. .
• "Quasi-experimental Design", in Trochim, William M. K., and James P. Donnelly. 2006.  The Research Methods Knowledge Base . 3rd edition. Mason, Ohio: Atomic Dog.  https://conjointly.com/kb/quasi-experimental-design/ Links to an external site.

Next: Knowledge Check: RCTs, Natural Experiments, Quasi-Experiments, Observational Studies

Reference Library

Collections

• See what's new
• All Resources
• Student Resources
• Assessment Resources
• Teaching Resources
• CPD Courses
• Livestreams

Study notes, videos, interactive activities and more!

Psychology news, insights and enrichment

Currated collections of free resources

Browse resources by topic

• All Psychology Resources

Resource Selections

Currated lists of resources

Study Notes

Types of Experiment: Overview

Last updated 6 Sept 2022

• Share on Facebook
• Share on Twitter
• Share by Email

Different types of methods are used in research, which loosely fall into 1 of 2 categories.

Experimental (Laboratory, Field & Natural) & N on experimental ( correlations, observations, interviews, questionnaires and case studies).

All the three types of experiments have characteristics in common. They all have:

• an independent variable (I.V.) which is manipulated or a naturally occurring variable
• a dependent variable (D.V.) which is measured
• there will be at least two conditions in which participants produce data.

Note – natural and quasi experiments are often used synonymously but are not strictly the same, as with quasi experiments participants cannot be randomly assigned, so rather than there being a condition there is a condition.

Laboratory Experiments

These are conducted under controlled conditions, in which the researcher deliberately changes something (I.V.) to see the effect of this on something else (D.V.).

Control – lab experiments have a high degree of control over the environment & other extraneous variables which means that the researcher can accurately assess the effects of the I.V, so it has higher internal validity.

Replicable – due to the researcher’s high levels of control, research procedures can be repeated so that the reliability of results can be checked.

Limitations

Lacks ecological validity – due to the involvement of the researcher in manipulating and controlling variables, findings cannot be easily generalised to other (real life) settings, resulting in poor external validity.

Field Experiments

These are carried out in a natural setting, in which the researcher manipulates something (I.V.) to see the effect of this on something else (D.V.).

Validity – field experiments have some degree of control but also are conducted in a natural environment, so can be seen to have reasonable internal and external validity.

Less control than lab experiments and therefore extraneous variables are more likely to distort findings and so internal validity is likely to be lower.

Natural / Quasi Experiments

These are typically carried out in a natural setting, in which the researcher measures the effect of something which is to see the effect of this on something else (D.V.). Note that in this case there is no deliberate manipulation of a variable; this already naturally changing, which means the research is merely measuring the effect of something that is already happening.

High ecological validity – due to the lack of involvement of the researcher; variables are naturally occurring so findings can be easily generalised to other (real life) settings, resulting in high external validity.

Lack of control – natural experiments have no control over the environment & other extraneous variables which means that the researcher cannot always accurately assess the effects of the I.V, so it has low internal validity.

Not replicable – due to the researcher’s lack of control, research procedures cannot be repeated so that the reliability of results cannot be checked.

• Laboratory Experiment
• Field experiment
• Quasi Experiment
• Natural Experiment
• Field experiments

You might also like

Field experiments, laboratory experiments, natural experiments, control of extraneous variables, similarities and differences between classical and operant conditioning, learning approaches - social learning theory, differences between behaviourism and social learning theory, ​research methods in the social learning theory, our subjects.

• › Criminology
• › Economics
• › Geography
• › Health & Social Care
• › Psychology
• › Sociology
• › Teaching & learning resources
• › Student revision workshops
• › Online student courses
• › CPD for teachers
• › Livestreams
• › Teaching jobs

Boston House, 214 High Street, Boston Spa, West Yorkshire, LS23 6AD Tel: 01937 848885

• › Contact us
• › Terms of use
• › Privacy & cookies

© 2002-2024 Tutor2u Limited. Company Reg no: 04489574. VAT reg no 816865400.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 09 January 2024

Low-carbon transition and energy poverty: quasi-natural experiment evidence from China’s low-carbon city pilot policy

• Yiming Xiao 1 , 2 ,
• Zhijun Feng 1 ,
• Xinying Li 3 &
• Shangrui Wang 4  

Humanities and Social Sciences Communications volume  11 , Article number:  84 ( 2024 ) Cite this article

2161 Accesses

2 Citations

1 Altmetric

Metrics details

• Environmental studies
• Science, technology and society

Low-carbon transition stands as a vital strategy for the global community to address the challenge of climate change, inevitably affecting residents’ daily lives. However, there is a notable gap in the quantitative analysis of the low-carbon transition’s impact on energy poverty in developing countries, limiting policymakers’ understanding of the inherent mechanism and their ability to take informed actions. This study investigates the low-carbon city pilot (LCCP) policy, China’s key low-carbon initiative, as a quasi-natural experiment, using the difference-in-differences (DID) method to examine its impact on residents’ energy poverty conditions. Utilizing panel data from 4807 households in the CHARLS dataset, this study effectively integrated household-level and city-level data. Benchmark regression indicates that the LCCP policy exacerbates energy poverty among residents. Further analysis reveals the pivotal role of energy infrastructure and expenditure in bridging the nexus between the LCCP policy and energy poverty, providing crucial insights into the potential pathways through which this policy impacts energy poverty. Additionally, heterogeneity analysis indicates that the impacts of LCCP policy are more pronounced in eastern cities, non-resource cities, and high administrative-level cities, as well as in the communities suffering from subpar governance quality. By leveraging reliable survey data and robust quantitative methods, this study not only broadens the methodology of energy poverty studies but also offers valuable insights for developing countries to safeguard residents’ energy welfare amid low-carbon transitions.

Similar content being viewed by others

Exploring the determinants of energy poverty in Indonesia’s households: empirical evidence from the 2015–2019 SUSENAS

Empirical analysis of the role of the environmental accountability system in energy conservation and emission reduction in China

Costs and health benefits of the rural energy transition to carbon neutrality in China

Introduction.

Low-carbon transition has been widely endorsed by the international community as a crucial lever to mitigate global warming (He et al., 2022 ; Olabi and Abdelkareem, 2022 ). Currently, global efforts in the low-carbon transition have transformed energy structure and bolstered the use of clean and renewable energy, thus aiding in achieving carbon reduction goals (Yu et al., 2022 ; Zhang et al., 2022 ). However, in light of the classic “energy trilemma” predicament, efforts toward low-carbon transition, at times, have unintentionally impacted energy security and energy equity in certain regions (Mišík, 2022 ; Xie et al., 2022 ). As countries implement these low-carbon strategies, their energy systems and even whole socioeconomic systems have become increasingly unstable and vulnerable (Frilingou et al., 2023 ; Magacho et al., 2023 ; Semieniuk et al., 2021 ; Sovacool et al., 2019 ). During the COVID-19 pandemic and the exacerbation of geopolitical tensions, various countries have witnessed energy supply threats and energy market fluctuations, further intensifying the energy accessibility challenges for numerous populations (Belaïd, 2022b ; Carfora et al., 2022 ). Recent data reveals a startling 20% increase in the global population lacking sufficient energy service in daily life (Siksnelyte-Butkiene, 2022 ). Consequently, both scholars and policymakers must recognize the unforeseen repercussions of the low-carbon transition, particularly its implications for vulnerable groups in developing countries.

When residents grapple with challenges in getting enough energy services to sustain their daily lives, they are defined as trapped in an energy poverty condition (Liang and Asuka, 2022 ; Sy and Mokaddem, 2022 ). As the primary indicator assessing a resident’s energy welfare, energy poverty encompasses the difficulties residents face in accessing or affording fundamental modern energy services (González-Eguino, 2015 ; Nussbaumer et al., 2012 ). According to previous studies, energy poverty, underscored by indoor air pollution and diminished thermal comfort, disrupts residents’ daily activities, severely affecting their physical and psychological well-being (Xiao et al., 2021 ; Zhang et al., 2021 ). Furthermore, it also leads to a decline in productivity, thereby potentially exacerbating social inequities and hindering development in disadvantaged regions (Du et al., 2022 ; Liu et al., 2022 ; Shahzad et al., 2022 ). Recognizing the gravity of this issue, the United Nations ( 2012 ) considers universal access to modern energy services as a major goal by 2030.

Scholars have approached the tension between low-carbon transition and energy poverty from perspectives of equity and justice (Heffron, 2022 ). Since related policies were mainly formulated and executed by predominant governmental and corporate entities, the voice of the general populace is marginalized, further obstructing the realization of distributive, recognition, and procedural justice (Sovacool, 2021 ; Sovacool and Dworkin, 2015 ). During the low-carbon transition, on the one hand, the construction of wind/solar farms has encroached upon the arable lands that residents rely on for sustenance, exacerbating their impoverished conditions (Argenti and Knight, 2015 ; Gorayeb et al., 2018 ). On the other hand, such a transition has not only elevated the cost of energy production, transmission, and storage but also heightened the unpredictability of the energy system, inevitably increasing the risk of energy disruption and the economic burden of vulnerable groups (Geels et al., 2017 ; Mohseni et al., 2022 ; Tian et al., 2022 ). Recognizing the challenges the low-carbon transition posed, Belaïd ( 2022b ) has probed into the new forms of inequalities birthed by transition policies, offering an integrated framework for harmonizing the low-carbon transition and energy poverty governance in developing countries. Existing research advises policymakers to ensure residents’ welfare during the low-carbon transition, especially addressing the energy poverty issues confronted by vulnerable groups. Yet, current research still contains the following three gaps:

Firstly, previous studies on the impacts of the low-carbon transition on energy poverty often remain limited to qualitative discussions, lacking quantitative analysis. Secondly, academia primarily addresses the direct impact of the low-carbon transition, with a scant exploration into its underlying mechanisms or the heterogeneous effects under diverse governance scenarios. Thirdly, the focal point of most research predominantly rests on the developed countries, overlooking the specific challenges faced by vulnerable groups in developing countries. These research gaps hinder the governance implications of the pertinent conclusions, necessitating deeper exploration.

This study examines the relationship between low-carbon transition and energy poverty in developing countries, using China’s low-carbon city pilot (LCCP) policy as a quasi-natural experiment. Specifically, employing the difference-in-differences (DID) method, we assess the LCCP policy’s impact on residents’ energy poverty conditions. We use panel data from 4807 households containing middle-aged or senior members in the China Health and Retirement Longitudinal Study (CHARLS) and match the data with the LCCP policy implementation in China’s cities, shedding light on the macro-policy’s micro-impacts. Moreover, we explore the underlying mechanisms by which the LCCP policy exerts its impacts, emphasizing the two mediating variables, including energy expenditure and infrastructure. Lastly, we conduct a heterogeneity analysis to understand the policy’s impacts in cities and communities with different characteristics.

The study makes three significant contributions to existing literature. Firstly, this study offers a quantitative insight into the significant implications of low-carbon transitions on energy poverty in developing countries. With some cities in China adopting the LCCP policy and others yet to, China’s LCCP initiative emerges as an ideal quasi-natural experiment to probe the effects of such transitions (NDRC, 2014 ). While earlier scholars predominantly embraced qualitative analysis or case studies, this study conducts a deeper and more reliable analysis, providing quantitative evidence for the relationship between low-carbon transition and energy poverty (Ravigné et al., 2022 ; Upham et al., 2022 ). Secondly, this study innovatively combines city-level pilot policies with household-level data, examining the micro-impacts of macro-policy. In previous research, scholars either pursued macro analysis using regional data or probed individual factors impacting energy poverty using household data (Dong et al., 2021 ; Zhao et al., 2022 ). Anchored by reliable survey data and robust methods, this study broadens the methodology for energy poverty research. Finally, the quantitative analysis not only aids China’s policymakers in assessing the eventual impact of their LCCP policy on residents’ welfare but also provides valuable reference for developing countries charting their future low-carbon transition pathways.

The remainder of this paper is structured as follows. Section “Literature review, theoretical basis, and research framework” reviews existing literature and proposes the theoretical basis and research hypotheses. Section “Methodology and data” introduces the methodology, including the model construction, variable selection, and data source. Section “Results and discussion” presents the results, as well as a discussion of the main findings. Section “Robustness test” presents the robustness tests. Section “Conclusion and policy recommendations” summarizes the main conclusions and offers relevant policy implications.

Literature review, theoretical basis, and research framework

Literature review, determinants of energy poverty.

Energy poverty, also called fuel poverty, is a central theme highlighting residents’ welfare, which has attracted increasing scholarly interest. Since Boardman pioneered the 10% indicator for the energy poverty condition in the United Kingdom, various standards like the 2M indicator (double the median share of household expenditure on energy), low-income high cost (LIHC) indicator, and minimum living costs (MIS) indicator have emerged to determine whether residents are living in energy poverty or not (Boardman, 1991a ; Castaño-Rosa et al., 2019 ; Hills, 2012 ; Moore, 2012 ). To fully cover residents’ daily energy needs, Nussbaumer et al. ( 2012 ), drawing inspiration from the Oxford Poverty and Human Development Initiative, introduced the multidimensional energy poverty index (MEPI), encapsulating a range of daily energy necessities, from cooking, heating, entertainment to communication. Furthermore, based on the LIHC indicator, Belaïd and Flambard ( 2023 ) integrated three aspects, including income, energy, and housing costs, presenting a more holistic conceptual framework. Scholarly refinements in energy poverty indicators lay a solid foundation for further analysis.

The determinants of energy poverty have been studied from macro or micro perspectives. At the macro level, factors like regional economic development, resource allocations, and technological development played pivotal roles in determining the energy poverty of certain regions (Liang and Asuka, 2022 ; Wang and Hao, 2018 ; Xiao et al., 2023 ). At the micro level, factors such as household income, age, educational level, as well as societal belief were found to be associated with residents’ energy poverty conditions (Awaworyi Churchill and Smyth, 2022 ; Belaïd, 2022a ; Belaïd and Flambard, 2023 ; Fry et al., 2022 ; Hasanujzaman and Omar, 2022 ). Yet, current studies focus predominantly on general factors, neglecting the impacts of government-led policies represented by the low-carbon transition. Such oversights limit the depth and clarity of insights into fluctuations in residents’ energy poverty conditions.

Challenges brought by low-carbon transition

Although the low-carbon transition has engendered numerous positive effects for the societal ecosystem, researchers have begun casting light upon the trade-off between such a transition and residents’ welfare. As early as 1991, Boardman ( 1991b ) argued that the introduction of carbon tax policy in the United Kingdom could cast a shadow upon the welfare of impoverished households. In the process of implementing a low-carbon policy, if stakeholders fail to simultaneously enhance energy efficiency, some households may be triggered into the predicament of high energy expenditure (Ürge-Vorsatz and Tirado Herrero, 2012 ). Nguyen et al. ( 2019 ) revealed that as Vietnamese households progressed from traditional to modern energy systems, there was a marked escalation in expenditure-based energy poverty. In the solar energy industry, the fabrication of thin-film solar panels, while advantageous for certain regions’ low-carbon transition, also harbors potential health risks for manufacturing workers (Mulvaney, 2014 ). In fact, numerous endeavors aimed at energy transition, including architecture modifications, household solar panels, and electric vehicles, have precipitated varying degrees of discrimination and injustice among people, with a more conspicuous impact on vulnerable groups (Sovacool, 2021 ).

Diving deeper into the effects of the low-carbon transition on residents’ welfare, researchers have pursued comprehensive studies from the perspective of energy justice. Setyowati ( 2021 ) examined the Indonesian government’s efforts to achieve energy justice during the low-carbon transition and found that these endeavors inadvertently led to the further exclusion and disempowerment of energy-poor communities in energy-related decisions. In China’s context, Wang and Lo ( 2022 ) investigated the country’s journey toward justice during the energy transition, using the case of the environmental organization “Friends of Nature.” They suggested that China’s approach is distinctively different from the West, primarily based on Confucian self-cultivation. Sovacool et al. ( 2019 ) studied low-carbon initiatives in various countries, including France’s nuclear power, the UK’s smart meters, Norway’s electric vehicles, and Germany’s solar energy, and identified 120 energy injustices and introduced a strategic framework that includes distributive justice, procedural justice, cosmopolitan justice, and recognitional justice to ensure a fairer transition. While many scholars elucidate deprivation and inequity during the low-carbon transition by case studies, the lack of quantitative data makes it difficult to truly understand the degree to which certain low-carbon transition practices contribute to energy poverty.

China’s low-carbon city pilot policy

As the world’s most populous developing country, China’s move toward a low-carbon transition might place a considerable burden on its residents (Bai et al., 2023 ). The “Coal-to-Gas” initiative launched in 2017 inadvertently resulted in a shortage of natural gas, leading to an inability for many households in northern China to heat their rooms (Luo et al., 2021 ; Wang and Ren, 2020 ). Furthermore, China’s environmental protection law (Ma et al., 2022 ), as well as local environmental regulations (Xiao et al., 2023 ), have intensified energy poverty issues, particularly for households dependent on non-clean energy. However, a comprehensive quantitative analysis of the impact of China’s low-carbon transition policy on residents’ energy poverty is still lacking.

China’s Low-Carbon City Pilot (LCCP) policy forms a critical part of the country’s broader low-carbon transition strategy (Yang et al., 2023b ). In pursuit of exploring efficient pathways towards carbon emission reduction, the National Development and Reform Commission selected cities to roll out the LCCP policy in 2011, 2013, and 2017. Directed by the central government, each pilot city, which reflected its own socio-economic characteristics, set individual carbon peaking objectives, established comprehensive greenhouse gas emission tracking systems, and employed both legal and economic mechanisms to encourage stakeholders to act accordingly (NDRC, 2014 ). The LCCP policy’s overarching ambition is to overhaul the energy framework, augment energy efficiency, and achieve tangible reductions in greenhouse gas emissions.

The LCCP policy demands more in-depth exploration regarding its implications for residents. However, academic investigations have predominantly focused on the policy’s broader outcomes, such as carbon emissions, ecological preservation, energy efficiency, innovation, and sustainable growth, highlighting its positive effects based on provincial or city-level data (Yang et al., 2023a ; Zhang, Feng, et al. 2022 ; Zhu and Lee, 2022 ). Previous studies risk overlooking the intricate impacts on residents’ energy welfare. Considering that residents utilize various forms of energy—like electricity, gasoline, and coal—accounting for around 20% of the overall societal energy usage (Shen and Shi, 2018 ), the implications of the LCCP policy on the energy system inevitably cascade down to residents, influencing their energy welfare.

Theoretical basis

Drawing upon existing research, this study centers on the theories of quasi-public goods and energy justice (Belaïd, 2022b ; Belaïd and Flambard, 2023 ; Xiao et al., 2023 ). Both theories, grounded in human rights perspectives, offer a qualitative explanation for the latent correlation between governmental actions toward low-carbon transition and the energy poverty conditions of residents.

The theory of quasi-public goods concerns those goods that lie between the private and public domains (Buchanan, 1965 ; Savas, 1999 ). Unlike clear-cut public or private goods, quasi-public goods are partly non-rivalrous and non-excludable. Currently, utilities such as energy, water, and communication exhibit characteristics of quasi-public goods, with energy being a prime example (Zhao et al., 2015 ). Energy is vital for residential life, requiring residents to bear associated costs for their daily consumption. However, the energy sector is largely dominated by suppliers who possess inherent monopolistic characteristics (Wang and Chen, 2012 ). Given that the infrastructures of electricity and natural gas in specific regions serve a multitude of users and are irreplaceable in function, residents face stark limitations in choosing suppliers and struggle to find better suppliers based on free-market principles. As governments advocate for low-carbon transitions, energy suppliers might face increased costs, raising terminal energy prices. Due to the monopolistic nature of the energy sector, residents cannot easily switch to cheaper alternatives, thus risking increased energy costs, supply interruptions, and subsequent energy poverty.

Energy justice, viewed as the “ethical turn” in current energy policies and related research, aims to address the marginalization of vulnerable populations in policy formulation and implementation (Hartwig et al., 2023 ). Instead of viewing energy policies solely as technical solutions to climate issues, energy justice sees energy systems as a socially embedded phenomenon calling for a politically and morally informed response (McHarg, 2020 ). This perspective underscores the importance of prioritizing vulnerable groups during the low-carbon transition and addressing the inherent injustices and inequalities (Bouzarovski and Simcock, 2017 ; Jenkins et al., 2021 ; Sovacool et al., 2023 ). McCauley et al. ( 2013 ) and Jenkins et al. ( 2016 ) initially framed energy justice in terms of distribution, recognition, and procedure. Later scholars have added restorative and cosmopolitan justice to this framework (Heffron, 2022 ). Given the quasi-public nature of energy, the impact of low-carbon transition on residents’ welfare is unavoidable. Energy justice enhances this argument, incorporating the justice dimension into the core values of governance, providing policymakers with a framework to identify and counter the ethical dilemma of low-carbon transition.

In summary, these theories provide an integrated consideration of climate, economy, and ethics for the formulation and implementation of energy policy. Quasi-public goods theory highlights the added burden residents face due to low-carbon transitions, while energy justice theory offers ethical benchmarks to address this issue. Using these theories as a foundation, this study investigates the nexus between low-carbon transitions and energy poverty in developing countries, utilizing quantitative analysis informed by China’s LCCP policy.

Research hypotheses

To fill the research gap, this study treats LCCP policy as a quasi-natural experiment and employs the DID approach to delve deeper into the impacts of LCCP policy on the residents’ energy poverty conditions, thereby advancing the understanding of the effects of low-carbon transition on energy poverty in developing countries.

Fundamentally, the LCCP policy is composed of a series of concrete emission reduction measures, forming a comprehensive policy system (Li et al., 2018 ; Wang et al., 2015 ). To meet stringent emission goals, local governments employ legal constraints and financial support to urge various stakeholders to reduce emissions, thereby driving the transformation of the societal energy structure (Feng and Chen, 2018 ; Khanna et al., 2014 ; Song et al., 2020 ). Although the main implementers of LCCP policy are the government and related enterprises, with few direct restrictions imposed on residents, residents will inevitably be affected by the aforementioned measures as the ultimate consumers of energy (Sovacool, 2021 ).

Primarily, the LCCP policy can exacerbate residents’ energy poverty conditions by increasing necessary living energy expenditure. On the one hand, in a bid to optimize industrial and energy structure, the government propels solar power, natural gas, electricity, and other advanced energy to supplant outdated energy sources such as coal (Li et al., 2018 ). Some archaic enterprises may even face constraints or closures, inevitably leading to an energy supply shortage. In fact, inherent governance defects have further intensified this shortage resulting from energy structure upgrading (Luo et al., 2021 ). In China, the domestic natural gas shortage and electricity shortage that occurred in 2017 and 2021, respectively, are concrete manifestations of this predicament. On the other hand, energy enterprises, in order to comply with government emission reduction requirements and ensure normal operation, may invest more funds into technology upgrades and facility renovations, thereby escalating energy production costs (Amores-Salvadó et al., 2014 ; Sarkis and Cordeiro, 2001 ). Consequently, these enterprises pass on these costs to consumers when providing energy services, causing residents to bear the economic cost of cities’ low-carbon transition (Zhang, 2018 ). In fact, energy prices have nearly doubled during some regions’ low-carbon transition, leaving residents facing severe energy poverty issues (Frondel et al., 2015 ).

However, it is necessary to note that modern energy infrastructures established by the LCCP policy could potentially alleviate residents’ energy poverty conditions. During policy implementation, governments encourage enterprises and other stakeholders to construct modern infrastructures for energy production, transmission, and distribution (Li et al., 2018 ). In China, as a result of the construction of large-scale power grids and natural gas networks, numerous residents have transitioned from using solid fuels such as coal or straw to modern energy (Yang et al., 2020 ). Previous research has demonstrated that well-developed energy infrastructures are crucial prerequisites for residents to get rid of energy poverty (Lippert and Sareen, 2023 ). Thus, energy infrastructure should also be taken into consideration when exploring the impact of LCCP policy on energy poverty.

Given the above analysis, the exact impacts of LCCP policy on energy poverty and the intermediary mechanisms still warrant further exploration. Therefore, we propose three hypotheses as follows, and the impact path is shown in Fig. 1 .

Impact path of the LCCP policy on energy poverty.

Hypothesis 1: The LCCP policy exacerbates residents’ energy poverty condition .

Hypothesis 2: The LCCP policy exacerbates residents’ energy poverty condition through increasing energy expenditure .

Hypothesis 3: The LCCP policy alleviates residents’ energy poverty conditions through energy infrastructure construction .

Methodology and data

Model construction, general form of difference-in-differences model.

The implementation of specific public policies may impact certain groups while leaving other groups unaffected. Thus, it can be likened to a particular “treatment” administered to subjects in a medical experiment. Much like research in natural sciences, events in social science studies that alter the environment of individuals or cities in society are often referred to as quasi-natural experiments. If a specific public policy is seen as a quasi-natural experiment, then by comparing the individuals affected by the policy (treatment group) with the individuals unaffected (control group), one can discern the effects brought forth by the policy (Zhou and Chen, 2005 ).

The DID method is often employed to investigate the effects of public policy implementation from the perspective of quasi-natural experiments. Specifically, the DID method uses the dual differences in cross-sections and time series introduced by the public policy to identify the policy’s “treatment effect” (Zhou and Chen, 2005 ). Its merit lies in circumventing the endogeneity issues when using policy as an explanatory variable and effectively controlling the interaction between dependent and independent variables. The DID model with panel data can account for unobservable individual heterogeneity among samples and control for unobservable factors that change over time, thereby achieving an unbiased estimation of policy effects (Fan et al., 2017 ). The general form of the DID model is shown in Eq. ( 1 ). Herein, \(y_{it}\) represents the dependent variable. The interaction term \(\left( {G_i \times D_t} \right)\) indicates if the region of residence for individual i implemented a specific policy in year t . A value of 1 confirms this, while 0 negates it. \(X_{it}\) includes control variables that could impact the dependent variable. μ and ε , respectively, represent the fixed effect and the error term.

Difference-in-differences model for LCCP policy

The objective of this study is to delve into the impact of LCCP policy on residents’ energy poverty conditions. Drawing from previous analysis, the effect of the LCCP policy can be perceived as a quasi-natural experiment. Given that selected pilot cities implemented the LCCP policy, their residents are inevitably under its sway. Conversely, residents of no-pilot cities remain unaffected. Thus, residents in pilot cities can be categorized as the treatment group, and those in non-pilot cities can be categorized as the control group. Utilizing the DID method, we can scrutinize the impact of LCCP policy by investigating differences before and after policy intervention, as well as differences between treatment and control groups at the same time point (Q. Shen et al., 2023). The benchmark DID model is shown in Eq. ( 2 ).

Herein, i and t denote specific residents and years, respectively. \({\rm {MEPI}}_{it}\) signifies the energy poverty condition experienced by resident i in the year t . \({\rm {LCCP}}_{it}\) denotes whether the city where resident i lives implemented the LCCP policy in the year t , and a value of 1 indicates affirmation, whereas 0 indicates negation. \(C_{it}\) embodies control variables that could impact the residents’ energy poverty. \(\mu _i\) and \(\sigma _t\) correspondingly represent the fixed effects of residents and years, while \(\varepsilon _{it}\) constitutes the error term. In this model, the coefficient \(\beta _1\) captures the shock of LCCP policy on energy poverty, with a positive value indicating an exacerbation effect, a negative value indicating an alleviation effect, and an insignificant value suggesting no substantial impact.

Regarding the intermediary effects of energy expenditure and energy infrastructure, we construct the following model, as depicted in Eqs. ( 3 ) and ( 4 ), to delve into the intermediary mechanisms.

In this model, \({\rm {Mediat}}_{it}\) represents the mediating variables. The coefficient \(\beta _1\) captures the impact of LCCP policy on mediating variables, while the coefficient \(\beta _4\) captures the impacts of mediating variables on residents’ energy poverty. A statistically significant value for both coefficients indicates the existence of intermediary effects, while an insignificant value suggests no such effect.

The applicability of difference-in-differences model

When examining the impact of the LCCP policy on the energy poverty of residents, it is imperative to meet the following two fundamental prerequisites: (1) Random City Selection for Pilots : The process of selecting low-carbon pilot cities should be random, free from biases that might affect the dependent variable. Current literature and official statements suggest that policymakers have not considered residents during pilot city selection (Deng and Zhan, 2017 ). Our analysis of energy poverty conditions across low-carbon pilot cities shows varied values, indicating that city selection is random to some extent. (2) Parallel trends : Prior to the implementation of the LCCP policy, residents’ energy poverty conditions in pilot cities should have a similar trend as those in no-pilot cities. This will be further elaborated upon in the section “Parallel trend test”.

Variable selection

Dependent variable.

Residents’ energy poverty condition serves as the dependent variable in this model, referring to the challenges residents confront in accessing or affording modern energy services. We adopt the multidimensional energy poverty index (MEPI) framework, the widely accepted measurement proposed by Nussbaumer et al. ( 2012 ), to measure residents’ energy poverty condition (Zhang, Appau et al., 2021 ). Specifically, we refine some indicators of MEPI to reflect China’s unique circumstances more precisely. Finally, we developed a modified MEPI indicator system, including five dimensions (cooking, room temperature, household appliances, education/entertainment, and communication) and 10 specific indicators. Considering each dimension holds significant importance in household living, we assign each dimension an equal weight of 0.2 (Zhang et al., 2019 ). However, for indicators within each dimension, we employ the entropy method to assign weights, thereby avoiding subjective biases within specific dimensions (Feng et al., 2022 ; Zhang, Shu et al., 2021 ). The MEPI indicator system and corresponding weights of indicators are shown in Table 1 .

According to the MEPI indicator system in this study, if a household’s condition meets the criterion for residents’ energy poverty, we will assign the indicator value to 1; otherwise, it will be assigned to 0. Specifically, if a household (1) uses non-modern energy sources (coal, straw, etc.) in cooking; (2) has no air conditioning; (3) has poor thermal comfort (too cold or hot); (4) has no refrigerator; (5) has no washing machine; (6) has no hot water supply; (7) has no television; (8) has no computer; (9) has no mobile phone; (10) has no internet, the corresponding indicator’s value is assigned to 1. Finally, these values are aggregated according to their respective weights to calculate the final MEPI, as shown in Eq. ( 5 ). The higher the MEPI of the residents’ households, the more severe their energy poverty condition.

Independent variable

The LCCP policy serves as the independent variable within this model. As previously mentioned, when a specific city was chosen as an LCCP pilot in a certain year, the variable LCCP for that year and all subsequent years will be assigned to 1; otherwise, it will be assigned to 0. China’s LCCP policy has undergone three batches: the first batch commencing in 2011, the second in 2013, and the final batch in 2017. The first batch was primarily aimed at provincial administrative regions, while the third batch had an excessively brief duration, both being unsuitable for this study (Zhao and Wang, 2021 ). Hence, this research selects the second batch of low-carbon pilot cities as a treatment group, while cities not identified as low-carbon pilot cities are utilized as a control group. In consideration of the availability of household-level data from the CHARLS database, 13 cities were finally chosen as the experimental group, and 85 cities as the control group.

Mediating variable

Energy expenditure is a mediating variable. Within China’s economic situation, the price of transportation fuels, such as petrol, often fluctuates due to market dynamics (Ju et al., 2017 ). In contrast, residential electricity prices largely retain their stability, primarily due to governmental constraints (Li et al., 2023 ). Consequently, for a household, expenditures on domestic electricity can function as a reference benchmark, while expenditures on transportation might effectively serve as an indicator reflecting energy price fluctuations. Accordingly, we utilize the ratio between the transportation fee and the electricity fee of a household to measure energy expenditure.

Energy infrastructure is another mediating variable. Regarding infrastructural developments in China, natural gas, an innovative fuel advocated by governments in recent years, its pipeline construction can serve as a relatively precise barometer of the progress in energy infrastructure (Dong, Jiang et al., 2021 ; Dong et al., 2017 ). Thus, we utilize household natural gas supply as a measurement of energy infrastructure.

Control variable

Eight control variables are incorporated at the city level and household level, thereby enhancing the accuracy of our parameter estimates and alleviating biases derived from omitted variables. In light of previous research, at the city level, we incorporate variables including economic development, population, industrial structure, and societal consumption (Dong et al., 2021 ; Ren et al., 2022 ; Zhao et al., 2022 ). Specifically, we (1) use per capita GDP to denote economic development, (2) use the year-end total population as a measure of population, (3) use the ratio of the secondary industry’s added value to GDP as a measure of industrial structure, and (4) use the total retail sales of consumer goods to represent societal consumption. At the household level, we include (5) household income, (6) household size, (7) marital status, and (8) the age of respondents (Abbas et al., 2020 ; Hong et al., 2022 ; Rahut et al., 2019 ).

Data source

This study utilizes household-level data from the China Health and Retirement Longitudinal Study (CHARLS) conducted by Peking University in collaboration with other institutions. This exhaustive survey employs a multistage stratified sampling methodology and rigorous survey process, guaranteeing regional representation and data quality (Peking University, 2023 ). CHARLS commenced its benchmark survey in 2011 and followed up in 2013, 2015, and 2018. The dataset encompasses households from 28 provinces, and more than 400 communities, offering rich information with a substantial sample size, fulfilling the requirements of this study.

CHARLS predominantly focuses on China’s households containing middle-aged or elderly members and collects household-level data, including income, consumption, and other routine activities. Considering the traditional Chinese family structure where middle-aged or elderly individuals often cohabitate with their offspring or kin, the CHARLS dataset aptly mirrors the typical Chinese household composition, portraying the evolving aging society in China (Wu, 2022 ; Yi and Wang, 2003 ). Therefore, if the energy poverty conditions of the households in CHARLS were confirmed to be impacted by LCCP policy, it would underscore the potential of low-carbon transition to alter energy poverty landscapes in developing countries. CHARLS publicly disclosed the cities where these households were located when starting the longitudinal survey. Leveraging this information, we can easily match these households with their respective cities, further establishing panel data to investigate the impact of the LCCP policy on residents’ energy poverty conditions (Li et al., 2022 ).

Our study incorporates household-level data from four waves of CHARLS surveys (conducted in 2011, 2013, 2015, and 2018) that maintained continuous tracking of these households. We utilize the primary characteristics and energy consumption data of these households for the dependent variable MEPI, mediating variables, and household-level control variables. Additionally, we gather city-level control variables—including per capita GDP, population, industrial structure, and societal consumption—from national and city statistical yearbooks.

Utilizing DID regression on household-level panel data, we surpass the scope of previous region-based studies, enabling us to capture dynamic processes at the micro level and thus facilitating a deeper analysis. Table 2 illustrates the variable measurements. Tables 3 and 4 provide the descriptive statistics and characteristics of key variables. To reduce heteroskedasticity, we apply logarithmic transformations (Numan et al., 2023 ) for variables including per capita GDP, population, societal consumption, and household income. Finally, we have collected panel data from 4807 households from the years 2011, 2013, 2015, and 2018, yielding a total of 19,228 observations. The MEPI for these households ranges between 0 and 1, with a mean value of 0.330, thereby delineating a representative snapshot of energy poverty among Chinese residents. These households are distributed across a range of city types, including 13 pilot cities and 85 non-pilot cities, thus offering a wide-ranging representation of the manifold city types within China.

Result and discussion

The evolution of residents’ energy poverty condition.

Using the previously outlined MEPI indicator system, we are able to calculate the MEPI index for households and trace the energy poverty condition of 4807 households from 2011 to 2018. As depicted in Fig. 2 , the Sankey diagram illuminates the overall evolution of energy poverty within these sampled households, as well as the relative proportion of households experiencing varying degrees of poverty.

The evolution of residents’ energy poverty condition.

Upon a comprehensive overview in Fig. 2 , the period from 2011 to 2018 witnessed a gradual decline in severe-energy-poverty households with an MEPI over 0.75, paralleled by an increasing trend of no-energy-poverty households with an MEPI below 0.25. This implies a gradual alleviation of the overall energy poverty situation in China. However, throughout the 8-year interval from 2011 to 2018, despite the increasing number of no-energy-poverty households, there persistently existed a segment of originally no-energy-poverty households transitioning into light, moderate, or severe energy poverty during 2011–2013, 2013–2015, or 2015–2018. Particularly during 2013–2015, around 20% of originally no-energy-poverty households transitioned into light energy poverty, 7% into moderate, and 1% into severe energy poverty. The count of households transitioning into poverty during 2013–2015 exceeded that of any other interval before or after. This suggests a possible existence of an exogenous shock significantly impacting residents’ energy poverty conditions, which could likely result from several cities being designated as low-carbon pilots since 2013.

In the following section, we will employ the DID approach to explore whether the LCCP policy can lead to a change in residents’ energy poverty conditions.

Benchmark regression

Table 5 delineates the benchmark regression results of LCCP policy impact on energy poverty based on household-level panel data. Moving from column (1) to column (3), the coefficients of LCCP policy are significantly positive, irrespective of whether time-fixed or household-fixed effects are controlled. Furthermore, in column (4), when we simultaneously control both time-fixed and household-fixed effects, the coefficient of LCCP policy is 0.0218 at the 1% level. In other words, compared to the control group, the LCCP policy has exacerbated the energy poverty condition of residents in the pilot cities by 0.0218, thereby supporting Hypothesis 1. Sovacool et al. ( 2022 ) expound that the low-carbon transition is not a panacea devoid of detriments, and some actions towards low-carbon transition may indeed precipitate fresh inequities and risks. Our empirical analysis uncloaks the aggravating impact of the LCCP policy on residents’ energy poverty, viewed from the perspective of inhabitants’ welfare.

Control variables at both the city and household levels are incorporated into the DID model to mitigate omitted variable bias. The regression results in column (4) of Table 5 reveal that economic development (GDP), household income (INCOM), household size (HOUSIZE), and marital status (MARRIAG) exert significant influence on residents’ energy poverty conditions. Among these, higher economic development, higher household income, and larger household size serve to alleviate energy poverty, consistent with previous studies (Ren et al., 2022 ; Zou and Luo, 2019 ). Intriguingly, the absence of marital relationships appears to alleviate energy poverty, which could be explained from a feminist perspective: within married households, women are typically tasked with energy consumption-related domestic labor (Amigo-Jorquera et al., 2019 ; Robinson, 2019 ). However, women’s labor is often undervalued, leading to a lack of motivation within these households to upgrade their energy sources (Heltberg, 2005 ). In contrast, within unmarried or divorced households, women assume control of energy upgrades, thus effectively liberating themselves from energy poverty (Azhgaliyeva et al., 2021 ). In addition, the insignificance of other control variables might be attributed to complex nonlinear relationships (Yang et al., 2023a ).

Intermediary mechanism

The aforementioned regression confirms that the LCCP policy can exacerbate residents’ energy poverty conditions. Delving further, we elucidate the intermediary mechanism through the regression presented in Table 6 . Columns (1) and (2) scrutinize the mediating effect of energy expenditure, columns (3) and (4) scrutinize the mediating effect of energy infrastructure, whereas column (5) gauges the joint impact of both on energy poverty. Results indicate that both energy expenditure and infrastructure play significant intermediary roles, while their effects are diametrically opposed. On one hand, the LCCP policy significantly enhances energy expenditure, subsequently exacerbating energy poverty. Hypothesis 2 is thus confirmed. On the other hand, the policy bolsters the construction of energy infrastructure, thereby alleviating energy poverty, and Hypothesis 3 is verified. Taken together, LCCP policy could exacerbate energy poverty, which is consistent with the previous benchmark regression.

Heterogeneity analysis based on city characteristics

This study encompasses 98 cities in China. Cities located at different geographical positions exhibit substantial variations in their resource endowment, scales, and administrative levels. To delve deeper into whether the LCCP policy’s impacts differ across cities with distinct characteristics, we conduct a comprehensive heterogeneity analysis as follows.

Cities’ natural conditions, including geographical location and resource endowment, are taken into consideration. Cities are categorized into eastern, central, and western regions, referenced from previous studies (State Council, 2000 ; Zheng and Shi, 2017 ). Subsequently, cities are bifurcated based on their resource endowments into non-resource and resource-dependent cities, in alignment with the National Resource-based City Sustainable Development Plan issued by the State Council ( 2013 ). The regression results are represented in columns (1)–(5) of Table 7 .

The implementation of the LCCP policy in China’s eastern region could markedly intensify energy poverty, with a significant increase of approximately 0.0425 at a 1% level. Yet, this policy’s impacts on energy poverty in the central and western regions remain negligible. As the most economically vibrant region of China, the eastern region exhibits a keen response to supply-demand dynamics in energy pricing (Cai et al., 2023 ; He et al., 2016 ). Consequently, policy shifts have a swift and palpable impact on energy consumption at the household level. In contrast, the central and western regions with lower levels of economic development and marketization (Ren et al., 2018 ), exhibit a certain “inertia” in energy prices, and the energy poverty condition of residents in these regions also tends to remain unchanged during LCCP policy implementation.

In non-resource cities, the LCCP policy significantly exacerbates energy poverty, with a coefficient of 0.0345, whereas this impact is not significant in resource-dependent cities. Non-resource cities rely on imported energy from other cities or regions, which extends the energy supply chain and escalates acquisition costs (Qiu et al., 2021 ). Consequently, the disruption to their energy supply and household energy consumption by LCCP policy is more pronounced. On the other hand, resource-dependent cities usually satisfy their energy needs locally or nearby, ensuring shorter supply chains and swift demand response, further effectively mitigating the LCCP policy’s impact on the entire energy system and residents’ energy poverty. Interestingly, most resource-dependent cities are located in central and western China, while non-resource cities are chiefly located in the east, and the regression results for these two city types could offer mutual corroboration. Broadly speaking, non-resource cities, particularly those in the east, should be cautious when implementing the LCCP policy, paying keen attention to the energy welfare of their residents.

Heterogeneity analysis of cities’ social conditions, including administrative level and city scale, is also conducted. Cities are classified by administrative levels: high-level (sub-provincial city or municipality) and low-level (prefecture-level city). Furthermore, we partitioned cities into small (populations under 5 million), medium (populations between 5 and 10 million), and large (populations over 10 million), and the corresponding regression results are presented in columns (1)–(5) of Table 8 .

As shown in Table 8 , the impact of LCCP policy is significant in high-level or big cities, in contrast to low-level or small cities. The above two regressions can be explained together: high-level city or large city tends to be more developed, leading its residents to adopt large amounts of modern energy (Liu et al., 2012 ; Ouyang and Hokao, 2009 ). However, the energy composition of these cities is relatively monolithic, largely relying on single sources such as electricity or natural gas, with limited options for energy substitution. Overdependence on single sources risks plunging these cities’ residents into energy poverty during supply fluctuation caused by LCCP policy. Conversely, small cities with low administrative levels, despite some degree of energy poverty, exhibit a broader energy composition in residents’ daily lives, including electricity, natural gas, liquefied petroleum gas, biogas, etc. (Cui et al., 2019 ; Peidong et al., 2009 ), thus these residents’ energy poverty conditions are less sensitive to the LCCP policy targeted to the specific type of energy.

Heterogeneity analysis based on community characteristics

349 distinct communities are included in this study. As the fundamental administrative unit in China’s society, the community is linked to every resident’s daily life. The LCCP policy’s impact could vary significantly across communities with diverse characteristics. Initial surveys by CHARLS exhaustively charted the inherent features of communities, providing a complete dataset for community heterogeneity analysis.

From the perspective of community governance, communities are stratified based on public service quality (high or low, dependent on whether officials can speak Mandarin or not), and public expenditure (high or low, dependent on whether it exceeds the 20,000 Yuan threshold). The regression results are outlined in columns (1)–(4) of Table 9 . The regression results show a significant impact of the LCCP policy, which amplifies residents’ energy poverty in communities with low public service quality and low public expenditure. This suggests that the impacts of the LCCP policy vary with changes in community governance. In simpler terms, when grassroots governance is inadequate, the negative impacts of the LCCP policy become significantly more prominent. Grassroots governments, such as community administrators, serve as a ‘shield’ in mitigating potential energy poverty risks among residents during the low-carbon transition (Martiskainen et al., 2018 ).

In addition, the regression analysis presented in columns (5) and (6) of Table 9 reveals contrasting impacts of the LCCP policy on residents of urban and rural communities. Specifically, while the impact on urban residents’ energy poverty proves insignificant, it is significant in rural communities. These findings align with previous heterogeneity analyses of public service quality and public expenditure. The significant urban-rural gap and governance practice in China give urban residents an advantage in accessing abundant resources and favorable energy policies, thereby enabling them to mitigate the potential deleterious effects of the LCCP policy (Lu et al., 2022 ; Yao and Jiang, 2021 ). Conversely, residents in rural areas, especially those in remote and sparsely populated regions, are often overlooked by energy policymakers, leaving them at the lower end of the energy ladder (Li and Ma, 2023 ; Tang and Liao, 2014 ). Despite China’s recent progress made through “Targeted poverty alleviation” policy, which has lifted many rural residents out of absolute poverty, the existing rural energy infrastructure, including natural gas networks and power grids, is still inadequate for meeting residents’ daily energy needs (Li et al., 2019 ; Liu and Mauzerall, 2020 ). Consequently, during the supply shortfalls caused by LCCP policy, rural residents are often forced to resort to outdated energy sources like coal, exacerbating their plunge into energy poverty.

This research employs the DID method to delve into the impact of the LCCP policy on residents’ energy poverty conditions and its underlying mechanisms. Our findings offer valuable insights into the delicate tension between low-carbon transitions and energy poverty in developing countries, enriching the understanding of both scholars and policymakers.

This study analyzes 4807 continuously tracked household samples from the CHARLS dataset, offering a snapshot of China’s diverse households. Specifically, 79.08% of these households are married, and 78.75% have up to four members—a reflection of the smaller family units after China’s one-child policy in the 1980s. Approximately 50% of households reported a per capita income above 4000 Yuan, a figure that rose between 2011 and 2018, echoing China’s economic growth. However, the 8-year CHARLS survey reveals that, although there was an overall decrease in energy-poverty households from 2011 to 2018, some households transitioned into energy poverty. While broader studies suggest that China’s recent socio-economic growth has lessened its energy poverty issues (Liang and Asuka, 2022 ; Zhao et al., 2021 ), our analysis uncovers subtle ‘shadows’—households at risk of returning to energy poverty—overlooked in regional data.

This study found that the LCCP policy significantly exacerbated residents’ energy poverty condition, providing the first quantitative demonstration of the LCCP’s negative impacts at the household level. Our findings contrast with previous studies (Dong et al., 2021 ; Dong, Ren et al., 2021 ). Employing province-level data and general regression methods, they deduced that low-carbon transition mitigated energy poverty (Dong, Jiang et al., 2021 ). However, such regional data inadequately captures the intricacies of residents’ energy poverty. Additionally, gauging low-carbon transition via natural gas consumption fails to directly represent the overall implementation of the low-carbon transition policies, thus affecting the reliability of associated conclusions. Complementing earlier research, our study harnesses the LCCP—most representative low-carbon policy in China—to directly probe its consequences on residents’ energy poverty, yielding more precise conclusions. This beckons policymakers to weigh the possible ramifications on residents’ energy welfare in upcoming low-carbon endeavors and advises circumspection before embracing unreviewed low-carbon strategies.

This study experimentally identifies two potential mediating pathways in the relationship between LCCP policy and energy poverty: energy infrastructure and energy expenditure. Regression results reveal that LCCP policy can alleviate energy poverty through the enhancement of energy infrastructure, but exacerbate it by escalating energy expenditure. This can be explained as follows: To comply with government mandates regarding the LCCP policy, energy enterprises must augment their investments in infrastructure, refining energy production, and transportation, which subsequently elevates the cost of energy supply. These surging costs are then passed on to residents, subjecting households sensitive to energy price fluctuations to the energy poverty trap. Compared with previous research, this study delves into the relationship between variables utilizing household-level data, significantly augmenting scholars’ preliminary qualitative analysis on the ramifications of low-carbon transition for resident welfare. Earlier investigations indicated that, with the rise in natural gas and oil prices, households previously affording these energy forms have resorted to coal as an alternative (Kapsalyamova et al., 2021 ; Turdaliev and Janda, 2023 ). This research further elucidates the trend of residents downgrading their daily energy source due to escalating prices, linking this argument to the broader issue of energy poverty.

However, some scholars, adopting a system dynamics perspective, underscore the dynamic feedback interplay between the aforementioned variables, including low-carbon transition, energy expenditure, and energy poverty (Che et al., 2023 ; Venkateswaran et al., 2018 ). Che et al. ( 2023 ) created causal loop diagrams to capture the interplay between energy poverty and various socio-economic factors, accentuating that energy poverty is influenced not merely by an array of factors and multifarious pathways but also exerts its own influence on the broader system. These perspectives highlight a limitation of this study: the path we identified from the LCCP policy through energy expenditure to energy poverty, perhaps reflects the associations among variables rather than the direct causality. Nonetheless, our study offers valuable insights for policymakers seeking to intervene in the adverse impacts of low-carbon transitions.

Heterogeneity analyses reveal that eastern cities, non-resource cities, high-level cities, and larger cities manifest a pronounced risk of residents descending into energy poverty after LCCP policy enforcement. These cities typically have a vibrant energy market where energy supply is predominantly market-driven. Consequently, this study supports the notion that an active energy market can enhance energy poverty risks since energy service for residents is considered a quasi-public good (Xiao et al., 2023 ). This finding aligns with scholars who warn against unchecked marketization of quasi-public goods like energy and highlight the importance of sustained government oversight to ensure residents’ energy welfare (Luo, 2008 ; Zhao et al., 2015 ).

Robustness test

To ensure the reliability of DID regression, we conduct robustness tests, including parallel trends test, anticipation effects, placebo test, PSM-DID approach, outliers excluding, and variable substitution as follows.

Parallel trend test

The prerequisite for DID regression is the satisfaction of the parallel trend assumption (Liu et al., 2022 ; Zhao and Wang, 2021 ). In the context of our study, in the absence of the LCCP policy, the trend of energy poverty among residents in the pilot city should be similar to that of the non-pilot city. We employ Jacobson et al. ( 1993 ) method to perform the parallel trends test, as illustrated in Fig. 3 . The results reveal that the coefficient prior to the policy shock is close to zero and statistically insignificant, indicating no divergence between the pilot and non-pilot cities before policy implementation. The coefficients significantly rise to positive values when the policy is implemented, suggesting that the LCCP policy initially exacerbates energy poverty. However, the coefficients gradually decrease in the second and fifth post-implementation years, signaling a diminishing impact of policy. Overall, this method validates the parallel trends assumption.

Parallel trend test.

Anticipation effects

The absence of anticipation effects is another pivotal prerequisite for the DID method. If anticipation effects exist, they could cause estimation bias, making it difficult to determine whether the effects we observed in the treatment group are due to anticipation actions or actual policy implementation. Therefore, excluding anticipation effects is of utmost importance for our study. We will discuss the anticipation effects through two aspects: policy practice and data analysis.

Firstly, China’s pilot city policy is a conventional approach to policy exploration and policy learning (Wang and Yang, 2021 ). In China’s actual governance context, when selecting low-carbon pilot cities, the central government primarily considers regional representativeness (Fang, 2015 ), and the energy poverty condition does not fall into policymakers’ consideration. In previous quantitative studies, scholars have confirmed that the selection of low-carbon city pilots is not related to the cities’ own low-carbon development status before being selected (Deng and Zhan, 2017 ), indicating that the governments of pilot cities did not take relevant actions that may influence their conditions before LCCP policy implementation. Consequently, residents, at the furthest end of the policy impact scope, are even less likely to be prematurely affected. Therefore, based on the pilot-selection logic of the central government and previous studies, we have substantial grounds to confirm the absence of anticipation effects.

Secondly, we conduct a quantitative comparison, analyzing the average energy poverty condition in pilot cities and non-pilot cities before LCCP policy implementation. As shown in Fig. 4 , there are no significant differences between pilot cities and no-pilot cities when the LCCP policy has not been implemented. Specifically, the average energy poverty condition of all cities included in the study was 0.3494 in 2011. In pilot cities like Guilin and Suzhou, levels were lower than 0.3494, while in Hulunbuir and Ganzhou, levels were higher, presenting a relatively uniform distribution, similar to that in non-pilot cities, indicating no anticipation effects before policy implementation. Therefore, we can confirm the absence of anticipation effects.

Average energy poverty conditions of cities before policy implementation.

Placebo test

To mitigate the effects of random factors on energy poverty and substantiate the policy-driven impact, we conduct a placebo test through random sampling (Wang et al., 2023 ). We randomly select artificial pilot cities and policy implementation times from the samples, thereby randomizing the impact of the LCCP policy. We perform the corresponding DID regression and repeat the random process 500 and 1000 times, with the distribution of coefficients shown in Fig. 5 . The coefficients cluster around zero, markedly less than the previously estimated value of 0.0218. Most regression coefficients have p -values exceeding 0.1, indicating insignificance at the 10% level. Therefore, the LCCP policy’s impact on residents’ energy poverty is not accidental and is not influenced by other random factors.

Placebo test.

PSM-DID approach

PSM-DID approach is used to counter the selection bias of the treatment group and reduce endogeneity issues (Dong et al., 2022 ). Initially, we conduct logit regression using control variables as covariates to calculate propensity matching scores. We then use the scores to match the treatment group and control group via nearest neighbor, radius, and kernel method, respectively. Lastly, we execute three DID regressions, as shown in Table 10 . The estimated values from all matching methods are similar to benchmark regression, confirming that the LCCP policy significantly exacerbates energy poverty, further reinforcing the robustness of our findings.

Outliers excluding and variable substitution

To mitigate the impact of outliers on the regression results, we apply a 1%, 5%, and 10% bilateral tail shrinkage treatment to the dependent variable. As shown in columns (1)–(3) of Table 11 , the coefficients of the LCCP policy are significant at the 1% level after this treatment. Simultaneously, we replace the dependent variable MEPI with cooking fuel type, another indicator of household energy poverty. The coefficient of the LCCP policy, as shown in column (4), remains significant. These findings thus substantiate the robustness of our conclusions.

Conclusion and policy recommendations

To examine the relationship between low-carbon transition and energy poverty in developing countries, this study employs China’s LCCP policy as a quasi-natural experiment. Drawing from 4-year household survey data from CHARLS, we leveraged DID models to examine the impact of the LCCP policy on residents’ energy poverty conditions from a micro perspective.

The main conclusions are as follows: (1) The energy poverty conditions of the 4807 households involved in our study experienced notable shifts from 2011 to 2018, and a significant number of residents saw their energy poverty conditions worsen. (2) The DID regression underscores that the LCCP policy notably exacerbates residents’ energy poverty. This conclusion holds true even after various robustness tests, including parallel trend test, placebo test, PSM-DID, and other methods. (3) According to intermediary mechanism analysis, energy infrastructure and energy expenditure play critical roles in the relationship between LCCP policy and energy poverty, offering valuable insight into the potential pathways of LCCP policy’s impact. (4) City heterogeneity analysis shows that LCCP policy has stronger impacts in eastern, non-resource, larger, and high-level cities. In addition, community heterogeneity analysis underscores a more severe impact of the LCCP policy in communities with inadequate grassroots governance.

Policy implications

Considering the energy poverty issue brought about by the low-carbon transition in developing countries, this study illuminates the following policy implications for future low-carbon practices and energy poverty governance.

According to this study, there exists a delicate balance between advancing low-carbon transitions and ensuring residents’ energy welfare. Developing countries’ governments should adopt measured practices toward low-carbon transition, as well as assess the energy poverty risk of their residents. The empirical data presented in this study calls for a rethinking of current low-carbon strategies, represented by LCCP policy, and setting appropriate targets in light of local conditions. In practice, low-carbon policies that neglect residents’ welfare might not only lead to resource misallocation but also incite public opposition. For instance, when China’s “coal-to-gas” policy resulted in daily heating issues, the government was compelled to suggest a more flexible approach, prioritizing coal, electricity, or gas based on regional suitability. Such frequent changes in policy resulted in substantial wastage of both administrative and financial resources. Therefore, before embarking on future low-carbon-related policies, it is imperative for governments to meticulously evaluate the potential impact on residents’ energy welfare.

This study reveals the potential path of low-carbon transition impacting energy poverty: Regulations related to such transition invariably increase energy sector supply costs. These additional costs are transferred to final consumers, thereby increasing residents’ economic burden, and even worsening their energy poverty conditions. Thus, governments should pay close attention to energy price fluctuations during policy implementation, set up early warning mechanisms for energy price increasing and supply disruption, and take effective actions to protect households from these situations. Additionally, it is imperative for the government to provide financial support to households, lessening the economic pressure associated with the low-carbon transition. Given the unique nature of energy as a quasi-public good, the government should maintain its role as a “gatekeeper” of residents’ welfare, institutionalizing protections for the daily energy needs of residents, leveraging adaptive measures like universal service funds and housing renovation grants, thereby ensuring residents’ energy welfare during the low-carbon transition.

Beyond direct financial support, information support can also safeguard against the risks of energy poverty during the low-carbon transition. According to Kyprianou et al. ( 2019 ), Spain has begun to offer residents advice about energy services, including whether their energy contracts are suitable for their own needs, and how to improve household energy efficiency. Compared to the financial measures, information support measures, aimed at enhancing public awareness and enriching energy-service knowledge, provide a more economical way to alleviate energy poverty. Drawing from Spain’s policy, developing countries like China should also integrate such measures, which may yield long-term policy effects, into their policy frameworks. Additionally, our study reveals that well-governed communities can mitigate the potential impacts of low-carbon transition on residents’ energy poverty, highlighting the importance of community support. Therefore, governments should recognize community officials as key participants in energy poverty governance, acting as a bridge for the government to provide information support to the residents. Moreover, community officials can also collect residents’ feedback about their living conditions, and provide invaluable first-hand data for optimizing existing strategies.

Finally, this study also highlights the heterogeneous impacts of low-carbon transition across varied city characteristics, suggesting that governments should consider the specific socio-economic condition disparities of regions. Previous studies have shown that, compared with state-led governance for energy poverty, regional autonomy governance contained more measures directed at vulnerable consumer groups (Kyprianou et al., 2019 ). In other words, when regions have more administrative power to design their policies, they are more likely to implement diverse strategies that are better suited to local conditions. Therefore, considering the regional heterogeneity in China, it is necessary for the central governments to delegate the formulation of action plans to local governments. For instance, in bustling metropolises reliant on energy imports, local governments should establish mechanisms to monitor and regulate energy supply, ensuring its continuity and stabilizing energy prices for residents during market fluctuations. In less-developed yet resource-rich cities, local governments should focus on developing modern energy infrastructure accessible to vulnerable groups, ensuring they have access to efficient energy sources.

Data availability

Original data for this study are available in the China Health and Retirement Longitudinal Study: http://charls.pku.edu.cn/ .

Abbas K, Li S, Xu D, Baz K, Rakhmetova A (2020) Do socioeconomic factors determine household multidimensional energy poverty? Empirical evidence from South Asia. Energy Policy 146:111754. https://doi.org/10.1016/j.enpol.2020.111754

Article   Google Scholar  

Amigo-Jorquera C, Guerrero-González MJ, Sannazzaro J, Urquiza-Gómez A (2019) Does energy poverty have a female face in Chile? Tapuya: Lat Am Sci Technol Soc 2(1):378–390. https://doi.org/10.1080/25729861.2019.1608038

Amores-Salvadó J, Castro GM, Navas-López JE (2014) Green corporate image: moderating the connection between environmental product innovation and firm performance. J Clean Prod 83:356–365. https://doi.org/10.1016/j.jclepro.2014.07.059

Argenti N, Knight DM (2015) Sun, wind, and the rebirth of extractive economies: renewable energy investment and metanarratives of crisis in Greece. J R Anthropol Inst 21(4):781–802. https://doi.org/10.1111/1467-9655.12287

Awaworyi Churchill S, Smyth R (2022) Protestantism and energy poverty. Energy Econ 111:106087. https://doi.org/10.1016/j.eneco.2022.106087

Azhgaliyeva D, Mishra R, Karymshakov K, Kerimray A, Kapsalyamova Z (2021) What determines coal consumption for residential heating in Kazakhstan and the Kyrgyz Republic? Australas J Environ Manag 28(4):410–432. https://doi.org/10.1080/14486563.2021.1989328

Bai C, Zhan J, Wang H, Liu H, Yang Z, Liu W, Wang C, Chu X, Teng Y (2023) Estimation of household energy poverty and feasibility of clean energy transition: evidence from rural areas in the Eastern Qinghai-Tibet Plateau. J Clean Prod 388:135852. https://doi.org/10.1016/j.jclepro.2023.135852

Belaïd F (2022a) Mapping and understanding the drivers of fuel poverty in emerging economies: the case of Egypt and Jordan. Energy Policy 162:112775. https://doi.org/10.1016/j.enpol.2021.112775

Belaïd F (2022b) Implications of poorly designed climate policy on energy poverty: global reflections on the current surge in energy prices. Energy Res Soc Sci 92:102790. https://doi.org/10.1016/j.erss.2022.102790

Belaïd F, Flambard V (2023) Impacts of income poverty and high housing costs on fuel poverty in Egypt: an empirical modeling approach. Energy Policy 175:113450. https://doi.org/10.1016/j.enpol.2023.113450

Boardman B (1991a) Fuel poverty: from cold homes to affordable warmth. Pinter Pub Limited

Boardman B (1991b) Fuel poverty is different. Policy Stud 12(4):30–41. https://doi.org/10.1080/01442879108423600

Bouzarovski S, Simcock N (2017) Spatializing energy justice. Energy Policy 107:640–648. https://doi.org/10.1016/j.enpol.2017.03.064

Buchanan JM (1965) An economic theory of clubs. Economica 32(125):1. https://doi.org/10.2307/2552442

Cai H, Feng Z, Zhou W, Chen J, Chen Z (2023) Understanding the spatial polarization pattern of technology transfer network in the Guangdong–Hong Kong–Macao Greater Bay area. Growth Change 54(1):4–25. https://doi.org/10.1111/grow.12636

Carfora A, Scandurra G, Thomas A (2022) Forecasting the COVID-19 effects on energy poverty across EU member states. Energy Policy 161:112597. https://doi.org/10.1016/j.enpol.2021.112597

Article   CAS   PubMed   Google Scholar  

Castaño-Rosa R, Solís-Guzmán J, Rubio-Bellido C, Marrero M (2019) Towards a multiple-indicator approach to energy poverty in the European Union: a review. Energy Build 193:36–48. https://doi.org/10.1016/j.enbuild.2019.03.039

Che X, Geng P, Wang D, Fan C, Yuan Y (2023) Integrated decision-making about China’s energy poverty alleviation based on system dynamics. Energy Strategy Rev 45:101011. https://doi.org/10.1016/j.esr.2022.101011

Cui T, Wang C, Nie P (2019) Economic analysis of marsh gas development in countryside of China: a case study of Gongcheng County in Guangxi Province. Energy Rep. 5:462–466. https://doi.org/10.1016/j.egyr.2019.04.012

Deng R, Zhan J (2017) Does the low carbon pilot policy promote the performance of carbon emissions reduction in pilot cities?—Based on the difference-in-difference method. Syst Eng 35(11):68–73

Google Scholar  

Dong F, Li Y, Qin C, Zhang X, Chen Y, Zhao X, Wang C (2022) Information infrastructure and greenhouse gas emission performance in urban China: a difference-in-differences analysis. J Environ Manag 316:115252. https://doi.org/10.1016/j.jenvman.2022.115252

Article   CAS   Google Scholar  

Dong K, Jiang Q, Shahbaz M, Zhao J (2021) Does low-carbon energy transition mitigate energy poverty? The case of natural gas for China. Energy Econ 99:105324. https://doi.org/10.1016/j.eneco.2021.105324

Dong K, Ren X, Zhao J (2021) How does low-carbon energy transition alleviate energy poverty in China? A nonparametric panel causality analysis. Energy Econ 103:105620. https://doi.org/10.1016/j.eneco.2021.105620

Dong X, Pi G, Ma Z, Dong C (2017) The reform of the natural gas industry in the PR of China. Renew Sustain Energy Rev 73:582–593. https://doi.org/10.1016/j.rser.2017.01.157

Du J, Song M, Xie B (2022) Eliminating energy poverty in Chinese households: a cognitive capability framework. Renew Energy 192:373–384. https://doi.org/10.1016/j.renene.2022.04.106

Fan D, Wang W, Liang P (2017) Analysis of the performance of carbon emissions trading rights in China: an evaluation based on the difference-in-dfference model. China Environ Sci 37(6):2383–2392

Fang Q (2015) Kang Yanbing, Expert from NDRC: China’s low-carbon city pilot and urban low-carbon transformation. http://cn.chinagate.cn/news/2015-10/09/content_36768456.htm

Feng Z, Cai H, Chen Z, Zhou W (2022) Influence of an interurban innovation network on the innovation capacity of China: a multiplex network perspective. Technol Forecast Soc Change 180:121651. https://doi.org/10.1016/j.techfore.2022.121651

Feng Z, Chen W (2018) Environmental regulation, green innovation, and industrial green development: an empirical analysis based on the spatial Durbin model. Sustainability 10(1):223. https://doi.org/10.3390/su10010223

Article   MathSciNet   Google Scholar  

Frilingou N, Xexakis G, Koasidis K, Nikas A, Campagnolo L, Delpiazzo E, Chiodi A, Gargiulo M, McWilliams B, Koutsellis T, Doukas H (2023) Navigating through an energy crisis: challenges and progress towards electricity decarbonisation, reliability, and affordability in Italy. Energy Res Soc Sci 96:102934. https://doi.org/10.1016/j.erss.2022.102934

Frondel M, Sommer S, Vance C (2015) The burden of Germany’s energy transition: an empirical analysis of distributional effects. Econ Anal Policy 45:89–99. https://doi.org/10.1016/j.eap.2015.01.004

Fry JM, Farrell L, Temple JB (2022) Energy poverty and retirement income sources in Australia. Energy Econ 106:105793. https://doi.org/10.1016/j.eneco.2021.105793

Geels FW, Sovacool BK, Schwanen T, Sorrell S (2017) The socio-technical dynamics of low-carbon transitions. Joule 1(3):463–479. https://doi.org/10.1016/j.joule.2017.09.018

González-Eguino M (2015) Energy poverty: an overview. Renew Sustain Energy Rev 47:377–385. https://doi.org/10.1016/j.rser.2015.03.013

Gorayeb A, Brannstrom C, De Andrade Meireles AJ, De Sousa Mendes J (2018) Wind power gone bad: critiquing wind power planning processes in northeastern Brazil. Energy Res Soc Sci 40:82–88. https://doi.org/10.1016/j.erss.2017.11.027

Hartwig M, Emori S, Asayama S (2023) Normalized injustices in the national energy discourse: a critical analysis of the energy policy framework in Japan through the three tenets of energy justice. Energy Policy 174:113431. https://doi.org/10.1016/j.enpol.2023.113431

Hasanujzaman M, Omar MA (2022) Household and non-household factors influencing multidimensional energy poverty in Bangladesh: demographics, urbanization and regional differentiation via a multilevel modeling approach. Energy Res Soc Sci 92:102803. https://doi.org/10.1016/j.erss.2022.102803

He J, Li Z, Zhang X, Wang H, Dong W, Du E, Chang S, Ou X, Guo S, Tian Z, Gu A, Teng F, Hu B, Yang X, Chen S, Yao M, Yuan Z, Zhou L, Zhao X, Zhang D (2022) Towards carbon neutrality: a study on China’s long-term low-carbon transition pathways and strategies. Environ Sci Ecotechnol 9:100134. https://doi.org/10.1016/j.ese.2021.100134

He L, Yang X, Yin F, Zhong Z, Jin L (2016) Regulatory effect and comparison of comprehensive energy price indices on Chinese provincial carbon intensity: an empirical analysis based on panel data from 30 provinces. Resour Environ Yangtze Basin 25(6):877–888

Heffron RJ (2022) Applying energy justice into the energy transition. Renew Sustain Energy Rev 156:111936. https://doi.org/10.1016/j.rser.2021.111936

Heltberg R (2005) Factors determining household fuel choice in Guatemala. Environ Dev Econ 10(3):337–361. https://doi.org/10.1017/S1355770X04001858

Hills, J (2012) Getting the measure of fuel poverty: final report of the fuel poverty review. Centre for Analysis of Social Exclusion

Hong X, Wu S, Zhang X (2022) Clean energy powers energy poverty alleviation: evidence from Chinese micro-survey data. Technol Forecast Soc Change 182:121737. https://doi.org/10.1016/j.techfore.2022.121737

Jacobson LS, LaLonde RJ, Sullivan DG (1993) Earnings losses of displaced workers. Am Econ Rev 83(4):685–709. JSTOR

Jenkins KEH, Sovacool BK, Mouter N, Hacking N, Burns M-K, McCauley D (2021) The methodologies, geographies, and technologies of energy justice: a systematic and comprehensive review. Environ Res Lett 16(4):043009. https://doi.org/10.1088/1748-9326/abd78c

Article   CAS   ADS   Google Scholar  

Jenkins K, McCauley D, Heffron R, Stephan H, Rehner R (2016) Energy justice: a conceptual review. Energy Res Soc Sci 11:174–182. https://doi.org/10.1016/j.erss.2015.10.004

Ju K, Su B, Zhou D, Wu J (2017) Does energy-price regulation benefit China’s economy and environment? Evidence from energy-price distortions. Energy Policy 105:108–119. https://doi.org/10.1016/j.enpol.2017.02.031

Kapsalyamova Z, Mishra R, Kerimray A, Karymshakov K, Azhgaliyeva D (2021) Why energy access is not enough for choosing clean cooking fuels? Evidence from the multinomial logit model. J Environ Manag 290:112539. https://doi.org/10.1016/j.jenvman.2021.112539

Khanna N, Fridley D, Hong L (2014) China’s pilot low-carbon city initiative: a comparative assessment of national goals and local plans. Sustain Cities Soc 12:110–121. https://doi.org/10.1016/j.scs.2014.03.005

Kyprianou I, Serghides DK, Varo A, Gouveia JP, Kopeva D, Murauskaite L (2019) Energy poverty policies and measures in 5 EU countries: a comparative study. Energy Build 196:46–60. https://doi.org/10.1016/j.enbuild.2019.05.003

Li H, Wang J, Yang X, Wang Y, Wu T (2018) A holistic overview of the progress of China’s low-carbon city pilots. Sustain Cities Soc 42:289–300. https://doi.org/10.1016/j.scs.2018.07.019

Li J, Chen C, Liu H (2019) Transition from non-commercial to commercial energy in rural China: insights from the accessibility and affordability. Energy Policy 127:392–403. https://doi.org/10.1016/j.enpol.2018.12.022

Li J, Ma W (2023) Sharing energy poverty: the nexus between social interaction-oriented gift expenditure and energy poverty in rural China. Energy Res Soc Sci 101:103131. https://doi.org/10.1016/j.erss.2023.103131

Li L, Song X, Li J, Li K, Jiao J (2023) The impacts of temperature on residential electricity consumption in Anhui, China: does the electricity price matter? Clim Change 176(3):26. https://doi.org/10.1007/s10584-023-03500-9

Article   ADS   Google Scholar  

Li Y, Ning X, Wang Z, Cheng J, Li F, Hao Y (2022) Would energy poverty affect the wellbeing of senior citizens? Evidence from China. Ecol Econ 200:107515. https://doi.org/10.1016/j.ecolecon.2022.107515

Liang Q, Asuka J (2022) A multidimensional energy poverty measurement in China—based on the entropy method. Energy Sustain Dev 71:554–567. https://doi.org/10.1016/j.esd.2022.11.005

Lippert I, Sareen S (2023) Alleviation of energy poverty through transitions to low-carbon energy infrastructure. Energy Res Soc Sci 100:103087. https://doi.org/10.1016/j.erss.2023.103087

Liu H, Mauzerall DL (2020) Costs of clean heating in China: evidence from rural households in the Beijing–Tianjin–Hebei region. Energy Econ 90:104844. https://doi.org/10.1016/j.eneco.2020.104844

Liu J, Feng H, Wang K (2022) The low-carbon city pilot policy and urban land use efficiency: a policy assessment from China. Land 11(5):604. https://doi.org/10.3390/land11050604

Liu J, Jain V, Sharma P, Ali SA, Shabbir MS, Ramos-Meza CS (2022) The role of Sustainable Development Goals to eradicate the multidimensional energy poverty and improve social Wellbeing’s. Energy Strategy Rev 42:100885. https://doi.org/10.1016/j.esr.2022.100885

Liu Z, Liang S, Geng Y, Xue B, Xi F, Pan Y, Zhang T, Fujita T (2012) Features, trajectories and driving forces for energy-related GHG emissions from Chinese mega cites: the case of Beijing, Tianjin, Shanghai and Chongqing. Energy 37(1):245–254. https://doi.org/10.1016/j.energy.2011.11.040

Lu H, Zhao P, Hu H, Zeng L, Wu KS, Lv D (2022) Transport infrastructure and urban-rural income disparity: a municipal-level analysis in China. J Transp Geogr 99:103292. https://doi.org/10.1016/j.jtrangeo.2022.103292

Luo G, Liu S, Yan X, Guo Y (2021) Institutional constraints to China’s low carbon transition: a case study of China’s coal-to-gas program. Struct Change Econ Dyn 57:121–135. https://doi.org/10.1016/j.strueco.2021.01.005

Luo M (2008) New welfare. Adm Law Rev 4:123–131

Ma R, Deng L, Ji Q, Zhai P (2022) Environmental regulations, clean energy access, and household energy poverty: evidence from China. Technol Forecast Soc Change 182:121862. https://doi.org/10.1016/j.techfore.2022.121862

Magacho G, Espagne E, Godin A, Mantes A, Yilmaz D (2023) Macroeconomic exposure of developing economies to low-carbon transition. World Dev 167:106231. https://doi.org/10.1016/j.worlddev.2023.106231

Martiskainen M, Heiskanen E, Speciale G (2018) Community energy initiatives to alleviate fuel poverty: the material politics of Energy Cafés. Local Environ 23(1):20–35. https://doi.org/10.1080/13549839.2017.1382459

McCauley D, Heffron R, Stephan H, Jenkins K (2013) Advancing energy justice: the triumvirate of tenets. Int Energy Law Rev 32(3):107–110

McHarg A (2020) Energy justice: understanding the ‘ethical turn’ in energy law and policy. In: McHarg A (ed) Energy justice and energy law. Oxford University Press, pp. 15–30

Mišík M (2022) The EU needs to improve its external energy security. Energy Policy 165:112930. https://doi.org/10.1016/j.enpol.2022.112930

Mohseni S, Brent AC, Kelly S, Browne WN (2022) Demand response-integrated investment and operational planning of renewable and sustainable energy systems considering forecast uncertainties: a systematic review. Renew Sustain Energy Rev 158:112095. https://doi.org/10.1016/j.rser.2022.112095

Moore R (2012) Definitions of fuel poverty: implications for policy. Energy Policy 49:19–26. https://doi.org/10.1016/j.enpol.2012.01.057

Mulvaney D (2014) Are green jobs just jobs? Cadmium narratives in the life cycle of photovoltaics. Geoforum 54:178–186. https://doi.org/10.1016/j.geoforum.2014.01.014

NDRC (2014) Promoting the pilot and demonstration projects for low-carbon development and shift the pattern of economic development. https://www.ndrc.gov.cn/xwdt/xwfb/201402/t20140214_956300.html

Nguyen TT, Nguyen T-T, Hoang V-N, Wilson C, Managi S (2019) Energy transition, poverty and inequality in Vietnam. Energy Policy 132:536–548. https://doi.org/10.1016/j.enpol.2019.06.001

Numan U, Ma B, Sadiq M, Bedru HD, Jiang C (2023) The role of green finance in mitigating environmental degradation: empirical evidence and policy implications from complex economies. J Clean Prod 400:136693. https://doi.org/10.1016/j.jclepro.2023.136693

Nussbaumer P, Bazilian M, Modi V (2012) Measuring energy poverty: focusing on what matters. Renew Sustain Energy Rev 16(1):231–243. https://doi.org/10.1016/j.rser.2011.07.150

Olabi AG, Abdelkareem MA (2022) Renewable energy and climate change. Renew Sustain Energy Rev 158:112111. https://doi.org/10.1016/j.rser.2022.112111

Ouyang J, Hokao K (2009) Energy-saving potential by improving occupants’ behavior in urban residential sector in Hangzhou City, China. Energy Build 41(7):711–720. https://doi.org/10.1016/j.enbuild.2009.02.003

Peidong Z, Yanli Y, Jin S, Yonghong Z, Lisheng W, Xinrong L (2009) Opportunities and challenges for renewable energy policy in China. Renew Sustain Energy Rev 13(2):439–449. https://doi.org/10.1016/j.rser.2007.11.005

Peking University (2023) China Health and Retirement Longitudinal Study. http://charls.pku.edu.cn/

Qiu S, Lei T, Yao Y, Wu J, Bi S (2021) Impact of high-quality-development strategy on energy demand of East China. Energy Strategy Rev 38:100699. https://doi.org/10.1016/j.esr.2021.100699

Rahut DB, Ali A, Mottaleb KA, Aryal JP (2019) Wealth, education and cooking-fuel choices among rural households in Pakistan. Energy Strategy Rev 24:236–243. https://doi.org/10.1016/j.esr.2019.03.005

Ravigné E, Ghersi F, Nadaud F (2022) Is a fair energy transition possible? Evidence from the French low-carbon strategy. Ecol Econ 196:107397. https://doi.org/10.1016/j.ecolecon.2022.107397

Ren S, Li X, Yuan B, Li D, Chen X (2018) The effects of three types of environmental regulation on eco-efficiency: a cross-region analysis in China. J Clean Prod 173:245–255. https://doi.org/10.1016/j.jclepro.2016.08.113

Ren Y-S, Jiang Y, Narayan S, Ma C-Q, Yang X-G (2022) Marketisation and rural energy poverty: evidence from provincial panel data in China. Energy Econ 111:106073. https://doi.org/10.1016/j.eneco.2022.106073

Robinson C (2019) Energy poverty and gender in England: a spatial perspective. Geoforum 104:222–233. https://doi.org/10.1016/j.geoforum.2019.05.001

Sarkis J, Cordeiro JJ (2001) An empirical evaluation of environmental efficiencies and firm performance: pollution prevention versus end-of-pipe practice. Eur J Oper Res 135(1):102–113. https://doi.org/10.1016/S0377-2217(00)00306-4

Savas E (1999) Privatization and public–private partnerships. Chatham House

Semieniuk G, Campiglio E, Mercure J, Volz U, Edwards NR (2021) Low‐carbon transition risks for finance. WIREs Clim Change 12(1):e678. https://doi.org/10.1002/wcc.678

Setyowati AB (2021) Mitigating inequality with emissions? Exploring energy justice and financing transitions to low carbon energy in Indonesia. Energy Res Soc Sci 71:101817. https://doi.org/10.1016/j.erss.2020.101817

Shahzad U, Gupta M, Sharma GD, Rao A, Chopra R (2022) Resolving energy poverty for social change: research directions and agenda. Technol Forecast Soc Change 181:121777. https://doi.org/10.1016/j.techfore.2022.121777

Shen K, Shi Q (2018) The impacts of population structure and household size on residential energy consumption: evidence based on provincial-level panel data. Popul Res 42(6):100–110

ADS   Google Scholar  

Siksnelyte-Butkiene I (2022) Combating energy poverty in the face of the COVID-19 pandemic and the global economic uncertainty. Energies 15(10):3649. https://doi.org/10.3390/en15103649

Song Q, Qin M, Wang R, Qi Y (2020) How does the nested structure affect policy innovation?: empirical research on China’s low carbon pilot cities. Energy Policy 144:111695. https://doi.org/10.1016/j.enpol.2020.111695

Sovacool BK (2021) Who are the victims of low-carbon transitions? Towards a political ecology of climate change mitigation. Energy Res Soc Sci 73:101916. https://doi.org/10.1016/j.erss.2021.101916

Sovacool BK, Bell SE, Daggett C, Labuski C, Lennon M, Naylor L, Klinger J, Leonard K, Firestone J (2023) Pluralizing energy justice: incorporating feminist, anti-racist, Indigenous, and postcolonial perspectives. Energy Res Soc Sci 97:102996. https://doi.org/10.1016/j.erss.2023.102996

Sovacool BK, Dworkin MH (2015) Energy justice: conceptual insights and practical applications. Appl Energy 142:435–444. https://doi.org/10.1016/j.apenergy.2015.01.002

Sovacool BK, Martiskainen M, Hook A, Baker L (2019) Decarbonization and its discontents: a critical energy justice perspective on four low-carbon transitions. Clim Change 155(4):581–619. https://doi.org/10.1007/s10584-019-02521-7

Sovacool BK, Newell P, Carley S, Fanzo J (2022) Equity, technological innovation and sustainable behaviour in a low-carbon future. Nat Hum Behav 6(3):326–337. https://doi.org/10.1038/s41562-021-01257-8

Article   PubMed   Google Scholar  

State Council (2000) Implementing several policies and measures for the large-scale development of the Western Region. https://www.gov.cn/gongbao/content/2001/content_60854.htm

State Council (2013) National Resource-based City Sustainable Development Plan (2013–2020) https://www.gov.cn/zwgk/2013-12/03/content_2540070.htm

Sy SA, Mokaddem L (2022) Energy poverty in developing countries: a review of the concept and its measurements. Energy Res Soc Sci 89:102562. https://doi.org/10.1016/j.erss.2022.102562

Tang X, Liao H (2014) Energy poverty and solid fuels use in rural China: analysis based on national population census. Energy Sustain Dev 23:122–129. https://doi.org/10.1016/j.esd.2014.08.006

Tian J, Yu L, Xue R, Zhuang S, Shan Y (2022) Global low-carbon energy transition in the post-COVID-19 era. Appl Energy 307:118205. https://doi.org/10.1016/j.apenergy.2021.118205

Turdaliev S, Janda K (2023) Increasing block tariff electricity pricing and propensity to purchase dirty fuels: empirical evidence from a natural experiment. Eastern Eur Econ 1–21. https://doi.org/10.1080/00128775.2023.2165948

United Nations (2012) International year of sustainable energy for all. https://www.un.org/en/events/sustainableenergyforall/background.shtml

Upham DP, Sovacool PB, Ghosh DB (2022) Just transitions for industrial decarbonisation: a framework for innovation, participation, and justice. Renew Sustain Energy Rev 167:112699. https://doi.org/10.1016/j.rser.2022.112699

Ürge-Vorsatz D, Tirado Herrero S (2012) Building synergies between climate change mitigation and energy poverty alleviation. Energy Policy 49:83–90. https://doi.org/10.1016/j.enpol.2011.11.093

Venkateswaran J, Solanki CS, Werner K, Yadama GN (2018) Addressing energy poverty in India: a systems perspective on the role of localization, affordability, and saturation in implementing solar technologies. Energy Res Soc Sci 40:205–210. https://doi.org/10.1016/j.erss.2018.02.011

Wang J, Song Z, Zhang Y, Hussain RY (2023) Can low-carbon pilot policies improve the efficiency of urban carbon emissions?—A quasi-natural experiment based on 282 prefecture-level cities across China. PLoS ONE 18(2):e0282109. https://doi.org/10.1371/journal.pone.0282109

Article   CAS   PubMed   PubMed Central   Google Scholar  

Wang Q, Chen X (2012) China’s electricity market-oriented reform: from an absolute to a relative monopoly. Energy Policy 51:143–148. https://doi.org/10.1016/j.enpol.2012.08.039

Wang R, Ren L (2020) Logic and its unexpected consequences of local environmental policy excessive implementation—case study from the “Coal to Gas” Policy in 2017. J Public Manag 18(1):13

CAS   Google Scholar  

Wang S, Yang D (2021) Policy experimentation in China: the political economy of policy learning (No. w29402; p. w29402). National Bureau of Economic Research

Wang X, Lo K (2022) Civil society, environmental litigation, and Confucian energy justice: a case study of an environmental NGO in China. Energy Res Soc Sci 93:102831. https://doi.org/10.1016/j.erss.2022.102831

Wang Y, Hao F (2018) Does Internet penetration encourage sustainable consumption? A cross-national analysis. Sustain Prod Consum 16:237–248. https://doi.org/10.1016/j.spc.2018.08.011

Wang Y, Song Q, He J, Qi Y (2015) Developing low-carbon cities through pilots. Clim Policy 15(suppl 1):S81–S103. https://doi.org/10.1080/14693062.2015.1050347

Wu F (2022) Intergenerational support and life satisfaction of older parents in China: a rural–urban divide. Soc Indic Res 160(2–3):1071–1098. https://doi.org/10.1007/s11205-021-02672-0

Xiao Y, Feng Z, Wu H, Wang S (2023) Every rose has its thorn: do environmental regulations exacerbate regional energy poverty? J Clean Prod 419:138285. https://doi.org/10.1016/j.jclepro.2023.138285

Xiao Y, Wu H, Wang G, Wang S (2021) The relationship between energy poverty and individual development: exploring the serial mediating effects of learning behavior and health condition. Int J Environ Res Public Health 18(16). Scopus. https://doi.org/10.3390/ijerph18168888

Xie L, Hu X, Zhang X, Zhang X-B (2022) Who suffers from energy poverty in household energy transition? Evidence from clean heating program in rural China. Energy Econ 106:105795. https://doi.org/10.1016/j.eneco.2021.105795

Yang F, Zhang S, Sun C (2020) Energy infrastructure investment and regional inequality: evidence from China’s power grid. Sci Total Environ 749:142384. https://doi.org/10.1016/j.scitotenv.2020.142384

Article   CAS   PubMed   ADS   Google Scholar  

Yang S, Jahanger A, Hossain MR (2023a) How effective has the low-carbon city pilot policy been as an environmental intervention in curbing pollution? Evidence from Chinese industrial enterprises. Energy Econ 118:106523. https://doi.org/10.1016/j.eneco.2023.106523

Yang S, Jahanger A, Hossain MR (2023b) Does China’s low-carbon city pilot intervention limit electricity consumption? An analysis of industrial energy efficiency using time-varying DID model. Energy Econ 121:106636. https://doi.org/10.1016/j.eneco.2023.106636

Yao Y, Jiang L (2021) Urbanization forces driving rural–urban income disparity: Evidence from metropolitan areas in China. J Clean Prod 312:127748. https://doi.org/10.1016/j.jclepro.2021.127748

Yi Z, Wang Z (2003) Dynamics of family and elderly living arrangements in China: new lessons learned from the 2000 census. China Rev 3(2):95–119. JSTOR

Yu B, Fang D, Kleit AN, Xiao K (2022) Exploring the driving mechanism and the evolution of the low‐carbon economy transition: lessons from OECD developed countries. World Econ 45(9):2766–2795. https://doi.org/10.1111/twec.13263

Zhang D, Li J, Han P (2019) A multidimensional measure of energy poverty in China and its impacts on health: an empirical study based on the China family panel studies. Energy Policy 131:72–81. https://doi.org/10.1016/j.enpol.2019.04.037

Zhang H, Feng C, Zhou X (2022) Going carbon-neutral in China: does the low-carbon city pilot policy improve carbon emission efficiency? Sustain Prod Consum 33:312–329. https://doi.org/10.1016/j.spc.2022.07.002

Zhang Q, Appau S, Kodom PL (2021) Energy poverty, children’s wellbeing and the mediating role of academic performance: evidence from China. Energy Econ 97:105206. https://doi.org/10.1016/j.eneco.2021.105206

Zhang Z (2018) Environmental regulation, price transfer, and pollution burden of Chinese manufacturers: evidence based on pollution fees of key monitoring firms. Ind Econ Res 4:65–75. https://doi.org/10.13269/j.cnki.ier.2018.04.006

Zhang Z, Hu G, Mu X, Kong L (2022) From low carbon to carbon neutrality: a bibliometric analysis of the status, evolution and development trend. J Environ Manag 322:116087. https://doi.org/10.1016/j.jenvman.2022.116087

Zhang Z, Shu H, Yi H, Wang X (2021) Household multidimensional energy poverty and its impacts on physical and mental health. Energy Policy 156:112381. https://doi.org/10.1016/j.enpol.2021.112381

Zhao C, Wang B (2021) Does China’s low-carbon pilot policy promote foreign direct investment? An empirical study based on city-level panel data of China. Sustainability 13(19):10848. https://doi.org/10.3390/su131910848

Zhao H, Guo S, Li H (2015) Economic impact assessment of wind power integration: a quasi-public goods property perspective. Energies 8(8):8749–8774. https://doi.org/10.3390/en8088749

Zhao J, Dong K, Dong X, Shahbaz M (2022) How renewable energy alleviate energy poverty? A global analysis. Renew Energy 186:299–311. https://doi.org/10.1016/j.renene.2022.01.005

Zhao J, Jiang Q, Dong X, Dong K (2021) Assessing energy poverty and its effect on CO 2 emissions: the case of China. Energy Econ 97:105191. https://doi.org/10.1016/j.eneco.2021.105191

Zhao J, Wang J, Dong K (2022) The role of green finance in eradicating energy poverty: ways to realize green economic recovery in the post-COVID-19 era. Econ Change Restruct. https://doi.org/10.1007/s10644-022-09411-6

Zheng D, Shi M (2017) Multiple environmental policies and pollution haven hypothesis: evidence from China’s polluting industries. J Clean Prod 141:295–304. https://doi.org/10.1016/j.jclepro.2016.09.091

Zhou L, Chen Y (2005) The policy effect of tax-and-fees reforms in rural China: a difference-in-differences estimation. Econ Res J 8:44–53

Zhu C, Lee C-C (2022) The effects of low-carbon pilot policy on technological innovation: evidence from prefecture-level data in China. Technol Forecast Soc Change 183:121955. https://doi.org/10.1016/j.techfore.2022.121955

Zou B, Luo B (2019) Rural household energy consumption characteristics and determinants in China. Energy 182:814–823. https://doi.org/10.1016/j.energy.2019.06.048

Download references

Acknowledgements

This work was supported by the Ministry of Education of Humanities and Social Science project (No. 21YJC630022), and the China Postdoctoral Science Foundation funded project (No. 2023M743362).

Author information

Authors and affiliations.

School of Economics and Management, Dongguan University of Technology, 523808, Dongguan, China

Yiming Xiao & Zhijun Feng

School of Management, University of Science and Technology of China, 230026, Hefei, China

Yiming Xiao

School of Data Science, City University of Hong Kong, 999077, Hong Kong, China

School of Public Policy & Management, Tsinghua University, 100084, Beijing, China

Shangrui Wang

You can also search for this author in PubMed   Google Scholar

Contributions

YX: conceptualization, methodology, and original draft writing. ZF: formal analysis, review, and supervision. XL: data curation and review. SW: review.

Corresponding author

Correspondence to Zhijun Feng .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

This paper did not include any studies with human or animal participants conducted by all the authors.

Informed consent

Additional information.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Xiao, Y., Feng, Z., Li, X. et al. Low-carbon transition and energy poverty: quasi-natural experiment evidence from China’s low-carbon city pilot policy. Humanit Soc Sci Commun 11 , 84 (2024). https://doi.org/10.1057/s41599-023-02573-2

Download citation

Received : 25 June 2023

Accepted : 18 December 2023

Published : 09 January 2024

DOI : https://doi.org/10.1057/s41599-023-02573-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Investigating the role of subsistence renewables in alleviating power poverty within nigeria’s energy-mix strategy.

• David Oluseun Olayungbo
• Ayodele Adekunle Faiyetole
• Adenike Anike Olayungbo

Sustainable Energy Research (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Asymmetric effectiveness of price limits: evidence from a quasi-natural experiment

• Original Research
• Published: 22 August 2024

Cite this article

• Siyuan Tang   ORCID: orcid.org/0000-0002-3453-1155 1  

This study examines the effectiveness of price limits using a quasi-natural experiment in China, where ChiNext stocks’ daily price limits changed from 10 to 20%. The empirical results show asymmetric performance of the upper and lower limits. Widening the upper limit range can alleviate delayed price discovery and trading interference, whereas an equivalent lower limit can exacerbate these adverse effects. A wider upper limit range increases volatility spillovers but a wider lower-limit range does not. Such asymmetry is more pronounced when there is higher investor sentiment, a greater possibility of manipulation, and eligibility for short selling. Additional analyses suggest that widening the limit range provides investors with a greater opportunity to buy (sell) more on the upper (lower) limit hitting day and sell (buy) more on the following day, resulting in a greater order imbalance primarily caused by institutional investors. The results indicate that the daily price limit leads to market inefficiencies when the price rises but reduces stock price crashes and acts as a price stabilizer.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Do price limits help control stock price volatility.

Performance of price limits: Evidence from cross-listed stocks in China

Short-term price overreactions: identification, testing, exploitation.

Deb et al. ( 2010 ) and Kim and Park ( 2010 ) analyze major global markets and provide a rationale for implementing price limit systems in high regulatory risk markets with more irrational investors. Chen et al. ( 2005 ) document the asymmetric effectiveness of price limits in bullish and bearish periods. Chen et al. ( 2024 ) show that the effect of circuit breakers on overall welfare varies with investor motives.

Many capital markets worldwide have encountered the costs associated with price limits. Studies have identified these costs in various stock exchanges. In Europe, examples include the Spanish (Kim et al. 2008 ) and Athens (Phylaktis et al. 1999 ) stock exchanges. In Asia, significant effects have been observed in exchanges like Tokyo (Kim and Rhee 1997 ), Taiwan (Huang et al. 2001 ; Kim and Limpaphayom 2000 ), South Korea (Berkman and Lee 2002 ), and Shenzhen and Shanghai in China (Chen et al. 2019 ; Hou et al. 2020 ), among others.

Amidst a market crash, the selling pressure predominantly targets stocks with higher market capitalization, as highlighted by Lauterbach and Ben‐Zion ( 1993 ). Stocks that are easily manipulated are more inclined to be sold on the day when the lower limit is hit, aiming to mitigate further losses.

There is less discussion about the effectiveness of lower limit hits than that on upper limit hits generally. This is partly because lower limit hits exhibit similar characteristics to upper limit hits (Kim and Rhee 1997 ) or due to relatively insufficient sample data for lower limit hits (Kim et al. 2013 ).

Mostly, no price limit is imposed on new shares on their first listing day. Since December 13, 2013, a 44% price limit has been implemented for new shares on the first trading day, which is not applicable to new shares on the ChiNext market after August 24, 2020, or stocks on STAR. For more discussion on IPO pricing, see Qian et al. ( 2022 ).

“ST” stands for special treatment in the context of the Shanghai and Shenzhen Stock Exchanges’ Listing Rules. This status is applied to companies facing unusual financial situations or other exceptional circumstances that might lead to delisting, or when the company’s future becomes uncertain, posing a risk to investor rights. Typically, a company receives the ST designation if it reports negative net income for two consecutive years, signaling heightened risk to investors.

In the initial five trading days following their listing, newly listed shares on the ChiNext market are not subject to any price limits.

The empirical results using the full sample are represented in Panel A in Internet Appendix B . The results using the full sample have higher significance than those using the PSM sample. The results using the matching sample with more covariates are represented in Panel B in Internet Appendix B .

For stock price experiencing “no change” by definition, I define Continuation i,t as zero. Since the occurrence of no-change price behavior is infrequent, the results remain unaffected regardless of whether “no change” is categorized as “continuation” or “reversal.”.

Since the regression sample is not a panel dataset, firm fixed effects are not controlled in the main regression. See Panel C in Internet Appendix B for the results of controlling firm fixed effects.

Because the primary tests for the three hypotheses contain different sample observations, the number of observations reported in Panel B represents the total number of all observations for control variables during the sample period. 12,545 is the number of quarterly observations, and 712,710 is the number of daily observations.

The number of observations for each subgroup is inadequate when the price movement margin of 1% is used, especially for the ChiNext market after the event date. Internet Appendix C reports the results using a 1% price movement margin to classify stocks. The significance of the regressions weakens to some extent, which may be because of the scarcity of the observations.

The results of the univariate tests conducted over a 21-day event window, spanning from Day -10 to Day + 10, are detailed in the Online Appendix. Specifically, Internet Table D1 presents these results with TradeInterfer as the dependent variable, whereas Internet Table D2 focuses on Volatility as the dependent variable.

The results of Eq. ( 2 ) over the 21-day period are presented in Appendix 2. For the upward price movement, Columns (1) and (3) present the coefficients of Post  ×  GEM  ×  Dum20 ( β 11 ) and Post  ×  GEM  ×  Dum16 ( β 7 ), and Columns (2) and (4) present their t-values. From Day 1 to Day 4, β 11 in Column (1) is significantly positive on Day 1 and Day 2, close to the significance levels on Days 3 and 4, whereas β 7 in Column (3) is insignificant during the post-limit period, indicating a wider upper price limit increases the spillover of stock volatility up to four days after the price limit hit. However, for the lower price limit, β 11 in Column (5) is not significant during the post-limit period, indicating that a wider lower price limit does not affect stock spillovers. On Day 1, β 11 in Column (5) exhibits a negative value, with a t-value of -1.62. This indicates even a decrease in volatility spillover after widening the lower price limit range.

Through an analysis of the market index and investor sentiment, I distinguish between bullish and bearish market periods. The patterns depicted in Fig.  3 in Appendix 3 indicate that both the market index and investor sentiment experience higher (lower) values after (before) the event date, suggesting a bullish (bearish) market period following (preceding) the event date.

Naturally, this effect exists on the ChiNext board like other confounding events as well. However, applying a difference-in-differences approach guarantees the results are not affected.

The findings of delayed price discovery for the other two subgroups ( S 0.08 /S 0.18 and S 0.06 /S 0.16 ) are presented in Internet Appendix A . There is no price delay observed for stocks in the two subgroups during upward price movements, as shown in Internet Table A1 . However, the ChiNext and non-ChiNext stocks both exhibit price delays during downward price movements after the event date, as presented in Internet Table A2 . Moreover, stocks with wider price movement range show a significant price delay, which is consistent with the results of lower limit hits, indicating that a wider downward limit range exacerbates the price delay.

Post + 0 ( Post + 1 ) equals one when a stock-day observation falls in the time range from August 24, 2020 to August 23, 2021 (August 24, 2021–August 23, 2022) and zero otherwise.

The samples here are segmented into four-year intervals, with each interval demarcated by August 24 as the annual dividing line.

Amihud is not incorporated as a control variable in this regression.

I replaced the dummy variables Dum20 and Dum16 in Eq. ( 2 ) with the continuous variables Return20 and Return16 . When Dum20 and Dum16 are set to one, Return20 and Return16 take the corresponding stock returns; when Dum20 and Dum16 are set to zero, Return20 and Return16 also take the value of zero. This approach helps in analyzing the non-parametric possibilities of return changes. As only Eq. ( 2 ) allows for this kind of research design, I only examine the dependent variables TradeInterfer or Volatility when applying GAMs. The regression results are presented in Internet Appendix E . The empirical findings demonstrate that considering non-parametric possibilities does not alter the previously drawn conclusions.

Ackert LF, Hunter WC (1994) Rational price limits in futures markets: tests of a simple optimizing model. Rev Financ Econ 4(1):93–108. https://doi.org/10.1016/1058-3300(94)90008-6

Article   Google Scholar  

Ackert LF, Huang Y, Jiang L (2015) Investor sentiment and price limit rules. J Behav Exp Fin 5:15–26. https://doi.org/10.1016/j.jbef.2015.01.001

Altig D, Baker S, Barrero JM et al (2020) Economic uncertainty before and during the COVID-19 pandemic. J Public Econ 191:104274. https://doi.org/10.1016/j.jpubeco.2020.104274

Amihud Y (2002) Illiquidity and stock returns: cross-section and time-series effects. J Financ Mark 5(1):31–56. https://doi.org/10.1016/S1386-4181(01)00024-6

Andersen TG, Bollerslev T, Diebold FX et al (2001) The distribution of realized stock return volatility. J Financ Econ 61(1):43–76. https://doi.org/10.1016/S0304-405X(01)00055-1

Baker M, Wurgler J (2007) Investor sentiment in the stock market. J Econ Perspect 21(2):129–151. https://doi.org/10.1257/jep.21.2.129

Berkman H, Lee JBT (2002) The effectiveness of price limits in an emerging market: evidence from the Korean stock exchange. Pac-Basin Financ J 10(5):517–530. https://doi.org/10.1257/jep.21.2.129

Brennan MJ (1986) A theory of price limits in futures markets. J Financ Econ 16(2):213–233. https://doi.org/10.1016/0304-405X(86)90061-9

Chan CC, Fong WM (2006) Realized volatility and transactions. J Bank Finance 30(7):2063–2085. https://doi.org/10.1016/j.jbankfin.2005.05.021

Chan SH, Kim KA, Rhee SG (2005) Price limit performance: evidence from transactions data and the limit order book. J Empir Financ 12(2):269–290. https://doi.org/10.1016/j.jempfin.2004.01.001

Chen G-m, Rui OM, Wang SS (2005) The effectiveness of price limits and stock characteristics: evidence from the Shanghai and Shenzhen stock exchanges. Rev Quant Financ Acc 25:159–182. https://doi.org/10.1007/s11156-005-4247-7

Chen T, Gao Z, He J et al (2019) Daily price limits and destructive market behavior. J Econom 208(1):249–264. https://doi.org/10.1016/j.jeconom.2018.09.014

Chen H, Gu M, Ni B (2023) How price limit affects the market efficiency in a short-sale constrained market? Evidence from a quasi-natural experiment. J Empir Financ 73:22–39. https://doi.org/10.1016/j.jempfin.2023.05.003

Chen H, Petukhov A, Wang J et al (2024) The dark side of circuit breakers. J Financ 79(2):1405–1455. https://doi.org/10.1111/jofi.13310

Cho DD, Russell J, Tiao GC et al (2003) The magnet effect of price limits: evidence from high-frequency data on Taiwan stock exchange. J Empir Financ 10(1–2):133–168. https://doi.org/10.1016/S0927-5398(02)00024-5

Chou P-H, Chou RK, Ko K-C et al (2013) What affects the cool-off duration under price limits? Pac-Basin Financ J 24:256–278. https://doi.org/10.1016/j.pacfin.2013.01.004

Deb SS, Kalev PS, Marisetty VB (2010) Are price limits really bad for equity markets? J Bank Finance 34(10):2462–2471. https://doi.org/10.1016/j.jbankfin.2010.04.001

Engelberg JE, Reed AV, Ringgenberg MC (2018) Short-selling risk. J Financ 73(2):755–786. https://doi.org/10.1111/jofi.12601

Fama EF (1989) Perspectives on October 1987, or, what did we learn from the crash? Irwin, Homewood

Google Scholar  

Farag H (2013) Price limit bands, asymmetric volatility and stock market anomalies: evidence from emerging markets. Glob Financ J 24(1):85–97. https://doi.org/10.1016/j.gfj.2013.03.002

Haruvy E, Noussair CN (2006) The effect of short selling on bubbles and crashes in experimental spot asset markets. J Financ 61(3):1119–1157. https://doi.org/10.1111/j.1540-6261.2006.00868.x

Hastie T, Tibshirani R (1986) Generalized additive models. Stat Sci 1(3):297–310

Hastie T, Tibshirani R (1987) Generalized additive models: some applications. J Am Stat Assoc 82(398):371–386. https://doi.org/10.1080/01621459.1987.10478440

Hou K, Moskowitz TJ (2005) Market frictions, price delay, and the cross-section of expected returns. Rev Financ Stud 18(3):981–1020. https://doi.org/10.1093/rfs/hhi023

Hou K, Li X, Zhong W (2020) Price limits and asymmetry of price dynamics—high frequency evidence from the Chinese stock market. Emerg Mark Financ Trade 56(7):1447–1461. https://doi.org/10.1080/1540496X.2018.1553163

Huang Y-S, Fu T-W, Ke M-C (2001) Daily price limits and stock price behavior: evidence from the Taiwan stock exchange. Int Rev Econ Financ 10(3):263–288. https://doi.org/10.1016/S1059-0560(00)00082-4

Jia S, An Y, Yang L et al (2024) Price limit relaxation and stock price crash risk: evidence from China. Financ Res Lett 59:104715. https://doi.org/10.1016/j.frl.2023.104715

Kahneman D, Tversky A (2013) Prospect theory: an analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I (pp 99–127). World Scientific. https://doi.org/10.1142/9789814417358_0006

Kim KA (2001) Price limits and stock market volatility. Econ Lett 71(1):131–136. https://doi.org/10.1016/S0165-1765(00)00403-1

Kim KA, Limpaphayom P (2000) Characteristics of stocks that frequently hit price limits: empirical evidence from Taiwan and Thailand. J Financ Mark 3(3):315–332. https://doi.org/10.1016/S1386-4181(00)00009-4

Kim KA, Park J (2010) Why do price limits exist in stock markets? A Manipulation-Based Explanation. Eur Financ Manag 16(2):296–318. https://doi.org/10.1111/j.1468-036X.2008.00456.x

Kim KA, Rhee SG (1997) Price limit performance: evidence from the Tokyo stock exchange. J Financ 52(2):885–901. https://doi.org/10.1111/j.1540-6261.1997.tb04827.x

Kim YH, Yagüe J, Yang JJ (2008) Relative performance of trading halts and price limits: evidence from the Spanish stock exchange. Int Rev Econ Finance 17(2):197–215. https://doi.org/10.1016/j.iref.2007.06.003

Kim KA, Liu H, Yang JJ (2013) Reconsidering price limit effectiveness. J Financ Res 36(4):493–518. https://doi.org/10.1111/jfir.12021

Kodres LE, O’Brien DP (1994) The existence of Pareto-superior price limits. Am Econ Rev 84:919–932

Kumar A, Lee CM (2006) Retail investor sentiment and return comovements. J Financ 61(5):2451–2486. https://doi.org/10.1111/j.1540-6261.2006.01063.x

Lauterbach B, Ben-Zion U (1993) Stock market crashes and the performance of circuit breakers: empirical evidence. J Financ 48(5):1909–1925. https://doi.org/10.1111/j.1540-6261.1993.tb05133.x

Lee CM, Ready MJ, Seguin PJ (1994) Volume, volatility, and New York stock exchange trading halts. J Financ 49(1):183–214. https://doi.org/10.1111/j.1540-6261.1994.tb04425.x

Lehmann BN (1989) Commentary: volatility, price resolution, and the effectiveness of price limits. In: Edwards FR (ed) Regulatory reform of stock and futures markets. Springer, The Netherlands, pp 107–111

Chapter   Google Scholar  

Liang H, Liu S, Zhan X et al (2024) The effect of gambling preference on stock volatility: from the evidence of looser price limits regulation. Emerg Mark Financ Trade. https://doi.org/10.1080/1540496X.2024.2308181

Luo Y, Ren H, Yang H (2024) Price limits, informed trading, and information consumption. Res Int Bus Financ 71:102416. https://doi.org/10.1016/j.ribaf.2024.102416

Ma CK, Rao RP, Sears RS (1989) Volatility, price resolution, and the effectiveness of price limits. Regul Reform Stock Futur Market: Spec Issue J Financ Serv Res. https://doi.org/10.1007/978-94-009-2193-1_7

Ma Y, Qian W, Luan Z (2021) Could increasing price limits reduce up limit herding? Evidence from China’s capital market reform. Financ Res Lett 42:101909. https://doi.org/10.1016/j.frl.2020.101909

Miller EM (1977) Risk, uncertainty, and divergence of opinion. J Financ 32(4):1151–1168. https://doi.org/10.2307/2326520

Phylaktis K, Kavussanos M, Manalis G (1999) Price limits and stock market volatility in the Athens stock exchange. Eur Financ Manag 5(1):69–84. https://doi.org/10.1111/1468-036X.00080

Qian Y, Ritter JR, Shao X (2022) Initial public offerings Chinese style. J Finan Quant Anal 59(1):1–38. https://doi.org/10.1017/S002210902200134X

Seasholes MS, Wu G (2007) Predictable behavior, profits, and attention. J Empir Financ 14(5):590–610. https://doi.org/10.1016/j.jempfin.2007.03.002

Shefrin H, Statman M (1985) The disposition to sell winners too early and ride losers too long: theory and evidence. J Financ 40(3):777–790. https://doi.org/10.1111/j.1540-6261.1985.tb05002.x

Tang S (2023) Price limit performance: new evidence from a quasi-natural experiment in China’s ChiNext market. Int Rev Financ Anal 89:102747. https://doi.org/10.1016/j.irfa.2023.102747

Titman S, Wei C, Zhao B (2022) Corporate actions and the manipulation of retail investors in China: an analysis of stock splits. J Financ Econ 145(3):762–787. https://doi.org/10.1016/j.jfineco.2021.09.018

Wan X, Zhang J (2022) The effect of relaxing daily price limit: evidence from the ChiNext market of China. Econ Lett 215:110509. https://doi.org/10.1016/j.econlet.2022.110509

Wong WK, Liu B, Zeng Y (2009) Can price limits help when the price is falling? Evidence from transactions data on the Shanghai stock exchange. China Econ Rev 20(1):91–102. https://doi.org/10.1016/j.chieco.2008.09.002

Zhang X, Wang Z, Hao J et al (2022) Price limit and stock market quality: evidence from a quasi-natural experiment in the Chinese stock market. Pac-Basin Financ J 74:101778. https://doi.org/10.1016/j.pacfin.2022.101778

Zhang X, Li X, Hao J et al (2023) Price limit change and magnet effect: the role of investor attention. Financ Res Lett 53:103577. https://doi.org/10.1016/j.frl.2022.103577

Download references

Acknowledgements

Siyuan Tang thanks for support from National Natural Science Foundation of China (Grant numbers [72102197]).

Author information

Authors and affiliations.

School of Economics and Management, Southwest Jiaotong University, Chengdu, 610031, Sichuan, China

Siyuan Tang

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Siyuan Tang .

Ethics declarations

Conflict of interest.

I confirm that this work is original and has not been published elsewhere, nor is it currently under consideration for publication elsewhere. I declare there is no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 118 KB)

See Fig.  3 .

Time-varying trends of market condition. This figure plots the time-varying trends in the SCI 300 index (black line) and investor sentiment (gray line) before and after widening the price limit range. The left-hand y-axis plots the SCI 300 index values for each trading month and the right-hand y-axis plots the investor sentiment values for each trading month. The x-axis plots the month relative to the event (August 2020)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Tang, S. Asymmetric effectiveness of price limits: evidence from a quasi-natural experiment. Rev Quant Finan Acc (2024). https://doi.org/10.1007/s11156-024-01333-w

Download citation

Accepted : 24 July 2024

Published : 22 August 2024

DOI : https://doi.org/10.1007/s11156-024-01333-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Price limits
• Price discovery
• Trading activity
• Emerging market

JEL Classification

• Find a journal
• Publish with us
• Track your research