Intended for healthcare professionals

  1. Research
  2. Using natural...
  3. Using natural experiments to evaluate population health and health system interventions: new framework for producers and users of evidence
CCBY Open access
Research Methods & Reporting

Using natural experiments to evaluate population health and health system interventions: new framework for producers and users of evidence

BMJ 2025; 388 doi: https://doi.org/10.1136/bmj-2024-080505 (Published 28 March 2025) Cite this as: BMJ 2025;388:e080505
  1. Peter Craig, professor of public health evaluation1,
  2. Mhairi Campbell, systematic reviewer1,
  3. Manuela Deidda, research fellow2,
  4. Ruth Dundas, professor of social epidemiology1,
  5. Judith Green, professor of sociology3,
  6. Srinivasa Vittal Katikireddi, professor of public health and health inequalities1,
  7. Jim Lewsey, professor of medical statistics4,
  8. David Ogilvie, professor of public health research5,
  9. Frank de Vocht, professor of epidemiology and public health67,
  10. Martin White, professor of population health research5
  1. 1MRC/CSO Social and Public Health Sciences Unit, School of Health and Wellbeing, University of Glasgow, Glasgow, UK
  2. 2School of Health and Wellbeing, University of Glasgow, Glasgow, UK
  3. 3Wellcome Centre for Cultures and Environments of Health, University of Exeter, Exeter, UK
  4. 4Health Economics and Health Technology Assessment, School of Health and Wellbeing, University of Glasgow, Glasgow, UK
  5. 5MRC Epidemiology Unit, University of Cambridge, Cambridge, UK
  6. 6Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK
  7. 7NIHR Applied Research Collaboration West, Bristol, UK
  1. Correspondence to: M Campbell mhairi.campbell{at}glasgow.ac.uk
  • Accepted 22 January 2025

Natural experiments are widely used to evaluate the impacts on health of changes in policies, infrastructure, and services. The UK Medical Research Council (MRC) and National Institute for Health and Care Research (NIHR) have published a new framework for conducting and using evidence from natural experimental evaluations. The framework defines key concepts and describes recent advances in designing and planning evaluations of natural experiments, including the relevance of a systems perspective, mixed methods, and stakeholder involvement. It provides an overview of the strengths, weaknesses, applicability, and limitations of the range of methods now available, and makes good practice recommendations for researchers, funders, publishers, and users of evidence.

Unlike true experiments that are conducted by researchers for scientific purposes, natural experiments occur when infrastructure, policies, or services are introduced or changed by governments or healthcare systems. Interventions of this kind are sometimes amenable to randomised controlled trials, for example, if the advantages of randomisation can be negotiated with policy makers or providers at the planning stage and the findings are likely to be transferable across several contexts. Although the randomised controlled trial remains an important method, there are occasions when a trial will not be appropriate or feasible for answering questions about infrastructure, policy, or service changes. However, provided that the intervention divides a population into groups that are otherwise similar, researchers can evaluate the health effects of the changes in a natural experimental evaluation. Natural experiments therefore generate valuable opportunities for evaluating population health, health systems, and other interventions, including those that are, for practical or ethical reasons, not suitable for investigation using randomised controlled trials.

One example of a natural experiment is the introduction by the Scottish government in 2018 of a minimum price at which a unit of alcohol could legally be sold. This was expected to reduce alcohol consumption, with most impact on the heaviest drinkers who tend to drink the lowest priced alcohol. A natural experimental evaluation has been conducted comparing trends in alcohol related deaths and hospital admissions in Scotland, before and after minimum unit pricing was introduced, with trends in England, which did not have a similar policy.1 In addition to policy changes, natural experimental approaches can be used to evaluate changes to health systems and broader infrastructure. An example of changes to health systems is the study by Doyle and colleagues, which analyses the effectiveness of emergency hospital care using ambulance callouts as a form of quasi-random assignment of patients to different hospitals.2 An example of changes to broader infrastructure is the study by Ogilvie and colleagues of the health impacts of a new urban motorway. This study uses a combination of repeat cross sectional and cohort analyses of surveys of residents in intervention and control areas, ethnography, and controlled interrupted time series analysis of routine police road traffic casualty data3 (table 1 and table 2 give overviews of the roles of different quantitative and qualitative methods).

Table 1

Quantitative methods for evaluating natural experiments

View this table:
Table 2

Contributions of qualitative methods to natural experimental evaluations with examples

View this table:

Guidance on using natural experiments was published by the UK Medical Research Council (MRC) in 2012,24 and several other overviews of methods and approaches have been published since then.252627 In recent years, there has been a substantial increase in the number of evaluations of natural experiments, advances in analysis methods, and in the application of whole system approaches to evaluation, greater availability of large administrative or routinely collected datasets, and demand for evaluation of natural experiments delivered at a population level. Whereas the 2012 guidance and subsequent overviews have focused on quantitative methods for measuring the effect of interventions, we believe there is a need for a broader framework that also considers the design and planning of natural experimental evaluations, the role of qualitative, mixed methods and economic evaluation, the use of routinely collected data, and the implications for evidence synthesis.

In this article, we present a new framework that provides an integrated guide for using a natural experimental approach to evaluating population health and health system interventions, covering the whole process from study design and planning through to reporting and dissemination. The framework provides a resource for researchers conducting evaluations, users of evaluation evidence, and evaluation commissioners deciding whether a natural experimental approach would meet their needs. The framework also provides information to help journal editors, funders, and peer reviewers understand the strengths and limitations of funding proposals and articles reporting natural experimental evaluations. A detailed version of the framework funded by the UK MRC and the National Institute for Health and Care Research (NIHR) has been published by the NIHR Journals Library.28

Summary points

  • Natural experiments that occur when policies, services, or infrastructure are introduced, modified, or withdrawn generate valuable opportunities for evaluation

  • The UK Medical Research Council and National Institute for Health and Care Research published a new framework to help producers and users of evidence make the best use of natural experiments

  • The new framework updates and extends previous guidance by taking account of recent developments in approaches to evaluating complex interventions and showing how they can be applied in natural experimental studies

  • Recommendations for good practice include the use of mixed methods, careful attention to the context in which interventions take place, stakeholder engagement, open science practices, and continued investment in research data infrastructures

Development of the framework

We convened a writing group of researchers with expertise in epidemiology, health economics, public health and sociology, and methodological expertise in statistics, qualitative research, and evidence synthesis. An advisory group of stakeholders experienced in using natural experimental evidence and with methodological expertise provided input at key stages of the project. We held online workshops and a consultation to obtain expert opinion on developing the framework. The workshops helped configure initial drafts of the framework, then during the consultation, participants were invited to review each chapter of the framework, with the additional feedback used to further refine the content. Participants in the workshops (n=21) and consultation (n=44) comprised researchers and other relevant stakeholders in Europe, Africa, the Americas, and Australasia, including members of research funding boards, policy makers, journal editors, and representatives of national and local governments. Further details of the methods used to develop the framework are available in online supplementary file 1 and the detailed report.28 The workshops and consultation helped guide the use of a broad definition of natural experiment, refine a framework for planning and conducting evaluations of natural experimental studies, and specify in detail the role and content of analytic methods in evaluations.

Framework for natural experimental evaluations

Natural experimental evaluations are valuable for many reasons. They can be used to study interventions under real world conditions, to evaluate very long term outcomes, and to investigate outcomes that were not the main purpose of the intervention—for example, the impacts on health of changes in education or social welfare policies. These evaluations allow retrospective examination of policies or interventions, and the evaluation of large scale or irreversible interventions such as national policy changes or large public infrastructure investments. They are the approach of choice when a controlled trial is not possible or ethical, or when previous political or financial commitments can make explicit experimentation unattractive to decision makers.

Concepts and definitions

The new framework uses a broad definition of natural experiments. We use the term to refer to events or processes outside the control of a researcher that divide a population into exposed or unexposed groups, or groups with differing degrees of exposure. A natural experimental evaluation uses data emerging from the introduction, delivery, or withdrawal of a natural experiment to evaluate the impact of the intervention on an outcome or outcomes. Other authors have proposed narrower definitions, such as those based on a list of study designs or on methods that can address unobserved confounding,29 or that require intervention assignment to be “as-if randomised.”30 We prefer a broader definition for two reasons. Firstly, study design labels do not necessarily indicate study quality, and lists of methods tend to be arbitrary and rapidly become outdated. Secondly, “as-if randomisation” can be difficult to define precisely.31 “As-if randomisation” captures an important feature of natural experiments, but represents one end of a spectrum of possible natural experiments rather than characterising the whole range of opportunities that can usefully be exploited with the right choice of methods.24 The focus on specific events or processes (known as assignment or allocation processes) that determine exposure distinguishes natural experimental evaluations from the broader range of observational studies.

Design and planning

We recommend an approach to planning natural experimental evaluations adapted from the MRC/NIHR framework for the development and evaluation of complex interventions.32 That framework highlights the value of a complex systems perspective for evaluation because considering a natural experiment event as a disruption within a system can help identify the breadth of potential intended and unintended impacts, as well as the role of context in shaping the effects of the intervention. Figure 1 presents the stages of identifying and appraising opportunities for a natural experimental evaluation and working out a feasible and appropriate design. Three important phases in the scoping and planning of natural experimental evaluations are identifying and theorising natural experiments, assessing their evaluability, and conducting feasibility studies for a future evaluation.

Fig 1
Fig 1

Framework for planning natural experimental evaluations: adaptation of UK Medical Research Council and National Institute for Health and Care Research (MRC/NIHR) framework for developing and evaluating complex interventions32

Identifying natural experiment—A variety of circumstances can provide opportunities for a natural experimental evaluation.24 We outline five kinds of opportunity that have been widely used. One is where there is a clear division in the presence or type of exposure between otherwise similar subpopulations by time or place of implementation; for instance, when policies such as state level gun control laws are implemented in some areas but not others.33 Individual level allocation mechanisms such as eligibility criteria embedded within a policy—for example, means tests that define entitlement to social security benefits3435—provide a second kind of opportunity. A third is the phased implementation of a policy across a population, such as implementation of the UK social security benefit Universal Credit.3637 A fourth is when randomisation is built into the policy, as in the case of a lottery used to allocate housing places.38 A fifth is when flaws or shortcomings emerge in policy delivery, such as database errors3940 or false negative test results in the UK’s Test and Trace programme for covid-19.

Assessing evaluability of natural experiment and feasibility of evaluation—A formal evaluability assessment is one way of ensuring that natural experimental evaluations are well designed and address relevant questions. Evaluability assessment is a systematic, collaborative approach to evaluation planning that is increasingly used in public health research.41 The assessment helps to identify key uncertainties that the evaluation should address, develop a theory of how the intervention works,32 and reach consensus about the plausible overall and distributional (ie, equity) effects the intervention could produce, the potential influence of the evaluation on future policy decisions, and how the results might contribute to the wider evidence base.4243 Assessing the evaluability of the natural experiment can help to ensure a shared understanding with stakeholders of what an evaluation can and cannot deliver. The process enables information to be gathered on intervention delivery and the availability of, and access to, monitoring data, and establishes a clear understanding of the assignment process for the intervention. A thorough assessment of the feasibility of the evaluation should be conducted to assess the practicalities of implementing the evaluation design, such as whether routinely collected data adequately capture differences in exposure and outcomes, that the data will enable sufficient statistical power in analyses, and whether alternative methods can be used if the preferred option is not feasible.43 As with randomised trials, the funding for natural experimental evaluations might have to incorporate contingency in case the proposed evaluation is unviable; for example, by including an explicit breakpoint in the award when a formal decision of whether to continue would be made.

Protocols and preregistration—It is best practice to develop a protocol, or some other form of prior study plan appropriate to the methods being used, and to place it in the public domain before analysis commences. Natural experimental evaluations commonly use several datasets and methods of analysis, and are often retrospective. Publishing analysis plans before data analysis begins enables users to see which analyses reflect previous hypotheses and which have been informed by emerging findings. Protocols can of course be adapted, if need be, provided that amendments are systematically recorded to maintain a transparent record of how the study design has evolved.

Engaging stakeholders—In evaluations of natural experiments, there will be a diverse range of stakeholders involved at different stages of the intervention and the evaluation. Relevant stakeholders might include legislators, policy makers, organisations, and individuals responsible for implementation, institutions enabling access to necessary datasets, advocacy groups, representatives of communities affected by the intervention, and the evaluation funder.44 Involvement of such stakeholders throughout the evaluation maximises the likelihood of findings being relevant, understood, taken up, and used for decision making. To avoid conflicts of interest, clear boundaries should be agreed for stakeholder involvement and the protocol or evaluation plan made publicly available.45

Taking complex systems perspective—Population health and health system interventions typically have several components and their impacts are moderated by interactions with elements of the wider system in which they are implemented. When evaluating a natural experiment, considering such interactions can help researchers understand why the intervention succeeds or fails to achieve its intended impact,46 or why impacts vary from one setting to another.47 For example, if we want to evaluate the introduction of a tobacco tax, we might consider how smokers, retailers, producers, smugglers, the mass media, think tanks, tobacco control advocates, and the cigarette taxation system might react to the introduction of the tax in ways that could dampen or amplify its effects. Taking a systems perspective involves including processes in the evaluation to build an understanding of how the intervention interacts with its context to produce impact. This can include using system mapping4849 (eg, developing a causal loop diagram) to create a robust theory of change for the intervention, identifying outcomes to measure (intended and unintended), or using a system dynamics model to simulate the evolution of the system over time.50

Methods for natural experimental evaluations

Evaluation designs that use both qualitative and quantitative methods are needed to provide an understanding of how the intervention effects were achieved given interactions between the intervention and elements of the wider system. Use of mixed methods can strengthen the estimation of effect sizes by providing a detailed understanding of the assignment process and how far it can be expected to generate otherwise comparable groups of exposed and unexposed units. Single or multiple qualitative and quantitative methods can be combined within an evaluation in several ways, including sequential exploratory, sequential explanatory, parallel convergent (triangulation), and integrated approaches.5152 The value of a mixed methods approach is greater if planned in advance,53 for instance, by including an integration work package in the project plan and explicitly earmarking resources for it.54 For example, in research to increase understanding of sugar-sweetened beverage taxation policies, simultaneous consideration of several types of data could be used to investigate whether the tax reduces the number of sugar-sweetened beverage consumers, and whether this reduction leads to an increase in the political acceptability of further taxes on sugar-sweetened beverages.55

Quantitative methods

The research question and the nature of the assignment process will determine what methods can be used to obtain quantitative effect estimates in a natural experimental evaluation.56 A useful planning tool is the target trial framework, which matches elements of study design to the components of a hypothetical randomised trial.57 A variety of study designs and quantitative analytical methods are available. Each method has strengths and limitations, and will be more or less applicable in specific circumstances. An overview is provided in table 1. When selecting methods, it is therefore best to avoid thinking in terms of a hierarchy. Instead, the choice will be determined by the research questions, the circumstances of the evaluation, and the availability of data. Often, the ideal data for a natural experimental evaluation will not be available. Because the assignment processes used in natural experimental evaluations are rarely random, threats to internal validity caused by selective exposure to the intervention are always a concern. Use of a combination of methods (each with differing strengths and limitations) might help address the threats. When reporting the analysis, it is important to state the treatment effect being evaluated by specifying the causal contrast or estimand(s);56 and to explain for users whether the main interest is in the average effect of the intervention on an individual or the average effect of the intervention on the population.58

Economic evaluation

Economic evaluations should ideally be conducted in conjunction with evaluations of effectiveness of the natural experiment because there are resource constraints on the implementation of policies. Designing, conducting, and reporting economic evaluations of natural experiments generate specific challenges. Because natural experimental evaluations often assess effects that are by-products rather than directly intended outcomes of the intervention, economic evaluation will often require a broad, “societal,” perspective rather than a sector specific perspective. For example, in an evaluation of the health impacts of Universal Credit, a UK social security programme intended to prevent poverty and provide incentives to work, the economic evaluation examines health and wellbeing, as well as income, employment, and economic productivity outcomes.59 Such a broad perspective will require data on multisectoral costs and outcomes, which could be hard to obtain. Routinely collected data can enable a long time horizon over which outcomes are calculated. If suitable data are available, methods such as distributional cost effectiveness analysis60 and extended cost effectiveness analysis can be used to investigate the equity impact of an intervention. Challenges involved in designing and conducting an economic evaluation alongside natural experimental evaluation are outlined by Deidda and colleagues,61 and guidance has recently been developed for identifying appropriate use and application of complex system models in economic evaluations.62

Qualitative methods

Qualitative methods can strengthen natural experimental evaluations in several ways. They can help researchers to characterise the intervention, understand assignment processes, explore mechanisms, identify threats to validity, help define parameters, and interpret and strengthen causal claims. Evaluations should therefore be planned and conducted in an integrated way to ensure that the qualitative components are incorporated throughout to achieve maximum use of the qualitative data. Below we highlight the components of an evaluation to which qualitative methods contribute beyond their use in process evaluations.63Table 2 provides examples of using qualitative methods. Note that the framework focuses on the use of qualitative methods within evaluations whose primary aim is to estimate effect sizes of interventions, rather than studies whose main goal is to address questions such as the contexts in which interventions work.4764

A key use of qualitative methods is to characterise the intervention in order to understand its rationale and the organisational, historical, political, and social and policy context, including co-occurring interventions, in which it is implemented.6566 Qualitative system mapping techniques (eg, group model building67) can be used to identify the elements of contexts that are important to include in the evaluation68 and can help to identify important preconditions, assumptions, and potential mechanisms of effect, given the structure of the system.69 Qualitative methods can also inform the selection of exposed and unexposed populations, and choice of appropriate outcomes and indicators.70 This helps ensure that the chosen quantitative indicators capture what is intended, such as outcomes that are important for stakeholders, and that their strengths and limitations are well understood. In some natural experimental evaluations, qualitative methods are used to generate data on outcomes, to enable triangulation, or to identify secondary outcomes not captured in quantitative datasets or unanticipated at the outset.71 Qualitative data might also serve as primary evidence on changes in knowledge, understanding, or practices that are associated with an intervention when quantitative data on such outcomes are not available (table 2).

Most importantly, qualitative methods can help us to understand mechanisms and mediators, and help explain change. Analysis of qualitative data can explore why the intervention did, or did not, lead to anticipated outcomes through making inferences about causal processes from comparisons within the case,72 drawing on approaches such as process tracing73 or analytic induction.74 Appropriate qualitative analysis, in the light of theories of change, strengthens causal inferences and claims about transferability.

Reporting, critical appraisal, and evidence synthesis

Reporting—Accurate, clear, and comprehensive reporting of the natural experiment and the evaluation is crucial for the best use and understanding of the evaluation. Details of reporting guidance likely to be useful to researchers conducting natural experimental evaluations are provided in supplementary file 2.

Critical appraisal—A systematic assessment of the design, conduct, and analysis of a study might be required to understand the rigour of an individual study or undertaken as part of evidence synthesis. No single critical appraisal tool can fully assess the risk of bias of all natural experimental evaluation study designs.75767778 For some types of natural experimental evaluations, and some systematic reviews, an appraisal framework more like those used for qualitative research might be more appropriate.7980

Evidence synthesis—Synthesising evidence from natural experimental evaluations requires consideration of how to manage the expected diversity in study design and characteristics. For some review topics, it might be more valuable to examine whether there is any evidence for an effect or to explore intervention mechanisms, for example, rather than to estimate an overall effect size within a meta-analysis. Approaches for synthesis without meta-analysis will often be useful; for example, guidance provided by Cochrane8182 and the RAMESES (Realist and Meta-narrative Evidence Syntheses: Evolving Standards) group.83 The specifics of the evidence synthesis questions and studies involved will determine the appropriate methods to use, with a mixed methods design often useful.8485

Certainty of evidence—If the aim of evidence synthesis is to estimate an effect size, then it is usually appropriate to summarise confidence in the overall effect estimate; that is, to formally assess the overall certainty of the findings. The framework most often used is GRADE (Grading of Recommendations, Assessment, Development, and Evaluations).86 Challenges in using GRADE with natural experimental evaluations—including the difficulty of classifying risk of bias for study designs not typically used in epidemiology, selecting outcomes for synthesis, and the lack of differentiation in certainty assessments for population health and health system interventions—are being addressed by the GRADE Public Health Group by developing further guidance and provision of training.87

Data infrastructure and information governance

Natural experimental evaluations often use data collected for other purposes. Such sources include administrative or commercial datasets, population surveys, and data collected from point of sale, traffic sensors, fitness apps, and so on. Secondary analysis of such datasets enables interventions to be evaluated retrospectively using data whose unit cost is a small fraction of the cost of collecting primary data. However, negotiating access to such datasets can often be a time consuming, expensive, and uncertain process, especially if the research involves combining data from several sources. An alternative is to use secure research platforms known as Trusted Research Environments, which are designed to curate data securely and to provide researchers with efficient access. For example, the Brazilian Centre for Data and Knowledge Integration for Health (CIDACS) stores, processes, and links identified data and has secure procedures to provide access to deidentified or anonymised data linking social benefit programmes with deaths, births, and infectious diseases (https://cidacs.bahia.fiocruz.br/en/).

Good practice considerations

Good practice considerations, derived from the content of the updated framework, are provided for different users. Box 1 presents a condensed form of these recommendations. The main messages for planning, commissioning, conducting, reporting, and using evidence from natural experimental evaluations have been grouped according to the key audience, concentrating on messages that are practical and implementable.

Box 1

Good practice considerations

All producers and users of natural experimental evaluations should:

  • Understand the design and planning processes of an evaluation of a natural experiment, including how to identify opportunities for natural experimental evaluation, select the most appropriate evaluation approach, and assess the feasibility of the evaluation.

  • Consider the needs and perspectives of the full range of stakeholders in the intervention and its evaluation.

  • Recognise the respective strengths of quantitative, qualitative, and integrated analytical approaches, incorporating perspectives from diverse disciplines, such as economics, social sciences, and epidemiology, for investigating the impacts of natural experiments.

Researchers conducting natural experimental evaluations should:

  • Be aware of the circumstances that are likely to give rise to good opportunities for a natural experimental approach. Adopt methods that are appropriate to the data available and to the processes that determine exposure to the intervention of interest.

  • Consider adopting a systems approach to evaluating natural experiments.

  • Consider using a combination of methods, including alternative methods of effect estimation, robustness checking, and a mixture of qualitative and quantitative methods.

  • Adopt open science practices, publishing a protocol or plan of the study in advance in open access journals or repositories.

  • Clearly report the natural experiment event and all stages of the evaluation, including its planning, protocol, analyses, and results, using established reporting standards if available, ensuring key details are in plain language appropriate for the evidence users.

  • Include a health equity perspective in the evaluation. Be aware that evaluation of the strength of evidence from natural experimental evaluations should be based on detailed appraisal of the strengths and limitations of the study methods for the specific evaluation, not on generic hierarchies of study design.

Research funders and commissioners supporting and investing in natural experimental evaluations should:

  • Encourage best practice when commissioning or funding natural experimental evaluations, for example, by requiring that a protocol or study plan is available before starting the analysis, that findings and analytic scripts are published in open access journals or other suitable platforms, and that the relevant reporting guidelines are followed.

  • Establish processes within funding bodies to facilitate flexible and timely responses to prospective natural experimental evaluation opportunities.

  • Support capacity building for natural experiments by investing in infrastructure and the workforce.

  • Negotiate with data owners to make routinely collected data available and linkable to other datasets for evaluations of policies and programmes.

  • When commissioning natural experimental evaluations, be prepared to be flexible and pragmatic, and accept that both the evaluability of the natural experiment and the feasibility of the evaluation require assessment.

  • Flexibility might also be required when considering the start date and timescale of the research because policy interventions can be delayed, changed, or withdrawn, and the effects of each will require consideration in an evaluation.

Journal editors, policy makers, practitioners, and other decision makers publishing and using evidence from natural experimental evaluations should:

  • Provide guidance for authors and reviewers on requirements for reports of natural experimental evaluations.

  • Use evidence from high quality natural experimental evaluations when this is the most appropriate or available form of evidence, being aware of any limitations of the evaluation.

  • Incorporate evaluation plans into the implementation of new policies and programmes.

RETURN TO TEXT

Conclusion

We have provided an overarching framework for evaluating population health and health system interventions as natural experiments, within which more detailed guidance on specific methods and techniques can be situated. The new framework is aligned with the updated MRC/NIHR framework for evaluating complex interventions in emphasising the value of understanding interventions as events in complex systems, and adopting a plurality of methods with the aim of providing evidence that is useful for decision making. To develop the framework, we consulted widely with producers and users of evidence whose feedback confirmed the usefulness of adopting a broad definition of a natural experiment, rather than limiting the approach to a narrow range of circumstances and techniques. Interest in natural experimental approaches has expanded markedly since the first edition of the guidance was published in 2012. With further investment in research capacity and infrastructure, continued methodological innovation, and a growing appreciation of the range of opportunities presented by service and policy developments, we believe there is scope for considerable further expansion.

Acknowledgments

We thank the participants of the workshops and online consultation for their contributions. We acknowledge the support and insight of the advisory group and the oversight group. Full acknowledgments are provided in the framework document. Some of this paper has been reproduced from the framework document.

Footnotes

  • Contributors: All authors made a substantial contribution to all stages of the development of the framework, contributing to development, drafting, and final approval of the framework and this paper. PC led the conceptualisation of the project, led on concepts and definitions, design and planning, infrastructure, data governance, and led the collation and editing of the framework. MC drafted evidence synthesis and good practice considerations, developed material for and helped facilitate the expert workshops, organised the online consultation, contributed to the content, collation, and editing of the framework, and wrote the first draft of this paper. MD led on economic evaluation. RD led on study registration and protocols. JG led and drafted qualitative methods. SVK drafted mixed methods, co-led on concepts and definitions, qualitative methods, and evidence synthesis. JL co-led quantitative methods. DO co-led design and planning and evidence synthesis. FdV co-led quantitative methods. MW contributed to the conceptualisation of the project, led on ideas for content on guidance for policy makers and other decision makers, and capacity building for optimal use of natural experimental evaluations. PC chaired the expert workshops, MC, MD, RD, SVK, JL, and DO contributed to the expert workshops. PC is the guarantor of this work. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding: This project was funded by the National Institute for Health and Care Research (NIHR) Public Health Research (PHR) programme and Medical Research Council (MRC) UK (MC_PC_21009). PC, MC, RD, and SVK are supported by the Medical Research Council (MC_UU_00022/2) and the Scottish Government Chief Scientist Office (SPHSU17). SVK is also supported by the European Research Council (949582). RD is supported by UK Prevention Research Partnership (MR/S037608/1). JG is supported by Wellcome Trust (Centre Grant 203109/Z/16/Z). DO and MW are supported by the Medical Research Council (unit programme MC_UU_00006/7). The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: support from National Institute for Health and Care Research Public Health Research programme, Medical Research Council, Scottish Government Chief Scientist Office, European Research Council, UK Prevention Research Partnership, and Wellcome Trust for the submitted work. JG was a member of the NIHR Public Health Research (PHR) Funding Board (2017-23), the O’Brien Institute for Public Health, Calgary, Canada, International SAB (2017-22), and the NIHR PHR Behavioural Science Scientific Advisory Board (2019-23). SVK and JL are members of the NIHR PHR Funding Board (2020-25). SVK was a member of the NIHR Long COVID funding board (2021-22) and the NIHR Policy Research Units funding board (2023). FdV is a member of the NIHR PHR Funding Board (2017-28), the Swedish Research Council Public Health Funding Committee, the UNSCEAR Expert Group on the evaluation of diseases of the circulatory system from radiation exposure (CircuDis), and a committee member of the Health Council of the Netherlands Future Hazards. RD is a member of the Wellcome Trust Population Health Advisory Board (2022-25), the NIHR Population Health Career Scientist Committee, and the NIHR Advisory Board for Evaluation and co-creation to optimise use and benefits of the Healthy Start Scheme. MW was a member of the NIHR PHR research funding panel (2009-20), director of the PHR programme and chaired the Prioritisation Board (2014-20), member of the NIHR Strategy Board (2014-20), member of the MRC Population Health Sciences Strategy Group (PHSG) (2014-20), member of the MRC Public Health Intervention Development (PHIND) panel (2014-18).

  • Patient and public involvement: The project was methodological. Throughout the course of the project there was engagement with appropriate audiences, such as relevant policy makers, practitioners, and local and national government representatives. The workshops and consultation exercise included input from evidence users as well as researchers.

  • Dissemination to participants and related patient and public communities: The authors plan to continue dissemination of the research through a launch event webinar, presentations at conferences and through social media to stakeholders who generate or use evidence, including policy makers, practitioners, and local and national government representatives. The authors are committed to disseminating the findings in formats accessible to relevant audiences to promote broader engagement in natural experimental evaluations.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

http://creativecommons.org/licenses/by/4.0/

This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/.

References

  1. ↵
    1. Wyper GMA,
    2. Mackay DF,
    3. Fraser C,
    4. et al
    . Evaluating the impact of alcohol minimum unit pricing on deaths and hospitalisations in Scotland: a controlled interrupted time series study. Lancet2023;401:1361-70. doi:10.1016/S0140-6736(23)00497-X. pmid:36963415
    OpenUrlCrossRefPubMed
  2. ↵
    1. Doyle J,
    2. Graves J,
    3. Gruber J,
    4. Kleiner S
    . Measuring returns to hospital care: evidence from ambulance referral patterns. J Polit Econ2015;123:170-214. doi:10.1086/677756. pmid:25750459
    OpenUrlCrossRefPubMed
  3. ↵
    1. Ogilvie D,
    2. Foley L,
    3. Nimegeer A,
    4. et al
    . Health impacts of the M74 urban motorway extension: a mixed-method natural experimental study. Public Health Research2017;5. doi:10.3310/phr05030 pmid:28459514
    OpenUrlCrossRefPubMed
    1. Hanckel B,
    2. Ruta D,
    3. Scott G,
    4. Peacock JL,
    5. Green J
    . The Daily Mile as a public health intervention: a rapid ethnographic assessment of uptake and implementation in South London, UK. BMC Public Health2019;19:1167. doi:10.1186/s12889-019-7511-9 pmid:31455316
    OpenUrlCrossRefPubMed
    1. Jones S,
    2. Hamilton S,
    3. Bell R,
    4. et al
    . What helped and hindered implementation of an intervention package to reduce smoking in pregnancy: process evaluation guided by normalization process theory. BMC Health Serv Res2019;19:297. doi:10.1186/s12913-019-4122-1. pmid:31072363
    OpenUrlCrossRefPubMed
    1. Penney T,
    2. Adams J,
    3. Briggs A,
    4. et al
    . Evaluation of the impacts on health of the proposed UK industry levy on sugar sweetened beverages: developing a systems map and data platform, and collection of baseline and early impact data.National Institute for Health Research, 2018.
    1. Christie N,
    2. Steinbach R,
    3. Green J,
    4. Mullan MP,
    5. Prior L
    . Pathways linking car transport for young adults and the public health in Northern Ireland: a qualitative study to inform the evaluation of graduated driver licensing. BMC Public Health2017;17:551. doi:10.1186/s12889-017-4470-x pmid:28592258
    OpenUrlCrossRefPubMed
    1. Ogilvie D,
    2. Panter J,
    3. Guell C,
    4. Jones A,
    5. Mackett R,
    6. Griffin S
    . Health impacts of the Cambridgeshire Guided Busway: a natural experimental study. Public Health Research2016;4. doi:10.3310/phr04010 pmid:26764445
    OpenUrlCrossRefPubMed
    1. de Vocht F,
    2. McQuire C,
    3. Brennan A,
    4. et al
    . Evaluating the causal impact of individual alcohol licensing decisions on local health and crime using natural experiments with synthetic controls. Addiction2020;115:2021-31. doi:10.1111/add.15002 pmid:32045079
    OpenUrlCrossRefPubMed
    1. Jones A,
    2. Steinbach R,
    3. Roberts H,
    4. Goodman A,
    5. Green J
    . Rethinking passive transport: bus fare exemptions and young people’s wellbeing. Health Place2012;18:605-12. doi:10.1016/j.healthplace.2012.01.003. pmid:22321902
    OpenUrlCrossRefPubMedWeb of Science
    1. Guell C,
    2. Panter J,
    3. Ogilvie D
    . Walking and cycling to work despite reporting an unsupportive environment: insights from a mixed-method exploration of counterintuitive findings. BMC Public Health2013;13:497. doi:10.1186/1471-2458-13-497 pmid:23705951
    OpenUrlCrossRefPubMed
    1. Green J,
    2. Perkins C,
    3. Steinbach R,
    4. Edwards P
    . Reduced street lighting at night and health: a rapid appraisal of public views in England and Wales. Health Place2015;34:171-80. doi:10.1016/j.healthplace.2015.05.011. pmid:26057894
    OpenUrlCrossRefPubMed
    1. Burges Watson D,
    2. Adams J,
    3. Azevedo LB,
    4. Haighton C
    . Promoting physical activity with a school-based dance mat exergaming intervention: qualitative findings from a natural experiment. BMC Public Health2016;16:609. doi:10.1186/s12889-016-3308-2. pmid:27440200
    OpenUrlCrossRefPubMed
    1. Benton JS,
    2. Cotterill S,
    3. Anderson J,
    4. et al
    . Impact of a low-cost urban green space intervention on wellbeing behaviours in older adults: a natural experimental study. Wellbeing Space Soc2021;2:100029doi:10.1016/j.wss.2021.100029.
    OpenUrlCrossRef
    1. Macintyre VG,
    2. Cotterill S,
    3. Anderson J,
    4. Phillipson C,
    5. Benton JS,
    6. French DP
    . “I Would Never Come Here Because I’ve Got My Own Garden”: older adults’ perceptions of small urban green spaces. Int J Environ Res Public Health2019;16:1994. doi:10.3390/ijerph16111994 pmid:31195595
    OpenUrlCrossRefPubMed
    1. Hewer RMF,
    2. Hill S,
    3. Amos A,
    4. et al
    . Student perceptions of smoke-free school policies in Europe—a critical discourse analysis. Crit Public Health2022;32:509-22. doi:10.1080/09581596.2020.1856332.
    OpenUrlCrossRef
    1. Halkjær S,
    2. Lueg R
    . The effect of specialization on operational performance. Int J Oper Prod Manage2017;37:822-39. doi:10.1108/IJOPM-03-2015-0152.
    OpenUrlCrossRef
    1. Alvarado M,
    2. Penney TL,
    3. Unwin N,
    4. Murphy MM,
    5. Adams J
    . Evidence of a health risk “signalling effect” following the introduction of a sugar-sweetened beverage tax. Food Policy2021;102:102104. doi:10.1016/j.foodpol.2021.102104. pmid:34404960
    OpenUrlCrossRefPubMed
    1. Green J,
    2. Roberts H,
    3. Petticrew M,
    4. et al
    . Integrating quasi-experimental and inductive designs in evaluation: A case study of the impact of free bus travel on public health. Evaluation2015;21:391-406. doi:10.1177/1356389015605205.
    OpenUrlCrossRef
    1. Gibson K,
    2. Pollard TM,
    3. Moffatt S
    . Social prescribing and classed inequality: a journey of upward health mobility?Soc Sci Med2021;280:114037. doi:10.1016/j.socscimed.2021.114037 pmid:34033978
    OpenUrlCrossRefPubMed
    1. Cummins S,
    2. Clark C,
    3. Lewis D,
    4. et al
    . The effects of the London 2012 Olympics and related urban regeneration on physical and mental health: the ORiEL mixed-methods evaluation of a natural experiment. Public Health Research2018;6. doi:10.3310/phr06120 pmid:30422451
    OpenUrlCrossRefPubMed
  22. Guevara T, Sarmiento OL, Higuera D, et al. Urban transformations and health: results from the TransMiCable evaluation: Urban Health in Latin America (SALURBAL) project, 2020. Available from: https://drexel.edu/~/media/Files/lac/Publications/TransMiCableENG.ashx?la=en
    1. Armstrong B,
    2. Bonnington O,
    3. Chalabi Z,
    4. et al
    . The impact of home energy efficiency interventions and winter fuel payments on winter- and cold-related mortality and morbidity in England: a natural equipment mixed-methods study. Public Health Research2018;6:1-110. doi:10.3310/phr06110 pmid:30329254
    OpenUrlCrossRefPubMed
  24. ↵
    1. Craig P,
    2. Cooper C,
    3. Gunnell D,
    4. et al
    . Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. J Epidemiol Community Health2012;66:1182-6. doi:10.1136/jech-2011-200375. pmid:22577181
    OpenUrlAbstract/FREE Full Text
  25. ↵
    1. Basu S,
    2. Meghani A,
    3. Siddiqi A
    . Evaluating the health impact of large-scale public policy changes: classical and novel approaches. Annu Rev Public Health2017;38:351-70. doi:10.1146/annurev-publhealth-031816-044208. pmid:28384086
    OpenUrlCrossRefPubMed
  26. ↵
    1. Bärnighausen T,
    2. Tugwell P,
    3. Røttingen J-A,
    4. et al
    . Quasi-experimental study designs series-paper 4: uses and value. J Clin Epidemiol2017;89:21-9. doi:10.1016/j.jclinepi.2017.03.012. pmid:28365303
    OpenUrlCrossRefPubMed
  27. ↵
    1. Craig P,
    2. Katikireddi SV,
    3. Leyland A,
    4. Popham F
    . Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Annu Rev Public Health2017;38:39-56. doi:10.1146/annurev-publhealth-031816-044327. pmid:28125392
    OpenUrlCrossRefPubMed
  28. ↵
    1. Craig P,
    2. Campbell M,
    3. Deidda M
    . Using natural experiments to evaluate population health interventions: a framework for producers and users of evidence. Public Health Res (Southampt)2025 (in press).
  29. ↵
    1. Tugwell P,
    2. Knottnerus JA,
    3. McGowan J,
    4. Tricco A
    . Big-5 quasi-experimental designs. J Clin Epidemiol2017;89:1-3. doi:10.1016/j.jclinepi.2017.09.010 pmid:29110785
    OpenUrlCrossRefPubMed
  30. ↵
    1. Dunning T
    . Natural experiments in the social sciences: a design-based approach.Cambridge University Press, 2012doi:10.1017/CBO9781139084444.
    OpenUrlCrossRef
  31. ↵
    1. Albers PN,
    2. Rinaldi C,
    3. Brown H,
    4. et al
    . Natural experiments for the evaluation of place-based public health interventions: a methodology scoping review. Front Public Health2023;11:1192055. doi:10.3389/fpubh.2023.1192055 pmid:37427271
    OpenUrlCrossRefPubMed
  32. ↵
    1. Skivington K,
    2. Matthews L,
    3. Simpson SA,
    4. et al
    . A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ2021;374:n2061. doi:10.1136/bmj.n2061. pmid:34593508
    OpenUrlFREE Full Text
  33. ↵
    1. Humphreys DK,
    2. Gasparrini A,
    3. Wiebe DJ
    . Evaluating the impact of Florida’s “stand your ground” self-defense law on homicide and suicide by firearm: an interrupted time series study. JAMA Intern Med2017;177:44-50. doi:10.1001/jamainternmed.2016.6811. pmid:27842169
    OpenUrlCrossRefPubMed
  34. ↵
    1. Katikireddi SV,
    2. Molaodi OR,
    3. Gibson M,
    4. Dundas R,
    5. Craig P
    . Effects of restrictions to Income Support on health of lone mothers in the UK: a natural experiment study. Lancet Public Health2018;3:e333-40. doi:10.1016/S2468-2667(18)30109-9 pmid:29976327
    OpenUrlCrossRefPubMed
  35. ↵
    1. Codreanu MA,
    2. Waters T
    . Do work search requirements work? Evidence from a UK reform targeting single parents.Institute for Fiscal Studies, 2023doi:10.1920/wp.ifs.2023.0223.
    OpenUrlCrossRef
  36. ↵
    1. Wickham S,
    2. Bentley L,
    3. Rose T,
    4. Whitehead M,
    5. Taylor-Robinson D,
    6. Barr B
    . Effects on mental health of a UK welfare reform, Universal Credit: a longitudinal controlled study. Lancet Public Health2020;5:e157-64. doi:10.1016/S2468-2667(20)30026-8 pmid:32113519
    OpenUrlCrossRefPubMed
  37. ↵
    1. Brewer M,
    2. Dang T,
    3. Tominey E
    . Universal Credit: welfare reform and mental health.IZA Institute of Labor Economics, 2022.
  38. ↵
    1. Nguyen QC,
    2. Rehkopf DH,
    3. Schmidt NM,
    4. Osypuk TL
    . Heterogeneous effects of housing vouchers on the mental health of US adolescents. Am J Public Health2016;106:755-62. doi:10.2105/AJPH.2015.303006 pmid:26794179
    OpenUrlCrossRefPubMed
  39. ↵
    Fetzer T, Graeber T. Measuring the scientific effectiveness of contact tracing: Evidence from a natural experiment. P Natl A Sci 2021;118(33):e2100814118. doi:10.1073/pnas.2100814118
    OpenUrlAbstract/FREE Full Text
  40. ↵
    1. Fetzer T
    . Measuring the epidemiological impact of a false negative: evidence from a natural experiment.University of Warwick, Department of Economics, 2021.
  41. ↵
    1. Leviton LC,
    2. Khan LK,
    3. Rog D,
    4. Dawkins N,
    5. Cotton D
    . Evaluability assessment to improve public health policies, programs, and practices. Annu Rev Public Health2010;31:213-33. doi:10.1146/annurev.publhealth.012809.103625 pmid:20235852
    OpenUrlCrossRefPubMedWeb of Science
  42. ↵
    1. Ogilvie D,
    2. Adams J,
    3. Bauman A,
    4. et al
    . Using natural experimental studies to guide public health action: turning the evidence-based medicine paradigm on its head. J Epidemiol Community Health2020;74:203-8. doi:10.1136/jech-2019-213085. pmid:31744848
    OpenUrlAbstract/FREE Full Text
  43. ↵
    1. Ogilvie D,
    2. Cummins S,
    3. Petticrew M,
    4. White M,
    5. Jones A,
    6. Wheeler K
    . Assessing the evaluability of complex public health interventions: five questions for researchers, funders, and policymakers. Milbank Q2011;89:206-25. doi:10.1111/j.1468-0009.2011.00626.x pmid:21676021
    OpenUrlCrossRefPubMedWeb of Science
  44. ↵
    1. Campbell M,
    2. Katikireddi SV,
    3. Hoffmann T,
    4. Armstrong R,
    5. Waters E,
    6. Craig P
    . TIDieR-PHP: a reporting guideline for population health and policy interventions. BMJ2018;361:k1079. doi:10.1136/bmj.k1079 pmid:29769210
    OpenUrlFREE Full Text
  45. ↵
    1. Maani N,
    2. Petticrew M,
    3. Galea S
    1. Cullerton K,
    2. White M
    . Understanding and managing corporate conflicts of interest. In: Maani N, Petticrew M, Galea S, eds. The commercial determinants of health.Oxford University Press, 2022: 307-18.
  46. ↵
    1. Rutter H,
    2. Savona N,
    3. Glonti K,
    4. et al
    . The need for a complex systems model of evidence for public health. Lancet2017;390:2602-4. doi:10.1016/S0140-6736(17)31267-9. pmid:28622953
    OpenUrlCrossRefPubMed
  47. ↵
    1. Shaw SE,
    2. Paparini S,
    3. Murdoch J,
    4. et al
    . TRIPLE C reporting principles for case study evaluations of the role of context in complex interventions. BMC Med Res Methodol2023;23:115. doi:10.1186/s12874-023-01888-7 pmid:37179308
    OpenUrlCrossRefPubMed
  48. ↵
    1. Vandenbroeck P,
    2. Goossens J,
    3. Clemens M
    . Foresight: tackling obesities: future choices-building the obesity system map.Government Office for Science, 2007doi:10.1037/e602972011-001.
    OpenUrlCrossRef
  49. ↵
    1. Kiekens A,
    2. Dierckx de Casterlé B,
    3. Vandamme A-M
    . Qualitative systems mapping for complex public health problems: a practical guide. PLoS One2022;17:e0264463. doi:10.1371/journal.pone.0264463 pmid:35213648
    OpenUrlCrossRefPubMed
  50. ↵
    1. Homer JB,
    2. Hirsch GB
    . System dynamics modeling for public health: background and opportunities. Am J Public Health2006;96:452-8. doi:10.2105/AJPH.2005.062059. pmid:16449591
    OpenUrlCrossRefPubMedWeb of Science
  51. ↵
    1. Creswell JW,
    2. Clark VLP
    . Designing and conducting mixed methods research.Sage publications, 2011: 53-106.
  52. ↵
    1. Fetters MD,
    2. Curry LA,
    3. Creswell JW
    . Achieving integration in mixed methods designs-principles and practices. Health Serv Res2013;48(6pt2):2134-56. doi:10.1111/1475-6773.12117. pmid:24279835
    OpenUrlCrossRefPubMedWeb of Science
  53. ↵
    1. Bazeley P
    . Integrating analyses in mixed methods research.Sage publications, 2018doi:10.4135/9781526417190.
    OpenUrlCrossRef
  54. ↵
    1. Alvarado M,
    2. Penney TL,
    3. Astbury CC,
    4. Forde H,
    5. White M,
    6. Adams J
    . Making integration foundational in population health intervention research: why we need “Work Package Zero”. Public Health2022;211:1-4. doi:10.1016/j.puhe.2022.06.028 pmid:35985222
    OpenUrlCrossRefPubMed
  55. ↵
    1. Alvarado M,
    2. Marten R,
    3. Garcia L,
    4. Kwamie A,
    5. White M,
    6. Adams J
    . Using systems thinking to generate novel research questions for the evaluation of sugar-sweetened beverage taxation policies. BMJ Glob Health2023;8(Suppl 8):e012060. doi:10.1136/bmjgh-2023-012060. pmid:37813450
    OpenUrlFREE Full Text
  56. ↵
    1. de Vocht F,
    2. Katikireddi SV,
    3. McQuire C,
    4. Tilling K,
    5. Hickman M,
    6. Craig P
    . Conceptualising natural and quasi experiments in public health. BMC Med Res Methodol2021;21:32. doi:10.1186/s12874-021-01224-x pmid:33573595
    OpenUrlCrossRefPubMed
  57. ↵
    1. Hernán MA,
    2. Robins JM
    . Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol2016;183:758-64. doi:10.1093/aje/kwv254. pmid:26994063
    OpenUrlCrossRefPubMed
  58. ↵
    1. Remiro-Azócar A,
    2. Heath A,
    3. Baio G
    . Conflating marginal and conditional treatment effects: Comments on “Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study”. Stat Med2021;40:2753-8. doi:10.1002/sim.8857 pmid:33963582
    OpenUrlCrossRefPubMed
  59. ↵
    1. Craig P,
    2. Barr B,
    3. Baxter AJ,
    4. et al
    . Evaluation of the mental health impacts of Universal Credit: protocol for a mixed methods study. BMJ Open2022;12:e061340. doi:10.1136/bmjopen-2022-061340. pmid:35396318
    OpenUrlAbstract/FREE Full Text
  60. ↵
    1. Cookson R,
    2. Griffin S,
    3. Norheim OF,
    4. Culyer AJ,
    5. Chalkidou K
    . Distributional cost-effectiveness analysis comes of age. Value Health2021;24:118-20. doi:10.1016/j.jval.2020.10.001. pmid:33431145
    OpenUrlCrossRefPubMed
  61. ↵
    1. Deidda M,
    2. Geue C,
    3. Kreif N,
    4. Dundas R,
    5. McIntosh E
    . A framework for conducting economic evaluations alongside natural experiments. Soc Sci Med2019;220:353-61. doi:10.1016/j.socscimed.2018.11.032. pmid:30513485
    OpenUrlCrossRefPubMed
  62. ↵
    1. Breeze PR,
    2. Squires H,
    3. Ennis K,
    4. et al
    . Guidance on the use of complex systems models for economic evaluations of public health interventions. Health Econ2023;32:1603-25. doi:10.1002/hec.4681. pmid:37081811
    OpenUrlCrossRefPubMed
  63. ↵
    1. Moore GF,
    2. Audrey S,
    3. Barker M,
    4. et al
    . Process evaluation of complex interventions: Medical Research Council guidance. BMJ2015;350:h1258. doi:10.1136/bmj.h1258 pmid:25791983
    OpenUrlFREE Full Text
  64. ↵
    1. Wong G,
    2. Westhorp G,
    3. Manzano A,
    4. Greenhalgh J,
    5. Jagosh J,
    6. Greenhalgh T
    . RAMESES II reporting standards for realist evaluations. BMC Med2016;14:96. doi:10.1186/s12916-016-0643-1. pmid:27342217
    OpenUrlCrossRefPubMed
  65. ↵
    1. Hawe P,
    2. Shiell A,
    3. Riley T
    . Theorising interventions as events in systems. Am J Community Psychol2009;43:267-76. doi:10.1007/s10464-009-9229-9. pmid:19390961
    OpenUrlCrossRefPubMedWeb of Science
  66. ↵
    1. Craig P,
    2. Di Ruggiero E,
    3. Frolich KL,
    4. et al
    . Taking account of context in population health intervention research: guidance for producers, users and funders of research.National Institute for Health Research, 2018.
  67. ↵
    1. Vennix JAM
    . Group model-building: tackling messy problems. Syst Dyn Rev1999;15:379-401. doi:10.1002/(SICI)1099-1727(199924)15:4<379::AID-SDR179>3.0.CO;2-E.
    OpenUrlCrossRef
  68. ↵
    1. Ng SW,
    2. Colchero MA,
    3. White M
    . How should we evaluate sweetened beverage tax policies? A review of worldwide experience. BMC Public Health2021;21:1941. doi:10.1186/s12889-021-11984-2. pmid:34702248
    OpenUrlCrossRefPubMed
  69. ↵
    1. Maini R,
    2. Mounier-Jack S,
    3. Borghi J
    . How to and how not to develop a theory of change to evaluate a complex intervention: reflections on an experience in the Democratic Republic of Congo. BMJ Glob Health2018;3:e000617. doi:10.1136/bmjgh-2017-000617. pmid:29515919
    OpenUrlAbstract/FREE Full Text
  70. ↵
    1. Bell R,
    2. Glinianaia SV,
    3. Waal ZV,
    4. et al
    . Evaluation of a complex healthcare intervention to increase smoking cessation in pregnant women: interrupted time series analysis with economic evaluation. Tob Control2018;27:90-8. doi:10.1136/tobaccocontrol-2016-053476. pmid:28202783
    OpenUrlAbstract/FREE Full Text
  71. ↵
    1. Moffatt S,
    2. White M,
    3. Mackintosh J,
    4. Howel D
    . Using quantitative and qualitative data in health services research - what happens when mixed method findings conflict? [ISRCTN61522618]. BMC Health Serv Res2006;6:28. doi:10.1186/1472-6963-6-28. pmid:16524479
    OpenUrlCrossRefPubMed
  72. ↵
    1. Green J,
    2. Hanckel B,
    3. Petticrew M,
    4. Paparini S,
    5. Shaw S
    . Case study research and causal inference. BMC Med Res Methodol2022;22:307. doi:10.1186/s12874-022-01790-8. pmid:36456923
    OpenUrlCrossRefPubMed
  73. ↵
    1. Mahoney J
    . The logic of process tracing tests in the social sciences. Sociol Methods Res2012;41:570-97doi:10.1177/0049124112437709.
    OpenUrlCrossRefWeb of Science
  74. ↵
    1. Robinson WS
    . The logical structure of analytic induction. Case study method: key issues, key texts.Sage Publications, 1951.
  75. ↵
    1. Sterne JA,
    2. Hernán MA,
    3. Reeves BC,
    4. et al
    . ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ2016;355:i4919. doi:10.1136/bmj.i4919. pmid:27733354
    OpenUrlFREE Full Text
  76. ↵
    ROBINS-E Development Group. Risk Of Bias In Non-randomized Studies - of Exposure (ROBINS-E), 2023. Available from: https://www.riskofbias.info/welcome/robins-e-tool
  77. ↵
    Effective Public Health Practice Project. Quality assessment tool for quantitative studies. Hamilton, Canada, 2022. Available from: http://www.ephpp.ca/PDF/Quality%20Assessment%20Tool_2010_2.pdf
  78. ↵
    Cochrane Effective Practice and Organisation of Care (EPOC). Suggested risk of bias criteria for EPOC reviews. EPOC resources for review authors, 2017. Available from: https://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/Resources-for-authors2017/suggested_risk_of_bias_criteria_for_epoc_reviews.pdf
  79. ↵
    1. Noyes J,
    2. Booth A,
    3. Flemming K,
    4. et al
    . Cochrane Qualitative and Implementation Methods Group guidance series-paper 3: methods for assessing methodological limitations, data extraction and synthesis, and confidence in synthesized qualitative findings. J Clin Epidemiol2018;97:49-58. doi:10.1016/j.jclinepi.2017.06.020 pmid:29247700
    OpenUrlCrossRefPubMed
  80. ↵
    Critical Appraisal Skills Programme (CASP). Qualitative Checklist, 2018. Available from: https://casp-uk.net/casp-tools-checklists/qualitative-studies-checklist/
  81. ↵
    McKenzie JE, Brennan SE. Synthesizing and presenting findings using other methods. Cochrane handbook for systematic reviews of interventions 2019:321-47.
  82. ↵
    Noyes J, Booth A, Cargo M, et al. Qualitative evidence. Cochrane handbook for systematic reviews of interventions 2019:525-45.
  83. ↵
    1. Pawson R,
    2. Greenhalgh T,
    3. Harvey G,
    4. Walshe K
    . Realist review—a new method of systematic review designed for complex policy interventions. J Health Serv Res Policy2005;10(Suppl 1):21-34. doi:10.1258/1355819054308530 pmid:16053581
    OpenUrlCrossRefPubMed
  84. ↵
    1. Sandelowski M,
    2. Voils CI,
    3. Barroso J
    . Defining and designing mixed research synthesis studies. Res Sch2006;13:29.pmid:20098638
    OpenUrlPubMed
  85. ↵
    1. Heyvaert M,
    2. Hannes K,
    3. Onghena P
    . Using mixed methods research synthesis for literature reviews: the mixed methods research synthesis approach.Sage Publications, 2016.
  86. ↵
    1. Guyatt G,
    2. Oxman AD,
    3. Akl EA,
    4. et al
    . GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol2011;64:383-94. doi:10.1016/j.jclinepi.2010.04.026 pmid:21195583
    OpenUrlCrossRefPubMed
  87. ↵
    1. Hilton Boon M,
    2. Thomson H,
    3. Shaw B,
    4. et al.,
    5. GRADE Working Group
    . Challenges in applying the GRADE approach in public health guidelines and systematic reviews: a concept article from the GRADE Public Health Group. J Clin Epidemiol2021;135:42-53. doi:10.1016/j.jclinepi.2021.01.001. pmid:33476768
    OpenUrlCrossRefPubMed
Back to top