Reading and conducting instrumental variable studies: guide, glossary, and checklist

Clinical and public health implications

Researchers increasingly use large datasets of electronic medical records, registries, or administrative claims data to provide evidence about the effects of interventions on patient outcomes. An important limitation of these datasets is that while the large sample size allows for very precise results, they frequently have inadequate measures of critical confounders. Confounders are variables that affect the likelihood of receiving the intervention and that also affect the outcome (eg, previous neuropsychiatric diagnoses and the likelihood of being prescribed varenicline rather than nicotine replacement therapy for smoking cessation). Patients rarely receive interventions entirely randomly. Key confounders, such as morbidity and other indications for intervention, are often challenging or impossible to measure with sufficient accuracy from diagnosis or billing codes or are unmeasured or unmeasurable. Thus, matching individuals receiving the intervention with sufficiently comparable controls can be difficult or impossible. As a result, observational analysis of large scale databases could provide unreliable evidence about interventions’ comparative effectiveness and safety. This issue is challenging for clinicians and patients because they need reliable evidence of the causal effects of different interventions to make well informed decisions. Instrumental variables can provide an alternative source of evidence about the effects of different interventions and, while less precise than other approaches, might be less affected by individual level biases such as confounding by indication, where the indications for intervention also affect the likelihood of an outcome.

Why use an instrument?

Most observational methods, such as multivariable adjusted regression or propensity score analysis, assume that it is possible to measure a sufficient set of confounders to account for all differences in the outcome between individuals given the intervention and control, except for those caused by the intervention.1819 However, the correct set of confounders is not always known, and even if they have been identified, measuring and accounting for baseline differences is extremely difficult, which can result in multivariable-adjusted and propensity score analyses having serious biases and providing misleading results.

For example, COX-2 inhibitors were developed to cause fewer gastrointestinal complications than traditional NSAIDs and marketed to patients and physicians accordingly. As a result, patients prescribed these drug treatments typically were at higher risk of gastrointestinal complications at baseline. Thus, in observational datasets, patients prescribed COX-2 tended to have higher rates of gastrointestinal complications than patients prescribed NSAIDs, a difference that was not fully attenuated after adjustment for measured confounders. This result is because the pre-existing differences in the risk of gastrointestinal complications are very challenging to measure sufficiently, especially in electronic medical records, resulting in residual confounding by indication.

Alternatively, patients prescribed nicotine replacement therapy for smoking cessation differ from those prescribed drugs such as varenicline: patients prescribed nicotine replacement therapy tend to be more unwell, be older, and have poorer mental health.20 However, electronic medical records or other datasets often do not record these differences. For example, patients might discuss smoking cessation with their general practitioner when they have preclinical symptoms of heart disease; these symptoms might not be perfectly captured in medical records.

Instrumental variable analysis offers an approach to deal with these problems. It relies on a distinct set of assumptions from other methods, which do not require measuring or knowing all the potential confounders of the intervention and outcome.

What is an instrumental variable?

The following three assumptions define instrumental variables. Firstly, the instrument is associated with the intervention of interest (relevance); secondly, it shares no uncontrolled common cause with the outcome (independence); and thirdly, it only affects the outcome through the intervention (exclusion restriction). Instruments only need to be associated with the likelihood of receiving the intervention; they do not necessarily need to cause it.3 Instrumental variable analyses exploit naturally occurring variation (the instrument) to estimate the impact of the intervention on an outcome. This variation can be due to clinical or policy decisions unrelated to unmeasured confounders. Box 2 defines these assumptions, and figure 2 uses a directed acyclic graph to represent these assumptions. Assessing the plausibility of the assumptions is critical to determining whether a proposed instrumental variable is valid and is discussed in detail below.

Fig 2

Assumptions of multivariable adjusted and instrumental variable studies. Physicians’ prescribing preferences are typically unmeasured; thus, normally, studies use prescriptions issued to the physicians’ previous patients to proxy for their preferences. Multivariable adjustment (MV1) assumes that a sufficient set of confounders can be measured to control for all open paths between the intervention and the outcome. In contrast, instrumental variable analysis assumes that there is an instrument that associates with the intervention (relevance; IV1), has no uncontrolled common cause with the outcome (independence; IV2), and only affects the outcome through the intervention (exclusion restriction; IV3)

• Download figure
• Open in new tab
• Download powerpoint

These assumptions can be defined unconditionally, or more often conditionally, on other important covariates in a dataset; for example, physicians’ prescribing preferences are usually conditioned on a patient’s age. If these assumptions are violated, for example, by residual confounding of the instrument-outcome association, then the results of an instrumental variable analysis can be more biased than other approaches, such as multivariable adjustment and propensity score. Thus, a key challenge for authors and readers of instrumental variable studies is determining whether the assumptions are plausible for the research question.

Types of instruments

Numerous natural experiments have been proposed and assessed as potential instruments. These commonly include physician preference (eg, preference to prescribe one intervention versus another for a given diagnosis), access to intervention (eg, distance to a hospital with specific speciality staff or equipment), or randomisation (eg, in the context of a randomised controlled trial with non-compliance). Examples of these instruments are given below. Other sources of variation have also been used and are covered elsewhere.21

Assessment of instrumental variable assumptions

Directed acyclic graphs provide a convenient and transparent way to depict and explain the assumptions required for an applied instrumental variable analysis.343536 Researchers can adapt the structure used in figure 2 for specific research questions. Studies can then use empirical data to assess whether the three core assumptions for instrumental variables hold.

The first instrumental variable assumption (relevance) states that the instrument must be strongly associated with the likelihood of taking the intervention. The strength of the instrument-intervention association is easily testable. For example, in the study of drug treatments for smoking cessation, we found that physicians who had previously prescribed varenicline were 24 percentage points (95% confidence interval 23 to 25) more likely to prescribe varenicline to their subsequent patients than physicians who had previously prescribed nicotine replacement therapy. However, a difference in treatment rates across instrument values is insufficient to measure instrument strength because it does not reflect the sample size. In a small study of a few hundred patients, even a very large difference in treatment rates across the instrument’s value will provide very little information about the effects of treatment.

In contrast, the first stage partial F statistic of the regression of the intervention on the instrument indicates both the strength of the association and the total sample size. The first stage partial F statistic in an instrumental variable analysis is analogous to the sample size in a randomised controlled trial. Most instrumental variable estimation packages in Stata and R (such as ivreg2 or AER, respectively)3738 will report this F statistic by default. A value above 10 is considered strong and unlikely to lead to weak instrument bias.39 However, an F statistic above 10 does not guarantee that an instrumental variable study will have sufficient statistical power to detect an effect size of interest.

The remaining assumptions are untestable, so they cannot be proven to hold, but they are falsifiable.3140 An assumption is falsifiable if it is possible to use empirical data to disprove it. The independence assumption can be falsified by testing the instrument-covariate associations using covariate balance and bias component plots,4142 or randomisation tests.43 If the instrumental variable assumptions hold, no associations between the instrument and alternative pathways or other covariates that predict the outcome should be detected.44 The exclusion restriction is falsifiable by demonstrating that other variables are affected by the instrument, which also affects the outcome. For example, in a study of angiotensin-converting enzyme (ACE) inhibitors for cardiovascular disease, if physicians who are more likely to prescribe these inhibitors are also more likely to prescribe statins, which also affect cardiovascular disease, the exclusion restriction assumption would be violated.

Another way to falsify the independence and exclusion restriction assumptions is using negative controls to investigate whether the instrument predicts the outcome in subgroups of the population for which the instrument does not affect the likelihood of receiving the intervention. If evidence indicates that the instrument affects the outcome, even in subgroups where the instrument does not affect the likelihood of receiving the intervention, the instrumental variable assumptions are unlikely to be plausible (eg, by using patients who do not have hypertension (eg, children) who were treated for other indications by physicians who preferred ACE inhibitors). Falsification tests are useful indicators of how plausible the assumptions are likely to be, however, and failure to falsify an assumption does not prove it holds. For example, if the instrument was associated with an unmeasured confounder of the intervention and the outcome, this association would not be evident in a covariate plot that only included measured covariates.

A further way to assess the plausibility of assumptions is to investigate any differences (heterogeneity) in the effect sizes implied by different instruments. This approach requires more than one instrument (which, when there are more instruments than interventions, is technically known as being over-identified). If more than one instrument is affecting the likelihood of receiving the intervention (eg, physicians’ preferences and distance to the healthcare facility), the heterogeneity in the effects of the intervention implied by each instrument could indicate violations of the instrumental variable assumptions. Bonet’s instrumental variable inequality tests can also falsify binary interventions’ exclusion restriction and independence assumptions.45

How to generate instrumental variable estimates

Instrumental variables can test whether an intervention affects an outcome and estimate the magnitude of that effect. The simplest estimator is the instrument-outcome association (reduced form; box 1), which can be estimated using regression methods (eg, linear or logistic regression methods). This estimator does not estimate the magnitude of the effect of the intervention on the outcome. However, under the instrumental variable assumptions, it is a valid test of the null hypothesis that the intervention does not affect the outcome. An advantage of this test is that it is simple, requires the fewest and weakest assumptions, and can test for the existence of an effect. A disadvantage is that it does not provide a scale for the effect of the intervention on the outcome, limiting the interpretation of the results. Ideally, we want to know the average effect of the intervention (also known as the average treatment effect), and not just the effect of the instrument. For example, researchers and readers might be more interested in the effect of prescribing varenicline or nicotine replacement therapy (the intervention) on their current patient than the effect of physicians’ previous prescriptions for smoking cessation treatment (the instrument) on smoking cessation rates (the outcome).

Several instrumental variable estimators can estimate the average treatment effect. Some of the most used instrumental variable estimators are covered below. However, these methods were largely developed to estimate average treatment effects for normally distributed instruments, exposures, and outcomes assuming linear mechanisms. Although, in practice, these methods are widely used for binary outcomes or non-linear mechanisms (sometimes with the same or a different name), the interpretation can be difficult and more advanced methods might be required.

If only one instrument is available, then the average effect of the intervention on an outcome can be estimated using instrumental variable estimators, such as the Wald estimator, which is the ratio of the instrument-outcome association divided by the instrument-intervention association. This estimator rescales the instrument-outcome association to the intervention scale and indicates the effect of a unit change in the intervention on the outcome. For example, if patients prescribed smoking cessation treatments by physicians who previously prescribed varenicline were 1 percentage point more likely to cease smoking (the instrument-outcome association) and 10 percentage points more likely to be prescribed varenicline (the instrument-intervention association), then the Wald estimate would be −0.01 ÷ 0.1 = −0.1. This estimate would imply that prescribing varenicline increases the absolute probability of stopping smoking by 10 percentage points.

When a study has one or more instruments available, for example, if a study used physicians’ preferences and distance to healthcare facility as instruments, then the effects of the intervention on the outcome can be estimated using a two-stage least squares estimator. This estimator comprises two regressions or stages. The first stage is a regression of the intervention on the instruments, which can predict the intervention value based on the instrument values. The second stage is a regression of the predicted intervention status on the outcome. The estimated coefficient on the predicted value is the instrumental variable estimate of the effect of the intervention on the outcome. A simulated example and the formulas are provided in the supplementary materials. It is usually essential that both stages of instrumental variable analysis contain the same covariates.46 However, this approach will not account for the estimation error from the first stage and is likely to give incorrect standard errors and confidence intervals. Typically, most analyses use a package such as ivreg2 in Stata or AER or ivreg packages in R,3738 which compute the instrumental variable estimates in one step and integrates the estimation errors from both stages.

Different types of outcomes require different instrumental variable estimators, which rely on logic similar to the two-stage least squares estimator. Commonly used estimators include:

• Continuous outcomes: Mean differences (eg, effects of smoking cessation treatment on body mass index using physicians’ prescribing preferences20) can be estimated using additive structural mean models.2

• Binary outcomes: Causal risk differences, odds ratios, and risk ratios (eg, estimating the effects of coronary bypass surgery on mortality14) can be estimated using additive, logistical, and multiplicative structural mean models and control function approaches.334748

• Survival outcomes: Methods using instrumental variables with survival outcomes, which adopt a similar approach to two-stage least squares, or the control function approach,49 have been developed to allow for covariate and outcome dependent censoring.50 For example, estimating the effects of screening frequency on colorectal cancer diagnoses using international differences in screening policies.51

• Instrumental variable quantile regression: Non-linear effects of the intervention can be estimated using instrumental variable quantile regression.525354 For example, investigating whether the effects of a unit increase in body mass index on healthcare costs differ between underweight and overweight individuals.55

Methods for instrumental variable estimation is an area of active methodological development, spanning statistics, econometrics, and computer science. Examples include estimators combining instrumental variable analysis and matching56 and estimators using machine learning.575859

Data for instrumental variable studies

Instrumental variable studies typically require measures of the instrument, the intervention, and the outcome for individual level data analysis using the same sample of people. This straightforward approach allows the most flexibility to test and evaluate the instrumental variable assumptions. However, integrating additional external datasets can improve the power and precision of instrumental variable analyses using an approach known as two-sample instrumental variable analysis.60 This approach estimates the instrument-intervention association in one sample and the instrument-outcome association in another, from which the Wald estimator can be calculated. For example, a study could estimate the effects of policy reform on educational attainment using census data from the entire population but estimate the effects on health outcomes in a cohort study subsampled from the same underlying population.61 A two-sample instrumental variable analysis does not need measures of the intervention or the outcome in all samples, which can increase power considerably, particularly when the outcome is rare or difficult to measure.

Summary

Instrumental variable analysis can provide reliable evidence about the causal effects of an intervention, even if the intervention-outcome association is affected by unmeasured confounding. Key to conducting and reading instrumental variable studies is assessing the plausibility of the three core assumptions on instrumental variables. Does the instrument strongly associate with the intervention? Is there a rationale for why the instrument-outcome association is less likely to have confounding than the intervention-outcome association? Is there evidence that measured covariates are less strongly associated with the instrument than the intervention? Are alternative pathways available that could mediate the effects of the instrument?

Instrumental variable analysis can provide a valuable complement to other forms of observational analysis. Unlike other approaches, instrumental variables have distinct assumptions and can strengthen inferences when combined with other sources of evidence. The increasing amount of data available for clinical research means that there is a growing opportunity to use these methods to improve patient care.