Intended for healthcare professionals

  1. Research
  2. Mailed feedback to...
  3. Mailed feedback to primary care physicians on antibiotic prescribing for patients aged 65 years and older: pragmatic, factorial randomised controlled trial
CCBYNC Open access
Research

Mailed feedback to primary care physicians on antibiotic prescribing for patients aged 65 years and older: pragmatic, factorial randomised controlled trial

BMJ 2024; 385 doi: https://doi.org/10.1136/bmj-2024-079329 (Published 05 June 2024) Cite this as: BMJ 2024;385:e079329

Linked Editorial

Antimicrobial stewardship

This article has a correction. Please see:

  1. Kevin L Schwartz, assistant professor1 2 3 4,
  2. Jennifer Shuldiner, doctoral student5 6,
  3. Bradley J Langford, assistant professor1 3,
  4. Kevin A Brown, associate professor1 2 3,
  5. Susan E Schultz, senior analyst2,
  6. Valerie Leung, pharmacist1 7,
  7. Nick Daneman, professor1 2 6 8,
  8. Mina Tadrous, assistant professor2 5 9,
  9. Holly O Witteman, associate professor10 11,
  10. Gary Garber, professor12 13,
  11. Jeremy M Grimshaw, professor14 15,
  12. Jerome A Leis, associate professor8 12 ,
  13. Justin Presseau, associate professor14 15,
  14. Michael S Silverman, professor16,
  15. Monica Taljaard, professor14 15,
  16. Tara Gomes, assistant professor2 4 6 9,
  17. Meagan Lacroix, research coordinator5,
  18. Jamie Brehaut, professor14 15,
  19. Kednapa Thavorn, associate professor2 14 15,
  20. Sharon Gushue, senior methodologist17,
  21. Lindsay Friedman, research coordinator1,
  22. Merrick Zwarenstein, professor18,
  23. Noah Ivers, associate professor2 5 19
  1. 1Public Health Ontario, Toronto, ON, Canada
  2. 2ICES, Toronto, ON, Canada
  3. 3Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada
  4. 4Li Ka Shing Knowledge Institute, Unity Health Toronto, Toronto, ON, Canada
  5. 5Women’s College Hospital Institute for Health System Solutions and Virtual Care, Women’s College Hospital, Toronto, ON, Canada
  6. 6Institute of Health Policy, Management, and Evaluation, University of Toronto, Toronto, ON, Canada
  7. 7Michael Garron Hospital, Toronto East Health Network, Toronto, ON, Canada
  8. 8Sunnybrook Research Institute, Toronto, ON, Canada
  9. 9Leslie Dan Faculty of Pharmacy, University of Toronto, Toronto, ON, Canada
  10. 10Department of Family and Emergency Medicine, Laval University, Quebec City, QC, Canada
  11. 11Vitam Research Centre for Sustainable Health, Quebec City, QC, Canada
  12. 12Department of Medicine University of Toronto, Toronto, ON, Canada
  13. 13Department of Medicine, University of Ottawa, Ottawa, ON, Canada
  14. 14Clinical Epidemiology Program, Ottawa Hospital Research Institute, Ottawa, ON, Canada
  15. 15School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada
  16. 16Western University, London, ON, Canada
  17. 17Ontario Health, Toronto, ON, Canada
  18. 18Departments of Family Medicine and Epidemiology/Biostatistics, Schulich School of Medicine and Dentistry, Western University, London, ON, Canada
  19. 19Department of Family and Community Medicine, University of Toronto, Toronto, ON, Canada
  1. Correspondence to: K L Schwartz kevin.schwartz{at}oahpp.ca (@DrKevinSchwartz)
  • Accepted 12 April 2024

Abstract

Objectives To evaluate whether providing family physicians with feedback on their antibiotic prescribing compared with that of their peers reduces antibiotic prescriptions. To also identify effects on antibiotic prescribing from case-mix adjusted feedback reports and messages emphasising antibiotic associated harms.

Design Pragmatic, factorial randomised controlled trial.

Setting Primary care physicians in Ontario, Canada

Participants All primary care physicians were randomly assigned a group if they were eligible and actively prescribing antibiotics to patients 65 years or older. Physicians were excluded if had already volunteered to receive antibiotic prescribing feedback from another agency, or had opted out of the trial.

Intervention A letter was mailed in January 2022 to physicians with peer comparison antibiotic prescribing feedback compared with the control group who did not receive a letter (4:1 allocation). The intervention group was further randomised in a 2x2 factorial trial to evaluate case-mix adjusted versus unadjusted comparators, and emphasis, or not, on harms of antibiotics.

Main outcome measures Antibiotic prescribing rate per 1000 patient visits for patients 65 years or older six months after intervention. Analysis was in the modified intention-to-treat population using Poisson regression.

Results 5046 physicians were included and analysed: 1005 in control group and 4041 in intervention group (1016 case-mix adjusted data and harms messaging, 1006 with case-mix adjusted data and no harms messaging, 1006 unadjusted data and harms messaging, and 1013 unadjusted data and no harms messaging). At six months, mean antibiotic prescribing rate was 59.4 (standard deviation 42.0) in the control group and 56.0 (39.2) in the intervention group (relative rate 0.95 (95% confidence interval 0.94 to 0.96). Unnecessary antibiotic prescribing (0.89 (0.86 to 0.92)), prolonged duration prescriptions defined as more than seven days (0.85 (0.83 to 0.87)), and broad spectrum prescribing (0.94 (0.92 to 0.95)) were also significantly lower in the intervention group compared with the control group. Results were consistent at 12 months post intervention. No significant effect was seen for including emphasis on harms messaging. A small increase in antibiotic prescribing with case-mix adjusted reports was noted (1.01 (1.00 to 1.03)).

Conclusions Peer comparison audit and feedback letters significantly reduced overall antibiotic prescribing with no benefit of case-mix adjustment or harms messaging. Antibiotic prescribing audit and feedback is a scalable and effective intervention and should be a routine quality improvement initiative in primary care.

Trial registration ClinicalTrials.gov NCT04594200

Introduction

Antimicrobial resistance is a rising global public health crisis with an estimated 1.27 million attributable deaths per year worldwide.1 Overuse and misuse of antibiotics are important modifiable drivers of rising drug resistant infections. Most antibiotics are prescribed by primary care physicians.

Peer comparison audit and feedback on antibiotic prescribing is a potentially scalable and effective intervention. Effective audit and feedback incorporates behavioural science principles to drive behaviour change and improve the quality of patient care. Previously, a three arm trial of an antibiotic feedback letter mailed to the highest antibiotic prescribing physicians in Ontario, Canada, led to a 4% to 5% reduction in antibiotic prescriptions.2 The effect size of other trials evaluating antibiotic audit and feedback to reduce antibiotic prescribing resulted in variable effect sizes.34567 Various unanswered questions remain regarding the optimal design of audit and feedback, with relatively few trials comparing different audit and feedback designs in an attempt to improve the effect of these interventions.89 Results from a trial in Australia showed that feedback including peer comparison and a graph performed best and reduced antibiotic prescription rates by 12%.10 A trial from Scotland investigated the effect of behavioural change messaging and showed no effect of sending three compared with two reports over the year.11

The drivers of inappropriate antibiotic prescribing are complex and multifactorial. A knowledge gap is generally not the primary driver of over-prescribing, whereas, physician habit, perceived patient expectations, and fear of consequences, predominate as themes in the medical literature. Physicians tend to overestimate the potential benefits of antibiotics and underestimate the potential harms. Another common theme in qualitative studies of audit and feedback is that physicians articulate that the data do not reflect their unique practice and patient characteristics.12131415

Our primary objective was to evaluate whether providing family physicians with feedback on their antibiotic prescribing, using routinely collected data, compared with their peers, reduces antibiotic use. Our secondary objectives were to test whether further incremental reductions could be made in antibiotic prescribing by providing two more forms of feedback. Firstly, we assessed case-mix adjustment in feedback reports, to address physicians’ perception that the data do not reflect their unique patient populations. Secondly, we assessed emphasising harms associated with antibiotics, to address physicians’ tendency to underestimate antibiotic harms. We hypothesised that there would be no interaction between these modifications and therefore chose a factorial design to efficiently evaluate our secondary objectives.

Methods

Design

We conducted a pragmatic randomised controlled trial of mailed antibiotic audit and feedback reports to primary care physicians with an embedded 2x2 factorial experiment in Ontario, Canada.16 The trial was considered pragmatic because the design was created to directly inform policy and decision making.17 We evaluated the effectiveness of our audit and feedback intervention under usual conditions across the province. Eligible physicians and outcomes were identified through existing, routinely collected administrative data. The study protocol has been previously published and registered (NCT04594200).16

Participants and setting

Ontario is Canada’s most populous province (population 14.5 million in 2022). Residents can see a physician without incurring out of pocket costs for the visit. However, medications are only publicly funded for some patients, including all patients 65 years and older. Our data only includes publicly funded medications; therefore, this study was limited to community dwelling patients aged 65 years and older. We have previously shown a strong correlation between physician prescribing for patients of this age and their overall antibiotic prescribing (Spearman’s r for men were 0.80 and women were 0.84).16

Ontario Health is the government agency that oversees administration of the provincial healthcare system. As part of their quality improvement activities, primary care physicians can voluntarily sign up for MyPractice reports, which, since 2021, have included antibiotic prescribing indicators. At the time of this trial, approximately 4000 primary care physicians received these reports. To avoid duplication, physicians who had previously signed up for MyPractice reports were excluded from this trial. We also excluded physicians who had opted out of a previous feedback trial in Ontario,2 physicians with fewer than 100 unique patient visits to patients 65 years or older in the most recent year, or in two of the three prior years, and physicians with fewer than 10 antibiotic prescriptions to patients 65 years or older in the most recent year, or two of the three prior years. Data from these physicians would not have been sufficient for meaningful feedback. Finally, one month before the trial, an introductory letter was sent to all eligible physicians with the opportunity to opt out.

Interventions

We developed the intervention audit and feedback reports using an iterative process. Team members with expertise in audit and feedback, design science, behavioural science, trial design, patient care, and antibiotic prescribing met to develop prototype reports. Prototype reports were then informally reviewed by physician colleagues who belonged to the target audience. Physicians' comments and reactions were brought back to the group and informed iterative revisions of the reports.

Physicians in the control group were not notified that they were in a study, aside from the opt-out letter, and did not receive any audit and feedback on antibiotics. All eligible physicians in the intervention group were randomly assigned to receive case-mix adjusted feedback, harms messaging, neither, or both. Physicians in the intervention group received a mailed letter to their primary office location in January 2022. The same letter was sent out again in February 2022 in an attempt to increase engagement with the intervention. The letters included three years of data (from 1 March 2018 to 28 February 2021) on total antibiotic prescribing to their patients aged 65 years or older. The letter also provided the median antibiotic prescribing rate and the lowest quartile met by other primary care physicians in Ontario, which was described in the report as an achievable target. Our previous work identified that the average primary care physician could safely reduce their antibiotic prescribing by at least 24%.18 Antibiotic audit and feedback interventions are discouraged from using the mean as the sole comparator to avoid regression to the mean.19 We also provided physicians with their data for the proportion of their antibiotic prescriptions that were more than seven days and a table of recommended antibiotic durations for common infections. These elements were based on the success of a previous trial.2 The report included a graph, education on appropriate antibiotic prescribing and durations, evidence informed communication strategies as well as tools from Choosing Wisely Canada to help to improve antibiotic prescribing. The letter was co-signed by Ontario’s Chief Medical Officer of health, the president of the Ontario College of Family Physicians, and the chair of Choosing Wisely Canada (supplementary appendix 1).

We embedded a 2x2 factorial trial in the intervention arm to efficiently evaluate the independent effects of each factor in the absence of any hypothesised interaction. For physicians randomly assigned to the case-mix adjusted letter, we standardised their antibiotic prescribing rate using hierarchical regression modelling, which incorporated their number of patient visits per year, as well as patient age, sex, socioeconomic status, comorbidities, and practice setting. On the letter’s first page, it was emphasised to physicians that their data were adjusted to represent a fair comparison to physicians with similar patients and practice characteristics. Physicians not in the adjusted group received feedback on their raw antibiotic prescribing rate compared with that of their peers. We anticipated that this modification would address physician’s lack of acceptance of audit and feedback that did not adequately capture their patient and practice complexity.15 For physicians randomly assigned to receive harms messaging, we included an infographic highlighting the frequency of side effects and harms associated with antibiotics. This infographic highlighted the 30% risk of side effects from antibiotic use, the doubling of bacterial resistance rates, and predicted rising mortality from drug resistant infections in the future (supplementary appendix 1). The physicians who were randomly assigned to the non-harms group only received an infographic on the lack of benefits from unnecessary antibiotic prescribing. We anticipated that this modification would address the perceived imbalance of risks from unnecessary antibiotic prescribing.13 The various versions of the intervention are available in supplementary appendix 1.16

Allocation

An epidemiologist who was not otherwise involved in the study generated the allocation sequence. Randomisation was a simple random sampling method without replacement, with no block size specification or clustering. Randomisation was done simultaneously for all eligible physicians, stratified by participation in a previous trial from Ontario in 2018, which was limited to high antibiotic prescribers only.2 Physicians were randomly assigned 4:1 (intervention:control), and 1:1:1:1 within the intervention arm to each of the two factors described above (adjusted or unadjusted and harms or no harms), giving four experimental conditions.

Outcomes and data sources

The primary outcome was antibiotic prescribing rate defined as the total number of systemic oral antibiotic prescriptions per 1000 patient visits in patients aged 65 years or older from the time of mailing the intervention to six months after intervention. The antibiotic prescribing rate was selected as the primary outcome because accurate measurement was possible, the rate is known to drive antimicrobial resistance, and over prescribing of antibiotics is present in this population; therefore, an opportunity exists to safely reduce overall antibiotic use.1820 Secondary outcomes included the number of likely unnecessary antibiotic prescriptions per 1000 patient visits (table S1), the number of antibiotic prescriptions of more than seven days per 1000 patient visits, and the number of broad-spectrum antibiotic prescriptions per 1000 patients visits (table S2). Unnecessary antibiotic prescriptions were defined as having a physician International Classification of Diseases 9th edition billing claim in the Ontario Health Insurance Plan database for one or more codes for a condition that rarely or never requires antibiotics (eg, asthma, common cold, and bronchitis). This list was derived from previous research (table S1).18 We initially planned to include total antibiotic days of treatment as an outcome, however, since this metric is a combination of antibiotic prescribing rate and duration we omitted it because we felt no new information would be gained. In a secondary analysis, outcomes were measured from letter mailing to 12 months of follow-up. The data for all outcomes were derived from routinely collected administrative data at ICES (formerly, the Institute for Clinical Evaluative Sciences), an independent, non-profit research institute whose legal status under Ontario’s health information privacy law allows it to collect and analyse health care and demographic data without consent for health system evaluation and improvement. Antibiotic prescription data were from the Ontario Drug Benefit database which is more than 99% accurate.21 Additional ICES databases used include the Ontario Health Insurance Plan database to identify physician visits, the Registered Person Database to identify patient demographics, the Canadian Institutes for Health Information Discharge Abstract Database to identify patient admission to hospital and comorbidities, and the ICES Physician Database to identify prescriber characteristics. These datasets were linked using unique encoded identifiers and analysed at ICES. We evaluated the intervention fidelity through phone calls to a random sample of 3% (ie, 135) of physicians in the intervention group (up to two phone call attempts per physician) after February 2023 when data collection ended.

Sample size

We anticipated that we would have data for approximately 6000 eligible physicians and an average of 784 patient visits per physician over six months. Assuming in the control arm an antibiotic prescribing rate of 40 per 1000 patient visits and a between-cluster coefficient of variation of 75%, we would meet at least 80% power to detect a 7.5% relative reduction in the antibiotic prescribing rate comparing the intervention to usual care in a 4:1 allocation, using a two sided test for the difference between two Poisson rates.

Statistical analysis

We conducted a modified intention-to-treat analysis. We excluded outliers at the 99th percentile for antibiotic prescribing rate at baseline after randomisation from each arm to eliminate data errors from implausibly high numbers of antibiotic prescriptions attributed to a small number of physicians.25 We analysed the data at the level of the physician using Poisson regression. The dependent variable was the number of antibiotic prescriptions and the model was offset by the log of the physician’s total patient visits. To improve power and efficiency the models were further adjusted for the following prespecified covariates: the log of the baseline antibiotic prescribing rate (15 January 2021 to 14 January 2022), physician’s sex, years since medical school graduation, and our stratification variable of whether the physician was enrolled in a previous Ontario trial. A second exploratory model included terms for the presence or absence of each factor (case-mix adjusted data or harms messaging).22 We hypothesised a priori that no interaction would be shown between factors and to avoid an increased risk of type I error, we did not perform a two stage analysis evaluating for interaction effects.23 We conducted an at-the-margins analysis by estimating the effects of each factor compared with physicians not receiving that factor. We added a post-hoc analysis that included an interaction term between factors to test our hypothesis of no multiplicative interaction. We then added a post-hoc inside-the-table sensitivity analysis of the embedded trial as four separate groups. Intervention effects were expressed using relative risks and 95% confidence intervals (CIs). Prespecified subgroup analyses of the primary outcome were conducted stratified on physician years in practice (<11 years, 11-24 years, or ≥25 years), sex, neighbourhood income quintile of practice, tertiles of the number of patient visits, tertiles of proportions of rostered patient visits (the number of patient visits for the physicians’ own patients divided by all patient visits), tertiles of the proportion of patient visits of people aged greater than 85 years, rural versus urban practice location, tertiles of baseline antibiotic prescribing rates, and tertiles of baseline antibiotic prescribing rates from virtual patient visits. The significance of subgroup differences was assessed by including the interaction between the subgroup variable and the treatment effect in the statistical model. We did not account for multiple hypothesis testing because these subgroup analyses were considered exploratory. The data were analysed using SAS Enterprise Guide 9.4.

Patient and public involvement

This trial was funded through the Canadian Institutes for Health Research’s strategy for patient oriented research. Patient representatives were engaged at the grant development stage and patient insights were solicited on the overall study and intervention design (as described in our protocol for this trial).16 Patients were involved in providing feedback during the iterative process of designing the feedback reports. For example, patient representatives helped to refine the language used in the section of the feedback reports that emphasised communication tips that physicians could use to explain to patients why antibiotics are not recommended. The experiences of our patients engaged through this project were previously detailed in a podcast.24

Results

We identified 15 438 physicians within our administrative datasets with the specialty of general practice or family medicine. After excluding physicians who were inactive, with low numbers of visits or antibiotic prescriptions to the patient population, and those already signed up for feedback from Ontario Health, 6466 physicians were eligible. We were unable to include 1016 of these due to missing identification numbers or address information. An opt out introductory letter was mailed to 5450 physicians and 353 (6.5%) opted out of the trial or had an invalid address. 5097 physicians were randomly assigned to a group and 4076 assigned to the intervention group were mailed the intervention. After excluding 1% of outliers based on baseline antibiotic prescribing rate, data for 5046 physicians were analysed (fig 1).

Fig 1
Fig 1

Consort diagram of study inclusion and exclusion criteria

Overall, the physician characteristics and baseline prescribing practices in each group were well balance (table 1). The average physician’s age was 50.6 years in the intervention group and 51.2 years in the control group. More than half of physicians were male and almost half were in practice for 25 years or more. The average baseline prescribing rate by antibiotic prescriptions per 1000 patient visits was 63.5 (standard deviation 36.3) in the intervention group and 64.6 (38.9), in the control group.

Table 1

Baseline characteristics of included physicians and patients 65 years of age and older from 15 January 2021 to 14 January 2022

View this table:

The antibiotic prescribing rate was lower in the intervention group compared with the control group at 0-6 months and 0-12 months post intervention (fig 2). After six months of follow-up, the mean antibiotic prescribing rate of the intervention group compared with the control group was significantly lower (56.0 (39.2) v 59.4 (42.0) per 1000 patient visits; relative rate 0.95 (95% CI 0.94 to 0.96))(table 2). Mean patient visits per physician during the six month intervention period were 612.7 (527.4) for the intervention group and 626.6 (547.4) for the control group. The intervention also was significantly lower for antibiotic prescribing that was likely unnecessary (ie, for viral illnesses) (7.5 v 8.6 per 1000 patient visits, 0.89 (0.86 to 0.92)), antibiotic prescribing rate for long duration prescriptions (13.7 v 16.5, 0.85 (0.83 to 0.87)), and antibiotic prescribing rate for broad-spectrum prescriptions (26.0 v 28.4, 0.94 (0.92 to 0.95)). We observed similar results when including antibiotic prescriptions dispensed between baseline and 12 months of follow-up (0.96 (0.95 to 0.97)) (table S3).

Fig 2
Fig 2

Monthly antibiotic prescribing rates per 1000 patient visits, by intervention group, January 2019 to January 2023

Table 2

Comparison of outcomes of antibiotic prescribing rates at six months for the primary analysis and factorial trial. Values are mean (SD) unless stated otherwise

View this table:

In our prespecified subgroup analyses, no significant differences were reported in the intervention effectiveness among subgroups of physician’s sex, neighbourhood income group, patient visit volume, rural versus urban practice setting, and antibiotic prescribing for virtual visits (fig 3 and table S4). Physicians with high baseline antibiotic prescribing had significantly larger reductions in antibiotic prescribing effect sizes compared with lower antibiotic prescribing physicians (pinteraction=0.018). In the sensitivity analysis that stratified the cohort by whether the patient was rostered to the prescribing physician, we observed similar results in both groups (table S5). We broke down the analysis by quarter post-hoc. The intervention effect in quarter one for relative risk was 0.94 (95% CI 0.93 to 0.96) and in quarter four was 0.98 (0.96 to 0.99) (table S6).

Fig 3
Fig 3

Forest plot of stratified results for antibiotic prescribing rate at six months. Abx=antibiotic

In the embedded factorial trial, our primary analysis assuming no interaction found that physicians receiving adjusted data had a 1% relative increase in antibiotic prescribing (relative risk 1.01 (95% CI 1.00 to 1.03)) (table 2) but those receiving harms messaging had similar outcomes to those who did not (1.00 (0.99 to 1.01)). A significant interaction was noted between factors in the factorial trial (P<0.001). The results from the sensitivity analysis as a four arm trial are presented in table S7. The results were consistent with our primary finding of no significant decreases in antibiotic prescribing outcomes for either intervention (table S7).

Of 135 randomly sampled physicians in the intervention group that we called after sending the debrief letter, 76 (56%) could not be reached, 18 (13%) confirmed receipt of the feedback letter, and 41 (30%) either did not receive or were unsure if they received the intervention.

Discussion

Principal findings

In this large, pragmatic randomised controlled trial with more than 5000 primary care physicians, a mailed letter to physicians led to a significant 5% relative reduction in overall antibiotic prescribing rate compared with physicians in the control group. We observed improvements in antibiotic prescribing on all outcomes evaluated including an 11% relative reduction in unnecessary antibiotic prescriptions, 15% relative reduction in antibiotic durations more than seven days, and a 6% relative reduction on broad-spectrum antibiotic prescribing.

Comparison with other studies

These results are consistent with a previous trial from Ontario,2 as well as most other audit and feedback interventions on antibiotic prescribing in primary care.4510112526 Some audit and feedback interventions in Europe have been ineffective with no significant change in antibiotic prescribing.367 Two main explanations are likely for this discrepancy. Firstly, some jurisdictions with negative studies had significantly lower baseline antibiotic prescribing rates, suggesting less opportunity for change.27 The second reason is likely related to the design of the intervention. A theory put forward by Linder and Fox is that feedback designed to be more subtle and avoid offending physicians may be too vague to elicit the desired behaviour change.28 Best practice recommendations have been published on optimizing audit and feedback of antibiotics in primary care.29 Antibiotic audit and feedback should be simple and include a single central figure.10 Prescribers should be able to understand the data within seconds and connect the data directly to a desired action. Data that are too subtle, complex, provide multiple metrics or multiple comparators are less likely to be used by physicians and will not drive behaviour change.282930

Our attempts to maximize the effects of audit and feedback through design modifications did not lead to measurable differences in the embedded factorial trial. A common concern of audit and feedback recipients is that the data do not adequately reflect their specific patient and practice characteristics.1215 Our attempt to address this common concern through case-mix adjustment did not reduce antibiotic prescribing and paradoxically may have led to small increases in antibiotic use. Providing more complex data could possibly have led to lower engagement with the intervention and less behaviour change. Another possible explanation is that including the case-mix adjustment paradoxically reinforced the cognitive bias that their patients’ may be different, which led to increased prescribing. These results should be interpreted cautiously as the effect sizes were very small and of uncertain clinical significance.

Regarding harms information, data show that physicians frequently perceive an imbalance in risks by underestimating the harms of unnecessary antibiotic prescribing and overestimating the risks of not prescribing antibiotics.1231 A previous trial emphasising harms related to opioid prescribing led to significant reductions in opioid use.32 However, despite additional emphasis on antibiotic harms, further improvements in antibiotic prescribing were not observed in our study. A possible explanation for this finding is that our infographic did not adequately address this risk perception to promote behaviour change beyond the feedback of data and other messaging within the letter. A theory informed process will explore the possible reasons that behaviour did not change and will be published separately. We observed an unanticipated statistically significant interaction effect in the factorial trial between case-mix adjusted feedback and harms messaging emphasis. The clinical significance of this observation is unclear and post-hoc analyses were similar; however, results from the factorial analysis should be interpreted cautiously.

Policy implications

Based on this work, as well as other published literature, we believe antibiotic audit and feedback should be a required routine quality improvement initiative in primary care funded by governments or insurers responsible for funding healthcare systems. All prescribers can benefit from access to their antibiotic prescribing data, if implemented thoughtfully using audit and feedback best practice recommendations to optimize patient care.29 Our stratified analyses support that audit and feedback is effective across different physician groups (eg, age and sex), and practice types (eg, urgent care centres where physicians tend to see non-rostered patients as well as those with high and low baseline prescribing levels of rostered patients). Encouragingly, this intervention also appeared effective across neighbourhood income levels, supporting the role of audit and feedback to improve equitable access to appropriate antibiotic use.

Strengths and limitations of this study

Our observed effect size was modest given the limited engagement from clinicians in reviewing the feedback reports. In a previous study from Ontario, approximately a third of physicians opened the letter.2 In this study, we sent the same letter twice, one month apart, in an attempt to increase engagement. Unfortunately, we were unable to engage more than one third of mailed recipients. Previous studies that have increased the number of mailings, or modified introductory emails, have had no or minimal impact.1133 However, our significant findings were observed despite this low engagement with the intervention. Further research on ways to improve engagement with feedback reports should be a priority because this change may result in larger effect sizes.

Further research is needed into implementing co-interventions such as public education, communication interventions, point-of-care testing, and other quality improvement initiatives implemented and evaluated alongside audit and feedback to further reduce unnecessary antibiotic prescribing in primary care. A previous trial demonstrated the effectiveness of a simple table on antibiotic durations to reduce the duration of antibiotic prescriptions.2 Prolonged antibiotic durations are likely very amenable to change because these are a knowledge gap for some physicians.34 However, the reasons driving antibiotic initiation for respiratory tract infections are substantially more complex,12 and will require more complex solutions through engagement of multiple stakeholders and prescribers. Future studies should focus on assessing the cost-effectiveness of antibiotic audit and feedback interventions in primary care. This research will aid in reducing unnecessary antibiotic use and improve resource allocation in publicly funded healthcare systems.

This trial has some notable limitations. The intervention was implemented during the covid-19 pandemic with substantial changes in the burden and timing of respiratory illnesses and healthcare seeking behaviours. The intervention was delivered soon after the early winter peak in antibiotic prescribing, which subsequently increased during the intervention, likely due to the resurgence of respiratory infections following a large decrease during the covid-19 pandemic.3536The generalizability to a post-pandemic setting is uncertain. However, our findings are consistent with several studies from different times and jurisdictions. We excluded primary care physicians who have already volunteered for electronic prescribing feedback (which includes antibiotic indicators) making these results particularly generalizable to primary care physicians not already engaged in audit and feedback. The pragmatic nature of our trial make it generalizable to other countries with primary care systems that, like Canada’s, offer care that is free at point of access. We chose not to cluster randomise by practice and contamination was possible across study arms. However, physicians in the control group for the primary analysis had no access to their data and therefore any contamination would have biased the results towards the null. Physicians within one practice may have been randomised to different factors with the potential for contamination. Our study was not specifically powered for the factorial analysis and the results from this embedded trial should be considered exploratory and interpreted cautiously. Our primary outcome was overall antibiotic prescribing rate because we were unable to accurately determine the appropriateness of all prescriptions with administrative data. Previous research from Ontario identified that approximately 25% of antibiotic prescriptions written by primary care physicians are unnecessary, suggesting substantial opportunity for safe reduction in use.18 It is less likely that necessary antibiotic would be withheld and our secondary analysis supports that the observed reduction in use was likely driven by less unnecessary antibiotic use. A previous study did not identify any harms with a 12% reduction in antibiotic use in primary care.37 Our data were limited to patients 65 years or older and may not apply to all patient populations. However, we have previously showed strong correlations between antibiotic prescribing by family physicians in Ontario to patients 65 years or older and overall prescribing.16 Finally, the results of previous literature, including this study, are largely limited to high income countries and further research in low and middle income countries on antibiotic audit and feedback is needed.

Conclusions

Mailed antibiotic audit and feedback letters led to significant reductions in antibiotic prescribing by primary care physicians. This intervention is scalable to large populations of prescribers with the potential to reduce inappropriate antibiotic use, improve the quality of patient care, and slow the emergence of drug resistant infections. Antibiotic prescribing feedback should be a routine quality improvement expectation in primary care.

What is already known on this topic

  • Antibiotic audit and feedback in primary care can be effective, however, previous literature has produced variable effect sizes

  • Which design components of a feedback intervention can improve the effectiveness of this intervention is largely unknown

What this study adds

  • A mailed antibiotic feedback letter with peer comparison is effective at reducing antibiotic prescribing by primary care physicians

  • Including adjusted metrics for the feedback, or antibiotic harms messaging, did not result in further reductions in antibiotic prescribing

  • Antibiotic audit and feedback reports should be a routine quality improvement initiative for all primary care physicians

Ethics statements

Ethical approval

This study received ethics board approvals from Women’s College Hospital Research Institute (REB# 2020-0024-E) and Public Health Ontario (REB# 2020-026.03). Both approved a waiver of consent and an opt-out option for participation.

Data availability statement

The dataset from this study is held securely in coded form at ICES. While legal data sharing agreements between ICES and data providers (eg, healthcare organizations and government) prohibit ICES from making the dataset publicly available, access may be granted to those who meet pre-specified criteria for confidential access, available at www.ices.on.ca/DAS (email: das@ices.on.ca). The full dataset creation plan and underlying analytic code are available from the authors upon request, understanding that the computer programs may rely upon coding templates or macros that are unique to ICES and are therefore either inaccessible or may require modification.

Acknowledgments

This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health and the Ministry of Long term Care. This study also received funding from the Canadian Institutes for Health Research’s Strategy for Patient Oriented Research innovative clinical trials grant (#MYG-398514). This document used data adapted from the Statistics Canada Postal CodeOM Conversion File, which is based on data licensed from Canada Post Corporation, and/or data adapted from the Ontario Ministry of Health Postal Code Conversion File, which contains data copied under license from Canada Post Corporation and Statistics Canada. Parts of this material are based on data and information compiled and provided by the Ontario Ministry of Health. The analyses, conclusions, opinions, and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred. Parts of this material are based on data and/or information compiled and provided by Canadian Institutes for Health Information. However, the analyses, conclusions, opinions, and statements expressed in the material are those of the author(s), and not necessarily those of Canadian Institutes for Health Information. We thank IQVIA Solutions Canada Inc for use of their drug information file.

Footnotes

  • Contributors: All authors contributed to the study conception and design. SES performed the data analysis. KLS drafted the manuscript. All authors reviewed the manuscript and provided critical edits. All authors approved the final version. KLS and NI act as guarantors of the work. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

  • Funding: This study was funded by the Canadian Institutes for Health Research (#MYG-398514). The funder had no role in the study design, data collection, analysis, interpretation, writing, or decision for submission. The authors had full access to the data and take responsibility for the data accuracy and integrity.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: funding by the Canadian Institute for Health Research; JAL reports financial compensation for expert testimony by the Ontario Hospital Association, Ontario Ministry of the Attorney General, Seneca College and Choosing Wisely Canada. TG reports financial compensation from the Ontario Ministry of Health, Ontario College of Pharmacists, the Canadian Agency for Drugs and Technologies in Health, and Indigenous Services Canada. All other authors declare no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

  • Transparency: The lead author, KLS, affirms that the manuscript is an honest, accurate, and transparent account of the study being reported with no important omissions.

  • Dissemination to participants and related patient and public communities: All participants received a debrief letter in the mail about the study and with updated antibiotic feedback data. Our plans to disseminate results include an intention to produce an infographic, lay summary, social media posts, and video abstract which will be developed with the support of our lead patient partner.

  • Provenance and peer review: Not commissioned; externally peer reviewed.

http://creativecommons.org/licenses/by-nc/4.0/

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

References

  1. ↵
    1. Murray CJL,
    2. Ikuta KS,
    3. Sharara F,
    4. et al.,
    5. Antimicrobial Resistance Collaborators
    . Global burden of bacterial antimicrobial resistance in 2019: a systematic analysis. Lancet2022;399:629-55. doi:10.1016/S0140-6736(21)02724-0 pmid:35065702
    OpenUrlCrossRefPubMed
  2. ↵
    1. Schwartz KL,
    2. Ivers N,
    3. Langford BJ,
    4. et al
    . Effect of antibiotic-prescribing feedback to high-volume primary care physicians on number of antibiotic prescriptions: a randomized clinical trial. JAMA Intern Med2021;181:1165-73. doi:10.1001/jamainternmed.2021.2790 pmid:34228086
    OpenUrlCrossRefPubMed
  3. ↵
    1. Aghlmandi S,
    2. Halbeisen FS,
    3. Saccilotto R,
    4. et al
    . Effect of antibiotic prescription audit and feedback on antibiotic prescribing in primary care: a randomized clinical trial. JAMA Intern Med2023;183:213-20. doi:10.1001/jamainternmed.2022.6529. pmid:36745412
    OpenUrlCrossRefPubMed
  4. ↵
    1. Meeker D,
    2. Linder JA,
    3. Fox CR,
    4. et al
    . Effect of behavioral interventions on inappropriate antibiotic prescribing among primary care practices: a randomized clinical trial. JAMA2016;315:562-70. doi:10.1001/jama.2016.0275 pmid:26864410
    OpenUrlCrossRefPubMed
  5. ↵
    1. Hallsworth M,
    2. Chadborn T,
    3. Sallis A,
    4. et al
    . Provision of social norm feedback to high prescribers of antibiotics in general practice: a pragmatic national randomised controlled trial. Lancet2016;387:1743-52. doi:10.1016/S0140-6736(16)00215-4 pmid:26898856
    OpenUrlCrossRefPubMed
  6. ↵
    1. Hemkens LG,
    2. Saccilotto R,
    3. Reyes SL,
    4. et al
    . Personalized prescription feedback using routinely collected data to reduce antibiotic use in primary care: a randomized clinical trial. JAMA Intern Med2017;177:176-83. doi:10.1001/jamainternmed.2016.8040 pmid:28027333
    OpenUrlCrossRefPubMed
  7. ↵
    1. Marwick CA,
    2. Hossain A,
    3. Nogueira R,
    4. et al
    . Feedback of Antibiotic Prescribing in Primary Care (FAPPC) trial: results of a real-world cluster randomized controlled trial in Scotland, UK. J Antimicrob Chemother2022;77:3291-300. doi:10.1093/jac/dkac317 pmid:36172861
    OpenUrlCrossRefPubMed
  8. ↵
    1. Ivers NM,
    2. Grimshaw JM,
    3. Jamtvedt G,
    4. et al
    . Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med2014;29:1534-41. doi:10.1007/s11606-014-2913-y pmid:24965281
    OpenUrlCrossRefPubMed
  9. ↵
    1. Ivers N,
    2. Jamtvedt G,
    3. Flottorp S,
    4. et al
    . Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev2012;6:CD000259. doi:10.1002/14651858.CD000259.pub3 pmid:22696318
    OpenUrlCrossRefPubMed
  10. ↵
    1. Australian Government
    . Nudge vs Superbugs: a behavioural economics trial to reduce the overprescribing of antibiotics. 2018https://behaviouraleconomics.pmc.gov.au/projects/nudge-vs-superbugs-behavioural-economics-trial-reduce-overprescribing-antibiotics
  11. ↵
    1. Elouafkaoui P,
    2. Young L,
    3. Newlands R,
    4. et al.,
    5. Translation Research in a Dental Setting (TRiaDS) Research Methodology Group
    . An audit and feedback intervention for reducing antibiotic prescribing in general dental practice: The RAPiD cluster randomised controlled trial. PLoS Med2016;13:e1002115. doi:10.1371/journal.pmed.1002115 pmid:27575599
    OpenUrlCrossRefPubMed
  12. ↵
    1. Simeoni M,
    2. Saragosa M,
    3. Laur C,
    4. Desveaux L,
    5. Schwartz K,
    6. Ivers N
    . Coping with ‘the grey area’ of antibiotic prescribing: a theory-informed qualitative study exploring family physician perspectives on antibiotic prescribing. BMC Prim Care2022;23:188. doi:10.1186/s12875-022-01806-8 pmid:35902821
    OpenUrlCrossRefPubMed
  13. ↵
    1. Mehrotra A,
    2. Linder JA
    . Tipping the balance toward fewer antibiotics. JAMA Intern Med2016;176:1649-50. doi:10.1001/jamainternmed.2016.6254 pmid:27653497
    OpenUrlCrossRefPubMed
  14. ↵
    1. Rose J,
    2. Crosbie M,
    3. Stewart A
    . A qualitative literature review exploring the drivers influencing antibiotic over-prescribing by GPs in primary care and recommendations to reduce unnecessary prescribing. Perspect Public Health2021;141:19-27. doi:10.1177/1757913919879183 pmid:31633458
    OpenUrlCrossRefPubMed
  15. ↵
    1. Ivers N,
    2. Barnsley J,
    3. Upshur R,
    4. et al
    . “My approach to this job is...one person at a time”: Perceived discordance between population-level quality targets and patient-centred care. Can Fam Physician2014;60:258-66.pmid:24627384
    OpenUrlAbstract/FREE Full Text
  16. ↵
    1. Shuldiner J,
    2. Schwartz KL,
    3. Langford BJ,
    4. Ivers NM,
    5. Ontario Healthcare Implementation Laboratory study team
    . Optimizing responsiveness to feedback about antibiotic prescribing in primary care: protocol for two interrelated randomized implementation trials with embedded process evaluations. Implement Sci2022;17:17. doi:10.1186/s13012-022-01194-8 pmid:35164805
    OpenUrlCrossRefPubMed
  17. ↵
    1. Loudon K,
    2. Treweek S,
    3. Sullivan F,
    4. Donnan P,
    5. Thorpe KE,
    6. Zwarenstein M
    . The PRECIS-2 tool: designing trials that are fit for purpose. BMJ2015;350:h2147. doi:10.1136/bmj.h2147 pmid:25956159
    OpenUrlFREE Full Text
  18. ↵
    1. Schwartz KL,
    2. Langford BJ,
    3. Daneman N,
    4. et al
    . Unnecessary antibiotic prescribing in a Canadian primary care setting: a descriptive analysis using routinely collected electronic medical record data. CMAJ Open2020;8:E360-9. doi:10.9778/cmajo.20190175 pmid:32381687
    OpenUrlAbstract/FREE Full Text
  19. ↵
    1. Linder JA
    . Moving the mean with feedback: insights from behavioural science. NPJ Prim Care Respir Med2016;26:16018. doi:10.1038/npjpcrm.2016.18 pmid:27121229
    OpenUrlCrossRefPubMed
  20. ↵
    1. Silverman M,
    2. Povitz M,
    3. Sontrop JM,
    4. et al
    . Antibiotic prescribing for nonbacterial acute upper respiratory infections in elderly persons. Ann Intern Med2017;166:765-74. doi:10.7326/M16-1131 pmid:28492914
    OpenUrlCrossRefPubMed
  21. ↵
    1. Levy AR,
    2. O’Brien BJ,
    3. Sellors C,
    4. Grootendorst P,
    5. Willison D
    . Coding accuracy of administrative drug claims in the Ontario Drug Benefit database. Can J Clin Pharmacol2003;10:67-71.pmid:12879144
    OpenUrlPubMed
  22. ↵
    1. Collins LM,
    2. Dziak JJ,
    3. Kugler KC,
    4. Trail JB
    . Factorial experiments: efficient tools for evaluation of intervention components. Am J Prev Med2014;47:498-504. doi:10.1016/j.amepre.2014.06.021 pmid:25092122
    OpenUrlCrossRefPubMed
  23. ↵
    1. Kahan BC
    . Bias in randomised factorial trials. Stat Med2013;32:4540-9. doi:10.1002/sim.5869 pmid:23733397
    OpenUrlCrossRefPubMed
  24. ↵
    1. Johannesen J,
    2. Angl EN
    . “How did we do?”: A debrief on the role of Lived Experience Advisors in a healthcare research project. https://mattersofengagement.com/how-did-we-do-a-debrief-on-the-role-of-lived-experience-advisors-leas-in-a-healthcare-research-project/
  25. ↵
    1. Chang Y,
    2. Sangthong R,
    3. McNeil EB,
    4. Tang L,
    5. Chongsuvivatwong V
    . Effect of a computer network-based feedback program on antibiotic prescription rates of primary care physicians: A cluster randomized crossover-controlled trial. J Infect Public Health2020;13:1297-303. doi:10.1016/j.jiph.2020.05.027 pmid:32554035
    OpenUrlCrossRefPubMed
  26. ↵
    1. Dutcher L,
    2. Degnan K,
    3. Adu-Gyamfi AB,
    4. et al
    . Improving outpatient antibiotic prescribing for respiratory tract infections in primary care: a stepped-wedge cluster randomized trial. Clin Infect Dis2022;74:947-56. doi:10.1093/cid/ciab602 pmid:34212177
    OpenUrlCrossRefPubMed
  27. ↵
  28. ↵
    1. Linder JA,
    2. Fox CR
    . Simplicity matters-overengineering feedback can be counterproductive. JAMA Intern Med2023;183:220-2. doi:10.1001/jamainternmed.2022.6528 pmid:36877210
    OpenUrlCrossRefPubMed
  29. ↵
    1. Schwartz KL,
    2. Xu AXT,
    3. Alderson S,
    4. et al
    . Best practice guidance for antibiotic audit and feedback interventions in primary care: a modified Delphi study from the Joint Programming Initiative on Antimicrobial resistance: Primary Care Antibiotic Audit and Feedback Network (JPIAMR-PAAN). Antimicrob Resist Infect Control2023;12:72. doi:10.1186/s13756-023-01279-z pmid:37516892
    OpenUrlCrossRefPubMed
  30. ↵
    1. Desveaux L,
    2. Ivers NM,
    3. Devotta K,
    4. Ramji N,
    5. Weyman K,
    6. Kiran T
    . Unpacking the intention to action gap: a qualitative study understanding how physicians engage with audit and feedback. Implement Sci2021;16:19. doi:10.1186/s13012-021-01088-1 pmid:33596946
    OpenUrlCrossRefPubMed
  31. ↵
    1. Zetts RM,
    2. Garcia AM,
    3. Doctor JN,
    4. Gerber JS,
    5. Linder JA,
    6. Hyun DY
    . Primary Care Physicians’ Attitudes and Perceptions Towards Antibiotic Resistance and Antibiotic Stewardship: A National Survey. Open Forum Infect Dis2020;7:ofaa244. doi:10.1093/ofid/ofaa244 pmid:32782909
    OpenUrlCrossRefPubMed
  32. ↵
    1. Doctor JN,
    2. Nguyen A,
    3. Lev R,
    4. et al
    . Opioid prescribing decreases after learning of a patient’s fatal overdose. Science2018;361:588-90. doi:10.1126/science.aat4595 pmid:30093595
    OpenUrlAbstract/FREE Full Text
  33. ↵
    1. Daneman N,
    2. Lee S,
    3. Bai H,
    4. et al
    . Behavioral nudges to improve audit and feedback report opening among antibiotic prescribers: a randomized controlled trial. In: Open forum infectious diseases.Oxford University Press US, 2022. doi:10.1093/ofid/ofac111.
    OpenUrlCrossRef
  34. ↵
    1. Langford BJ,
    2. Quirk J,
    3. Carey S,
    4. Daneman N,
    5. Garber GE
    . Influencing duration of antibiotic therapy: A behavior change analysis in long-term care. Am J Infect Control2019;47:1409-14. doi:10.1016/j.ajic.2019.05.020 pmid:31327576
    OpenUrlCrossRefPubMed
  35. ↵
    Kitano T, Brown KA, Daneman N, et al. The impact of COVID-19 on outpatient antibiotic prescriptions in Ontario, Canada; An interrupted time series analysis. In: Open forum infectious diseases. Oxford University Press US; 2021:ofab533.
  36. ↵
    Zhang A, Surette MD, Schwartz KL, et al. The collapse of infectious disease diagnoses commonly due to communicable respiratory pathogens during the COVID-19 pandemic: a time series and hierarchical clustering analysis. In: Open Forum Infectious Diseases. 2022.
  37. ↵
    1. Gulliford MC,
    2. Prevost AT,
    3. Charlton J,
    4. et al
    . Effectiveness and safety of electronically delivered prescribing feedback and decision support on antibiotic use for respiratory illness in primary care: REDUCE cluster randomised trial. BMJ2019;364:l236. https://www.bmj.com/content/364/bmj.l236. doi:10.1136/bmj.l236 pmid:30755451
    OpenUrlAbstract/FREE Full Text
Back to top