Intended for healthcare professionals

  1. Research
  2. CONSORT Harms 2022...
  3. CONSORT Harms 2022 statement, explanation, and elaboration: updated guideline for the reporting of harms in randomised trials
Research Methods & Reporting

CONSORT Harms 2022 statement, explanation, and elaboration: updated guideline for the reporting of harms in randomised trials

BMJ 2023; 381 doi: https://doi.org/10.1136/bmj-2022-073725 (Published 24 April 2023) Cite this as: BMJ 2023;381:e073725

Linked Opinion

Future directions of research into harms in randomised controlled trials
  1. Daniela R Junqueira, research associate1,
  2. Liliane Zorzela, assistant clinical professor1,
  3. Susan Golder, associate professor2,
  4. Yoon Loke, senior lecturer3,
  5. Joel J Gagnier, associate professor4,
  6. Steven A Julious, professor5,
  7. Tianjing Li, associate professor6 7,
  8. Evan Mayo-Wilson, associate professor8,
  9. Ba Pham, research coordinator9,
  10. Rachel Phillips, senior lecturer10,
  11. Pasqualina Santaguida, assistant professor11,
  12. Roberta W Scherer, retired senior scientist12,
  13. Peter C Gøtzsche, director13,
  14. David Moher, senior scientist14 15,
  15. John P A Ioannidis, professor16,
  16. Sunita Vohra, professor1
  17. on behalf of the CONSORT Harms Group
    1. 1Faculty of Medicine and Dentistry, University of Alberta, Edmonton, AB, Canada
    2. 2Department of Health Sciences, University of York, York, UK
    3. 3Norwich Medical School, University of East Anglia, Norwich, UK
    4. 4Department of Epidemiology and Biostatistics, Department of Surgery, Western University, London, ON, Canada
    5. 5Design, Trials and Statistics, School of Health and Related Research (ScHARR), University of Sheffield, Sheffield, UK
    6. 6Department of Ophthalmology, School of Medicine, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
    7. 7Department of Epidemiology, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO, USA
    8. 8Department of Epidemiology, UNC Gillings School of Global Public Health, Chapel Hill, NC, USA
    9. 9Knowledge Translation Programme, Unity Health Toronto, Toronto, ON, Canada
    10. 10Faculty of Medicine, School of Public Health, Imperial College London, London, UK
    11. 11Department of Health Research Methods, Evidence and Impact (HEI), McMaster University, Hamilton, ON, Canada
    12. 12Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA
    13. 13Institute for Scientific Freedom, Hørsholm, Denmark
    14. 14Centre for Journalology, Clinical Epidemiology Programme, Ottawa Hospital Research Institute, Ottawa, ON, Canada
    15. 15School of Epidemiology and Public Health, University of Ottawa, Ottawa, ON, Canada
    16. 16Departments of Medicine, of Epidemiology and Population Health, of Biomedical Data Science, and of Statistics, Stanford University, Stanford, CA, USA
    1. Correspondence to: S Vohra svohra{at}ualberta.ca
    • Accepted 6 February 2023

    Randomised controlled trials remain the reference standard for healthcare research on effects of interventions, and the need to report both benefits and harms is essential. The Consolidated Standards of Reporting Trials (the main CONSORT) statement includes one item on reporting harms (ie, all important harms or unintended effects in each group). In 2004, the CONSORT group developed the CONSORT Harms extension; however, it has not been consistently applied and needs to be updated. Here, we describe CONSORT Harms 2022, which replaces the CONSORT Harms 2004 checklist, and shows how CONSORT Harms 2022 items could be incorporated into the main CONSORT checklist. Thirteen items from the main CONSORT were modified to improve harms reporting. Three new items were added. In this article, we describe CONSORT Harms 2022 and how it was integrated into the main CONSORT checklist, and elaborate on each item relevant to complete reporting of harms in randomised controlled trials. Until future work from the CONSORT group produces an updated checklist, authors, journal reviewers, and editors of randomised controlled trials should use the integrated checklist presented in this paper.

    Randomised controlled trials are the reference standard among study designs to investigate the benefits of interventions. These trials are the foundation for regulatory approval of drugs and are also important when evaluating surgical procedures, medical devices, psychological and behavioural interventions, social interventions, and complementary therapies. Ideally, randomised controlled trials should evaluate not only potential benefits of interventions, but also potential harms. However, these trials are often limited in their ability to evaluate harms because of the short duration of intervention and follow-up, restricted study populations (eg, excluding participants with comorbidities or receiving cointerventions), and lack of statistical power to assess rare events.1234 Nevertheless, prospectively collected data about harms in randomised controlled trials are important to inform knowledge synthesis and patient and provider decisions. The randomised design offers a clear advantage and a unique opportunity to study harms in a controlled setting. Despite these strengths, trials often fail to report harms,56 even serious ones. For example, more than half the deaths and suicides occurring in trials of psychiatric drugs were not reported in published trial reports.7

    The insufficient reporting of harms outcomes might be the result of inadequate planning and design compounded with major challenges in fully reporting diverse adverse events within a limited amount of space in journal articles. However, harms can seriously affect a patient’s quality of life8 and treatment adherence,910 and increase financial costs to patients and health systems.11 To allow patients and healthcare providers to make truly informed decisions, randomised controlled trial reports should describe evidence on benefits and harms.

    To promote better reporting of harms in randomised controlled trials, the Consolidated Standards of Reporting Trials (hereafter referred to the main CONSORT) statement includes one potentially subjective item on reporting harms (ie, all important harms or unintended effects in each group; for specific guidance see CONSORT for harms).12 However, this item did not do full justice to the importance of harms related issues,13 and in 2004, the CONSORT group developed an extension providing specific guidance for the complete reporting of harms in randomised controlled trials (the CONSORT Harms statement).13 Although the main CONSORT statement was revised in 2001 and 2010, CONSORT Harms has not been updated since 2004. Moreover, although CONSORT is endorsed by journals of diverse areas of healthcare research, CONSORT Harms is not mentioned explicitly in the submission instructions of key scientific journals.5

    Considering the ongoing inadequacies in reporting harms in randomised controlled trials, a CONSORT Harms initiative was formed to update CONSORT Harms to provide current and improved guidance on a minimum set of items about harms to be reported in trials. Furthermore, to enhance harms reporting in clinical trials, these items should be integrated into the main CONSORT statement instead of remaining a standalone extension. Here, we present the CONSORT Harms 2022 checklist, illustrating how these items can be incorporated into the CONSORT statement. Until the CONSORT statement is updated to incorporate more complete guidance for reporting harms, CONSORT Harms 2022 should replace CONSORT Harms 2004.

    Summary points

    • Prospectively collected data about harms in randomised controlled trials are important to inform knowledge synthesis and patient and provider decisions

    • Randomised controlled trials should measure and report benefits and harms of health interventions

    • This article describes CONSORT Harms 2022, a guideline to support better reporting of harms in randomised controlled trial publications, and elaborates on reporting guidance for each item relevant to the complete reporting of harms assessed in trials

    • The CONSORT Harms 2022 reporting checklist is described in a way to show how it can be integrated into the main CONSORT checklist. Until future work from the CONSORT group produces an updated checklist, trial authors, journal reviewers, and editors should use the integrated checklist presented in this paper

    Development of CONSORT Harms 2022

    We followed the strategy recommended by the EQUATOR (Enhancing the quality and transparency of health research) Network for the development of reporting guidelines.14 The study protocol was approved by the University of Alberta Health Research Ethics Board on 2 March 2018 (Pro00078962). The approval process included a meta-epidemiological overview of the literature on reporting of harms and a modified Delphi survey,1516 including two online rounds followed by an in-person consensus meeting. Consumers and patient representatives participated in the online Delphi rounds, the in-person meeting, and manuscript preparation; their input was received and incorporated into the final document.

    We first conducted a comprehensive meta-epidemiological overview to identify studies of harms reporting in randomised controlled trials.17 We assessed 13 reviews of harms reporting in trials. These reviews assessed 522 trials for their reporting of the items in CONSORT Harms 2004, which varied from 9% to 69%.17 Reporting of harms improved only slightly when comparing clinical trials published before and after the publication of CONSORT Harms 2004.17 Data from this overview showed that checklist items comprising multiple components presented challenges for reporting and identified additional relevant items for reporting harms in trials.17 Examples of items of CONSORT Harms 2004 including multiple components are item 6 (clarify how harms related information was collected, including (i) mode of data collection, (ii) timing, (iii) attribution methods, (iv) intensity of ascertainment, and (v) harms related monitoring and stopping rules); and item 8 (describe plans for analysing harms, including (i) presentation of absolute risk per arm, (ii) per adverse event type, (iii) per grade, (iv) per seriousness, (v) appropriate metrics for recurrent events, (vi) continuous variables, and (vii) scale variables). These findings were used to generate new items and potential modifications to the original CONSORT Harms checklist, resulting in a list of 26 items.

    Next, we deployed a modified Delphi process consisting of two online survey rounds followed by an in-person consensus meeting. The online rounds were completed using a questionnaire developed and managed using the REDCap18 electronic data capture tool hosted and supported by the Women and Children’s Health Research Institute at the University of Alberta. Participants were asked to assess the importance of each item on a five point Likert scale (ranging from 1=strongly disagree to 5=strongly agree). An agreement on the inclusion or exclusion of items of the CONSORT checklist was prespecified and considered if 68% of the respondents agreed or strongly agreed with the inclusion or exclusion of the item). Free text comments were collected to guide the revision of the items. Participants invited to complete the online Delphi rounds were identified by their areas of expertise or stakeholder relevance to the initiative (eg, methodologists, statisticians, epidemiologists, clinicians, journal editors, consumers or patient representatives, and members of the industry and health regulatory agencies). The list of invitees also included researchers involved in the development of CONSORT, CONSORT Harms 2004 and other CONSORT extensions, and PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) and PRISMA Harms.

    Delphi rounds findings

    We collected feedback from 211 and 92 respondents in the first and second Delphi rounds, respectively. In the first Delphi round, an additional question sought participants’ opinions on whether CONSORT Harms should be integrated into the main CONSORT statement or remain a standalone extension; 67% of the respondents recommended integration. All items received many comments and suggestions, including recommendations for inclusion of new items or amending the existing ones.

    In the first Delphi round, one item did not reach agreement for inclusion (to identify the assessment of harms in the title). In the second Delphi round, respondents disagreed on the inclusion of one item (to describe methods of attribution to determine causality of adverse events). Several respondents voiced concern that the length of CONSORT would be excessive if numerous harms items were incorporated in addition to CONSORT items in the main checklist. To determine whether adding several harms items to the CONSORT statement would make it excessively long, we developed a draft in which all harms items surveyed in the first round were incorporated into the main CONSORT checklist so that respondents could see the potential text in situ. With few exceptions, the CONSORT items only required minor revision to refer to harms. This draft was provided to respondents in the second Delphi round to facilitate the visualisation of the integrated checklists.

    The next and final iteration was the in-person consensus meeting held in Edmonton, Canada in September 2019. A subset of 18 researchers who had participated in the online rounds or were part of the steering committee attended a two day meeting. The composition of participants attending the consensus meeting was comparable to the group that completed the online Delphi rounds (participants included methodologists, statisticians, epidemiologists, clinicians, journal editors, members of industry and health regulatory agencies, and consumers or patient representatives). The group discussed the list of items and comments collected in the online Delphi rounds and followed a nominal group technique19 to reach consensus. The meeting was documented by two note takers and the content was collated and refined to generate the final checklist and explanation document.

    Terminology

    Consistent with the guidance provided in CONSORT Harms 2004, we define harms as the totality of possible adverse consequences of an intervention or therapy; they are the direct opposite of benefits, against which they must be compared.13 We recommend reporting on harms including information on methods of ascertainment (that is, whether harms were assessed systematically or non-systematically). Adverse events and adverse drug reactions are other terms used to describe harms and often used in regulatory research. As defined in the main CONSORT guideline, outcomes in a randomised controlled trial are also distinguished as prespecified and non-prespecified, where prespecified refers to outcomes explicitly defined in the protocol.12 Therefore, in the context of harms reporting, we recommend considering the two related aspects. Firstly, was the case definition for the harmful outcome prespecified? If yes, how was the harmful outcome monitored (systematic or non-systematic assessment)?

    Harms might be assessed systematically by measuring variables for all participants using standardised clinical examinations, questionnaires, and medical instruments.20 For systematically assessed harms, the trial report should provide the case definition of the outcome assessed (eg, insomnia), the specific tool or instrument used (eg, Insomnia Severity Index), the metric (eg, change from baseline, final value, time to event), the method of aggregation (eg, median, proportion), and time points for data collection.21 If cut-off points associated with increased harm (for harms, assessed as continuous variables such as hormone levels or bone mass index) or categories were analysed, these should also be defined. The non-systematic assessment of harms relies on the passive or unstructured reporting of adverse events, such as unprompted self-reporting by participants,22 and can meaningfully complement the systematic assessment of harms, particularly when generating signals for unexpected events.

    Other terms to describe harms are not recommended. For instance, the use of “safety” can be a misleading term used to diminish the importance of harms or to imply the absence of harms. Many trials casually note that the intervention assessed was effective and safe. The term “safe” might give the impression that harms are not caused by an intervention or could imply that the trialists or the sponsoring drug company judged that the potential benefits of the intervention assessed outweighed the potential harms (at least under the trial conditions). However, this does not mean that the intervention carries no risk of harms in the trial conditions, let alone during general clinical use in the real world. Therefore, the potentially misleading safe/safety terminology should be avoided. Similarly, the term “side effect” denotes an effect without identifying it as a harmful one, and implies that it is related to the mechanism of action of the intervention.2324 Importantly, patient representatives participating in all Delphi rounds confirmed they felt the term “side effect” downplays harms and should therefore not be used. Lastly, the term “risk” is used colloquially to denote uncertainty, especially of an undesirable event. As a statistical term, risk is a proportion and should not be conflated with the nature of an event; that is, interventions are associated with the “risk” (or probability) of benefits and harms.

    Table 1 summarises some of the terminology problems concerning the reporting of harms in randomised controlled trials. For instance, events described as “anticipated” or “unintended” lack specific meaning. The use of ambiguous terminology is problematic and should be avoided.

    Table 1

    Ambiguity of harms related terminology as currently used in publications of randomised controlled trials that should be avoided

    View this table:

    How to use the CONSORT Harms update

    Table 2 shows the CONSORT Harms checklist alongside the main CONSORT statement. Subsequently, we provide examples and explanations for the CONSORT Harms items and discuss special considerations for the reporting of harms in randomised controlled trials, including examples of good reporting when available. Several historical examples predate current recommendations and do not adhere entirely to the terminology recommended in this update.

    Table 2

    CONSORT Harms 2022 items versus main CONSORT 2010 checklist

    View this table:

    CONSORT Harms 2022 had the goal of updating and incorporating items of the CONSORT Harms 2004 extension into the items of the main CONSORT statement. Overall, 13 of 25 items of the main CONSORT were minimally modified by adding the phrase “benefits and harms” to assure harms are reported alongside potential benefits. Considerable modifications were made to the remaining 12 items of the title and abstract, introduction, methods (topics outcomes, blinding and statistical methods), results (topics participant flow, recruitment, numbers analysed, outcomes estimation, ancillary analyses), discussion (topic limitations), and other information (topic protocol). Three new items for CONSORT Harms 2022 were developed and incorporated into the reporting checklist (a side-by-side comparison of CONSORT Harms 2004 and 2022 can be found in appendix 1 in the supplementary material). An elaboration and explanation section specifically concerning harms was also developed. CONSORT Harms urges authors that data on harms should be completely reported for all assessed and detected harms, and, when appropriate to overcome space constraints in journal publications, supplementary information on harms should be made publicly available through online repositories.

    CONSORT Harms 2022: checklist items, explanation, and elaboration

    Title and abstract

    Item 1b

    CONSORT—Structured summary of trial design, methods, results, and conclusions (for specific guidance see CONSORT for abstracts).

    CONSORT Harms 2022: item 1b—Structured summary of trial design, methods, results of outcomes of benefits and harms, and conclusions (for specific guidance see CONSORT for abstracts).

    CONSORT Harms explanation—Abstracts are often all readers can openly access from a trial report. Because of barriers to accessing the scientific literature (eg, paywalls, language), some readers might be able to access only information available in abstracts, rather than the full text.29 Lack of accuracy and completeness of reporting outcomes results in abstracts can result in misleading impression of results to readers.30 For example, biases favouring the reporting of only benefits and the use of vague general statements (eg, “safe” or “well tolerated”) are common. This is true for both abstracts of published papers and abstracts presented at conferences.3132

    Recognising that there might be challenges in accommodating journal guidance and character limits for abstracts, authors should strive to provide consistent and complete information on important benefits and harms. Finally, if no harms were assessed or detected, this should be stated.

    Example of item 1b of CONSORT Harms 2022

    “16 (2%) of 958 women in the intravenous iron sucrose group and 13 (1%) of 976 women in the standard therapy group had serious maternal adverse events. Serious fetal and neonatal adverse events were reported by 39 (4%) of 961 women in the intravenous iron sucrose group and 45 (5%) of 982 women in the standard therapy group. At 6 weeks post-randomisation, minor side-effects were reported by 117 (16%) of 737 women in the intravenous iron sucrose group versus 155 (21%) of 721 women in the standard therapy group.”75 [In this example, “side effects” is used as a reference to harm outcomes; preferable terms are “harms” or “adverse events.”]

    Introduction

    Item 2b

    CONSORT—Specific objectives or hypotheses.

    CONSORT Harms 2022: item 2b—Specific objectives or hypotheses for outcomes of benefits and harms.

    CONSORT Harms explanation—Randomised controlled trials are often designed with a focus on potential benefits, including for sample size estimates. However, it is also relevant to the Introduction to describe what consideration has been given to harms outcomes when planning the trial.

    Examples of item 2b of CONSORT Harms 2022

    “The LIFE Study is the largest trial to evaluate the benefits of physical activity in older people. Serious fall injury was included in the LIFE Study as one of the prespecified secondary outcomes. By improving gait, balance, and lower extremity strength, physical activity may reduce the likelihood of falling and sustaining a serious injury, but it may also increase opportunities to fall and incur a serious injury. Here we report results associated with our hypothesis that a long-term physical activity program compared with a health education program reduces the risk of serious fall injuries among sedentary older people with functional limitations.”35

    “The aim of the present study is to compare the postoperative analgesic efficacy and incidence of nausea between the variable-rate feedback infusion mode (VFIM) and conventional fixed-rate infusion mode (CFIM) of ropivacaine/fentanyl-based PCEA in patients who underwent open gastrectomy.”76 [PCEA=patient-controlled epidural analgesia]

    Methods

    Item 6a

    CONSORT—Completely defined prespecified primary and secondary outcome measures, including how and when they were assessed.

    CONSORT Harms 2022: item 6a—Completely defined prespecified primary and secondary outcomes, for both benefits and harms, including how and when they were assessed.

    Item 6b

    CONSORT—Any changes to trial outcomes after the trial commenced, with reasons.

    No modifications in Consort Harms 2022.

    Item 6c (new item)

    CONSORT Harms 2022: item 6c—Describe if and how non-prespecified outcomes of benefits and harms were identified, including any selection criteria, if applicable.

    CONSORT Harms explanation (items 6a and 6c)—Previous recommendations have suggested that randomised controlled trial reports should focus on the most clinically important harms.28 However, selecting harms for analysis and reporting based on clinical importance can be problematic because it relies on the author’s judgment and allows for the application of selection criteria that might be decided post hoc. There might not be consensus on the most clinically important harms,33 and documentation for any independent evaluation of harms that has informed reporting—for example, the charters for independent data monitoring committee and outcome adjudication committee—should be provided.

    For prespecified and systematically assessed harms, authors should identify and define the outcomes according to their domains (case definitions), measurements, metrics, method of aggregation and time frames.2125 For non-systematically assessed harms (prespecified or non-prespecified), the report should elaborate on the specific methods used to collect non-systematically assessed harms (eg, by stating verbatim if indirect questioning was used or including a copy of the blank case report form) and the time points at which participants were asked about harms or had a possibility of reporting harms if no questions were asked. The information provided should be sufficient to allow others to use the same outcomes34; and supplementary material could be used to describe the information with the necessary details. For non-systematically assessed harms (prespecified or non-prespecified), the report should elaborate on the specific methods used to collect non-systematically assessed harms (eg, by stating verbatim if indirect questioning was used or including a copy of the blank case report form) and the time points at which participants were asked about harms or had a possibility of reporting harms if no questions were asked. The information provided should be sufficient to allow others to use the same outcomes34; and supplementary material could be used to describe the information with the necessary details.

    Outcome measurements refer to whether trialists used standardised instruments to assess harms, if and how new scales were developed, and the use of non-validated scales. Initiatives like the Consensus-based Standards for the selection of health Measurement Instruments (COSMIN) provide specific guidance on how to select outcome measurement instruments; additional guidance on outcomes reporting can be found in CONSORT Outcomes.3536 Validated scales or consensus guidelines might3738 enhance quality of measurement and assist in comparison with similar studies.39 Metrics refer to the measure used to characterise the results; for example, proportion of patients experiencing an event. For harms reporting, it is important to clarify whether the metric refers to the proportion of patients presenting with at least one harm event or the proportion of patients presenting with specific harm events. This reporting is particularly relevant when considering events that can recur (eg, headaches). When harms outcomes are aggregated, the methods must be fully and completely described. These methods could include timing, coding or body system subgrouping, frequency, seriousness, severity, and chronicity, as appropriate. If harms are defined as attributed or not to the intervention, the attribution methods should be described along with information on who completed such assessment (investigators, data safety monitoring boards, patients, sponsors, or any combination thereof). The lack of reporting of such details has been shown to seriously hamper comparability of harms outcomes.40

    Finally, the specific time frame for the assessment of harms should be specified. The time point for assessing harms events can be crucial, as illustrated by a systematic review evaluating the risk of heparin induced thrombocytopenia in postoperative patients.41 Heparin induced thrombocytopenia is a prothrombotic disorder related to antibody mediated platelet activation induced by heparin that occurs within 5–14 days of the start of heparin treatment. In one trial considered for inclusion in the review, heparin induced thrombocytopenia was assessed on postoperative days 1–5.42 The trial was not included in the systematic review because the time point was inappropriate to allow for the reaction to develop.

    Criteria to include harms in a trial report are often post hoc decisions based on arbitrary thresholds, such as “harms occurring in at least 5% of the RCT participants” or “grade 3 or above events.” We strongly recommend that thresholds are not applied.4 Instead, authors should provide a summary of the data on harms in the main report and report all harms in detail in supplementary materials. If any criteria or rule based approaches are used to select which harms were identified in a report, they should be prespecified and stated explicitly.

    When authors choose a subset of harms to report, this can be highly misleading and dramatically impact the public’s ability to identify and synthesise the harms of intervention. For interventions that target healthy individuals (for example, screening), any harm, however minor, could be important to capture and report because the balance between harms and benefits could easily lean towards harms in a low risk population. For interventions that improve major outcomes (for example, survival in patients with cancer), minor harms might not be as important to patients compared with potential benefits. However, harms deemed clinically mild can seriously affect a patient’s quality of life8 and treatment adherence,910 and potentially increase financial costs to patients and health systems.11 Finally, results should be reported for all prespecified outcomes, not just for the most frequent events or analyses that were statistically significant or interesting.

    Example of item 6a CONSORT Harms 2022

    “As secondary outcomes, we analyzed intraoperative complications, drainage time, length of hospital stay, postoperative pain, postoperative quality of life, and readmissions within 90 days. Drainage time was defined as the interval between surgery and the removal of the chest tube and was measured in days. Length of hospital stay was measured in days after surgery. Postoperative pain was evaluated by a visual analog pain scale on the first, second, and third postoperative days and at the 30-day outpatient visit. We also assessed the need for opioid use at the 30-day outpatient visit. Any hospitalization within the 90-day postoperative period was considered as readmission”77. [In this example, “complications” is used as a reference to harm outcomes; preferable terms are “harms” or “adverse events.”]

    Item 11a

    CONSORT—If done, who was blinded after assignment to interventions (eg, participants, care providers, those assessing outcomes) and how.

    CONSORT Harms 2022: item 11a—If done, who was blinded after assignment to interventions (eg, participants, care providers, those assessing outcomes of benefits and harms) and how.

    CONSORT Harms explanation—Blinding, or masking, is the process of withholding information about the intervention assignment from people involved in the conduct of a trial after participants are assigned to the intervention. Personnel and participants aware of intervention assignment might be influenced in their delivery of interventions as well as beliefs of intervention effects; therefore, blinding provides protection against bias due to deviations from intended intervention and in outcomes ascertainment.4344 People involved in a trial who could be blinded include study participants, and personnel such as people administering the intervention, laboratory technicians, data collectors, people assessing benefits and harms, and statisticians. Of note, despite the widespread use of terminologies such as single, double, or triple blind, there is no standard, widespread accepted meaning for these terms; therefore, authors should avoid them.45 Additionally, “masking” rather than “blinding” is the preferred terminology in ophthalmology trials.

    For some trials, it is not possible to keep either the participant or the person administering the intervention unaware of the assignment. In these cases, it might still be possible for people assessing benefits and harms (other than those assessed by the participants themselves) to remain blinded. Authors should report whether or not blinding of harms assessment occurred. In some cases, this assessment involves an external committee, such as an independent safety monitoring board; members of that committee make decisions about attribution of harms and their possible relationship with the intervention under study.

    Item 12a

    CONSORT—Statistical methods used to compare groups for primary and secondary outcomes.

    CONSORT Harms 2022: item 12a—Statistical methods used to compare groups for primary and secondary outcomes of both benefits and harms.

    CONSORT Harms explanation—Statistical methods should be described for each outcome investigated in a randomised controlled trial, including both primary and secondary benefits and harms. Additionally, it is important that a report states whether the methods were prespecified. If a statistical analysis plan is available, it should be dated and have revisions documented. If there are deviations from the statistical analysis plan, they should be documented. Supplementary material can be used to provide this information.

    For non-systematically assessed harms, it is common practice in trials to code and group events into broader categories before conducting statistical analyses. The report should describe how individual events were grouped, including details for any coding dictionary (eg, MedDRA). Grouping harms could help identify signals for biologically related harms or syndromes, but it might also allow trial investigators to obscure important harms by combining them with less important ones (eg, migraine might be more severe than headache, and bleeding might be more severe than bruising). Authors should clarify the level of grouping (eg, high level terms) and whether such groupings were prespecified in a protocol or statistical analysis plan or defined post hoc.46 Obscuring harms by coding can be problematic.4748

    When relevant, authors should plan and explain how they accounted for time and duration of events, duration of follow-up, and how recurrent events were managed in the trial analysis. Reporting such information allows a more accurate and comprehensive knowledge of harms and provides valuable insight into participants’ quality of life. Of note, harms occurring repeatedly and over an extended duration might have a detrimental impact on a patient’s quality of life, even if the harms are not considered serious or severe.

    Example of item 12a of CONSORT Harms 2022

    “Safety analyses: the primary safety variables were time-to-first treatment emergent adverse event (AE), serious AE, and AE leading to discontinuation from study treatment. The primary variable was analysed using a Cox regression model stratified by randomisation stratum, with treatment group, severity of asthma and region as fixed factors.”78 [In this example, “safety” is used as a reference to harm outcomes. We recommend against the use of the terminology “safety” variables. Our guidance is for complete reporting of “harms” as the possible adverse consequences of an intervention and the direct opposite of benefits.]

    Results

    Item 13a

    CONSORT—For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome.

    CONSORT Harms 2022: item 13a—For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for outcomes of benefits and harms.

    CONSORT Harms explanation—The flow of participants in the trial and the population analysed for the outcomes measured might not be identical for benefits and harms. Censoring or attrition might lead to different analysable populations, which should be described clearly. For harms, additional information on dose reductions of the allocated intervention might also be relevant to report, with emphasis on reasons, including if related to participants experiencing harms. It should be acknowledged that attrition is often due to a combination of harms and (perceived) lack of benefits, and it might be difficult to disentangle the relative contribution of these different reasons. Also, trials with long term follow-up should differentiate and describe attrition issues related to harms at early and late time points to improve accuracy of the information related to harms. Information on time to discontinuation due to harms might also be useful in some circumstances.

    Example of item 13a of CONSORT Harms 2022

    In this RCT of the effect of an intervention (vosoritide) to treat achondroplasia as compared with placebo, the population analysed for harms included all patients who received at least one dose of the intervention (vosoritide) or placebo (the so-called safety analyses).79 The population analysed for potential benefits included all randomised participants, constituting the so-called full analysis set according to intention-to-treat principles.

    The flow diagram (fig 1) details the numbers of participants who were randomly assigned and received intended treatment as defined for each population analysed.

    Fig 1
    Fig 1

    Flow diagram, relating to example 13a of CONSORT Harms 2022. [In this example, “safety” is used as a reference to harm outcomes. We recommend against the use of “safety analysis” or “safety population” terms. Our guidance is for complete reporting of the randomised population and number analysed for outcomes of benefits and harms regardless of whether harms were the primary or secondary outcome in the trial.]

    Item 14a

    CONSORT—Dates defining the periods of recruitment and follow-up.

    CONSORT Harms 2022: item 14a—Dates defining the periods of recruitment and follow-up for outcomes of benefits and harms.

    CONSORT Harms explanation—Information on the date of the randomisation and the completion of the study for the primary efficacy outcome does not suffice for the assessment of harms. For instance, the assessment of harms might be planned to take place during the entire study through non-systematic assessment, might occur during only part of the study duration, might occur at specific time points using systematic or non-systematic assessment, or might continue after the completion of follow-up for the main efficacy outcome. Reporting the periods of recruitment and follow-up for benefits and harms is crucial to allow comprehensive and accurate interpretation of the trials results.

    Example of item 14a of CONSORT Harms 2022

    “The trial involved five visits: Visit 1 on day 1 (screening, randomization and initial dosing), Visit 2 on day 2 (assessment of the primary endpoint), Visit 3 on day 4 (assessment of efficacy and safety parameters), Visit 4 on day 6 (end-of-treatment visit) and Visit 5 on day 8 to day 10 (follow up by telephone interview). Patients were asked to return all unused trial medication and their diaries at each visit.”80 [In this example, “safety” is used as a reference to harm outcomes. We recommend against the use of “safety”; preferable terms are “harms” or “adverse events.”]

    CONSORT Harms 2022 special consideration: run-in periods

    A meta-epidemiological study49 identified randomised trials with run-in periods and evaluated the characteristics of these run-in periods, and the completeness of reporting with regard to exclusions (number of and reasons for exclusions and baseline characteristics of those excluded). Of 470 PubMed indexed randomised controlled trial publications from 2014, 25 (5%) included some type of run-in period. In 23 out of 25 trials (88%), the run-in period was incompletely reported, mostly because of missing baseline characteristics. The authors recommended trial publications should state the number of excluded patients, reasons for exclusion, and baseline characteristics of the excluded patients.

    Run-in periods have been used in some randomised controlled trials to exclude patients before randomisation. A frequent approach is to give patients the trial drug and to exclude those who experience harms.50 Trials estimating the effects of initiating unknown interventions (v initiating placebo) might obtain different results compared with trials estimating the effect of continuing well tolerated interventions. Therefore, when run-in periods are used, adequate reporting of exclusions due to harms during or after the run-in period is essential to avoid underestimating the true incidence of harms.51 Harms should be reported for all the study periods, including the run-in period.

    Item 16

    CONSORT—For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups.

    CONSORT Harms 2022: item 16—For each group, number of participants (denominator) included in each analysis of outcomes of benefits and harms and whether the analysis was by original assigned groups and if any exclusions were made.

    CONSORT Harms explanation—Information about which participants were included in which analyses is essential. It is common practice in randomised controlled trial reports to describe the population analysed using a variety of terms, such as intention to treat, modified intention to treat, and per protocol analysis. In most trials, each of these will include a different set of participants and corresponding denominator; however, it is often not clear from trial reports which study population was included in the analyses of different outcomes. The most common terminology, the intention-to-treat analysis, by principle should analyse all participants in the group randomised, regardless of non-compliance, protocol deviations, withdrawal, and any other reason for exclusion after randomisation.52 Nevertheless, it is not always possible to measure outcome data on all participants.53 “Safety population” is an additional term often used in reports of randomised controlled trials to define an analysis approach that includes patients who received at least one dose of the drug or intervention. However, heterogeneity in the use of the different terms and missing data might confuse readers trying to identify the population at risk that was analysed.

    Example of item 16 of CONSORT Harms 2022

    “Results, Patient disposition, baseline demographics and study drug exposure. The integrated safety database included a total of 4439 patients who received at least one dose of study drug (placebo, n=1262; lasmiditan, n=3177). Of those who received a study dose, 97% assigned either placebo or to a lasmiditan dose group completed the study (i.e. all required follow-ups).”81 [In this example, we recommend against the use of the terminology “safety” database. Our guidance is for complete reporting of “harms” as the possible adverse consequences of an intervention and the direct opposite of benefits.]

    Item 17a

    CONSORT—For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% confidence interval).

    CONSORT Harms 2022: item 17a—For each primary and secondary outcomes of benefits and harms, results for each group, and the estimated effect size and its precision (such as 95% confidence intervals).

    Item 17a2 (new item)

    CONSORT Harms 2022: item 17a2—For outcomes omitted from the trial report (benefits and harms), provide rationale for not reporting and indicate where the data on omitted outcomes can be accessed.

    Item 17b

    CONSORT—For binary outcomes, presentation of both absolute and relative effect sizes is recommended.

    CONSORT Harms 2022: item 17b—Presentation of both absolute and relative effect sizes is recommended for outcomes of benefits and harms.

    Item 17c (new item)

    CONSORT Harms 2022: item 17c—Report zero events if no harms were observed.

    CONSORT Harms explanation (items 17a, 17a2, and 17c)—Harms are often dichotomised into binary or count data (see explanation for items 6a, 6c, and 12a), in which case it could be appropriate to follow the main CONSORT guidance to present relative (risk ratio (relative risk) or odds ratio) and absolute effect (risk difference) metrics with confidence intervals.12 However, for non-systematically assessed harms, measures of relative or absolute risk difference should be used with caution because these outcomes were not actively determined. Additionally, harms should be reported even when they are not directly comparable among intervention groups, thus precluding computing measures of relative or absolute effect. Therefore, for harms, we recommend that the reporting of items 17a and 17b be considered as appropriate and not solely based on whether the outcome was analysed as binary data.

    When presenting the results of harms, whether using binary or continuous data, it is important to separate the reporting into systematically and non-systematically assessed harms. Additionally, authors should always report events of serious harms. If attribution methods were applied, it is informative and adds to transparency that all harms are reported, even if events are deemed not related to the intervention (see explanation for items 6a and 6c). We also recommend that authors consider available visualisation methods for the communication of adverse events (illustrating data through figures rather than tables), which can offer a useful alternative to help summarise harm profiles.5455

    Reporting incidence (eg, proportion of people experiencing an event) can be misleading for recurring events. Therefore, it should be clear whether authors report number of participants presenting with at least one event (incidence) or number of events per unit of time at risk (rate), and whether recurrent events were included.

    Concerning the relevance of the time frame to specify and define harms (see item 6), authors should report exposure time and any differences in follow-up or exposure and how they accounted for these differences in the analysis. Studies with differential follow-up between intervention arms (eg, because of participant withdrawals) only provide the opportunity to collect information on harms for participants who remain in the study. In this scenario, using person time for exposure can be misleading if participants who continued an intervention are a subgroup who experienced few harms.

    Confidence intervals are useful for reporting systematically assessed harms and sometimes for non-systematically assessed harms (P values for results of non-systematically assessed harms are of little relevance because hypothesis testing would be of limited value). Confidence intervals could indicate that the result does not rule out an important clinical difference in one or both directions. Although P values might be provided in addition to confidence intervals, results should not be reported solely as P values.

    Finally, for prespecified and systematically assessed harms, authors should report results even if zero events were observed to promote transparency and to facilitate synthesis across trials.

    Example of items 17a and 17b of CONSORT Harms 2022

    Harms experienced by patients treated with albiglutide were compared with harms assessed in patients receiving placebo in a randomised controlled trial conducted by Hernandez et al.82 The trial publication reports the number and proportion of patients who experienced systematically assessed harms among patients who received at least one dose of albiglutide or placebo, along with the relative risk (RR) of harms with 95% confidence interval (CI).

    Table 3 presents the absolute risks and relative risks reported by Hernandez and colleagues and the estimates of risk difference, which is considered a preferable statistical measure when dealing with rare events. This is because relative risk estimates for rare events can be misleading in representing effects sizes; table 3 shows how the relative risk of severe hypoglycaemia of 0.56 relates to a risk difference of 0.005. The use of relative risks is also problematic when dealing with zero events in one or two arms.

    Table 3

    Prespecified systematic harms

    View this table:

    Example of item 17c of CONSORT Harms 2022

    Table 4

    Primary and secondary cardiovascular outcomes and safety outcomes in the overall population83

    View this table:

    Item 18

    CONSORT—Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre-specified from exploratory.

    CONSORT Harms 2022: item 18—Results of any other analyses performed for outcomes of benefits and harms, including subgroup analyses and adjusted analyses, distinguishing prespecified from exploratory.

    CONSORT Harms Explanation—When subgroup analyses are done, authors should report which subgroups were examined, the rationale for choosing these subgroups, and whether the direction and magnitude of differences between subgroups were prespecified. Selective reporting of subgroup analyses is an important source of bias. Subgroup analysis investigates if the intervention effects vary among the levels of some factor of interest, but very few trials are powered to detect subgroup differences, especially for harms outcomes.

    If multiple analyses are done, authors should indicate which analysis (eg, unadjusted or adjusted) is the primary one, and if it was prespecified in the statistical analysis plan or protocol. For example, whether variables adjusted for were prespecified in the protocol and known a priori to be strongly or moderately associated with the outcome should also be stated.

    Discussion

    Item 20

    CONSORT—Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses.

    CONSORT Harms 2022: item 20—Trial limitations, addressing sources of potential bias related to the approach to collecting or reporting data on harms, imprecision, and, if relevant, multiplicity or selection of analyses.

    CONSORT Harms explanation—When discussing trial limitations, authors should indicate whether the data are similarly valid and reliable for both benefits and harms. If data on harms were not systematically assessed, authors should draw particular attention to limitations in the assessment of harms, including the chance of both false positive and false negative results (lack or poor monitoring of harms does not provide evidence that no harms occurred).56

    In addition to risk of bias and imprecision that apply to all outcomes, trials often have specific limitations related to assessing, reporting, and analysing harms. Trials with adequate power to detect potential benefits might be underpowered to detect important differences in harms, including prespecified harms. Trials of short duration might not be able to detect harms that develop after prolonged treatment, and trials reporting only the proportion of participants who experienced one or more harms might conceal information about event rates that is important for decision making.5 Finally, threats to internal validity might affect harms and benefits differently. For instance, data are commonly missing for participants who discontinue interventions because of harms.

    Although harms are more likely to be underreported than overreported, estimates of the occurrence of harms might also be influenced by a nocebo effect, where the communication about potential harms inflates the reporting of these events.57 From this perspective, the informed consent process can contribute to the detection of specific harms if they are communicated with negative framing or in a way that promotes the reporting of placebo induced harms.58596061 Harms can also be overlooked or disregarded by investigators who are not convinced of attribution; all harms should be reported, rather than only those felt by investigators to be causally attributed to an intervention.

    Example of item 20 of CONSORT Harms 2022

    “Finally, although we found no increase in the risk of congenital abnormalities among babies of women treated with progesterone, the trial was not powered for such rare outcomes.”84

    Item 24

    CONSORT—Where the full trial protocol can be accessed, if available.

    CONSORT Harms 2022: item 24—Where the full trial protocol and other relevant documents can be accessed, including additional data on harms.

    CONSORT Harms explanation—A large portion of data about harms are not published.62636465 The inadequate reporting of harms is prevalent among pharmaceutical and non-pharmaceutical sponsored trials, a troublesome and widespread practice.6667 It might not be practical to report all data on harms in journal publications, but these data should be made freely available elsewhere for systematically and non-systematically assessed outcomes, ideally including anonymised individual participant data.

    Trial authors should report where data can be found. The FAIR data principles (findability, accessibility, interoperability, and reuse) are relevant to harms data and should be considered as open science practices and evolving legislation become more accepted.68

    Example of item 24 of CONSORT Harms 2022

    “Data sharing: The statistical analysis plan is available at https://doi.org/10.6084/m9.figshare.5375026.v1. The raw trial data are provided by the authors on a secure online repository (see supplementary appendix for link). Data include anonymised individual patient variables for results reported here, a read-me file with data dictionary and analyses scripts used in this study.”85

    Discussion

    Investigators continue to under-report harms.46566697071727374 The CONSORT extension for Harms was published in 2004 to highlight the inadequate reporting of harms in trials, and to promote guidance on the minimum set of information on harms trials should provide.13 A guideline is only helpful if its guidance is followed. While the CONSORT statement has been endorsed by over 500 medical journals, CONSORT extensions are endorsed by a small fraction of these. We evaluated the adherence to CONSORT Harms items in published clinical trials and found that half of the items failed to reach more than 50% compliance after 2004.17

    Because interventions have the potential to cause beneficial and harmful effects, we proposed to the CONSORT Group to incorporate harms items into the main CONSORT statement. The CONSORT group requested that Delphi participants in this study be offered the opportunity to comment. Most Delphi participants supported the incorporation of CONSORT Harms extensions into the main CONSORT checklist and the CONSORT Executive agreed. The main CONSORT statement will likely be revised, at which stage the removal of item 19 might be recommended considering the integration of CONSORT and CONSORT Harms. Until future work from the CONSORT group produces an updated checklist, trial authors, journal reviewers, and editors should use the integrated checklist presented in this paper (table 2).

    Ethics statements

    Ethical approval

    The study protocol was approved by the University of Alberta Health Research Ethics Board on 2 March 2018 (Pro00078962).

    Acknowledgments

    CONSORT Harms Group contributors: Stephen Evans, London School of Hygiene and Tropical Medicine, UK; Ann Fonfa, Annie Appleseed Project, USA; Thomas A Lang, Principal, Tom Lang Communications and Training International, USA; Elizabeth Loder, The BMJ, Brigham and Women’s Physician Organisation, UK; Laura Weeks, Canadian Agency for Drugs and Technologies in Health (CADTH), Canada. Group members supported the work by providing critical suggestions and modifications to the manuscript content. Ann Fonfa provided an advocate-consumer perspective.

    Contributors: DRJ was responsible for the study’s conceptualisation, methodology, investigation, data curation, formal analysis, project administration, writing of the original draft, and review and editing of the manuscript. LZ was responsible for the study’s conceptualisation, methodology, project administration, and review and editing of the manuscript. SG and YL were responsible for the study’s conceptualisation, methodology, critical review. JJG, SAJ, TL, EM-W, BP, RP, PS, RWS were responsible for the study’s validation, writing and critical review. PCG, DM, JPAI were responsible for the study’s conceptualisation, methodology, writing—review and editing. SV was responsible for the study’s conceptualisation, methodology, funding acquisition, supervision, writing—review and editing. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

    Footnotes

    • We thank all participants who completed the Delphi surveys, thus informing the development of this reporting guideline with their knowledge and expertise.

    • Funding: This study was supported by NACTRC (Northern Alberta Clinical Trials and Research Centre). The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

    • Competing interests: All authors have completed the ICMJE uniform disclosure form at https://www.icmje.org/disclosure-of-interest/ and declare: support from NACTRC for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

    • Patient and public involvement: Patients were invited to complete the Delphi surveys and given the opportunity to provide inputs and critical review.

    • Provenance and peer review: Not commissioned; externally peer reviewed.

    References

    1. ↵
      1. Hodkinson A,
      2. Kirkham JJ,
      3. Tudur-Smith C,
      4. Gamble C
      . Reporting of harms data in RCTs: a systematic review of empirical assessments against the CONSORT harms extension. BMJ Open2013;3:e003436. doi:10.1136/bmjopen-2013-003436. pmid:24078752
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. Gewandter JS,
      2. Smith SM,
      3. McKeown A,
      4. et al.
      Reporting of adverse events and statistical details of efficacy estimates in randomized clinical trials of pain in temporomandibular disorders: analgesic, anesthetic, and addiction clinical trial translations, innovations, opportunities, and networks systematic review. J Am Dent Assoc2015;146:246-54. doi:10.1016/j.adaj.2014.12.023. pmid:31488200
      OpenUrlCrossRefPubMed
    3. ↵
      1. Golder S,
      2. Loke YK,
      3. Wright K,
      4. et al
      . Reporting of adverse events in published and unpublished studies of health care interventions: a systematic review. PLoS Med2016;13:e1002127. doi:10.1371/journal.pmed.1002127 pmid:31488200
      OpenUrlCrossRefPubMed
    4. ↵
      1. Mayo-Wilson E,
      2. Fusco N,
      3. Hong H,
      4. Li T,
      5. Canner JK,
      6. Dickersin K
      . Opportunities for selective reporting of harms in randomized clinical trials: Selection criteria for non-systematic adverse events. Trials2019;20:553. doi:10.1186/s13063-019-3581-3. pmid:31488200
      OpenUrlCrossRefPubMed
    5. ↵
      1. Phillips R,
      2. Hazell L,
      3. Sauzet O,
      4. Cornelius V
      . Analysis and reporting of adverse events in randomised controlled trials: a review. BMJ Open2019;9:e024537. doi:10.1136/bmjopen-2018-024537. pmid:30826796
      OpenUrlAbstract/FREE Full Text
    6. ↵
      1. Ioannidis JP
      . Adverse events in randomized trials: neglected, restricted, distorted, and silenced. Arch Intern Med2009;169:1737-9. doi:10.1001/archinternmed.2009.313. pmid:19858427
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Hughes S,
      2. Cohen D,
      3. Jaggi R
      . Differences in reporting serious adverse events in industry sponsored clinical trial registries and journal articles on antidepressant and antipsychotic drugs: a cross-sectional study. BMJ Open2014;4:e005535. doi:10.1136/bmjopen-2014-005535. pmid:25009136
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Costello R,
      2. Patel R,
      3. Humphreys J,
      4. McBeth J,
      5. Dixon WG
      . Patient perceptions of glucocorticoid side effects: a cross-sectional survey of users in an online health community. BMJ Open2017;7:e014603. doi:10.1136/bmjopen-2016-014603. pmid:28373256
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Leporini C,
      2. De Sarro G,
      3. Russo E
      . Adherence to therapy and adverse drug reactions: is there a link?Expert Opinion on Drug Safety2014;13:41-55. doi:10.1517/14740338.2014.947260. pmid:28373256
      OpenUrlCrossRefPubMed
    10. ↵
      1. Campbell RT,
      2. Willox GP,
      3. Jhund PS,
      4. et al
      . Reporting of lost to follow-up and treatment discontinuation in pharmacotherapy and device trials in chronic heart failure: a systematic review. Circ Heart Fail2016;9:e002842. doi:10.1161/CIRCHEARTFAILURE.115.002842. pmid:27162229
      OpenUrlAbstract/FREE Full Text
    11. ↵
      1. Birnbaum HG,
      2. White AG,
      3. Reynolds JL,
      4. et al
      . Estimated costs of prescription opioid analgesic abuse in the United States in 2001: a societal perspective. Clin J Pain2006;22:667-76. doi:10.1097/01.ajp.0000210915.80417.cf. pmid:16988561
      OpenUrlCrossRefPubMedWeb of Science
    12. ↵
      1. Moher D,
      2. Hopewell S,
      3. Schulz KF,
      4. et al
      . CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ2010;340:c869. doi:10.1136/bmj.c869. pmid:20332511
      OpenUrlFREE Full Text
    13. ↵
      1. Ioannidis JP,
      2. Evans SJ,
      3. Gøtzsche PC,
      4. et al.,
      5. CONSORT Group
      . Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med2004;141:781-8. doi:10.7326/0003-4819-141-10-200411160-00009. pmid:15545678
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Moher D,
      2. Schulz KF,
      3. Simera I,
      4. Altman DG
      . Guidance for developers of health research reporting guidelines. PLoS Med2010;7:e1000217. doi:10.1371/journal.pmed.1000217. pmid:20169112
      OpenUrlCrossRefPubMed
    15. ↵
      1. Murphy MKBN,
      2. Black NA,
      3. Lamping DL,
      4. et al
      . Consensus development methods, and their use in clinical guideline development. Health Technol Assess1998;2:1-88. doi:10.3310/hta2030 pmid:9561895
      OpenUrlCrossRefPubMed
    16. ↵
      1. Trevelyan EG,
      2. Turner WA,
      3. Robinson N
      . Developing an acupuncture protocol for treating phantom limb pain: a Delphi consensus study. Acupunct Med2015;33:42-50. doi:10.1136/acupmed-2014-010668. pmid:25488185
      OpenUrlCrossRefPubMed
    17. ↵
      1. Junqueira DR,
      2. Phillips R,
      3. Zorzela L,
      4. et al
      . Time to improve the reporting of harms in randomized controlled trials. J Clin Epidemiol2021;136:216-20. doi:10.1016/j.jclinepi.2021.04.020. pmid:33984494
      OpenUrlCrossRefPubMed
    18. ↵
      1. Harris PA,
      2. Taylor R,
      3. Thielke R,
      4. Payne J,
      5. Gonzalez N,
      6. Conde JG
      . Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform2009;42:377-81. doi:10.1016/j.jbi.2008.08.010. pmid:18929686
      OpenUrlCrossRefPubMedWeb of Science
    19. ↵
      1. Harvey N,
      2. Holmes CA
      . Nominal group technique: an effective method for obtaining group consensus. Int J Nurs Pract2012;18:188-94. doi:10.1111/j.1440-172X.2012.02017.x. pmid:22435983
      OpenUrlCrossRefPubMed
    20. ↵
      1. Mayo-Wilson E,
      2. Fusco N,
      3. Li T,
      4. Hong H,
      5. Canner JK,
      6. Dickersin K,
      7. MUDS investigators
      . Harms are assessed inconsistently and reported inadequately part 1: systematic adverse events. J Clin Epidemiol2019;113:20-7. doi:10.1016/j.jclinepi.2019.04.022. pmid:31055175
      OpenUrlCrossRefPubMed
    21. ↵
      1. Zarin DA,
      2. Tse T,
      3. Williams RJ,
      4. Califf RM,
      5. Ide NC
      . The ClinicalTrials.gov results database—update and key issues. N Engl J Med2011;364:852-60. doi:10.1056/NEJMsa1012065. pmid:21366476
      OpenUrlCrossRefPubMedWeb of Science
    22. ↵
      FDAAA 801 and the Final Rule. Department of Health and Human Services. 42 CFR Part 11. Clinical trials registration and results information submission: final rule. 2016.
    23. ↵
      1. Chou R,
      2. Aronson N,
      3. Atkins D,
      4. et al
      . AHRQ series paper 4: assessing harms when comparing medical interventions: AHRQ and the effective health-care program. J Clin Epidemiol2010;63:502-12. doi:10.1016/j.jclinepi.2008.06.007. pmid:18823754
      OpenUrlCrossRefPubMedWeb of Science
    24. ↵
      1. Aronson JKFR,
      2. Ferner RE
      . Clarification of terminology in drug safety. Drug Saf2005;28:851-70. doi:10.2165/00002018-200528100-00003 pmid:16180936
      OpenUrlCrossRefPubMedWeb of Science
    25. ↵
      1. Chan AW,
      2. Tetzlaff JM,
      3. Gøtzsche PC,
      4. et al
      . SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. BMJ2013;346:e7586. doi:10.1136/bmj.e7586. pmid:23303884
      OpenUrlAbstract/FREE Full Text
      1. George GC,
      2. Barata PC,
      3. Campbell A,
      4. et al
      . Improving attribution of adverse events in oncology clinical trials. Cancer Treat Rev2019;76:33-40. doi:10.1016/j.ctrv.2019.04.004. pmid:31108240
      OpenUrlCrossRefPubMed
      1. Enrico D,
      2. Waisberg F,
      3. Burton J,
      4. Mandó P,
      5. Chacón M
      . Analysis of adverse events attribution and reporting in cancer clinical trials: a systematic review. Crit Rev Oncol Hematol2021;160:103296. doi:10.1016/j.critrevonc.2021.103296. pmid:33675904
      OpenUrlCrossRefPubMed
    28. ↵
      1. Lineberry N,
      2. Berlin JA,
      3. Mansi B,
      4. et al
      . Recommendations to improve adverse event reporting in clinical trial publications: a joint pharmaceutical industry/journal editor perspective. BMJ2016;355:i5078. doi:10.1136/bmj.i5078. pmid:27697753
      OpenUrlFREE Full Text
    29. ↵
      1. The PLoS Medicine Editors
      . The impact of open access upon public health. PLoS Med2006;3:e252. doi:10.1371/journal.pmed.0030252. pmid:16719556
      OpenUrlCrossRefPubMed
    30. ↵
      1. Ward LG,
      2. Kendrach MG,
      3. Price SO
      . Accuracy of abstracts for original research articles in pharmacy journals. Ann Pharmacother2004;38:1173-7. doi:10.1345/aph.1D416. pmid:15150375
      OpenUrlCrossRefPubMed
    31. ↵
      1. Fleming PS,
      2. Buckley N,
      3. Seehra J,
      4. Polychronopoulou A,
      5. Pandis N
      . Reporting quality of abstracts of randomized controlled trials published in leading orthodontic journals from 2006 to 2011. Am J Orthod Dentofacial Orthop2012;142:451-8. doi:10.1016/j.ajodo.2012.05.013. pmid:22999667
      OpenUrlCrossRefPubMed
    32. ↵
      1. Jørgensen AW,
      2. Jørgensen KJ,
      3. Gøtzsche PC
      . Unbalanced reporting of benefits and harms in abstracts on rofecoxib. Eur J Clin Pharmacol2010;66:341-7. doi:10.1007/s00228-010-0791-8. pmid:20162413
      OpenUrlCrossRefPubMed
    33. ↵
      1. Mayo-Wilson E,
      2. Golozar A,
      3. Cowley T,
      4. et al
      . Methods to identify and prioritize patient-centered outcomes for use in comparative effectiveness research. Pilot Feasibility Stud2018;4:95. doi:10.1186/s40814-018-0284-6. pmid:30026961
      OpenUrlCrossRefPubMed
    34. ↵
      1. Glasziou P,
      2. Meats E,
      3. Heneghan C,
      4. Shepperd S
      . What is missing from descriptions of treatment in trials and reviews?BMJ2008;336:1472-4. doi:10.1136/bmj.39590.732037.47. pmid:18583680
      OpenUrlFREE Full Text
    35. ↵
      1. Gill TM,
      2. Pahor M,
      3. Guralnik JM,
      4. et al.,
      5. LIFE Study Investigators
      . Effect of structured physical activity on prevention of serious fall injuries in adults aged 70-89: randomized clinical trial (LIFE Study). BMJ2016;352:i245. doi:10.1136/bmj.i245. pmid:26842425
      OpenUrlAbstract/FREE Full Text
    36. ↵
      1. Prinsen CA,
      2. Vohra S,
      3. Rose MR,
      4. et al
      . How to select outcome measurement instruments for outcomes included in a “Core Outcome Set”—a practical guideline. Trials2016;17:449. doi:10.1186/s13063-016-1555-2. pmid:27618914
      OpenUrlCrossRefPubMed
    37. ↵
      1. McDowell INC
      . Measuring health: a guide to rating scales and questionnaires.3rd ed. Oxford University Press, 2006doi:10.1093/acprof:oso/9780195165678.001.0001.
      OpenUrlCrossRef
    38. ↵
      1. Streiner DNC
      . Health measurement scales: a practical guide to their development and use.3rd ed. Oxford University Press, 2003.
    39. ↵
      1. Clarke M
      . Standardising outcomes for clinical trials and systematic reviews. Trials2007;8:39. doi:10.1186/1745-6215-8-39. pmid:18039365
      OpenUrlCrossRefPubMed
    40. ↵
      1. Bailey C,
      2. Peddie D,
      3. Wickham ME,
      4. et al
      . Adverse drug event reporting systems: a systematic review. Br J Clin Pharmacol2016;82:17-29. doi:10.1111/bcp.12944. pmid:27016266
      OpenUrlCrossRefPubMed
    41. ↵
      1. Junqueira DR,
      2. Zorzela LM,
      3. Perini E
      . Unfractionated heparin versus low molecular weight heparins for avoiding heparin-induced thrombocytopenia in postoperative patients. Cochrane Database Syst Rev2017;2017:CD007557. doi:10.1002/14651858.CD007557.pub3. pmid:28431186
      OpenUrlCrossRefPubMed
    42. ↵
      1. Konkle BA,
      2. Bauer TL,
      3. Arepally G,
      4. et al
      . Heparin-induced thrombocytopenia: bovine versus porcine heparin in cardiopulmonary bypass surgery. Ann Thorac Surg2001;71:1920-4. doi:10.1016/S0003-4975(01)02534-6. pmid:11426769
      OpenUrlCrossRefPubMedWeb of Science
    43. ↵
      1. Probst P,
      2. Grummich K,
      3. Heger P,
      4. et al
      . Blinding in randomized controlled trials in general and abdominal surgery: protocol for a systematic review and empirical study. Syst Rev2016;5:48. doi:10.1186/s13643-016-0226-4. pmid:27012940
      OpenUrlCrossRefPubMed
    44. ↵
      1. Saltaji H,
      2. Armijo-Olivo S,
      3. Cummings GG,
      4. Amin M,
      5. da Costa BR,
      6. Flores-Mir C
      . Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions. BMC Med Res Methodol2018;18:42. doi:10.1186/s12874-018-0491-0. pmid:29776394
      OpenUrlCrossRefPubMed
    45. ↵
      1. Lang TA,
      2. Stroup DF
      . Who knew? The misleading specificity of “double-blind” and what to do about it. Trials2020;21:697. doi:10.1186/s13063-020-04607-5. pmid:32758278
      OpenUrlCrossRefPubMed
    46. ↵
      1. Gamble C,
      2. Krishan A,
      3. Stocken D,
      4. et al
      . Guidelines for the content of statistical analysis plans in clinical trials. JAMA2017;318:2337-43. doi:10.1001/jama.2017.18556. pmid:29260229
      OpenUrlCrossRefPubMed
    47. ↵
      1. Le Noury J,
      2. Nardo JM,
      3. Healy D,
      4. et al
      . Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ2015;351:h4320. doi:10.1136/bmj.h4320. pmid:26376805
      OpenUrlAbstract/FREE Full Text
    48. ↵
      1. Schroll JB,
      2. Maund E,
      3. Gøtzsche PC
      . Challenges in coding adverse events in clinical trials: a systematic review. PLoS One2012;7:e41174. doi:10.1371/journal.pone.0041174. pmid:22911755
      OpenUrlCrossRefPubMed
    49. ↵
      1. Laursen DRT,
      2. Paludan-Müller AS,
      3. Hróbjartsson A
      . Randomized clinical trials with run-in periods: frequency, characteristics and reporting. Clin Epidemiol2019;11:169-84. doi:10.2147/CLEP.S188752. pmid:30809104
      OpenUrlCrossRefPubMed
    50. ↵
      1. Collister D,
      2. Rodrigues JC,
      3. Mbuagbaw L,
      4. et al
      . Prerandomization run-in periods in randomized controlled trials of chronic diseases: a methodological study. J Clin Epidemiol2020;128:148-56. doi:10.1016/j.jclinepi.2020.09.035. pmid:33002638
      OpenUrlCrossRefPubMed
    51. ↵
      1. Huo X,
      2. Armitage J
      . Use of run-in periods in randomized trials. JAMA2020;324:188-9. doi:10.1001/jama.2020.6463. pmid:32662850
      OpenUrlCrossRefPubMed
    52. ↵
      1. Gupta SK
      . Intention-to-treat concept: a review. Perspect Clin Res2011;2:109-12. doi:10.4103/2229-3485.83221. pmid:21897887
      OpenUrlCrossRefPubMed
    53. ↵
      1. Gravel J,
      2. Opatrny L,
      3. Shapiro S
      . The intention-to-treat approach in randomized controlled trials: are authors saying what they do and doing what they say?Clin Trials2007;4:350-6. doi:10.1177/1740774507081223. pmid:17848496
      OpenUrlCrossRefPubMedWeb of Science
    54. ↵
      1. Cornelius V,
      2. Cro S,
      3. Phillips R
      . Advantages of visualisations to evaluate and communicate adverse event information in randomised controlled trials. Trials2020;21:1028. doi:10.1186/s13063-020-04903-0. pmid:33353566
      OpenUrlCrossRefPubMed
    55. ↵
      1. Phillips R,
      2. Cro S,
      3. Wheeler G,
      4. et al
      . Visualising harms in randomised controlled trial publications: consensus and recommendations. BMJ2022;377doi:10.1136/bmj-2021-068983.
      OpenUrlAbstract/FREE Full Text
    56. ↵
      1. Loke YK,
      2. Mattishent K
      . If nothing happens, is everything all right? Distinguishing genuine reassurance from a false sense of security. CMAJ2015;187:15-6. doi:10.1503/cmaj.141344. pmid:25404394
      OpenUrlFREE Full Text
    57. ↵
      1. Petrie KJ,
      2. Rief W
      . Psychobiological mechanisms of placebo and nocebo effects: pathways to improve treatments and reduce side effects. Annu Rev Psychol2019;70:599-625. doi:10.1146/annurev-psych-010418-102907. pmid:30110575
      OpenUrlCrossRefPubMed
    58. ↵
      1. Faasse K,
      2. Huynh A,
      3. Pearson S,
      4. Geers AL,
      5. Helfer SG,
      6. Colagiuri B
      . the influence of side effect information framing on nocebo effects. Ann Behav Med2019;53:621-9. doi:10.1093/abm/kay071. pmid:30204841
      OpenUrlCrossRefPubMed
    59. ↵
      1. Mondaini N,
      2. Gontero P,
      3. Giubilei G,
      4. et al
      . Finasteride 5 mg and sexual side effects: how many of these are related to a nocebo phenomenon?. J Sex Med2007;4:1708-12. doi:10.1111/j.1743-6109.2007.00563.x. pmid:17655657
      OpenUrlCrossRefPubMedWeb of Science
    60. ↵
      1. Colagiuri B,
      2. McGuinness K,
      3. Boakes RA,
      4. Butow PN
      . Warning about side effects can increase their occurrence: an experimental model using placebo treatment for sleep difficulty. J Psychopharmacol2012;26:1540-7. doi:10.1177/0269881112458730. pmid:22992377
      OpenUrlCrossRefPubMed
    61. ↵
      1. Leibowitz KA,
      2. Howe LC,
      3. Crum AJ
      . Changing mindsets about side effects. BMJ Open2021;11:e040134. doi:10.1136/bmjopen-2020-040134. pmid:33526496
      OpenUrlCrossRefPubMed
    62. ↵
      1. Golder S,
      2. McIntosh HM,
      3. Duffy S,
      4. Glanville J,
      5. Centre for Reviews and Dissemination and UK Cochrane Centre Search Filters Design Group
      . Developing efficient search strategies to identify reports of adverse effects in MEDLINE and EMBASE. Health Info Libr J2006;23:3-12. doi:10.1111/j.1471-1842.2006.00634.x. pmid:16466494
      OpenUrlCrossRefPubMedWeb of Science
    63. ↵
      1. Pitrou I,
      2. Boutron I,
      3. Ahmad N,
      4. Ravaud P
      . Reporting of safety results in published reports of randomized controlled trials. Arch Intern Med2009;169:1756-61. doi:10.1001/archinternmed.2009.306. pmid:19858432
      OpenUrlCrossRefPubMedWeb of Science
    64. ↵
      1. Schroll JB,
      2. Penninga EI,
      3. Gøtzsche PC
      . Assessment of adverse events in protocols, clinical study reports, and published papers of trials of orlistat: a document analysis. PLoS Med2016;13:e1002101. doi:10.1371/journal.pmed.1002101. pmid:27529343
      OpenUrlCrossRefPubMed
    65. ↵
      1. Hodkinson A,
      2. Gamble C,
      3. Smith CT
      . Reporting of harms outcomes: a comparison of journal publications with unpublished clinical study reports of orlistat trials. Trials2016;17:207. doi:10.1186/s13063-016-1327-z. pmid:27529343
      OpenUrlCrossRefPubMed
    66. ↵
      1. Moskowitz A,
      2. Andersen LW,
      3. Holmberg MJ,
      4. Grossestreuer AV,
      5. Berg KM,
      6. Granfeldt A
      . Identification, collection, and reporting of harms among non-industry-sponsored randomized clinical trials of pharmacologic interventions in the critically ill population: a systematic review. Crit Care2020;24:398. doi:10.1186/s13054-020-03113-z. pmid:32641148
      OpenUrlCrossRefPubMed
    67. ↵
      1. Rodgers MA,
      2. Brown JV,
      3. Heirs MK,
      4. et al
      . Reporting of industry funded study outcome data: comparison of confidential and published data on the safety and effectiveness of rhBMP-2 for spinal fusion. BMJ2013;346:f3981. doi:10.1136/bmj.f3981. pmid:23788229
      OpenUrlAbstract/FREE Full Text
    68. ↵
      1. Wilkinson MD,
      2. Dumontier M,
      3. Aalbersberg IJ,
      4. et al
      . The FAIR Guiding Principles for scientific data management and stewardship. Sci Data2016;3:160018. doi:10.1038/sdata.2016.18. pmid:26978244
      OpenUrlCrossRefPubMed
    69. ↵
      1. Favier R,
      2. Crépin S
      . The reporting of harms in publications on randomized controlled trials funded by the “Programme Hospitalier de Recherche Clinique,” a French academic funding scheme. Clin Trials2018;15:257-67. doi:10.1177/1740774518760565. pmid:29498543
      OpenUrlCrossRefPubMed
    70. ↵
      1. McGauran N,
      2. Wieseler B,
      3. Kreis J,
      4. Schüler YB,
      5. Kölsch H,
      6. Kaiser T
      . Reporting bias in medical research—a narrative review. Trials2010;11:37. doi:10.1186/1745-6215-11-37. pmid:20388211
      OpenUrlCrossRefPubMed
    71. ↵
      1. Vedula SS,
      2. Li T,
      3. Dickersin K
      . Differences in reporting of analyses in internal company documents versus published trial reports: comparisons in industry-sponsored trials in off-label uses of gabapentin. PLoS Med2013;10:e1001378. doi:10.1371/journal.pmed.1001378. pmid:23382656
      OpenUrlCrossRefPubMed
    72. ↵
      1. Hum SW,
      2. Golder S,
      3. Shaikh N
      . Inadequate harms reporting in randomized control trials of antibiotics for pediatric acute otitis media: a systematic review. Drug Saf2018;41:933-8. doi:10.1007/s40264-018-0680-0. pmid:29737504
      OpenUrlCrossRefPubMed
    73. ↵
      1. de Vries YA,
      2. Roest AM,
      3. Beijers L,
      4. Turner EH,
      5. de Jonge P
      . Bias in the reporting of harms in clinical trials of second-generation antidepressants for depression and anxiety: a meta-analysis. Eur Neuropsychopharmacol2016;26:1752-9. doi:10.1016/j.euroneuro.2016.09.370. pmid:27659240
      OpenUrlCrossRefPubMed
    74. ↵
      1. Junqueira DR,
      2. Phillips R,
      3. Zorzela L,
      4. et al
      . Time to improve the reporting of harms in randomized controlled trials. J Clin Epidemiol2021;136:216-20. doi:10.1016/j.jclinepi.2021.04.020. pmid:33984494
      OpenUrlCrossRefPubMed
    75. ↵
      1. Neogi SB,
      2. Devasenapathy N,
      3. Singh R,
      4. et al
      . Safety and effectiveness of intravenous iron sucrose versus standard oral iron therapy in pregnant women with moderate-to-severe anaemia in India: a multicentre, open-label, phase 3, randomised, controlled trial. Lancet Glob Health2019;7:e1706-16. doi:10.1016/S2214-109X(19)30427-9. pmid:31708151
      OpenUrlCrossRefPubMed
    76. ↵
      1. Jang YK,
      2. Kim NY,
      3. Lee JS,
      4. et al
      . Comparison of postoperative pain and adverse effects between variable-rate feedback infusion and conventional fixed-rate basal infusion modes of patient-controlled epidural analgesia following open gastrectomy: a randomized controlled trial. Int J Environ Res Public Health2021;18:8777. doi:10.3390/ijerph18168777. pmid:34444526
      OpenUrlCrossRefPubMed
    77. ↵
      1. Terra RM,
      2. Araujo PHXN,
      3. Lauricella LL,
      4. Campos JRM,
      5. Trindade JRM,
      6. Pêgo-Fernandes PM
      . A Brazilian randomized study: robotic-assisted vs. video-assisted lung lobectomy outcomes (BRAVO trial). J Bras Pneumol2022;48:e20210464. doi:10.36416/1806-3756/e20210464. pmid:35830078
      OpenUrlCrossRefPubMed
    78. ↵
      1. Maspero J,
      2. Agache IO,
      3. Kamei T,
      4. et al
      . Long-term safety and exploratory efficacy of fevipiprant in patients with inadequately controlled asthma: the SPIRIT randomised clinical trial. Respir Res2021;22:311. doi:10.1186/s12931-021-01904-8. pmid:34895218
      OpenUrlCrossRefPubMed
    79. ↵
      1. Savarirayan R,
      2. Tofts L,
      3. Irving M,
      4. et al
      . Once-daily, subcutaneous vosoritide therapy in children with achondroplasia: a randomised, double-blind, phase 3, placebo-controlled, multicentre trial. Lancet2020;396:684-92. doi:10.1016/S0140-6736(20)31541-5. pmid:32891212
      OpenUrlCrossRefPubMed
    80. ↵
      1. Predel HG,
      2. Ebel-Bitoun C,
      3. Lange R,
      4. Weiser T
      . A randomized, placebo- and active-controlled, multi-country, multi-center parallel group trial to evaluate the efficacy and safety of a fixed-dose combination of 400 mg ibuprofen and 100 mg caffeine compared with ibuprofen 400 mg and placebo in patients with acute lower back or neck pain. J Pain Res2019;12:2771-83. doi:10.2147/JPR.S217045. pmid:31576162
      OpenUrlCrossRefPubMed
    81. ↵
      1. Krege JH,
      2. Rizzoli PB,
      3. Liffick E,
      4. et al
      . Safety findings from Phase 3 lasmiditan studies for acute treatment of migraine: Results from SAMURAI and SPARTAN. Cephalalgia2019;39:957-66. doi:10.1177/0333102419855080. pmid:31166697
      OpenUrlCrossRefPubMed
    82. ↵
      1. Hernandez AF,
      2. Green JB,
      3. Janmohamed S,
      4. et al.,
      5. Harmony Outcomes committees and investigators
      . Albiglutide and cardiovascular outcomes in patients with type 2 diabetes and cardiovascular disease (Harmony Outcomes): a double-blind, randomised placebo-controlled trial. Lancet2018;392:1519-29. doi:10.1016/S0140-6736(18)32261-X. pmid:30291013
      OpenUrlCrossRefPubMed
    83. ↵
      1. Solomon SD,
      2. McMurray JJV,
      3. Claggett B,
      4. et al.,
      5. DELIVER Trial Committees and Investigators
      . Dapagliflozin in heart failure with mildly reduced or preserved ejection fraction. N Engl J Med2022;387:1089-98. doi:10.1056/NEJMoa2206286. pmid:36027570
      OpenUrlCrossRefPubMed
    84. ↵
      1. Coomarasamy A,
      2. Devall AJ,
      3. Cheed V,
      4. et al
      . A randomized trial of progesterone in women with bleeding in early pregnancy. N Engl J Med2019;380:1815-24. doi:10.1056/NEJMoa1813730. pmid:31067371
      OpenUrlCrossRefPubMed
    85. ↵
      1. Nave AH,
      2. Rackoll T,
      3. Grittner U,
      4. et al
      . Physical Fitness Training in Patients with Subacute Stroke (PHYS-STROKE): multicentre, randomised controlled, endpoint blinded trial. BMJ2019;366:l5101. doi:10.1136/bmj.l5101. pmid:31533934
      OpenUrlAbstract/FREE Full Text
    Back to top