Early phase clinical trials extension to guidelines for the content of statistical analysis plans
BMJ 2022; 376 doi: https://doi.org/10.1136/bmj-2021-068177 (Published 07 February 2022) Cite this as: BMJ 2022;376:e068177- Victoria Homer, biostatistician1,
- Christina Yap, professor of clinical trials biostatistics2,
- Simon Bond, lead senior statistician3,
- Jane Holmes, senior medical statistician4,
- Deborah Stocken, professor of clinical trials research5,
- Katrina Walker, senior medical statistician5,
- Emily J Robinson, lead statistician6,
- Graham Wheeler, senior lecturer in clinical trials statistics7,
- Sarah Brown, principal statistician5,
- Samantha Hinsley, lead biostatistician8,
- Matthew Schipper, research professor of biostatistics9,
- Christopher J Weir, professor of medical statistics and clinical trials10,
- Khadija Rantell, statistical adviser11,
- Thomas Prior, principal statistician12,
- Ly-Mee Yu, associate professor13,
- John Kirkpatrick, senior principal statistician14,
- Alun Bedding, associate director14,
- Carrol Gamble, professor of medical statistics15,
- Piers Gaunt
, principal biostatistician1
- 1Cancer Research Clinical Trials Unit, University of Birmingham, Birmingham, UK
- 2Clinical Trials and Statistics Unit, Institute for Cancer Research, London, UK
- 3Cambridge Clinical Trials Unit, Cambridge, UK
- 4Oxford Clinical Trials Research Unit, Centre for Statistics in Medicine, Nuffield Department of Orthopaedics, Rheumatology and Musculoskeletal Sciences, University of Oxford, Oxford, UK
- 5Clinical Trials Research Unit, Leeds Institute of Clinical Trials Research, University of Leeds, Leeds, UK
- 6Royal Marsden Clinical Trials Unit, Royal Marsden NHS Foundation Trust, London, UK
- 7Imperial Clinical Trials Unit, Imperial College London, London, UK
- 8Cancer Research UK Glasgow Clinical Trials Unit, University of Glasgow, Glasgow, UK
- 9Departments of Radiation Oncology and Biostatistics, University of Michigan, Ann Arbor, MI, USA
- 10Edinburgh Clinical Trials Unit, Usher Institute, University of Edinburgh, Edinburgh, UK
- 11Medicines and Healthcare products Regulatory Agency, London, UK
- 12Early Development Oncology Statistics Department, Janssen Research and Development, Spring House, PA, USA
- 13Primary Care Clinical Trials Unit, University of Oxford, Oxford, UK
- 14Roche Products, Welwyn Garden City, UK
- 15Liverpool Clinical Trials Centre, University of Liverpool, Liverpool, UK
- Correspondence to: P Gaunt p.gaunt{at}bham.ac.uk
- Accepted 10 January 2022
This paper details guidelines for the content of statistical analysis plans (SAPs) for early phase clinical trials, presenting an extension to the guidelines for the content of SAPs in clinical trials by Gamble et al.1 Early phase clinical trials (phase I and non-randomised phase II) aim to determine the safety and initial indicators of efficacy of interventions before conducting phase III clinical trials, which have the potential to change practice. The undertaking of definitive late phase clinical trials is often a lengthy and costly process, because they ensure full scale evaluation of the interventions and can also include cost effectiveness analyses. Definitive clinical trials are predicated on accurate and robust conclusions from early phase clinical trials, with flaws in design and analysis potentially leading to interventions failing to demonstrate a benefit in phase III clinical trials. Consequently, the design, conduct, and analysis of early phase clinical trials does not solely affect that specific study. Conclusions from early phase clinical trials have implications for all related subsequent clinical trials. As such, these studies must be performed to the highest standards of rigour and quality, to ensure that correct decisions are taken forward.
Historically, phase I clinical trials were conducted without extensive statistical involvement and conformed to rule based designs (eg, the 3+3 design to determine the maximum tolerated dose).23 Recent recommendations propose that phase I studies should use model based designs4 such as the continual reassessment method,5678 or model assisted designs such as a modified toxicity probability interval design.9 Randomised dose finding phase I clinical trials (eg, that randomise to attain the optimal doses or dose schedules once safety has been assured10) and single arm phase II designs11 are also being used, all of which require considerable statistical input before, during, and at the analysis stage of the clinical trial. Use of these trial designs with greater statistical involvement has become more prevalent, accelerated by oncology clinical trials,12 however, examples are emerging across other disease areas.13
The International Council for Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use has provided the E9 guidelines (ICH E9), which state that “although the early phases of drug development consist mainly of clinical trials that are exploratory in nature, statistical principles are also relevant.”14 Because early phase clinical trials use statistical model based designs, the requirement for good quality SAPs, including additional statistical parameters and progression criteria to later phase research, becomes an even greater necessity,1516 with trial statisticians having a key role in designing, and undertaking analysis of, early phase clinical trials.
Guidelines for the content of SAPs by Gamble et al were published in 20171 and highlighted the need for a detailed SAP to improve transparency, clinical trial quality, and accuracy. These guidelines were developed with the primary intention of being applicable to the analyses of randomised controlled trials at later phases, and acknowledged that despite some recommendations being transferable, specific consideration and guidance are needed for early phase clinical trials. These guidelines were discussed at a statisticians operational group meeting of the UK Clinical Research Collaboration (UKCRC) registered Clinical Trials Unit (CTU) network in April 2018, confirming that specific consideration and guidance for early phase clinical trials was an area of unmet need. This need was based on the fact that early phase clinical trials often are not randomised, use adaptive designs, and otherwise have statistical considerations and requirements that differ in character from those of later phase, randomised controlled trials. This discussion led to this extension of those 2017 guidelines to fulfil the needs and considerations of SAPs for early phase clinical trials. Given the drug development pathway, early phase trials are more prevalent than late phase trials,17 highlighting the importance and impact of this guidance.
In this article, we describe the development of an extension to published guidelines for SAP content to broaden their applicability to early phase clinical trials. These recommendations aim to guide the authors of SAPs for all early phase studies, irrespective of the study design used (rule based, model based, model assisted, or randomised phase I trials; or single arm phase II designs). Beyond the scope of this extension are randomised phase II trials, given that they are covered by the existing SAP guidelines for randomised trials.
Summary points
Guidance for the content of statistical analysis plans (SAPs) for clinical trials was published in 2017 and focused on late phase, randomised controlled trials
The existing guidelines have been extended to broaden their applicability to early phase (phase I and non-randomised phase II) clinical trials
This extension is based on existing guidance; a comprehensive search to identify existing published protocols, SAPs, and SAP guidance; a survey of clinical trial funders and regulators; a survey of current practice by statisticians within Clinical Trials Units registered with the UK Clinical Research Collaboration; a critical appraisal and expert review meeting; and a pilot of the proposed guidelines
Of 55 original items in the current SAP content guidance, 30 have remained unchanged, 25 have been modified, and a further 11 new items have been proposed to ensure comprehensive and appropriate guidance for early phase clinical trials
Development of the guidance extension
This guidance document encapsulates the findings of a comprehensive search to identify existing published protocols, SAPs, and SAP guidance; a survey of clinical trial funders and regulators; a survey of current practice by statisticians within CTUs registered with the UKCRC; a critical appraisal and expert review meeting; and a pilot of the proposed guidelines. An overview of this process is given in appendix 1.
Literature review of existing guidance
A literature review was undertaken to identify peer reviewed publications of applicable guidelines, and example clinical trial protocols and SAPs (appendix 2 lists the search terms). Two statisticians searched the EQUATOR (enhancing the quality and transparency of health research) network repository for existing guidance, and PubMed for published SAPs and for protocols of early phase clinical trials (at phases I and II). Protocols were included in the search for statistical analysis information, as SAPs might not always have been written for some early phase clinical trials.
These searches were performed independently by two statisticians, with all papers categorised as a relevant and methodological paper, relevant and published protocol or SAP, or not relevant. Papers were classified as not relevant because they included late phase trials, cluster randomised trials, epidemiological studies, meta-analyses and systematic reviews, results papers, editorial publications, or other reasons of non-relevance decided at the discretion of the reviewers (eg, methodological papers, and SAPs for literature reviews). Any discrepancies regarding relevance categorisation were discussed and resolved by mutual agreement.
The literature review was initially performed in November 2019 and updated in October 2020. Of 610 papers returned by the literature review, 500 were excluded owing to non-relevance. Figure 1 shows the number of papers categorised according to each exclusion. The other reasons for exclusion included statistical and clinical trials methodology and literature reviews, SAPs for substudies of randomised controlled trials, and SAPs for substudies. The SAP and protocol search identified two papers perceived to be SAP guidelines: the first being the original guidelines for the content of SAPs,1 and the second being an overview to the typical content of SAPs for various study types (eg, randomised controlled trials and observational studies) rather than proposing recommended content for SAPs.18 The literature review found three published SAPs for early phase trials192021 and 105 published early phase protocols containing some statistical content. The three published early phase SAPs were all single arm, phase II clinical trials. Of the 105 published protocols, one included an SAP as an appendix,22 seven indicated that a separate SAP had been written (but did not make it available), and the remainder typically contained varying but limited statistical content. These findings are consistent with the notion that while the publication of SAPs has increased, these plans are overwhelmingly for late phase clinical trials.
Results of literature review (as of 22 October 2020) of existing guidance for content of statistical analysis plans (SAPs). *Other reasons of non-relevance include statistical and clinical trials methodology and literature reviews, statistical analysis plans for substudies of randomised controlled trials, and statistical analysis plans for substudies
Survey of clinical trial funders and regulators
The same clinical trial funders and regulators contacted during the original SAP guidance development1 were contacted via email in January 2020. Funders were initially contacted to gauge whether they fund early phase clinical trials. If a response was not received, up to two further reminder emails were sent. Surveys were sent to all regulators and those organisations who confirmed the scope of their funding considerations would extend to early phase clinical trials. Consultation with clinical trial funders led to the identification of two additional dedicated early phase clinical trial funders who were also approached. A list of organisations contacted is provided in appendix 3. The goal of these surveys was to ascertain funding and regulatory requirements of design, analysis, publication, and SAP contents for early phase clinical trials.
Of the 39 funding institutions contacted, 28 responded to our request for information, of which 15 indicated that they would fund early phase research. The European Medicines Agency and Medicines and Healthcare products Regulatory Agency provided their regulatory requirements for early phase clinical trials. Additionally, guidance was reviewed that pertained to the running and conduct of early phase clinical trials, such as multiple ICH documents,14232425 the Royal Statistical Society working party report on statistical issues in first-in-human studies,26 various consolidated standards of reporting trials (CONSORT) statements,27282930 and publicly available regulatory guidance.31 The predominant documents that funders referred to were ICH E914 and existing late phase guidance.1 The prevailing findings were that:
Dose escalation decisions, stopping criteria, and interim go-no-go criteria are often poorly documented, potentially resulting in ambiguous and non-robust decisions to escalate
The statistical design and analysis plan are often not clearly justified
Where model based approaches are used, the choice of model should be clearly justified, and the risk of overdosing must be quantified and justified to be acceptable, supported by simulation where applicable
Appropriate sample sizes for early phase clinical trials can be better justified statistically (eg, by simulation) as opposed to by mere custom and historical practice, and that sufficient detail regarding the sample size should be included (with supplemented code to help facilitate this where appropriate) to allow for full replication
The programming code for modelling should either be suitably referenced or made available in the SAP so that escalation decisions can be replicated and reproduced
Standard statistical principles (eg, the implication of interim analyses on the overall integrity of the clinical trial and type I error control) are still applicable.
Survey of Clinical Trials Units
The survey was developed based on the original SAP guidance survey1 and tailored to early phase clinical trials. The aim was to identify CTUs conducting early phase clinical trials and the current practice within those units for developing SAPs. The survey (appendix 3) was circulated to CTUs in the UKCRC network. A list of the 53 registered CTUs was accessed (May 2020) from the UKCRC website32 to cross check responders.
A senior statistician at each CTU was asked to complete the survey to reflect practices and majority opinion within the statistician’s CTU in May 2020. If no response was received then two reminder emails were sent via the mailing list to encourage responses, and then contacts at the unit were approached directly for a response. Example SAPs shared by CTUs conducting early phase clinical trials were collated and reviewed for content to establish the current level of detail provided. To ensure as much coverage for study design types and disease areas, examples were sought from multiple scenarios, including design based (eg, rule based, model based, or single arm phase II) and disease based (oncology or non-oncology) scenarios.
Of the 53 CTUs, 40 (75%) responded to the survey about their experiences of SAPs for early phase clinical trials. Of the 40 responders, 21 (53%) declared to design and analyse early phase clinical trials. On enquiry, the remaining 13 units who did not respond did not conduct early phase clinical trials and so were not pursued further. The prevailing practice of the 21 units statistically designing and conducting early phase clinical trials at the time of survey circulation was to have a generic template (or set of instructions) applicable to all phases of clinical trial, without specific instructions or sections for early phase clinical trials (n=15, 71%). Three (14%) units reported having no template. Importantly, only three (14%) units reported having a template specific to early phase clinical trials. Most units (n=20, 95%) highlighted the desire for early phase SAP guidance, and all indicated that they would use the guidance extension if it existed.
Critical appraisal and expert review meeting
The first draft of the guidance was produced using the literature reviews, funding and regulatory requirements, CTU examples, and authors’ experiences. An expert review panel from the United Kingdom and United States was convened, consisting of academic, pharmaceutical, NHS (UK only), and regulatory representatives (appendix 4).
The panel met virtually on 26 October 2020 with contribution and attendance from 16 statisticians from 14 organisations. Participants critically appraised the first draft of the extension in advance of the meeting in preparation of wider discussion with the group. Considerable discussion centred on the level of detail required from the guidelines and areas that required expansion or clarification. Comments were received from meeting attendees, and a further five statisticians who were unable to attend the meeting. Consensus was reached at this meeting regarding several areas for inclusion of content and recommendations. After incorporation of comments, attendees reviewed and provided feedback before finalising the draft guidance extension for piloting within CTUs.
Piloting of the early phase trial extension
After the expert review meeting, the guidelines extension was updated and piloted at six CTUs in the UK. The aim of the pilot was to ensure the guidance extension produced was fit for purpose, was appropriate to the needs of statisticians writing early phase clinical trial SAPs, and to identify any items requiring further clarification. The pilot was conducted between December 2020 to March 2021 whereby participating CTUs were invited to give formal feedback via email. Piloting covered both phase I and phase II clinical trial designs, and different disease areas. Feedback was universally positive, with minimal amendments (improved wording and clarifications) made to the guidelines extension, which were then finalised.
Ethics
Consistent with the development of the original SAP guidance, ethical approval was not sought for the distribution of surveys; instead, consent to take part was indicated by survey participation.
Early phase extension to existing SAP guidance
The resulting recommendations for the extension to existing SAP content guidance for early phase clinical trials are presented in table 1 (and supplementary checklist 1). Of 55 items proposed in the original SAP content guidance,1 30 items have remained unchanged, 25 items have been modified to better reflect early phase clinical trials, and a further 11 new items have been proposed. Substantial modifications and new items include:
Extension of content guidance for statistical analysis plans (SAP) from early phase clinical trials
Increased details regarding statistical design methodology, and model choice where appropriate
Update of outcome definitions to include definition of estimands in line with the principles outlined in ICH E9 (R1)
Inclusion of simulation reports incorporating operating characteristics, to justify statistical design or sample size where applicable
Inclusion of code required for novel methodology
Inclusion of dose transition pathways, where appropriate
Amendments to wording to be more neutral to both frequentist and bayesian methodology, to reflect that some early phase clinical trials designs, particularly phase I, are underpinned by bayesian methods.
Minor changes were made to items including updates to the descriptions to ensure pertinence to early phase clinical trials. Items with no comments in the “Recommended early phase clinical trials extension” column in table 1 (and supplementary checklist 1) indicates that the original item is appropriate and covers the necessary content for early phase clinical trials. An elaboration of each item within the extension guidelines is included in appendix 5, with example text covering various early phase clinical trial designs and disease areas. These examples are intended to be illustrative and are not an endorsement of the methods described.
Discussion
Critical appraisal of clinical trials is only possible if their design, conduct, and analysis are predefined thoroughly and clearly described. Planned trial analyses must be suitably predefined, typically in an SAP. Predefinition of analyses increases the credibility of results by minimising the opportunity of making data driven decisions, or by selecting estimands or methodology to produce a more positive trial outcome. With increased focus on transparency and concerns regarding reproducibility of results alongside the ability to reconstruct clinical trial design and analyses, guidance has been produced for the prospective reporting of SAPs for randomised controlled trials,33 which advocates that SAPs should be made publicly available. However, this guidance has generally focused on late phase clinical trials and the needs of early phase clinical trials have not explicitly been considered, acknowledging the need for extensions.1 The absence of transparency for early phase clinical trials and guidance tailored to these designs could result in biased results, which could in turn misinform decision making in clinical development. This SAP guidance extension has been developed to enable statisticians, triallists, and clinical investigators conducting early phase clinical trials to produce clear and relevant SAPs for those clinical trials. The desired outcome is that these clinical trials would be run with increased transparency and increase the likelihood of accurate conclusions.
A major update of the extension is the development of wording on outcome measures and estimands (appendix 5, section 6, item 26), following the ICH E9 (R1) addendum on estimands and sensitivity analyses in clinical trials.25 Although ICH E9 (R1) focuses on the analysis and interpretability of late phase trials, it clearly states that the same principles apply to single arm trials, therefore encompassing early phase designs. At the time of publication, estimands are not widely used in early phase clinical trials with working groups convened specifically to advise on this issue (eg, the Oncology Estimand Working Group34 and Estimand Implementation Working Group35). These groups will provide or publish guidance and examples pertaining to estimands for early phase clinical trials, which should be incorporated into the writing of SAPs in the future, and which might also necessitate revisions to this document.
Another important component to the extension is the simulation report template. All phases of clinical trials need to ensure that the trial will yield an accurate, unbiased result. For fixed or late phase clinical trials, this component typically manifests as a formal sample size calculation. However, for model assisted and model based designs of early phase trials, this calculation is not appropriate. Instead, simulations to assess the designs operating characteristics are needed to ensure that the clinical trial will yield a result and provide sufficient overdose control.36 A template for the suitable simulation report has not been developed as part of this project as they depend on multiple variables including disease area, trial question, trial design, and methodology. Instead, suggestions for content are provided in appendix 5, item 33. The same rationale applies to relevant code and a reports template, with suggestions provided in appendix 5, items 35 and 36, respectively.
In early phase clinical trials, critical decisions regarding the trial progression often need to be made at multiple time points, potentially as early as after the first patient has completed a specified evaluation period. Therefore, compared to late phase clinical trials, SAPs for early phase clinical trials will generally need to be written earlier. At the expert review meeting, the panel acknowledged that the first version of the SAP should be signed off before the trial opening, but this is not always feasible. The panel then recommended finalising the first version of the SAP before the first analysis of clinical trial data, for example, before evaluation of a potential first dose escalation or any interim analysis.
The panel acknowledged that the SAP should encompass all relevant points from table 1 (and supplementary checklist 1) but recommended that signposting should be used to indicate location of details captured elsewhere in documents such as in the clinical trial protocol or a simulation report, to avoid replication. The SAP is not a standalone document and should be read in conjunction with other trial related documentation.
In some scenarios (eg, rule based designs), an SAP might not be required for early phase clinical trials—for instance, if all the applicable content according to these guidelines is sufficiently detailed in the protocol. However, any minor change to that content as the clinical trial progresses would then necessitate a protocol amendment. The combination of running an efficient clinical trial and the level of detail appropriate for analysis of early phase clinical trials provided within this content justifies a standalone SAP.
Although not all UKCRC registered CTUs responded to our survey, this extension captures the opinions of all CTUs who design and analyse early phase clinical trials. Our work built on the original SAP guidelines, which included a Delphi survey.37 In developing the extension to early phase clinical trials, an additional Delphi survey was not considered necessary because our aim was to build on existing knowledge rather than requiring repetition of the process. Instead, we ensured that all relevant stakeholder groups (including CTUs, funders, and regulators) were included in the elicitation of information with the expert multidisciplinary panel assuming responsibility of ensuring detailed considerations of the produced guidelines.
Our expert panel comprised of more academic statisticians than regulatory or pharmaceutical statisticians. This under-representation was identified and more representatives from these areas were invited to participate on the panel. In addition, some individuals provided comments and responded to surveys but were not able to participate in the expert panel. Importantly, every expert panel member was invited to provide their opinions outside of the meeting and review the proposed guideline extension independently and share with their colleagues. When these views were combined, all views were weighted equally regardless of further participation in the extension’s development, and consideration was taken in ensuring that the opinions of all stakeholder groups were reflected in the final version, which all authors have contributed to and agreed on.
Given the multitude and complexity of designs for early phase clinical trials, this guidance was developed to be as generic and applicable as possible across all designs. A proportionate approach was taken, striking a balance between an increased number of items but not increasing beyond the minimum number of items needed to cover adequately the diverse features of early phase clinical trial designs and analysis requirements. Methodological developments within early phase clinical trials are increasingly prevalent, and therefore continued progress in this area is expected. The guidance might need to be updated when developments in statistical techniques emerge, and as such methodology and guidance documentation around early phase clinical trials might need to be reviewed periodically for necessary updates. Certain types of trials, such as bayesian or adaptive trials, are not covered by this extension or the original guidance and will require additional considerations incorporating available regulatory and published guidance. This guidance document provides a necessary extension to the SAP guidelines paper published by Gamble et al1 for early phase clinical trials. Adherence to this extended guidance will support those individuals working in early phase clinical trials in producing robust conclusions to ensure that correct decisions are taken forward.
Acknowledgments
We thank Peter Brocklehurst and Birmingham Clinical Trials Unit for their assistance in the initiation and support of this project; all UK Clinical Research Collaboration (UKCRC) registered Clinical Trials Units responding to our surveys and providing example documents; the various Clinical Trials Units (Institute of Cancer Research, Cambridge Clinical Trials Unit, Leeds Clinical Trials Research Unit, Royal Marsden Clinical Trials Unit, Centre for Statistics in Medicine at the University of Oxford, and Cancer Research Clinical Trials Unit) and the individuals at these units (Muneeb Mahmud and Jan Rekowski (Institute of Cancer Research), Simran Vaja (Cambridge Clinical Trials Unit), Amy Coulson (Leeds Clinical Trials Research Unit), Catey Bunce and Kabir Mohammed (Royal Marsden Clinical Trials Unit), Cathy Qi (Centre for Statistics in Medicine at the University of Oxford), and Amit Patel and Shanna Maycock (Cancer Research Clinical Trials Unit)) for piloting the guidelines; Louise Stanton and Louise Williams for facilitating communication with the UKCRC Clinical Trials Unit Network; Siân Lax (Cancer Research Clinical Trials Unit) for her help in preparing this manuscript; and all the individuals from funding and regulatory institutions who responded to our surveys.
Footnotes
Contributors: PG, CY, SB, DS, and CG wrote and were awarded the grant funding. VH and PG performed the literature review and wrote the survey, guidance, and manuscript. All authors attended the expert review meeting or provided comments; contributed to the underpinning research, and subsequent drafts of the guidance and manuscript; and approved the final version. VH and PG act as guarantors. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: This project is funded by the National Institute for Health Research (NIHR) Clinical Trials Unit Support Funding scheme. The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. This project was supported and endorsed by the UK Clinical Research Collaboration Registered Clinical Trials Unit Network. Staff at the Cancer Research Clinical Trials Unit are supported by a core funding grant from Cancer Research UK (C22436/A25354 and C22436/A28028), and further grants from Birmingham Biomedical Research Centre (funded by the NIHR) and Advanced Therapy Treatment Centre (funded by Innovate UK).
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: support from the NIHR for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. There is no additional data available.
Provenance and peer review: Not commissioned; externally peer reviewed.
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.