Development of ROBUST-RCT: Risk Of Bias instrument for Use in SysTematic reviews-for Randomised Controlled Trials
BMJ 2025; 388 doi: https://doi.org/10.1136/bmj-2024-081199 (Published 25 March 2025) Cite this as: BMJ 2025;388:e081199
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Dear Editor
Wang et al. 2025 introduce a critical tool aimed at assessing bias risk in RCTs as far as systemic reviews go. The ROBUST-RCT seeks to attain an equilibrium between methodological rigor with simplicity, thus addressing limitations in existing instruments [1].
Moreover, this tool offers simplicity and practicability. The structure and framework are straightforward and remain ideal for users without extensive methodological training. This approach ensures applicability across different systematic review contexts [1]. Also, the article provides a comprehensive development process that improves efficiency and utility.
Nevertheless, the following areas should be considered for improvement. Firstly, this article would benefit from a detailed comparison with existing tools, primarily the Cochrane risk-of-bias tool for randomized trials (RoB 2) [2]. Outlining certain advantages over established instruments would clarify ROBUST-RCT’s unique contributions. Also, empirical data demonstrating reliability is missing. Thus, an inter-rater reliability assessment with other instruments reinforces its credibility.
References
[1] Wang Y, Keitz S, Briel M, Glasziou P, Romina Brignardello-Petersen, Siemieniuk RAC, et al. Development of ROBUST-RCT: Risk Of Bias instrument for Use in SysTematic reviews-for Randomised Controlled Trials. BMJ. 2025 Mar 25;e081199–9.
[2] Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ [Internet]. 2019 Aug 28;366(1):l4898. Available from: https://www.bmj.com/content/366/bmj.l4898
[3] Flemyng E, Moore TH, Boutron I, Higgins JP, Hróbjartsson A, Nejstgaard CH, et al. Using Risk of Bias 2 to assess results from randomised controlled trials: guidance from Cochrane. BMJ Evidence-Based Medicine. 2023 Jan 24;bmjebm-2022-112102.
Competing interests: No competing interests
The ROBUST-RCT instrument is a valuable addition to the risk of bias (RoB) toolkit for randomized controlled trials (1). I would like to highlight a potential omission that may merit consideration in future iterations of this and other RoB assessment tools, however.
Some clinical trials evaluate interventions that influence the allocation of limited healthcare resources, such as diagnostic appointments, ICU beds, or staff time. In individually randomized trials, increased demand for these resources by patients in the intervention group can inadvertently reduce access for control group patients. This dynamic can introduce spillover effects that lead to biased overestimation of treatment benefit (2).
While ROBUST-RCT addresses risk of bias from imbalanced co-interventions due to unblinding of participants (core item 3), unblinding of healthcare providers (core item 4), or random chance (optional item 2), it does not address spillover arising from an intervention’s effect on availability of co-interventions within the trial setting. Such spillover can occur even in trials with adequate blinding and randomization, representing a distinct source of bias.
Although this form of spillover has been studied in economics as a source of bias in randomized controlled trials, it remains under-recognized in the clinical trials literature. Given the increasing emphasis on trials of care delivery practices, screening tests, and patient monitoring systems—especially in settings characterized by long wait times or staffing shortages—addressing this potential source of bias is increasingly important.
I hope future updates to ROBUST-RCT will consider whether such spillover effects merit inclusion either as an extension of optional item 2 or as a distinct non-core RoB domain. Additionally, other pillars of evidence-based medicine—including trial design guidance, reporting standards such as CONSORT, and other RoB tools—should consider addressing this potential threat to study validity.
References:
1. Wang Y, Keitz S, Briel M, et al. Development of ROBUST-RCT: Risk Of Bias instrument for Use in SysTematic reviews-for Randomised Controlled Trials. BMJ (2025). doi:10.1136/bmj-2024-081199
2. Mann, S. Negative spillover due to constraints on care delivery: a potential source of bias in pragmatic clinical trials. Trials 25, 833 (2024). doi:10.1186/s13063-024-08675-9
Competing interests: I conduct health services research examining the issue of spillover in clinical trials
Critical Analysis of the ROBUST-RCT Tool for Assessing Risk of Bias in Randomised Controlled Trials
Dear Editor
The ROBUST-RCT (Risk Of Bias instrument for Use in SysTematic reviews—for Randomised Controlled Trials) [1] was developed to address limitations in existing risk of bias (ROB) assessment tools [2,3] for randomised controlled trials (RCTs). Its aim is to simplify the assessment process and improve usability for systematic reviewers. While the tool is structurally simple and user friendly, several key limitations may affect its broader integration into systematic review methodology.
A major limitation concerns the level at which bias is assessed. ROBUST-RCT appears to focus on risk of bias assessment at trial-level rather than outcome-level. This poses challenges for integration with the GRADE (Grading of Recommendations Assessment, Development, and Evaluation) approach, which evaluates certainty of evidence at the outcome level. The authors do not clearly explain how to reconcile trial-level assessments with outcome-level certainty ratings. This lack of guidance creates uncertainty for reviewers aiming to align ROBUST-RCT assessments with GRADE methodology.
Another limitation relates to the structure of the tool. ROBUST-RCT comprises six core items and eight optional items but lacks a mechanism for generating an overall risk of bias rating. This absence is methodologically important, as overall risk of bias assessments—preferably at the outcome level, or at least at the study level—are essential for summarising study limitations in systematic reviews. Without such a rating, it is unclear how reviewers should integrate assessments across domains into a summary judgement. This limits the tool’s usefulness for stratified or sensitivity analyses, which often rely on overall bias categories to interpret potential bias across studies.
The inability to synthesise domain-level judgements into an overall rating also restricts analytic flexibility. Stratified analyses by risk of bias and sensitivity analyses exploring the effect of bias on pooled estimates are common practices in evidence synthesis. Without guidance on summarising bias across domains, reviewers may find it difficult to apply ROBUST-RCT in these contexts.
As the authors noted, tools designed for broad use strike a careful balance between usability and the methodological robustness required for systematic reviews. While simplification can enhance usability, it should not come at the expense of rigour. If a tool’s simplicity leads to assessments lacking depth or clarity, it risks being perceived as insufficiently robust by the evidence synthesis community.
In conclusion, while the ROBUST-RCT addresses several limitations of existing ROB assessment tools for RCTs [2,3] and offers enhanced usability through structural simplicity, crucial methodological gaps remain. Addressing these gaps, particularly around outcome-level assessment and the integration of a comprehensive overall risk of bias rating, will be critical for ensuring the robustness and credibility of systematic reviews utilising this tool.
References
[1] Wang Y, Keitz S, Briel M, Glasziou P, Romina Brignardello-Petersen, Siemieniuk RAC, et al. Development of ROBUST-RCT: Risk Of Bias instrument for Use in SysTematic reviews-for Randomised Controlled Trials. BMJ. 2025 Mar 25;e081199–9.
[2] Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ [Internet]. 2019 Aug 28;366(1):l4898. Available from: https://www.bmj.com/content/366/bmj.l4898
[3] Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011 Oct 18;343:d5928. doi: 10.1136/bmj.d5928.
Competing interests: No competing interests