A step-by-step approach for selecting an optimal minimal important difference

Rationale of the selection approach

As the most important aspect for selecting an anchor based MID, the committee prioritised the methodological rigor. Thus, we first apply the credibility assessment for all available MID estimates3132 and choose the most credible MIDs (first stage in fig 1 (step 1 to 30); appendix 2 presents the criteria for choosing the most credible estimates) for a given PROM (or subdomain). The median of the most credible MIDs for that PROM (or subdomain) constitutes the initial best guess as to an optimal MID.13

Fig 1

Selection process for an optimal, anchor based, minimal important difference (MID). The process has two stages: to choose the most credible MIDs by credibility (step 1 to 30), and then to explore contextualised factors (starting from step 30). *Step 1 aims to find the MID estimates with five ratings of “definitely yes/yes/definitely closely related” across the five core credibility criteria. Criteria questions include: (Q1) is the patient or necessary proxy responding directly to both the patient reported outcomes measure (PROM) and the anchor; (Q2) is the anchor easily understandable and relevant for patients or necessary proxy; (Q3) has the anchor shown good correlation with the PROM or (Q3.1) if the correlation is not reported, to what extent is the construct of the anchor closely related to the construct of the PROM); (Q4) is the MID precise; and (Q5) does the threshold or difference between groups on the anchor used to estimate the MID reflect a small but important difference (response options for Q1 are yes or no; all other criteria are rated on a 5 point adjectival scale with response options for “definitely yes” (“definitely closely related” for Q3.1), “to a great extent,” “not so much,” “definitely no” (“definitely not related” for Q3.1), and “impossible to tell” (appendix 2). †Step 4: the most credible estimates are the available MID estimates that receive the highest ratings across the five core credibility criteria. To arrive at the most credible estimates, the criteria are progressively relaxed, as follows: firstly, MID estimates should be chosen with five “definitely yes/yes/definitely closely related” ratings across the five credibility core criteria (Q1-5 (or using Q3.1 instead if correlation coefficient is not reported)); if, however, no estimates meet the five “definitely yes/yes/definitely closely related” ratings, this definition of the most credible MIDs can be relaxed to include MIDs that are rated as five “definitely yes/yes/definitely closely related” or “to a great extent” ratings across the core criteria; if no MIDs meet this relaxed definition, the criteria can be further relaxed to select estimates in lower credibility rank (see appendix 3 for the credibility ranks, higher rank means higher credibility)—ie, estimates with four “definitely yes/yes/definitely closely related” ratings across the core criteria; if not available again, MIDs should be selected with four “definitely yes/yes/definitely closely related” or “to a great extent” ratings across the core criteria, and so on (appendix 3). When MIDs with higher rank are available, do not go down the ranking system to select the MIDs with lower rank (eg, if there is one estimate at rank 1, this estimate would be the only one to be used). ‡Steps 5, 15, or 16: consistency is defined as 80% of the MID estimates lying within an absolute value of 10% of the PROM score above or below the median (ie, within a range of 20% of the median). §Steps 9, 22, or 23: near the median of the whole distribution means that the absolute difference between the median of the selected MID estimates and the median of the whole distribution is less than an absolute value of 10% of the PROM score. ¶Step 14: only applicable when, among the most credible MID estimates, there are estimates anchored to transition ratings—if, however, no estimates are anchored to transition ratings or all the estimates anchored to transition ratings are with long recall period (>4 weeks), skip this step and its downstream steps (ie, steps 14, 16, 20, 21, 23, 26, 27, and 29). **Steps 31 or 35: at least three most credible estimates per contextualised factor. ††Steps 34 or 41: to test whether variability is explained by contextualised factors, compare the distributions; for binary factors, use Wilcoxon rank sum test with a threshold P=0.10

• Download figure
• Open in new tab
• Download powerpoint

The committee recognised that optimal MIDs might vary by contexts.34 If such differences exist, the median of the most credible MIDs will not apply to all contexts. Evidence suggesting that contextual differences might exist include: the most credible MIDs are inconsistent, or the most credible MIDs are consistent with one another but their median is not near the median of all MIDs. Either of these findings requires that investigators seek contextualised factors that can explain the variability among MID estimates (second stage in fig 1 (starting from step 30)).

If investigators identify such contextualised factors, they will select the median of the most credible MIDs under each context as the optimal MIDs. If, however, they fail to identify contextualised factors that explain MID variability, they choose the median of the most credible MIDs as the optimal MID.

MID selection approach in detail

Figure 1 presents the complete processes of the selection approach and appendix 2 provides a more detailed narrative description for the selection process. We summarise the process below.

The process begins with identifying the most credible MIDs. Investigators should apply the credibility instrument,3132 count the number of the five core credibility criteria met, and select the MID estimates with the highest count (fig 1 (steps 1-4); appendix 2). The most credible MIDs are those with the highest ratings across the five core credibility criteria (ie, the highest credibility rank, appendix 3).

Appendix 2 elaborates on how investigators can assess each credibility criterion and select the most credible MID. Briefly, criterion 1 is rated yes or no. All other criteria are rated on a 5 point adjectival scale with response options for “definitely yes” (or “definitely closely related”) (highest credibility); “to a great extent”; “not so much”; “definitely no” (or “definitely not related”); and “impossible to tell” (lowest credibility). The best MIDs are those meeting five “definitely yes/yes/definitely closely related” ratings for the five core criteria (fig 1 (step 1)). To identify the most credible MIDs, however, investigators can progressively relax the criteria (fig 1 (step 4); appendix 2) until they find available MIDs with the highest credibility rank (higher rank means higher credibility, appendix 3). For example, no MIDs for KOOS-quality of life (Qol) met five “definitely yes/yes/definitely closely related” ratings (ie, rank 1). We thus relaxed our criteria and found two MIDs with highest credibility rank, referring to them as the most credible MIDs, which were those meeting five “definitely yes/yes/definitely closely related” or “to a great extent” ratings (ie, rank 2; appendices 3-4).

Investigators will then check the consistency of the selected MID estimates (fig 1 (step 5)) and compare their distribution to the distribution of all available MIDs (fig 1 (step 9)). The committee suggested that MIDs be considered consistent with one another if 80% of the estimates lie within an absolute value of 10% of the PROM score above or below their median (ie, within a range of 20% of the median; box 1). When the most credible MIDs are consistent (fig 1 (step 7)) and the median of the most credible MIDs is near the median of the whole distribution (fig 1 (step 13); when the absolute difference between the median of the most credible MIDs and the median of the whole distribution is less than an absolute value of 10% of the PROM score (box 1)), investigators can be confident that the median of the most credible MIDs represents the optimal MID (fig 1 (step 19); appendix 2). For example, we identified the most credible MIDs for the pain visual analogue scale (VAS-pain; 0-100) to be 17, 13.5, 12, 15, 13,16, 14, 17, and 20.4, of which the median was 15. By definition, they were consistent. Because their median was near the median of the whole distribution (ie, 15.7)—the absolute difference between them was within 10—we selected the median of the most credible, which was 15, as the optimal MID (appendix 4).

The most credible estimates could, however, be inconsistent and excessive variability might exist. If that is the case, investigators should try to explain the variability by further consideration of the credibility criteria. The most important credibility criterion focuses on the anchor validity: the correlation between the anchor and the PROM of interest.3134 Therefore, when the most credible estimates are inconsistent (fig 1 (step 6)), investigators will prioritise the correlation criterion to explain the remaining variability and, among the most credible estimates, select MIDs with high correlations with the anchor (r ≥0.5)31 (fig 1 (step 8); appendix 2).

The credibility assessment instrument has four additional extension credibility criteria that focus on the validity of transition rating anchors (appendix 2).3135 After restricting to MIDs with high correlations with the anchor (r ≥0.5),31 if substantial variability among the estimates remains (fig 1 (step 17)) or if no MIDs with high correlations are available (fig 1 (step 10)), because MIDs were often estimated on transition rating anchors (more than half in the MID inventory),33 investigators could further consider the additional credibility criteria.31 We suggest assessing the recall period for the transition rating anchors, because except for follow-up length, the relevant data for the other three additional criteria were rarely reported.3136 Investigators could remove the estimates anchored to transition ratings with a long recall period (>4 weeks)31 (fig 1 (step 14)) and determine whether the remaining most credible estimates prove consistent (fig 1 (step 16); appendix 2). If, however, no selected estimates are anchored to transition ratings or all the selected estimates anchored to transition ratings are with long recall period (>4 weeks), investigators would skip the assessments (ie, skip steps 14, 16, 20, 21, 23, 26, 27, and 29 in fig 1).

The further consideration of the credibility criteria regarding the anchor validity above, however, might not be necessary. Because there might be few estimates with a high correlation of 0.5 between the PROM and the anchor, it is possible that all the estimates using transition anchors have long recall period; more often, the most credible estimates would appear to be consistent. For example, in our worked examples (appendices 4-5), we did not further consider the correlation and recall period criteria. After applying the five core credibility assessments, the most credible MIDs were consistent (appendices 4-5).

At this stage, investigators will face one of three situations:

• The (newly) identified most credible MIDs are consistent, and their median is near the median of all MIDs (fig 1 (steps 13, 25, or 27); appendix 2). If this is the case (referring to the VAS-pain example above), the median of these (newly) identified most credible MIDs represents the optimal MID, applicable to all contexts (fig 1 (steps 19, 28, or 29); appendix 2).

• The (newly) identified most credible MIDs are consistent, but their median differs considerably from the median of the whole distribution (fig 1 (steps 12, 24, or 26); appendix 2).

• The (newly) identified most credible MIDs are inconsistent (fig 1 (steps 6, 17, or 20); appendix 2).

Intuitively, for the second situation, the substantial differences of the two medians might be attributed to MID credibility alone. Contextualised factors, however, could have a role. For example, all the most credible MIDs might be established in a same context (see the worked example below), and the different estimate in another context could be due to the lower credibility or rather due to the different context. Therefore, because either the second or third situation above suggests that the MIDs might be context dependent, further exploration seeking contextualisation as an explanation of variability is required (fig 1 (step 30); appendix 2).

Investigators will use the most credible MID estimates for exploring variability (fig 1 (step 31)). When, however, the number of credible estimates is insufficient for this exploration (fig 1 (step 33))—that is, fewer than three estimates per contextualised factor (box 1)—investigators will then use all available estimates for the exploration (fig 1 (step 35)). If the search for contextualised factors yields an explanation for the variability, the optimal MID will be context dependent: one optimal MID for each context (fig 1 (steps 40 or 44); appendix 2). If no contextual explanation is found, investigators will select the median of the most credible estimates among all available MIDs as the optimal MID (fig 1 (step 45); appendix 2).

Worked example of the selection approach

To further illustrate the selection approach, we describe a worked example here, using the pain subdomain of the WOMAC. Appendix 4 presents more worked examples, including VAS-pain, KOOS-Qol, and the 36 item short form survey-mental component summary (SF-36-MCS).

For WOMAC-pain, the data were obtained from the PROMID inventory (up to 2018).33 We describe the detailed process for generating the database elsewhere.33 Briefly, we searched all relevant database to summarise all available anchor based MIDs of PROMs from primary studies.33 We identified 13 studies (up to 2018) estimating MIDs for WOMAC-pain. By using different chosen thresholds on the anchor, different anchors, and more than one set of participants with different conditions or different analytical anchor based methods to estimate MID within one study, the 13 studies generated 67 estimates, of which authors expressed 45 MID estimates in absolute terms and 22 MID estimates relative to baseline scores (appendix 6).33 Our approach suggests selecting the optimal MIDs from absolute MIDs (appendix 1). We thus conducted the selection for the 45 absolute MIDs. Appendix 5 presents the entire selection process and appendix 7 provides the relevant data for conducting the selection. We describe the selection process below.

Typically, the WOMAC has five items for pain subdomain assessed on a 0-4 scale with a total score ranging from 0 to 20. Although using the same instrument, authors used different scoring system or made conversion of the scores, and thus expressed the results with different scales (scales ranging from 0 to 10, 0 to 20, 0 to 50, and 0 to 100, appendix 6). Therefore, we transformed each estimate into a scale of 0-100. After this transformation, the MIDs ranged from 0 to 35 (appendix 7).

After assessing MID credibility (fig 1 (step 1)), no estimates met five “definitely yes/yes/definitely closely related” ratings across the five core credibility criteria (appendices 5 and 7). By relaxing the criteria (fig 1 (step 4)), we found five estimates meeting five “definitely yes/yes/definitely closely related” or “to a great extent” ratings (appendices 5 and 7). Table 1 presents the absolute value of these most credible MID estimates. Using our definition for consistency (box 1), these most credible MID estimates were consistent (fig 1 (step 7); appendix 5). Because the median of the most credible estimates (28.1) was not near the median of the whole distribution (12.5) (box 1; fig 1 (step 12); appendix 5), we postulated that the MIDs for WOMAC-pain could be context dependent. We therefore explored the possibility that contextualised factors could explain MID variability.

Because the number of most credible MID estimates (n=5) was not sufficient for the exploration (fig 1 (step 33); appendix 5), we instead used all available estimates (fig 1 (step 35); appendix 5). We explored the impact of patient condition (knee v hip complaints) on the variability of the MID estimates by comparing the medians of the two groups using the Wilcoxon rank sum test. We found that the patients’ condition did not explain the variability. We then explored whether the intervention (surgical v non-surgical intervention) explained the variability and found a significant difference between MIDs generated in surgical versus non-surgical intervention settings (P=0.009, table 1; appendices 5 and 7).

Therefore, the selection and application of the optimal MID for WOMAC-pain was context dependent—the optimal MID differed depending on whether the patients were undergoing surgical or non-surgical intervention (fig 1 (step 44); appendix 5). In other words, we found two optimal MIDs: one selected from the MIDs estimated under the context of surgical intervention that should be exclusively used in the context of surgical intervention; and the other selected from the MIDs estimated under the context of non-surgical intervention that should be exclusively used in the context of non-surgical intervention.

Under the context of surgical intervention, we found that the most credible estimates were those meeting five “definitely yes/yes/definitely closely related” or “to a great extent” ratings across the five core credibility criteria; including estimates 29.26, 29.9, 20.5, 28.1, and 23.5, with a median of 28.1 (table 1). The most credible estimates for non-surgical intervention were those meeting four such ratings across the five core credibility criteria; including 11.8, 12.9, 6.4, 8.3, 13.51, 8.74, 15, 8.74, 7.09, and 4.1, with a median of 8.7 (table 1). The corresponding optimal MID for surgical intervention was 28.1 applicable to the context of surgical intervention, and the optimal MID for non-surgical intervention was 8.7 applicable to the context of non-surgical intervention.