By David Tuller, DrPH
Last month, The BMJ published a study of a rehab intervention for Long Covid in which the authors made claims that were not borne out by the data. The study was called “Clinical effectiveness of an online supervised group physical and mental health rehabilitation programme for adults with post-covid-19 condition (REGAIN study): multicentre randomised controlled trial” and was led was led by researchers from the University of Warwick in Coventry, England. (I critiqued it here.) In particular, the paper declared that the intervention was shown to be “clinically effective”—even though the results for the primary outcome did not meet the acknowledged threshold for a “minimally important” or “clinically important” difference.
This contradiction was noted in both a comment on the Science For ME forum and a rapid response appended to the article itself. Today, I sent the following letter to The BMJ’s editor-in-chief, Dr Kamran Abbasi. I cc’d Professor Gordon McGregor, the lead author, as well as my eleven co-signatories.
**********
Dear Dr Abbasi:
In February, The BMJ published “Clinical effectiveness of an online supervised group physical and mental health rehabilitation programme for adults with post-covid-19 condition (REGAIN study): multicentre randomised controlled trial,” by McGregor et al [1]. In the trial, 585 patients experiencing prolonged symptoms after having been hospitalized for covid-19 were assigned to either an eight-week intervention, consisting of more than ten one-hour online sessions, or to “usual care,” consisting of one half-hour online consultation.
The trial was unblinded and relied solely on subjective outcomes—a study design guaranteed to generate an unknown amount of bias. The primary outcome, health-related quality of life at three months, was assessed with the preference score (PROPr) derived from the Patient-Reported Outcomes Measurement Information System (PROMIS) [2]. While no biomarkers are available for “post-covid-19 condition,” the authors could have chosen to include among their secondary outcomes one or more objective measures of function, such as the six-minute walking test, step-test for fitness, actigraphy to assess distance moved, or employment status.
Given the bias inherent in the study design, it is surprising that the findings for the primary outcome were nonetheless so poor that the authors apparently felt compelled to misrepresent them. In the main intention-to-treat analysis, the authors report a mean difference in PROPr scores between the intervention and usual-care arms of 0.03 at both three months and 12 months. (The latter was a secondary outcome.) In the abstract’s conclusion, the authors state that the intervention was “clinically effective at improving health related quality of life at three and 12 months compared with usual care” in “adults with post-covid-19 condition.”
To begin with, it is inappropriate for the authors to extrapolate findings from patients who were hospitalized with covid-19 to the much larger number of patients with prolonged symptoms who were not hospitalized. Beyond that serious error of interpretation, the reported difference of 0.03 in PROPr scores between the two groups does not support the authors’ assertion that the intervention was “clinically effective.”
In the description of PROPr in the methods section, the authors note that “as with other preference based measures such as the EuroQol 5 dimension 5 level (EQ-5D-5L) instrument, a difference of 0.03 to 0.05 is considered to be clinically important.” However, the reference for this statement, an official “frequently-asked-questions” (FAQ) page for the PROMIS measures being used in the study, explicitly contradicts the authors’ claim that “a difference of 0.03 to 0.05 is considered to be clinically important” for PROPr [3].
Here is the explanation from the FAQ cited by the authors:
“The minimally important difference for PROPr has not been formally evaluated, but we currently recommend using 0.04. Most preference-based measures have minimally important difference thresholds between 0.03 and 0.05. Current work-in-progress suggests a minimally important difference of 0.04 would be appropriate, although a conservative estimate of 0.08 (half of a standard deviation in the PROPr dataset) could also be used.”
In other words, the authors’ assertion in the methods section that the currently recognized threshold for a “clinically important” or “minimally important” difference for PROPr is the same as for other preference-based measures is categorically false [4, 5]. In the article’s discussion section, the authors acknowledge as much but then provide an irrelevant rationale for having claimed clinical effectiveness anyway.
As they write: “Research completed since we started this study suggests a minimally important difference of 0.04 on the PROPr score between groups. Our observed differences of 0.03 (95% confidence interval 0.01 to 0.05) at three months and 0.03 (0.01 to 0.06) at 12 months are smaller than this suggestion. However, the complier average causal effect analysis showed a larger effect of 0.05 (0.01 to 0.09) at three months and 0.06 (0.01 to 0.10) at 12 months, suggesting that the true effect, in those fully complying with the intervention, might exceed this threshold.”
This argument is unsustainable. The authors cannot claim clinical effectiveness for their overall findings if this claim only applies to a more fully compliant subset of their sample and is not borne out by the intention-to-treat analysis. The paper—and in particular the abstract’s conclusion that the intervention was “clinically effective” for health-related quality of life—must be corrected to reflect that the primary outcome results did not meet the threshold for “clinically important” or “minimally important” difference recommended for the measure in question.
Sincerely,
David Tuller (corresponding author)
Center for Global Public Health
University of California, Berkeley
Berkeley, California, USA
davetuller@berkeley.edu
Nicola Baker
School of Health Sciences
University of Liverpool
Liverpool, England, UK
Todd Davenport
Department of Physical Therapy
University of the Pacific
Stockton, California, USA
David Davies-Payne
Department of Radiology
Starship Children’s Hospital
Auckland, New Zealand
Jonathan Edwards
Department of Medicine
University College London
London, England, UK
Mark Faghy
Human Sciences Research Centre
University of Derby
Derby, England, UK
Keith Geraghty
Centre for Primary Care and Health Services Research
Faculty of Biology, Medicine and Health
University of Manchester
Manchester, England, UK
Mady Hornig
Department of Epidemiology
Columbia University Mailman School of Public Health
New York, New York, USA
Brian Hughes
School of Psychology
University of Galway
Galway, Ireland
Leonard Jason
Center for Community Research
DePaul University
Chicago, Illinois, USA
Asad Khan
North West Lung Centre
Manchester University Hospitals
Manchester, England, UK
David Putrino
Department of Rehabilitation Medicine
Icahn School of Medicine at Mt Sinai
New York, New York, USA.
John Swartzberg
Division of Infectious Diseases and Vaccinology
School of Public Health
University of California, Berkeley
Berkeley, California, USA
1. McGregor G, Sandhu H, Bruce J, et al. Clinical effectiveness of an online supervised group physical and mental health rehabilitation programme for adults with post-covid-19 condition (REGAIN study): multicentre randomised controlled trial. BMJ 2024;384;e076506
2. Dewitt B, Jalal H, Hanmer J. Computing PROPr Utility Scores for PROMIS® Profile Instruments. Value Health 2020;23:370-8
3. PROPr: The PROMIS-Preference Score. What is this minimally important difference for PROPr? https://www.proprscore.com/faqs/; 2023
4. ME/CFS Skeptic. Thread: Clinical effectiveness of an online group physical & mental health rehab programme for post-covid-19 condition REGAIN study, 2024, McGregor+. Science for ME [Internet]; 9 Feb 2024; #23. Accessed at: https://www.s4me.info/threads/clinical-effectiveness-of-an-online-group-physical-mental-health-rehab-programme-for-post-covid-19-condition-regain-study-2024-mcgregor.37174/page-2#post-514588
5. Louise S. Rapid Response: Re: Clinical effectiveness of an online supervised group physical and mental health rehabilitation programme for adults with post-covid-19 condition (REGAIN study): multicentre randomised controlled trial — Conclusions are concerning. BMJ; 13 Feb 2024. Accessed at: https://www.bmj.com/content/384/bmj-2023-076506/rr