By David Tuller, DrPH
I have sent the following letter to Fiona Godlee, editorial director of BMJ and editor-in-chief of The BMJ, on behalf of Professors Brian Hughes and Vincent Racaniello as well as me. We were responding to the recent editorial regarding the new draft of ME/CFS clinical guidelines from the National Institute for Health and Care Excellence–as others have already done through BMJ’s rapid response function.
**********
Dear Dr Godlee:
The recent editorial from Professors Turner-Stokes and Wade about the new draft of clinical guidelines for ME/CFS from the National Institute for Health and Care Excellence (NICE) is problematic on multiple fronts. [1]
For three decades, a small group of medical professionals have forcefully promoted the claim that two specific interventions — cognitive behaviour therapy (CBT) and graded exercise therapy (GET) — can lead not just to improvement but to recovery from what they prefer to call chronic fatigue syndrome. In 2011, The Lancet published the main results from the largest randomised trial of CBT and GET for the illness, the so-called PACE trial; at first glance, the study appeared to support these views. [2]
The PACE investigators referred to their trial as the “definitive” test of their approach to therapeutic intervention for this disabling illness. Lancet editor Richard Horton hailed the research that appeared in his journal as “remarkable” and “utterly impartial”. [3] No less an eminence than Professor Sir Simon Wessely, who was involved with PACE but was not a co-author, proclaimed it “a thing of beauty”. [4]
Professor Wade himself declared the following about PACE in a statement released on his behalf by the Science Media Centre, a communications agency with close ties to the trial investigators: “Randomised controlled trials provide the best and only reliable evidence on safety and effectiveness of any intervention in any condition. The trial design in this study was very good, and means that the conclusions drawn can be drawn with confidence.” [5]
The PACE trial, of course, has since been widely discredited for “unacceptable methodological lapses” [6], and Professor Wade has apparently changed his mind about its probative value. The editorial he has co-authored argues that randomised controlled trials (RCTs) should not be viewed as acceptable when assessing “complex interventions for complex conditions.” It is noteworthy that he is publicly espousing this position after the renowned trial he previously championed has come to be regarded as fatally flawed.
The editorial’s main argument is that non-pharmacological treatments with multiple elements cannot be appropriately evaluated using the standard GRADE system. Trials of these therapies, the authors declare, are inevitably hampered by built-in limitations, such as the difficulty of blinding interventions and the need to use subjective outcomes in the absence of physiological measures. Whether or not GRADE is adequate for examining these kinds of non-pharmacological interventions, challenging the process and the rules after-the-fact raises questions about whether such protests are primarily principled or self-interested.
Indeed, the arguments advanced in this effort to discard a long-standing approach to the assessment of clinical trial evidence are easily dismissed.
First, the main objections raised about PACE and related studies have not been about the lack of blinding on its own or subjective outcomes on their own. It is the combination of the two in a single study that makes much of this research hard to interpret. This study design inevitably leads to unknown amounts of bias. When blinding is impossible, subjective findings must be accompanied by some sort of objective data to confirm that the results are genuine and not solely a factor of the study design.
The absence of biomedical tests for ME/CFS has not prevented investigators from incorporating a range of other objective measures. In PACE, the investigators chose four—how participants performed on a six-minute walking test and on a step-test, whether or not they were employed, and whether or not they were receiving social benefits. None of these measures matched the subjective reports of improvement from CBT or GET. The PACE authors subsequently dismissed their own objective measures as irrelevant or not objective after all, and claimed success anyway.
As part of the guideline development process, NICE commissioned an independent review of the evidence base for ME/CFS treatments. The independent review assessed dozens of studies, including PACE, that collectively covered 236 treatment outcomes–findings that have been used to support claims that CBT or GET were effective. Of these 236 outcomes, the evidence for 205 was deemed to be of “very low quality”, and 31 were deemed to represent evidence of “low quality”. Not a single outcome was supported by evidence that exceeded this abysmal threshold.
During the independent review, the most common methodological problem identified in this evidence base was “risk of bias.” This is not surprising, given the tendency of the PACE investigators and their like-minded colleagues to design studies relying on unblinded interventions and subjective outcomes.
Second, just because you believe that you cannot meet a high theoretical standard does not entitle you to demand that you be judged against a lower one. The fact that your research falls short does not make the theoretical standard less important. Nor does it magically infuse your efforts with robustness or diminish the need for reliable data in order to assess performance.
Professors Turner-Stokes and Wade argue that their “complex interventions” cannot easily be standardized and are therefore incompatible with RCTs. For them, appropriate therapy is “the equivalent of a well tailored [sic] suit, fitted to the individual patient.” This sounds a lot like the special pleading employed by homeopaths, crystal therapists, and chiropractors to argue that their treatments, too, are difficult to research. In other words, the logic used is akin to pseudoscience.
By stating that the “rigid structure” of RCTs is inappropriate for testing the effectiveness of rehabilitative strategies involving CBT and GET, Professors Turner-Stokes and Wade are throwing the once-vaunted PACE trial and all related research under the bus. They are also demanding that the rest of us change our own positions about the importance of maintaining research standards. Beyond stating that some people with the illness get better, they provide no reliable evidence that their individualised rehabilitative approach has improved patients’ health more than, say, the passage of time on its own.
This lax attitude to evidentiary standards has implications beyond the domain of ME/CFS. In a perplexing passage, Professors Turner-Stokes and Wade credit “appropriate support and rehabilitation” for improvements seen among people experiencing prolonged symptoms after an acute bout of Covid-19. Their claim is unwarranted. That some or many long-Covid patients have improved and even recovered, for reasons that remain unclear, proves nothing more than that some or many long-Covid patients improve and recover. Any definitive statement about effective treatment will need to await properly conducted research.
In our view, the concerns expressed by Professors Turner-Stokes and Wade about the draft NICE guidelines for ME/CFS are misguided. Complex therapies for complex conditions should be judged against the best available standards, and these standards should not be compromised simply to make a preferred treatment look better. In this regard, NICE is to be commended for evaluating the relevant research with objectivity and thoroughness.
The new draft NICE guidelines for ME/CFS are extremely welcome and long overdue.
Sincerely–
Brian Hughes, PhD
Professor of Psychology
National University of Ireland, Galway
Galway, Ireland
Vincent Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University
New York, New York, USA
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
School of Public Health
University of California, Berkeley
Berkeley, California USA
References
[1] https://www.bmj.com/content/371/bmj.m4774
[2] https://pubmed.ncbi.nlm.nih.gov/21334061/
[6] https://www.virology.ws/2018/08/13/trial-by-error-open-letter-to-the-lancet-version-3-0/
Comments
9 responses to “A Letter to BMJ From Prof Hughes, Prof Racaniello and Me”
Thank you for speaking up. It’s hard to believe we are still here in 2020.
In 2007, Simon Wessely seemed to think that RCTs were needed in mental health at least, and in relation to complex conditions -https://www.cambridge.org/core/journals/biosocieties/article/abs/defence-of-the-randomized-controlled-trial-in-mental-health/C6DE3096B2D61C5B05D9BEAA9E44724C . And then there’s Wessely and Everitt’s 2008 textbook where they stress the importance of well-conducted clinical trials in psychiatry. So I wonder, will Wessely wade in now and try to put Turner-Stokes, Wade and the BMJ straight or does he perhaps think that the need for such trials is confined to psychiatry and that ME and CFS lie outside that field? Surely this editorial must interest/concern him?
Silent knight?
Thank you for reminding of the importance of objectivity and measurability when it comes to recommendations in the coming update of NICE guidelines. Patients have been waiting years for this guideline to change its harmful recommendations -seems like this is finally happening through solid work.
The levels to which British medical science has plummeted! Has Godlee not read Ioannidis seminal work on “Why Most Published Research Findings Are False” ? (-https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124-)
In the hope she might bother to read this blog I’ll copy the abstract here… a little bit of homework for you, love, :
“There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.”
Thank you for writing this.
It’s like these people need an explanation what science is.
It must be very distressing for competent dedicated investigators to witness the debasement of scientific research by a growing pool of frauds and fools. Naturally they prefer to keep head down and stick to the lab bench. Who really wants to do office politics and drama?
Unfortunately head-in-the-sand won’t work. If honest scientists don’t get organized and wrest control of their field from the charlatans they will find themselves in another Dark Age. Even now the scientific knowledge base is so polluted in so many areas that only subject matter experts have any hope of separating what’s so from what’s not.
All my life I have heard that “Science is self-correcting”, but what I see instead, even as a non-scientist, is that Science is self-destructing. Like most people in the world, I can not be a subject matter expert in medicine. For self-protection I have to assume that whatever some self-appointed expert says is not trustworthy. I have to assume that all experts and authorities are frauds trying to get wealth and power at my expense, even though I know that some experts are not.
Widespread mistrust of everybody and every institution in “The West” is driving us
towards social disintegration. Regius Professor Sir Simon and his associates deserve some of the blame for their role in this breakdown of trust. But maybe there is a silver lining in his work: it boldly underscores that “Greed Is Good” is not a foundation for a stable society.
Thank you for having our backs. Defending ourselves to people we know is difficult enough.
Are Horton and Godlee related to Kellyanne Conway, by any chance?
What with all the ‘alternative facts’ and obsessive support for the discredited?
When did editing prestigious medical journals become a PR exercise on behalf of the favoured few?