By David Tuller, DrPH
What is going on with Cochrane’s much-delayed re-review of exercise therapy for the illness the organization is these days calling ME/CFS? It is more than three years since Cochrane announced plans to develop this completely new review, but no one outside the organization seems to have much idea of what’s happening with that project.
Maybe no one inside really does, either.
Patients are frustrated and angry—and rightly so. The existing review, as flawed as it is, still gets cited as evidence that exercise therapies have been shown to be effective treatments. Recently, for example, the authors of a propaganda piece presented in the guise of a “research agenda for post-COVID-19 fatigue” referenced the contested Cochrane review to assert this exact point and suggest the same interventions would be helpful for the new global cohort of long Covid patients. However, in a telling omission that highlights the weakness of their argument, the authors failed to mention the current ME/CFS guidelines from the UK’s National Institute for Health and Care Excellence (NICE). These were published in 2021 and reversed NICE’s prior exercise therapy recommendation; the agency assessed the quality of the evidence in favor of these interventions as uniformly “low” or “very low.”
Here’s a short recap of this saga. In 2015, Cochrane published an update of an exercise therapy review first done in 2004 for what was then being called CFS. The updated review was conducted based on the original protocol, which was published in 2001. The lead author of this updated review was Lillebeth Larun, a researcher with the Norwegian Institute of Public Health in Oslo. In 2017, Cochrane appended to the review series of exchanges between Larun and two very sharp patient-advocates who challenged the methodology and conclusions—Tom Kindlon and the late Robert Courtney.
Dissatisfied with Larun’s responses and her blanket rejection of valid concerns about the reported findings of benefit from exercise therapies, Courtney filed a formal complaint with Cochrane. After an extended delay, Cochrane’s editor-in-chief, David Tovey, determined by late 2018 that the review should be withdrawn based on the issues raised by Courtney. By then, Courtney had passed away. (I met with Tovey in the summer of 2018 and argued that the review should be dumped for a variety of methodological reasons.)
Tovey’s effort to have the review withdrawn was publicly revealed in October 2018 in a biased Reuters news article, which absurdly portrayed the evidence-based decision as a major defeat for Science writ large. Cochrane received huge pushback from the review authors, the Norwegian Institute of Public Health, and others invested in the crumbling GET/CBT treatment paradigm. Ultimately, Cochrane caved in to the pressure and did not withdraw the review but published an amended version in October, 2019. For this version, Cochrane required Larun and her co-authors to water down their conclusions.
“People who have exercise therapy probably have less fatigue at the end of treatment than those who receive more passive therapies,” concluded the revised review. “We are uncertain if this improvement lasts in the long term. We are also uncertain about the risk of serious side effects from exercise therapy.”
Even amended, the review is a mess. It includes a warning that, in contrast to the previous version, the amended review “places more emphasis on the limited applicability of the evidence to definitions of myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) used in the included studies, the long-term effects of exercise on symptoms of fatigue, and acknowledges the limitations of the evidence about harms that may occur.” In regular English, that means: Don’t take these recommendations at face value!
This resolution ultimately satisfied no one.
I presume Larun and her colleagues felt that being forced to weaken their reported findings cast doubts on the competence of their work—as it did, and as it should have. Meanwhile, Cochrane’s advisory essentially acknowledged that the review was not fit-for-purpose, and patient advocates declared—accurately—that it was a piece of crap. Like the PACE trial itself, which remains the largest study included in the analysis, the review relies on subjective outcomes in unblinded trials. Given the inevitable bias built into this study design, the findings are pretty useless.
**********
Promise to launch a completely new review process
The publication of the amended version in October 2019 was accompanied by a statement from Tovey’s replacement as Cochrane’s editor-in-chief, Karla Soares-Weiser, promising an immediate start to work on a completely new exercise review. Here’s what she wrote:
“Cochrane recognizes the importance of providing the best available evidence on interventions for ME/CFS to enable patients and clinicians across the world to make well-informed decisions about treatment. This amended review is still based on a research question and a set of methods from 2002, and reflects evidence from studies that applied definitions of ME/CFS from the 1990s. Having heard different views expressed about the evidence base for this condition, we acknowledge that the publication of this amended review will not resolve all the ongoing questions about this globally important health topic.
“We have decided, therefore, that a new approach to the publication of evidence in this area is needed; and, today we are committing to the production of a full update of this Cochrane Review, beginning with a comprehensive review of the protocol, which will be developed in consultation with an independent advisory group that we intend to convene. This group will involve partners from patient-advocacy groups from different parts of the world who will help us to embed a patient-focused, contemporary perspective on the review question, methods and findings.”
In pursuing this strategy, Cochrane designated a group of researchers and patient advocates and charged them with authoring the new review. Others were named to serve as members of an independent advisory group (IAG). At first, there were regular updates on progress. After that, there has been mostly silence. The last official update about what Cochrane is calling a “pilot” of “stakeholder engagement in high-profile reviews” was on June 30, 2021, as far as I can tell. (Perhaps there have been more recent updates made privately.)
The process was supposed to involve writing a new protocol. So far, any such protocol, if one exists, is being closely held inside Cochrane. Can Cochrane provide any actual information about this protocol? If there is one, who is reviewing it? The IAG? Others within or outside Cochrane? How will the review process work? How will differences between reviewers, the writing team, and the IAG be resolved? Does anyone know? And will there be an opportunity for patients and advocates to offer comments and suggestions?
And beyond the protocol, what is the process for doing the actual analysis and writing the draft review? Who will be reviewing and revising that draft? And again, will the patient and advocacy communities be able to offer comments before the final version is published?
Whatever the causes of these delays, they are unacceptable. It is not surprising that patients interpret the prolonged and unexplained lack of transparency to be a sign of disrespect and institutional indifference to their situation. I assume there is also likely disarray in Cochrane’s top ranks over how to handle this mess. Really, Cochrane has only itself to blame for having published work that treats arguably fraudulent studies like the PACE trial as sources of legitimate data. (Leading Cochrane-ites still seem to support this piece of crap.)
Now, if the Cochrane protocol for the new review exists, and if it’s great, perhaps a final review will be terrific as well. But we are three years into this process, and we don’t have a protocol yet. (Let’s stipulate that some delay would be understandable, given the pandemic.) Moreover, the protocol should be the easier part—conducting the review itself and analyzing the data is the real guts of the project. At the rate things are moving, it could be another five or ten years until a new review emerges from the black maw of the Cochrane process. It will likely be outdated the moment it is birthed into being.
Luckily, the new NICE clinical recommendations already cover what Cochrane is purportedly investigating. The NICE team reviewed all the literature and rendered an authoritative assessment. Meanwhile, Cochrane has demonstrated that moving things along in an appropriately open and efficient manner seems to be beyond its current capacities. I don’t know why this is and am not pointing fingers at any individuals. I tend to assume it’s an institutional or organizational failure. I also assume that the people actually dedicated to the project and trying to make it work are fed up with any internal roadblocks and obstacles.
In any event, whatever its merits in other domains, Cochrane has long failed patients miserably in this one. Perhaps it needs to consider scrapping the entire effort and retracting the existing review. Then it could fine-tune its stakeholder engagement process with a different group of patients and leave this field to others who place greater value on patient health and interpretable research than on the egos and anti-scientific pronouncements of professors and other pooh-bahs.
As it is, the current situation is untenable. The decision to leave the flawed review standing during this open-ended period was extremely unfortunate and damaging to patient well-being. The sooner this misbegotten document is disavowed for good, the better–not only for those with ME/CFS but for the long Covid patients with post-exertional malaise who could be harmed by purportedly “effective” exercise therapies.