By David Tuller, DrPH
*This is a crowdfunding month at UC Berkeley for my Trial by Error project. If you appreciate my work and would like to help support it, here’s the link for this November’s campaign.
I recently posted about Cochrane’s unsatisfactory responses to a request from the Science for ME forum that the organization withdraw “Exercise therapy for chronic fatigue syndrome.” a seriously flawed and unreliable 2019 review. The forum also requested an update on Cochrane’s delayed efforts to produce a new ME/CFS exercise review designed to replace this 2019 document; the latter, unfortunately, remains on Cochrane’s site and continues to get cited by promotors of the GET/CBT treatment paradigm. The forum found the organization’s responses inadequate and has filed a formal complaint with Cochrane about the matter.
In the meantime, after more than two years of silence about what has—or has not–been happening with the planned new review, Cochrane last week finally offered a glimpse into developments. The information arrived on November 10th in the form of a report from Hilda Bastian, a long-time Cochrane associate and the designated head of a team, called the Independent Advisory Group (IAG), that is charged with overseeing the development of the new review.
Bastian herself appointed the members of the IAG as well as the members of the team responsible for writing a protocol and the review itself. After promising regular progress reports, she and Cochrane stopped communicating publicly about the guideline development process; until last week, Bastian’s most recent update was in July, 2021. Given this information void, many patients felt dismayed, confused, depressed, and angered. And understandably so.
Bastian started the update with some terrible personal news, then touched on other factors that contributed to delays in communicating with the public as well as to an “unplanned hiatus” of the IAG’s. As far as I’m aware, this was the first public acknowledgement of such a hiatus. As she wrote:
“This report is long overdue, and so comes with deep apologies for my long silence. I was out of action for some time after the sudden death of my son late in 2021. That coincided with the upheaval for this project caused by the organisational changes and complaints discussed in this report. The combination of extended uncertainties and confidentiality while some processes were underway made meaningful reporting impossible. Those uncertainties included how long the unplanned hiatus for the IAG might last. The authors’ work on the draft protocol continued, however.”
Besides Bastian’s family tragedy, a major cause of the delay turned out to be objections to the entire project from interests presumably aligned with the GET/CBT ideological brigades—those with the most to lose from a new review. As Bastian explained:
“The reasons [for the delay] included the lengthy and confidential process following a detailed complaint that called for the process to be discontinued, and for the review to remain in its current form. The basis for incorporating an IAG into the editorial processes of a Cochrane review was challenged, as well as the need for an update, and the proposed editorial process. The complaint also argued that too many of the people who were authors of the review or members of the IAG, including me, were too biased in favour of criticisms of the current Cochrane review.”
And further: “Multiple committees within Cochrane’s organisation considered this complaint consecutively, which took around a year. The complaint was not upheld after thorough consideration by the Conflict of Interest Panel, a subgroup of the Editorial Board, and finally Cochrane’s Governing Board.
So let me understand this. After promising regular updates regarding a project vital to the ME/CFS patient community, Cochrane undertook a lengthy internal investigation without making any public communication, leaving everyone in the dark for an extended period of time. Was there really no way for the organization to disclose the existence of an obstacle that was delaying matters while preserving whatever confidentiality was required? I am perplexed that anyone would determine that the only or least the best way to handle this difficult situation was to ghost the entire patient community for more than two years. Beyond being unacceptable, this strategy was certainly not the way to build trust and confidence in the process going forward.
On the positive side (sort of): Bastian mentioned a proposal that could help ameliorate some of the possible damage from the decision to not withdraw the review. The plan is for the IAG to create a better editorial note for the review because, as she wrote, “some uses of the Cochrane review do not reflect critical nuance and limitations in its conclusions, and the goal of a note would be to try to reduce such misinterpretation.” (Of course, such an editorial note would not be necessary at all if Cochrane removed the review in the first place.)
Along with posting her report, Bastian opened a blog to serve as a communications platform on which she would respond to comments. That has led to a lively debate—with lots of sharp comments and generally patient but not always satisfactory responses. I don’t envy Bastian’s task here. Cochrane seems to have placed her in the uncomfortable position of having to field continuing criticisms of the 2019 review and to defend the hard-to-defend decision not to withdraw it.
Whatever editorial note has been or will be slapped on the 2019 review to alert readers to the problematic nature of its findings, it continues to be referenced by those seeking to promote the CBT/GET treatment framework. It should not have been published in the first place; once published, it should have been withdrawn. Bastian herself had nothing to do with the previous version and the decision-making around it. Nonetheless, because Cochrane handled the whole matter so badly and remains recalcitrant in its refusal to take further action, she is bearing the current brunt of complaints about what has and hasn’t happened.
But here’s the bottom line: As an institution, Cochrane seems unable to come to grips with the reality that unblinded trials relying solely on subjective outcomes are fraught with bias and do not provide reliable evidence for therapeutic interventions. This is especially so when the subjective findings are not only modest but are contradicted by poor or null results for objective outcomes in the same body of research—as is the case here. So passing judgement on these studies should not be a hard call. But apparently it is, at least for Cochrane.
That obviously raises some concerns about what kind of review could emerge from this ongoing process. We’ll have to see what happens going forward. But both the writing team and the IAG have some smart people on them, so there’s reason for at least measured optimism that the final product will be acceptable—notwithstanding Cochrane’s questionable behavior with regards to the 2019 version.