What’s Up With Cochrane’s Exercise Review?

By David Tuller, DrPH

On June 17th, Cochrane announced that it had received a revision of a much-contested review of exercise therapy for treatment of chronic fatigue syndrome (as the organization has long called the illness or cluster of illnesses also referred to as myalgic encephalomyelitis, CFS/ME, and ME/CFS). In a posted statement, Cochrane noted that “the process has taken longer than hoped; the amended review is being finalised and it will be published during the next 2 months.”

Well, it seems like the process continues to take “longer than hoped,” since it is now more than three months since that notice was posted. Publication of the revision is more than a month overdue. Perhaps Cochrane is planning to publish it imminently. Or perhaps the process of “finalising” the amended review has been more complicated than anticipated, raising further questions for Cochrane about whether it should endorse and publish another iteration of what has been a misbegotten mess from the start.

We don’t know what changes were made for the version mentioned in the June 17th statement. However, we do know that Dr David Tovey, Cochrane’s now-retired editor-in-chief, rejected a revision submitted in mid-May because it overstated the available evidence.

Critics–mainly very smart patients–have long made this same point about the 2014 review, which was an update of one published a decade earlier. Most of the eight studies included investigated graded exercise therapy. The review, which was conducted by Cochrane’s Common Mental Disorders Group, is so methodologically flawed that the best move would be for Cochrane to scrap the whole thing, transfer ME/CFS to a more appropriate section of the organization, and start from scratch.

In any event, Dr Tovey expressed his views in a set of e-mails released under Norway’s liberal freedom of information laws. (The review’s lead author, Lillebeth Larun, is a researcher at the Norwegian Institute of Public Health). On May 13th, Atle Fretheim, a director of research and innovation at Larun’s institute, sent Dr Tovey the long-awaited revision of the exercise review. On May 24th, Dr Tovey informed Dr Fretheim that Cochrane did not plan to publish it.

“With great regret and disappointment, I am afraid that I am not able to publish this version,” wrote Dr Tovey. “As we have discussed, the scrutiny that this review and this decision will be subject to means that we cannot publish a new version unless it is clear that the review is as watertight as humanly possible…In particular, I remain concerned that the presentation in general, but in particular of the short term results on fatigue, remains too optimistic.”

In discussing exercise therapy elsewhere in the e-mail exchange, Dr Tovey referred to it as “an intervention that really probably has very little to contribute”—a view at serious odds with the review’s claims of benefits. (Dr Tovey is now retired, and Dr Karla Soares-Weiser has assumed the position of Cochrane’s editor-in-chief.)

The wrangling over the review has embroiled Cochrane’s editorial leadership in the protracted dispute between advocates of integrity in science and the GET/CBT ideological brigades, who continue to have significant support among pooh-bahs within the UK academic and medical establishments. Experts in study design—that is, experts outside the precincts of the psychiatry departments at Oxford, Kings College London and some other select British institutions–recognize that unblinded trials relying on self-reported outcomes, like those included in the Cochrane review, generate unknown amounts of bias and cannot provide reliable evidence.

I should disclose that I have been told second-hand by a reliable source that Larun considers my work on the Cochrane review tantamount to “harassment.” If she does in fact believe this, it would demonstrate that she cannot distinguish between “harassment” and tough but fair criticism. I have never contacted Larun. When I was in Norway last November, I met with Dr Fretheim and another official from Larun’s agency, the Norwegian Institute of Public Health. I explained to them in sharp terms why I thought the Cochrane exercise review was deficient. Expressing my professional opinion about an influential paper to health officials cannot reasonably be defined as “harassment,” no matter how much the lead author dislikes my views.

*****

To recap briefly what’s happened in this saga till now: After Cochrane published the exercise review by Larun and three co-authors, two patients, Tom Kindlon and the late Robert Courtney, wrote extensive critiques in the correspondence. Larun provided deficient and intellectually bankrupt responses. Courtney then sent a formal complaint to Cochrane. Last year, after investigating Courtney’s concerns, Cochrane acknowledged their validity and asked Larun and her co-authors to address them. Unfortunately, Courtney was already dead. I have written most extensively about the Cochrane situation here. (In July of last year, I met with Dr Tovey to press the argument that the review was deficient in multiple ways.)

One of several key issues was the review’s claim that PACE was at low risk of selective outcome reporting bias—even though the investigators changed their primary outcomes well into the conduct of the trial. In the correspondence following publication of the review, Larun insisted that the primary outcomes in PACE were “pre-designated” because they “were drawn up before the analysis commenced and before examining any outcome data.” In advancing this claim, Larun was parroting the bogus argument often promoted by the PACE authors themselves to excuse their dramatic outcome-switching. Apparently it needs to be repeated that, in open label trials with self-reported outcomes, trends in results should be apparent to investigators long before they see the actual data.

By sometime last year, Dr Tovey had apparently decided that the review should be withdrawn, pending the revisions. Last October, Reuters UK science “reporter” Kate Kelland, a bestie of the so-called Science Media Centre, broke the news of the attempted withdrawal with a profoundly biased article that portrayed Cochrane, and Dr Tovey personally, as caving in to rabid patient activists. The Reuters article quoted from an e-mail exchange between Cochrane editors and the review team about the proposed withdrawal. Although Reuters did not disclose the source of the e-mails, they were presumably shared with Kelland by those with an interest in pushing back against Cochrane’s decision to withdraw the review.

Predictably, given Kelland’s ties to the SMC and apparent sympathy for the plight of the unhappy researchers, the article failed to discuss the methodological issues at stake or the scientific reasons behind the decision to withdraw the review. In response, Virology Blog organized and posted a statement in support of Cochrane’s decision; the statement outlined the review’s serious flaws and was signed by more than 40 scientists, clinicians and academics from Harvard, Columbia, London School of Hygiene and Tropical Medicine, Berkeley and many other leading institutions.

Unfortunately, the distorted Reuters article stigmatized patients and lent credence to the manufactured argument that Cochrane was surrendering to an irrational and anti-scientific rabble. In the face of this public smackdown, Cochrane switched gears. A week after the Reuters article appeared, the organization posted a notice that a revision was in the works–but it did not withdraw the current review. In late November, Cochrane announced that it had rejected this revision, leaving the situation in some limbo.

The next step was a March announcement of further changes to the review that were “aimed at improving the quality of reporting…and ensuring that the conclusions are fully defensible and valid to inform health care decision making.” Some version of that revision was the one rejected by Dr Tovey in May. Presumably a re-revised version is what was being referenced in the June 17th statement.

Incidentally, the phrasing of the March announcement suggests that the conclusions of the current version are not “fully defensible and valid to inform health care decision making.” Given that, it is fair to ask Cochrane how the organization justifies having backed down on its earlier decision to withdraw the review pending acceptable revisions.

*****

After Dr Tovey rejected the revision on May 24th, Dr Fretheim responded by proposing further possible tweaks to settle their differences. The subsequent exchange featured spirited and somewhat testy debates over the obstacles to publication—in particular a conflict over the quality and/or certainty of the evidence supporting exercise as a beneficial treatment for fatigue.

According to the 2014 review, the evidence that exercise therapy reduces fatigue is “moderate-quality”—even though all the studies included in the review were open label studies with self-reported outcomes. Since self-reported outcomes in open-label studies are by definition subject to unknown amounts of bias, the “evidence” cannot reasonably be described as of “moderate” quality.

Dr Tovey’s e-mails make clear that he considered the quality of evidence in support of the claims for exercise therapy and fatigue reduction to be “low,” not “moderate.” As he wrote, “the conclusion that this is moderate certainty evidence seems indefensible to me, and as we know, I am not alone in this…It seems absolutely clear to me that this is low certainty evidence.” In response, Dr Fretheim suggested a compromise, in which the evidence would be characterized as “low-moderate.” It is unclear if or how this issue has been resolved in the revision now in the pipeline for publication.

Besides that dispute, some other changes appeared to have been agreed upon. According to an attachment sent to Dr Tovey by Dr Fretheim, the revision would include a statement that “patients diagnosed using other criteria may experience different effects.”

If the revision is indeed published, it will be critical to highlight this point prominently. The trials included in the review recruited patients based on either the Oxford criteria or the Fukuda criteria. Both of these case definitions are fatigue-based. Unlike more recent case definitions, such as the 2011 International Consensus Criteria for ME, they do not require the presence of post-exertional malaise and are more likely to generate heterogeneous study samples. As has been frequently noted, exercise therapies could very well be helpful for many people whose fatigue arises from undiagnosed depression or anxiety disorders, or other conditions.

Although not discussed in the e-mails released, Cochrane should also have insisted that any revised review include the various objective outcomes from the studies. Given that objective outcomes in ME/CFS trials have repeatedly failed to match the benefits found in self-reported results, including them would have required a downgrade in the ratings for exercise therapy. The 2014 review was based on an earlier protocol that did not include these objective measures. But it was a mistake for Cochrane to approve the update without ensuring that objective measures were added to the analysis. Any revision without them will be biased toward the positive. Cochrane should understand that it is unacceptable to publish a review—or republish a review–while ignoring key findings that would have forced a reassessment of the reported results.

On May 29th, Dr Fretheim proposed submitting the matter to a mutually agreed upon “arbitrator.” Perhaps that has happened, perhaps not. No e-mails from after May have yet been released. At this point, Cochrane is in a fix. Its editors must know, or at least suspect, that publishing this new version could be a big, big mistake. If Cochrane endorses a revision that turns out to be more propaganda for the purported benefits of exercise therapy, its reputation is likely to suffer far greater damage than if it pulled the plug on this disaster now and began the process all over again.