The Cochrane Controversy

By David Tuller, DrPH

Cochrane–formerly called the Cochrane Collaboration–is respected worldwide for its systematic reviews of medical treatments. These reviews are often cited as the definitive source of information about treatment efficacy and safety. In taking on the thankless task of assessing the data on commonly used interventions, Cochrane performs an invaluable public health service and has advanced the cause of evidence-based decision-making in medicine.

But like any organization, Cochrane can get things wrong—as it has in the case of chronic fatigue syndrome. (Cochrane generally uses the term CFS, so I will also when referring to these systematic reviews.) Cochrane’s review of cognitive behavior therapy for CFS was published in 2008, pre-PACE. The most recent review of exercise therapies for CFS, which mainly included studies of graded exercise, was published in 2014. These systematic reviews and previous versions, all of which reported benefits from the treatments, were conducted by Cochrane’s Common Mental Disorders group.

Last month, The Times and The BMJ covered the growing international concerns about the PACE trial. Both publications ran articles about Virology Blog’s most recent open letter to The Lancet, which cited PACE’s “unacceptable” flaws and called for a fully independent reanalysis of the trial data. The letter was signed by 114 experts, ten members of Parliament, and 70 patient and advocacy organizations. To counter this sort of public criticism and support their unwarranted claims, the CBT/GET ideological brigades and their enablers regularly cite Cochrane’s systematic reviews.

Most recently, Professor Fiona Watt, executive chairwoman of the UK Medical Research Council, released a statement in response to The Times’ coverage of the Lancet open letter. The MRC, the main funder of PACE, has previously defended the conduct of the study. In her letter, Professor Watt reaffirmed this support without providing any response to the specific concerns raised about PACE—such as the paradox that 13 % of the participants were already “recovered” on the key outcome measure of physical function at baseline, before any treatment at all.

Professor Watt’s defense of PACE rested heavily on the fact that other researchers have similarly reported benefits from CBT and GET. She noted pointedly: “This evidence is summarised in three Cochrane reviews. Cochrane reviews are systematic reviews of primary research in human healthcare and health policy, and are internationally recognised as the gold standard in evidence-based healthcare.” [It is not clear which is the third Cochrane review being referenced here.]

It takes nothing away from Cochrane’s reputation to note that systematic reviews are only useful if the studies they include provide valid and reliable data. If the studies themselves are fundamentally flawed, and if the purported experts conducting and writing the reviews refuse to acknowledge or cannot understand these shortcomings, then any synthesis or summation will generate similarly problematic conclusions. This is what appears to have happened with the systematic reviews for CFS treatments conducted by the Common Mental Disorders group.

**********

Let’s dispense with one issue right away: This illness should not be housed in the Common Mental Disorders group. Whatever the historical reasons for this arrangement, it undoubtedly must lead observers to assume that Cochrane as an organization endorses the framing of CFS as a psychiatric illness. Patients object to the situation not because they are prejudiced against psychiatry and people with mental disorders–as the PACE authors and others have claimed—but because their illness is not a mental disorder and because the Common Mental Disorders group has already demonstrated its inability to assess the research accurately.

(Cochrane editor-in-chief David Tovey and psychologist James Coyne have previously engaged in a public debate over issues related to conflicts-of-interest of Cochrane reviewers in this domain. I am not addressing those issues in this post.)

The Common Mental Disorders group’s most recent version of the exercise systematic review drew skeptical scrutiny from very smart advocates soon after its 2014 publication. Patient-researchers Tom Kindlon and the late Robert Courtney, in particular, submitted cogent and comprehensive comments that exposed the systematic review’s serious flaws and refuted its unfounded claims. When Cochrane republished the review last year, it included the exchanges between the correspondents and Lillebeth Larun, the lead author.

Larun, a researcher and associate professor in the department of assessment interventions at the Norwegian Institute of Public Health, provided inadequate defenses to the concerns raised by Kindlon and Courtney. I won’t dissect the arguments here. But Larun’s response to one problem is worth highlighting for its audacity in re-purposing English to justify poor methodology.

In PACE, the investigators switched their methods of assessing their primary outcomes from those detailed in their protocol. These outcome switches—which produced numbers that favored a more positive interpretation of the results—took place after data collection. The PACE investigators have nonetheless referred to these revised assessment measures as “pre-specified” because, as they have explained, the changes were made before they examined their data.

The issue is significant because it impacts how Cochrane reviewers should assess PACE’s risk of bias. In Cochrane’s own guidelines for assessing a study’s risk of bias across multiple domains, the requirements for being considered at “low risk” of bias when it comes to reporting results include the following: “The study protocol is available and all of the study’s pre-specified (primary and secondary) outcomes that are of interest in the review have been reported in the pre-specified way.”

That sentence seems clear. “Pre-specified” in this case means “specified in the protocol before the beginning of the trial.” It is indisputable that PACE was not reported in this “pre-specified” way. Yet Larun, parroting the PACE authors, has chosen to re-define the word so that it can encompass what actually happened in the trial. Here is what she wrote about the outcome-switching: “These changes were drawn up before the analysis commenced and before examining any outcome data. In other words they were pre-specified, so it is hard to understand how the changes contributed to any potential bias.” She assessed PACE as having a “low risk” of bias.

Larun’s position is unsustainable. Clinical trial investigators write protocols so that everyone understands what the goal-posts are. No matter what Larun and the PACE authors might argue, “pre-specified” does not mean “specified post-data-collection-but-pre-data-viewing”—and that’s per Cochrane’s own risk-of-bias guidelines, which Larun apparently decided she could ignore. Moreover, PACE was an open-label trial relying on subjective outcomes. In such cases, investigators are likely to know the outcome trends long before they look at any actual data. In this context, to define the reported PACE outcome measures as “pre-specified” is ridiculous.

**********

With more than 600 participants, PACE was the largest treatment trial for the illness. Even so, removing it from the exercise systematic review would not change the overall conclusions. But both the exercise and CBT systematic reviews suffer from other deficiencies that render their findings suspect and essentially meaningless. Putting aside PACE’s outcome-switching and other unique flaws, the trial exemplifies two major problems that plague much or most of the CBT/GET research for this illness. The first is that many studies use overly broad case definitions. The second is that the studies are open-label trials that rely on subjective outcomes.

The first problem means that study samples are likely to include a heterogeneous collection of people suffering from chronic fatigue for any number of reasons, including depression and anxiety disorders, but not necessarily the illness supposedly being investigated. It is possible that some of these other participants could benefit from CBT and GET, complicating any efforts to interpret the findings.

The second problem—combining open-label status with subjective outcomes–means that positive self-reports from participants in treatment arms could easily be due to bias. Since participants know their treatment allocation as well as whether the treatment is supposed to help them, their responses are likely to be influenced by hopes and expectations. It is not clear why systematic reviews should include such trials at all, just as it is not clear why anyone would spend much money conducting them in the first place.

Do systematic reviews of pharmaceuticals generally include such trials and assess them as providing robust evidence with a low risk of bias? If not, why is that appropriate in the case of this illness and these studies? In any event, if Cochrane feels it must include these inherently unreliable trials in systematic reviews, then its guidelines should automatically designate them as having a high overall risk of bias, even if they boast other laudatory traits.

Systematic reviews that includes studies with these thorny problems will feature some of the same defects themselves. Unfortunately, such reviews will provide little or no information about people suffering from the illness in question as defined through more precise definitions. In this case, that means not only the CDC’s 1994 Fukuda definition (for CFS) but also two superior ones drawn up by international committees of experts–the 2003 Canadian Consensus Criteria (for ME/CFS) and the 2011 International Consensus Criteria (for ME). (There is also the US Institute of Medicine’s 2015 clinical case definition for systemic exertion intolerance disease, or SEID, but that’s another issue.)

And such systematic reviews will provide no information about whether objective measures support the positive subjective reports. Larun has acknowledged that including objective outcomes in the exercise systematic review would be helpful. However, to justify having excluded them from the current exercise review, she noted that they were excluded from the systematic review protocol. Of course, that reasoning just raises the question of why objective findings were excluded from the protocol.

When it comes to this illness, objective findings have generally not supported the published subjective results. Including objective data in the systematic review would therefore have required a downward reassessment of the reported benefits of the interventions. Perhaps that is one reason it was decided to leave these data out of both the protocol and the review.

**********

The Common Mental Disorders group has written a second exercise systematic review, using individual participant data from the various trials rather than just the published results. This IPD review was reportedly supposed to have been published last year, but it remains unpublished. It was known that Cochrane—to its credit—sent it out for peer review to people beyond the usual orbit. These further peer reviews were said to have been scathing. This would not be a surprise to anyone outside the biopsychosocial bubble-think.

Cochrane is aware that concerns about PACE and this entire field of research have extended beyond the patient and advocacy communities. It knows the US National Institutes of Health and the Institute of Medicine (now the National Academy of Medicine) released major reports three years ago that declared the illness to be organic and not psychological in nature. It knows that the US Centers for Disease Control has rejected CBT and GET as treatments for what it now calls ME/CFS; that the US Agency for Healthcare Research and Quality has downgraded its recommendations for CBT and GET after stratifying the analysis by case definition; and that the UK National Institute for Health and Care Excellence is pursuing a “full update” of its guidance.

In other words, international support for the CBT/GET paradigm is crumbling. Yet members of the Common Mental Disorders group still champion these treatments, basing their arguments on deficient research. This presents a challenge for Cochrane. The challenge involves not just what to do with the unpublished IPD review but how to handle the published reviews as well. These reviews, and in particular the exercise review, continue to exert a harmful impact on patient treatment options, as I noted in a recent post about the Mayo Clinic. That will continue as long as CBT/GET promoters can hide behind Cochrane’s skirts.

In the near future, Cochrane needs to make some tough decisions, announce its plans, and then clean up the mess created by the Common Mental Disorders group. Should the current systematic reviews be withdrawn? (Absolutely, from my perspective, with the reasons clearly outlined.) Or should they be slapped with warning labels while experts unaffiliated with the Common Mental Disorders group reconsider the entire enterprise and develop a new strategy for assessing studies and analyzing the data?

Would any subsequent systematic reviews be required to differentiate results based on case definition? Would these reviews highlight objective outcomes? If open-label trials relying on subjective outcomes are to be included, would they be appropriately assessed as having a high overall risk of bias?

Professor Watt’s recent defense of PACE suggests that deference to authority still outweighs scientific reasoning in powerful sectors of the UK medical-industrial complex. Cochrane will likely face pushback for seeking to address the flaws of these systematic reviews, so taking corrective action won’t be easy. But to protect patients’ health, it must be done.