The 2018 PACE Reanalysis and the SMC’s Expert Appraisals

By David Tuller, DrPH

It has been almost two years since BMC Psychology published a key reanalysis of raw data from the PACE trial. Given the significance of this paper (of which I was the least important of seven co-authors), I figured it wouldn’t hurt to highlight it again.

The heroic Alem Matthees, a patient in Perth, Australia, succeeded in liberating the relevant data through a Freedom of Information request, but only after Queen Mary University of London spent £250,000 in legal fees in its efforts to prevent access. The reanalysis documented that the benefits for CBT and GET reported in multiple PACE papers were either exaggerated or illusory when the data were assessed per the methods detailed in the trial’s published protocol.

The paper, which appeared in March of 2018, was called “Rethinking the treatment of chronic fatigue syndrome: a reanalysis of findings from a recent major trial.” Its publication ensured that these criticisms of PACE had been vetted through the recognized scientific process; that obviously was not the case with my 15,000-word investigative journalism series about the trial, which ran on Virology Blog in October of 2015. Anyone who now cites the PACE trial as evidence of the effectiveness of the CBT/GET paradigm also has an obligation to cite the peer-reviewed reanalysis that challenges the core findings.

When the reanalysis was published, the Science Media Centre invited two experts to comment: Chris Ponting, a professor of medical bioinformatics at the University of Edinburgh, and Jon Stone, a consultant neurologist and an honorary senior lecturer at the same university. (He is now an honorary professor at the University of Edinburgh.) The SMC also posted a statement from the PACE team, which featured a few short paragraphs of their typical evasions. These included the argument that the published protocol in which they detailed their outcome measures was only a “preliminary” analysis plan. Apparently, no one should have been bothered that they weakened key outcome measures after gathering data and then refused to provide the less appealing results they would have obtained without these changes.

The reanalysis was published around the time that Professor Ponting was assuming the position of vice chair of the CFS/ME Research Collaborative. Given the responsibilities of that role and the questionable behavior of his methodologically and ethically challenged predecessor, I was pleased to read his positive critique. I assumed the SMC, a cheerleader for the CBT/GET ideological brigades, did not expect a review that would extol the reanalysis and explicitly reference the PACE trial’s “nonsensical” quality.

Here’s what Professor Ponting wrote:

In 2011 the PACE group interpreted their randomised trial data to mean that cognitive behaviour therapy (CBT) and graded exercise therapy (GET) “can safely be added to [specialist medical care] to moderately improve outcomes for chronic fatigue syndrome [CFS, or M.E./CFS].” Now Wilshire and coauthors report on a new analysis of the PACE trial’s data. This reanalysis was required in part because the trial group had revised their analysis from the plan published in their protocol. This revision meant that, in theory, some trial participants nonsensically could be accepted on to the trial as patients, yet then be considered to have recovered by the trial’s end despite their symptoms not improving, or even deteriorating. Wilshire et al. provide evidence that “the effects of CBT and GET were very modest” and not statistically reliable overall if we apply procedures very close to those specified in the original published protocol. Their analysis also revealed that recovery rates according to the protocol definition were much lower than previously published and that CBT and GET did not lead to recovery.

Importantly, Wilshire et al. provide a plausible explanation even of these modest effects. Specifically, they argue that they are explicable simply from the raised expectations of CBT and/or GET participants that their treatments would be effective. Expectations are heightened, they argue, when participants are not blinded to their treatment and are assured that their treatment is effective, as was the case for the PACE trial. The absence of meaningful gains in objective outcome measures such as fitness indicates that gains on self-report measures may well be unreliable. This could largely explain both the modest effects seen after 1 year and the disappearance of these effects subsequently. The authors also make the case that the lack of substantial and enduring effects of CBT and/or GET seen from a trial of the size of PACE implies that these therapies are unlikely to be commonly effective.

Shortly after I read Professor Ponting’s assessment of the reanalysis, I e-mailed him and thanked him for his honesty and integrity.

**********

In his SMC statement, Professor Stone took a harsher view of the reanalysis and seemed protective of PACE. As he acknowledged in an accompanying disclosure, he is a former student and current colleague of Professor Michael Sharpe, a lead PACE investigator. He also collaborates with Professor Trudie Chalder, another lead PACE investigator. (Professor Chalder famously–and falsely–declared at the 2011 PACE press conference that twice as many in the CBT and GET groups as in the other groups got “back to normal.” To my knowledge, she has never publicly explained or apologized for that mischaracterization.)

From Professor Stone’s comments, it would be fair to wonder whether his sense of loyalty has impacted his scientific judgement in this context. The SMC would have done both Professor Stone and itself a favor had it sought input from a more neutral observer. Below, I have posted Professor Stone’s remarks about the reanalysis in italics, with my responses in bold following each paragraph:

This reanalysis of data from the PACE trial shows that using more stringent outcome measures, patients with CFS/ME had a better outcome at one year with Cognitive Behavioural Therapy(CBT), Graded Exercise Therapy (GET) than the control group, but the proportion improved was not as large as in the original report (20%/21% vs 10% in this reanalysis). This is not a surprising result. That would be the expected effect of using more restrictive criteria in any similar trial. The apparent benefit will be reduced for those undergoing the intervention, and also for the controls.

This framing omits a salient detail: these “more restrictive criteria” were those that the trial investigators themselves outlined in their protocol, abandoning them in favor of weaker ones only after gathering data. As Professor Stone noted, it is not surprising that analyzing the data with more restrictive criteria would reduce the apparent effectiveness of CBT/GET. By the same token, it could not have been surprising to the investigators themselves that their decision to weaken their outcome criteria would generate more appealing results. The corollary of Professor Stone’s point is that the PACE investigators must have or should have known that their multiple outcome changes would inevitably make their findings look better.

Has CBT and GET been oversold for CFS/ME? There is no doubt we need better treatments. They only have a modest treatment effect overall and are certainly not suitable for all patients with the condition. The evidence that these treatments lead to a better outcome than controls at 2.5 years is also lacking, both from the studies original authors and from this analysis. But that is because the trial only lasted one year. Cohort data and the post-hoc subgroup analysis in this reanalysis can’t be used to resolve that issue, but does at least provide evidence that benefit in the CBT/GET groups was sustained, even if the other groups subsequently caught up.

It is the PACE investigators and their colleagues who have oversold CBT/GET. Members of the PACE team spent years advising disability insurers that patients who have not undergone CBT/GET should not receive disability and other social welfare payments. Such categorical recommendations seem unreasonable for treatments that, per Professor Stone, “only have a modest treatment effect overall and are certainly not suitable for all patients with the condition”–and that also seem to have no long-term benefits.

As for the follow-up findings, the main result in a clinical trial, even with post-trial follow-ups, is a comparison between the groups, not the “within-group” comparison cited by Professor Stone. It was inappropriate as a matter of science for the PACE investigators to spin their findings and present these “within-group” data as the main result, and it was inappropriate as an editorial matter for the journal to approve the study for publication in that form. The follow-up’s main result was that there were no long-term differences between the groups, with whatever needed caveats and limitations. In fact, this result matches other long-term follow-up data from related research.

One of the assertions of this reanalysis, that treatment effects seen may simply be the result of unblinded self-report measures, would not explain why patients undergoing adaptive pacing therapy (APT) did worse than those receiving CBT and GET in the original study who were also unblinded to therapy. It is disappointing that the pacing therapy data was not reanalysed as part of this study.

The PACE investigators themselves have deployed similar reasoning–that the reported results contradicted expectations, since before the trial began participants expressed favorable attitudes toward the so-called pacing intervention. (I use “so-called” because APT is not pacing; it is the PACE trial’s operationalized version of it.) Yet in mounting such arguments, the PACE investigators and Professor Stone have ignored something that the reanalysis specifically discussed and that Professor Ponting highlighted: Participants in the CBT/GET arms were routinely primed about the purported benefits of the treatments, while those receiving APT were not. Such differential messaging to participants would be likely to induce bias in their subjective responses.

The reanalysis further noted that APT performed similarly in the PACE trial to so-called “standard medical care.” There was no need to analyze these APT data separately in order to assess the validity of the claims of benefits from CBT/GET.

We need better treatments for CFS/ME, be they biological treatments directed at the associated pathophysiology of the condition, or more effective forms of rehabilitation. Until we have these, the question is whether it is better to offer a modestly effective treatment supported by data from many other trials, with a realistic discussion of its pros and cons, than none at all.

According to the PACE trial’s descriptions of CBT and GET and previous writings of the main investigators, these interventions presume that there is no “associated pathophysiology” that accounts for the perpetuation of the symptoms–just unhelpful cognitions and the effects of deconditioning. That viewpoint implies that research into “associated pathophysiology” for the condition would be pointless. Acknowledgement of the existence or possible existence of “associated pathophysiology” would undermine the theoretical rationale for the CBT/GET paradigm, contradict key material in the PACE treatment manuals for both therapists and participants, and require significant changes in the design and implementation of the interventions.

Regarding the “many other trials” that support the use of CBT/GET, they all share a fundamental design flaw with PACE–they are open-label trials relying on subjective outcomes. This combination of traits is known to create a serious risk of bias, for multiple reasons. Moreover, in PACE and related trials, objective outcomes have overwhelmingly failed to support subjective reports of benefits–yet these outcomes have routinely been dismissed or minimized by the investigators themselves. In short, there is little or no valid evidence from this body of research that CBT and GET are effective treatments.

The PACE Trial has been subject to an extraordinary degree of hostility. On the basis of this reanalysis, this was not warranted–the basic difference in outcomes are still present. Additionally, it is worth reflecting that positive trials of CBT for fatigue in conditions such as Multiple Sclerosis with similar treatment effects do not mean that MS is a psychological condition. The same is true for CBT/GET and CFS/ME.

The major criticisms of the PACE trial’s methodological and ethical missteps have been very much warranted. Large segments of the international scientific community, including the US Centers for Disease Control and Prevention, have rejected the PACE findings and the CBT/GET treatment paradigm. In PACE, the investigators weakened key outcome measures after collecting data and then refused to provide the analyses they promised in their published protocol. They accused patients of being “vexatious” for pursuing their legal right to seek access to publicly funded trial data.

When a tribunal dismissed the PACE investigators’ arguments in scathing terms and ordered them to release the data, it turned out patients were right–the outcome changes made the results look much better than they otherwise would have. In the reanalysis, the evidence that the treatments led to “recovery” disappeared. The extremely modest reported benefits were shown to be well within what would be expected from bias or an expectations effect, especially given the positive messages delivered to those in the CBT/GET arms.

It would be silly to interpret these disastrous results as confirming “the basic difference in outcomes,” but Professor Stone is free to do so.

One quirk of PACE was that 13% of participants were already “within normal range” or “recovered” on the key variable of self-reported physical function when they entered the trial. This perplexing paradox was not disclosed in the paper. As far as I have seen, the PACE investigators have never admitted that this overlap is problematic or raises any questions of interpretation. While Professor Ponting cited this unique and “nonsensical” feature of the PACE trial in his comments, Professor Stone did not. Perhaps he believes it is standard practice for a significant number of trial participants to have met key outcome thresholds at baseline and that people who request adequate explanations for such statistical anomalies are demonstrating “an extraordinary degree of hostility.”

Regarding fatigue in connection to other diseases, no credible authority (as far as I know) has suggested that CBT for MS-related fatigue will “reverse” the MS and lead to “recovery” from it. But that is the claim that has routinely been made for this therapy in the domain under discussion, including in the PACE trial. The comparison with MS-related fatigue is not readily applicable to the debate.