By David Tuller, DrPH
*This is a crowdfunding month at UC Berkeley for my Trial by Error project. If you appreciate my work and would like to help support it, here’s the link for this November’s campaign.
As I pointed out last month when I reviewed a ridiculous study of “psychosomatic therapy” for “persistent somatic symptoms,” the Dutch psychologizers can’t seem to stop churning out poor-quality research purporting to support claims that their interventions are effective. Before the psychosomatic therapy study was the one investigating cognitive behavior therapy (CBT) for fatigue associated with long Covid, in which the authors declared success but failed to disclose that they had null results for their one objective measure—how far participants moved, as assessed by actigraphy.
Now we have another study of this ilk, this time a meta-analysis of individual patient data from a batch of studies of CBT for ME/CFS: “Does the effect of cognitive behavior therapy for chronic fatigue syndrome (ME/CFS) vary by patient characteristics? A systematic review and individual patient data meta-analysis.” The lead and senior authors are the same as for the recent study of CBT for long Covid –Dr Tanja Kuut and Professor Hans Knoop, both from the University of Amsterdam.
The new meta-analysis was published by Psychological Medicine, a journal that is essentially an in-house publication for the stream of shoddy papers emerging from members of the CBT/GET ideological brigades. (The Journal of Psychosomatic Research performs a similar function.) To make it more of an inside job, all eight studies included in the meta-analysis were done by investigators from the same group of researchers that includes Dr Kuut and Professor Knoop.
Given the background of both the authors and the journal, low expectations for this meta-analysis would be warranted. And such low expectations are rewarded. The authors seem to think that by slicing and dicing old data in new ways they will come up with refurbished findings that bolster their case. But this notion is a delusion.
Here is a quote from Professor Knoop in a press release about the study: “These results give a clear picture that CBT can be beneficial for a substantial number of patients.” But Professor Knoop and his colleagues still fail to grasp or acknowledge an essential problem–studies that are unblinded and rely solely on subjective outcomes for their claims of success are prone to an unknown amount of bias. The results are therefore hard if not impossible to interpret. All eight studies included in this meta-analysis featured this design. Meta-analyzing data fraught with bias does nothing to improve the quality of the data or minimize the bias.
To justify this reliance on subjective outcomes, the paper includes the following statement: “All case definitions of ME/CFS rely on reports of patients of subjectively experienced symptoms. Therefore the efficacy of interventions aimed at symptoms of ME/CFS can only be determined with patient-reported outcome measures.”
This is simply untrue. The efficacy can also be assessed by objective indicators of actual functioning—such as actigraphy, which measures physical movement. The investigators themselves know this, since they have used actigraphy in previous research. Other possible indicators include employment status, social benefits status, and assessments of physical fitness and walking ability. None are perfect. But they are certainly relevant to determining the efficacy of “interventions aimed at symptoms of ME/CFS.”
In previous research of CBT for ME/CFS from members of the Dutch group of investigators, the actigraphy results have been poor, in contrast to the positive subjective reports. Those findings were left out of the original trial reports—as they are left out of this meta-analysis. Ignoring null objective findings while touting positive subjective ones is the epitome of what would be called cherry-picking. This self-serving decision is unacceptable and raises questions about the integrity of the authors as well as of the meta-analysis itself.
The eight studies included a total of 1298 participants. The primary outcome was fatigue severity; the secondary outcomes were functional impairment and physical functioning. The studies confirmed the basic principle that unblinded trials testing therapeutic interventions and relying on subjective outcomes will tend to yield positive results. No surprises there! According to the meta-analysis of moderating factors, the effects of CBT varied with age, functional impairment status, and whether participants’ activity levels were what the investigators called “fluctuating.” The meta-analysis found no differences in outcomes according to whether or not participants reported having post-exertional malaise (PEM).
The article is one of a number recently that tried to push back against the 2021 ME/CFS guidelines issued by the National Institute for Health and Care Excellence (NICE). These guidelines reversed NICE’s previous recommendations for CBT and GET as curative treatments, although they allow for CBT as a supportive intervention. As part of its decision-making process, NICE placed greater weight on studies that required the presence of PEM. The authors of the new meta-analysis make it clear they believe that decision to be unjustified, asserting that “our findings do not support the decision of NICE to downgrade evidence from studies not mandating the presence of PEM in their updated guideline.” In reality, their findings are neither here nor there regarding this matter. The authors claim that most of the participants in the combined sample experienced PEM, but it is unclear how or how carefully this trait was assessed in each of the included studies.
The meta-analysis also notes this: “In the Cochrane risk of bias tool studies are penalized if the outcome assessor (the patient) was aware of the intervention received. However, this limitation is inherent to the evaluation of behavioral/psychotherapeutic interventions using a subjective outcome measure.”
It is true that this limitation is inherent to the study design mentioned—an unblinded trial relying on subjective outcomes. But that recognition doesn’t mitigate the bias generated by this combination of factors. These authors have themselves chosen not to include objective measures of functioning. Given that choice, they don’t then get to exempt their work from being penalized for this obvious deficiency. Like other offerings from Professor Knoop and his minions, this methodologically defective paper is a piece of crap. The reported benefits for CBT cannot be taken at face value.