By David Tuller, DrPH
I am reposting below a blog about a “living systematic review” of interventions for Long Covid, which was published by The BMJ in late 2024. This systematic review figured prominently in the opening of the recent Wired piece about mind-body interventions for Long Covid. (I’ve posted about the Wired piece here, here and here.) The journalist—Alan Levinovitz, a professor of religion at James Madison University in Virginia—presented the systematic review’s conclusions uncritically. He did not discuss the review’s serious shortcomings, which raise questions about the reliability of its conclusions.
Here’s what Levinovitz wrote in Wired about the systematic review near the top of his story:
“The BMJ’s systematic review of ways to treat long Covid lists two as supported by moderate evidence, cognitive behavioral therapy and physical exercise. But if you attended the third annual Long Covid International Conference in Boston—as I did, late last year—you’d think the BMJ was encouraging medical malpractice. During two days of presentations, the world’s leading scientific authorities brought up exercise only to warn against it. Cognitive behavioral therapy received just one mention: ‘not recommended.’”
For Levinovitz, this negative attitude toward cognitive behavior therapy (CBT) and exercise is odd and hard to understand. That’s not surprising, given his apparent belief that 60-80% of Long Covid cases are “psychogenic,” per an X posting last year. However, what is actually odd and hard to understand is why major journals like The BMJ continue to publish work featuring errors that would be obvious to first-year epidemiology students at Berkeley.
Perhaps the Long Covid International Conference attendees didn’t focus on these interventions because they knew that many of their patients had already tried them and had not improved, or had even gotten worse. Or perhaps the attendees already knew not to take these studies at face value. In unblinded trials of non-pharmacological interventions, reports of modest improvements in subjective measures are to be expected–even in the absence of an actual therapeutic effect. The study design alone ensures the presence of an unknown amount of bias, so findings must be taken with a huge amount of skepticism.
My colleague and friend Brian Hughes, a professor of psychology at the University of Galway, in Galway, Ireland, was less enamored of the BMJ systematic review than Levinovitz. Back when it was published, he issued a sharp critique on his blog, The Science Bit. His post was titled “That BMJ review of Long Covid therapies does not show what it says it does.” Here’s a taste of his view:
“The authors are able to produce their findings only by looking at studies they themselves declare are biased. When these biased studies are excluded, the so-called evidence disappears…The BMJ review reveals that there is, in fact, NO UNBIASED EVIDENCE THAT CBT OR PHYSICAL REHABILITATION IMPROVE SYMPTOMS OF LONG COVID.”
Below is my blog about the systematic review—first posted on December 3, 2024. The review’s findings on exercise and CBT were largely based on two trials.
As I pointed out in that blog, the main trial in the systematic review that appeared to support exercise had been slapped with a major correction earlier in 2024. The correction made it completely clear that the trial’s findings only applied to patients who had been hospitalized for COVID-19, not to the larger Long Covid population. This key information does not appear in the most high-profile sections of the systematic review–even though the review was published after the correction.
In the trial that reported modest improvements in self-reported fatigue with CBT, the investigators had null results for their sole objective results–how far participants moved, as measured by a wearable device. Then, in a troubling lack of transparency, they failed to report these null results in the published paper. The systematic review, of course, also failed to mention the null results.
In any event, if experts at the Long Covid International Conference had read the BMJ systematic review, it wouldn’t be surprising if they ignored the conclusions–even though Levinovitz found the situation perplexing.
**********
Trial By Error: Yet Again BMJ Recommends CBT and Exercise for Long Covid (December 3, 2024)
What is going on at The BMJ? In May, the journal corrected an obvious error in a paper about a prominent Long Covid mental and physical rehabilitation trial called REGAIN. The trial was conducted among patients who had been hospitalized for Covid-19, but key sections of the paper generalized the findings to all Long Covid patients—a huge lapse that should have been caught by peer-reviewers, not to mention The BMJ’s own editorial team. Given that the great majority of Long Covid patients have not been hospitalized, the correction essentially undermined any claim that the REGAIN findings could be extrapolated to everyone with the illness.
Now The BMJ has published a major review of interventions for Long Covid that compounds the original mistake. The review’s abstract claims that, based on “moderate certainty evidence,” the REGAIN intervention “probably” improves Long Covid symptoms–without noting that the trial participants had been hospitalized. The review mentions this highly salient fact only deep in the text of the paper. But the broad statements in the abstract and elsewhere, which essentially extrapolate the purported benefits to all Long Covid patients, seems to have raised no questions among peer reviewers. Nor did this excessively expansive interpretation of the REGAIN results lead to any apparent concern among editors at The BMJ, who presumably should have known that the referenced trial, published by their own journal earlier this year, already bore an embarrassing correction for having misrepresented its findings.
This means the review itself requires a similar correction—and I plan to urge the journal to take such action.
The review—which also cites a blog post of mine while misrepresenting my qualifications–is called “Interventions for the management of long covid (post-covid condition): living systematic review.” (Can someone please explain to me the difference between a “living” systematic review and a “dead” one?) The review includes 24 clinical trials, with four each involving drug interventions, dietary approaches, and medical devices or technologies; three involving behavioral interventions, including cognitive behavior therapy (CBT); and eight involving physical activity or rehabilitation. One included trial—REGAIN—involved a combination of mental and physical rehabilitation.
As with REGAIN and the review’s positive recommendation for mental and physical health rehabilitation, a single CBT trial formed the basis for the review’s conclusion that there was “moderate certainty evidence” the therapy would “probably” improve symptoms. Beyond those two trials, another one suggested that intermittent aerobic activity was more beneficial than “continuous exercise.” None of the other 21 trials yielded actionable evidence, per the review.
In a blistering take-down on his blog, The Science Bit, Professor Brian Hughes, a psychologist at the University of Galway, pointed out that the relevant psycho-behavioral studies included in the review were all at high risk of bias in various ways. The review itself concedes the point in this acknowledgement buried in the text:
“The evidence addressing CBT and physical and mental health rehabilitation was…at high risk of bias due to lack of blinding and imbalances in the degree of interactions between patients and healthcare providers between arms.”
Here’s a rough translation of that sentence into English:
“If you offer one group of sick patients an intervention that they believe might help them feel better, and you offer another group of sick patients nothing, those who receive the intervention they believe might help them feel better will be much more likely afterwards to report that they feel better than those who receive nothing.”
Responses at “high risk of bias” are obviously an unreliable and inappropriate basis for making confident predictions about the effects of an intervention. So it is perplexing that the review cites such questionable data to assert that anything has been shown by “moderately certain evidence” or is “probably” going to occur.
**********
Null findings on CBT trial’s sole objective measure
Here’s how the authors explain why they wrote the review:
“Some patients and healthcare providers have questioned the credibility of interventions in published trials, such as exercise and cognitive behavioural therapy (CBT). Trustworthy systematic reviews that clarify the benefits and harms of available interventions are critical to promote evidence based care.”
The first sentence of that passage cites three supporting references—one of which is a post I wrote about a dodgy Dutch trial, reported in a 2023 paper; that trial turns out to be the sole basis for the review’s positive conclusions about CBT. But contrary to the sentence’s claim about “patients and healthcare providers” questioning the research, I do not fall into either of those categories. As a public health academic at the University of California, Berkeley, I have some training in basic epidemiology, unlike many—although obviously not all–patients and healthcare providers.
In one sense, this is an inconsequential error. I mean, who cares? However, the error does have the convenient effect of obscuring a key point–many well-qualified academics, and not solely patients and healthcare providers, have highlighted the serious flaws that mar the body of research on CBT, graded exercise, and other psycho-behavioral approaches in both ME/CFS and Long Covid. There’s a reason why the National Institute for Health and Care Excellence (NICE), in an assessment prepared for its 2021 ME/CFS guidelines, found the quality of all the evidence for CBT and GET to be “very low” or merely “low”—certainly not the level required for meaningful clinical guidelines.
In the case of the CBT study highlighted in the review as having “moderate certainty evidence,” the participants—as I noted in the cited blog post–did not move any more at the end of the treatment than before, as measured by body monitors. These null findings suggested that any marginal self-reported improvements were likely an artifact of the bias-inducing study design rather than reflective of genuine changes. The trial investigators chose not to report the results from this outcome in the main paper—an egregious failure of judgment and a likely example of research misconduct.
And yet this is the trial the review cites to justify its recommendation for CBT.
The review trots out the usual rationale for explaining the purported effectiveness of CBT and graded activity:
“CBT and graduated physical activity are offered to patients with long covid and ME/CFS based on the observation that patients often reduce activity in response to their symptoms. Consequently, patients may become physically deconditioned, develop disrupted sleep-wake patterns, and hold unhelpful beliefs about fatigue. Interventions such as CBT and supervised physical activity which gradually reintroduce patients to activity may help with reconditioning, regularising patterns of activity, optimising rest and sleep, and addressing patients’ unhelpful beliefs about fatigue and activity.”
The problem is that, as NICE noted, the evidence in support of this approach—however appealing the interventions sound in theory–is extremely weak. The theory is essentially what Professor Sir Simon Wessely and colleagues proposed back in the late 1980s—and the discredited and fraudulent PACE study was said to be the “definitive” test of that proposition. In fact, the review’s self-delusion is evident in its irony-free reference to PACE as the “only” known trial that found benefits for “all interventions found to be effective” in the review–that is, both mental and physical rehabilitation approaches. After everything that has happened in this field, the blanket declaration that PACE documented benefits is laughable—and unjustifiable from a scientific perspective.
The NICE’s 2021 ME/CFS guidelines themselves serve as a rebuttal to the arguments advanced in the review. However, the review fails to mention this authoritative NICE document–another oversight that should have been called out by peer reviewers or The BMJ’s own editors.
All in all, pure propaganda.
