By David Tuller, DrPH
I often find myself responding to crap studies–such as a Norwegian study called “Brief Outpatient Rehabilitation Program for Post–COVID-19 Condition: A Randomized Clinical Trial,” from Nerli et al., published last December by JAMA Network Open. The senior author was Professor Vegard Wyller, the dean of the Norwegian wing of the CBT/GET/Lightning Process ideological brigades. The trial was as bad as one would expect, given its provenance. (Here’s my post about it.)
It is refreshing when other academics issue their own challenges to the kinds of self-evident methodological lapses that mar this study and so much of this research. Richard Aubrey White, a Harvard-trained biostatistician and a researcher at the Norwegian Institute of Public Health (NIPH), has just published a tough take-down of Nerli et al.—to which the lead author has responded with more of the usual blah blah.
The article appeared in Forskning.no, an online publication focused on news about Norwegian and international research, and includes a note that White is not writing on behalf of his employer. (NIPH has been a hotbed of support for the so-called “biopsychosocial” paradigm, so White’s opinion might be unpopular in some quarters.)
White’s critique—called “Promising treatment for long covid without evidence”–highlights similar issues to those I raised. (I am reliant on Google Translate, so don’t hold me responsible for any mistranslations.)
“A recent study by Tom Farmen Nerli and others suggests treatment ‘based on a cognitive and behavioral approach.’ The study has raised hopes, but a closer examination reveals serious methodological weaknesses.
“Clinically significant” is a crucial term when assessing treatment effectiveness. It means that the treatment produces a real, noticeable improvement in the patient’s health—not just a difference that can be measured statistically. It is not enough to demonstrate a change; it must be large enough to have practical significance for quality of life and function.
“The Nerli study uses a physical function scale (SF-36-PF) in which participants self-assess the extent to which their health limits them in everyday activities such as climbing stairs, carrying goods or vacuuming.
“In the analysis plan, written before the results were known, the authors state that an improvement of at least 10 points on this scale is necessary for a treatment to be considered clinically significant.
“In this study, the treatment group achieved an average improvement of only 9.2 points – below the threshold value. They have indeed measured an effect, but according to the authors’ own criteria it is not large enough to have practical significance.
“Yet the authors present the treatment as clinically significant and effective in the article. This is clearly misleading.”
White also points out that the design of the study lends itself to bias, and that modestly positive results should not therefore be regarded as evidence of the impact of the intervention itself:
“Since participants self-assessed the outcomes (for example, “Is your health such that it limits you from moderate activities such as vacuuming?”), and knew whether they received the intervention or not, it is unclear how much of the intervention’s modest effect is due to the placebo effect and response bias.
“This means that participants may have overestimated their improvement because they expected an effect, or wanted to confirm the researchers’ assumptions. Objective measures, such as the number of steps per day, would likely have shown even weaker results.”
According to White, the implications are clear. The study’s findings indicate “the need for biomedical treatment strategies aimed at, among other things, viral persistence, post-acute inflammation, autoimmunity, thrombosis, reactivation of latent viruses, dysbiosis, intestinal translocation and mitochondrial dysfunction.”
Finally, White writes: “Misrepresentation, weak methods and data secrecy are not just technical details, but serious mistakes with major consequences for those struggling with long covid. Only honest and open science can form the basis for informed choices for patients, healthcare professionals and politicians.”
The response from the corresponding author, Tom Nerli, is unconvincing. He fails to acknowledge the key point behind White’s charge of misrepresentation—that the primary outcome fell below the threshold deemed clinically significant by the investigators themselves. And Nerli doesn’t address the concern about an unblinded trial that relies solely on subjective outcomes. Instead, he cites similar research, as if multiple poorly designed studies can collectively correct the problem.
Finally, one of his suggestions is a real whopper of delusional thinking: “The results from our study are so promising that it should be offered to the patient group that currently has no other documented effective treatment available.”
There’s clearly a huge gap in the two versions of reality being presented in this exchange.