By David Tuller, DrPH
If the results for a trial’s primary outcome do not meet the threshold for what is considered a “clinically significant” benefit, it goes without saying—or at least it should–that investigators have no legitimate grounds for promoting their intervention as “effective.” This is especially true when the trial in question is unblinded and the measure involved is subjective. This combination of elements is a recipe for bias. Given the placebo effect, such trials should be expected to yield modestly positive results solely as an artefact of the study design.
And yet…the die-hard members of the CBT/GET ideological brigades apparently feel free to ignore these basic principles. And so do the high-impact journals that keep publishing their research.
Last month’s example—“Brief Outpatient Rehabilitation Program for Post–COVID-19 Condition: A Randomized Clinical Trial”—was published by JAMA Network Open. The senior author is Norway’s Vegard Bruun Wyller, a professor of pediatrics at the University of Oslo. The study included 314 patients with “mild to moderate” cases meeting the WHO’s broad criteria for Long Covid, or what the organization calls post-COVID-19 condition (PCC).
Half of the participants received a program of two to eight clinical sessions “based on a cognitive and behavioral approach.” The other half didn’t receive the intervention–just “care as usual.”
The intervention included offering participants “cognitive reassurance that bodily symptoms do not necessarily indicate a disease but rather a disorder that is temporary and amendable” and explaining that “certain infections (eg, COVID-19) could trigger maladaptive responses and diverse, unpredictable, and bothersome symptoms (eg, fatigue, dyspnea, and brain fog).” During sessions, “cognitive behavioral therapy–trained physiotherapists supervised the patients by using nondirective communication, socratic dialogue, and guided discovery, prompting patients to infer that recovery would require an active pursuit of physical and mental tasks, thereby fostering positive stimuli expectancies.”
The trial’s primary outcome was the SF-36 Physical Function Subscale (SF-36-PFS)—a commonly used measure in these studies. Higher scores on the 100-point scale represent better self-reported health. As the paper and the study protocol both noted, a 10-point change on the SF-36-PFS is considered “clinically significant.” Changes less than 10 points are, by definition, considered clinically insignificant—that is, too small to be meaningful or even noticeable to the individual.
In this trial, the difference between the changes in the intervention and non-intervention groups on the SF-36-PFS at the end is 9.2 points–below the 10-point threshold pre-designated as “clinically significant.” But the abstract reports instead that the SF-36-PFS scores “improved statistically and clinically significantly in the intervention group.” The same phrase is repeated in a highlights box headlined “Key Points.”
Note the clever wording—the phrase apparently refers to the change within the intervention group from baseline to post-intervention and suggests that it exceeded the 10-point threshold for clinical significance. But change within the intervention group is not the metric of interest in a clinical trial. You don’t need to conduct a clinical trial in order to measure change in a group receiving an intervention. You conduct a clinical trial in order to compare the change in the intervention group with the change in some other group.
In a clinical trial, therefore, the metric that matters—as Professor Wyller and editors of JAMA Network Open surely know–is the difference between the changes in the intervention and non-intervention groups. And in this case, the difference was not clinically significant–even in a trial with bias built into the design.
Here’s the claim from the abstract’s conclusion: “In this randomized clinical trial, a brief outpatient rehabilitation program with a cognitive and behavioral approach in patients with PCC was effective and safe.” Given that the primary outcome did not reach the level of clinical significance, it is unwarranted and deceptive to claim the intervention overall was “effective.”
Beyond the abstract, the fact that the difference between the groups fell below the clinically significant threshold is also not mentioned in either the discussion or limitations sections. The only mention of the 10-point threshold itself is in the section on statistical calculations. (The abstract mentions that the difference between the groups is 9.2 points, but the number on its own, and without the needed context, is meaningless to readers.)
In short, the investigators are seriously misrepresenting their findings. And JAMA Network Open is allowing them to do so.
That misrepresentation is far from the only problem with this trial. Among other issues, almost a quarter of the participants—83 of 314, or 74%–did not provide final data. Since they were presumably motivated to get better, the high drop-out rate raises questions about why so many decided not to follow through.
This trial, however unimpressive the findings, has received lots of attention.
Along with the study, JAMA Network Open published an “invited commentary” from Trudie Chalder, King’s College London’s factually and mathematically challenged professor of cognitive behavior therapy. It is full of her usual blah blah, like this bit of PACE-style propaganda: “Over the past few decades, substantial evidence has amassed for the efficacy of CBT for symptoms in the context of somatic conditions, including chronic fatigue syndrome.”
And as often happens with research from these investigators, the news coverage has been glowing and gullible. NRK—the Norwegian Broadcasting Corporation—published a credulous article headlined “New study gives hope.” The headline on a MedPage Today article was “Long COVID Symptoms Improve With Outpatient Intervention.” Of course, neither mentioned that the results on the primary outcome were not clinically significant. I assume this trial’s inflated claims will be included in the next iteration of the recent “living” systematic review of Long Covid interventions, which is itself in need of a major correction.