By David Tuller, DrPH
I’ve recently spent some time lambasting a Long Covid study in The BMJ that claimed a rehab program addressing both physical and mental health was “clinically effective”—even though the primary outcome results fell below the recommended level for what would be considered “minimal clinically important difference” on the measure in question. Now another high impact journal has published another Long Covid study that also engages in unacceptable methodological shenanigans.
Here’s the title of the new study, which examines the impact of physical activity (PA) in young women with what is referred to as post-covid condition (PCC): “Higher level of physical activity reduces mental and neurological symptoms during and two years after COVID-19 infection in young women.” And here’s the summation of the findings from the abstract: “In conclusion, a higher level of regular PA can reduce…the number of mental and neurological symptoms in PCC underlying the importance of regular PA, even in this and likely other viral disease conditions.”
The authors of the paper are from Semmelweiss University in Budapest; the senior author is also affiliated with the Hungarian University of Sports Science in Budapest and New York Medical College in Valhalla, New York. The paper was published this week by Scientific Reports, a prestigious journal under the Nature publishing umbrella. Not surprisingly, the journal claims to have an “extensive network of expert peer reviewers” and promises that “our editorial team provides rigorous, objective and constructive peer review.”
These claims are bullshit, at least as far as this paper is concerned. This is clear not just from the title but from the conclusion of the study as well. The use of the verb “reduce” in both indicates a causal link. In other words, the title and conclusion are declaring that the greater level of physical activity led to the reduction of symptoms in women. That would be great, if it were a valid statement. Unfortunately for the study authors, it is a statement that cannot legitimately be made based on the study itself—a cross-sectional survey.
Cross-sectional surveys can only find associations. They are not designed to illuminate causal relationships. The reason is obvious. If you’re measuring two things at the same time, it can be impossible to know the order of events. Which caused what? Who knows?
Here’s an easy example. A cross-sectional survey of gay men might find that those with HIV were less likely to use condoms than gay men without HIV. (Let’s postulate that this survey was done years ago, before the advent of pre-exposure prophylaxis for HIV muddied the waters a bit.) A possible assumption—similar to the one made by the authors of the Hungarian study—might be that the failure to use condoms led to the HIV infections.
But to assert such a relationship as a fact would be completely unjustified. I mean, it’s possible, of course, that the chain of events occurred in that order for some of the respondents. But the association between lack of condom use and HIV infection can also be explained the other way around—those who know they have HIV are less likely to use condoms when they have sex because they’re not worried about contracting HIV. In the case of such a survey, the causal relationships are most likely running in both directions; both explanations are viable given the data from the survey. But it is impossible to know which came first.
Similar interpretive confusion about the meaning of associations plagues many of the biopsychosocial studies of the illness or cluster of illnesses referred to as ME/CFS. Studies often show that those who attribute their condition to physical disease are more likely to have worse symptoms. This has been interpreted by clueless investigators—King’s College London’s mathematically challenged Trudie Chalder, a professor of cognitive behavioural therapy, comes to mind as a prime example—as evidence that the belief of physical disease is itself leading to negative outcomes. But the alternate interpretation—that those who are physically sicker are correctly identifying the source of their ill more and are more likely to be doing much worse—is just as reasonable an assumption. The truth is that the data do not allow anyone to distinguish between the two explanations.
The study in Scientific Reports analyzed the responses of 802 women from 18 to 34; of the more than 50% who had experienced an acute bout of Covid-19, the average time since infection was almost two years. The investigators divided the participants into categories of low, moderate and high physical activity based on how many minutes a week they reported engaging in physical activity. At the same time, they assessed dozens of symptoms using the WHO’s form for what the agency calles “post COVID condition.” Around a quarter of the women who had been infected reported ongoing persistent or intermittent symptoms that were “primarily neurological and mental,” according to the study.
In outlining the goal of the reesarch, the investigators wrote the following: “This study aimed to examine the hypothesis that regular PA reduces the number of symptoms during and after COVID-19 infection in young women.” It should have been obvious to any smart person reading the draft that the cross-sectional study design did not allow for any such examination. It is disturbing that this paper passed peer review in this form. The title and the entire thrust of the paper require corrections.