By David Tuller, DrPH
Norway’s got a double whammy going on. First there’s the group of investigators that seems to have had trouble determining whether their newly published research on CBT and music therapy was an actual randomized trial or merely a feasibility study. (More on that below.) Then we have Dagbladet, a widely read tabloid, promoting a new study of the Lightning Process–with the same senior investigator as the music therapy research. Dagbladet has so far published two stories about the matter (here and here), with perhaps more on the way.
The Dagbladet stories are stuffed with misstatements and omissions about ME, about patients, about the Lightning Process, and about the 2017 Lightning Process trial–too many to address here. The first article mentions me in referencing my criticism of research on this woo-woo intervention. I am referred to as a journalist and a blogger. I certainly don’t object to those terms. But as in previous articles, my academic credentials are not mentioned. Nor is the fact that the investigation of crap research–like the CBT and music therapy study–is in my job description.
The story mentions that the Norwegian ME Association has supported my crowdfunding efforts. The implication is that I am willing to trash research into the Lightning Process because of this financial support–rather than because it is crap. I guess Dagbladet is more concerned with my source of funding than with the 2017 study’s documented ethical and methodological violations. In contrast, the reporter does not appear to be concerned about the key involvement of a prominent Lightning Process practitioner in the proposed Norwegian study. Such a person would clearly have every financial and professional incentive to want to prove that this goulash of neurolinguistic programming, osteopathy and life-coaching has what looks like scientific backing.
As far as I know, the reporter made no effort to contact me. I am a very easy person to find. My academic credentials and my professional title at UC Berkeley are readily available to anyone who knows how to do an online search. Today I plan to send a letter to the editor.
**********
On Monday, I wrote about the protocol and statistical analysis plan for that CBT and music therapy thing. The peer reviews also make for interesting reading and shed some light on the project’s downward slide from full-scale randomized trial to feasibility study. BMJ Paediatrics Open has an open review policy—that is, the peer reviewers are not anonymous. The authors see their names, and so does anyone who wants to review the reviews, which are appended to the article on the journal’s site. There are major advantages to this system in terms of transparency and accountability.
But it can also lead to embarrassment when a reviewer, like one of the two who assessed this manuscript, writes this: “I haven’t read beyond the abstract.” BMJ prides itself on the rigor of its peer review process. It is hard to square that pride with BMJ Paediatrics Open’s apparent decision not to press the peer reviewer to read beyond the abstract and, if that failed, to reassign the manuscript to someone else. (I am assuming this because nothing in the record indicates any such follow-up occurred.)
Given his candor, I don’t fault the reviewer, although perhaps it would have been better to decline the invitation to review in the first place. It was BMJ Paediatrics Open’s responsibility to decide if such limited scrutiny could pass BMJ’s purportedly high standards for quality and integrity. Editors either didn’t bother to read the review carefully, or they determined that a review of just the study’s abstract was sufficient.
Whatever the reason, other observers might view the failure to seek further input as a disturbing lapse in editorial judgement and an abrogation of BMJ’s obligation to readers–and to the medical literature. This failure suggests that disdain for or indifference to proper quality control–so clearly demonstrated in the company’s disastrous handling of the 2017 Lightning Process study–is perhaps systemic and not limited to a single BMJ journal or editor.
In the draft submitted to the journal for review, the title billed the study not as a “feasibility study” but as “an exploratory randomized trial.” (Had they gotten decent results, perhaps they would have dropped the word “exploratory.”) The first reviewer praised multiple aspects of the study but also noted the following:
“I struggle to understand from the aims of the study and the way the study is described whether this was intended as a feasibility study – i.e. to look at feasibility (can this be done?), acceptability (how do participants experience it?) and to give some indication of potential effect sizes to power a future larger scale trial, or whether this was intended as a fully powered trial. Throughout, I think this needs to be clarified for the reader and interpretations/conclusions drawn in light of what the aim was.”
Here is how the authors responded to this point: “Thank you. We agree – this study should be regarded a feasibility study, and the manuscript has been rephrased accordingly.”
Perceptive editors would have noticed that this response was non-responsive. The reviewer did not ask how the study “should be regarded” now that it was already done. The reviewer asked whether the study started out as a fully powered trial or as a feasibility study. She wanted the authors to clarify this point, not to fudge it. The interpretation and conclusions, she noted, needed to be drawn “in light of what the aim was”–not in light of how the authors reframed that aim after-the-fact.
And here is how the published paper describes the aim: “The aim of the present study was to explore the feasibility of this mental training programme in adolescents suffering from CF after acute EBV infection, and to provide preliminary estimates of effects as a basis for a full-scale clinical trial in the future.”
This is demonstrably not the case. As I noted in my earlier post, the protocol and the other documents include no mention of this trial being a feasibility study. Despite the reviewer’s straightforward request, the revised version did not clarify that the study was intended to be a fully powered trial. Instead, the authors rewrote the paper–and the history of the research–as if they’d intended from the start to conduct a feasibility study.
The authors should be expected to account for this mischaracterization of their research. So should the editors who accepted the article for publication while ignoring a reviewer’s alert that he hadn’t reviewed the actual paper. Did anyone at the journal read the supporting trial documentation? My best guess is no.
Comments
13 responses to “Norway’s Double Whammy of Fuzzy Science”
Once again BMJ has proven that their compliance to basic scientific rules is non existant.
Both articles should be stored where they belong: the round archive aka the dustbin..
It is strange that these authors, at least one having a conflict of interest, are allowed publishing this non scientific garbage.
Who are the people behind the scene at BMJ pushing this nonsense?
I don’t think it would be vexatious to ask BMJ whether this kind of peer review is typical of their standards. 1 who didn’t read beyond the abstract yet still found the whole thing puzzling and the other one a like-minded colleague of the authors. The average undergrad paper gets more scrutiny than this.
This is not what people have in mind when they think peer review. It’s equivalent to a security theater, I can’t imagine anyone would be OK with such low level of scrutiny on topics they consider of importance. Or is this a matter of exemptions? Because this is not specific to any particular paper, it’s clearly a pattern so what are the features of this pattern and where are those features written? How does someone get their hands on optional requirements and exemptions from normal standards? Is it a special club? Is there a membership fee to that club? A secret handshake?
Because this sure isn’t normal.
yes, I’m planning to pursue that point specifically. It is hard to claim this has been adequately peer reviewed when one of the reviewers acknowledging that only the abstract was peer-reviewed. This appears to be a systemic BMJ problem with peer review.
Wrt Dagbladet, as opposed to BMJ, does an equivalence of the Science Media Centre operate in Norway? It’s just that I feel I see a pattern here….
Standards certainly seem to be slipping at the BMJ company. From riches to rags, methinks.
With so many ‘journals’ publishing papers, the standards for peer-review and simply for getting published have never been lower.
Savvy practitioners in a field know which journals are junk; administrators and bureaucrats don’t – and it shows.
Lady Shamble:
There is no officially known equivalence to the SMC in Norway.
But the LP coach that shall do the LP study in Norway, worked as a journalist before she became ill, and as such had a large network. She worked for another tabloid paper, VG,
When she became a LP coach she published some op-eds in that paper. A few years ago it seemed that VG changed their opinion. Just before Christmas that year they published a very good article on ME. So maybe she was turned down by her old paper and now shifted to Dagbladet.
Interestingly the doctoral thesis of senior investigator of the study, was the first ever scientific paper that Phil Parker, the inventor of LP, quoted on his website along with a statement from the investigator that was very pro-LP.
There is in this case something strange going on behind the scenes.
Good response, and work!
A trifle, but the two stories you are linking to in section one, is the same link.
“Dagbladet has so far published two stories about the matter (here and here), with perhaps more on the way.”
Here is the other link:
https://translate.googleusercontent.com/translate_c?depth=1&pto=aue&rurl=translate.google.com&sl=no&sp=nmt4&tl=en&u=https://www.dagbladet.no/tema/vil-stoppe-me-forskning/72469419&usg=ALkJrhjFXbfzAB_PbGEWGt0RjrSzDs3duw
Dagbladet did also publish a plus-article (behind paywall) 15th may, about a patient (the patient was of course a PhD.) that recovered form ME with LP.
Biased.
Ref. my last post. “Dagbladet did also publish a plus-article (behind paywall) 15th may, about a patient (the patient was of course a PhD.) that recovered form ME with LP.”
I am sorry. The patient was not treated with LP, but psychotherapy.
Regarding Dagbladets bias in ME/CFS-coverage:
They’ve made a huge case out of the fact that some patients have started a campaign to change how the LP study is conducted (they’re trying to make it more rigorous and less likely to be biased). Dagbladet’s article about this was something along the lines of: “Dagbladet Reveals: ME-activists are trying to stop research”. Sensationalist and a ridicolous angle, but for the uninitiated it’s difficult to tell just how biased this piece is.
Now, the interesting part is that a group of Norwegian BPS-doctors, with Henrik Vogt as the leader, according to multiple sources tried to stop a biomedical study that Fluge and Mella were doing two years ago. Dagbladet have been notified of the BPS-doctors attempt to thwart biomedical research, and been given multiple email screenshots, but they’ve chosen not to write anything about it, or even investigated it in the slightest. I guess it doesn’t fit with the narrative they’re trying to push?
They’re willing to write about patients trying to change a research design into a more rigorous one, but they’re not willing to write about doctors trying to stop a research project of researchers with a competing theory…
Charles: The study that Vogt & co tried to stop was published last week
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7201056/pdf/fmed-07-00162.pdf
I tried in comments on another website that just repeated Dagbladet to point out that a) They ME-patients do not try to stop the study
b) That Vogt tried to stop the Cyclophosphamide study.
@Eirik.
I know. And the only media that’s covered this positive study (that the leader of Recovery Norway tried to bury), is ScienceNorway, which seems to be the only somewhat balanced news source left.
Something to ponder: with starting Recovery Norway Henrik Vogt stated that “he wanted to give patients hope”. That is of course an admirable goal, but I think it’s strange that he doesn’t see how he’s actually taking hope from millions of patients worldwide by trying to thwart research that suggests a biological basis for the illness. It’s almost as if he’s trying to hide something? Trying to prevent information from reaching the public?
And it’s doubly annoying that the media with higher circulation won’t write about this. There are some powerful and influential doctors who stood behind Vogt in this attempt: e.g. Preben Aavitsland from Folkehelseinstituttet, which has been an important person in developing Norway’s approch towards handling Covid-19. He’s a very powerful figure in Norwegian medical circles and it seems as though the media is scared of writing anything that could be interpreted as negative about him.
But I guess they can’t keep this under wraps forever? Will be interesting to see what the larger public will make of this once it they find out.
I wonder whether BMJ advertising/sponsorship income is an element here?
BMJ are explicit that advertisements for products making therapeutic claims without marketing authorisation or CE marking (or local equivalent) need to have had their claims substantiated in full length research papers published in peer reviewed journals
https://authors.bmj.com/policies/advertising-sponsorship/
https://learning.bmj.com/learning/info/advertisers_sponsors.html