By David Tuller, DrPH
On November 12, I received my latest letter from BMJ’s so-called research integrity office about the pile of potential research misconduct otherwise known as the pediatric study of cognitive behavior therapy and music therapy as a treatment for chronic fatigue after acute EBV. This study was published in April by BMJ Paediatrics Open and immediately came under sharp and justified criticism–including from me.
Earlier this week, I posted the letter I received from the research integrity office. I am posting it again below, along with my response. Of course, my beef isn’t with the research integrity coordinator. She’s just doing her job, presumably at the direction of BMJ higher-ups. The fault in this disaster lies squarely with those at the top, who seem to prefer obfuscation to transparency when addressing methodological and ethical lapses in papers published under the BMJ umbrella. At least, that has been my experience.
**********
BMJ Paediatrics Open‘s most recent letter to me
Dear Dr Tuller
Thank you for your email.
As you will know, there is controversy among publishers around when to publish expressions of concern. The retract and republish approach is recommended by COPE’s Retraction Guidelines and in this case, we felt that it would be better to undertake these simultaneously. We did not feel there was an urgent patient safety issue which required immediate retraction of the paper.
The Editor in Chief, Imti Choonara, would welcome a brief e-response to the article making your point about the outcome of post-exertional malaise and we will ask the authors to respond. We have satisfied ourselves that this does not represent research misconduct.
You will see from our previous response to you that this new submission has undergone full editorial and peer review. We are not going to post the peer reviews on this paper. The peer review process relies on trust and readers may be reassured that we have learned from our previous errors and ensured our processes are more robust.
We now consider this matter closed and we will not engage in any further correspondence regarding the editorial process. Should you wish to respond in the journal regarding the post-exertional malaise outcome, however, you would be welcome.
Kind regards,
Simone Ragavooloo
BMJ Research Integrity Team
On behalf of Fiona Godlee, Editorial Director
**********
My response to BMJ
Dear Ms Ragavooloo:
BMJ Paediatics Open’s actions regarding this study of cognitive behavior therapy and music therapy for chronic fatigue after acute EBV continue to be troubling enough to warrant further comment.
As you note, BMJ Paediatrics Open has pursued a “retract and replace” approach. The replacement paper includes many of the faults of the retracted one, but I will focus on two issues–the presence of the outcome measure of post-exertional malaise (PEM) in the retracted and replacement papers, and the retraction notice’s account of why the investigators misrepresented their full-scale trial by describing it in the retracted paper as a feasibility study.
Among the many concerns initially raised about the retracted paper was that PEM was not included as an outcome measure in the trial registration and protocol. That means there was no legitimate rationale for including PEM as an outcome measure in the retracted paper, just as there is no legitimate rationale for including it in the replacement paper. Including outcome measures that are not mentioned in a trial’s registration and protocol is a violation of core research principles and is barred by BMJ’s own rigorous standards.
The proper remedy would be to correct the replacement paper by removing the errant outcome measure, perhaps also explaining why BMJ Paediatric Open‘s editorial process has now twice failed to catch this specific lapse. In your most recent letter, you suggest instead that the proper remedy is for BMJ Paediatrics Open to publish an exchange of letters between the investigators and me rather than to fix the problem. This proposed remedy is insufficient in such circumstances and fails to adequately safeguard the medical literature. The suggestion also indicates that BMJ Paediatrics Open is reluctant to adhere to BMJ’s well-articulated policies on addressing selective outcome reporting. I therefore decline the invitation to submit a letter for publication to BMJ Paediatrics Open.
Regarding the retraction notice, it only addressed one issue—that the investigators presented the research as a feasibility study when in fact it was designed as a full-scale clinical trial. In the notice, BMJ Paediatrics Open absolved the investigators of responsibility for this misrepresentation. (For unclear reasons, the retraction notice ignored the PEM outcome measure and other concerns raised about the retracted paper, and the replacement paper similarly leaves multiple issues unaddressed.)
The retraction notice stated the following: “We identified a mistake in the editorial process which led to this misrepresentation of the research that was undertaken. Unfortunately, editorial errors in judgement happen.We wish to transparently acknowledge our error.” No further explanation or details were provided about this “mistake.” It is disingenuous and just plain silly for BMJ Paediatrics Open to declare that it is “transparently” acknowledging an error while not being transparent about what the error was and how it happened.
Moreover, a review of the relevant available documentation undermines the retraction notice’s characterization. If BMJ Paediatrics Open has information documenting that an editorial “mistake” can explain why the investigators themselves chose to misrepresent their research, the journal should certainly make it public.
Under BMJ Paediatrics Open’s guidelines, peer reviews are posted alongside papers themselves. The peer reviews posted with the retracted paper were not ambiguous. One of the reviewers acknowledged not having read “beyond the abstract.” (BMJ Paediatrics Open chose not to mention this breakdown of the peer review process in the retraction notice, indicating that its declared commitment to transparency has limits.) The other reviewer expressed confusion about the draft and asked a straightforward question: Was this research designed as a feasibility study or as a fully powered trial?
Two answers to this question were available to the investigators. One of these answers was right, and the other was wrong. One answer was true, and the other was false. As we know, the investigators designed this study as a full-scale trial. Yet they responded to the reviewer’s request for clarification by wrongly and falsely portraying their research as a feasibility study.
It is thus puzzling that the retraction notice attributed the misrepresentation to a “mistake in the editorial process.” What was the editorial “mistake” that caused the investigators to pretend they conducted a feasibility study when they did not? During the draft revision phase, did a BMJ editor advise or force the investigators to be dishonest in describing their research? If that is what happened, readers should certainly be informed. If not, perhaps the investigators should be asked to account for their untruthfulness in downgrading their full-scale trial into a mere feasibility study.
In sum, BMJ Paediatrics Open’s opaque retraction notice has failed to resolve many of the issues of concern. The journal has also disregarded its own guidelines by not posting the peer reviews for the replacement paper. In response to my question on this point, your letter confirms this departure from usual practice but offers no justification. That does not inspire confidence in the editorial integrity of BMJ Paediatrics Open‘s most recent scrutiny of this research, especially given the previous mishaps.
Your letter notes in conclusion that “readers may be reassured that we have learned from our previous errors and ensured our processes are more robust.” Unfortunately, it is hard to take seriously the argument that readers should be “reassured” on any score. The record of this debacle does not provide evidence that BMJ and BMJ Paediatrics Open have learned a thing from their mistakes or that editorial processes are now “more robust.” On the contrary.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
Comments
4 responses to “My Letter to BMJ Paediatrics Open about the CBT-Music Therapy Study”
“The peer review process relies on trust ”
Whenever anyone says “Trust Me”, I know I’m about to be stabbed in the back.
I don’t think BMJ understands how the internet works. They don’t seem to understand that this rubbish research and their mishandling of it will not go away. CIA Agent Stan (from animated series “American Dad”) is not going into the bowels of the CIA to fix BMJ’s Permanent Record.
Eventually the lawyers will get involved. They will be interested in BMJ’s role in harming patients. They won’t have to look very hard to find the evidence.
“You will see from our previous response to you that this new submission has undergone full editorial and peer review. We are not going to post the peer reviews on this paper. ”
– Here you can see X.
– May I see X?
– No you may not.
Well, OK then. That clarifies it as much as pocket sand to the eyes. I mean we literally cannot see that the submission has undergone full editorial and peer review, because reasons. It’s very fitting for such an insulting response to contradict itself in the next sentence, a sort of performance art nod to research of the same level of integrity. I’m sorry I mean “integrity”, another word whose definition has lost all meaning in “evidence”-based medicine.
Thanks for pursuing this. You deserve a medal.
I think the revised version of the paper is still outrageously bad.
A) The small sample size makes it hard to draw conclusions from the data – we could just be seeing random noise due to small sample size. Referees certainly can reject a paper and tell the authors to come back when they’ve done a better experiment, and I think that should have happened here.
B) Although the small sample size makes it hard to tell if the effect is real or random noise, there are signs that the treatment made patients worse, not better. That’s a serious red flag for the treatment under evaluation that should get some attention in the write-up, and not be swept under the carpet. In particular, the recommendation for follow up experiments ought to say that this possible treatment harm ought to be further investigated. For example, deliberately not doing activigraphy in the next experiment because the authors now suspect that it will reveal evidence of a treatment harm that they are trying to hide is seriously not OK.
C) The write up uncritically accepts that CBT is an effective treatment for CFS, despite their own data hinting that it isn’t working and the research literature giving indications of a controversy in this area. (See, especially, the new draft NICE guidelines on CFS. Granted, the NICE guidelines might have come out to late to be cited here, but there were earlier indications of problems). If your data hints that the treatment might be making patients worse, your review of prior work ought to be more critical of what exactly was demonstrated in prior experiments, in order to reconcile previous results with your own findings.