By David Tuller, DrPH
Update (7/6/18): I sent a follow-up e-mail to Dr Godlee yesterday to correct an inaccuracy in what I wrote about the 2011 report on PACE in The BMJ. I have included that follow-up e-mail at the end of this post.
**********
Earlier today, I e-mailed the following letter to Fiona Godlee, the editor-in-chief at The BMJ and editorial director of BMJ. I cc’d the Health Research Authority, the CFS/ME Research Collaborative, NICE, Carol Monaghan, etc.
The subject line: BMJ’s failure to address problems with pediatric studies of ME/CFS
**********
Dear Dr. Godlee—
I’m sorry if this sounds like an impertinent question, but I have to ask: Does BMJ care about the health of children? Frankly, the only conclusion I can draw from my recent efforts to address the methodological and ethical violations in two BMJ papers involving children with ME/CFS is that concerns about reputational damage have overridden other considerations, at least in this domain of inquiry. This is an unattractive conclusion, but as far as I can determine it is the only one that fits the facts.
You are likely familiar with both matters. In recent weeks, I have cc’d you on multiple e-mails to various agencies, organizations and officials regarding the articles—the 2011 school absence study in BMJ Open and last year’s Lightning Process study in Archives of Disease in Childhood. As you know, the problems have been authoritatively documented on Virology Blog. Yet editors at the two journals have demonstrated that they are either unwilling to or incapable of addressing the concerns. This abrogation of editorial responsibility is alarming, given that both papers have the potential to impact health policy regarding ME/CFS in children. As just one example, NICE recently included the Lightning Process in a list of non-pharmacological treatments to be investigated for the new ME/CFS guidance—a decision obviously based on the Archives study.
Let’s briefly review the papers and the journals’ inadequate responses when questioned about their egregious flaws.
The investigators of the school absence study exempted it from having to undergo ethical review on the grounds that it qualified as “service evaluation.” Editors should have recognized this claim to be false, given that the study included a hypothesis, generalizable conclusions, and an analysis of primary data drawn from interviews with known individuals—all characteristics of research that requires ethical review. Yet BMJ Open published the paper as service evaluation even after a peer-reviewer expressed strong reservations about the lack of ethical review.
Since the questions arose last year, BMJ Open has compounded its initial mistake by doubling and tripling and quadrupling down. Editors have repeatedly maintained that they were correct to treat the study as service evaluation, despite the fact that the journal itself published it under the heading of “research.” Remarkably, in its presentation of the issue to the Committee on Publication Ethics, BMJ Open argued that one reason the study qualified as service evaluation was that it only involved anonymized data. Since this statement was clearly at odds with the methodology described in the paper, it created the impression BMJ Open editors had not even read the study they were aggressively defending.
The COPE statement also maintained that my reporting about the study had been “inaccurate, misrepresentative and damaging to the publisher’s reputation.” I don’t doubt that what I wrote could have been damaging to the publisher’s reputation, as it should have been. But I have been eagerly awaiting BMJ Open’s evidence that my reporting was “inaccurate” or “misrepresentative.” (Like everyone, I can make mistakes, and when I do I seek to correct them—unlike BMJ, apparently.)
No such evidence has been provided, which I take as BMJ Open’s acknowledgement that the statement about my work was untrue, so I hope to receive an apology for this misstatement in due course. For now, I’ll settle for an answer to this question: Do you believe it is acceptable for your editors to make unsupported accusations about those expressing legitimate criticisms while disseminating demonstrably false statements about studies they have published?
In response to BMJ Open’s opaque account of the events, COPE advised editors to review the methodology with the investigators to determine if the study did or did not qualify as service evaluation. Following this advice would have inevitably exposed the fallacy of BMJ Open’s position, so presumably it was not what editors wanted to hear. Instead of complying, they closed the COPE case. I suppose they were trying to pretend that the issue had been resolved.
The journal did encourage Professor Vincent Racaniello and I to submit a letter for publication, offering the following bait: A promise that editors would then seek a point-by-point rebuttal from the investigators. (Professor Racaniello is a microbiologist at Columbia and the host of Virology Blog.) From our perspective, BMJ Open already had an obligation to seek a point-by-point rebuttal or some sort of explanation from the investigators, so we failed to understand why such a letter from us was required to trigger this necessary editorial action.
In any event, we offered to submit a letter for publication if the journal agreed to seek input from the appropriate agency—the Health Research Authority—about whether the study qualified as service evaluation or not. We did not hear back from BMJ Open. That appeared to confirm our assumption that the journal was not sincerely seeking clarity on the question but simply wanted to publish an exchange of opinions between us and the investigators so it could be viewed as acting as an honest broker of scientific “debate.”
Let’s be clear: There can be no such debate. The school-absence study is research and required ethical review. That is a fact, not an opinion or a question with multiple possible answers. Moreover, given its past actions, BMJ Open cannot credibly present itself as an honest broker in this matter. BMJ now needs to publicly answer two important questions: How could BMJ Open have published the school absence study as service evaluation in the first place? And why has the journal continued to claim that the decision to publish it as service evaluation was correct and repeatedly dismissed the concerns of those who accurately argued otherwise?
Now let’s take the Archives of Disease in Childhood paper on the Lightning Process. We should be able to agree on a brief set of facts about this study: 1) Fifty-six out of the 100 participants were recruited before trial registration. 2) This violated BMJ’s strict policy against publishing papers in which participants were recruited before trial registration. 3) The investigators swapped primary and secondary outcomes based on the data provided by these 56 participants recruited before trial registration, a recipe for producing biased results. 4) No one reading the paper without the supporting documentation would have any awareness of points 1, 2, and 3.
None of these methodological anomalies are acceptable, of course. The editor of Archives of Disease in Childhood was alerted to the issues five months ago, in an open letter signed by 21 experts from Stanford, Harvard, Berkeley, Columbia, UCL, QMUL, etc. He promised to investigate. More than a month ago, when you forwarded him the e-mail I had sent about the study to those involved in developing the new NICE guidance for ME/CFS, the editor responded that the journal would respond when “fully ready.” Since the facts are easy to ascertain, this lengthy delay suggests that Archives does not have a genuine interest in assessing them.
Let’s remember something. The Lightning Process was created by a guy who previously trained people to develop their healing powers through the use of Tarot, spiritual guides, and other rather unusual modalities. It is a combination of neuro-linguistic programming, osteopathy, and positive affirmations. Practitioners claim to be able to cure not only this illness but MS and a host of other serious diseases. They have been cited multiple times for false advertising.
Archives of Disease in Childhood has now provided all Lightning Process practitioners with bragging rights that their pseudo-scientific intervention is effective—based on a study that explicitly violated BMJ policy regarding trial registration and guaranteed its own positive findings by outcome-swapping. So why are ME/CFS patients constantly accused of being anti-science and irrational and “vexatious” for not accepting findings produced by this kind of data-torturing? When it comes to this domain of research, it is clear to non-biased observers that it is the researchers themselves who are best compared to climate-change deniers, not the patients valiantly challenging their terrible science.
Really, Dr Godlee, enough is enough. It is time to stop stonewalling, acknowledge and address the issues with these papers, and make a clean breast of it—that is, if children’s health does in fact matter to BMJ. (And I haven’t even mentioned the fact that The BMJ itself has never corrected a false statement in its news section about the 2011 Lancet paper—that 30 % of those in the study were cured. Since the Lancet paper itself explicitly did not report on “recovery” and that claim was taken from the specious commentary written by Dutch colleagues of the authors, this statement needs to be fixed—even at this late date.)
In short, I’m afraid your journals have lost their way in how they approach this particular illness. The current treatment paradigm based on PACE/CBT/GET is collapsing before our eyes. As that process continues, those who have enabled such scientific misadventures, including BMJ, should expect to suffer some serious collateral damage. The longer you wait to acknowledge these issues and take corrective action, the worse the inevitable reputational fallout is likely to be.
In the interests of transparency, I have cc’d this letter to others with whom I have already communicated about the problem with these BMJ studies. That includes those involved in developing the new NICE guidance, members of the executive board of the CFS/ME Research Collaborative, officials at the Health Research Authority, and Carol Monaghan MP.
Thanks for your quick attention to this matter. I would be more than happy to have a phone conversation or an in-person meeting to go over the documentation and discuss these very serious issues.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
**********
Follow-up letter:
Dear Dr. Godlee–
In the letter I sent the other day, I inaccurately cited the reporting of the 2011 PACE results in The BMJ. The sentence in my letter referring to this trial report should have indicated not that The BMJ reported that 30% of study participants were “cured” but that 28-30% of those assigned to the two active interventions of interested were “cured.”
I apologize for the error. For full transparency, I will post this e-mail on Virology Blog as an update. I am sure, however, that you understood the intended meaning. Either way, the sentence in The BMJ is inaccurate and must be corrected. As the PACE authors explicitly stated in the correspondence several months later, the 2011 paper did not address “recovery.”
The BMJ should have indicated that the “cure” claim came from the commentary accompanying the paper, which was written by Dutch colleagues of the PACE authors. Remarkably, The Lancet has defended this specious statement by claiming it only reflects the opinions of the commentators and not the actual findings of the study–even though the commentary itself clearly presented the claim as a fact.
I look forward to your response to this and all the other concerns raised in my previous letter about the Lightning Process and school absence studies.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley