By David Tuller, DrPH
I have been trying to convince editors at two BMJ journals to take responsibility for poor decisions. Despite serious nudging and prodding, I have been unsuccessful. The two papers I have criticized as being fraught with methodological and/or ethical missteps are these: “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial,” published last year in Archives of Disease in Childhood; and “Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics,” published in 2011 in BMJ Open.
These papers both involved research with children. They both have the potential to impact public policy and medical treatment. So getting the issues aired publicly and then adequately addressed is of some importance—at least to me. BMJ does not appear to share that sense of urgency.
For reasons that remain unclear, editors at the two journals have chosen so far to ignore the obvious facts. (At least publicly; I have no idea what they think privately.) So has Fiona Godlee, editor in chief of The BMJ and editorial director at BMJ Company. According to BMJ’s website, Dr Godlee is “responsible for ensuring the editorial integrity of all or [sic] our products and services.” If so, she should immediately address the issues with these two papers. Silence is not a viable long-term option, especially when the evidence in both cases is irrefutable.
Two weeks ago, I warned some of those involved in the NICE guidance process about the Lightning Process study, since the agency’s draft scoping report had mentioned this training modality as a possible non-pharmacological treatment for ME/CFS. That initial e-mail led to a follow-up exchange yesterday with Nick Brown, the editor of Archives of Disease in Childhood. Dr Brown indicated that the journal was still investigating the Lightning Process study and would “respond fully” when ready. He did not explain why the investigation had taken more than four months and how much longer it would last.
This morning, I sent an e-mail about BMJ to Scottish National Party MP Carol Monaghan. MP Monaghan, a science teacher by profession, has a keen interest in research accountability as well as in the plight of ME/CFS patients. In February, she organized a hearing in the House of Commons on the flaws of the PACE trial. During that event, she referred to PACE as “one of the biggest medical scandals of the 21st century”—an undoubtedly accurate assessment even though the century still has a long way to go. In March, I posted a Q-and-A with her on Virology Blog.
In today’s e-mail to MP Monaghan, posted below, I alerted her to BMJ’s apparently casual approach toward violations of scientific principles in high-impact studies on children. As with the NICE e-mail, I cc’d Fiona Godlee.
**********
Dear MP Monaghan—
I know from your public actions and our recent exchanges that you have great concern about the validity and reliability of the published evidence on ME/CFS, including but not limited to the PACE trial. Given your interest in research integrity, I wanted to bring your attention to two studies published by journals in the BMJ stable–Archives of Disease in Childhood and BMJ Open.
Both studies involve children with the illness, and both violate core scientific and/or ethical principles. Yet because they are in the literature, they could impact public health policies and the medical treatments available to kids, if they haven’t already. Although the editors of the two journals have been informed of the issues, they have so far failed to take the appropriate and necessary corrective steps. This failure is arguably placing children at risk of harm and is, to me, inexplicable and inexcusable.
The Archives of Disease in Childhood paper, called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial,” was published last year. The BMJ Open paper, called “Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics,” was published in 2011.
In the Archives of Disease in Childhood study of the Lightning Process, more than half the participants were recruited as part of a feasibility study starting almost two years before trial registration. Even as these feasibility study participants were folded into the full trial, primary and secondary outcome measures were swapped based on their results–a recipe for biasing the findings. The paper in Archives of Disease in Childhood reported positive results but did not disclose these relevant details—a disturbing omission. BMJ has a longstanding policy against publishing trials in which participants were recruited before registration—a policy clearly breached in the Lightning Process study.
In the BMJ Open study, the investigators sought to test the hypothesis that children with chronic fatigue syndrome could be identified through school absence records. But they exempted the study from ethical review on the false grounds that it was “service evaluation” and not “research.” The published paper included a hypothesis and generalizable conclusions, and relied on primary data collected directly from known participants. According to the U.K. guidelines, a study with these features cannot be classified as “service evaluation” but qualifies as “research” and requires ethical review.
A peer reviewer raised tough questions about the study’s claim that it was “service evaluation” rather than “research.” Although the investigators failed to provide an adequate response for the lack of ethical review, BMJ Open overlooked the issue and accepted the paper anyway. Ironically, the journal published it under the heading of “research” but has since defended the decision to categorize it as “service evaluation.”
You can read more details about the documented problems with these studies, and letters of concern sent to the editors of the two journals, in the following Virology Blog posts:
Trial By Error: The SMILE Trial’s Undisclosed Outcome-Swapping
Trial By Error: A Letter to Archives of Disease in Childhood
Trial By Error: No Ethical Review of Crawley School Absence Study
I have cc’d Fiona Godlee, editor-in-chief of The BMJ and editorial director of BMJ Company, on this e-mail. Perhaps she can explain why BMJ journals have not yet acknowledged the obvious: The Archives study violated BMJ policy on prospective trial registration and the BMJ Open study was not “service evaluation,” so neither should have been accepted for publication in the first place.
Does Dr Godlee believe these incidents suggest a need for greater scrutiny of BMJ’s editorial and peer-review processes, at least when it comes to this domain of inquiry? Does she have suggestions for what recourse should be available when journals fail in their responsibility to monitor themselves, with potentially negative public health and medical consequences?
I am raising these issues with you now because this BMJ research is at risk of being given credence in deliberations on public policy involving children, such as the current NICE effort to develop a new guidance for ME/CFS. Like PACE, these flawed studies have no place being cited in support of any public health initiatives or recommendations on medical care for people with this illness. But because of apparent editorial deficiencies at the journals in question, these papers somehow passed peer review, were accepted for publication, and remain in the literature.
Thank you for your continuing interest in this matter.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley