By David Tuller, DrPH
Let me say this directly: It is way past time for Archives of Disease in Childhood to consign the pediatric Lightning Process study to the trash bin.
As I have pointed out repeatedly, the Bristol University investigators recruited more than half their participants before trial registration, swapped outcome measures based on the early results, and then failed to disclose these critical details in the paper. More than a decade ago, all the major medical journals banned the publication of any studies in which recruitment pre-dates trial registration. For that reason alone, Archives of Disease in Childhood, a BMJ journal, should not have published it in the first place.
In addition to retracting the paper, the journal should offer a full and transparent account of what went wrong. How did the paper pass peer review in the first place? Why has it taken so long for Archives to clean up the mess? What changes has Archives implemented in its review processes, if any?
Beyond the publication realm, the appropriate authorities need to review the conduct and reporting of the Lightning Process study as a case of possible research misconduct. The investigators’ choices appear to meet one or more thresholds for what is defined in ethical codes as research misconduct–such as withholding information that would significantly impact interpretation of the results. Given the known facts, an inquiry into possible research misconduct is warranted. It is hard to understand how members of an experienced team of investigators from Bristol University could fail to recognize the implications of what they did.
Whatever. In the meantime, I have been keeping up with my letter-writing. On Saturday, I sent the following letter to Dr Nick Brown, the editor-in-chief of Archives of Disease in Childhood, and cc-d some others:
Dear Dr Brown—
On June 3, 2018–that is, a year ago–you responded to a letter I had sent to people involved in the development of new ME/CFS guidelines under the auspices of the National Institute for Health and Care Excellence. In my letter, I had alerted them to serious methodological and ethical violations in a high-profile 2017 paper in Archives of Disease in Childhood, a BMJ journal. The paper, from investigators at the University of Bristol, was called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.”
Along with people involved in the NICE process, I had cc-d Dr Fiona Godlee, BMJ’s editorial director. You indicated in your e-mail that Dr Godlee had passed on my letter to you. You assured the various recipients that Archives would “respond fully when ready” to the concerns I had raised.
Later that month, Archives posted an “editor’s note” that acknowledged the concerns, explained that the investigators had provided “clarifications,” and indicated that the matter was under “editorial consideration.” The editor’s note was clearly intended as an interim response, not the promised full response. For unexplained reasons, it was not visible from the paper itself but was posted somewhere in what I have referred to as the editorial equivalent of Siberia.
In April of this year, Current Opinion in Pediatrics published a review called “Child and adolescent chronic fatigue syndrome/myalgic encephalomyelitis: where are we now?” This review highlighted the Lightning Process as “effective,” based solely on the Archives paper. Presumably, the review authors did not notice the obscurely located editor’s note.
(I presume the review authors did not notice the editor’s note because anyone reading it would likely have become alarmed about the validity and reliability of the paper’s reported findings. Certainly such readers would likely have recognized that a decision to cite the Archives findings authoritatively could come back to haunt them.)
The new review in Current Opinion in Pediatrics could impact ongoing public health policy decision-making involving a highly vulnerable population–children with a debilitating and stigmatizing illness. It is therefore urgent to finally address the situation in order to safeguard the medical literature and prevent further dissemination of information based on flawed science. While Archives continues to deliberate the fate of the published paper, the now-appointed NICE committee has been holding meetings to debate and develop the new ME/CFS guidelines.
I have noted repeatedly that it takes half an hour, if that, to examine the relevant trial documents and ascertain what happened. Let’s recap. The investigators recruited more than half the participants before registration, in the process violating a strict policy that BMJ professes to observe. They then swapped outcome measures based on data from these early participants, which allowed them to present positive rather than null results for their reported primary outcome. Then they failed to mention the pre-registration recruitment and the mid-trial outcome-swapping in the paper itself; of course, including these details would have likely doomed the chances of publication in a major journal.
Dr Brown, to those who have been waiting patiently on your year-old promise to “respond fully when ready,” can you explain why Archives is not yet ready to respond fully, given the prolonged period of editorial consideration? Will you now commit to a deadline by which Archives can be ready to respond fully to the plain set of facts outlined above?
In seeking to expedite resolution of this public health issue and protect children from exposure to recommendations arising from this unacceptable piece of research, I have cc’d the following: Four physicians involved with the NICE committee, along with two patient representatives; Dr Godlee; Dr Terry Segal, the senior author of the review that, to its misfortune, cited the Archives paper uncritically; Sue Paterson, director of legal services at University of Bristol; two members of Parliament who have expressed concerns about the poor quality of much ME/CFS research, along with a parliamentary aide; and Professor Chris Ponting, vice chair of the UK CFS/ME Research Collaborative, a position he assumed following the tenure of the Lightning Process trial’s lead investigator.
Thank you for your attention to this matter.
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
4 responses to “Time to Retract the LP Study; Letter to Archives of Disease in Childhood”
I agree that “It is hard to understand how members of an experienced team of investigators from Bristol University could fail to recognize the implications of what they did” especially as online information suggests that one of the co-authors had previously been Director of the Bristol Randomised Trials Collaboration (BRTC) and was Professor of Medical Statistics and Clinical Trials at a different Russell Group university at the time of publication of the study.
I don’t think I’ll hold my breath on this either though, but in the eventuality that there is an unexpected turnaround, then didn’t several news outlets report this study positively – Guardian, Daily Mail, Daily Telegraph, BBC News – and the NHS seemed to me to be on the positive side -https://www.nhs.uk/news/medical-practice/lightning-process-could-help-children-chronic-fatigue-syndrome-study-claims/, so shouldn’t they all have questions to answer too? The media doesn’t seem to have a great track record in checking out the facts in matters relating to ME or CFS, does it? Could that have anything to do with the SMC? You’d have thought they’d take more care especially when the health of vulnerable children is at stake.
It is a bit of a worry with those that sit on the NICE guidelines and those that think anything to do with the SMILE trial will help children.
Even more distressing is the fact UCLH is being used to filter children most effected to a stay in their hospital with distressing results that are hidden. It is about time that a decline in ability was noted when children are taken into hospital for treatment when they have ME/CFS and their journey through the hospital stay with their ability logged in detail. That CFS clinics can explain on their websites the full process of PEM and why and how it happens. Importantly explaining how their treatments are going to impact on the child’s ability and PEM and what support they will be giving and why. How else are the parents going to given informed consent?
The same for the research. How can a parent give consent or explain to their child if they have not been given the full facts of PEM?
In 2011 Hampshire Trading Standards requested that the UK Advertising Standards Authority (ASA) give a ruling on the website http://www.lightningprocess.com, arguing that the information on the site was misleading in four areas. The ASA upheld two of the four challenges. They concluded that although there seemed to be some evidence of participant improvement during trials conducted, the trials were not controlled, the evidence was not sufficient enough to draw robust conclusions, and more investigation was necessary; consequently, the website’s claims at the time were deemed misleading.
I’m strongly leaning on the fact that the flaws in the SMILE trial also plagued PACE (and then some) plays a huge part in why doing the right thing is being resisted here. And your role in both makes the whole thing extra awkward for people who know they committed unsustainable malpractice and would rather not be held accountable for it.
This trial is clearly the most unethical of the bunch, yet its flaws are found in the whole body of research supporting the psychosocial model. The weird resistance to upholding this garbage piece of voodoo is inexplicable without the broader context, especially the widest of all: the replicability crisis in psychological research, which threatens the whole MUS effort.
As Zapp Brannigan so aptly said: If we hit that bullseye, the rest of the dominoes will fall like a house of cards. Checkmate.