By David Tuller, DrPH
Earlier today, Professor Racaniello e-mailed the following letter to Nick Brown, the editor-in-chief of Archives of Disease in Childhood, one of the journals from the BMJ Group. Archives recently published a study of the Lightning Process in kids with ME/CFS.
**********
Dear Dr. Brown:
In September, Archives of Disease in Childhood published a study called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial” [1]. Like many studies of this illness, it was an open-label trial with self-reported outcomes—a study design known to be highly vulnerable to bias.
The researchers have received criticism for their initial decision to conduct a pediatric trial of the Lightning Process–a commercial, pseudo-scientific intervention that combines elements of life-coaching, osteopathy, and neurolinguistics programming. The intervention encourages participants to report that it has made them better, another source of potential bias. Beyond the decision to study the Lightning Process, the study suffered from major cross-contamination between treatment arms and significant loss-to-follow-up, among other problems.
In addition to these issues, we are also concerned about a methodological anomaly reported on Virology Blog by David Tuller from the University of California, Berkeley: An earlier feasibility trial was extended into the full trial even as the researchers swapped primary and secondary outcomes following a review of the feasibility trial results [2]. The researchers received ethical approval for these study design changes but failed to mention them in the Archives paper, although the changes enabled the findings to be presented in a much more positive light than would otherwise have been the case.
These changes are only apparent from a review of other trial-related documents, such as the registration [3], the feasibility trial protocol [4] and the full-trial protocol [5]. Our specific concerns include the following:
*The researchers registered the study as a prospective trial. But more than half the sample were apparently participants recruited for the feasibility trial [6], starting almost two years before the registration date. The Archives paper did not mention this fact or explain the discrepancy in dates.
*In the feasibility trial protocol, school attendance at six months was the primary outcome, and self-reported physical function at six months was a secondary outcome. After reviewing results from the feasibility trial, the researchers swapped these outcome measures for the full trial—school attendance at six months became a secondary outcome, and self-reported physical function at six months became the primary outcome. The Archives paper did not disclose the outcome-swapping that occurred after more than half the sample had provided data as participants in the feasibility trial.
*As its main finding, the Archives paper highlighted the reported benefits in the Lightning Process group for self-reported physical function at six months, the new primary outcome. Self-reported school attendance at six months, now a secondary outcome, yielded null results and received scant mention in the Archives paper. Thus the outcome-swapping allowed the researchers to report positive results for their primary outcome. Had they not switched the outcomes after more than half the sample had provided data, they would have had to report null primary outcome results
*In both the feasibility trial protocol and the full-trial protocol, the researchers promised to seek official school attendance records as a complement to self-reported school attendance. The Archives paper did not provide official school attendance records, even though it highlighted benefits at 12 months in self-reported school attendance. The researchers offered no explanation for the absence of these official school attendance records. Interestingly, it was stated in a recent news account that the official school attendance records corroborated the self-reported school attendance results, suggesting that these data were in fact collected.
*Not surprisingly, the media coverage of the Archives paper focused on the reported benefits in the Lightning Process arm for the revised primary outcome of self-reported physical function at six months. The null results for school attendance at six months, the original primary outcome, were largely ignored. The outcome-swapping that occurred after more than half the sample had provided data—and that the Archives paper did not mention—ensured that the reported findings received more positive media coverage than they otherwise would have.
Transparency is essential in scientific research. In this case, the Archives paper failed to disclose salient and relevant information about the composition of the study sample, the outcome-swapping, and the official school attendance records. These missing details, if provided, would help others to more properly assess the validity and reliability of the reported findings.
The Archives paper must be corrected to acknowledge that the sample included the feasibility trial participants and that the review of their data informed the decision to switch the outcome measures. The paper should also present separate results for participants in the feasibility trial and those recruited afterwards. And the corrected version must clarify the status of the official school attendance records; if these data were collected, as the recent news account suggested, they should also be reported.
We urge Archives of Disease in Childhood to take these concerns seriously and address them promptly.
Sincerely—
Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University
New York, NY, USA
Lisa F. Barcellos, MPH, PhD
Professor of Epidemiology
School of Public Health
California Institute for Quantitative Biosciences
University of California, Berkeley
Berkeley, CA, USA
Robin Callender Smith, PhD
Professor of Media Law
Centre for Commercial Law Studies
Queen Mary University of London
Barrister and Information Rights Judge
London, UK
Lily Chu, MD, MSHS
Independent Consultant
Burlingame, CA, USA
Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University
Stanford, CA, USA
Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine
University College London
London, UK
Valerie Eliot Smith
Barrister and Visiting Scholar
Centre for Commercial Law Studies
Queen Mary University of London
London, UK
Rebecca Goldin, PhD
Professor of Mathematics
George Mason University
Fairfax, VA, USA
Leonard A. Jason, PhD
Professor of Psychology
DePaul University
Chicago, IL, USA
Michael W. Kahn, MD
Assistant Professor of Psychiatry
Harvard Medical School
Boston, MA, USA
Nancy Klimas, MD
Director, Institute for Neuro-Immune Medicine
Nova Southeastern University
Director, Miami VA Medical Center GWI and CFS/ME Program
Miami, FL, USA
Bruce Levin, PhD
Professor of Biostatistics
Columbia University
New York, NY, USA
Steven Lubet
Williams Memorial Professor of Law
Northwestern University Pritzker School of Law
Chicago, IL, USA
Marlon Maus, MD, DrPH, FACS
Director, DrPH Program
School of Public Health
University of California, Berkeley
Berkeley, CA, USA
Patrick E. McKnight, PhD
Professor of Psychology
George Mason University
Fairfax, VA, USA
Zaher Nahle, PhD, MPA
Vice President for Research and Scientific Programs
Solve ME/CFS Initiative
Los Angeles, CA, USA
Philip B. Stark, PhD
Professor of Statistics
University of California, Berkeley
Berkeley, CA, USA
John Swartzberg, MD
Clinical Professor Emeritus
School of Public Health
University of California, Berkeley
Berkeley, CA, USA
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
University of California, Berkeley
Berkeley, CA, USA
William Weir, FRCP
Infectious Disease Consultant
London, UK
Carolyn Wilshire, PhD
Senior Lecturer
School of Psychology
Victoria University of Wellington
Wellington, New Zealand
**********
[1] Crawley EM, Gaunt DM, Garfield K, et al. 2018. Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial. Archives of Disease in Childhood; 103(2):155-164
[2] Tuller D. 2017. Trial by error: the SMILE trial’s undisclosed outcome-swapping. Virology Blog; Dec 13. Accessed at: https://www.virology.ws/2017/12/13/trial-by-error-the-smile-trials-undisclosed-outcomes/
[3] ISRCTN Registry: ISRCTN8145620. July 31, 2012. Specialist Medical Intervention & Lightning Evaluation: Comparing specialist medical care with specialist medical care plus the Lightning Process for Chronic Fatigue Syndrome or Myalgic Encephalopathy (CFS/ME). Accessed at: http://www.isrctn.com/ISRCTN81456207
[4] Crawley EM, et al. July 2010. SMILE Protocol v6: Assessing the feasibility and acceptability of comparing the Lightning Process with specialist medical care for Chronic Fatigue Syndrome or Myalgic Encephalopathy (CFS/ME)–pilot Randomised Controlled Trial. Accessed at: http://www.bristol.ac.uk/media-library/sites/ccah/migrated/documents/smprotv6final.pdf
[5] Crawley EM, Mills N, Hollingworth W, et al. 2013. Comparing specialist medical care with specialist medical care plus the Lightning Process® for chronic fatigue syndrome or myalgic encephalomyelitis (CFS/ME): study protocol for a randomised controlled trial (SMILE Trial). Trials; 14:444
[6] Crawley EM, Mills N, Beast L, et al. 2013 The feasibility and acceptability of conducting a trial of specialist medical care and the Lightning Process in children with chronic fatigue syndrome:: feasibility randomized controlled trial (SMILE study). Trials; 14:415.
Comments
28 responses to “A Letter to Archives of Disease in Childhood”
Wow amazing. Love how many people signed as well. Right on!!! Thank you
!!
P.S. you have my signature as well.
Fantastic work. The potentially harmful psychobabble parading as science has to be exposed. And journals held responsible for what they publish. Thank you to all the signatories.
Nice to see so many high profile signatories. Having said that I’ll be amazed if a BMJ publication does anything other than ignore it on the basis that they clearly think they are above the usual rules that govern the rest of us..and certainly that govern ethical trials.
Thanks a lot to all the signatories of this great letter.
Thank you Vincent Racaniello, David Tuller, and all those who have signed this powerful letter.
Excellent!
This research is, to my regret, representative of many more fatigue studies based on the biopsychosocial model like the studies into the effectiveness of CBT with a graded activity protocol.
The fact that so many scientists subscribe this letter means that the fatigue researchers cannot pretend anymore that it is only a small number of militant patients criticising there (poor) work.
I agree, but it all adds to the growing audit trail of such dismissals, which one day may prove very helpful.
Excellent letter, thank you.
Thank you to the growing list of people challenging the bad science perpetrated by the devotees of the psychosocial model of ME/CFS. Misleading reporting of how the research was conducted and effectively outcome switching midtrial on top a profoundly unreliable experimental design, let’s hope that this SMILE trial proves to be the reductio ad absurdum of this whole cult.
Thanks to David and all the other signatories. Incompetence in research, and publishing research, must be exposed to protect people from these charlatans.
Thanks, David,for sharing this with us, and thanks to all those who signed the letter.
Thank you. Don’t give up, keep fighting these fake treatments. The sooner they’re exposed, the sooner efforts will be directed towards finding real treatments for patients. I can’t stress enough how much we need healthy well respected professionals to take this fight into academia, because there is only so much we sick patients can do.
Bravo!
This is all adding up to an overwhelming body of evidence which one day soon will be produced in a court of law. I hope the BMJ get the significance of all this
I hope this gets submitted as a response on the journal website. That means that even if it is not accepted for publication, readers will still be aware of it.
I hope it is also posted on PubMed Commons. This means that it is under the abstract on PubMed so the attention of a lot of people will be drawn to it.
Good to see so many signatures from Berkeley.!
Especially after Bristol University’s complaint to Berkeley about Tuller.
How do study a treatment that involves patients being instructed to believe they are feeling better? Certainly not by asking if they feel better, and especially not with children.
Not that I would want LP to be studied. I’m just pointing out how the basic foundations for obtaining useful results are once again not there.
When you look at the list of co-authors for this study there are people listed who I’d have thought would be above omitting findings/results, accepting participants switched from a feasibility study, and swapping outcome measures towards more positive results and media coverage. It seems that there are no less than 4 University professors involved with this paper including Professor Alan Montgomery who is now Professor of Medical Statistics and Clinical Trials at Nottingham. Did he really think that this paper was up to scratch? And I hope that those Feasibility study participants whose data was used in this paper were informed of all the protocol changes and then properly consented into the full Trial by signing the updated consent forms. If not then I suspect there may be some ethical issues here as well as these methodological ones.
David, would it be possible for you to send a copy of this letter to Bristol University’s research and ethics committee, or whatever they call themselves, or even to the Vice Chancellor? And to the ethics committee that approved the trial and the changes? And to all the authors of the paper – as someone else pointed out, one wonders whether they were all fully aware of the unethical nature of both carrying out such a trial on children, and of the outcome switching. It would be great if any of these would distance themselves publicly from the trail.
Maybe the pressure on academics is so bad now that even university professors will increase their published output by putting their names to studies without checking exactly how they were conducted. Even if it is, there can surely be no such reason or excuse for Professor Montgomery putting his name to this paper, given the issues cited on this blog. In 2012, he was Director of the Bristol Randomised Trials Collaboration (BRTC), a fully registered UKCRC Clinical Trials Unit (CTU), which, amongst other things, has responsibility for how clinical trials in Bristol are conducted. http://www.uhbristol.nhs.uk/news/bristol%E2%80%99s-clinical-trials-units-awarded-full-registration/ But perhaps he considers that the sort of research practice detailed in Racaniello’s excellent letter IS acceptable/fine?
As for the Ethics Committee, I think there is quite a clear picture emerging that these committees do not work to effectively protect research participants anymore. Their remit seems to be to get as much research through as possible, to not stand in the way of universities or researchers. You do have to wonder whether there are vested interests at play, but perhaps they’re just lazy or overworked too.
Excellent letter, thank you so much Vincent Racaniello, David Tuller and everybody who signed! <3
I’m so grateful that other scientists are starting to look closely at these papers and are willing to speak out against shoddy science.
Some of us patients have spoken out against the smoke-and-mirrors for years, but were dismissed, accused of being militant activists, accused of making vexatious charges, accused of harassing the papers’ authors with the intent of damaging their reputations, and even called names (“chronic fatigue jihadists”). It was like gaslighting on a large scale. It was a frustrating, no-win position, because if we didn’t speak out, the papers would stand unchallenged and if we did speak out, it was used as evidence that we were crazy and shouldn’t be listened to.
It’s a relief to have scientists with no dog in the race, except for a belief in scientific integrity, read these papers objectively.
I don’t believe that most of us patients are seeking a particular outcome. We just want to arrive at the truth about our disease, with the goal of finding effective treatments. But that can only be achieved by good, solid, well designed, unbiased research. It is frustrating to have the extremely small amount of research funding allocated to this disease wasted on shoddy junk science, which then confounds the results of well conducted research.
Thanks so much to Vincent Racaniello, David Tuller, and all who signed the letter for advocating for scientific integrity.
Impressive. (y)
Thank you to David Tuller and all signatories.
Wow! Thanks to all for this great letter!
Such a clear explication of the bias in this study. Have they no shame in endangering very sick children in this way. Bravo for pointing out the errors and for collecting a truly impressive list of signers!