By David Tuller, DrPH
Last week I posted three comments sent to Dr Fiona Godlee, editorial director of BMJ, in support of retraction of the biased and discredited Lightning Process. All three–Professor Ola Saugstad of University of Oslo, Professor Vincent Racaniello of Columbia University, and Professor Elisa Oltra of University Catolica de Valencia San Vicente Martir–were among the 55 experts who signed Virology Blog’s recent open letter to Dr Godlee. That letter expressed concern and dismay that Archives in Disease in Childhood, the BMJ journal that published the study two years ago, had acknowledged the methodological violations documented on Virology Blog but nonetheless republished the original, seriously compromised findings. When I sent the letter to Dr Godlee, I cc’d many of the signers.
Since Professor Oltra sent her note, additional calls for retraction of the paper have come from: Dr Alan Gurwitt, a retired Yale psychiatrist; Dr Marlon Maus* [corrected shortly after posting from Maos–sorry, Marlon!], a colleague from Berkeley’s School of Public Health; and Professor Jonathan Edwards of University College London. In his letter to Dr Godlee, Professor Edwards mentioned Cochrane’s revised risk-of-bias tool, just published by BMJ, and noted a troubling detail–the revision’s lead author, Professor Jonathan Sterne of Bristol University, was also a co-author of the Lightning Process study.
*****
Dear Dr. Godlee:
This message is to add to the others you have already received regarding the correction and republication of the Lightning Process study, originally published by Archives of Disease in Childhood in September 2017. That article was entitled “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical for paediatric chronic fatigue syndrome: randomized controlled trial.”
I fully agree with the points outlined in Dr Tuller’s letter, which many other clinicians and scientists also signed. This study must be retracted in order to protect the interests of children. The decision to republish the study with the original findings was grossly inappropriate.
I am a retired psychiatrist and one of several co-authors of “Myalgic Encephalomyelitis/Chronic Fatigue Syndrome Diagnosis and Management in Young People: A Primer,” published in Frontiers in Pediatrics (June 2017). In the process of reviewing the literature for that primer, I reviewed what was available on the Lightning Process. I came to the conclusion that this treatment was utter nonsense. Its proposed use for ME/CFS emanates from the misguided belief in the UK that the illness has psychological rather than pathophysiological underpinnings.
Sincerely,
Alan Gurwitt, MD
Psychiatrist in Private Practice (retired)
Associate Clinical Professor (retired)
Yale Child Study Center
Yale School of Medicine
New Haven, Connecticut, USA
**********
Dear Dr. Godlee,
As an ophthalmologist and Professor of public health I wish to add my voice to that of others regarding the unsound study “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical for paediatric chronic fatigue syndrome: randomized controlled trial.” After reviewing the background for the study and its conclusions I concur completely with the interpretation by Dr. Tuller and others. The medical profession is rife with examples of the victimization of patients that have been diagnosed with psychiatric diseases when in fact later studies and research have proven them physical diseases. I believe that this may be such case and we must utilize the precautionary principle and I suggest you consider retracting the study until further evidence is obtained.
Sincerely,
Marlon Maus, MD, DrPH, FACS
Professor, School of Public Health, UC Berkeley
*****
Dear Fiona,
I note the recent publication in BMJ of an account of the finalised Risk of Bias 2 tool (RoB2) used in the context of the GRADE system used by Cochrane and NICE for assessing quality of clinical trial evidence. The document focuses on reducing the stringency of RoB assessment principally in unblinded trials.
My immediate reaction to this was that it is a retrograde step, since, increasingly, unblinded trials appear to be given more credence than they deserve. My immediate reaction was then followed by a more serious concern when it was brought to my notice that the corresponding author, Jonathan Sterne from Bristol, is also an author on the paper reporting on the SMILE trial of the Lightning Process that has been the topic of previous correspondence and is now again.
Amongst the proposals for increasing leniency when considering bias in trials, the RoB2 document re-assesses the impact of changing outcome measures after trial initiation but before data analysis and also makes reference to problems with bias due to beliefs held by the patient or treatment delivery team when outcome measures are subjective. It will have not have escaped your notice that amongst the many flaws in the SMILE trial are the issues of outcome switching midstream and the problem of severe risk of expectation bias relating to subjective outcome measures.
So I find myself writing to you again to support the request from David Tuller, and now fifty or more other colleagues, to take the problems with the SMILE trial seriously. I think I had previously suggested that a retraction or clear indication of the flaws of the trial was needed. Looking at the revised manuscript I cannot see that anyone would consider the problem adequately addressed since the inappropriate conclusion remains the same.
Things are beginning to look surreal. It is almost as if the SMILE trial was an exercise in demonstrating that scrutiny by GRADE and RoB systems can be circumvented with impunity.
The GRADE system starts from the premise that a randomised trial provides high quality evidence unless it suffers from one or more defects including bias. The weakness of this can be illustrated by proposing a trial in which patients are randomised to being taught that they will only get better if they think and say they feel better, whether they do or not, or to being told to say how they really feel, and then using how the patient says they feel as outcome. One might think such a trial would be absurd, and never proposed. Yet this appears to be more or less what the SMILE trial is. It is pretty much how patients describe their experience. On the other hand, as Dr Oltra points out, we are not told what really went on, and it looks as if according to RoB2 the risk of bias is not scored high if we do not know what went on!
RoB2 does mention the possibility that bias might arise from patient or therapist beliefs but gives as examples the rather extreme cases of a physiotherapist assessing the benefit of her own treatment or a patient having a belief in homeopathy. There appears to be no recognition of the fact that expectation bias due to beliefs about the value of any sort of treatment are ubiquitous in trials (and in any scientific experiment, including work in the lab) and are the basic reason why we blind ourselves to test and control. In ME/CFS we have seen dramatic responses due to expectation bias with conventional drugs with no therapeutic effect, including rituximab and anti-virals. The RoB2 analysis appears either very naïve or disingenuous.
As we are all aware, the SMILE trial had a highly unsatisfactory structure, being initiated as a ‘feasibility study’, with a large number of patients recruited, and then being registered as a ‘formal study’ after switching of outcomes in the knowledge of the progress of the first 50 or so patients. This might fall under Dr Sterne’s allowed situation of before data analysis but it is a classic cherry picking scenario.
‘Feasibility trials’ and ‘pragmatic trials’ appear to be increasingly popular. As far as I can see these techniques are designed to make trials look as if they have a valid structure for gathering reliable evidence when they almost certainly do not. I have a strong impression that ‘methodological experts’ associated with clinical trials units and other related departments may have a conflict of interest in terms of co-authorship on publications of poor quality trials.
I had for some time thought that the problems of poor quality trials in ME/CFS were at the periphery of academic medicine. Now I get the strong feeling that they may be typical of a general process of degrading the quality of clinical science in the UK. We appear to be moving towards acceptance of methodology that for decades we have known yields meaningless results. I get the impression that bodies like Cochrane and the BMJ are sleepwalking into a situation where they rubber stamp commercial ventures of no merit. In this wider context I can only emphasise the view of all those copied above, that the SMILE trial paper should be retracted and that a fully independent investigation should be completed as soon as possible.
Yours sincerely,
Jonathan Edwards
Comments
21 responses to “More Experts Urge Godlee to Retract Lightning Process Study”
Thank you!
Thank you!
Brilliant! Simple masterful!
Such important issues, especially Professor Edward’s points.
If the British establishment, including the BMJ, continue to protect bad science such as the Lightning Process study, we will seen British science increasingly become an irrelevance on the fringe of real work, a backwater of misinformation and propaganda.
“Things are beginning to look surreal. It is almost as if the SMILE trial was an exercise in demonstrating that scrutiny by GRADE and RoB systems can be circumvented with impunity.”
As someone whose ME started in childhood and has continued to progress over the last 50 years I make 2 observations: i) I’m so glad that Crawley et al weren’t around when I first developed the disease and ii) things have been feeling very surreal for me for many decades now. ME is a grim disease to have and regularly comes out ‘last’ in tables of QOL. Given that, the quicker this sham ‘research’ [cobbled by people who seem to have no scientific integrity but do seem to wield a lot of power such that their own nakedness is ‘blinded’ (to them) even if their studies aren’t] is retracted, the better.
Fantastic. Thank you so much!!
Brilliant! Thank you so much!
This really isn’t a good look for the UK medical establishment. For so long we’ve seen them try to frame the reported problems with ME research as abusive patients and their advocates persecuting saintly doctors and researchers. Now, thanks to David and all those who’ve signed the letter and put pen to paper to write to Fiona Godlee themselves, it’s clearly a matter of the quality and soundness of the science involved, or rather the lack of. By these accounts, the LP study fails miserably and needs retracting. How long can Godlee hold out I wonder? Does she feel any responsibility for the health and safety of children with ME?
Recently Godlee wrote an article in the BMJ (https://www.bmj.com/content/366/bmj.l5382) – “Brexit: doctors’ duty is to inform patients” in which she appeared to agree with Peter English and colleagues (who had also written an article on the same subject in relation to Brexit) that doctors’ first duty is to the patients that they serve. So would she agree that their first duty shouldn’t be to the reputation of other doctors/researchers, nor should it be to the reputation of a journal or journals that they happen to be the Editor/Editor-in-chief of or Editorial Director of/over?
When there are individuals with strong powers of persuasion who are informally let loose on vulnerable children with apparent ‘success’, who then are seen to be suitable therapists to use to continue in a allegedly more formal way because the outcomes suit a particularly dear model to the authors and others with vested interests, I feel this is like exposing these children to bogey men with no sense of right and wrong.
The cherry picking referred to above is exactly that. It is unethical to the core and at every level.
Brilliant letters. Thank you.
When there are individuals with strong powers of persuasion who are informally let loose on vulnerable children with apparent ‘success’, who then are seen to be suitable therapists to use to continue in a allegedly more formal way because the outcomes suit a particularly dear model to the authors and others with vested interests, I feel this is like exposing these children to bogey men with no sense of right and wrong.
The cherry picking referred to above is exactly that. It is unethical to the core and at every level.
Brilliant letters. Thank you.
Alan Gurwhitt wrote:
“This study must be retracted in order to protect the interests of children. ”
Somebody please get Godlee to stand on a circle and say this over and over.
SMILE was and still is child abuse – plain and simple.
Re my last comment – this blue circle would do – (https://twitter.com/bmjlearning?lang=en)
I sure hope someone comes up with the idea of legal action to stop the fraud and the harm. Per Fink reveals that litigation will get his attention:
” they are bringing lawsuits against our research articles. This means that researchers who are otherwise interested in the field will stay away. They can’t stand the harassment” [1]
What are the lawsuits he is referring to, and have they been successful?
[1] https://translate.google.com/translate?sl=da&tl=en&u=https%3A%2F%2Fdagensmedicin.dk%2Fprofessor-i-funktionelle-lidelser-patienter-kan-ikke-selv-bestemme-hvad-de-fejler%2F%3Ffbclid%3DIwAR3Up96o2rulcVqAIj_3RHbIwDOJ-tFBJaH0CcYhtkJM3ArIk262rAn0Aog
*Very Maude Flanders voice*: won’t somebody think of the children?!
Seriously, how is it possible that such unchecked psychological abuse be “ethically approved”, publicly funded (in the case of the trial) and condoned by the British Medical Association, by way of the BMJ?
It has been pointed out that the substance of the LP must remain secret. Telling children to keep secrets about sham therapy they received is incredibly ethically bankrupt. Were the peer-reviewers at BMJ made aware of the content of the LP? Either answer is troubling, a black box “treatment” being tested cannot reasonably pass any form of peer review, whether the reviewers are not aware of its content or are privately informed of it. Did they sign confidentiality agreements about the contents of the commercial LP? Or simply not bother with “reviewing” a black box?
What’s remarkable is how similar SMILE is with PACE, in style and substance. Now the two are joined at the hip by way of Crawley and Sharpe himsel seemingly condoning LP as just as good as his own CBT/GET combo.
But to involve children in this has to stop. This is just about the worst possible to conduct medical research, as ethically bankrupt as it is intellectually. Any trial with this shoddy methodology would prove the “benefits” of literally anything, from whole-bee-colony-stinging therapy all the way to foot-smashing therapy.
Thank you so much to everyone who sent letters to Dr. Godlee.
I just can’t believe we even have to argue about this- it’s like an episode of Twilight Zone.
Thank you so much
Thank you for taking the time and effort to write your own letters. The story of this study is so incredible. We need respected, credible people to speak up. Thank you.
Edwards really does write very well argued pieces.
Tragic for UK medical science, that Lancet, BMJ, and now even Cochrane, have chosen to be champions of quackery instead.
Dear me, is this the result of letting alternative medicine and other rubbish become acceptable? Why is this fight still going on?
hi