By David Tuller, DrPH
Same-Day Update:
I forwarded our response to Dr Brown’s letter (see below) to Dr Fiona Godlee, BMJ’s editorial director. I also cc-d others on my e-mail to Dr Godlee. Here’s what I wrote:
Dear Fiona–
Dr Brown, the editor-in-chief of Archives of Disease in Childhood, sent a message to Professor Racaniello and me last Thursday about the results of the in-depth investigation of the Lightning Process study. Professor Racaniello sent Dr Brown our response this morning. I am forwarding you a copy of that response.
As you know, the investigators of the Lightning Process study violated BMJ policy on prospective trial registration, selected their primary outcome after recruiting and collecting data from more than half their participants, and failed to mention these details in the published paper. So we applaud the journal’s acknowledgement of the study’s documented flaws. However, the decision to republish the paper with its original findings intact is unacceptable and potentially harmful to children, as we explain in our letter.
I am forwarding our response as well to some of those involved with the issue who have been cc-d on earlier letters about this matter: Sue Paterson, director of legal services at Bristol University; Professor Alan Montgomery, the senior author of the Lightning Process study; four physicians involved with the effort to develop new ME/CFS guidelines under the auspices of the National Institute for Health and Care Excellence; two patient representatives on the relevant NICE committee; Dr Terry Segal, senior author of a recent systematic review that highlighted the Lightning Process study; Tom Whipple, a Times science reporter who has written about ME/CFS; Professor Chris Ponting, vice chair of the CFS/ME Research Collaborative; Carol Monaghan MP, who has expressed alarm at the poor quality of much ME/CFS research; and Professor Racaniello.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
**********
In December, 2017, I wrote a long post about the methodological and ethical violations committed by a team of Bristol University investigators in their conduct and reporting of a pediatric trial of the Lightning Process. The study was published three months earlier by Archives of Disease in Childhood, a BMJ journal, under the following title: “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.”
The issues I raised in the post involved retrospective registration, outcome-swapping, and the failure to disclose salient details in the published paper. On January 30, 2018, Professor Racaniello sent a letter of concern to Dr Nick Brown, the journal’s editor-in-chief, signed by more than 20 experts (including me).
Dr Brown promised to look into the matter, and he kept his word. On June 19, 2018, the journal posted an editor’s note acknowledging the validity of the issues raised and explaining that the situation was under “editorial consideration” in light of “clarifications” provided by the investigators. Unfortunately, this editor’s note was not visible from the paper itself, so was likely not seen by most readers.
I repeatedly highlighted the obscure placement of this editor’s note in blog posts and private messages. Last October, I finally received a short note from Dr Fiona Godlee, BMJ’s editorial director. Dr Godlee suggested that the organization’s “systems” did not make it easy for the IT department to address the problem–although she assured me she would see what she could do. Apparently she wasn’t able to do anything, since the editor’s note never moved.
Last Thursday, Professor Racaniello and I received an e-mail from Dr Brown. He alerted us to the news that a “corrected” version of the Lightning Process study had just been posted, along with a lengthy correction notice and a new editor’s note explaining the changes. He thanked us for bringing the issues to the journal’s attention. I have posted that message below, followed by our response, which Professor Racaniello sent this morning.
I consider it a major victory to have Archives of Disease in Childhood acknowledge that everything I documented is true. The correction notice itself is unusual in its length and in the sheer number of points it needs to cover. But allowing the investigators to re-affirm their original reported findings despite the study’s disqualifying methodological and ethical violations represents a perplexing abrogation of BMJ’s editorial responsibilities.
**********
Dear Professor Racaniello and Dr Tuller
Thank you both once again for making us aware of your concerns about this paper
Having acknowledged your comments as valid, we embarked on the (highly unusual) process of subjecting the paper to a full re-review, both methodological and senior statistical
During this process, it became clear that a number of facets of the manuscript needed to be addressed and clarified. To do this to both the reviewers’ and our own satisfaction has required a number of iterations, multiple discussions and a substantial amount of time.
Our conclusion, in short, was that the issues did warrant, as you suggested, the publication of a corrected version to take the place of the original. As a courtesy, we wanted to let you know that this has (literally) just gone ‘live’. The new (corrected) version is linked to both a full correction note (in which every change to the original is listed) and an editor’s note detailing the decisions leading to and chronology around the process we have undertaken
Thank you again for your comments and for your patience during the (unavoidably) lengthy process
Nick Brown
**********
Dear Dr Brown—
Thank you for your letter. We are pleased that Archives of Disease in Childhood has published a correction and editor’s note confirming what Virology Blog documented in December, 2017: The investigators of the Lightning Process study violated BMJ policy on prospective trial registration, swapped outcome measures after gathering data from early trial participants, and failed to disclose these salient details in the published paper.
However, we are surprised you believe addressing these concerns is only a matter of “clarification” and that you have therefore allowed the study’s reported findings to stand. This decision appears to call into question previous statements from BMJ leaders about the critical role of prospective trial registration in protecting the integrity of the published literature. Examples include the following:
*In 2013, Dr Godlee testified before the House of Common’s Science and Technology Committee about BMJ’s efforts to implement a zero-tolerance policy for trials in which recruitment of participants preceded registration. “As far as we are aware, in the last two years we have not published any trial that has not been prospectively registered,” testified Dr Godlee–a claim she could not truthfully make today.
*In a 2017 article, Dr Elizabeth Loder, The BMJ’s head of research, discussed how the medical and research communities might render these trials “a thing of the past.” One solution, proposed Dr Loder, “might be to treat them as the research equivalent of medical ‘Never Events’…These are ‘particularly shocking medical errors (such as wrong-site surgery) that should never occur.’”
*Last year, Dr Trish Groves, BMJ Open’s former editor-in-chief, tweeted the following: “Prospective registration of #clinicaltrials isn’t red tape: it’s done for sound ethical and scientific reasons.”
When it comes to the Lightning Process study, you seem to regard the registration issue as a “red tape” mix-up that can be resolved through a simple “clarification while the findings themselves remain intact. In this case, BMJ has apparently abandoned the “sound ethical and scientific reasons” behind its principled position on trial registration and adopted a more expedient or flexible approach. In the editor’s note, you declare that non-adherence to the prospective registration requirement does not warrant retraction–but you offer no rationale for this relaxed interpretation of long-standing BMJ policy.
Your current stance is not compatible with the tough, no-nonsense views articulated by Dr Godlee, Dr Loder and Dr Groves. As Dr Loder noted in her 2017 article, “Trustworthy clinical trials = prospectively registered trials that faithfully report their prespecified outcomes.” In a 2018 BMJ Open article, Dr Loder and colleagues described prospective registration as “an important safeguard” against selective reporting of results. The investigators of the Lightning Process study not only failed to register their trial prospectively but also failed to “faithfully report” their prespecified outcomes.
These investigators swapped primary and secondary outcomes after collecting data from early participants, who were initially recruited for a feasibility study and then included in the full trial sample. For the feasibility study, school attendance at six months was designated as the primary outcome, and self-reported physical function at six months as a secondary outcome. At the end of the full trial, school attendance at six months yielded null results—but this measure had been demoted to a secondary outcome. The new primary outcome, self-reported physical function at six months, had positive results.
In a clinical trial, it is problematic to identify the primary outcome after collecting data from more than half the participants. Whatever the intentions of the investigators, their undisclosed outcome-swapping allowed them to present positive rather than null results for what they reported as their primary outcome. It is irrelevant whether or not they analyzed outcomes data from the feasibility study before deciding to swap measures for the full trial. In open-label trials relying on self-reported measures, like the Lightning Process study, trends in outcomes are generally apparent to investigators long before the data are formally scrutinized.
In the editor’s note, you discuss the journal’s extensive and lengthy review of the multiple issues raised about the study. With regards to the outcome-swapping, you explain that the journal’s review involved “seeking assurance from the authors that the change in primary outcome was not influenced by (positive) findings in the feasibility phase.”
This statement is striking. Individuals subject to potential bias are often not considered impartial and authoritative arbiters of whether this potential bias has influenced their decision-making. Perhaps you and Dr Godlee believe it is reasonable and appropriate to take such “assurance from the authors” at face value while ignoring the self-evident impact of their methodological choices on the reported results. We do not.
Beyond the truism that humans tend to be blind to their own biases, it is worth noting that these investigators withheld important information about trial registration and outcome-swapping in their public version of events. In a significant departure from reality, the published paper portrayed the trial as if it were prospectively registered, with pre-designated primary and secondary outcomes. This sort of misrepresentation does not inspire confidence in any subsequent assurances of impartiality.
Archives of Disease in Childhood would not have published this paper in the first place except for two major failures: 1) The investigators’ failure to provide editors with an accurate account of how they conducted the trial; and 2) The journal’s failure to detect and address unacceptable deficiencies, despite its obligation to subject the paper to rigorous processes of peer review and editorial oversight.
Given the undisputed facts, BMJ’s decision to publish a “corrected” version of this study that re-affirms the original findings is scientifically and ethically indefensible. It is also potentially harmful to the health and well-being of children.
Because of public interest, this exchange will be posted on Virology Blog and forwarded to others engaged with the issue.
Best—
Vincent Racaniello, Ph.D. | Higgins Professor
Department of Microbiology & Immunology
Columbia University, New York
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
Comments
11 responses to “Our Exchange with BMJ Journal about “Correction” of LP Study”
It’s so hard not to stray into the land of ‘ad hominem’ isn’t it? Neither patients with ME nor parents of patients with ME are stupid. The former are unwell and the latter are probably worried out of their wits for not being able to access good, properly peer reviewed, ethically generated research advice for treatments which might ameliorate their children’s suffering. But ‘stupid’ they ain’t.
If it wasn’t all so serious this would be funny. Grown men and women, masquerading as ‘academics’ allowing mindless drivel like the LP Study to see the light of day..and still, despite every error being spelled out to them, & more than once, reluctant to retract a paper which had no business in any publication bar perhaps an edition of ‘Paranormal Magazine’.
One glaring omission here is how such mistakes made it through the review process in the first place. If this truly were a mistake, there would be genuine concern about understanding how obvious breaches to BMJ’s rules and standards went “unnoticed”. We see the opposite of that.
What actually happened is they got caught, were not allowed to just let the issue die of old age and made cosmetic changes that entirely avoid the actual issues without taking any concerns with how those issues could happen at all.
Especially when this pattern happens all over Crawley’s research. Psychosocial research is graded on a curve that grants it the least amount of scrutiny despite a clear need to apply the maximum level. Subjective narrative-driven research by biased ideologues with a history of deceit is the least reliable type of research and here is a perfect example of why it amounts to a gassy landfill of Potemkin medicine: zero accountability, exemption from the usual rules and optional requirements.
Meanwhile vulnerable children are the only ones facing consequences, without recourse or informed consent. This is moral bankruptcy.
Were the kids given the opportunity to opt in or to opt out at retrospective registration? That’s what I want to know. There’s a big ethical difference between the two. I wonder, did either ADC or BMJ bother to ask the researchers this key question? Perhaps the kids did opt in, but I can’t see this mentioned in the ‘corrected’ paper or in the ‘Notice of correction and clarification’ (unless I missed it), so it’s as clear as mud to me.
Thank you!
I applaud and thank you for your continued work on this matter, David. I’m sorry it’s such a sad farce. In my view, there is no way they can disavow this particular article because it would draw a lot of attention to the fact that they’d taken the Lightning Process seriously in the first place. And the Lightning Process is just too uncomfortably similar to the BPS approach itself. One is straight up hucksterism and magical thinking. The other is science-y magical thinking. And it’s not a coincidence, the history of psychosomatic “medicine” is tightly bound to popular beliefs in what was called “mind power” in the late 19th century. People buy into BPS b.s. for the same reasons that people buy books like The Secret. The former is a little more highbrow but it’s just as hollow no matter how many half-baked studies you drape over it.
I think what you published made a bunch of sense. However,
what about this? suppose you wrote a catchier title?
I mean, I don’t want to tell you how to run your blog, however
what if you added something that makes people want more?
I mean Trial By Error: Our Exchange with BMJ Journal about “
Correction” of LP Study is kinda plain. You could look at Yahoo’s home page and see how they write
news headlines to get viewers to open the links. You might try adding
a video or a related picture or two to get readers excited about
what you’ve written. Just my opinion, it would bring
your posts a little bit more interesting.
Thank you for continuing to address these research and now editorial malpractices.
That these are only part of the flaws in this study, that subjective outcomes are meaningless, indeed irrational, in evaluating an intervention that instructs participants to lie about their symptoms, that any intervention that instructs children to lie about their symptoms raises child protection concerns and that an intervention’s contents are not actually reported because of commercial sensitivity make it essentially unreplicable by independent researchers all render its conclusions profoundly suspect, over and above its violations of the BJM standards that should have meant it was unpublishable had the researchers not in the first incidence misled the Journal.
The Bristol researchers and the BMJ seem determined to give credibility to a dangerous alternative psychotherapy that makes unjustified claims about curing a large number of biomedical conditions and already has been involved advertising standards disputes and negative judgements in a number of countries. It is not acceptable that a British University and the BMJ are now effectively colluding to provide a spurious validation to the Lightening Process.
We further have anecdotal evidence of serious harm arising in children as a result of participation in the Lightening Process that the company and faith healer behind it do not report, indeed deny the existence of. For some children the repeated over exertion that is a likely consequence if this intervention is actually taken seriously by a child with ME, a condition that is by definition worsened by over exertion, can see the deterioration of a potentially recoverable disease into an irreversible life long disability.
Such research should be the reductio ad absurdum of the use of subjective outcome measures in open label trials, instead you have an internationally respected journal bending over backwards to kept this research in press despite its research malpractice being incompatible with their own declare policies and standards, in this situation one must ask the question why?
I suspect we are now seeing a desperate attempt at protecting reputations by denial and obfuscation at the expense of a risk of very real harm to vulnerable children.
The new new editorial comments and the authors’ changes ask us to believe subjective assurances from the authors that they were not subject to unconscious biases as justification for allowing the study’s conclusions to stand in the literature. Essentially the authors are saying ‘we believe our results even though we diverted from the BMJ standards and generally accepted scientific practice and did not tell anyone’, and the Journal is replying ‘OK darling, if you say so everything is fine’.
Is this remotely reasonable, given the authors previously misled both the Journal and its readers?
Either the authors knew they were breaching both the BMJ’s standards and generally accepted scientific good practice, in which case they deliberately sought to mislead, so why should we believe them now, or they completely failed to understand that these standards and practices are in place to prevent subconscious biases influencing research, so why should we now accept their subjective assurance that they were not influenced by such biases that they previously appeared not to understand.
I’m liking the same-day update and also Peter Trewhitt’s comments above, especially the either…..or paragraph. But can anyone who is in the know answer my question about opt-in/ opt-out?
“It is also potentially harmful to the health and well-being of children.”
I’m afraid this is still missing a much wider and more important point that affects *all* M.E. (and wider ‘MUS’) patients, and is going largely unreported or recognised:
The ‘study’ is being allowed to conclude that an intervention that is well known to be unscientific woo, is, nevertheless, effective at improving the lives of M.E. patients. *This is certain to be taken, in the wide, stubbornly sceptical, medical community, to ‘prove’ that M.E. is ‘all in the mind’*: which it most certainly is not!
Furthermore: The study is a danger way beyond the paediatric M.E. world, in that it gives an air of scientific respectability to the large number of unscrupulous charlatan woo merchants who prey on the sick for their own enrichment.
It is lamentably irresponsible, and unforgivable, that a major scientific journal is helping to both promote and perpetuate an obvious confidence trick, in a way that will just encourage even more tricksters to set up shop and start ripping off patients and their parents and carers, themselves!
This is a much more serious matter than is being presented: it adds to the climate of scientific illiteracy that is all pervasive in this Internet Age; it tarnishes the respectability of one of the core sources of reliable medical knowledge; it paints an entire patient group as hysterical and incapable of judging their own state of health, and likely to be susceptible to brainwashing; it encourages yet more woo merchants to have a go: thereby tending to perpetuate the danger to *all* patients seeking knowledge and possible treatment.
The study simply cannot be allowed to stand!
One is left to wonder at the bidability, or simple gullibility, of the medical publishing world in their dealings with those who seek to promote psychologically based pseudoscience ‘treatments’, via modes of ‘research’ that require little thought or effort, and even less science, whilst maximising the ‘reward output’ for the effort put in–so perpetuating the attraction of the whole pseudoscientific field to the unscrupulous or just plain lazy!
The link is to my detailed critique of the Lightning Process, supporting much of what has been said above.
I also have read in one of Crawley’s accounts that a lot of children didn’t want to get involved with the trial, and I assumed that this meant they didn’t… I hope this assumption is right.
The Lightning Process is dangerous and very damaging not because NLP ‘doesn’t work’, it is because it does. People can get carried away with the powerful motivational strategies and belief-change strategies into actually trying to pretend they aren’t ill, throwing away their aids, exercising….and do themselves terrible, irremediable harm, as many adults who have gone through it can testify.
Parker should be sued, jailed….there are no words for a person who makes money through this abusive use of a system of powerful psychological techniques.
The proper use of any supportive form of psychotherapy for PWME, including NLP, is to help the patient to free themselves from the powerful cultural belief that ‘exercise is good for you no matter what’. We have to ‘reframe’ exercise as TOXIC, and rest as the most effective strategy we have…to allow our bodies the energy needed to fight whatever pathogen they are dealing with in this disease. Ramsay prescribed complete rest from the inception, and he was right. PWME also need help to realise that losing the ability to do physical things, help people as they want to …does not make them less valuable as human beings…we need help to keep our sense of identity in the face all the changes forced upon us by our illness. This is the appropriate use of any form of psychotherapy for a person with ME.
There certainly are serious child protection issues…the children are told they must say they feel good no matter how they actually feel, and told the treatment ‘won’t work if they tell anyone what it is’. Then they are pressured to answer a questionnaire about how much the treatment has helped. This meets Bateson’s four criteria for a schizophrenogenic ‘double bind’…The person is asked to do two opposite things (lie/tell the truth), can’t comment on the situation they are in (‘the process won’t work if you tell anyone what it i), and can’t escape from the situation (repeated phone calls from the researchers asking for the answers to the questionnaire. According to Bateson, it is this combination that creates psychotic episodes. I believe there has been one suicide of a child following the Lightning Process.
http://www.positivehealth.com/article/cfs-me/me-cfs-nlp-and-the-lightning-process-in-the-looking-glass
People keep asking why this kind of flawed research gets a free pass. Study the history of the ‘welfare to work’ policies developed and encouraged when Tony Blair was changing government attitudes to disability support, the involvement of Simon Wessely from that time onward, (now ‘mental health czar’ under Teresa May), the involvement of Layard and the idea that because the long-term unemployed often had more mental health problems than people in jobs, that forcing the long-term unemployed into work would improve their mental health…and that approach now applied to the chronically ill…forcing them back to work will cure their illness. Study Simon Wessely’s many roles…President of Royal Colleges, board of the SMC, board of the Wellcome Trust, wife on MRC for many years, author of the advice for Joint Commissioners of services for people with medically unexplained symtoms. Layard is strongly behind the push for IAPT. Read Monbiot on ‘Invasion of the Entryists’…right wing establishment figures posing as Radical Left….
This stuff….the BPS model….is deeply, deeply embedded within the British establishment…there is nothing accidental about any of this…
I’ll post this now, but can post links to all this for anyone interested in following it up.