More Letters About BMJ’s Flawed Pediatric Studies

By David Tuller, DrPH

This morning I sent three more e-mails alerting interested parties to my concerns about two BMJ studies of children with ME/CFS. When it comes to research, kids are already a vulnerable population, and those with a stigmatizing illness even more so. That’s why it is both surprising and troubling that BMJ appears to have little interest in addressing the ethical and/or methodological violations that Virology Blog has documented in these pediatric studies.

I first sent an e-mail to the executive board of the CFS/ME Research Collaborative. Given that the CMRC is likely to continue to play a role in both research and health policy going forward, I believe members of the organization’s leadership should be alerted to serious problems in high-profile studies about the illness, especially those involving children.

Next I sent an e-mail to Jonathan Montgomery and Teresa Allen, who are, respectively, the chair and the interim chief executive of the Health Research Authority, an arm of the National Health Service that, among other activities, oversees the process through which investigators receive approval from local research ethics committees. Finally, I sent an e-mail to Professor Dorothy Bishop, a well-known neuropsychologist at Oxford, who provided a statement last September for the Science Media Centre’s round-up of expert opinion about one of the studies.

As I have indicated repeatedly, the BMJ journals in question—Archives of Disease in Childhood and BMJ Open–have abrogated their editorial obligations. At this point, BMJ editorial director Fiona Godlee has a responsibility to explain how the journals got things so wrong in the first place and why they have refused to acknowledge the problems even when presented with irrefutable documentation. She should further explain why BMJ in this instance has appeared to place its reputational and institutional concerns above children’s interests and wellbeing.

I have cc’d Dr Godlee on my e-mails so she can respond directly, should she so choose. On my e-mail to Professor Bishop I also cc’d Edward Sykes, the Science Media Centre’s head of mental health and neuroscience.

**********

Letter to CMRC Executive Board

Dear Executive Board Members of the CMRC:

Many of you were on the receiving end of a series of contentious e-mails I sent last year after I had been falsely accused of libel. Now that the situation has changed, I’d like to congratulate the CMRC on its decision to pursue a new direction. Professor Ponting’s public statements about ME/CFS have been very welcome and have provided significant grounds for optimism. I hope we can have a more productive relationship going forward.

As you all likely know, I have been a sharp critic of the PACE trial. But when it comes to publishing poor research in the ME/CFS field, it would be unfair to point fingers solely at The Lancet. Given the CMRC’s central role in the ME/CFS domain, I felt it would be appropriate to bring your attention to two problematic pediatric studies published in BMJ journals–Archives of Disease in Childhood and BMJ Open.

Both studies involved children suffering from this illness—a stigmatized group within an already vulnerable population. And both studies violated core scientific and/or ethical principles. Yet this research could impact public health policies and the medical treatments available to kids, if it hasn’t already, and trigger follow-up studies. Although the editors of the two journals have been informed of the issues, they have so far failed to take the appropriate and necessary corrective steps. This failure is arguably placing children at risk of harm.

The Archives of Disease in Childhood paper, called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial,” was published last year. The BMJ Open paper, called “Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics,” was published in 2011.

In the Archives of Disease in Childhood study of the Lightning Process, more than half the participants—56 out of 100–were recruited as part of a feasibility study starting in September, 2010. The trial itself was registered in July, 2012. Even as these feasibility study participants were folded into the full trial, primary and secondary outcome measures were swapped based on their results–an obvious recipe for biasing the findings. The Archives of Disease in Childhood paper did not disclose these relevant methodological details—a disturbing omission.

To protect against bias, BMJ and other leading journals have longstanding policies against publishing trials in which participants were recruited before registration. The Lightning Process study clearly breached this policy. Yet Archives of Disease in Childhood has nonetheless provided practitioners of the Lightning Process with bragging rights that their pseudo-scientific program has proven effective in treating children in a randomized trial. The news coverage about the study has likely driven more desperate parents to spend their hard-earned money on the Lightning Process.

In the BMJ Open study, the investigators sought to test the hypothesis that children with chronic fatigue syndrome could be identified through school absence records. But they exempted the study from ethical review on the false grounds that it was “service evaluation” and not “research.” The study relied not on anonymous secondary data but on primary data collected directly from identified children, and it included a hypothesis and generalizable conclusions. According to U.K. guidelines, a study with these features cannot be classified as “service evaluation” but qualifies as “research” and requires ethical review.

A peer reviewer for BMJ Open raised tough questions about the study’s claim that it was not “research” but “service evaluation” and therefore exempt from ethical review. Although the investigators failed to provide adequate responses, BMJ Open overlooked the issue and accepted the paper anyway without ethical review. Ironically, the journal published it under the heading of “research” but has since defended the decision to categorize it as “service evaluation.”

You can read more details about the documented problems with these studies, and letters of concern sent to the journal and signed by more than a dozen scientists, clinicians and other experts, in the following Virology Blog posts:

Trial By Error: The SMILE Trial’s Undisclosed Outcome-Swapping
Trial By Error: A Letter to Archives of Disease in Childhood
Trial By Error: No Ethical Review of Crawley School Absence Study
Trial By Error: A Letter to BMJ Open

I have cc’d Fiona Godlee, editor-in-chief of The BMJ and editorial director of BMJ, on this e-mail. Perhaps she can explain why her editors have not yet acknowledged the obvious: The Archives of Disease in Childhood study violated BMJ policy on prospective trial registration, and the BMJ Open study is not “service evaluation.” Neither paper should have been accepted for publication in the first place.

I am raising these issues with the CMRC now because these BMJ studies are at risk of being given credence in deliberations on children’s public health issues or of being cited in support of further research. Like PACE, these studies have no place being considered in the development of any public health initiatives, clinical guidelines, or recommendations on medical care. Nor should they serve as the basis for further research by CMRC-affiliated scientists or others.

Thank you for your consideration of this matter.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

**********

Letter to Health Research Authority

Dear Jonathan Montgomery and Teresa Allen–

I am a research fellow in public health and journalism at the Center for Global Public Health at the University of California, Berkeley. For the last couple of years, I have been investigating the PACE trial and the associated body of research into GET and CBT as treatments for the illness variously known as CFS, ME, CFS/ME, and ME/CFS

As far as I’ve been told, your remit is to address issues going forward and not to investigate past concerns. However, it would be perhaps useful to your work to review instances in which the research ethics committee (REC) process has failed to ensure optimal results. For that reason, I want to bring your attention to two studies published by BMJ journals—one in Archives of Disease in Childhood and the other in BMJ Open.

Both studies involved children suffering from this illness—in other words, these participants are members of an already vulnerable population with a stigmatized condition. And both studies violated core scientific and/or ethical principles. Yet these flawed studies are in a position to impact public health policies and the medical treatments available to kids, if they haven’t already. Although the editors of the two journals have been informed of the concerns, they have so far failed to take the appropriate and necessary corrective steps. This failure is arguably placing children at risk of harm.

The Archives of Disease in Childhood paper, called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial,” was published last year. In this case, a subcommittee of the local REC approved an application for a substantial protocol amendment that guaranteed the introduction of bias into the study

The BMJ Open paper, called “Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics,” was published in 2011. In this case, the investigators cited an REC opinion from earlier research to exempt from ethical review this completely unrelated study with a very different data collection method, falsely claiming that it was “service evaluation” and not “research.”

In the Archives of Disease in Childhood study of the Lightning Process, more than half the participants—56 out of 100–were recruited as part of a feasibility study starting in September, 2010. The trial was registered in July, 2012–almost two years later. At the time of registration, the investigators also sought permission from the local REC to simultaneously extend the feasibility study into the full trial while swapping primary and secondary outcome measures. Since this was a clear recipe for bias, the investigators should not have made this request; once made, it should have been denied. Instead, a two-member subcommittee of the REC approved it. In a troubling omission, the Archives paper did not disclose that the outcome measures were swapped midway through based on data from more than half the trial participants.

To protect against bias, BMJ and other leading journals have longstanding policies against publishing trials in which participants were recruited before registration. The Lightning Process study clearly breached this policy. Yet the Archives paper has nonetheless offered practitioners of the Lightning Process bragging rights that their pseudo-scientific program has proven to be effective in a randomized trial in treating children. The news coverage about the study has likely driven some desperate parents to spend their hard-earned money on the Lightning Process in hopes of curing their kids.

In the BMJ Open study, the investigators sought to test the hypothesis that children with chronic fatigue syndrome could be identified through school absence records. But they exempted the study from ethical review on the false grounds that it was “service evaluation” and not “research,” citing an earlier REC opinion that had nothing to do with the new research or the method of data collection. The study included a hypothesis and generalizable conclusions and relied on primary data collected directly from identified participants, not on anonymous secondary data. According to the applicable guidelines, a study with these features cannot be classified as “service evaluation” but qualifies as “research” and requires ethical review.

A peer reviewer for BMJ Open raised tough questions about the study’s claim that it was not “research” but “service evaluation” and exempt from ethical review. Although the investigators failed to provide an adequate response, BMJ Open overlooked the issue and accepted the paper anyway without an ethical review. Ironically, the journal published it under the heading of “research” but has since defended the decision to categorize it as “service evaluation.”

You can read more details about the documented problems with these studies, and letters of concern sent to the journal editors by more than a dozen scientists, clinicians and other experts, in the following Virology Blog posts:

Trial By Error: The SMILE Trial’s Undisclosed Outcome-Swapping
Trial By Error: A Letter to Archives of Disease in Childhood
Trial By Error: No Ethical Review of Crawley School Absence Study
Trial By Error: A Letter to BMJ Open

I have cc’d Fiona Godlee, editor-in-chief of The BMJ and editorial director of BMJ Company, on this e-mail. Perhaps she can explain why these journals have not yet acknowledged the obvious: Neither paper should have been accepted for publication in the first place.

As I mentioned, my understanding is that looking back at historical mistakes is not within your charge. But I wanted to alert you to these studies so you can design the system to prevent abuses of the REC process from occurring in the future. When journals refuse to acknowledge obvious problems such as those documented in the two studies of concern here, they should face repercussions for placing reputational and institutional concerns above the public interest.

Thank you for your consideration of this matter.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley

**********

Letter to Professor Dorothy Bishop

Dear Professor Bishop—

I am a research fellow in public health and journalism at the Center for Global Public Health at the University of California, Berkeley. For the last couple of years, I have been investigating the PACE trial and the associated body of research into GET and CBT as treatments for the illness variously known as CFS, ME, CFS/ME, and ME/CFS.

Last September, Archives of Disease in Childhood published a study called “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.” As part of the Science Media Centre’s promotion of the study, you provided a statement that offered cautious support for the findings and was widely disseminated to the media. While you expressed concerns about the Lightning Process itself, you noted that the reported benefits “do seem solid” and that the statistical analysis appeared to adhere to “a high standard.”

In December, thanks to some clues provided by sharp-eyed patients and advocates, I documented on Virology Blog that the Lightning Process study violated basic methodological and scientific principles. Here is a link to that post:

Trial By Error: The SMILE Trial’s Undisclosed Outcome-Swapping

Those violated principles included BMJ’s own strict policy of not publishing trials in which participants were recruited before registration. In the Lightning Process study, 56 out of 100 children were recruited prior to trial registration as part of a feasibility study. Moreover, primary and secondary outcome measures were swapped based on data provided by these feasibility study participants–an effective method of biasing the final results. In a troubling omission, the published paper failed to mention these salient and important details.

Along with 20 other experts, I alerted Archives of Disease in Childhood to these anomalies on January 30. Here is a link to that letter:

Trial by Error: A Letter to Archives of Disease in Childhood

Although it has now been more than four months, Archives of Disease in Childhood has not yet responded or taken corrective action. Nor has it agreed to set a date by which it will do so. The delay is inexplicable: This was a high-profile trial likely to impact treatments provided to children, and checking the relevant documents would take no more than a few minutes. If the concerns were based on a misunderstanding of the facts, the journal could have cleared up the matter immediately. Under the circumstances, the inevitable assumption is that BMJ’s ethical compass is not currently in working order, at least when it comes to this domain of inquiry.

In deciding to highlight this study with a briefing, a roundup of expert opinion and a statistical analysis, the SMC similarly failed to conduct appropriate oversight. Presumably no one at the organization read the supporting documents and realized that more than half the study’s participants were recruited before the trial registration and that the investigators swapped their outcomes based on results from those early enrollees. But the SMC has been alerted to the issues and, like BMJ, has so far failed to act.

The SMC publicity push led to major media interest. These reports understandably touted the positive results for the revised primary outcome of self-reported physical function but ignored the null results for the original primary outcome—school attendance at six months. Had primary and secondary outcome measures not been swapped after more than half the participants had provided data, the news coverage would obviously have been different. As it is, Lightning Process practitioners now have bragging rights that this pseudo-scientific intervention incorporating neurolinguistics programming is effective in treating this serious illness in kids.

Here are links to a post about the SMC’s strategy for promoting the study and one about my recent letter to the SMC concerning the issue:

Trial By Error: My Questions for the Science Media Centre

Trial By Error: My Letter to the Science Media Centre about BMJ Study

BMJ and the SMC need to stop stonewalling and address the situation. Given the role of your own comments in promoting this deeply flawed study, you could help reverse some of the damage and champion the cause of ethical scientific conduct by reviewing the trial registration and other relevant documentation, acknowledging the study’s disqualifying missteps, and revising your previous statement, which was presumably based on incomplete information.

I have cc’d Fiona Godlee, editorial director at BMJ, and Edward Sykes, the SMC’s head of mental health and neuroscience, so they can respond directly to these concerns, should they choose to.

Thanks for your prompt attention to this matter. –

Best—David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley