Bristol Investigating Crawley Papers

By David Tuller, DrPH

In the last year, I have raised multiple concerns about studies from the University of Bristol with the UK Health Research Authority, among others. The HRA is the National Health Service arm that oversees approvals for research in England; the agency’s role is to ensure that research receives a favorable opinion from an ethics committee before it begins.

Yesterday, the agency confirmed to me that, after reviewing the matter, it has raised questions with University of Bristol officials about a number of these studies and has asked them to investigate the situation. The HRA is now awaiting the outcome of that investigation.

In other words, some of Professor Crawley’s work is now under formal scrutiny. This is a serious inquiry, although of course it is difficult to predict what the results will be, given the sorry history of this whole mess. I would expect this investigation to take at least a couple of months, and perhaps more. It’s not always clear to me how things work in the UK, but people can presumably make their own assessments about whether the HRA would have chosen to request an investigation if it believed everything in these studies were kosher.

This is not the first indication that my efforts to draw attention to these problematic studies are possibly bearing fruit. Last June, Archives of Disease in Childhood slapped an editor’s note on the Lighting Process study it published in 2017. (Archives of Disease in Childhood is a BMJ journal.) The investigators claimed that the study was prospectively registered even though 56 out of 100 participants were recruited before trial registration–meaning any claims that it was prospective were self-evidently untrue. It is hard to understand how experienced researchers could have committed such an elemental error.

Moreover, the investigators swapped primary and secondary outcomes after more than half the participants had been recruited, thereby biasing their reported results. Yet the paper failed to disclose these details and presented the study as if it were, in fact, prospective. BMJ, like all top medical journals, has a strict, long-standing policy that it will not consider for publication any studies that are not prospectively registered. And undisclosed outcome-swapping more than half-way through a study is unacceptable by any standard.

Here’s the relevant part of the editor’s note: “Since publication, the study has been criticised for failing to meet ICMJE and BMJ policy on trial registration and for not fully adhering to CONSORT guidance on trial reporting. The journal has been criticised for not detecting these issues during editorial and peer review. We have acknowledged these comments and reviewed our processes in relation to this paper and relating to EQUATOR guidance in general. In addition, we have received clarifications from the authors which are under editorial consideration.”

It is noteworthy that the editor’s note does not refute the concerns I have raised. The statement is most easily interpreted as an admission that these concerns are in fact valid; otherwise there would be no need for a period of “editorial consideration” to consider the “clarifications” provided by the authors. Yet the editor’s note is not linked to or visible from the paper itself. No one reading the paper would have any idea such an editor’s note existed unless they happened to click on the “responses” page. And that means they would not know that significant questions about the study’s methodological and ethical choices had been raised–and “acknowledged” by the journal itself, as the statement indicates.

It is unacceptable that this critical editor’s note has been sequestered away in a section where few people are likely to see it. It is also unacceptable that this period of “editorial consideration” has now lasted more than seven months, with no end or resolution in sight. It takes only a few minutes to check the relevant documentation and determine what happened here. Why hasn’t this matter been addressed and resolved already?

This stonewalling creates the strong impression that the editorial team is more concerned with avoiding reputational damage than protecting children’s health. The same is true of BMJ’s unwillingness to address the problems with the 2011 school absence study in BMJ Open, which was exempted from ethical review on the false grounds that it was service evaluation and not research. BMJ Open not only defended publication of the paper without ethical review but, in the process of doing so, made false claims about the paper’s methodology while simultaneously accusing me of disseminating misinformation.

**********

Fiona Godlee, BMJ’s editorial director, is aware that the situation is problematic and needs to be dealt with. She suggested as much in a note she sent me in October in response to various open letters I had sent her and other BMJ editors. Again, it is noteworthy that she mounted no defense or refutation of the points I have raised about studies published in her journals.

I haven’t mentioned this note previously because, for one reason, Dr Godlee indicated that she was anxious to get to the bottom of things. So I thought I might give her the benefit of the doubt. Yet I have seen no evidence that BMJ has taken further steps since then. So there’s no particular reason not to share our exchange at this point:

Here’s what Dr Godlee wrote on October 12:

Dear David, I am conscious that I have not directly responded to your recent emails. I wanted to let you know that we are working closely with…the HRA…and are doing our best to resolve the situation. There are matters beyond our immediate control that make this challenging as I am sure you understand…You were concerned that the notice on the ADC [Archives of Disease in Childhood] paper was not prominent enough. We are looking into whether this can be made more prominent–our systems don’t make this entirely easy! Please be assured that we are actively working on the various aspects of this case and will do all we can to reach a timely resolution. We will also do our best to keep you informed of progress. All best wishes, Fiona Godlee

**********

I answered Dr Godlee’s note a few days later. (I have deleted some passages from my response.) Here’s what I wrote:

Dear Fiona—

Thank you for your note. I very much appreciate hearing from you directly. I apologize in advance for this long message, but I have been trying to get BMJ to address these issues for some time and I therefore have a few things to explain…I’m pleased that things seem to be progressing, given the troubling nature of the methodological and ethical violations that I have documented.

I have frankly been rather surprised at the way the editors involved with both the school absence and Lightning Process studies have responded to date. Since last fall, I have written more than 20 blog posts about these two studies. (Here is a link to all my Virology Blog posts about ME/CFS, including those: https://trialbyerror.org/archives/.) No one should have to invest this amount of time and energy to convince reputable publications that they need to address such obvious irregularities. I’m afraid that this protracted and wearying process has created the impression that your editorial team has been more concerned about protecting reputations than about the health and wellbeing of vulnerable children. If BMJ’s attitude toward this mess is now shifting, that is certainly a positive development.

Since I am neither an epidemiologist nor a statistician but an investigative reporter with a public health doctorate from Berkeley, I have vetted my concerns about PACE and the other studies I have criticized with colleagues at the University of California, Columbia, Harvard and elsewhere. As you can see from the credentials of the experts who signed the various open letters to The Lancet and Psychological Medicine as well as Archives of Disease in Childhood and BMJ Open, eminent researchers and scientists in the US and elsewhere are genuinely shocked at the deficiencies of all these studies. From our perspective, the situation is a real-life version of “the emperor has no clothes,” and that description would seem to apply to much if not most of those occupying high-level positions in the UK medical and academic establishments.

[paragraph deleted]

The main reason these matters need to be dealt with as soon as possible is that information about the Lightning Process and school absence studies has been widely disseminated, at least in part because both papers were promoted by the good folks at the Science Media Centre. This research has the potential to influence policy, if it hasn’t already. That means children have been or are being impacted by BMJ studies that should never have been published in the first place.

[Paragraph deleted]

Since you have now opened up a dialogue with me, I should let you know—in the interests of transparency–that I am working on more open letters to you…The first is a request for an apology from BMJ Open for its actions in response to inquiries about the school absence study. In particular, I am owed an apology for the journal’s decision to call my reporting “inaccurate” and “misrepresentative” in its statement to COPE—not to mention the fact that this COPE statement included the untrue claim that the data used in the study were collected anonymously. Since anyone who read the study would immediately know that children and their families were interviewed in person, I am curious why BMJ Open editors thought the COPE statement was an appropriate description of what happened. (More on this issue below.)

Now to the points you have raised in your letter. You suggest that I am concerned that the editor’s note in Archives of Disease in Childhood is not “prominent enough.” That is not exactly so. My concern is that it is not “prominent” at all. This editor’s note has been posted where no one is likely see it unless they are looking for it. Furthermore, I am confused by the statement that BMJ’s systems “don’t make this entirely easy.” Why would that be? When BMJ journals need to issue a correction, isn’t it appended directly to the article in question? If so, why is that possible to do with a correction but somehow complicated to do with an editor’s note? (In fact, my main problem with the editor’s note is that Dr Brown chose to post it rather than retract the study as soon as he had confirmed the facts–more on that below.)

In addition, I am afraid I do not understand why BMJ feels that its actions are constrained by “matters beyond [y]our immediate control.” This argument does not seem to apply to either of the studies in question.

Regarding the school absence paper…I do not see why BMJ Open should need more time to acknowledge that it failed in its basic oversight function by publishing the school absence study without ethical review….

Regarding the Lightning Process study, the editor’s note makes clear that Archives of Disease in Childhood has received a response from the investigators and that the matter is now under “editorial consideration.” The editor’s note was published four months ago. How much “editorial consideration” is needed to determine that this paper violated core scientific principles? After all, the company’s own policy states that “in accordance with the ICMJE Recommendations, BMJ will not consider reports of clinical trials unless they were registered prospectively before recruitment of any participants.” The resolution of this issue does not therefore seem to be dependent on factors beyond your immediate control. BMJ Open needs no further information to take action, unless the journal has for some reason outsourced its editorial decision-making to other parties.

Around the time I raised these questions about the Lightning Process study, BMJ Open published a paper on the importance of requiring prospective registration for all trials being considered for publication. To highlight this point, the former editor-in-chief of BMJ Open tweeted that “prospective registration of #clinicaltrials isn’t red tape: it’s done for sound ethical and scientific reasons.” Dr Groves is right, of course. One corollary of that principle is that any cases in which the opposite occurred should be addressed immediately for the same “sound ethical and scientific reasons.” Moreover, the Lightning Process investigators omitted crucial details from the published paper, which was written up as a prospective trial and linked to an ISRCTN registration that falsely categorized the trial as prospective…

[Three paragraphs deleted]

Under the circumstances, it is hard to understand why Archives of Disease in Childhood would consider any resolution but retraction—along with a full accounting of what happened and why the journal failed to notice these problems. In fact, retraction is the only solution that will allow Archives of Disease in Childhood to maintain that it has adhered to BMJ’s unequivocal policy about prospective registration. As I suggested in my most recent open letter to Dr Brown, I hope BMJ now understands the importance of asking editors and peer reviewers to read not just drafts of papers but supporting documentation—trial registration, research ethics committee applications, and so on. Otherwise BMJ journals will likely experience more such embarrassing episodes.

[Much of the following information, about my request for an apology, was also part of an open letter to Dr Godlee on this issue, so regular readers might find these passages familiar.]

Now I’ll address my request for an apology related to BMJ Open’s defense of the school absence study—the subject of my next open letter to you. Since last year, BMJ Open has repeatedly obfuscated the issues and denied any problems. This happened first when a reader contacted the journal with valid complaints. After the journal brushed her off, she contacted me. I quickly verified that the study was indeed research that required ethical review. After all, the paper included a formal hypothesis, the title itself expressed a generalizable conclusion, and the lead investigator personally interviewed participants. These features are all well-known hallmarks of research, not service evaluation.

I then contacted Dr Groves. It seemed from the start that she hoped the whole matter would go away or that I would lose interest or…well, I don’t really know what she thought. She stated in correspondence with me that BMJ Open itself had reviewed the matter and had apparently satisfied itself that the study was appropriately categorized as service evaluation—despite the fact that it was published under the heading of “research.” I noted in response that this internal BMJ Open review could not have been very thorough. Any editor who took ten minutes to read the study should have known immediately that it should never have been exempted from ethical review.

In something of a scholarly version of Rashomon, the journal then presented its own misleading account to COPE. In response, COPE suggested that BMJ Open should ascertain how the data were collected to determine whether the study could qualify as service evaluation. At that point, BMJ Open apparently chose to close the COPE case without further action. In a subsequent letter to me, Dr Groves cited the COPE statement as evidence that BMJ Open had taken the issue seriously—without mentioning the journal’s apparent failure to follow COPE’s advice.

In February, Professor Vincent Racaniello, a Columbia microbiologist and host of Virology Blog, sent an open letter to BMJ Open, signed by multiple colleagues from Berkeley and elsewhere, to draw attention to the journal’s negligence. In her response, Dr Groves suggested that we submit a letter for publication. Professor Racaniello and I saw no reason to do so, since it seemed likely that the journal would publish our letter, along with a letter from the Bristol investigators, and then wash its hands of the matter. This approach would have allowed BMJ Open to present itself as a neutral arbiter of an unresolved dispute. But of course the facts are indisputable. BMJ Open published a research paper that exempted itself from ethical review on false grounds, and the journal has continued to defend this indefensible decision.

In her final communication, Dr Groves explained that a letter from us would allow BMJ Open to request a detailed response to our concerns from the investigators. We did not understand why a publishable letter from us was needed in order for the journal to seek adequate explanations from the investigators. Naturally, we declined to participate in BMJ Open’s apparent effort to extricate itself from this thorny situation while evading responsibility for publishing the paper in the first place.

I only appealed to…the Health Research Authority when it became clear that BMJ Open had no interest in admitting fault and that Archives of Disease in Childhood was taking an inordinately long time to investigate the Lightning Process study. I am concerned that the school absence study could lead to further efforts by investigators at Bristol and elsewhere to interview children and their parents without ethical review. But I am especially worried that the continuing delay in addressing the issues with the Archives of Disease in Childhood study could have potentially harmful consequences for the very ill children whose desperate parents have sought out Lightning Process practitioners based on BMJ’s seal-of-approval. As I have repeatedly pointed out, the Lightning Process founder has claimed that he has been able to “step into other people’s bodies over the years to assist them in their healing with amazing results.” Now BMJ has endowed his current magical modality—a goulash of neuro-linguistic programming, osteopathy, self-affirmations, and life-coaching–with an aura of scientific credibility.

As I wrote in one of my open letters, enough is enough. It is time for BMJ to take stock, acknowledge its mistakes, and do what it can to make things right. I apologize again for the length of this letter, but these things needed to be said. I look forward to further communications from you about BMJ’s progress in coming to a timely resolution of these matters. (I have just arrived in London today and will be in England until October 30. I would be happy to meet with you at your convenience if that would be helpful.)

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley