A Bit More About Bristol’s Investigation

By David Tuller, DrPH

Yesterday I reported that Bristol University, at the request of the UK Health Research Authority, is investigating a number of studies conducted by Professor Esther Crawley. The results of this investigation are expected in two months or so.

Today I can disclose that the scrutiny involves papers linked to a specific research ethics committee (REC) reference: 07/Q2006/48. That is the number of a 2007 REC opinion issued for a study titled “What happens to children with CFS/ME? The study of a longitudinal cohort of children who access a paediatric CFS/ME service. Version 2.” The investigators were seeking permission to add some more questionnaires to those already being filled out by (or about) pediatric patients at the specialized Bath clinic run by Professor Crawley.

I have also been informed that this is an “independent” investigation. While the specific meaning of “independent” in this context is not fully clear, I interpret this to indicate that it is not an internal review under the direct control of Bristol. My assumption is that it involves an assessor unaffiliated with Bristol or a panel dominated by non-Bristol experts.

The HRA’s scope is relatively narrow. The agency does not assess scientific quality and the appropriateness of this or that methodological approach; rather, in cases like this, it focuses on whether proper procedures have been followed by investigators seeking ethical approval for studies as well as by RECs assessing such applications. The agency does not consider questions that it views as falling within the purview of study funders and journal editors. Any findings it makes must be assessed with that limited scope in mind.

I did not file formal complaints with the HRA but cc’d the agency on various open letters I sent to this or that person. Most of this cc-ing involved my concerns about two Bristol studies–the school absence study published in BMJ Open in 2011, and the Lightning Process study published in Archives of Disease in Childhood, another BMJ journal, in 2017. The school absence study, as it turned out, was one of 11 papers I have identified that cited REC reference 07/Q2006/48.

The concerns I raised about the promiscuous use of REC reference 07/Q2006/48 clearly fall within the HRA’s jurisdiction. The 2007 opinion indicated that the proposed study qualified as “service evaluation” and was not “research.” That meant it was exempt from the kind of ethical review generally required for studies involving human subjects. According to official guidance, service evaluation studies are just that—evaluations of services provided. By definition, they do not include hypotheses, generalizable conclusions, or collection of data from individuals known to the investigators.

As I have previously reported, the Bristol researchers cited that same REC opinion to exempt all the other studies from ethical review as well—even though the opinion referred to “this project.” In other words, the opinion appeared to be applicable only to the specific study contemplated in 2007. Some or all of the other studies appear to have little or nothing to do with the activities described in the 2007 ethical review application that led to REC reference 07/Q2006/48. Some of the studies involved adults, so it seems self-evident—at least to me—that it would not or at least should not be permissible to exempt them from ethical review based on an REC opinion about pediatric research.

In any event, the use of a single REC opinion to exempt so many studies from ethical review is rather unusual, whatever the circumstances. It is not surprising that this pattern would raise eyebrows, once someone noticed. Perhaps the investigators thought no one would bother to check obscure REC reference numbers. If so, they didn’t count on the intrepid sleuth who first alerted me to this perplexing situation.

The school absence study has been the subject of many blog posts in this space. The paper included a formal hypothesis and generalizable conclusions. To collect data for the study, Professor Crawley met in person with a host of students and parents. Any experienced researcher knows–or should be expected to know—that a study featuring these characteristics cannot be classified as service evaluation. Any experienced journal editor should know this as well.

In accepting research that exempted itself from ethical review on apparently illegitimate grounds, editors at BMJ Open failed in their editorial oversight functions. And this disturbing failure continues, since BMJ editors, including editorial director Fiona Godlee, have stonewalled in what appears to be an effort to avoid responsibility for their poor choices.

Why should anyone care about such technicalities as violations of research ethics practices and requirements? After all, the entire area of informed consent and ethical review is fraught and controversial in many areas of research. This situation, however, involves a highly vulnerable population; Professor Crawley’s study subjects were not only minors, but minors suffering from or possibly suffering from a stigmatizing illness. In the case of the school absence study, Professor Crawley interviewed dozens of children while evading her obligation to have her approach vetted and approved by those whose job it is to protect human subjects. Depending upon what the Bristol investigation finds after reviewing more documentation than I have been able to access, this apparent breach is potentially a very serious matter.

For reasons that escape me, Dr Godlee informed me in an October e-mail that addressing some of the issues I had raised were beyond her control—and that she was sure I would understood. As I pointed out in my response to her, that argument makes little sense to me. BMJ Open made a problematic editorial decision, and then vigorously defended this misstep–in the process providing false information about the study methodology to the Committee on Publication Ethics while simultaneously accusing me of disseminating inaccurate claims and harming the journal’s reputation.

My guess–and this is purely a guess–is that Dr Godlee would like to delay resolution of the matter until the Bristol investigation is over; then she would perhaps be able to pin the whole mess directly on Bristol and Professor Crawley rather than acknowledging BMJ Open’s own negligence. Perhaps there is another logical explanation for Dr Godlee’s apparent abdication of her professional obligations, but if there is one I can’t imagine what it would be.

BMJ Open’s accusation that I have disseminated misinformation about this study is categorically false—despite my requests, no one has provided me with documentation for this untrue statement and I have not received an apology. On the other hand, I understand and accept BMJ Open’s concern that my investigation has harmed the journal’s reputation. Harming reputations has never been my goal; I’m doing what I’m doing to try to correct bad science. But I recognize that some reputational impact on journals, universities and individuals is an inevitable and unfortunate result of my efforts to discredit studies that do not measure up to core principles of scientific inquiry.

At this juncture, I find myself in a somewhat odd position for a journalist. I have a professional obligation to seek comment from those whose work I am criticizing. At the same time, I understand or at least presume that any e-mails from me distress Professor Crawley. In fact, as I was told by the reporter currently preparing an upcoming article about the controvery, Professor Crawley considers my strategies for drawing attention to the flaws in her work to be a form of “harassment.”

That charge, I gather, encompasses not only my decision to investigate Professor Crawley’s research but also the fact that I sent her many e-mails seeking comment. I dropped that approach long ago when I realized it was pointless and that she would never respond. I assume the “harassment” charge also includes the many strongly worded e-mails I sent to the CFS/ME Research Collaborative in relation to Professor Crawley’s public accusation that I was writing “libellous blogs.”

Her definition of “harassment” might also include the fact that I attended a public lecture she gave and asked a question she didn’t like. She also might consider my decision to disseminate widely my criticism of her work as another form of “harassment.” Given all that, I cannot in fact seek comment from her about this story, even though that would be the appropriate journalistic approach.

Seeking comment from Bristol is another problematic issue. Bristol has filed multiple complaints about what it apparently regards as my bad “behavior”—although no one from Bristol’s legal department has ever explained to me what this bad behavior consists of and why it warrants some form of sanction. Given the seriousness of the charges being leveled against me, I am not comfortable reaching out to the university for possible comment. (Not that anyone would be likely to comment anyway.)

So why have I spent so much time examining Professors Crawley’s work? This should not be hard to understand. Professor Crawley is the most prominent British pediatrician among the GET/CBT ideological brigades. Her research has heavily impacted the treatment of children diagnosed with this illness. If you’re a journalist and public health expert worried about the treatment of sick children and their families, an examination of her influential work is the logical place to start.

Nonetheless, I recognize that anything I do in this context is likely to generate charges—whether from Professor Crawley or other researchers in this domain–that I am engaging in “harassment.” It goes without saying that I reject that characterization of my work.

Regarding the Lightning Process study, I have documented that the trial was not in fact prospectively registered, given that more than half the subjects were recruited before trial registration. I have also documented that the investigators engaged in undisclosed outcome-swapping after more than half of the participants were recruited. Last June, Archives of Disease in Childhood posted an “editor’s note” about the paper and promised that further decisions would be made after “editorial consideration” of “clarifications” provided by the investigators. (Of course, we’re still waiting for this prolonged period of “editorial consideration” to be concluded.)

In looking at the trial, the HRA determined that the REC was informed of the changes in study design and followed the appropriate procedures in approving them. Yet the HRA has also made clear that this determination does not absolve the investigators for not disclosing their methodological choices in the paper itself. Nor does it absolve peer-reviewers, journal editors and publishers for their own actions in the matter. The HRA simply does not consider such issues. They are outside its domain of activity, and the agency therefore has nothing to say about them one way or the other.

So let’s sum up: My investigation has resulted in an independent investigation of what could be up to 11 papers from Professor Crawley’s team; five of them, including the school absence study, were published in top BMJ journals. My investigation has also forced one of those BMJ journals to alert readers to significant concerns about yet another Bristol paper. Both these developments indicate that I have at the least raised valid concerns that require further scrutiny. But I understand why it would be much easier to accuse me of “harassment” than to acknowledge the legitimacy and accuracy of my critiques.