By David Tuller, DrPH
Last November 12, I published a post called “How Bristol Investigators Avoided Ethical Review.” That post addressed the series of eleven studies conducted by Bristol investigators and exempted from ethical review on questionable grounds. The lead investigator of these studies was Professor Esther Crawley, Bristol’s methodologically challenged pediatrician, whose work was the subject of a report released last week by the National Health Services’s Health Research Authority.
That investigation confirmed my argument that these studies all qualified as “research” and not “service evaluation.” Professor Crawley has now been directed to make corrections to the ethics statements in all eleven studies. While this remedy falls significantly short of what I believe is appropriate, it would nonetheless be a humbling and perhaps humiliating task for any serious investigator to have to correct so many studies at one go. And of course this mandate comes just a few months after Archives of Disease in Childhood appended a 3000-word correction to the dung-heap known as the Lightning Process trial–also from Professor Crawley and her colleagues.
I will address my concerns about the new report on Professor Crawley’s work at another time. For now, I thought it might be useful to reprise last year’s post (published in full below). The post offers an in-depth explanation of the issues involved, including the distinction between “research” and “service evaluation.”
**********
November 12, 2018:
I have written many posts about BMJ Open’s 2011 school absence study,
which reported that school absence records could be useful in identifying children with chronic fatigue syndrome. However, for reasons not yet adequately explained, the investigators exempted the study from ethical review on the grounds that it qualified as “service evaluation.” To support the claim, they cited a 2007 letter from the regional research ethics committee (REC), which was unrelated to the study in question.
Studies involving data from human subjects are often categorized as either “service evaluation” or “research.” Those in the former group are generally supposed to evaluate the provision of health care or other services through analyses of anonymous data; they are exempt from ethical review. If a study is testing a hypothesis, produces generalizable conclusions and/or involves data collected from participants known to the investigators, it falls under the category of “research,” not “service evaluation,” and requires ethical review.
The school absence study was conducted by investigators from the University of Bristol. It featured a formal hypothesis, generalizable conclusions, and in-person collection of data from participants. It was clearly “research,” as understood by the Health Research Authority, the arm of UK the National Health Service that serves as the arbiter of research ethics. The study should never have been defined as service evaluation and published without ethical review. Yet BMJ Open has repeatedly defended the decision to publish it.
As it turns out, the 2007 REC letter has been cited to exempt at least ten other studies from ethical review as service evaluation. (In some cases, the claimed exemption was only for part of the study.) Perhaps exemption from ethical review could be justified in some instances. But given the Bristol investigators’ inappropriate decision to cite the 2007 REC letter in exempting the school absence study from ethical review, further scrutiny of these studies would seem to be warranted. (A smart source introduced me to the issues involving the 2007 REC letter.)
*********
Here’s the background:
In 2007, the North Somerset and South Bristol REC approved a request to add questionnaires to those already being used during pediatric care at the Bath CFS/ME clinical service. At the time, the clinical service was conducting assessments with several questionnaires at entry and at twelve months. The REC application sought approval for the Bath CFS/ME clinical service to add further assessments at six weeks and six months, arguing that this would be useful for measuring and improving the delivery of care.
In the documents, the title of the research project associated with the application was: “What happens to children with CFS/ME? The study of a longitudinal cohort of children who access a paediatric CFS/ME service.” After reviewing the application, the North Somerset and South Bristol REC sent a letter dated May 1, 2007. Here’s the operative phrasing from this letter: “Members [of the REC] considered this project to be service evaluation. Therefore it does not require ethical review by a NHS Research Ethics Committee or approval from the NHS R&D office.”
The 2007 REC letter referred to “this project”—i.e., the expanded schedule of pediatric assessments at the Bath CFS/ME clinical service. It did not present itself as a precedent for avoiding ethical review for future, as-yet-undescribed projects. Yet that is how members of the Bristol team of investigators seem to have used it.
Besides the school absence study, here are the other papers (in no particular order) that have cited this 2007 REC letter to claim exemption from ethical review:
https://www.ncbi.nlm.nih.gov/pubmed/19452195
http://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-11-217
http://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-11-308
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3665909/
https://www.ncbi.nlm.nih.gov/pubmed/23619200
http://bmjopen.bmj.com/content/5/10/e008830.full
https://www.ncbi.nlm.nih.gov/pubmed/27655658
https://www.ncbi.nlm.nih.gov/pubmed/19001477
https://www.ncbi.nlm.nih.gov/pubmed/26800634
https://hqlo.biomedcentral.com/articles/10.1186/s12955-018-1028-2
Some of these studies involved data from adults, so it is hard to understand why they would be considered exempt from ethical review based on an REC letter related to routine pediatric questionnaires. Five of the exempted studies were published in BMJ journals—three in Archives of Disease in Childhood and two in BMJ Open, including the school absence study. This doesn’t surprise me. For almost a year, I have been criticizing both journals for lax oversight.
In the case of Archives of Disease in Childhood, our ongoing discussion involves an unrelated study from members of the same team of investigators—a trial of the Lightning Process as a pediatric treatment for children with chronic fatigue syndrome. This trial violated BMJ’s strict policy on prospective registration but is still in the literature. In the case of BMJ Open, the issue has been the school absence study and its lack of ethical review. That study helped trigger my interest in how the same 2007 REC letter was used to exempt other research from ethical review.
**********
Besides the BMJ Open school absence paper, BMJ published four more studies that specifically cited the 2007 REC letter. None seem to meet the criteria for service evaluation of pediatric care.
The second BMJ Open paper on the list involved data from adults as well as children and included this explanation of the finding of the 2007 REC letter: “The North Somerset and South Bristol Research Ethics Committee decided that collection and analysis of CFS/ME patient data constituted service evaluation and did not require ethical review by a NHS Research Ethics Committee or approval by NHS Research and Development offices.”
This expansive reading of the 2007 REC letter would seem to go beyond the meaning and intent of the letter’s finding that “this project”—the collection of specific pediatric data for a longitudinal cohort study–qualified as “service evaluation.” Ethics declarations from other papers in the group use similarly broad terms in describing the 2007 REC letter to exempt themselves from ethical review on the grounds of service evaluation.
The three studies published in Archives of Disease in Childhood and exempted from ethical review on service evaluation grounds do not seem to have focused on the evaluation of pediatric services. Instead, they presented generalizable conclusions about children with the illness.
One of the studies concluded this, for example: “Depression is commonly comorbid with CFS/ME, much more common than in the general population, and is associated with markers of disease severity. It is important to screen for, identify and treat depression in this population.” Studies with generalizable conclusions are normally categorized as “research” and require ethical review.
It is unclear why a narrowly written 2007 REC letter about a particular pediatric research project at one clinical service seems to have been treated as something of an invitation to bypass ethical review in a range of pediatric and adult studies. Bristol University should assess whether each study in this group was or was not appropriately exempted from ethical review based on the finding from the 2007 REC letter. The university should also determine whether members of its faculty commonly use this method to exempt their studies from ethical review.
For its part, BMJ has an independent responsibility to explain why it published multiple studies that exempted themselves from ethical review on questionable grounds, all citing the same 2007 REC letter. Editors need to ask themselves some tough questions. Here are a few:
*What lapses in procedures for assessing and reviewing manuscripts might have allowed these studies to slip through? What red flags were missed?
*What is current policy about fact-checking ethics declarations, especially when they are used to exempt studies from ethical review?
*In the case of the 2011 school absence study, the abstract itself provides sufficient evidence to determine that this was “research” requiring ethical review and not “service evaluation.” Why doesn’t BMJ Open apologize for making an error in judgement and address the issue honestly?
*What steps can BMJ take to prevent such potential embarrassments in the future and to bolster public confidence in the integrity of its editorial decision-making?
Comments
3 responses to “How Bristol Investigators Avoided Ethical Review (Reprise)”
Thanks for the reminder. I noticed on the s4me website that someone had written to the HRA about the panel investigation at Bristol and the HRA replied saying that they had “accepted the recommendations of the panel” and that the case was now closed. The HRA may well think that they have closed this case and they may well refuse to engage with any further communications in relation to it but they seem to forget that their aim, as given on their website -https://www.hra.nhs.uk/about-us/ is “to make the UK a great place to do health research, TO BUILD CONFIDENCE AND PARTICIPATION IN HEALTH RESEARCH, and so improve the nation’s health.” (My capitalization). They seem rather too focussed on the first part of that by facilitating health researchers to do whatever they want (see -https://www.youtube.com/watch?v=A3-uKBng7kU at 16.22) – while ignoring the ‘building confidence and participation’ bit.
How can the public ever trust the HRA when they have accepted this panel’s findings, a panel that they laughably describe as ‘independent’? According to the report of the review, the ‘Terms of reference setting out the purposes and principles of the review were agreed in advance between the HRA, the University and the members of the panel’. Since when did a ‘defendant’ in a case, an incontrovertibly biased party, agree the terms of reference for that case? And what’s with the “It is not within the panel’s remit to determine whether there has been misconduct by any person”? What the Dickens? Of course that statement is ambiguous in itself – do they mean the panel can’t determine if there’s been any misconduct or more specifically if there’s been misconduct by a particular person? (Surely all misconduct is perpetrated by people?) But then these people seem to play with ambiguity for a pastime. As I see it, there was no strong case for the ‘prosecution’ put forward and it appears to me that evidence was withheld by Bristol University. And they made this case all about the PI with no mention of the other researchers involved, (including Bristol University professors who should have read their own papers and known better), and no apparent consideration of whether the HRA’s/REC’s decisions might have been wrong.
The public know a stitch-up when they see one. I, for one, will refuse any future participation in health research because I now believe the HRA and health research establishment to be rotten to the core. I will be advising others to so the same, not that it’ll make much difference. The medical research establishment now appears so busy circumventing ethical rules and purloining our data via ‘service evaluation’ and Big Data projects that I suspect any informed patient consent will soon be a thing of the past. (Just watch the Wessely bit on that video immediately before the Wisely bit).
Oh my! CT’s link to that particular time in the comment above is priceless! If there were ever two people in cahoots and caught in flagrante (in the ‘malfeasance in public office’ sense) it would seem to be those two! I wonder who is boss? I think the HRA response last week gives that game away doesn’t it? The tail is wagging the dog. (IMO obviously! 😉 )
I could use cliche’s about this situation ad infinitum but what’s really happening here, what has HAPPENED here, is the demolition of medical science in the UK. In an intellectual sense that’s saddening, but in a moral and ethical sense it’s appalling, in fact I’d go so far as to say it’s ‘evil’ . These people… those people in that video at 16 minutes and 22 seconds… that collusion.. (shakes head) ….
Very sick people, encompassing many different diagnoses, are being let down. I know that people will have lost their lives due to these machinations. The 30 plus years of Wessely School interference has cost people with ME their quality of life, and sometimes their actual lives… whether by suicide because they can no longer live with a disease which is so heinous but for which there are no therapies (because the collusion I have mentioned above has led to a wasteland of Junk Science when research might otherwise have found useful therapies that could at least numb the suffering to some extent) or by the progression of the disease itself (and again had the last 30 years had no input from The Wessely School and like minded dupes, then treatments that could have halted that progression might have been developed. )
I don’t why so many many people behave so badly. I ponder that, in some, ‘narcissistic behaviour’ is at play and they really care not a jot about how their actions affect others. Some are mere pawns..dupes who are too stupid to see how they are being played. Some I guess are afraid of not acquiescing to those they perceive to be pulling the strings. After all we have quite a few verifiable examples of people being attacked behind the scenes with letters to their employers… some people want to keep their jobs. Some people however do not bend when attacked in this way. Some like Tuller keep going. Thank goodness for his tenacity!
If the HRA or the ‘panel’ had any moral bone in their collective body they would have demanded retraction of all 11 of the studies using that 2007 letter because they have already agreed that Tuller’s charges are true. ie: ‘they’re guilty m’lord’. But they didn’t, did they? And what does that say?
Putting the case of the school absence study to one side – ( to anyone with any sense that should have been designated as research necessitating full REC ethical approval and oversight) – it appears that the Bristol investigation panel have rested their conclusions regarding the other studies on researchers being allowed to conduct research without ethical approval if they use anonymized data previously gathered for service evaluation purposes.
Surely that should depend on what material has been collected for the supposed ‘service evaluation’ in the first place? Service evaluation is supposed to use data gathered in the course of ROUTINE care. But Crawley and her team were collecting large amounts of data on a wide range of themes via the ‘National Outcomes Database’ or ‘NOD’ from services all around the country, and selectively using it in their research studies which averaged around one paper per year for 10/11 years. The amount of data collected increased considerably over time and NHS clinics and adult patients not mentioned in REC ref 07/Q2006/48 were added to the database. The report indicates that the REC/HRA gave permission for extra data collection points to be added under ‘service evaluation’ designation. NB They initially seemed to think this needed ethical approval so prompting the REC application, but then for some bizarre reason appear to have changed their minds. So it looks to me as if the HRA/REC has facilitated the use of extra non-routine patient data for research purposes, data that could be collected without informed consent. The Bristol panel have apparently found that the increase in the amount and range of data collected and the inclusion of adults in the database (and the subsequently published research studies) were all ok because the HRA had given Crawley and her team carte blanche for the extension of the service evaluation in that ‘January’ letter of ambiguous meaning. But should the HRA have ever given that carte blanche? And wasn’t it incumbent on Crawley and her team to go back to the HRA to seek clarification if they thought any research project of theirs might require ethical approval? Also, isn’t it inconceivable that Crawley and her colleagues didn’t know when they were collecting the data (in all but the first few years) that they intended to use it for research papers? The data collection involved continual recruitment of patient data to the database from 2004 up until a few years ago, and their production of research papers seems to have been somewhat akin to a factory production process too, that from 2007 onwards was going on at the same time. At the time that newly referred patients were being asked upfront (pre-assessment) for their data, the research team must have had more than an inkling that they would be accessing it for research purposes down the track. That, in my book, is not secondary use.
So is it the case that if a doctor wants to do a research project (that isn’t a clinical trial) using patient data then all they have to do is collect the required data from their patients for a few years without consent, claim that the data is routinely collected at their clinic anyway, and then publish research papers on the back of it to their heart’s content? Doesn’t it matter what the data is, how intrusive it is, how much there is, if it’s a burden on the patients to complete the questionnaires, if there’s no informed consent, if it’s irrelevant to the evaluation of the service being provided or if it could be used for a purpose that could undermine the patient’s situation and/or trust? On the last 2 points – what exactly has the employment and welfare benefit data in the NOD got to do with service evaluation? None of this would seem to matter to the panel investigating this case at Bristol because they don’t appear to have asked any questions about the nature of the data that was collected or whether patients were informed as to how it would be used. But more worryingly, none of this appears to matter to the HRA either, since they have accepted the panel’s findings and closed the case.