By David Tuller, DrPH
Jonathan Sterne, a professor of medical statistics and epidemiology at Bristol University, is the corresponding author of Cochrane’s revised “risk of bias” tool, which BMJ published online on August 28th. Whatever the merits or defects of this revision, Professor Sterne’s involvement as the first of more than two dozen authors has to be considered unfortunate.
(Added: On the defect side, this revision makes it easier for unblinded studies relying on self-reported outcomes to be assessed as having “low risk” of bias. In fact, studies with this design are at high risk of bias. Yet they are favored by members of the CBT/GET ideological brigades. More on that in a later post.)
Professor Sterne has co-authored 11 papers with his Bristol colleague, Professor Esther Crawley, according to a search on PubMed. I have documented that at least two of these papers breached core principles of scientific inquiry–the Lightning Process trial published two years ago by Archives of Disease in Childhood, and a 2011 study in BMJ Open that exempted itself from ethical review on specious grounds. Neither article would likely have been published had the journals involved conducted sufficiently rigorous peer review to identify the disqualifying flaws.
[I should disclose here that Professor Crawley has publicly accused me of libel, defamation, and engaging in behavior warranting police intervention. The Bristol vice-chancellor’s office has made at least three complaints about me to the Berkeley chancellor but has provided no apparent evidence of wrong-doing or inaccuracy. I note that I remain an academic appointee in good standing at Berkeley’s Center for Global Public Health, and I continue my investigation with the support of my department and academic colleagues.]
In the conduct and reporting of the Lightning Process trial, Professor Crawley, Professor Sterne and their co-authors recruited more than half of their sample before registration, swapped primary and secondary outcome measures after gathering data from these early participants, and then failed to mention these salient details in the published paper, as I documented shortly after it was published. Last month, Archives of Disease in Childhood, a BMJ journal, posted a correction/clarification of more than 2800 words, along with a lengthy editor’s note seeking to justify the decision to re-publish–rather than retract–the primary results.
The journal’s action downplayed the significance of the Bristol team’s misbehavior and ignored the extent to which Professor Crawley, Professor Sterne and their co-authors had biased their reported findings with their outcome-swapping. This deficient editorial response dismayed many who care about research integrity and the well-being of children. That’s why dozens of scientists, academics and other experts, including several of my Berkeley colleagues, signed Virology Blog’s recent open letter to Dr Fiona Godlee, BMJ’s editorial director. The letter called the decision to republish the original Lightning Process trial findings “scientifically and ethically indefensible.”
As for BMJ Open’s school absence study, the three authors–including Professor Crawley and Professor Sterne–exempted it from ethical review on the false grounds that it qualified as “service evaluation.” (Virology Blog documented this two years ago, thanks to a reader’s tip). Service evaluation, which is exempt from ethical review, mostly relies on anonymous databases to evaluate the provision of specific services to clients or patients. Under widely accepted criteria, if a study features a hypothesis, generalizable conclusions, and/or non-anonymous data collection through interviews or other means, it qualifies as “research” and cannot be exempted from ethical review as “service evaluation.”
Experienced researchers, including Professor Sterne and Professor Crawley, should be well aware of this distinction between “research” that requires ethical review and “service evaluation” exempt from it. Since the school absence study included a hypothesis, generalizable conclusions, and in-person interviews conducted by Professor Crawley, it did not meet any standard criteria for service evaluation. It is therefore hard to understand why the authors, including Professor Crawley and Professor Sterne, would choose to exempt this study from ethical review and publish it as service evaluation. (It is also hard to understand why BMJ Open editors would stonewall, make untrue claims, cast doubts on my reporting, and deny the obvious when confronted with the incontrovertible facts.)
Bristol University is currently conducting an investigation into the school absence study and a number of others that were similarly exempted from ethical review as service evaluation. This investigation was initiated based on concerns I raised with the Health Research Authority, an arm of the National Health Service. I hope to hear something about the results of this investigation in the near future.
It is indisputable that studies involving hypotheses, generalizable conclusions, and in-person interviews with more than 100 children and/or their family members cannot reasonably be defined as service evaluation and exempted from ethical review. But as I have discovered in the last few years, investigations conducted by people enmeshed in the same systems that generated the problematic research in the first place often serve to white-wash, obfuscate or justify bad behavior rather than expose it for what it is.
In any event, until Professor Sterne acknowledges and explains his own role in the conduct and reporting of these two unpalatable studies, it is hard to take seriously any pronouncements about how to assess “risk of bias.” Here’s what I’d like to ask, as a start:
Why did Professor Sterne sign off on the Lightning Process trial paper, in which critical information about retrospective registration and outcome-swapping were withheld from the public version of events? Is Professor Sterne aware that prospective registration and pre-designated outcomes are considered essential in reducing the “risk of bias”? Regarding the school absence paper, does Professor Sterne really believe that a study featuring a hypothesis, generalizable conclusions, and in-person interviews conducted by the lead investigator can reasonably be defined as “service evaluation” and appropriately exempted from ethical review?
These are legitimate questions not only for Professor Sterne but for Cochrane, BMJ, and Professor Sterne’s many co-authors on this “risk of bias” revision. Given that both the Lightning Process and school absence studies are fraught with the kinds of methodological and ethical irregularities that should be obvious to first-year epidemiology students, it is unclear how Professor Sterne can currently serve as a credible authority on anything.
Comments
9 responses to “Lead Author of Cochrane’s New Bias Guideline is LP Study Co-Author”
It was also pointed out that this guideline considers randomization sufficient to attain a low risk of bias status in clinical trials. Is this a misunderstanding? It is very hard to believe that anyone could seriously propose this.
Only amateurs play the refs.
Real pros become the refs and rewrite the whole rulebook in order to retroactively clear all past wrongdoings and guarantee future success.
Looking forward to the Cochrane review by Miss Cleo showing astrological enema treatment for tinnitus is better than nothing and of low bias. And a churn of studies surprisingly showing peanut butter as a promising treatment for hair loss based on questionnaires from participants showing they feel like they have more hair, sponsored by Hair club for men and Kraft. No bias. No bias. You’re the bias.
Throwing the scientific method and centuries of progress out the window. What could possibly go wrong?
What?!
This is the same bloke who didn’t realize that retrospective registration and outcome switching was not allowed? This is insane.
No amount of randomisation impacts on the high risk of bias when using subjective measures in unblinded trials.
No matter how good or sophisticated your statistical analysis if your data is subject to bias your results are worthless.
This is nothing but a cynical attempt to lend spurious credibility to bad research.
If the House of Commons Science and Technology Committee doesn’t urgently investigate this screamingly obvious example of a lack of UK research integrity then, in my opinion, they’re not fit for purpose. Patients can’t have any confidence in the research system and the UK medical establishment when a BMJ owned journal refuses to retract, and allows as an exception, a study that admits to not complying with ICMJE and BMJ guidance on trial registration, especially when the lead author of Cochrane’s ‘Risk of Bias’ tool was co-author of the study so should have been fully aware of that guidance and when we know he advised on the trial methodology for the feasibility study. This stinks to high heaven, surely the Committee must acknowledge that. I’d suggest that they cross examine all 4 professors involved with the LP trial – Sterne, Crawley, Montgomery and Hollingworth -, Fiona Godlee, Nick Brown, Bristol University’s Head of Research Governance and representatives from the HRA who allowed this study on children in the first place, knowing that a significant proportion of ME/CFS patients had reported getting much worse as a result of trying the Lightning Process. If Norman Lamb is still Chair when Parliament gets back to work then he should set this up as soon as possible. The public needs protecting and we are watching. Perhaps someone should start a petition?
Oh, and of course they need to invite David to give evidence, to make sure the detail is right.
“Perhaps someone should start a petition?” As someone who has indulged in the petition strategy to create a ‘data set’ of sorts in order to affect change, but is also mindful that doing so needs to be carefully considered (so many people get ‘petition fatigue’ and so often petitions are badly worded or lack proper information making their results uninterpretable), I really do think that the time is right for a petition to directly attack what is happening here. Couch Turnip is right… the House of Commons Science and Technology Committee MUST urgently seek to investigate these matters. I concur with CT wrt the personnel who should be compelled to answer questions from that committee too. I am aware that creating a well worded petition isn’t the easiest job in the world. If someone with more energy is ‘game’ I think the time is nigh.
These documents may be of interest:
July 2018 – “Quarter of universities not reporting on potential malpractice” [https://www.parliament.uk/business/committees/committees-a-z/commons-select/science-and-technology-committee/news-parliament-2017/research-integrity-report-publication-17-19/]
Also, in this document – “House of Commons Science and Technology Committee Research Integrity Sixth Report of Session 2017–19” dated 26th June 2018 – on page 45 we learn how the Russell Group and Universities UK respectively argued against a UK regulatory body to help ensure research compliance and against “policing researchers via a compliance-led policy “.
It seems that UKRI (https://www.ukri.org/about-us/) are now pressing ahead with forming its own Research Integrity Committee instead of an ‘independent body’ being established [https://www.parliament.uk/documents/commons-committees/science-technology/Correspondence/190625-Letter-to-Chair-from-Chris-Skidmore-on-Research-Integrity.pdf]. I’d interpret this as the research institutions, universities and industry making sure they’re in control. They plan to have this committee set up and operating by Summer 2020.
What faith can the public possibly have in this self-regulatory approach when a Russell group University – Bristol University – in conjunction with the HRA , BMJ and ADC seems to have got away with all sorts with regard to the Lightning Process and School Absence studies for ME/CFS?
“It seems that UKRI (https://www.ukri.org/about-us/) are now pressing ahead with forming its own Research Integrity Committee instead of an ‘independent body’ ”
More self regulation.. a very worrying move indeed.