by David Tuller, DrPH
David Tuller is academic coordinator of the concurrent masters degree program in public health and journalism at the University of California, Berkeley.
‘The PACE trial is a fraud.’ Ever since Virology Blog posted my 14,000-essord investigation of the PACE trial last October, I’ve wanted to write that sentence. (I should point out that Dr. Racaniello has already called the PACE trial a “sham,” and I’ve already referred to it as “doggie-poo.” I’m not sure that “fraud” is any worse. Whatever word you use, the trial stinks.)
Let me be clear: I don’t mean “fraud” in the legal sense—I’m not a lawyer–but in the sense that it’s a deceptive and morally bankrupt piece of research. The investigators made dramatic changes from the methodology they outlined in their protocol, which allowed them to report purported “results” that were much, much better than those they would have been able to claim under their originally planned methods. Then they reported only the better-looking “results,” with no sensitivity analyses to analyze the impact of the changes—the standard statistical approach in such circumstances.
This is simply not allowed in science. It means the reported benefits for cognitive behavior therapy and graded exercise therapy were largely illusory–an artifact of the huge shifts in outcome assessments the authors introduced mid-trial. (That’s putting aside all the other flaws, like juicing up responses with a mid-trial newsletter promoting the interventions under investigation, failing to obtain legitimate informed consent from the participants, etc.)
That PACE suffered from serious methodological deficiencies should have been obvious to anyone who read the studies. That includes the reviewers for The Lancet, which published the PACE results for “improvement” in 2011 after what editor Richard Horton has called “endless rounds of peer-review,” and the journal Psychological Medicine, which published results for “recovery” in 2013. Certainly the deficiencies should have been obvious to anyone who read the trenchant letters and commentaries that patients routinely published in response to the egregious errors committed by the PACE team. Even so, the entire U.K. medical, academic and public health establishments refused to acknowledge what was right before their eyes, finding it easier instead to brand patients as unstable, anti-science, and possibly dangerous.
Thanks to the efforts of the incredible Alem Matthees, a patient in Perth, Australia, the U.K.’s First-Tier Tribunal last month ordered the liberation of the PACE trial data he’d requested under a freedom-of-information request. (The brief he wrote for the April hearing, outlining the case against PACE in great detail, was a masterpiece.) Instead of appealing, Queen Mary University of London, the home institution of lead PACE investigator Peter White, made the right decision. On Friday, September 9, the university announced its intention to comply with the tribunal ruling, and sent the data file to Mr. Matthees. The university has a short window of time before it has to release the data publicly.
I’m guessing that QMUL forced the PACE team’s hand by refusing to allow an appeal of the tribunal decision. I doubt that Dr. White and his colleagues would ever have given up their data willingly, especially now that I’ve seen the actual results. Perhaps administrators had finally tired of the PACE shenanigans, recognized that the study was not worth defending, and understood that continuing to fight would further harm QMUL’s reputation. It must be clear to the university now that its own reputational interests diverge sharply from those of Dr. White and the PACE team. I predict that the split will become more apparent as the trial’s reputation and credibility crumble; I don’t expect QMUL spokespeople to be out there vigorously defending the unacceptable conduct of the PACE investigators.
Last weekend, several smart, savvy patients helped Mr. Matthees analyze the newly available data, in collaboration with two well-known academic statisticians, Bruce Levin from Columbia and Philip Stark from Berkeley. Yesterday, Virology Blog published the group’s findings of the single-digit, non-statistically significant “recovery” rates the trial would have been able to report had the investigators adhered to the methods they outlined in the protocol. That’s a remarkable drop from the original Psychological Medicine paper, which claimed that 22 percent of those in the favored intervention groups achieved “recovery,” compared to seven percent for the non-therapy group.
Now it’s clear: The PACE authors themselves are the anti-science faction. They tortured their data and ended up producing sexier results. Then they claimed they couldn’t share their data because of alleged worries about patient confidentiality and sociopathic anti-PACE vigilantes. The court dismissed these arguments as baseless, in scathing terms. (It should be noted that their ethical concerns for patients did not extend to complying with a critical promise they made in their protocol—to tell prospective participants about “any possible conflicts of interest” in obtaining informed consent. Given this omission, they have no legitimate informed consent for any of their 641 participants and therefore should not be allowed to publish any of their data at all.)
The day before QMUL released the imprisoned data to Mr. Matthees, the PACE authors themselves posted a pre-emptive re-analysis of results for the two primary outcomes of physical function and fatigue, according to the protocol methods. In the Lancet paper, they had revised and weakened their own definition of what constituted “improvement.” With this revised definition, they could report in The Lancetthat approximately 60 % in the cognitive behavior and graded exercise therapy arms “improved” to a clinically significant degree on both fatigue and physical function.
The re-analysis the PACE authors posted last week sought to put the best possible face on the very poor data they were required to release. Yet patients examining the new numbers quickly noted that, under the more stringent definition of “improvement” outlined in the protocol, only about 20 percent in the two groups could be called “overall improvers.”. Solely by introducing a more relaxed definition of “improvement,” the PACE team—enabled by The Lancet’s negligence and an apparently inadequate “endless” review process–was able to triple the trial’s reported success rate..
So now it’s time to ask what happens to the papers already published. The editors have made their feelings clear. I have written multiple e-mails to Lancet editor Richard Horton since I first contacted him about my PACE investigation, almost a year before it ran. He never responded until September 9, the day QMUL liberated the PACE data. Given that the PACE authors’ own analysis showed that the new data showed significantly less impressive results than those published in The Lancet, I sent Dr. Horton a short e-mail asking when we could expect some sort of addendum or correction to the 2011 paper. He responded curtly: “Mr. Tuller–We have no such plans.”
The editors of Psychological Medicine are Kenneth Kendler of Virginia Commonwealth University and Robin Murray of Kings College London. After I wrote to the journal last December, pointing out the problems, I received the following from Dr. Murray, whose home base is KCL’s Department of Psychosis Studies: “Obviously the best way of addressing the truth or otherwise of the findings is to attempt to replicate them. I would therefore like to encourage you to initiate an attempted replication of the study. This would be the best way for you to contribute to the debate…Should you do this, then Psychological Medicine will be most interested in the findings either positive or negative.”
This was not an appropriate response. I told Dr. Murray it was “disgraceful,” given that the paper was so obviously flawed. This week, I wrote again to Dr. Murray and Dr. Kendler, asking if they now planned to deal with the paper’s problems, given the re-analysis by Matthees et al. In response, Dr. Murray suggested that I submit a re-analysis, based on the released data, and Psychological Medicine would be happy to consider it. “We would, of course, send it out to referees for scientific scrutiny in the same manner as we did for the original paper,” he wrote.
I explained that it was his and the journal’s responsibility to address the problems, whether or not anyone submitted a re-analysis. I also noted that I could not improve on the Matthees re-analysis, which completed rebutted the results reported in Psychological Medicine’s paper. I urged Dr. Murray to contact either Dr. Racaniello or Mr. Matthees to discuss republishing it, if he truly wished to contribute to the debate. Finally, I noted that the peer-reviewers for the original paper had okayed a study in which participants could be disabled and recovered simultaneously, so I wasn’t sure if the journal’s assessment process could be trusted.
(By the way, Kings College London, where Dr. Murray is based, is also the home institution of PACE investigator Trudie Chalder as well as Simon Wessely, a close colleague of the PACE authors and president of the Royal College of Psychiatrists*. That could explain Dr. Murray’s inability or reluctance to acknowledge that the “recovery” paper his journal peer-reviewed and published is meaningless.)
Earlier today, the PACE authors posted a blog on The BMJ site, their latest effort to salvage their damaged reputations. They make no mention of their massive research errors and focus only on their supposed fears that releasing even anonymous data will frighten away future research participants. They have provided no evidence to back up this unfounded claim, and the tribunal flatly rejected it. They also state that only researchers who present “pre-specified” analysis plans should be able to obtain trial data. This is laughable, since Dr. White and his colleagues abandoned their own pre-specified analyses in favor of analyses they decided they preferred much later on, long after the trial started.
They have continued to refer to their reported analyses, deceptively, as “pre-specified,” even though these methods were revised mid-trial. The following point has been stated many times before, but bears repeating: In an open label trial like PACE, researchers are likely to know very well what the outcome trends are before they review any actual data. So the PACE team’s claim that the changes they made were “pre-specified” because they were made before reviewing outcome data is specious. I have tried to ask them about this issue multiple times, and have never received an answer.
Dr. White, his colleagues, and their defenders don’t yet seem to grasp that the intellectual construct they invented and came to believe in—the PACE paradigm or the PACE enterprise or the PACE cult, have your pick—is in a state of collapse. They are used to saying whatever they want about patients—Internet Abuse! Knife-wielding! Death threats!!–and having it be believed. In responding to legitimate concerns and questions, they have covered up their abuse of the scientific process by providing non-answers, evasions and misrepresentations—the academic publishing equivalent of “the dog ate my homework.” Amazingly, journal editors, health officials, reporters and others have accepted these non-responsive responses as reasonable and sufficient. I do not.
Now their work is finally being scrutinized the way it should have been by peer reviewers before this damaging research was ever published in the first place. The fallout is not going to be pretty. If nothing else, they have provided a great gift to academia with their $8 million** disaster—for years to come, graduate students in the U.S., the U.K. and elsewhere will be dissecting PACE as a classic case study of bad research and mass delusion.
*Correction: The original version of the post mistakenly called the organization the Royal Society of Psychiatrists.
**Correction: The original version of the post stated that PACE cost $8 million, not $6.4 million. In fact, PACE cost five million pounds, so the cost in dollars depends on the exchange rate used. The $8 million figure is based on the exchange rate from last October, when Virology Blog published my PACE investigation. But the pound has fallen since the Brexit vote in June, so the cost in dollars at the current exchange rate is lower.
Comments
22 responses to “The Real Data”
🙂 Best quote: “. . . they have covered up their abuse of the scientific process by providing non-answers, evasions and misrepresentations—the academic publishing equivalent of “the dog ate my homework.” “
As a currently active clinical trials statistician, working frequently with all phases of clinical trial, and blinded or unblinded, and especially planning and analyzing double blind randomized clinical trials in the pharmaceutical industry, it is mind boggling that the PACE investigators made changes to the planned analysis in an open label study. We view as potentially “tainting” a -blinded- study to change a primary endpoint or the planned analysis. By the ICH “E3” (international conference on harmonization) guidelines are required to clearly state any changes and the reasons why any changes were made to the planned analysis. I have on one occasion had to make the argument to a narrow minded client that —- the definition and preparation of a valid statistical analysis was implicit in the informed consent and in submissions to the IRB and prior to database lock and regardless whether data was blinded or not and before or viewing any data (blinded or not) that there is an ethical obligation to the patients to prepare the most sound statistical analysis possible – and that failing to do so it is then unethical to ask patients to participate in the trial. As a member of the American Statistical Association, we have published our ethical guidelines https://www.amstat.org/asa/files/pdfs/EthicalGuidelines.pdf
My wife has had ME for around 10 years, but somehow the PACE trial, and its significance, has passed us by until recently. As a layperson, I cosily believed that medical trials were always undertaken with best intent, competence and integrity. Whenever I heard the Lancet mentioned in the news, I would feel a certain reverence for what I believed to be THE premiere medical journal, upholding and promoting all the best ideals of the profession it represents. Then, last month, I started reading about the PACE trial. I am not especially naive, but we all have our blind spots I guess.
Although I do not fully understand the medical and mathematical detail of the PACE trial, the excellent work by yourself and others means I can digest more than enough to see what a breathtakingly poor piece of science the trial was. It feels more like vandalism given the damage it has done – all at public expense, both financial and in terms of trust.
At the most lenient end of the spectrum, the PACE trial was astonishingly incompetent. But the follow-up behaviour during the last 5 years suggests wilful deception and obfuscation; a considerable escalation beyond “mere” incompetence. Instead of striving to improve ME sufferers’ lives, the motivations seem far more self-centred, almost narcissistic.
I am also concerned about motivations for some of the funding of the PACE trial.
I really do hope now, that all aspects of any wrong doing in and around the PACE trial will get thoroughly investigated, with no cover ups. But most of all I hope that ME sufferers will start to get a better deal – up until now they have been getting a very raw deal.
Finally I have to pay tribute to Alem Matthees. Of all the PACE trial authors’ many and varied incompetencies, misjudging Mr. Matthees’ skill, courage and determination, will probably prove to be their greatest.
Thank you. You’ve just said what I’ve been longing to say, but have felt compelled to be more circumspect, lest I be accused of being unhinged, a cyber-bully, part of the “ME/CFS jihad,” etc., when what I actually object to is junk science.
Unfortunately, it doesn’t end here. Esther Crawley and colleagues are even now recruiting patients to do another PACE-like study, only this time on children with ME/CFS. They call it “a trial investigating the effectiveness and cost effectiveness of Graded Exercise Therapy compared to Activity Management for paediatric CFS/ME: A feasibility randomised controlled trial,” and it is referred to as the MAGENTA trial.
The patient information for MAGENTA responds to the question “I have read that GET is harmful. Is that true?” by saying “No, so long as it is provided by appropriately trained healthcare professionals, as it is in this study,” and referencing the Cochrane review, which was based in large part on the PACE trial. Five of the 8 studies they reviewed used the Oxford criteria, which the NIH has said should be retired because “continuing to use the Oxford definition may impair progress and cause harm.” It also links to another review by Peter White, published in the Journal of Psychosomatic Research, which also uses the PACE trial as evidence for the efficacy of GET.
As long as the PACE trial paper is not retracted, they can and will continue to get away with things like this. To knowing use flawed data to justify conducting a similar trial on sick children is immoral and a violation of medical research ethics.
http://www.bristol.ac.uk/ccah/research/childdevelopmentdisability/chronic-fatigue/magenta-trial/
As a plasma physics researcher from Princeton, disabled almost 28 years ago by CFS, I KNOW I can’t increase energy output in any kind of a regular way (the GET – graduated exercise therapy recommended by PACE) – and not pay for it with an extended period of complete collapse. I do as much as I can, and have used CBT since my graduate school days to manage a complicated life, but neither GET nor CBT have made the slightest sustained improvement in my daily life.
They are management strategies: do as much as you can (so as not to get any more deconditioned than necessary) and manage your life as well as you can (because otherwise the disease takes the rest of your life).
If these strategies worked, every person with ME/CFS on the planet would be well by now. We are neither lazy nor crazy – we would flock to the treatment – IF IT WORKED.
To see these useless strategies recommended by, for example, the Mayo clinic – supported by this travesty of non-scientific trash – is the most frustrating part of all. No cause, no cure, and the recommended makes things worse? No wonder we feel invisible and ignored. And blamed for our ‘problems.’
I don’t think it at all a coincidence that GET and CBT would save the heath insurers gobs of money.
Thank you for weighing in. The underlying problem with PACE seems to be very low standards in psychosomatic research.
“A show trial is a public trial in which the judicial authorities have already determined the guilt of the defendant.” – Wikipedia
Substitute medical for judicial, false illness beliefs for guilt, and patients for defendants, and you have PACE. No surprise then that the results are as bogus as any of Stalin’s Moscow trial verdicts.
“It means the reported benefits for cognitive behavior therapy and graded exercise therapy were largely illusory–an artifact of the huge shifts in outcome assessments the authors introduced mid-trial.”
I think it’s been established that the changes to the thresholds of improvement and recovery were made after the trial ended. The point of debate is whether or not the authors saw the protocol-designated results before they switched to the lower thresholds.
There are currently two petitions to stop MAGENTA. Please sign and share widely.
One is for UK people only – 10,000 signatures will force a reply from government:
https://petition.parliament.uk/petitions/166601
The other is global and can be signed by anyone (UK people can sign both):
https://my.meaction.net/petitions/stop-graded-exercise-therapy-trials-for-me-cfs
We must protect children from GET – and we must get these researchers off the PACE gravy train. The sole justification given in the MAGENTA protocol for the trial is that GET was shown to be “moderatively effective” in adults in PACE.
Thank you, David, for this and all your other writings on PACE. Your involvement has been pivotal on bringing this scandal to a wider audience.
Richard Horton’s response to you is disgraceful. So is Robin Murray’s. They don’t appear to be fit to do their jobs, or to be in such positions of public influence.
What arrogant and stupid disregard of patients.
Thanks for all your efforts to unmask this study. As a CFS patient myself, I’ve tried to keep an open mind, but I’ve been skeptical of the results since they seemed so obviously contrary to my own experience. It’s not surprising that it turns out the authors cooked the books–but it’s disappointing that Lancet has been such a willing accomplice in the process.
I think an interesting paper could be written on the strange phenomenon where journal editors back studies that have been proven to all others to be broken. It’s a very peculiar situation and it leaves you wondering what the reasoning is behind it.
PACE is obviously a joke on so many levels. Yet the journals let their reputations be eroded way past the point where this is clear. What are the reasons for this?
Another terrific piece that lifted my spirits no end, knowing the truth is being told. THANK YOU, THANK YOU, THANK YOU!
Regarding the MAGENTA trial, shouldn’t this kill it stone dead:-
http://www.legislation.gov.uk/uksi/2004/1031/regulation/31/made
The Medicines for Human Use (Clinical Trials) Regulations 2004
->Suspension or termination of clinical trial
->>31.—(1) If, in relation to a clinical trial—
->>>(b)the licensing authority have information raising doubts about the safety or scientific validity of the trial, or the conduct of the trial at a particular trial site,
->>the licensing authority may, by a notice served in accordance with paragraph (2), require that the trial, or the conduct of the trial at a particular trial site, be suspended or terminated.
If the PACE trial fallout does not count as “information raising doubts about the safety or scientific validity of the trial”, then I cannot imagine what else would. If MAGENTA is allowed to proceed then surely the licensing authority would be just as culpable as the PACE trial authors.
To the layperson it looks like sections of the scientific research establishment behave similarly to the corrupt autocracies that rule over many nations. They extract rents from the populations they rule over, create ideologies that may not be questioned and generally regard anyone who questions their rule (ideologies) as entities to be punished and discredited. Clearly there is one set of rules for the insiders and an entirely different set of rules for the outsiders. It’s comical how these folks pretend to be fair and objective when they behave like a mafia (without the killings). It’s also interesting how the press allows these folks to engage in vague ad hominem attacks on their critics without any critical analysis.
Thank you David, and all involved, for your work in this very important matter. It is incomprehensible that this trial came to be, was ‘peer-reviewed’ and published. Definitely anti-science.
I would imagine the worse thing that could be said to a trial investigator is your last phrase:
“If nothing else, they have provided a great gift to academia with their $8 million disaster—for years to come, graduate students in the U.S., the U.K. and elsewhere will be dissecting PACE as a classic case study of bad research and mass delusion.”
Now we must ensure that children are saved from the dangers of GET, we must put a stop to MAGENTA. Please sign petitions in Sasha’s comment below.
Thank you for your instructive comment! Please stand up against the fraud of the PACE trial as vocally as you are able! Cheers!
Thank you for not mincing words, Dr. Tuller! Excellent post! Thank you for publishing, Prof. Racaniello!
I’m very grateful to Dr Tuller for the all the time, skill, passion and energy he has brought to the issues: his particular background as a journalist and a public health expert probably helped him to understand and communicate issues that many others might have struggled with.
Wow – best blog yet from David Tuller! I predict that both Lancet and Psychological Medicine will end paying dearly for their arrogance and lack of integrity.
Oh dear – I’ve just up-voted myself, which I really didn’t mean to do? Can I undo it?
Thank you for all the work you have done in exposing this scandal which has inflicted so much suffering on so many desperate and vulnerable patients around the world.