On October 23rd, virology blog published the third installment of David Tuller’s investigative report about the PACE study of treatments for ME/CFS. In the post, Dr. Tuller demonstrated that the key finding of an economic analysis of the PACE trial, published in PLoS One in 2012, was almost certainly false. The finding–that cognitive behavior therapy and graded exercise therapy were cost-effective treatments–relied on an inaccuracy in the paper about whether the results of sensitivity analyses were “robust.”
Since the publication of the virology blog series, the PACE study has come under sustained and blistering public criticism for its many flaws. The PLoS One paper is currently the center of attention as a result of the researchers’ insulting response to Dr. James Coyne, a well-known psychologist and PLoS blogger. Dr. Coyne requested data to verify the results from the PLoS One paper, and was told that his request was “vexatious.” The researchers have called patients “vexatious” for years, of course, but the effort to use this strategy against a respected researcher has caused an uproar. Several colleagues and I, including Dr. Tuller, cited this rejection recently in our own request for a different set of PACE-related data.
Because of the open data policies of the PLoS journals, requesting data on that basis was a smart move by Dr. Coyne, and he has done a brilliant job of rousing support for the larger issue of access to data in scientific research. The PACE authors must recognize by now that at some point they will have to release all of their data.
The PLoS One study reported that cognitive behavior therapy and graded exercise therapy, the two treatments long favored by the main investigators, were more cost-effective than other approaches. The investigators have routinely cited these findings in promoting use of the two treatments. The truth or falseness of these claims from the PLoS One study are at the heart of the current controversy
In fact, it is already clear that the claim is highly unlikely to withstand serious scrutiny, based on the public record. In the October 23rd post, Dr. Tuller demonstrated that subsequent public comments of the lead author contradicted a critical statement in the paper about the PLoS One study’s sensitivity analyses..
The relevant excerpt from virology blog is below:
In another finding, the PLoS One paper argued that the graded exercise and cognitive behavior therapies were the most cost-effective treatments from a societal perspective. In reaching this conclusion, the investigators valued so-called “informal” care—unpaid care provided by family and friends–at the replacement cost of a homecare worker. The PACE statistical analysis plan (approved in 2010 but not published until 2013) had included two additional, lower-cost assumptions. The first valued informal care at minimum wage, the second at zero compensation.
The PLoS One paper itself did not provide these additional findings, noting only that “sensitivity analyses revealed that the results were robust for alternative assumptions.” Commenters on the PLoS One website, including Tom Kindlon, challenged the claim that the findings would be “robust” under the alternative assumptions for informal care. In fact, they pointed out, the lower-cost conditions would reduce or fully eliminate the reported societal cost-benefit advantages of the cognitive behavior and graded exercise therapies.
In a posted response, the paper’s lead author, Paul McCrone, conceded that the commenters were right about the impact that the lower-cost, alternative assumptions would have on the findings. However, McCrone did not explain or even mention the apparently erroneous sensitivity analyses he had cited in the paper, which had found the societal cost-benefit advantages for graded exercise therapy and cognitive behavior therapy to be “robust” under all assumptions. Instead, he argued that the two lower-cost approaches were unfair to caregivers because families deserved more economic consideration for their labor.
“In our opinion, the time spent by families caring for people with CFS/ME has a real value and so to give it a zero cost is controversial,” McCrone wrote. “Likewise, to assume it only has the value of the minimum wage is also very restrictive.”
In a subsequent comment, Kindlon chided McCrone, pointing out that he had still not explained the paper’s claim that the sensitivity analyses showed the findings were “robust” for all assumptions. Kindlon also noted that the alternative, lower-cost assumptions were included in PACE’s own statistical plan.
“Remember it was the investigators themselves that chose the alternative assumptions,” wrote Kindlon. “If it’s ‘controversial’ now to value informal care at zero value, it was similarly ‘controversial’ when they decided before the data was looked at, to analyse the data in this way. There is not much point in publishing a statistical plan if inconvenient results are not reported on and/or findings for them misrepresented.”
Comments
8 responses to “Revisiting the PLoS One economics analysis of PACE”
The CFS Research and Treatment Unit is a partnership between King’s College London and South London and Maudsley NHS Trust http://www.kcl.ac.uk/innovation/groups/projects/cfs/index.aspx data that disagrees with assertions made in PLOS One must have consequences for the universities partnership with the NHS (national health service)
Thank you for this very interesting commentary. The treatment by the PACE authors of their own planned analyses here is reminiscent of all the others that they’ve ditched or dismissed, even though they presumably sat down with serious intent and with some heavy-duty statisticians for their £5 million, publicly funded trial when they drew up their study protocol and analysis plan.
To decide, all of a sudden, that they were wrong about their plans and on such a scale is beyond any scientific norm. This trial now has no credibility left and it’s imperative that independent researchers get the raw data and run the protocol-specified analyses.
Patient deserve good science, not this pile of nonsense.
What a shameful waste of public money.
You continue the ME/CFS misinformation and dragging ME into the CFS debacle which greatly harms ME patients. That patient groups or even journalists do this uncritically is one thing, but science and medicine should be able to understand the issues and implications.
This blog is about viruses and viral disease. What does PACE have to do with it?
PACE is clearly stated on CFS. Without the ME/CFS nonsense is would not apply (or misapply) to ME patients. `ME/CFS is far more harmful, which this PACE frenzy only furthers, than any one trial, no matter how bad. PACE authors aren’t the only ones doing harm.
Queen Mary, University of London wrote about the PACE trial: http://www.wolfson.qmul.ac.uk/current-projects/pace-trial
“What is the PACE trial?
This large-scale trial is the first in the world to test and compare the
effectiveness of four of the main treatments currently available for people
suffering from chronic fatigue syndrome (CFS), also known as myalgic
encephalomyelitis (ME). These are adaptive pacing therapy, cognitive behaviour
therapy, graded exercise therapy, and standardised specialist medical care (see
below). All of the treatments offer ways for patients to deal with and improve
the symptoms of CFS/ME and its effects on disability. The participants in the
trial are randomly allocated to one of the treatments and then given a 12-month
programme involving appointments with specialised doctors and, for three of the
four treatment groups, therapists. Participants’ progress is closely monitored
by specially-trained research nurses or assistants. The five-year trial will
involve 600 participants, aged 18 and over, in Scotland and England. All have to
be referred from the specialist hospital CFS/ME clinics involved in the trial
and these are based in Edinburgh, Oxford and three London hospitals.”
Case in point. This is the same ME/CFS it’s just a name misinformation. Repeating misinformation over and over does not make it true. This is on CFS – that they call ME – and then misapplies CFS research to ME patients.
You have a point, disqus, as the PACE trial entry protocol excluded patients with infection or inflammation (!) [1] “Methods In our parallel-group randomised trial, patients meeting Oxford criteria for chronic fatigue syndrome” [2] and thus excluded any really ill patients… But as Shakespeare said, all those years ago, a rose by any other name would smell as sweet (I paraphrase, I’m sure), and since this illness is conflated with MDD and others, we do have to address the misnomers..
[1] http://www.guardian.co.uk/discussion/comment-permalink/9627125
[2] https://dl.dropboxusercontent.com/u/44533954/PACEarticle.pdf
As far as I can see the Plos paper, published in 2012, does not take into account the long term follow up data published recently, (but presumably available in 2012), which shows no long term benefit from either CBT or GET when compared with APT or SMC. Therefore any so called economic benefit is at best short lived. Surely the researchers should take this into account in any re-analysis of the data, making it imperative that all data be released, not just the parts requested. Thank you for your continuing work on this issue.
But yes, as you pointed out, it is NOT just about a name so quote does not apply.