By David Tuller, DrPH
David Tuller is academic coordinator of the concurrent masters degree program in public health and journalism at the University of California, Berkeley.
The PACE authors have long demonstrated great facility in evading questions they don’t want to answer. They did this in their response to correspondence about the original 2011 Lancet paper. They did it again in the correspondence about the 2013 recovery paper, and in their response to my Virology Blog series. Now they have done it in their answer to critics of their most recent paper on follow-up data, published last October in The Lancet Psychiatry.
(They published the paper just a week after my investigation ran. Wasn’t that a lucky coincidence?)
The Lancet Psychiatry follow-up had null findings: Two years or more after randomization, there were no differences in reported levels of fatigue and physical function between those assigned to any of the groups. The results showed that cognitive behavior therapy and graded exercise therapy provided no long-term benefits because those in the other two groups reported improvement during the year or more after the trial was over. Yet the authors, once again, attempted to spin this mess as a success.
In their letters, James Coyne, Keith Laws, Frank Twist, and Charles Shepherd all provide sharp and effective critiques of the follow-up study. I’ll let others tackle the PACE team’s counter-claims about study design and statistical analysis. I want to focus once more on the issue of the PACE participant newsletter, which they again defend in their Lancet Psychiatry response.
Here’s what they write: “One of these newsletters included positive quotes from participants. Since these participants were from all four treatment arms (which were not named) these quotes were [not]…a source of bias.”
Let’s recap what I wrote about this newsletter in my investigation. The newsletter was published in December 2008, with at least a third of the study’s sample still undergoing assessment. The newsletter included six glowing testimonials from participants about their positive experiences with the trial, as well as a seventh statement from one participant’s primary care doctor. None of the seven statements recounted any negative outcomes, presumably conveying to remaining participants that the trial was producing a 100 % satisfaction rate. The authors argue that the absence of the specific names of the study arms means that these quotes could not be “a source of bias.”
This is a preposterous claim. The PACE authors apparently believe that it is not a problem to influence all of your participants in a positive direction, and that this does not constitute bias. They have repeated this argument multiple times. I find it hard to believe they take it seriously, but perhaps they actually do. In any case, no one else should. As I have written before, they have no idea how the testimonials might have affected anyone in any of the four groups—so they have no basis for claiming that this uncontrolled co-intervention did not alter their results.
Moreover, the authors now ignore the other significant effort in that newsletter to influence participant opinion: publication of an article noting that a federal clinical guidelines committee had selected cognitive behavior therapy and graded exercise therapy as effective treatments “based on the best available evidence.” Given that the trial itself was supposed to be assessing the efficacy of these treatments, informing participants that they have already been deemed to be effective would appear likely to impact participants’ responses. The PACE authors apparently disagree.
It is worth remembering what top experts have said about the publication of this newsletter and its impact on the trial results. “To let participants know that interventions have been selected by a government committee ‘based on the best available evidence’ strikes me as the height of clinical trial amateurism,” Bruce Levin, a biostatistician at Columbia University, told me.
My Berkeley colleague, epidemiologist Arthur Reingold, said he was flabbergasted to see that the researchers had distributed material promoting the interventions being investigated, whether they were named or not. This fact alone, he noted, made him wonder if other aspects of the trial would also raise methodological or ethical concerns.
“Given the subjective nature of the primary outcomes, broadcasting testimonials from those who had received interventions under study would seem to violate a basic tenet of research design, and potentially introduce substantial reporting and information bias,” he said. “I am hard-pressed to recall a precedent for such an approach in other therapeutic trials. Under the circumstances, an independent review of the trial conducted by experts not involved in the design or conduct of the study would seem to be very much in order.”
Comments
34 responses to “More Nonsense from The Lancet Psychiatry”
In the Ampligen trials, I was in a compassionate care, open label “study.” But there were a lot of patients in the double-blind study in the same doctor’s (the late Dr. Bellesorte) office. We were under strict orders not to even TALK to the patients in the double-blind, to avoid any risk of prejudicing the outcome. Given the inherently subjective nature of a study using psychotherapy and exercise, I am astounded that they can be so cavalier about posting a newsletter saying ANYTHING about study outcomes to participants while the study was in progress.
Thank you for your continued good work in investigative journalism. And thank you, Dr. Racaniello, for giving this important conversation a forum.
I think the PACE trial authors have actually answered all the criticisms satisfactorily. A lot of the criticism has come from people who have misunderstood aspects of the trial and what its conclusions are. I think CBT and GET are the best treatments available at the moment as long as they are properly tailored to the individual patient’s level of ability.
Sure, they’ve trained a bunch of patients to say they feel better on surveys, despite the fact they’re not performing better on objective tests nor are returning to work or going off disability. Did I understand the aspects of the trial correctly?
There is so much momentum now to the exposure of PACE for the flawed, shoddy thing that it is, and we owe you, David Tuller, a huge debt of gratitude for getting this recent ball rolling. Many, many thanks also to all others who have joined the battle with their wise words. It cannot be said too often just how much hope this has all given to people like myself who have battled so long with this wretched illness and its associated equally wretched politics.
I really do fail to see how anyone in the health professions, or health policy makers, could any longer claim this dog’s breakfast of a trial as the “best available evidence” any longer. Their time is up, the hole is big, and they should stop digging.
“Where patients followed courses that only related to CBT, GET or Pacing;
more reported an improvement in symptoms following their Pacing course
than did those who attended either of the other courses. CBT resulted in
91% of participants feeling their ME/CFS symptoms were unaffected or
made worse, GET 88%, and Pacing 55%. GET was deemed to have led to a
worsening of symptoms by more patients than those who
attended either CBT or Pacing courses (graph 3:4:2a).”
http://www.meassociation.org.uk/2015/05/23959/
http://iacfsme.org/PDFS/Reporting-of-Harms-Associated-with-GET-and-CBT-in.aspx
Outside of the PACE trial I think GET and maybe CBT is sometimes incorrectly applied, and that is why it can lead to worsening symptoms.
Another point is that the testimonials section includes a recommendation from #10 Downing Street. On the surface, it may appear to be a neutral statement in regards to specific treatment, but it implies that PACE is a good, well-designed trial, which encourages people to see the entire trial in a positive light, but most especially the treatment(s) the trial is designed to validate. The fact that there are so many holes that can so easily be found in the trial design clearly indicates that this was a poor trial, and that people going into it should have been warned about it, not go into it with a positive expectation.
I think you need to re-read the criticisms and the response to them again, then. Many points were glossed over, or ignored entirely, and excuses given for methodology changes, presenting seriously disabled patients as recovered, and a whole slew of other such complaints were beyond ludicrous. To say that they’ve answered the criticisms satisfactorily makes me suspect that either you’re associated with the team or that this is a paid response.
Some patients experienced significant improvement in symptoms, not necessarily a recovery, I think this is surely better than nothing
Some patients were selected based on Oxford criteria and likely did not have ME/CFS at all. It is therefore not unexpected that patients experiencing symptoms which CBT and GET might help with would indeed experience an improvement.
If you say you feel less fatigued, but you’re not able to exert more energy, what’s the point?
What leads you to make this astounding theory, exactly? Do you have evidence to back it up, or is this just wishful thinking in the hopes of discounting evidence within the trial that failed to support their pet theory?
Personal opinion, based on own experience and other patients’ anecdotes, and a reading of the GET therapists’ and participants’ manuals in the PACE trial
You’re missing the point. People will also report improvement in symptoms when given an inert placebo, while their underlying disease remains unchanged. Especially when high expectations are set, patients are pressured, or followed by a therapist that sets goal, or when they fear they might be labelled as non-cooperative, etc. PACE intentionally introduced or reinforced some of these factors in the CBT and GET groups.
I suspect this person is digging for inflammatory remarks, as is often the case. These remarks are then used to conflate all of the unanswered and legitimate criticism that this comment has done thing to refute with militancy. Best to steer clear.
Attracting inflammatory exchanges to Dr Racaniello’s blog can be seen as a way of impugning his reputation if not immediately moderated. He will be tarred as a willing conspirator to militant behaviour.
All very subtle stuff, but this is how you protect junk Science. Define what is reasonable, invite your opponents into making mistakes and address nothing that you cannot defend.
And in case it’s not clear: in PACE, there were no clinically significant improvements in the more objective measures of health and function.
GET is widely reported to be harmful by patients. PACE was designed to almost certainly underreport the actual harm associated with GET. For example, any harm occurring in the second half of the study was ignored according to the protocol.
You strike me as someone who hasn’t really read the criticism and takes the words of the PACE authors at face value.
David – you raise a key issue concerning the PACE authors’ response to the point in our letter about the leaflet and equipoise. The point is that the groups have quite different expectations to start with. When randomly assigned to their groups, when asked if their ‘Treatment is Logical’, those who assigned to Adaptive Pacing Therapy (APT) gave 84% endorsement, Cognitive Behavioural Therapy (CBT) 71% and Graded Exercise Therapy (GET) 84% – by contrast Specialised Medical Care (SMC) was endorsed as a ‘logical treatment’ by fewer than half – 49%. And following treatment, only 50% were satisfied with SMC compared with 82-86% for the other three conditions. Further, at the end of treatment, the proportion dissatisfied with treatment was greater in the SMC than the other three groups *combined*. This is covered in my blog http://keithsneuroblog.blogspot.co.uk/2015/11/pace-thoughts-about-holes.html
My point is that the SMC becomes akin to a wait-list control and this is well-known to often induce detrimental effects in participants – and this is only likely to be worsened by receipt of the leaflet outlining how great everyone is doing, while those who see their assignment as logical are buoyed-up by the positive statements in the leaflet.
Nobody is arguing that the impact of a leaflet could predict the whole pattern of results – the key point is the introduction of unequivocal presence of bias …against Specialised Medical Care and in favour of the other three treatments
The questions aren’t going away. The reason they wont answer directly is because what they did is entirely indefensible. There is no way to make it look good, so instead they just try to shift onto something else. But the same questions will just get louder until they are answered directly.
Thanks for your continued great coverage.
Thank you for you continued efforts to reveal the PACE flaws
Noticeably, nothing they’ve said refutes (or even addresses) either your suggestion, or my suggestion elsewhere in this thread that they’re associated with the PACE team or are a paid responder. I agree that inflammatory responses are unhelpful, though, and the fact that this person has only anecdotal evidence to support their claim eliminates any factual weight from their comment.
With respect,that may be your opinion but you haven’t included any points that would substantiate it. No quotes from the trial authors,no contrasting of GET vs another treatment – though admittedly I wouldn’t care much for the latter as the value of GET shouldnt be derived from how it compares to other treatments for a condition largely regarded as untreatable except by some experimental drugs still in fairly primitive trials ( a lot of customisation of dosage schedule may be needed for rituximab for example)
That’s becoming a pattern here unfortunately, the other articles have randomly generated usernames from people who apparently have never ( in their travels of the vast number of websites with this commenting system and its universal username and sign in system) commented before. E.G. one swearing that the lax criteria used (diagnostic) by the PACE folks invalidates all criticism because it doesn’t include aspects of the condition like Post exertional malaise and thus they were examining “chronic fatigue” not CFS or ME but funnily enough,the world over we find the PACE findings applied by doctors and medical bodies to patients diagnosed with all the stricter sets than the crazy Oxford diagnostics
You talking about disqus here? That persona is all over the internet saying CFS is not ME/CFS or CFS/ME and all of those names are bogus and nothing to do with M.E. but, of course they are.. obfuscations.
Whereas I’ve never seen jmtc before and that comment is laughable, given the evidence. Answered the criticisms satisfactorily? Is jmtc having a laugh? So many questions are fended off with pathetic excuses…
Significant improvements? Like a subjective 6 or 8%? Without objective outcome measures and with a lot of flannel about how good everyone else was doing? I call you as a troll, sir.
What is glossed over is the fact that CFS is NOT ME as in the ME/CFS fraud (which has NO established meaning or officially recognized definition). PACE is more honest in what it was studying (even though it was a flawed trial) than the bogus ignorant “ME/CFS”nonsense.
jmtc you are incorrect, the PACE trial authors have not answered all the criticisms satisfactorily. The flaws and defects in the PACE trial remain and totally discredit the treatments and the authors. CBT and GET are not the best treatments available and are actually ineffective, incapable of delivering recovery and are quite harmful. Please educate yourself about these facts at http://www.me-ireland.com/bogus.htm#pace
jmtc you are incorrect again, your perception of ‘significant improvement’ is based on fraudalent changes to the 4 original protocols for recovery and ignoring the 3 objective markers for recovery. Changing these renders all outcomes suspect, null, void, and meaningless. The flaws and defects in the PACE trial remain and totally discredit the treatments and the authors. CBT and GET are not the best treatments available and are actually ineffective, incapable of delivering recovery and are quite harmful. Please educate yourself about these facts at http://www.me-ireland.com/bogus.htm#pace
jmtc you are incorrect, the PACE trial authors have not answered all the criticisms satisfactorily. The flaws and defects in the PACE trial remain and totally discredit the treatments and the authors. CBT and GET are not the best treatments available and are actually ineffective, incapable of delivering recovery and are quite harmful. Please educate yourself about these facts at http://www.me-ireland.com/bogus.htm#pace
your subjective opinions and views are not backed up by hard evidence. You are full of presumptions, assumptions and suppositions. The flaws and defects in the PACE trial remain and totally discredit the treatments and the authors. CBT and GET are not the best treatments available and are actually ineffective, incapable of delivering recovery and are quite harmful. Please educate yourself about these facts at http://www.me-ireland.com/bogus.htm#pace
your personal or subjective opinions and views are not backed up by hard evidence. You are full of presumptions, assumptions and suppositions. The flaws and defects in the PACE trial remain and totally discredit the treatments and the authors. CBT and GET are not the best treatments available and are actually ineffective, incapable of delivering recovery and are quite harmful. Please educate yourself about these facts at http://www.me-ireland.com/bogus.htm#pace
It is not only the PACE authors who have long demonstrated great facility in evading questions they don’t want to answer.Dr. Hans Knoop seems to have the same skills; he refused to answer my questions
This week I asked dr. Hans Knoop twice on linkedin for a
reaction on the severe flaws the PACE-trial is known for at this moment, and on
his role towards PACE. Today I added an
email containing my questions.
Hans Knoop is a clinical psychologist and head of the
Knowledge (and treatment) Centre for Chronic Fatigue (NKCV), part of Radboud
University Medical Centre in Nijmegen. Amongst other publications on CFS, he
published an article in The Lancet in 2011, together with his collegue
professor Gijs Bleijenberg. They showed their enthousiasm about the outcomes of
the PACE trial and added something extra to the results.
The questions I asked him were:
1.
What is your reaction on the fact that severe
flaws have been found in the PACE-trial?
2.
Why didn’t you mention these severe flaws?
3.
How can it be explained that several outcome
parameters, described in the protocol, were scratched from the outcome measurements?
4.
What is your explanation concerning the
fact that you and Bleijenberg added an
extra percentage to the supposed positive outcomes of this trial?
5.
What is your explanation for using rather
flexible criteria for “severely ill” or “recovered” on SF36PF?
He
didn’t answer. Why would that be?
a.
He didn’t see my questions yet? That would be nice. Surely I can expect his
answers very soon!
b.
The described problems were there and he didn’t
see them? That would be strange for a scientist publishing about the outcomes of this trial.
c.
The problemes were there, he saw them and he
decided not to mention them? That would be even stranger for a scientist
publishing about the outcomes of this trial.
d.
The described problems are not there? In that
case there is a lot of evidence that has to be refuted!
I know the reason now! He wrote me that he has no time to answer questions posed by just somebody by mail. He is to busy doing his research and his lessons.
Yes, exactly! The best evidence available shows that GET is the most harmful and least helpful of all interventions for ME’cfs’ and among several other fatal flaws, the PACE study did not adequately monitor harm caused by GET and CBT, which can include death for patients with this disease.
You are either a troll, a paid shill or have not read the criticisms. There can be no other conclusion.
From the little I’ve heard from PACE participants, what the researchers grandly term “Specialised Medical Care” would be better described as “Standard (Non-Specialised) Medical Care” – or better yet, “No Treatment At All”.
I’d assumed when reading the protocols and reports that the “SMC” arm had involved doctors actively working on genuinely medicalised care for their participants. I got the impression they’d be treating symptomatically – pain relief, beta blockers, muscle relaxants, or whatever seemed a logical match for the patient. That would have been a nice change, and worth investigating as a treatment possibility – going back to the 1980s and before, when GPs were expected to try out problem-solving skills every so often. But no. From what I can gather, the meetings were basically what you’d get from the majority of GPs: “Well NICE says there aren’t any drugs licensed for it, so we can’t do anything for you. You’ll just have to cope.”
Small wonder those guys were dispirited and felt like they’d been left to wait. They had! SMC wasn’t a “treatment arm” as advertised; it was an unacknowledged control group and the patients felt it acutely.