By David Tuller, DrPH
David Tuller is academic coordinator of the concurrent masters degree program in public health and journalism at the University of California, Berkeley.
Last week’s post on FITNET-NHS and Esther Crawley stirred up a lot of interest. I guess people get upset when researchers cite shoddy “evidence” from poorly designed trials to justify foisting psychological treatments on kids with a physiological disease. I wanted to post some additional bits and pieces related to the issue.
*****
I sent Dr. Crawley a link to last week’s post, offering her an opportunity to send her response to Dr. Racaniello for posting on Virology Blog, along with my response to her response. So far, Dr. Racaniello and I haven’t heard back—I doubt we will. Maybe she feels more comfortable misrepresenting facts in trial protocols and radio interviews than in addressing the legitimate concerns raised by patients and confronting the methodological flaws in her research. I hope Dr. Crawley knows she will always have a place on Virology Blog to present her perspective, should she choose to exercise that option. (Esther, are you reading this?)
*****
From reading the research of the CBT/GET/PACE crowd, I get the impression they are all in the habit of peer-reviewing and supporting each others’ work. I make that assumption because it is hard to imagine that independent scientists not affiliated with this group would overlook all the obvious problems that mar their studies—like outcome measures that represent worse health than entry criteria, as in the PACE trial itself. So it’s not surprising to learn that one of the three principal PACE investigators, psychiatrist Michael Sharpe, was on the committee that reviewed—and approved—Dr. Crawley’s one-million-pound FITNET-NHS study.
FITNET-NHS is being funded by the U.K.’s National Institute for Health Research. I have no idea what role, if any, Dr. Sharpe played in pushing through Dr. Crawley’s grant, but it likely didn’t hurt that the FITNET-NHS protocol cited PACE favorably while failing to point out that it has been rejected as fatally flawed by dozens of distinguished scientists and clinicians. Of course, the protocol also failed to point out that the reanalyses of the trial data have shown that the findings published by the PACE authors were much better than the results using the methods they promised in their protocol. (More on the reanalyses below.) And as I noted in my previous post, the FITNET-NHS protocol also misstated the NICE guidelines for chronic fatigue syndrome, making post-exertional malaise an optional symptom rather than a required component—thus conflating chronic fatigue and chronic fatigue syndrome, just as the PACE authors did by using the overly broad Oxford criteria.
The FITNET-NHS proposal also didn’t note some similarities between PACE and the Dutch FITNET trial on which it is based. Like the PACE trial, the Dutch relied on a post-hoc definition of “recovery.” The thresholds the FITNET investigators selected after they saw the results were pretty lax, which certainly made it easier to find that participants had attained “recovery.” Also like the PACE trial, the Dutch participants in the comparison group ended up in the same place as the intervention group at long-term follow-up. Just as the CBT and GET in PACE offered no extended advantages, the same was true of the online CBT provided in FITNET.
And again like the PACE authors, the FITNET investigators downplayed these null findings in their follow-up paper. In a clinical trial, the primary results are supposed to be comparisons between the groups. Yet in the follow-up PACE and FITNET articles, both teams highlighted the “within-group” comparisons. That is, they treated the fact that there were no long-term differences between the groups as an afterthought and boasted instead that the intervention groups sustained the progress they initially made. That might be an interesting sub-finding, but to present “within-group” results as a clinical trial’s main outcome is highly disingenuous.
*****
As part of her media blitz for the FITNET-NHS launch, Dr. Crawley was interviewed on a BBC radio program by a colleague, Dr. Phil Hammond. In this interview, she made some statements that demonstrate one of two things: Either she doesn’t know what she’s talking about and her misrepresentations are genuine mistakes, or she’s lying. So either she’s incompetent, or she lacks integrity. Not a great choice.
Let’s parse what she said about the fact that, at long-term follow-up, there were no apparent differences between the intervention and the comparison groups in the Dutch FITNET study. Here’s her comment:
“Oh, people have really made a mistake on this,” said Dr. Crawley. “So, in the FITNET Trial, they were offered FITNET or usual care for six months, and then if they didn’t make a recovery in the usual care, they were offered FITNET again, and they were then followed up at 2 to 3 years, so of course what happened is that a lot of the children who were in the original control arm, then got FITNET as well, so it’s not surprising that at 2 or 3 years, the results were similar.”
This is simply not an accurate description. As Dr. Crawley must know, some of the Dutch FITNET participants in the “usual care” comparison group went on to receive FITNET, and others didn’t. Both sets of usual care participants—not just those who received FITNET—caught up to the original FITNET group. For Dr. Crawley to suggest that the reason the others caught up was that they received FITNET is, perhaps, an unfortunate mistake. Or else it’s a deliberate untruth.
*****
Another example from the BBC radio interview: Dr. Crawley’s inaccurate description of the two reanalyses of the raw trial data from the PACE study. Here’s what she said:
“First of all they did a reanalysis of recovery based on what the authors originally said they were going to do, and that reanalysis done by the authors is entirely consistent with their original results. [Actually, Dr. Crawley is mistaken here; the PACE authors did a reanalysis of “improvement,” not of “recovery”]…Then the people that did the reanalysis did it again, using a different definition of recovery, that was much much harder to reach–and the trial just wasn’t big enough to show a difference, and they didn’t show a difference. [Here, Dr. Crawley is talking about the reanalysis done by patients and academic statisticians.] Now, you know, you can pick and choose how you redefine recovery, and that’s all very important research, but the message from the PACE Trial is not contested; the message is, if you want to get better, you’re much more likely to get better if you get specialist treatment.”
This statement is at serious odds with the facts. Let’s recap: In reporting their findings in The Lancet in 2011, the PACE authors presented “improvement” results for the two primary outcomes of fatigue and physical function. They reported that about 60 percent of participants in the CBT and GET arms reached the selected thresholds for “improvement” on both measures. In a 2013 paper in the journal Psychological Medicine, they presented “recovery” results based on a composite “recovery” definition that included the two primary outcomes and two additional measures. In this paper, they reported “recovery” rates for the favored intervention groups of 22 percent.
Using the raw trial data that the court ordered them to release earlier this year, the PACE authors themselves reanalyzed the Lancet improvement findings, based on their own initial, more stringent definition of “improvement” in the protocol. In this analysis, the authors reported that only about 20 percent “improved” on both measures, using the methods for assessing “improvement” outlined in the protocol. In other words, only a third as many “improved,” according to the authors’ own original definition, compared to the 60 percent they reported in The Lancet. Moreover, in the reanalysis, ten percent “improved” in the comparison group, meaning that CBT and GET led to “improvements” in only one in ten participants—a pretty sad result for a five-million-pound trial.
However, because these meager findings were statistically significant, the PACE authors and their followers have, amazingly, trumpeted them as supporting their initial claims. In reality, the new “improvement” findings demonstrate that any “benefits” offered by CBT and GET are marginal. It is preposterous and insulting to proclaim, as the PACE authors and Dr. Crawley have, that this represents confirmation of the results reported in The Lancet. Dr. Crawley’s statement that “the message from the PACE trial is not contested” is of course nonsense. The PACE “message” has been exposed as bullshit—and everyone knows it.
The PACE authors did not present their own reanalysis of the “recovery” findings—probably because those turned out to be null, as was shown in a reanalysis of that data by patients and academic statisticians, published on Virology Blog. That reanalysis found single-digit “recovery” rates for all the study arms, and no statistically significant differences between the groups. Dr. Crawley declared in the radio interview that this reanalysis used “a different definition of recovery, that was much harder to reach.” And she acknowledged that the reanalysis “didn’t show a difference”—but she blamed this on the fact that the PACE trial wasn’t big enough, even though it was the largest trial ever of treatments for ME/CFS.
This reasoning is specious. Dr. Crawley is ignoring the central point: The “recovery” reanalysis was based on the authors’ own protocol definition of “recovery,” not some arbitrarily harsh criteria created by outside agitators opposed to the trial. The PACE authors themselves had an obligation to provide the findings they promised in their protocol; after all, that’s the basis on which they received funding and ethical permission to proceed with the trial.
It is certainly understandable why they, and Dr. Crawley, prefer the manipulated and false “recovery” data published in Psychological Medicine. But deciding post-hoc to use weaker outcome measures and then refuse to provide your original results is not science. That’s data manipulation. And if this outcome-switching is done with the intent to hide poor results in favor of better ones, it is considered scientific misconduct.
*****
I also want to say a few words about the leaflet promoting FITNET-NHS. The leaflet states that most patients “recover” with “specialist treatment” and less than ten percent “recover” from standard care. Then it announces that this “specialist treatment” is available through the trial—implicitly promising that most of those who get the therapy will be cured.
This is problematic for a host of reasons. As I pointed out in my previous post, any claims that the Dutch FITNET trial, the basis for Dr. Crawley’s study, led to “recovery” must be presented with great caution and caveats. Instead, the leaflet presents such “recovery” as an uncontested fact. Also, the whole point of clinical trials is to find out if treatments work—in this case, whether the online CBT approach is effective, as well as cost-effective. But the leaflet is essentially announcing the result–“recovery”—before the trial even starts. If Dr. Crawley is so sure that this treatment is effective in leading to “recovery,” why is she doing the trial in the first place? And if she’s not sure what the results will be, why is she promising “recovery”?
Finally, as has been pointed out many times, the PACE investigators, Dr. Crawley and their Dutch colleagues all appear to believe that they can claim “recovery” based solely on subjective measures. Certainly any definition of “recovery” should require that participants can perform physically at their pre-sickness level. However, the Dutch researchers refused to release the one set of data—how much participants moved, as assessed by ankle monitors called actometers–that would have proven that the kids in FITNET had “recovered” on an objective measure of physical performance. The refusal to publish this data is telling, and leaves room for only one interpretation: The Dutch data showed that participants did no better than before the trial, or perhaps even worse, on this measure of physical movement.
This FITNET-NHS leaflet should be withdrawn because of its deceptive approach to promoting the chances of “recovery” in Dr. Crawley’s study. I hope the advertising regulators in the U.K. take a look at this leaflet and assess whether it accurately represents the facts.
*****
As long as we’re talking about the Dutch members of the CBT/GET ideological movement, let’s also look briefly at another piece of flawed research from that group. Like the PACE authors and Dr. Crawley, these investigators have found ways to mix up those with chronic fatigue and those with chronic fatigue syndrome. A case in point is a 2001 study that has been cited in systematic reviews as evidence for the effectiveness of CBT in this patient population. (Dr. Bleijenberg, a co-investigator on the FITNET-NHS trial, was also a co-author of this study.)
In this 2001 study, published in The Lancet (of course!), the Dutch researchers described their case definition for identifying participants like this: “Patients were eligible for the study if they met the US Centers for Disease Control and Prevention criteria for CFS, with the exception of the criterion requiring four of eight additional symptoms to be present.”
This statement is incoherent. (Why do I need to keep using words like “incoherent” and “preposterous” when describing this body of research?) The CDC definition has two main components: 1) six months of unexplained fatigue, and 2) four of eight other symptoms. If you abandon the second component, you can no longer refer to this as meeting the CDC definition. All you’re left with is the requirement that participants have suffered from six months of fatigue.
And that, of course, is the case definition known as the Oxford criteria, developed by PACE investigator Michael Sharpe in the 1990s. And as last year’s seminal report from the U.S. National Institutes of Health suggested, this case definition is so broad that it scoops up many people with fatiguing illnesses who do not have the disease known as ME/CFS. According to the NIH report, the Oxford criteria can “impair progress and cause harm,” and should therefore be “retired” from use. The reason is that any results could not accurately be extrapolated to people with ME/CFS specifically. This is especially so for treatments, such as CBT and GET, that are likely to be effective for many people suffering from other fatiguing illnesses.
In short, to cite any findings from such studies as evidence for treatments for ME/CFS is unscientific and completely unjustified. The 2001 Dutch study might be an excellent look at the use of CBT for chronic fatigue*. But like FITNET-NHS, it is not a legitimate study of people with chronic fatigue syndrome, and the Dutch Health Council should acknowledge this fact in its current deliberations about the illness.
*In the original phrasing, I referred to the intervention mistakenly as ‘online CBT.’
Comments
27 responses to “A Follow-Up Post on FITNET-NHS”
Thanks very much for this article.
Thanks again Dr Tuller!!
An argument that I don’t think has got enough attention: The ‘Other Care’ group in the FITNET study were also receiving CBT. I find this astonishing, because the researchers have clearly used the impressive ( although misleading) results from FITNET to suggest that CBT is the effective intervention here. This further propagates the myth that ME is largely psychological. Further, around a third of the ‘Other Care’ patients were receiving GET, and another third physio. Since we know these treatments can be harmful, it is the lack of ‘other care’ that could be significant here. I wonder if this is something you could highlight if you write about this again – the fact that we need to look more closely at the control group in this study.
Hi Paul–I wrote about that in the earlier post–that half the comparison group had CBT and a third GET, so they were comparing online CBT to the “evidence-based” treatments they had been recommending for years. very bizarre “control” group.
Thank you for airing this horrific disease and how people with Myalgic Encephalomyelitis are either forgotten or misdiagnosed. I have no idea how Crawley and friends are able to overcome resistance from the ME Community and organize more research funding without any accountability for the well being of those trying to live with this disease. This Collaborative of Psycho Babble feed on newly diagnosed families who haven’t had enough time to understand the complexities of ME. The Collaborative operate with more “smokes & mirrors” than China. They sit on and/or draft biased protocol for the Cochrane Review which shows the benefit of GET & CBT, cover each other’s reviews and publish in the Lancet who offers protection for their “scientific data”.
both this and the earlier observations are being brought to the attention of the ad hoc ME/CFS committee of the Health Council of the Netherlands, as well as possible future articles on flaws of studies of the Dutch NKCV and the British BPS-exponents, both past and current.
Wow, wow, wow. Too many things you’ve written down that are so amazing.
It’s amazing how linked they all are. And of course if you use the same type of Oxford variation criteria meta-analyses also turn out amazing.
It’s so horrible. We need help. We need a cure. And for that we need research funding. No biomedical research funding since 2005 in the Netherlands because of the 2005 health council. And Crawley is absorbing the ones in the UK as well.
What a mess, what a mess. Thank you David. Grateful for not letting them get away with this. It needs to stop.
Thanks again for another clearly explained article.
Not pulling any punches! I like it 🙂
Thank you again David for this sensible blogpost.
I always wonder why, if researchers must use subjective outcomes, they don’t simply ask patients, “Do you now consider yourself to be recovered?” Perhaps the answers would not be to their liking? Yet I’m sure that is the question that other patients considering any treatment option would like to hear answered! I would.
Surely the clouds of obfuscation surrounding these therapies are now getting well blown away?
Some of them do ask that. But if patients are told constantly they will recover from treatment, then they are more likely to report that they “recovered,” whether or not they can actually do any more.
Thanks Dr Tuller. I must have missed that. Yeah, not enough being made of this I think – the control group could easily suffer due to GET and Physio, accounting for a difference between groups.
They make their own definition of ‘recovery’. This is one of the biggest problems, and ethical questions hanging over this body or research.
Great stuff. Great to hear
David, thank you so much for your hard work.
I suspect that the entire CBT approach is built on creating and documenting placebo responses and conflating them with genuine improvement of the underlying illness. So they need a case definition that includes many cases that will improve even without effective treatment. I am wondering if there is any data on the natural history of different case definitions. I vaguely recall that there was a paper on early intervention with CBT/GET in CFS. I think it suggested there was high regression to the mean. Unfortunately I can’t find this paper.
Thank you again, David. We desperately need your articles to appear in the national media.
It’s amazing to see how easily you can shut the mouths of this kind of pretentious “doctors” that won’t hesitate to earn fame and $$$$ at expense of innocent people’s health (and what is worse, at expense of children’s health).
I’m a member of the spanish #millionsmissing team (#millonesausentes) and we would love to translate all your awesome articles about the PACE case and now FITNET (I heard some are calling it SHITNET??) so please, if there is a way we could translate the great tools you’ve created to defend pwme from harmful ideologies and practices in the medical community, let me know!
Insight into how the smoke & mirrors work http://quillette.com/2016/02/15/the-unbearable-asymmetry-of-bullshit/
Many thanks for another informative blog, uncovering yet more irregularities with the FITNET studies. They just keep on giving, don’t they? One thing I don’t think you’ve mentioned yet is a couple of surprising statistics concerning the patients in the 2012 Dutch study. Classic M.E. usually has a sudden onset triggered by a virus or other infection, but if you look at the patients on this study (see table 1), about 60% of them had gradual onset and only about 18% started with an infection. This suggests (surprise, surprise) that we may be talking about a high proportion of patients with generic fatigue rather than M.E. It is also interesting that over 40% of the patients were attending school full time in the fortnight before the study began, which gives the impression that a lot of them were not very ill to start with.
Thank you David for continuing to invest your time in writing about this. These issues are all out in the open for anyone to see, yet you are one of the few who is willing to point it out.
These people have been stuck in their insular little mutual admiration society for too long and I think they really truly do believe their own bullshit at this point. There was nobody to call them out on it before, but that is starting to change.
“Beware the aphorism of the blinkered investigator ‘Don’t blind me with science, my mind is made up’” — James F. Mowbray
A concise and logical article exposing the illogical mindset of the GET/CBT crowd and their attempt to fake science to fit their own ignorance and pad their pockets. Thank you, Mr. Tuller!
When I first heard of FITNET, I misread it as FITBIT, the line of wrist pedometers, and thought “oh good, they’re actually going to use objective data.” Wrong! Of course they won’t because they wouldn’t be able to cram the results through statistic filter and force them to fit their narrative.
Thanks again to David Tuller. Keep the pressure up, keep pushing back, keep them honest.
The more I read your ‘Trial by Error’ serie, David, the more I wonder about how medicine, in particular psychiatry built up this empire where patients are less human than them, where patients have less rights and less knowledge than them. This empire needs to be defeated. Sadly, 25 years or so this empire reigned on ME, and made us the patients its victims.
It is truly unbelievable that here we are in 2016, still waiting for biomarkers, still waiting for a first FDA approved treatment, and still hoping that the other side of medicine, and science and politics stand up and say something, and especially stop the bad science from happening, from publishing, and from reaching med schools and doctors offices from all around the world.
i truly hope David that we have reached the tipping point and tht things will get much easier from here. Thank you as always for telling the truth.
Great post, David. When reading about the machinations of the BPS school, this quote from PJ O’Rourke often comes to mind:
“Politics is the business of getting power and privilege without possessing merit.”
Crawley, Wessely, Sharpe et al aren’t interested in helping patients or undertaking serious, meaningful research. Rather, their sole interest is in gaming the system to gain power, funding and prestige. And, as we’ve seen, they are exceptionally good at it. They even peer-review each other’s work and approve new funding applications for yet more of their clique’s terrible research.
In this way, mediocre researchers and incompetent physicians have been able to obtain sinecures, knighthoods and build taxpayer-funded empires far beyond what they would have merited on their talents and abilities alone.
The only way to challenge them is tear down the whole stinking edifice by continually embarrassing them and those that control their purse-strings. In this, you are doing an exceptional job.
David, thank you for another great post. The amount of effort, energy and talent that has been used to show what is going on is incredible.
In my opinion, the upshot is:
1) They can always quote the PACE trial because it has not been retracted from The Lancet
and
2) The PACE was a fraud and, until they’re brought up on fraud charges, it would appear that nothing will change.
“Be sceptical, ask questions, demand proof. Demand evidence. Don’t take anything for granted. But here’s the thing: When you get proof, you need to accept the proof. And we’re not that good at doing that.”
Michael Specter
Hi, you are more than welcome to translate everything you want, as long as there’s a link included to the original. Is there a big CBT/GET movement in Spain as well?
Hi A.B. Here’s a link to the IACFSME-2016 conference program. On page 77 (Poster 8), there’s a summary of a presentation by James N Baraniuk MD: “Population-Based CFS Prevalences Using Different Criteria and the Styles Questionnaire: The Oxford criteria grossly over-estimate CFS/ME prevalence”.
http://iacfsme.org/ME-CFS-Primer-Education/News/IACFSME-2016-Program.aspx
His conclusion: “The Oxford criteria grossly overestimate CFS/ME prevalence and include unacceptable numbers of CFSLWIFS (CFS Like With Insufficient Fatigue Syndrome), CIF (Chronic Idiopathic Fatigue) and control subjects. Study outcomes and treatment recommendations based on Oxford criteria cannot be generalized to CFS or ME. FM (2011) and CFS criteria overlap significantly.”
It is worth noting that Esther Crawley frequently dilutes the selection criteria even further for her studies and ‘research’ papers, which makes them even less applicable to those with ME/CFS who ipso facto have an organically-based energy production problem and exertion intolerance.
Following your last Blog post on Fitnet NHS, I submitted the following complaint to the National Institute for Health Research:
“I would like to bring to your attention, the concerns and criticisms of the Fitnet NHS trial (being conducted by Prof Esther Crawley & colleagues at the University of Bristol), contained in the following article by Dr David Tuller: https://www.virology.ws/2016/11/21/trial-by-error-continued-the-new-fitnet-trial-for-kids/ I would also draw your attention to the “Fitnet” area of the University of Bristol website, which, in my opinion, propagates misinformation – particularly in the “flyers” promoting the trial – with the specific intention of biasing the outcome of the study: http://www.bristol.ac.uk/ccah/research/childdevelopmentdisability/chronic-fatigue/fitnet-nhs/gpinfo/ I would therefore welcome your comments following a thorough investigation of the detailed content of this complaint.”
Yesterday, I got the following reply:
“Dear Mr Watton,
Thank you for your email regarding the FITNET-NHS study. The content within your email was thoroughly reviewed. All applications to the HTA Programme undergo a comprehensive assessment as described in the ‘general assessment criteria’ document available on our website at http://www.nets.nihr.ac.uk/funding. This includes review by both professionals and members of the public. We are content that the FITNET-NHS trial has met the criteria listed. We are also content the study is accurately following NICE guidelines.
We are unable to enter into further correspondence on these issues, but we thank you again for voicing your concerns.
Best wishes,
Maggie Shergill
Research Manager (Monitoring) | NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC)”
What hope is there when a national institution, charged with overseeing research, is incapable of understanding what is required in order to ensure that research is done well.