By David Tuller, DrPH
*October is crowdfunding month at Berkeley. I conduct this project as a senior fellow in public health and journalism at the university’s Center for Global Public Health. If you would like to support the project with a donation to Berkeley (tax-deductible for US taxpayers), here’s the place: https://crowdfund.berkeley.edu/project/27513
Two years ago, the North Bristol NHS Trust conducted a survey among current and recent attendees of clinical services for patients with CFS/ME (as the illness was then being called). The survey included one key question: Did respondents believe that specialist care should be available for CFS/ME patients?
According to the trust’s report on the survey, 99.8 % of the 612 respondents from April to July 23rd agreed that such services should be available, with only one person responding in the negative. (In a longer report, the trust indicated that it received 610 responses by July 23rd; if this discrepancy was explained, I didn’t notice it.) The survey did not ask specifically whether the respondents approved of the treatments being offered, in particular a steady increase in physical exertion called graded exercise therapy and a CFS/ME-specific form of cognitive behavior therapy. Both interventions have long been promoted as cures.
Why bother discussing this survey now? Well, the issue relates to the continuing saga over the new-but-hijacked ME/CFS guidelines from the National Institute for Health and Care Excellence (NICE), the UK agency charged with developing clinical advice about various medical conditions.
in November 2020, NICE published its draft guidelines for what it now calls ME/CFS. The draft recommended against the two interventions most prominently endorsed in the 2007 guidelines, in particular GET and CBT. In the subsequent public comment period, a wide range of interest groups expressed their views on the draft. NICE considered these concerns, produced a final version, and designated August 18th as the publication date.
Instead, in an abrupt divergence from its usual process, NICE announced a publication delay less than 24 hours beforehand. This decision came after fierce objections to the new version from pooh-bahs at some of the UK’s powerful medical associations. (These physician groups are known grandly as “royal colleges,” but they function as trade organizations serving the professional interests of their members.) On Monday, NICE is hosting a “roundtable” discussion among relevant parties—although some stakeholders that asked to be included were not.
North Bristol NHS Trust’s report on the 2019 survey included criticism of other research in which patients have reported harms following these standard interventions, and in particular GET. In the process, the trust’s report also raised questions about the NICE draft issued in November 2020. It is certainly possible that this survey will be referenced at the upcoming roundtable by those parties seeking to subvert the guideline development process and gut the new recommendations.
So let’s take a closer look.
A closer look at the Bristol survey
The North Bristol NHS Trust’s reporting of the survey is seriously deficient. The trust noted the number of responses (whether 610 or 612) but failed to provide information about the response rate–a noticeable and surprising omission. In short, how many people received the questionnaire and did not respond? Or did the service just send the questionnaire to patients they hoped or believed would fill it out as desired?
Hard to know, since here is all we’re told about this point: “Patients were invited to respond to the short survey by emails sent from NHS service email accounts…The majority of patients were those of the Bristol CFS/ME Service and parents of young people attending the Paediatric CFS/ME Service at Bath were also invited to participate.” This passage does not offer sufficient context to allow for a genuine assessment of the findings.
Moreover, the survey was apparently launched primarily as a campaigning and advocacy tool. According to the trust, “this survey was conducted because we were aware of a risk of closure of NHS CFS/ME specialist services.” In other words, the trust wanted to generate data that would support a particular policy goal–persuading the NHS to continue to fund CFS/ME specialist services. Perhaps that’s why the main question was asked in a leading fashion that seemed clearly designed to elicit the desired response.
Here’s how the survey introduced the issue:
“Research shows that adults and children with CFS/ME are more likely to improve with specialist treatment…We believe that adults and children with Chronic Fatigue Syndrome or ME (CFS/ME) should have a choice and be able to access specialist NHS services if they want this.“
Then the respondents were asked this question: “Do you think that adults and children with Chronic Fatigue Syndrome or ME (CFS/ME) should have access to specialist NHS services for assessment and treatment?“
This is laughable. It would be hard to concoct a more effective method of ensuring biased responses. The framing of the question leaves minimal room for “no” as an answer. It shouldn’t be surprising that patients want specialist services, and of course they’re likely to agree that they should have access to them. That doesn’t necessarily mean they like or want the services provided at Bristol and Bath, and no one should presume that it does.
To bolster its argument, the North Bristol NHS Trust noted that only a few people criticized the interventions. But the survey did not ask about them. Instead, it left a blank space with this instruction: “If you want to say anything else, please add some free text below.” The trust analyzed these comments by theme, but failed to indicate how many people added any text at all. This parsimonious approach to detail makes it hard to assess the reported findings.
Indeed, that prompt to add “free text” does not represent a serious effort to obtain actionable information—especially because respondents were not anonymous to those conducting the survey. How many recent or current patients (or patents of patients) would be willing to openly declare to those running their clinic that they disliked the interventions or found them to be useless or even harmful?
In late July, someone posted the survey on social media—at which point, the North Bristol NHS Trust report noted, the proportion of negative comments about the services and the interventions rose dramatically. Almost 30% of the 282 later respondents answered “no” to the main question and many reported concerns about the interventions. These thumbs-down assessments were much more in line with multiple other surveys of patients’ experiences than the Bristol survey. According to the trust, the discrepancy between the first and second batches of responses should raise concerns—but apparently only about the second batch.
Here is what the North Bristol NHS Trust concluded:
“The shift in negative responses…provides compelling evidence that the responses of current and recent NHS patients differ greatly from the responses of people responding via social media. We think that this evidence should be taken into account by the NICE Guideline review in considering the scope of the data submitted from surveys which involved recruitment via social media. It is clear that these surveys do not reflect the experiences of all NHS patients who are accessing specialist CFS/ME Services. It is likely that there are a number of sources of bias involved, and certainly the responses to our survey after it became public suggests that the experiences of actual NHS patients is very different from the experiences of those who heard about the survey via social media.”
This position comes across as an effort to ignore inconvenient data. To my knowledge, it has not been argued that responses to questionnaires disseminated on social media are representative of NHS patients accessing specialist services, nor of the overall patient population. This lack of representativeness, while undoubtedly a limitation, does not render the testimony meaningless or negate the reported experiences.
In contrast, North Bristol NHS Trust appears to assume without justification that its own survey findings are in fact representative of NHS patients attending specialist services. But no one should be able to make this argument with a straight face, or pretend that these results were not themselves subject to enormous bias in the framing of the relevant question.
Beyond that, did North Bristol NHS Trust contact patients who dropped out of clinical services before completing treatment? How about those who went through treatment a year before, or two years, or five years? We simply don’t know—just as we don’t know how many current and recent patients actually received the survey but failed or refused to respond, for whatever reason or reasons.
Will representatives of the medical trade associations really highlight this problematic survey during the roundtable discussion in an effort to impeach other data? Who knows? If they make that choice, their reliance on such flimsy data would expose the weakness of their position. Hopefully, attendees with a more robust grasp of scientific methodology will be able to address the survey’s multiple shortcomings and set everyone straight.
11 responses to “North Bristol NHS Trust’s Biased Survey of Patients Attending “CFS/ME Specialist Services””
Well said. I completed a form when I completed my CBT at Leeds ME clinic. The form was handed to me by the CBT therapist and he waited for me to complete it. I felt constrained to give positive feedback. I felt a negative response would lead to adverse comments on my file.
By the way, attending those clinics led to worsening of my ME. I became almost totally housebound. I was simply not well enough to take the taco to the other side of town and then to go through the one to one session every week. It was way too exhausting. I assumed I would be ok as I wrongly assumed ME experts wouldn’t suggest anything that might harm my health. I had never had mental health issues and I’d already worked out for myself any of the useful stuff in those CBT sessions from reading self help books.
I would never submit to such nonsense now. I would not undergo such a useless (for me) therapy and I would not feel constrained to give a positive response to the feedback forms. Not now. Never again.
NICE and its acolytes are showing an all too human trait by displaying self-interest. They have forgotten that they are being paid to put other people first. I would like to disseminate a rumour that the government are looking to downsize NICE and out-source it to India in the hope that it will become a self-fulfilling prophesy. How is it possible that people who do harm are allowed to continue but the odd one or two whose treatment improves physical symptoms (like my dentist in London) are having to go to the High Court?
It should be shocking that “researchers” are conducting such a blatant public relations/lobbying campaign, and using patients to do it. Instead it is normal and expected.
I’m sure that the people involved are quite surprised to be criticized for manipulating patients and public opinion. After all, “Everyone’s doing it”. And getting away with it. For decades.
Is it possible that this survey might have been created with the primary aim of discrediting the Forward ME online survey that the ME Association posted about at the beginning of April 2019 -https://meassociation.org.uk/2019/04/forward-me-and-oxford-brookes-university-announce-results-of-patient-survey-on-cbt-and-get-in-me-cfs-03-april-2019/ ? And that the key question might have just been a ruse? And that the social media leak might have been a carefully choreographed stunt?
Or do I just have an overactive imagination?
Surely they don’t believe this to be supportive scientific data, still ? How to teach them about authentic studies/data.
So pleased to have your overview so I can easily stay informed on matters that impact my life.
Most patients attending NHS #ME/CFS Clinics are unaware of the dangers of treatments used there. If they were aware, why would they even bother attending? Similarly, they’d be unaware of the cynical framing of questions in this questionnaire. Surely @NICEComms can see this?
One thing i am ashamed off is that when attending such clinics i was too scared to tell them the truth about how awful what they were saying was and how badly it was affecting me. I had had so many bad experiences in the past that i had learnt to just smile and nod and get the hell out of there. But that doesn’t help the clinic or the poor patients that came after me.
The last therapist i saw was nice, she wasn’t like some of the more sadistic ones i’ve encountered in the past, and i genuinely think she thought she was helping. And i know how awful it is when health care practitioners turn against you, it can make getting any help very difficult at all. I didn’t have the cognitive ability to write a letter / email explaining i wasn’t complaining about her, just their whole therapy… Where do you even start with that?!
It’s one of the reasons i hope they get a version of the yellow card scheme so the therapy itself can be reported easily for the harms it causes, rather than the only option being to report the therapist. And obviously, an end to curative CBT / GET…
If i had got this survey and i knew they would know it was me filling it out, i’m not certain i wouldn’t have said what they wanted to hear, or have had the brain power to work out the leading question was asking me to endorse CBT/GET.
Some of the same tricks were used by the Phil Parker Group, which produced misleading claims for efficacy with the lightning process. Immediately following 3 days of something like brainwashing, telling attendees to think positively and believe that they were well, they were asked if they felt better. The phenomenon of client deference and gratitude is well known.
No subjective evaluation should ever be conducted by anyone with a vested interest. Nor should it be conducted in the setting at which the therapy took place, or without sufficient time for a client to process the experience and evaluate for themselves – what, if any, difference it has made to their lives.
While social media surveys tend to show selection bias, it doesn’t follow that other surveys are necessarily better. Desiging and executing surveys requires skill.
Even so and in spite of selection bias, the Vermoeidheid Kliniek in the Netherlands in one year received a score of 4 out of 10 for their services, this only from patients that hadn’t quit. Usually surveys of that type produce a solid 7.
Specialist ME/CFS clinics giving these ‘curative’ treatments have been operating all over England for at least 20 years. It follows that there must be a cohort of thousands of fully recovered ME/CFS patients…..where are they?
Do you need expert help in gaining access/passwords to Facebook, gmail, Instagram, yahoo-mail, snap-chat, twitter, Hotmail, various blogs, icloud, apple accounts etc. Password retrieval, smartphone,tablet portable device hacks, ip tracking and general tracking operations……….contact: hack.truth77 AT g mail . com I got access to my email after 48hours.