Cochrane Update, and Caroline Struthers’ Latest Letter

By David Tuller, DrPH

The situation at Cochrane appears to remain fluid. Last week, the organization posted a notice that it is seeking a new editor-in-chief. The current occupant of the position, Dr David Tovey, is leaving in May. Dr Tovey has been at Cochrane for a decade, so this move could easily have been a long-planned departure.

Since last year at least, Dr Tovey has been grappling with the fate of the contested review of exercise treatments for chronic fatigue syndrome, as Cochrane has called the illness. The announcement of this change comes while the situation with the exercise review is still uncertain. So is the overall disposition of the illness within Cochrane, which has housed it in the Common Mental Disorders section. Perhaps Dr Tovey will be able to set matters on a more fruitful path before he takes his leave.

To recap: The Cochrane review of exercise therapy was last updated in 2014. Last year, Cochrane had determined that concerns raised about the review were legitimate, and Dr Tovey decided that it should be temporarily withdrawn while the authors revised it. That move was pre-empted in October by biased news coverage that failed to examine the many scientific and methodological deficiencies of the exercise review; instead, the coverage portrayed Cochrane as buckling under pressure from irrational, anti-scientific patients.

This charge was unfair. In response, Virology Blog posted an open letter of support for Cochrane that was signed by more than 40 scientists, academics and other experts. Amid the uproar, Cochrane did not withdraw the exercise review but asked for revisions from the authors. In November, Cochrane rejected that revised version as inadequate but left the current version on its website.

In December, Cochrane took a related step. It withdrew the published protocol for a second exercise review of CFS—an expanded version known as an “individual patient data” review. Based on that protocol, the IPD review had already been written. However, it was known to have received some scathing critiques from peer-reviewers and had not been published. The withdrawal of the protocol for that IPD review was a tacit acknowledgment that Cochrane had no plans to publish the review itself.

With all this going on, Cochrane further signaled that the illness could be moved out of the Common Mental Disorders section to another area, with an announcement planned by the end of the year. No such announcement has occurred. So as of now, the long-term fate of the exercise review is in limbo, as is the future home of the illness within Cochrane.

I would assume that, behind the scenes, Dr Tovey has been under enormous pressure from the GET/CBT ideological brigades and their enablers in the UK academic and medical establishments. These pooh-bahs have defended this “eminence-based” treatment paradigm, as psychology professor Brian Hughes at National University of Ireland Galway has called the tendency to assess research based on prestige rather than quality. They refuse to acknowledge the obvious–that the evidence base falls apart under the scrutiny of experts outside their domains of influence. And that PACE really is a piece of crap.

For defenders of the status quo, Cochrane is a bulwark against the collapse of the GET/CBT approach, now that international experts recognize that PACE does not meet minimal scientific standards. Last summer, after Virology Blog posted its most recent open letter to The Lancet about the trial’s “unacceptable methodological lapses,” the Medical Research Council cited Cochrane as supporting evidence for PACE. Earlier this month, the Health Research Authority cited the Cochrane exercise review’s positive rating of PACE–without mentioning that this assessment is one of the issues in dispute.

This unresolved state of affairs is untenable. Perhaps Dr Tovey will be able to take the necessary steps to fix things before his upcoming departure, or perhaps his departure means that further decisions about the issue will be bumped into the future. While I hope for the former, I am concerned about the possibility of the latter.

In the meantime, debate continues. A few days ago, Caroline Struthers sent a lengthy response to the letter she received from Dr Tovey last month regarding a complaint she had filed about the exercise review.

Struthers was a member of the Cochrane Consumers’ Executive from 2015-2016. I am reposting her response in full here, with her permission.

**********

18 February 2019

Dear Governing Board and Chair of the Cochrane Library Oversight Committee
cc. David Tovey, Editor-in-Chief; Mark Wilson, Cochrane CEO; Co-chairs of the Cochrane Council

Re: Complaint about the Cochrane Exercise for Chronic Fatigue Syndrome Review

In the absence of a complaints procedure or a process for the independent assessment of my complaint, I expect you are aware that I also submitted it to the Charity Commission.

In the meantime, please consider the complaint and David’s response in light of the comments below which follow the same numbering system as the original.

I appreciated seeing the notes on the agenda of the Governing Board meeting which took place 19-20 January in London outlining the remit of the Cochrane Library Oversight Committee (CLOC). To quote from these notes, the remit of the Committee is to:

Consider and where appropriate mediate on, matters of tension between the Governing Board and the Editor-in-Chief, at the request of either party, that relate to the editorial process or published content of or in The Cochrane Library.

The notes also say that membership of the CLOC should include at least one patient or patient advocate. Please can you let me know which of the current membership fit that description and how?

I am disappointed by the lack of interest shown in this issue demonstrated by the Board co-chairs passing it back to David to respond to despite it being addressed specifically to them, and my having already been in correspondence about it with David and members of the Central Executive for well over a year. There also seems to be little sense of urgency or concern for people with ME displayed by the Chief Executive Officer, the Cochrane Council (including the two patient representatives) or the Governing Board.

The recent decision by HRA to cite the exercise review as evidence that PACE was viewed as a well-conducted trial indicates that this is an urgent matter. In the letter to Rt. Hon Norman Lamb MP of the Science and Technology Committee, the HRA stated that “The robustness of the PACE trial has been considered in a Cochrane review that classified it as high quality”. The review is therefore continuing to influence health care policy as well as the opinions of politicians and health officials because of the failure of the authors to respond appropriately to concerns about their work, and lack of necessary action by Cochrane. An editor’s note on the review, which can only be discovered if you click through the “what’s new” link, indicates a resubmission from the authors intended to account for the concerns raised about their work still doesn’t meet the standards expected of Cochrane reviews, and so was rejected for publication. Yet this old and uncorrected version of the review is still being cited as “evidence” that PACE was a high-quality trial.

Many of the issues I am raising here were expressed in Virology Blog’s open letter of support for Cochrane on 23 October 2018, which was posted after the announcement that Cochrane had decided to temporarily withdraw the review. That letter was signed by more than 40 scientists, academics and other experts from Harvard, Stanford, Cornell, Johns Hopkins, Uppsala, University College Dublin, and many other leading institutions.

I don’t anticipate David’s response, or my comments on it which follow, will cause the necessary “tension” between the Board and the Editor-in-Chief for either party to request the impartial scrutiny of the Cochrane Library Oversight Committee. I only hope that you, as members of the Board of Trustees, will put the interests of patients and the public – the beneficiaries of the charity – first, and advise David to withdraw the Exercise for CFS review, as he stated he would do in October. I also ask, once again, for Cochrane to issue an apology to people with ME for the distress and harm caused by Cochrane’s inaction.
Before I respond to David’s answers to my complaint in turn below, I’d like to thank the many people whose work I’ve drawn upon in this response, particularly the late Robert Courtney.

1. I criticised that the conclusion of the Exercise for CFS review was that Graded Exercise Therapy (GET) is beneficial when the supporting evidence (ie. the methodological quality of the included trials) is extremely poor.

David spoke of the review being published “in 2004 following the usual rigorous editorial process and in line with the expected standards of the time including peer review”, but it is clear that there were serious problems with this process and the role of patients within it. By whose standards of the time was this process rigorous? Not by the standards of those patients who had been attempting to raise concerns about the quality of research in this area since the 1990s.

Instead of arguing that the way Cochrane’s processes were followed was acceptable, Cochrane should recognise that the problems surrounding this review indicate that urgent changes must be made to its processes to avoid causing more problems for patients.

In his reply David picked extracts from the review (not the plain language summary), possibly to demonstrate how measured the review’s conclusions had been.

“It concludes that for one outcome, fatigue, there is benefit (low to moderate quality evidence) and that in the trials, reports of serious harm were rare. For other outcomes the certainty of the evidence was low or very low. The Abstract of the review explicitly states that data on harms are ‘sparse’.”

David’s choice of extracts drew my attention to the plain language summary which has been published in ten languages. It upgrades the quality of the evidence for the benefit of exercise from “low to moderate” to “moderate” for reducing fatigue. It also states that exercise has a “positive effect” on physical function, sleep and overall health. So the plain language summary indicates that exercise is beneficial for four outcomes not one.

Readers are not informed of the fact, highlighted by Robert Courtney in one of his comments on the review, that exercise therapy at follow-up was not associated with a significant improvement for any of the pre-specified outcome measures apart from self-reported sleep. The summary of main results states that

A positive effect of exercise therapy was observed both at end of treatment and at follow‐up with respect to sleep (Analysis 1.12; Analysis 1.13), physical functioning (Analysis 1.5; Analysis 1.6) and self‐perceived changes in overall health (Analysis 1.14; Analysis 1.15).

In fact, analyses 1.6 and 1.15 show no significant effect, even though these outcomes relied on data from subjective self-report outcomes from unblinded trials. Furthermore for the review’s prespecified primary outcome of standardised mean differences in fatigue questionnaire scores, there was also a null result at follow-up, although this was not made clear to readers.

The summary also states that exercise “didn’t make symptoms worse” and “Serious side effects were rare”. This creates a different impression from what David quoted from the abstract which was that “data on harms are `sparse’”.

Considering the problems now recognised with much of the research into exercise therapy for CFS, a plain language summary that overstates benefit and is overly reassuring about potential side effects is not a good look for a Cochrane review.

2. I questioned the decision of the review authors to rely almost entirely on data from subjective outcomes when the included studies were unblinded.

David’s response was to point out that the limitations of the reliance on subjective outcomes in open label trials are recognised in the review and are reflected appropriately in the GRADE Summary of Findings tables.

So the authors recognised and documented the limitations caused by reliance on subjective outcomes in open label trials, yet they had decided to focus on subjective outcomes in their review, despite the presence of objective outcomes in the same studies? My understanding is that before the Larun review was published a patient had raised concerns about the divergence between the more positive results seen for subjective self-report outcomes, which are prone to problems with bias, and the largely null results for objective outcomes which are less prone to bias yet these concerns were left unaddressed in the review.

In the same point I questioned the decision of the authors to ask for and acknowledge the advice of the principle investigator of the PACE trial who has an obvious interest in a favourable assessment of it. David’s response was that triallists are often consulted during the review process and there is nothing in the Cochrane guidance to prohibit it.

The fact that Cochrane guidance doesn’t currently prohibit something doesn’t make it OK. In this case it seems part of an inappropriately close relationship between those who have developed and tested a proposed treatment and those who are supposed to be providing independent scrutiny of their work.

3. I stated that relying on subjective outcomes in open label trials is “terrible methodology”, and inexplicable in a review of open label trials.

David’s response was that this was “simply an opinion” and was not shared by his methodological advisors.

He explained in his response to point 2 above that the authors recognised these limitations and reported them in the appropriate points in the review. Again, this begs the question: Why did the review authors choose to limit the review outcomes to subjective ones (with the exception of healthcare usage, which for which there was a null result)?

The PACE trial measured three outcomes that the trial researchers had classed as being ‘more objective’. PACE reported that the addition of exercise therapy led to no improvement in their objective measure of fitness, or in employment outcomes. For the six minute walking test there was a statistically significant improvement with GET, but it failed to meet the criteria for a clinically significant improvement they used for their primary outcomes (0.5 of standard deviation at baseline).

Just because the review authors made the decision not to add objective measures to the protocol they inherited from previous versions of the review, doesn’t make it the right decision. In fact, it would have been easy to foresee that the omission of null results from objective outcomes would bias the reported findings of the review to the positive. Again, who were the independent peer reviewers and methodologists who approved the review authors’ decision not to analyse objective outcomes from studies they were including?

There were aspects to the conduct of the PACE trial that raise particular concern about the biasing of data for subjective self-report outcomes. A recent paper from Carolyn Wilshire summarises these issues, drawing attention to the fact that, mid-trial, “GET participants were told that ‘in previous research studies, most people with CFS/ME felt either “much better” or “very much better” with GET’ and that GET was ‘one of the most effective therapy strategies currently known’”.

It seems that such problems did not just afflict PACE, the largest study included in the Larun review. For example, the second largest trial, FINE, made use of what was described as “Rousing Reassurance”. This involved telling patients things such as “every exercise is strengthening your body”, “there is no disease” and “from the moment you walk out of this room your recovery is beginning”.

As well as potentially biasing participants’ self-report outcomes, such reassurance also seems likely to affect how staff viewed participants. There are indications that this has happened, such as a paper from the FINE trial which quotes a supervisor saying: “There have been one or two times where I have been worried because they have got angry at the patients…that anger has been communicated to the patients. Their frustration has reached the point where they sort of boiled over… there is sort of feeling that the patient should be grateful and follow your advice, and in actual fact, what happens is the patient is quite resistant and there is this thing like you know ,“The bastards don’t want to get better””

By promoting unduly positive views of the quality of research like PACE, and the benefits of exercise therapy for CFS patients, Cochrane risks playing a role in biasing the way society responds to an entire patient population.

It’s clear that many CFS patients are concerned about the low standards often applied to the research into their condition. Their views need to be taken seriously, and engaged with, rather than superseded by the opinions of those selected by Cochrane. If Cochrane has found independent methodologists who wish to defend trials such as PACE and FINE from concerns about the potential for bias distorting subjective self-report outcomes, then they should be expected to do so as part of an open discussion with both patients and those methodologists who disagree.

Professor Jonathan Edwards Emeritus Professor of Connective Tissue Medicine at UCL had this to say on the Science for ME blog.

The problem of unblinded trials with subjective end points is not ‘just an opinion’. It is the only sensible view and is embedded in all evidence-based medicine. With due respect if Tovey’s advisors do not share this view they are incompetent. Again with due respect if Tovey is not aware of this he is not qualified to act as an editor in an organisation like Cochrane. He seems to have acted fairly so far but in this instance he appears to be indicating that he has no grasp of reliable evidence. That makes Cochrane something of a basket case maybe.

The problem of subjective end points with no blinding is as basic as it gets. It is the reason we have blinded trials. No trial with this design of a drug would be taken seriously. Moreover the problem is much worse with therapist-delivered treatments, not less bad, because of the role playing and manipulation of attitudes that is inherent in the human interaction.

I am afraid this hits rock bottom.

I think an apology is due, together with an admission that the comment about ‘just an opinion’ was not appropriate. If a Cochrane official makes a statement like this it effectively destroys the Cochrane reputation entirely.

Last year 114 scientists, academics, clinicians, ten MPs and the dozens of patient and advocacy organisations signed an open letter to Dr Richard Horton, editor of The Lancet. That letter called out PACE’s “unacceptable methodological lapses”. What can you share about the expertise and justifications used by the Cochrane methodological advisors who don’t share this opinion?

David explained that it is common for Cochrane Reviews to use patient-reported outcomes. I have no problem with this when there are validated measures of core outcomes, and the outcomes are appropriate and important to patients. I do have a problem when a Cochrane review primarily relies on subjective outcomes alone when included trials cannot be blinded. If the results from relevant objective measures contradict subjective measures, as they did in the case of PACE trial, you have an additional reason for concern about the design and conduct of the trial. This review team deliberately chose to rely on subjective outcomes and downplay objective outcomes when they didn’t have to. Without a good explanation, it risks creating the impression that there was a desire to rely on outcomes which would be likely to favour the intervention.

This is an extract from the Cochrane Handbook

8.12.1 Rationale for concern about bias

In empirical studies, lack of blinding in randomized trials has been shown to be associated with more exaggerated estimated intervention effects, by 9% on average, measured as odds ratio (Pildal 2007). These studies have dealt with a variety of outcomes, some of which are objective. The estimated effect has been observed to be more biased, on average, in trials with more subjective outcomes (Wood 2008).

This is from a blog about blinding by Saul Crandon posted on 26 June 2017 on the Students for Best Evidence site (https://www.students4bestevidence.net/blinding-comprehensive-guide-students/)

What about when blinding is not possible?

In cases where blinding is not possible or feasible, the outcome measures must be objective! If you are reading a study that is unblinded, with subjective outcome measures, then you may as well stop reading it and move on. This is because, if a patient is aware they are receiving the active intervention and the outcome measure is subjective, such as ‘how much pain they are experiencing’, their reporting is likely to be biased. Knowledge of the group assignment can consciously or subconsciously cause the patient to feel better and report improved subjective pain tolerance. This is not a reliable study design and the results should not be interpreted with any certainty.

This blog is on an excellent website whose partners include the UK Cochrane Centre and many other Cochrane Centres throughout the world. The blogger seems to have a better grasp on the limitations of subjective outcomes in unblinded trials than the authors of this review, or indeed Cochrane’s independent methodological advisors

Professor Edwards also gave me permission to quote his simple illustration of why blinding matters for those who find the technical concepts difficult.

A simple test of the adequacy of a trial design is whether it is likely that a trial of an arbitrary alternative therapy, such as cranial osteopathy, with this design, would produce a spurious ‘significant’ result from systematic bias using conventional statistics. For trials such as PACE the historical evidence is that the answer is yes. Moreover, we have an example, in the SMILE trial, of an arbitrary alternative therapy for ME/CFS, designed by an osteopath (the Lightning Process), giving a positive result which apparently ‘surprised’ the principle investigator. I am embarrassed that she should be surprised since I had hoped that all our trainees would understand that this is the reason for blinding trials (which is the same as the reason for blinding trials with subjective outcomes because other trials do not need blinding).

David replied to this point by stating that the authors followed the review protocol. However, the review authors did in fact alter the way the primary outcome was assessed, changing from a planned analysis in the protocol which revealed a null results at follow-up to an analysis which allowed for largely positive results to be reported. If this poorly justified change from what was specified in the protocol could be made, then accounting for important methodological concerns raised by patients about the reliance on subjective outcomes should also have been possible.

4. I stated that the principal investigator of the PACE Trial and advisor on the review, and his co-investigators were proponents of the exercise intervention, advisors to the insurance industry, and the PACE trial was partly funded by the Department of Work and Pensions (DWP) who have never funded a trial before or since.

Encouraging the view that sick and disabled people would need less financial support if they were encouraged to adopt the right attitudes and behaviour seems to have helped secure support for recent welfare reforms. The DWP had founded these reforms on a biopsychosocial model of disability which emphasised the need for those with conditions such as CFS to engage with interventions like Graded Exercise Therapy with positive expectations. These reforms, and their impacts upon disabled people, have now been condemned by a UN enquiry for leading to “grave and systematic violations” of disabled people’s rights. At the same time, insurance companies have been refusing claims from ME/CFS patients who do not wish to undergo additional exercise therapy, with patients having reported PACE primary investigator Peter White’s personal involvement in this. White was a Chief Medical Officer at Swiss Re-insurance, and an archived copy of their coverage of his presentation to them on the PACE trial includes advice on how to use mental health exclusions to limit pay-outs to claimants with CFS.

I also outlined other flaws in the conduct of PACE which I will repeat more fully here. David didn’t explain why the review authors ignored all of them in their risk of bias assessment. I missed a few out in the first version of my complaint, and I also know there are more problems than the ones I have listed below.

• Outcomes were switched after the trial had started.

• The PACE triallists didn’t declare their conflicts of interest to the study participants, breaching the Declaration of Helsinki, which they had committed to abiding by in their protocol.

• They used the Oxford diagnostic criteria for CFS which meant that a large proportion of the trial participants may not be representative of the patient population with ME/CFS. The National Institutes of Health Pathways to Prevention Workshop on ME/CFS recently concluded that “Continuing to use the Oxford definition (of ME/CFS) may impair progress and cause harm”

• They moved the boundaries for inclusion and recovery meaning that for at least one outcome 13% of participants were simultaneously ill enough to take part in the trial, and well enough to be classed as recovered. The justification provided for this was that “approximately half the general working age population” had an SF36-PF score of under the protocol recovery cut off of 85, but the data they cited showed that this was true of only 18% of the general working age population. Six years after publication, this error remains uncorrected, though PACE data that was released under court order has revealed null results for the protocol specified recovery criteria in a 2017 paper by Wilshire et al. Wilshire’s paper also explains problems with other aspects of the PACE team’s revised recovery criteria, which appears to have been devised post-hoc without this being explicitly declared in their paper

• They decided to abandon one of the few objective outcome measures (actigraph) despite using it at baseline. When challenged, they argued that the decision was made as actometers were too burdensome for patients. The trial’s recently released Trial Management Group minutes (meeting #11 4/11/2004) show that the decision was based on then unpublished data from another research group which found that CBT for CFS was able to alter subjective self-report outcomes, but did not lead to an improvement in objectively measured activity levels. This was taken to show that actometers were “not useful for outcome”.

• They promoted the effectiveness of their favoured therapies during the trial with leaflets.

David’s response was that it is not unusual for scientific research for triallists or trial sponsors to have vested interests in the outcome of a study. I am well aware of this, and I am sure we can all agree it is nothing to celebrate. Nor is it something to gloss over as par for the course. It is something I thought Cochrane would inform readers of and attempt to actively control for. I would not expect Cochrane authors to seek and acknowledge the advice of the primary investigator of the main included study.

David goes on to say that the PACE study was subject to the usual risk of bias and certainty evaluations as part of the standard peer review process. Yet as both Tom Kindlon and Robert Courtney made clear in their comments on the review, the Cochrane risk of bias tool was not correctly applied to the PACE trial. It was rated at low risk of bias as if it had reported all results for the outcome measures specified before recruitment had begun. This was clearly not the case, and the review authors chose to ignore it. It is important to try to understand how this mistake in rating the risk of bias occurred, and why it was left uncorrected for so long after it was pointed out.

David then informed me that Cochrane reviews are not intended to make recommendations. Again, I am well aware of this. However, Cochrane promotes itself as the most reliable and trustworthy source of evidence – the “home of evidence” in fact. Therefore Cochrane has a duty to ensure this evidence is unbiased and unaffected by conflict of interest. If the results of a Cochrane review backs up a decision-maker’s motivation to save money, for example by limiting disability benefit payments to patients who refuse a treatment Cochrane evidence has indicated is effective, that source of information will be favoured by that decision-maker over any other source. I don’t think it’s appropriate to deny Cochrane has any responsibility for the decision-maker’s behaviour in this case.

5. I stated that the PACE trial is now being used as a teaching example of how not to do a randomised trial

David’s response was that conflict of interest and doing unplanned analyses after data collection was not unique to PACE. I have given examples of many other problems with the PACE trial which merit special attention. I have no problem with post-hoc analyses, if they are labelled as such, and all the pre-specified analyses are done and reported. The attitude that Cochrane reviewers can ignore these travesties of conduct and reporting because they are common in trials, particularly of psychological and other non-drug interventions is, again, deeply worrying.

6. I pointed out the problem caused by the review’s management by the Common Mental Disorders Group

David said its placement there didn’t imply anything about causation. I wasn’t suggesting it did. But it does imply that the treatment under review is assumed to be an appropriate choice for ME/CFS.

I am pleased the review is being transferred to another group. I think Neuromuscular would probably be the most suitable, however I have heard it is likely to be the Pain, Palliative and Supportive Care Group. Considering this group’s Co-ordinating Editor is also one of the two people working with David on oversight of the exercise review, I want to mention concerns that some patients have in relation to his 2015 book on physical sensations. This includes a chapter on chronic fatigue that seems to partly reflect the culture patients have been raising concerns about. The PACE trial researchers’ positive results were presented uncritically with CBT and GET described as challenging patients’ beliefs and bodily sensations. I realise that the problems with PACE were less widely discussed in 2015, but they were still present and important. If the limitations of the supposedly supporting evidence for claims about treatment efficacy are not properly explained then we risk devaluing and over-riding patients’ own responses to their illness with misguided claims about what the ‘evidence-based’ approach is. Cochrane should be operating as a bulwark against this, and speaking out against exaggerated claims made by researchers. In the case of ME/CFS it seems that low standards for evidence have been accepted even in the face of patients telling us that this is harming them. The past work of the Pain Group’s editor indicates a possible bias towards the norms of a research culture that has not served ME patients well, and which has encouraged an uncritical acceptance of work like PACE.

Is Cochrane consulting patients about this? Patients have been involved in reviewing the now thankfully withdrawn individual patient data review, so they would be ideal people to start with.

It is interesting to note that NICE put ME/CFS condition in a category of its own. https://www.nice.org.uk/guidance/conditions-and-diseases. Sensible. No doubt ME/CFS reviews will have to be shoehorned somehow into the Cochrane Review Group system even if there is nowhere suitable. The Cochrane Review Group system is often unhelpful, and confusing to the outside world. I hope at least a new review team will be recruited.

7. I said that I knew a person with ME who had been in correspondence with David Tovey and Rachel Churchill since before the review was originally published in 2014 warning of the huge potential problems with it. I asked why there was no apparent involvement of patients in the review, despite offers of help well in advance of publication. I then made the mistake of also implying I could supply copies of correspondence between David and this person. I should have made it clear that I have never been privy to the content of any discussions between them, and so could not and would not supply copies of any correspondence or notes without both parties’ permission. I apologise for any misunderstanding.

David did not answer the question about why patients were actively excluded from involvement in the review. It is Cochrane policy to involve patients in reviews as much as possible, and the common excuse trotted out by review groups for not doing so is that it’s too difficult to find “consumers” with the necessary skills. I would like an explanation of why then these offers of help were deliberately rejected. If the problems with the way patient concerns about reviews are treated are not to be examined openly, then that it is something that Cochrane need to look at internally rather than ignore.

8. I complained that Cochrane Editors seemed unconcerned by level of criticism submitted via the feedback system, or whether the author responses were appropriate or not.

David responded by saying that the feedback mechanism had worked as intended and that it was the author’s responsibility to respond. If I were Editor-in-Chief I would be deeply concerned by the inadequate and borderline rude responses of the Cochrane authors to irrefutable criticism of the review. If this is what it looks like when the feedback mechanism is working appropriately, then how bad do things need to be before it is thought to have gone wrong?

Who has the responsibility to judge if the authors’ have responded appropriately or not? It seems little oversight was provided, although I’ve been told that concerns were raised about the authors’ poor responses as soon as they were published.

9. I mentioned the case of Robert Courtney who provided detailed, rigorous and irrefutable criticism via the feedback system and received such a poor response from the author he made an official complaint.

David replied that this complaint was treated seriously by the Cochrane editors and that work is ongoing as a consequence of the editorial assessment of this complaint. This puts the responsibility entirely on those providing critical feedback to follow up if the author’s response is inadequate, and it’s only then that Cochrane takes it seriously. It was obvious to anyone who read the feedback of Courtney and others who made critical comments that the authors’ responses were inadequate.

10. I stated I have been complaining privately to David and his team about this review and other issues (such as the pointless support of reviews of homeopathy, and the inadequate control of conflict of interest) for years. I was often left with the impression that these concerns were not being taken seriously, with e-mails often receiving no response. I also pointed out the huge amount of criticism of the PACE trial by Dr David Tuller and others.

David stated that his editors have taken a balanced and cautious approach focussing solely on the quality of the science. My criticism, and that of scientists, trial methodologists, statisticians and ME experts, not to mention an extremely well-informed patient community is also focussed on the quality of the science. This has been made much more difficult by Cochrane’s failure to recognise or criticise the problems to be found in this field of research, allowing institutions that wished to dismiss concerns about PACE to do so seemingly with Cochrane’s endorsement. If Cochrane’s editors had been focussing on the quality of the science, then this review would not have been published in this form, and a lot of trouble could have been saved.

David said that the amount of time he and his team have taken to investigate complaints has been appropriate. I and many others don’t agree.

11. I mentioned an article in the BMJ about the open letter to The Lancet organized by Dr David Tuller and signed by around 200 organisations and scientists calling for a reanalysis of the PACE trial. I forgot to mention in my original complaint that there was also a whole issue of The Journal of Health Psychology (Special Issue: The PACE Trial Volume 22 Issue 9, August 2017) dedicated the PACE trial. An accompanying editorial (Special Issue on the PACE trial) described the low numbers and quality of pro-PACE submissions of the 20 people approached from the pro-PACE side. This, combined with the unwillingness of the pro-PACE authors to make revisions after peer review, meant that there were only two articles out of 17 defending PACE.

David said that Cochrane editors have a responsibility to make decisions on the basis of the science, not opinions, however, strongly held. I am not sure if he thinks the opinions, strong or otherwise, of the hundreds of people who signed the open letter to The Lancet or commented on the BMJ article, or wrote one of the many articles criticising PACE in The Journal of Health Psychology were based on anything other than the science.

Don’t you think patients would have been delighted with the results of the PACE trial if the science were sound? The acceptance of this bad science as having validity has meant the establishment of harmful policies and exaggerated claims about a treatment’s efficacy and safety. People assume that because Cochrane has not “called out” the bad science, that the science was good. Cochrane has let patients down.

12. I mentioned that David made the welcome announcement in October 2018 that he was planning to withdraw the review pending investigation. I and many others were extremely disappointed and confused that this didn’t happen.

David explained in his response that two of the authors had persuaded him not to.
He defended it by saying the team is focused on not being inappropriately influenced by individuals on either side of the long-standing disputes around this condition and review.
But being influenced by the authors is OK? It seems those now making judgements on this review within Cochrane are those who played a role in approving publication of the previous review, or who have already cited PACE uncritically in their own work. Does anyone with a history of being willing and able to speak out about the clear problems with this influential review have any role in Cochrane’s current process for oversight?

It seems that another factor which led to the review not being withdrawn was correspondence from Trygve Ottersen, Acting Director of the Norwegian Institute of Public Health, the institution that currently funds Cochrane Norway. Ottersen argued that it would be unreasonable to withdraw the review as a result of applying higher standards for the assessment of Larun’s work than for other Cochrane reviews. Yet the old standards which allowed this review to be published are difficult to defend. Standards for research should be set in consultation with patients, who should not have the low standards of others imposed upon the work which affects their lives.

It is important that what David refers to as a “duty of care” to contributors does not lead to an attempt to protect them and their work from justified criticism. Cochrane’s business model relies on attracting volunteers to conduct reviews, so there may be particular hesitancy to take action which risks embarrassing review authors or the institutions that support them. Cochrane must be willing to weather that storm if it is to avoid prioritising the interest of researchers over patients.

13. I asked why the critique of the HPV review prompted an “urgent investigation”, but Mark Vink’s critique of this review didn’t.

David’s reply here was revealing. He said that the investigation into Robert Courtney’s complaint was “very similar in its rigour and the importance ascribed to it to the response to the Jorgensen et al HPV article”. Very similar apart from speed. If Courtney’s complaint was taken as seriously and is being investigated as rigorously as the Jorgensen article, why was the investigation into the HPV article fast-tracked, whereas the investigation into Courtney’s complaint which preceded it by many months is still ongoing? This looks like discrimination in favour of protecting Cochrane’s reputation rather than in protecting patients.

14. I mentioned the growing concern of MPs about the poor treatment of people with ME as a result of the acceptance of the flawed evidence of the PACE trial. In a Westminster debate in June last year, the MP Carol Monaghan stated that when the full truth about the PACE Trial comes out it will be one of the biggest medical scandals of the 21st Century

There was subsequent House of Commons debate on 24th January 2019 with the motion

“That this House calls on the Government to provide increased funding for biomedical research into the diagnosis and treatment of ME, supports the suspension of Graded Exercise Therapy and Cognitive Behaviour Therapy as means of treatment, supports updated training of GPs and medical professionals to ensure they are equipped with clear guidance on diagnosis of ME and appropriate management advice to reflect international consensus on best practice, and is concerned about the current trends of subjecting ME families to unjustified child protection procedures.”

The motion was carried unanimously by 40 MPs with no one speaking against it.

David responded by repeating that Cochrane editors and authors are focusing solely on the science and the evidence and should not be swayed unduly by opinion of any sort. Except the opinion of Cochrane’s anonymous and independent methodological advisors, of course. Again, the implication is that opinions against PACE are not based on science. In fact this seems to be a worrying example of British science needing political pressure to take the more rigorous approach required to better serve patients.

He adds that Cochrane strongly values its independence from political interference. However, it is the influence of those already working with Cochrane who seem most likely to sway important judgements, rather than political interference. It seems that there are academics with reputations and financial interests to protect who have found Cochrane’s systems and processes amendable to influence, or else they are simply designed in a way which serves their interests.

No medical issue is going to be debated in Parliament like this without something having gone seriously wrong which is adversely affecting patients, carers, services and/or the economy. The PACE trial is a major concern which Cochrane seems to be attempting to tip-toe around, rather than taking a stand against some of the unacceptable failings that have done so much to harm patient trust.

15. My final point mentioned the publication of the protocol for an Exercise for CFS review using individual patient data (IPD) which included most of the PACE triallists in the review team, plus a prominent Cochrane member who is a proponent of exercise as a treatment for ME.

Thankfully this review protocol has been withdrawn and the review will never be published.
David remarked that IPD reviews generally include the triallists as part of the author team and that this is “accepted practice”. Again, this accepted practice is wholly inappropriate and unacceptable.

Since the IPD protocol was published, the PACE authors have used it, and their association with Cochrane, to help dismiss concerns about their work. Legitimate freedom of information requests from patients related to the PACE trial have been described as ‘vexatious’ in decision notices citing Cochrane’s work. These unjust decisions have now in turn been cited by the Health Research Authority as evidence that the PACE researchers have responded reasonably to requests for information and data from their work. Cochrane has played a pivotal role in the cascade of misinformation spread amongst UK institutions about the PACE trial and the patients who have rightly criticised it.

In summary, Cochrane is a charity whose beneficiaries are patients and the public. This is not true of academic institutions, and it certainly isn’t of publishing companies like Elsevier. Yet Cochrane has stonewalled criticism and stood on the side-lines impassively watching as patients and advocates struggle for truth and justice against old-fashioned eminence-based medicine.

Sincerely,

Caroline Struthers