Another Nudge to Mahana’s Gasteroenterology Advisers

By David Tuller, DrPH

Back in February, I wrote to some science advisers to Mahana Therapeutics to alert them to the company’s unjustified claims about the impact of its web-based program on the symptoms of irritable bowel syndrome. Two of them–Dr Peter Lu and Dr Carlo di Lorenzo–are gastroenterology specialists at Nationwide Children’s Hospital in Columbus, Ohio, a renowned medical center.  I did not hear back.

Given the coronavirus situation, I have refrained from continuing to pursue this matter for the last two months. During that same period, Mahana has continued to misrepresent the web-based program on its site. The company presumably still hopes this program will pass muster with regulatory officials in the US and the UK. Perhaps Mahana believes health agencies will not notice that the reported benefits for the program over treatment-as-usual in reducing symptom severity cannot accurately be defined as clinically significant.

Over the weekend, I sent Dr Lu and Dr di Lorenzo a follow-up nudge, similar to one I recently sent to another gastroenterology adviser, Dr Mel Heyman from University of California, San Francisco. I cc’d Mahana’s CEO, Rob Paull, and Professor Racaniello.

**********

Now I’m trying to get back to work.

Let’s recap: Mahana announced in January that it had licensed the web-based CBT program from King’s College London. My concerns involved statements in a press release and on Mahana’s website that clearly exceeded the data from the relevant study. The press release described the web-based program’s impacts on symptom severity as “substantial” and “durable.” The website called them “dramatic” and “potentially game-changing.”

As I have pointed out, these descriptions cannot be justified. At 12 months, the reported benefits of the web-based program over treatment-as-usual on the study’s measure of symptom severity were statistically significant but clinically insignificant. At 24 months, these reported benefits were neither statistically nor clinically significant. Given the weak results for this core indicator, it is hard to understand why Mahana decided to license the program in the first place.

It now appears that Mahana has changed the website’s description of the program’s purported benefits and no longer characterizes them as “dramatic” and “potentially game-changing.” I am pleased to see this change. However, the site currently states that “patients enrolled in a minimal-contact digital CBT program experienced significant and clinically meaningful reduction in the severity of their IBS.”

This is an empty statement, since it would be true if only two patients in the web-based arm had achieved this “significant and clinically meaningful reduction” in symptom severity. It could also easily be said about the treatment-as-usual arm, since at least two patients in that arm also achieved “significant and clinically meaningful reduction” in symptom severity. The relevant question is whether the program provides clinically meaningful benefits over and above what is achieved through treatment-as-usual–and the main results for the symptom severity scale suggest that it does not.

The two data points Mahana includes on the website about these purported benefits are not the central findings for IBS symptom severity. To cite them out of context in this manner is rather misleading, as I have previously explained on Virology Blog.

According to Mahana’s website, “66% of patients reported significant and clinically meaningful reduction in the severity of their IBS.”

It is true that 66% of those in the web-based CBT arm who responded at 12 months had a reduction in the scores on the IBS Symptom Severity Scale of 50 or more points–the threshold for clinical significance. But it is not true that most of those changes can be attributed to the web-based program, which is what the statement implies. Mahana does not mention that 44% of those in the treatment-as-usual arm who reported at 12 months also had a reduction in score of 50 or more on the same scale. Given those numbers, it seems likely that many or most of the 66% in the web-based arm could have reported those improvements anyway.

Moreover, the site does not make clear that only 70% of the study sample provided data at 12 months. We can’t know what the final results for the remaining 30%–those considered “lost to follow-up”–would have been. That means we have no idea how those who dropped out from the web-based program arm felt about the intervention or whether it helped them. At 12 months, as I have already noted, the mean score for the web-based group in the intention-to-treat analysis was 35.2 points lower than for the treatment-as-usual group—quite a bit less than the 50-point difference that is considered a clinically significant improvement.

Mahana’s website also states: “On average, reduction in IBS severity was twice that of patients receiving medical care as usual.”

Again true, and again misleading. When improvements are small, improvements that are twice the size are also small. Just because something doubles does not mean the change is of much or any clinical significance. The more telling statistic is often not the relative difference between groups–the kind cited in the above claim about average reductions of IBS severity–but the absolute difference. In this case, as I’ve already noted, the absolute difference in score between the means of the groups was 35.2 points–well under the 50 points that is considered a clinically significant change.

Unfortunately for Mahana, this web-based CBT program cannot be accurately marketed as having clinically meaningful impacts on symptom severity beyond treatment-as-usual. The coronavirus epidemic certainly heightens the potential appeal of effective web-based therapies, but the data from this study cannot be twisted to mean what they don’t. Patients–and customers–deserve honesty, not hype.

Best–David