In case you missed it, there’s a 1400 word comment from Prof. Tim Noakes on my previous blog post. Seeing as the bulk of his comment is entirely unrelated to the subject of that blog post, I thought it offered a handy opportunity to provide an additional example of reasoning gone wrong, this time both in basic logic, and again in science. (Also, I’d need another 800 or so words to match his word-count.) So, below you’ll find block-quotes of his full comment, and an explanation of the errors committed. If you want to see his quote in context, please visit the original piece, Lessons in bad science – Tim Noakes and the SAMJ.
Apologies, but this will be somewhat lengthy. I’ll try to keep each unit of quote and response comprehensible on its own, though. Those of you who get bored, please do scroll down and read the bit headed (in bold) “A very important bit” before you leave.
And a reminder – my post the other day wasn’t about the diet itself. This post isn’t about the diet either. It would be fantastic if Noakes/Taubes and the rest were correct in this instance, in that they would have re-discovered or popularised a highly cost-effective way to treat an a highly significant public health problem. Or, various problems, including obesity and diabetes. That would be something to be celebrated, and I’ll be one of those celebrating.
However, if we can arrive at that outcome while supporting (and reinforcing) the scientific method and basic logic, surely that’s an even better outcome?
Noakes begins by re-stating a powerful anecdote:
What is really so funny is that this is a report of how 127 people felt their lives had been dramatically improved by following a particular diet. Included were 14 who claimed they had been “cured” of Type 2 diabetes – confirmed in 3 cases I investigated further. To my knowledge the SAMJ has never before carried a report in which patients with an “incurable” condition (type 2 diabetes) were cured of that condition. One doctor who had told his wife he would be dead in 7 years because he had 5 “incurable” conditions, was completely healed of all conditions (no more medications required) when he restricted his carbohydrate intake.
Everything else he had ever tried (according to his conventional medical training) including treatment by the best medical specialists in Cape Town had done little for his health. Naturally this medical practitioner who had never in 57 years been exposed to this information (why not?), concluded that the dietary advice I gave him had produced a “miracle”. He now includes this method in his treatment options for his patients with obesity/diabetes/metabolic syndrome. He now informs me at least monthly of how much success he is having with this dietary treatment for these patients.
After the words “felt”, “claimed” and then the quotation marks around “cured” in the first two sentences, the thing you’ll note about this anecdote is that it uses language that is entirely inappropriate to the level of evidence available. Imagine yourself to be someone who has Type 2 diabetes, or who is overweight, and who then reads the two paragraphs above. If your level of scientific literacy was that of the average person, you’d come away thinking that there’s something akin to certainty that this diet is effective.
If you’re a marketer, this tactic is completely understandable, and appropriate. But science should be a domain of reason and evidence, not of hyperbole, and not of presenting contested evidence as if it obviously demonstrates something that it is not known to demonstrate. Second, as I’ve said in my original piece, if the evidence exists, you wouldn’t need the anecdotes. Unless, of course, Noakes has so little confidence in the acumen of his peers that he thinks that they would be equally persuaded by either.
Paragraph two contains the quite typical injection of conspiracy theory, with its suggestion that something must be afoot for this medical practitioner to not have heard this dietary advice before. And yes, it’s possible that something was and is afoot – that there is a systematic bias against this approach to diet. If so, that’s a problem that should be remedied. But it has no bearing on whether the advice is in fact good or bad advice. It can have a bearing on how much evidence we have, in that research might have been stymied or inappropriately directed. (Noakes, however, keeps insisting that the evidence is clear, so it seems that this problem is surmountable.)
I wrote the article to alert my colleagues to the fact that there is a simple dietary option that might be able to reverse the very conditions that our profession finds so difficult to treat – obesity, type 2 diabetes, metabolic syndrome. I also referred to the extensive scientific literature showing why and how this dietary intervention does and should work for people with these conditions. The explanations are simple, obvious and proven.
But then perhaps you need a training in the medical sciences to understand those mechanisms, Without some understanding of biochemistry, it is not possible to follow that argument. What one cannot understand, one naturally dismisses as irrelevant.
As I’ve asked before, if the “extensive scientific literature” makes the case, why do we need the anecdotes? As above, the use of the word “proven” massively overstates the case, and again indicates a failure to understand why he comes across as pseudoscientific in these regards. If it were “proven”, there would be far less disagreement in the medical profession than there seems to be about this sort of diet’s efficacy.
In general, though, to quote from a comment I made on the previous post, “the issue is that if non-anecdotal evidence for the same conclusion exists (as he asserts), that evidence would be sufficient by itself. So, either it doesn’t exist, and he needs to rely on the anecdotal evidence (which teaches a bad lesson regarding scientific reasoning), or it does exist, but he thinks that the anecdotal evidence adds weight to the conclusion (which it doesn’t, as he should know)”.
The last paragraph contains another fairly typical tactic for Noakes, namely an attempt to discredit an opponent through focusing on something irrelevant or personal, as we saw in the “fat-shaming” comment in the previous post. Because there are two possibilities here: first, that he’s right that we need medical training to understand this. If so, I’m mystified as to why Noakes does all this public speaking to laypeople, and also that he writes on these matters in (almost exclusively) lay publications and books as much as he does, seeing as he knows none of those audience-members can understand what he’s saying.
Alternatively, he’s subtly suggesting I’m too thick to “get it”. But again, seeing as my post was not about the diet, but about what evidence and arguments look like, I’d have to protest and tell him that I “get that” very well, and that of all the medical practitioners who have commented or Tweeted about the post in question, everyone except Noakes thinks I’m on to something. Not, I again remind you, about whether the diet is good or bad, seeing as I don’t express a view on that, but simply that this sort of “research” or “study” sets a very low standard in terms of what we should aspire to as scientists.
At no point in the article is the claim ever made that this is an attempt at a scientific proof of a particular diet. That is why the title includes the words – Occasional Survey. It is simply a group of case reports showing that some patients achieve remarkable cures for their intractable medical conditions simply by following advice, the key point of which is that it normalises hunger. For the truth is that these patients are not dying of obesity etc, so much as they are dying of hunger. Once their hunger is controlled by simple dietary advice, they can start to cure the conditions caused by the overconsumption of addictive, highly processed, carbohydrate-rich foods (made worse by their insulin-resistant state).
I have been in science long enough to understand how people try to divert attention from the message. I wrote about this extensively in Challenging Beliefs. First they always question the methods. The methods I used in this study are entirely appropriate for the extremely limited goals of this paper. That simple goal was to show that some people benefit dramatically and in some cases miraculously from this simple advice. Whether or not they would have benefitted equally from other advice is utterly irrelevant since I am not trying to prove (in this article) that one treatment is better than another. Of course I would guess that 100% of the 127 had all tried the conventional advice and it had failed for them. But I only made that claim if I it was supported by the information I had.
A group of “case reports”? I don’t know about you, but that seems an awfully strong description for a series of self-reported and completely uncontrolled and in most cases unverified narratives. But it’s nevertheless a legitimate description, with a track record in medical literature.
However, because a large group of case reports, as in this case, can create an impression of generalisability or significance where there might be none, we find Johns Hopkins, for example, requiring IRB (institutional review board) clearance for any case studies (or a case series, in this instance) involving more than three participants. No clearance is mentioned in this case.
Again, I remind you that I’m simply saying that the study offers little of scientific merit, and that the SAMJ erred in accepting it for publication – not that the anecdotes are false. (It’s that we can’t know whether they are false or not that is part of the problem.)
As for “I have been in science long enough to know” – I refer you to the point about deflection and conspiracy mentioned earlier. The fact that methods are questioned isn’t evidence that the scientist is a martyr for the truth, as Noakes seems to want to imagine himself here. As Occam’s Razor suggests, it might also be because the methods are questionable.
It’s entirely relevant to question whether the participants would have benefited equally from other advice, precisely because 127 cases offers an impression of significance or generalisability. The goal was “to show that some people benefit dramatically and in some cases miraculously from this simple advice” (my emphasis) – and how do you show that through 127 unverified self-reported anecdotes? If the science already shows this, then it can stand alone, with the anecdotes as illustrations if one so desires. Noakes says (in the paper) that this “data” is “of value” and “challenges current conventional wisdom” – and yes, it would, if we had reason to believe it was replicable. It might well be replicable, but the anecdotes are not evidence for that conclusion.
A key point about South African medical ethics is that if there is more than one treatment options it is ethically unacceptable for a South African practitioner to prescribe only one. My ethical responsibility as an educator and scientist is to bring the attention of my colleagues to the established fact that there is more than one option for the treatment of obesity, diabetes and metabolic syndrome and that the scientific evidence for this is well established in the literature (as recently accepted by the highest Swedish medical authorities).
Having been involved in high-level research ethics myself, of course I’d agree in the main. Except, Noakes is leaving something crucial out of the summary: it’s not only when there is simply “more than one treatment option”. Instead, it’s when there is “more than one effective/proven/viable/etc. treatment option”. This might well become known to be one of those options, perhaps even the best one. But it isn’t known to be that as yet, which is a reality Noakes again evades in the above quotation.
A very important bit
Above, Noakes says “as recently accepted by the highest Swedish medical authorities”. This, in a nutshell, demonstrates his rather casual relationship with reality when it comes to promoting the conclusion he wishes to. You’ll note, as a starting point, that the language is unambiguous – a trusting reader will be left utterly convinced that the Swedes have accepted LCHF as obviously the recommended diet. So, let’s look at the evidence. The quote from his paper reads as follows:
The Swedish National Board of Health and Welfare has concluded that ‘low carb diets can today be seen as compatible with scientific evidence and best practice for weight reduction for patients with overweight or diabetes type 2, as a number of studies have shown effect in the short term and no evidence of harm has emerged … ’
It’s a direct quote, so you’d expect a reference (and quotation marks, which might look a little alien to some potential readers). We have both in this case, and the reference given is to the Swedish Board in question…. oops. No, sorry, my mistake – the reference is to a blog post titled Low-carb for You. The Swedes are eating more butter! In another interesting development, the full quote reads (my emphasis):
Professor Christian Berne, one of Sweden’s leading diabetes experts, had carefully investigated the case against Dr. Dahlqvist and presented his findings to the Swedish National Board of Health and Welfare. He said, “…a low-carbohydrate diet can today be said to be in accordance with science and well-tried experience for reducing [obesity] and type 2 diabetes…a number of trials has shown no effects in the shorter run and that no evidence for it being harmful has emerged in systematic literature researches performed so far. [There is] no scientific support yet for treatments in excess of 1 year. A thorough evaluation of long time treatment results is therefore an important demand on the practitioner.”
So what we learn here is:
- In the source Noakes refers to in order to support a very strong claim, we find Berne reporting findings to the Board
- But that quote is presented by Noakes as a resolution of the Board, rather than an opinion expressed to the Board
- An important bit of the quote is left out, because it’s inconvenient (namely, that there is no scientific support for treatments “in excess of one year“)
- Notice that this question – around long-term efficacy – was a central theme of my previous blog post that inspired this Noakes essay in response – and his own source makes the same point
- Lastly, we learn that quotation marks don’t mean the same thing for Noakes as they might to you, in that a sentence like “a low-carbohydrate diet can today be said to be in accordance with science and well-tried experience for reducing [obesity] and type 2 diabetes” morphs, in Noakes’ version, into “low carb diets can today be seen as compatible with scientific evidence and best practice for weight reduction for patients with overweight or diabetes type 2” (I’m not asserting here that he changed the meaning – it’s just odd to quote-and-not-quote at the same time).
The SBU (Swedish Council on Health Technology Assessment) have however published guidelines on diet and obesity – but unfortunately, they’re not yet available in English, so Noakes couldn’t easily have quoted from them in an English-language journal. But let’s see what Google translate can do for us, seeing as we’re able to take such liberties on a blog post.
“SBU has previously addressed food for people with diabetes [link to English pdf]. The results for people with obesity and diabetes, pointing at large in the same direction.”
That direction, according to the English diabetes report, includes bits like this (my emphasis):
- Scientific evidence is not available to evaluate the long-term safety of moderate and extreme low-carbohydrate diets. This includes cardiovascular morbidity and other complications of diabetes.
- There is strong scientific evidence that lifestyle intervention, combining a low-fat diet with high fiber intake and increased physical activity, prevents diabetes in people that would otherwise be at high risk for the disease
Back to the Google translate version of the obesity report, which also says (my emphasis):
- In the short term (six months) is advice on strict or moderate carbohydrate diet more effective for weight loss than low-fat diets advice. In the long run there are no differences in efficacy between weight loss tips on strict and moderate carbohydrate diet, low-fat diets, högproteinkost, Mediterranean diet, diet focuses on low-glycemic load diet or a high proportion of monounsaturated fats.
- After that obese people have lost weight they can maintain their weight better with advice on low-fat diets with low glycemic index and / or high protein content than with low-fat diets with high glycemic index and / or low protein content. There is no basis for assessing whether even advice, eg, low-carbohydrate diet and the Mediterranean diet is effective in preventing weight gain after weight loss
Once again, it might one day be common knowlege that Noakes et. al. are right. But that day doesn’t seem to have arrived yet, and in the meanwhile, it certainly looks as if support is being appropriated where it doesn’t quite (as yet) exist. This, again, is bad science, if not simply dishonest.
Back to the Noakes comment
The reasons why this information is not taught more widely across the world is not material to this article and whether or not there is a conspiracy is not relevant. The point is that students in South Africa (as in most other countries) are currently taught only one side of a two-sided story. As far as patient care is concerned, that is unethical.
Sentence one, and then two and three seem somewhat contradictory. If students are being misled into providing unethical patient care, surely that must be relevant? But in any case, my point was that if something was obvious, and as evidence-based as Noakes keeps asserting, it would be taught all over the world. He constantly refers to conspiratorial reasons why that isn’t the case, rather than considering the possibility that others don’t think the evidence is as clear as he thinks it is.
Three years ago I decided that it my ethical responsibility to acknowledge publicly that my advice on high carbohydrate diets for runners, widely read in Lore or Running, was wrong for those with insulin resistance/type 2 diabetes/metabolic syndrome since it would contribute to their ill-health in the long term as it has to mine. This article is one outcome of that admission.
I could have kept quiet and hidden my error but I chose not to. Now that this article has been published in the SAMJ (and I have spoken about it at the most recent SAMA conference), South African medical practitioners, perhaps for the first time, have been exposed to the evidence that there is an alternative option that they might like to consider in future for the treatment of these conditions.
The result is that if the 127 patients reported here are any indication, many patients in South Africa with these conditions will be offered another treatment option that before they would not have been offered. I suspect that many will do much better on that therapy than if they continue to follow advice that does not work (for them).
Nothing to add here, except to repeat that the point is precisely that we have no reason to believe that the 127 patients reported here in fact are any indication.
So this focus of this discussion should not be about whether or not I am a good scientist who understands what is and what is not good science.
Erm, no. I choose what the focus of the discussion is on my own website, thanks.
Fortunately in science, there are simple markers of our standing as scientists that are based on hard measureables and not on the opinions of others. These are the h-Index and the number of citations. Anyone who wishes to determine my status as a scientist is welcome to find those numbers and what they mean. Those are measures of scientific influence over a life-time, not as the result of one single good or bad article.
Yes, they are indeed “simple” markers. “Crude” might be another appropriate word, as Prof. Noakes knows full well. They offer a valuable heuristic in making snap judgments, but they aren’t any sort of guarantee of sense or quality in perpetuity. I don’t dispute, and have no reason to dispute, that Noakes has done tremendous work in the past. But that tells us nothing about the present topic, except to make it statistically more probable that he’s worth paying attention to here than many others might be, because of that track record. This doesn’t mean that – once you look at a particular case – you need to grant extra authority to that person if they present weak arguments.
To do so would be to commit the informal fallacy of making (or rather, falling for) an appeal to authority. It’s particularly disappointing when the authority him or herself makes the appeal on their own behalves.
It is sad that this article which should be a celebration of how simple dietary advice may be able to reverse intractable medical conditions in some people (it would have been valuable even if it had reported just a single “cure”) has been used by some to argue what a dreadful scientist I am, who is trying to push some sort of devious agenda that has no scientific basis.
It’s arguably more sad when eminent scientists start practicing bad science, and then doubly so when they defend it as weakly as this. “This article”, meaning my blog post, is about that topic, not about the diet. Readers, or subjects of a post, don’t get to say what a post “should be” about. Furthermore, nobody is being described as being “devious”, or having an “agenda”. Someone is being described as making poor arguments for a conclusion.
My agenda is clear. I want my profession to teach more than one option for the management of obesity/diabetes/metabolic syndrome and to understand that our current dietary advice is in my opinion the cause of so much of our ill-health.
My agenda is also clear. I teach critical thinking, and the Noakes paper, and his responses to criticisms of it (and, general criticisms on social media) provide great classroom examples of how reputations are no guarantee of good sense.
I have spent 3 years researching this topic and am happy that the scientific evidence supporting this position is as powerful as any evidence I have touted in the past (see Challenging Beliefs). However the topic is much more important that anything I have ever tackled since it is the single most important medical problem in the world and is currently out of control and getting worse by the day.
Agreed entirely, which is why I want the science supporting it to be beyond reproach, especially on such obvious grounds.
What this paper shows it that there may be simple answers for what seem to be intractable conditions. That is why the final sentence of the article calls for a properly funded and designed study to test the hypothesis (not proof) advanced by the finding of this Occasional Survey.
As argued above, the occasional survey doesn’t present a case for doing so, because of the quality of the data. Furthermore, we are told by Noakes that non-anecdotal evidence exists – and if this is the case, the anecdotes are superfluous. This was the topic of my last post, so I won’t repeat the argument here.
I would be only too happy if that trial disproves me. But if it shows that a carbohydrate-restricted diet can reverse intractable obesity and some cases of Type 2 diabetes, then we will have shown that the causes of the obesity and diabetes epidemics are much simpler than we believe and that we might be able to do something to protect our future generations from these diseases. Of course, it there is a conspiracy, then it will do all it can to insure that we do not ever make that finding.
Agreed entirely. And of course a scientist should be happy to be corrected (even if it’s sometimes difficult to swallow).
Perhaps we can move the debate forward by focusing on what this paper actually found and how that might be of value in trying to understand what is causing the obesity and diabetes epidemics globally. Then we will be making a positive contribution to the future health of the world.
No, there are different debates. Noakes, and his peers, can move that debate forward. My focus is critical thinking, logical fallacies, the standard of education and so forth. That’s the debate I’m going to “move forward”, and these sorts of examples are great resources for doing so. In fact, they allow for us to make a positive contribution to the current – and future – health of the world too, by helping to inculcate and reinforce clear and cogent reasoning, without which medical science is doomed.
Thanks for the opportunity to express myself more fully and I look forward to your contribution to that agenda should you think it sufficiently important.
It’s vital. But also, that agenda is not my field. This one is.
The various Faculty Boards at the University of Cape Town are currently considering alternative models for UCT student admissions. These models arise from a debate the University has been having for some time now, regarding whether “race” is still the most effective identifier of likely disadvantage available to us. Some participants in this debate argue that race has become an increasingly crude proxy for disadvantage, resulting in a large number of false positives (which in turn has the effect of shrinking the number of places available for people who are actually disadvantaged, whatever their race might be).