The art of the absence of evidence

Barrett Newman’s Voice of Fire: http://en.wikipedia.org/wiki/Voice_of_Fire#/media/File:Voice_of_Fire_photo.jpg

This study comes from Ian Lane who asked whether the authors were, “entitled to believe that their conclusion is ‘true’.”

The more you learn, the more you realize the limitations of what you know. You can’t know your limits until you’ve pushed up against them and you realize there’s a spot where the rules are not as hard nor fast. That being said, when it comes to interpreting scientific literature, there are no hard and fast rules once you get past the basics (the problem being that many people haven’t gone past them). My statistics professors often said something along the lines of, “…and this is where we get into the art of statistics…” because there are often many options on what direction to take an analysis and making the choice is an art, not a science.

I like this study because it is an illustration of how the art of interpretation works. It’s a case study. I neither endorse nor deny the claims of the author in this case. I don’t think this study is conclusive, but it might be ‘conclusive enough’.

Johnston CS, Tjonn SL, Swan PD et al. Ketogenic low-carbohydrate diets have no metabolic advantage over nonketogenic low-carbohydrate diets. American Journal of Clinical Nutrition, 83:1055-61, 2006. http://ajcn.nutrition.org/content/83/5/1055.full.pdf+html

I like that this is an old study because it means that there have (hopefully) been developments since its publication and that the information in it might be somewhat obsolete, because the focus of this entry is not on the conclusions. I’m not going to focus much on the study population, or even most of the outcomes. This is more about seeing the forest, not the trees.

Methods (in brief):

Subjects with BMI>25 were recruited for the study, stratified by age, gender, and BMI. They were then randomly assigned to one of two diets: a low-carb ketogenic diet, or a low-fat non-ketogenic diet. If you want the specifics of the diets, you can look it up.

This was a 6-week trial with subjects receiving all of their food from the research team. All subjects were classified as ‘sedentary’ and remained that way throughout the study. A further 10-week endpoint was also defined, with subjects being told to follow their assigned diet for an additional 4 weeks on their own.

Body weight and ‘fat mass’ were measured using a bioimpedance scale ever week. Subjects also filled out questionnaires about how they felt over the last week in terms of mood.

The researchers did report a power calculation, but not a sample size estimation. They estimated that with 20 subjects, that they would have a 70% power to detect a change of 0.6mmol/L change in LDL cholesterol. They did not comment on their primary outcome of their conclusions which was weight loss.

[I’m not sure why this is meaningful. A 70% power is way below the usual standard and definitely points to a hypothesis-generating type of study, as opposed to a confirming one. Flag number 1.]

Results (just the major ones):

Between the two diets, the average weight lost in each group over the first 6 weeks was 6.3kg (SD=0.6) in the ketogenic group and 7.2 kg(SD=0.8) in the non-ketogentic group. This difference between group was not (unsurprisingly) statistically significant.

The average weight lost in the subsequent four weeks was 0.1kg in the ketogenic group and 1.4kg in the non-ketogenic group (variances not reported). This difference between groups was also (unsurprisingly) not statistically significant.

The difference in LDL cholesterol levels (reported graphically, but no actual numbers reported) were (unsurprisingly) not statistically significant. Graphically, they looks quite different, with the mean LDL cholesterol hovering just under 4mmol/L in the ketogenic group and just under 3mmol/L in the non-ketogenic group.

[The authors also reported that LDL cholesterol levels were correlated with beta-hydroxybuterate levels, which DID differ in a “statistically significant” way between groups, but what this means, or why it’s relevant, I have no idea, since LDL is the factor of interest, not a possible proxy for it. Sorry, but if the direct measurement doesn’t pan out, you don’t get to fall back onto a proxy; unless that proxy itself is a direct risk factor, in which case, it’s not really a proxy and the correlation becomes unnecesssary. Flag number 2.]

Authors’ conclusions

The first sentence of the discussion is, “These data show that, under isocaloric conditions, total weight loss and fat loss did not differ significantly by diet treatment.”

This is the crux of the paper. The question that remains to be answered is whether or not this statement is supported.

The obvious criticism of this paper is the low sample size. Twenty subjects hardly seems conclusive. Also, there’s the “rule” that you, “can’t positive prove a negative.” For proponents of ketogenic diets, this seems like ripe ammunition to throw the study out. Admittedly, there are reporting issues here as well which make some of the assessments of the authors’ statements difficult (standard things like how subjects were randomly allocated, what level of BMI were they stratified on, who was blinded) The two flags I’ve written above also make me leery of this study—I feel that reporting extraneous details without explaining why they’re there points to a larger issue of possible poor understanding of the implications of those details.

But a low sample size isn’t a hard and fast rule. For instance, if a study shows that a meaningful difference between two groups is also “statistically significant” then regardless of the sample size, it had enough power to detect that difference statistically. There might be issues with _generalizability_ (the degree to which the results can be extended to people who were not directly in the study), but from a significance point of view, it’s all good.

Similarly, in this study, the low sample size isn’t necessarily a fatal blow. This is where things get artistic. Art is perceived after a deeper understanding of context. What looks like a red line on a blue background can be child’s drawing, or a work of profound meaning. So what’s the context here?

For weight loss in the first 6 weeks: An average difference of roughly 1kg between groups after 6 weeks of dieting in sedentary, overweight people (possibly in favour of the non-ketogenic diet.) I think you’d be hard pressed to sell that as a relevant difference.

For weight loss in the 4 weeks after: Virtually no weight loss in the ketogenic group compared to 1.4kg or 3 pounds of weight loss on a self-administered diet over 4 weeks in the non-ketogenic group. There’s a part of me that says that this is relevant. SOME weight loss is better than NO weight loss and 3 pounds over 4 weeks with whatever imperfections the individual brings to the table when they do it on their own might actually be relevant, particularly, if it has the potential to spawn more adherence and commitment to what could be a sustainable change. Lots of possibles here, yes; but that’s how the art of science is done, folks.

For LDL cholesterol: It’s been a long time since I’ve had to look at cholesterol numbers. So I asked Spencer (http://drspencer.com) what his opinion was in terms of a meaningful reduction in LDL if one started around 4mmol/L. His answer (not knowing I was talking about this study or any study in particular) was, “3-3.5mmol/l”; which _was_ achieved in this study (but by the non-ketogenic group.) So, again a relevant difference, but insufficient power (which we knew was going to happen when they mentioned their low power problem at the beginning.)

So where do we go from here? In a ‘perfect’ world where prepared food appears at your door, it seems unlikely that there is a difference between ketogenic and the non-ketogenic diets used in this study over 6 weks. While you COULD say that a larger study is needed to detect a difference of roughly 2 pounds, it doesn’t seem that that is going to be a useful study if these numbers are accurate in terms of the population-at-large. Even if you could show a statistical difference of an additional 2 pounds lost, it’s not likely going to be in favour of ketogenic diets since the difference is so small.

What is question-generating in this study, is the fact that non-ketogenic dieters appears to continue to lose weight while the ketogenic dieters might not have when left to their own devices; and that there might be substantial LDL benefits to the non-ketogenic diet compared with the ketogenic one. This is where things start to get interesting and the “more research is needed” statement appears because we don’t know whether these differences are truly present outside the study sample in the general population or whether they’re just flukes.

However, I don’t think the authors overstate the limitations of their study when they state that there does not appear to be any “metabolic advantage” with ketosis dieting when compared to their non-ketosis diet—because for all intents and purposes, there really doesn’t appear to be one even after taking context into consideration; despite the methodological issues with the study itself.