A comprehensive guide on spotting shoddy science, by example.
On August 16, 2018, The Lancet published a paper, titled Dietary carbohydrate intake and mortality: a prospective cohort study and meta-analysis
Immediately, people started reaching out to me with links from news sites saying things like “New study finds that popular low carb diets could shorten life.” Before reading the study, I already knew what kind of study it was… I considered this my call to action. Although the world is more open-minded than ever towards reconsidering stale public health recommendations, it’s evident that there is still a need for guides that can help keep minds open and progress on track.
This is my analysis and response towards the study and the researchers who were involved. This is my attempt at teaching the average Joe/Jane (maybe that’s you) to critically assess studies for themselves and come up with their own takeaways.
In part 1, we’ll look at the social context of this and similar studies by exploring:
- How irrelevant the study is
- The culture and process behind how a study is officially submitted, legitimized, & published
- How authors of a scientific paper might leverage it to get a different message across
- How all of the above applies to the new Lancet study
If you don’t read any further, if you only walk away with one fact, understand this: This study, in no shape or form, represents what the collective “we” refer to us a low-carb diet.
That fact doesn’t excuse the many other issues I have with this, and similar, studies. In my eyes, it certainly doesn’t excuse the behavior of the researchers outside of the paper itself, including first author listed in the study, Sara B Seidelmann, MD. Before going into that, though, let’s get this out of the way.
Potato Potato? Look’s the same to me!
A ketogenic diet is understood as is a diet that is low enough in carbohydrate intake to induce a metabolic state where the primary and preferred fuel sources of the body are fat and a byproduct of the breakdown of fat, called ketone bodies. The modern (as in “2018 modern”) interpretation of a low-carb is typically understood to be ketogenic. Finally, the upper limit, measured by weight, of carbohydrates that the average person can consume and still be in a state of “ketosis” is somewhere between 20-40 grams, as seen in countless other ketogenic studies. It’s common that the limit is towards the lower end of that range. This would conservatively be around 4-8% of your caloric intake on a reasonable, or even low, caloric intake.
Conversely, the study defines their “lowest carbohydrate” group consisting of the same 3086 people since 1987, as an average of 37% calorie intake from carbohydrates. The group’s self-reported average daily calorie intake was a mere 1558 calories (more on that absurdly low & unrealistic number in Part 2). Even with such a low average caloric intake, 37% carbs would be 144 grams of carbohydrate. Nobody on a low carb diet would dare call that a low carb diet.
“Potato Potato.” If you see that expression written out, the two words look the same. But when you experience it in person, you hear “po-TAY-toe po-TAH-toe.” If you haven’t read the study, “low carb” in the study looks like “low carb” in the wild.
This just scratches the surface of the flawed data interpretation. But this isn’t what Part 1 is about. Once again, we can revisit that topic in more detail in Part 2. Keep in mind, the rest of Part 1… is pretty much irrelevant now. There’s no need to further pick apart the study and the researchers in order to defend low-carb diets, given that we now know that the study had no low carb-diet group in it at all. But let’s do it anyway in case more studies like this pop up. Let’s talk about the researchers and the manipulated public perception of the study’s findings.
Q: How do you make yourself stand out?
Many things about the paper itself along with everything surrounding the media coverage of it give off a fishy smell. Science is supposed to be unbiased in its attempt to find truth. In this case, it appears to me like they had an opinion and intentionally framed existing data in a way that sounds, at the surface level, to prove that their opinion was correct.
Before I get into the specifics of what I mean, consider this. The #1 Google search for diet in 2018 is “ketogenic diet”. If you were a researcher or team of researchers trying to accomplish the following…
- gain recognition
- increase funding for your team
- make your current funding sources happy
…would you not target something everyone and their mothers are focused on? Especially when it’s as easy as rehashing old information and reframing it? To clarify, what I mean by old information is that the researchers used data from a different study and found what they were looking for within that old data. In this case, all you’d need to do is cause enough controversy over the ketogenic diet in order to accomplish your goals. That’s what it feels like the authors did here. I can live with that – ‘you gotta do what you gotta do’.
However, the way that this study was handled by the authors is NOT good practice in science…
A: Attack the most popular kid in the yard
Sidenote: The scientific community, those responsible for a paper's review process, will do their best to prevent duplicate studies from being published unless they feel that the new study has enough of a difference/distinction to be considered novel and beneficial towards science.
Definition: Abstract - Scientific term for an official synopsis of the rest of the paper
Definition: All-cause-mortality - All deaths, statistically, without regard to the exact cause of death. In this case, all deaths were tallied despite whether there is a clear link to nutrition or not
If you’re the researcher, in order to attack the ketogenic diet and make a splash, this is what you need to do. Suggest that what you consider “low carb” in your study is the same as what the rest of the world considers “low carb.” The fact of the matter is, the majority of people will never look further than the headline. If you read the news headlines, you’ll probably conclude that even the news sites didn’t read the study. Only an actual reader would have noticed that the study has one subtle scientific contribution that makes it “unique enough to be worth publishing”, compared to similar studies. The majority of the paper is spent suggesting that low-carb diets inherently are a primary cause of early all-cause mortality between their participant groups. The paper only contains a relatively small number of statements suggesting differences in all-cause-mortality based on fat/protein source (plant vs. animal), including within the low-carb group. The animal vs plant food source was the distinguishing observation that likely got the study approve, despite how little screen-time it was given. Otherwise, it would have been written off as a complete repeat study.
Here is how the summary starts off within the paper:
“Low carbohydrate diets, which restrict carbohydrate in favour of increased protein or fat intake, or both, are a popular weight-loss strategy. However, the long-term effect of carbohydrate restriction on mortality is controversial and could depend on whether dietary carbohydrate is replaced by plant-based or animal-based fat and protein. We aimed to investigate the association between carbohydrate intake and mortality.”
- Do they manage to tie what they define as low-carb to what is popularly low carb, despite the glaring fact that their low carb groups were not low carb? Yes – Check.
- Do they mention the distinguishing aspect of this study vs older similar studies (meaning the animal-vs-plant protein issue)? Yes – Check.
- Do they then overshadow that with their real objective in a closing sentence that brings the focus back to “low carb is the cause of mortality?” Yes – Check.
Great! They’ve ensured that people think this is about the ketogenic diet.
If their primary distinction is animal vs plant in low-carb-high-mortality populations, they should have left that last sentence out. The other studies that they reference have already tried to prove that last point (the key word there there is tried, by the way). If you read no further, you’d be left to think that this study is mostly about low carb in general. The proof is in the news response. Let’s look at some headlines:
- Low-carb diets could shorten life, study suggests
BBC News – Aug 16, 2018
- Low and high carb diets increase risk of early death, study finds
WPSD Local 6 – Aug 17, 2018
- Eating Some Carbs, But Not Too Many, Could Help You Live Longer …
Live Science – Aug 17, 2018
- Avoid low-carb diets if you want to live longer, study suggests
Springfield News Sun - Aug 17, 2018
- Sorry, Gwyneth. Diets that replace carbs with protein and fat are giving wrong advice, new study says
Galgary Herald – Aug 17, 2018
The requirements of fact
“But wait, isn’t that the media’s fault for not reading more carefully?”
Yes and no. As much as I hate to say it, what I’m about to highlight is a very common strategy, especially surrounding such a controversial topic as diet.
I am fortunate that my wife is a Stanford University PhD recipient (in Organic Chemistry if you’re curious) that has worked in a couple different academic research labs. She’s given me insight into the process of having your work published in a scientific journal. I can’t stress enough the importance of understanding that all legitimized papers go through a series of anonymous peer reviews and edits on behalf of the publishing journal. They make sure nothing blatantly incorrect sneaks by. Keep in mind, a journal’s reputation is on the line if they were to allow too much “bad science” to slip by. The reviewers make sure that the data is sound and factual and that if a claim is made, it is supported by that data. If and idea is not 100% fact, it has to be framed as a suggestion. As you might imagine, there is some grey area there.
Keep in mind, people that read these papers at the end of the day are NOT robots. Everyone will read a paper with a certain amount of bias and preconceived ideas and “truths”. Thus, it’s not difficult for a biased author to use language that is “swaying” to get the reader to take on the author’s own opinions, well beyond the facts.
Also, understand that doctors, scientists, and researches are all simply people with their own motivations at the end of the day. Sometimes it’s simply a student completing a thesis in order to graduate. Sometimes it’s simply getting published to flesh out one’s CV. Sometimes the biggest motivation is to be recognized. Dare I say that egos exist amongst researchers too!
Putting all of those things together, here’s the takeaway. Not every idea in a legitimized study is required to be 100% fact. There is a certain amount of story telling and embellishing of a message that a writer can get passed a review process. Ideas can be buried amongst facts, making it hard for a non-keen eye to spot the difference. This is the case for all studies, both well and poorly conducted. In fact, all studies at the very least have a “Discussion” section of some sort, usually at the end, where the authors can give a little bit of insight into what they think and how they interpreted the data. Although, that section is universally understood as being reserved for author interpretation, so it has less risk of opinion being mistaken as supported fact.
Token of Credibility
Part 2 will talk more about this type of study and its very valid place in science. Here’s the short version. The type of study I’m referring to here is called, an “epidemiology” study which is conducted via what’s called an “observational” study design. These types of studies play a critical role in the scientific process, but cannot and should not replace patient-based experimental studies that would aim to validate or invalidate the hypotheses derived from the previous studies. Basically, these studies look back at some preexisting population with preexisting habits in order to try and find certain patterns that are worth following up on. Observational studies are supposed to end there one they’ve found a likely pattern. Their purpose is not to prove anything, but to raise questions to be taken up by separate follow up experiments.
It’s not uncommon to see an observational study skirting around disguised as “proof.” Even without saying “this is proof,” an observational study can easily convince those that don’t know the difference that proof was found. Sometimes, this is not the fault of the researcher at all. It can sometimes the fault of the media that picks it up and spins the message. They “cherry pick” out of context bits that sound compelling.
But I hate to break it to you, sometimes this is the plan of the author all along. Sometimes all the author needs is a paper, any paper; it doesn’t even matter what the paper actually says, to an extent. Having the paper published gives them the spotlight and a token of credibility to say what they really want to say, unhindered by a peer review process.
Here’s an example of this in action from the first author herself, as quoted by BBC news:
“Low-carb diets that replace carbohydrates with protein or fat are gaining widespread popularity as a health and weight-loss strategy. However, our data suggests that animal-based low carbohydrate diets, which are prevalent in North America and Europe, might be associated with shorter overall life span and should be discouraged.” - Sara B Seidelmann, MD
Ok, so she managed to make a mention of the animal/plant-based part to check that off. The cynic in me also believes that she also did do her best to make sure the audience attributes what she’s talking about to all “popular low-carb diets”, the ketogenic diet included. What else is said by the authors?
“These findings bring together several strands that have been controversial. Too much and too little carbohydrate can be harmful but what counts most is the type of fat, protein, and carbohydrate,” - Prof. Walter Willett, co-author of the study
Here are the reasons I am pointing these quotes out. Scientists and researchers are our key-holders into a greater future. Along with the responsibility of reporting on factual findings, they also bear the responsibility of being incredibly careful with their words. Scientists (authors and researchers) are societally trusted sources because of their hard-earned credentials. With such trust, they must be socially responsible with their ‘off the cuff’ comments, which are often extrapolated from what is written in their papers. Remember that an observational study is the basis for a hypothesis, not a basis for a conclusion. Yet both the author’s used phrases like “should be discouraged” and “but what counts is” as if they have made up their minds, definitively. This should be a red flag.
Decades ago, Sen. George McGovern is quoted saying, "Senators don't have the luxury that the research scientist does of waiting until every last shred of evidence is in." Although I agree that it is impossible to wait for every shred of evidence, McGovern waited for almost no evidence at all. This was the committee and mentality that led to premature recommendations to cut fat out of the diet which then led to an increase in carb consumption and downward spiral of public health. Let's not repeat history. Read more about it from this well cited article found here.
When your words are intended for the trusting public eye, definitive words like “is” and “should” must be avoided in the case of observational-study-generated-hypotheses. Suggesting recommended courses of action to the public during the hypothesis step is an even more egregious act, if we are supposed to follow The Scientific Method. Many of these “is/should” statements stem from a bias based on older conventional theories, theories which are not addressed by their data at all. The study is not a clinical experiment that aims to directly test any theories whatsoever. The authors are overstepping the bounds of what their study has allowed them to hypothesize by suggesting courses of action. Historically, recommending courses of action before a hypothesis is properly tested has lead to disaster. Just look at what the McGovern Report has done for our diet recommendations for the past 5 decades.
The damage is done
What’s the big deal? You’re nitpicking on tiny words! They mentioned the distinction of animal vs plant food sources. Medical doctors are allowed an opinion, too! Well…
Just as we were finally making progress.
What was in that study anyway?
We will take a critical look at that in Part 2. If you’ve never read a study before, get ready. If you have, still… get ready. It can be daunting to assume that you, the layperson, can sift through technical literature and come to your own legitimate conclusions, but I’m here to show you that unless it’s rocket science, it’s not really rocket science. It’s also worth mentioning, some of what I mentioned in Part 1 may have seemed nit-picky or overly speculative of the researcher’s intentions, but once you see the actual information in the study, you’ll likely agree that the entire thing is suspect.