Memory based dietary assessment methods such as NHANES are “fundamentally and fatally flawed”, argues researcher

In a critical review* led by Edward Archer PhD, MS (a research fellow at the University of Alabama at Birmingham), and published in the journal Mayo Clinic Proceedings​, ​the authors argue that memory based dietary assessment methods (such as the 24-hour dietary recall surveys used in NHANES, and food frequency questionnaires used in epidemiological studies) are “fundamentally and fatally flawed​” and should not be used to inform public policy such as the dietary guidelines for Americans.              

NHANES data does not correlate with objective measures of energy and nutrient intakes​

The evidence is now overwhelming that NHANES 24-hour dietary recall data (collected by USDA researchers who ask Americans what they ate the previous day) bears little relation to objective measures of the actual energy and nutrient intakes of the same subjects, claim the authors.

For example, self-reported energy intakes reported in a high percentage of NHANES records are lower than the subjects’ actual energy requirements (which are calculated using validated techniques based on basal metabolic rates), Dr Archer told FoodNavigator-USA.

And this would mean that most people are losing weight. Yet we know that the opposite is true.   

Such contradictions are also evident in scores of studies in which free-living subjects’ self-reported energy intakes (based on dietary intake surveys & food frequency questionnaires) are routinely lower than even their resting​ energy requirements (eg. the number of calories they need to maintain their current weight if they do literally nothing all day​), yet body-weight data collected by researchers show the subjects in question maintaining or gaining weight, which proves they are misreporting energy intakes, he said.

Misreporting is systemic​

A 2013 study​​ led by Dr Archer and published in PLOS One revealed that across the 39-year-old history of NHANES, energy intake data from 67.3% of women and 58.7% of men was “not physiologically plausible​”, he added.

And some groups – for example obese women – are under-reporting up to 865 calories a day, making their dietary recall data so inaccurate that it is arguably “completely​ worthless​”, Dr Archer told FoodNavigator-USA.

But this isn't just about calories, he said. Self-reported dietary intake data is also used to estimate population-wide nutrient intakes (eg. Are Americans getting enough calcium, iron, zinc?), which then feed into public policy, "so that policy is based on flawed data".​

NHANES data is not ‘better than nothing’​

But what does he say to researchers arguing that surveys such as NHANES are a pretty good approximation​ of what Americans eat, as long as we take into account that respondents typically under-report calories by a certain percentage, for example?

“I would argue that they are not 'better than nothing'​,” said Dr Archer. “They are worse than nothing, because they give us the illusion of knowledge.”​

Moreover, “there’s no such thing as typical under-reporting,​” added Dr Archer, who has a PhD in exercise science and Bachelor's and Master's degrees in psychology, and spends much of the paper exploring why memory-based recall data is unreliable. 

If we knew, for example, that everyone under-reports calories by 10%, the data might be more useful, he said.  

But he added: “The problem is that there is over-reporting and under-reporting in every category and in every population subgroup, and there is no pattern that’s consistent over time.​ [Misreporting also varies between countries and cultures, he said, citing data from 10 different countries].”

There is a huge amount of institutionalized inertia​

Meanwhile, he claimed, “Physical activity, cardio-respiratory fitness and exercise are major determinants of health and are largely ignored or improperly measured by federally funded nutrition researchers.”​

So are memory-based dietary intake surveys really a complete waste of time? As a tool for accurately reporting what Americans eat, yes, he argued, “although they do provide an interesting insight into people’s perceptions​ about what they eat.”​

But if this misreporting is widely acknowledged (Dr Archer cites scores of papers to this effect), then why aren’t more people listening?

According to Dr Archer: “There is a huge amount of institutionalized inertia. If you’ve spent your entire career using this data, you’re not going to stop now.”​

Adam Drewnowski: 'Everyone should just take a deep breath and relax.'​

However, one researcher who has conducted scores of analyses of NHANES data, told FoodNavigator-USA that “everyone should just take a deep breath and relax.”​

Adam Drewnowski, PhD Director, Center for Public Health Nutrition at the University of Washington, said: “First, NHANES is a pretty good approximation of what people eat and a great tool of monitoring how the US diet has changed over time. ​

“Second, people not remembering what they ate yesterday is NOT the major problem.  The problem is that what they ate yesterday may bear no relation to what they ate the day before and the day before that.  So one day of intake (or even two) does not give you a full picture for any one individual - but does provide an OK estimate for the population. There are modeling methods that estimate "habitual" diet from NHANES data.”​

He also noted that the dietary guidelines are not based exclusively on NHANES data (the DGAC report for example frequently refers to the CDC’s data on biochemical indicators of nutrients).

USDA ARS: NHANES is designed to provide population means of average dietary intake​

Meanwhile, David M. Klurfeld, Ph.D., National Program Leader, Human Nutrition at USDA’s Agricultural Research Service, told FoodNavigator-USA that there was “nothing new​” in Dr Archer's paper, which he claimed “ignored that NHANES is designed to provide population means of average dietary intake rather than their incorrect extrapolation to individual values”.​

He added: “The assumption Dr. Archer used that we need 100% accuracy for individuals to arrive at the population average is incorrect. Two recalls, as currently used in NHANES, from the USDA Automated Multiple Pass Method (AMPM) generate accurate energy reporting from 78% of men and 74% of women. Knowing the degree of under-reporting, primarily influenced by body-mass index and gender, allows adjustment of the population average to approximate the true intake.”​

Meanwhile, USDA validates its techniques, he said: “USDA published validation of the AMPM in 2008 for estimating total energy in over 500 subjects using doubly labeled water, the gold standard for estimating energy requirements. For the total sample, underestimation was 11%; for normal weight people the error was only 3%.​

“Concluding that memory cannot be used for assessing dietary intake is analogous to recommending that physicians no longer take a health history from their patients.”​

*​Archer, E., Pavela, G., Lavie, CJ., Blair, SN. The Inadmissibility of ‘What We Eat In America’ (WWEIA) and NHANES Dietary Data in Nutrition & Obesity Research and the Scientific Formulation of National Dietary Guidelines.​​ (Mayo Clinic Proceedings)