Saltar al contenido principal
Pan y Hierro Nutrition and exercise with sources

Blog

How to read a nutrition study without getting fooled

The CONSORT and STROBE checklists to judge a trial or an observational study, and the four questions this site asks itself.

An article on the front page says “supplement X improves memory”. Another says “food Y is associated with less disease”. Both are published, both have a journal behind them, and both can be misread. This article explains the tools that exist to tell them apart.

Two checklists that solve half the problem

There are standardized reporting guidelines which are, essentially, checklists of what a study should have reported. They are what this site uses when deciding whether a figure is publishable.

CONSORT is the reporting guideline for randomized trials [1]. The current version is CONSORT 2025 [6], which replaces the 2010 one, which is limited to parallel-group trials [1]. It requires things like:

  • How participants were assigned to groups (real randomization).
  • Whether blinding was used, and to what degree.
  • How many participants entered each group and how many were lost.
  • The sample size calculated in advance.

STROBE is the reporting guideline for observational studies [2], which is a different kind of question. It requires things like:

  • Design and setting, with data sources and dates.
  • Variables: how outcomes and predictors were defined.
  • Bias: what was done to control confounding.
  • Sensitivity: whether sensitivity analyses were done.

The difference between the two is not a detail: a randomized trial and an observational cohort answer different questions and have different biases. An article that mixes them, or that calls something a “study” without saying which kind, is already telling you something.

The four questions this site asks

When evaluating a figure before publishing it, this site always asks the same four questions. They are what separate a headline from a finding.

1. Is it a randomized trial or an observation? The creatine meta-analysis cited in this blog includes only randomized trials, with a minimum duration of 6 weeks and direct measures of hypertrophy (MRI, CT or ultrasound) [3]. The number of studies and the way of measuring are published precisely so it can be judged [3].

2. How many people, and how many were lost? Back to that meta-analysis: 10 studies and 44 results analyzed, all in healthy adults who did resistance training [3]. The pooled effect was 0.11, with a credibility interval of −0.02 to 0.25 [3]. An interval that includes zero is the first thing to look at in a meta-analysis, and it is exactly what many headlines do not mention.

3. Is the outcome measured the one in the headline? This site is especially careful here. That same analysis deliberately excluded total lean mass measures, such as DXA or bioelectrical impedance, because lean mass includes body water and can inflate the creatine result [3]. Had the meta-analysis relied on those measures, the effect would have looked larger. Measuring something different is not the same thing.

4. Does the effect size support the claim being made? Here is the most common pattern: a statistically significant effect can be very small. The difference between “there was a difference” and “the difference matters to me” is not automatic, and the two questions are answered separately.

What this will not settle for you

It is worth saying, because the easy mistake is turning these checklists into a shopping list:

  • A systematic review does not stop having bias if the studies it includes are low quality. The creatine meta-analysis relies on 10 studies; the steps one, on 15 cohorts with 47,471 participants [7]. They are different things, and there is methodological review documenting how badly the risk-of-bias tools are applied [5].
  • The methodological guideline does not make a study good. It only tells you whether it reported everything it did.
  • And they do not replace reading the methods abstract before the headline.

How this applies to “diet and health” studies

A pattern that repeats in nutrition research: almost everything associated with health-relevant outcomes is observational, for a practical reason: you cannot randomly assign the diet of thousands of people for years.

An example from this site itself: the systematic review on ultra-processed foods and metabolic disease [4]. It includes 15 studies: fourteen observational (nine cross-sectional and five prospective) and one randomized trial, with narrative synthesis instead of meta-analysis. It is observational design in what matters, and that has to be said.

That is why an association like “eating a lot of ultra-processed food is associated with more hepatic steatosis and with metabolic syndrome in part of the studies” [4] is not the same as “eating ultra-processed food causes disease”. That is the leap from a hypothesis to a causal fact.

What you take away

  1. CONSORT (today in its 2025 version) for randomized trials and STROBE for observational ones: they are checklists of what the study should have reported [1][2][6].
  2. Randomized trials and observations answer different questions, with different biases [1][2].
  3. Look at the interval: one that includes zero (like the creatine one, −0.02 to 0.25) tells a different story from the headline [3].
  4. Check that the outcome measured is the one in the headline: the creatine meta-analysis deliberately excluded total lean mass measures, which include water and distort the effect [3].
  5. An observational result is an association, not a cause [4].

Next time you see a health headline, these questions will serve you better than the headline itself.

More in: the practical nutrition guide, with the figures and sources from the Nutrition section.

Related articles

Sources for this article (7) 7 references
  1. Schulz KF, Altman DG, Moher D; CONSORT Group (2010) CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMJ 340:c332. See source (opens in a new tab) DOI: 10.1136/bmj.c332 Guía de notificación de ensayos paralelos (CONSORT): checklist para saber si un ensayo clínico es fiable Licence: Artículo de acceso abierto en PMC (BMJ, CC BY 4.0). Se cita con enlace. Consulted:
  2. von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP (2007) Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. BMJ 335:806-808. See source (opens in a new tab) DOI: 10.1136/bmj.39335.541782.AD Guía de notificación de estudios observacionales (STROBE): sirve para juzgar si un estudio de nutrición cuenta todo lo que hizo Licence: Artículo de acceso abierto en PMC (BMJ, CC BY 4.0). Se cita con enlace. Consulted:
  3. Burke R, Piñero A, Coleman M, et al. (2023) The Effects of Creatine Supplementation Combined with Resistance Training on Regional Measures of Muscle Hypertrophy: A Systematic Review with Meta-Analysis. Nutrients 15(9):2116. See source (opens in a new tab) DOI: 10.3390/nu15092116 Revisión sistemática y metaanálisis (10 estudios, ≥6 semanas,RM/TC/US): efecto muy pequeño de la creatina sobre hipertrofia medida por imagen Licence: Artículo de acceso abierto en PMC (MDPI, CC BY 4.0). Se cita con enlace. Consulted:
  4. Grinshpan LS, Eilat-Adar S, Ivancovsky-Wajcman D, Kariv R, Tirosh I (2024) Ultra-processed food consumption and non-alcoholic fatty liver disease, metabolic syndrome and insulin resistance: A systematic review. Journal of Hepatology Reports 13(1):100964. See source (opens in a new tab) DOI: 10.1016/j.jhepr.2023.100964 Revisión sistemática (ultraprocesados y enfermedad metabólica) Licence: Artículo de acceso abierto (texto completo en PubMed Central, CC BY-NC-ND 4.0). Se cita con enlace. Consulted:
  5. Igelström E, Campbell M, Craig P, Katikireddi SV (2021) Cochrane's risk of bias tool for non-randomized studies (ROBINS-I) is frequently misapplied: A methodological systematic review. Journal of Clinical Epidemiology 140:22-32. See source (opens in a new tab) DOI: 10.1016/j.jclinepi.2021.08.022 Revisión sistemática metodológica (124 revisiones) sobre el uso incorrecto de la herramienta ROBINS-I de riesgo de sesgo Licence: Artículo de revista con derechos de autor. Se cita con enlace al editor; no se reproduce texto salvo cita breve verificable. Consulted:
  6. Hopewell S, Chan AW, Collins GS, Hróbjartsson A, Moher D, Schulz KF, et al.; CONSORT Group (2025) CONSORT 2025 statement: updated guideline for reporting randomised trials. BMJ 389:e081123. See source (opens in a new tab) DOI: 10.1136/bmj-2024-081123 Versión vigente de la guía de notificación de ensayos aleatorizados (CONSORT); sustituye a la de 2010 Licence: Artículo de acceso abierto en PMC (BMJ). Se cita con enlace. Consulted:
  7. Paluch AE, Bajpai S, Bassett DR, et al. (2022) Daily steps and all-cause mortality: a meta-analysis of 15 international cohorts. The Lancet Public Health 7(3):e219-e228. See source (opens in a new tab) DOI: 10.1016/S2468-2667(21)00302-9 Metaanálisis de 15 cohortes (47.471 adultos): el riesgo baja hasta 6.000-8.000 pasos/día en mayores de 60 y 8.000-10.000 en menores de 60; el beneficio se satura según la edad Licence: Artículo de acceso abierto en PMC (Elsevier, The Lancet Public Health). Se cita con enlace. Consulted:

How to cite

Pan y Hierro (2026). How to read a nutrition study without getting fooled. https://panyhierro.com/en/blog/how-to-read-a-study/. Updated October 12, 2026.